2023-06-22T07:35:14.726 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-22T07:35:14.727 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --name rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi --verbose --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116 -- /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/orig.config.yaml 2023-06-22T07:35:14.782 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T07:35:14.886 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116 branch: wip-rishabh-improvements-authmon description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7311116' kernel: client: branch: testing flavor: default kdb: true sha1: 1c622d92b6f18ce44a7c33802686ff61c3c273ab ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-rishabh-improvements-authmon ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: 8627af3e0adcb765a3c249fcc209cba9f4873e1b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 8627af3e0adcb765a3c249fcc209cba9f4873e1b kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-improvements-authmon sha1: 8627af3e0adcb765a3c249fcc209cba9f4873e1b owner: scheduled_rishabh@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 sha1: 8627af3e0adcb765a3c249fcc209cba9f4873e1b sleep_before_teardown: 0 suite: fs suite_branch: wip-rishabh-improvements-authmon suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 8627af3e0adcb765a3c249fcc209cba9f4873e1b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 076bbebc42a14f7d568aaa78eabb0038327bcb23 timestamp: 2023-06-21_23:38:17 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-06-22T07:35:14.886 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa; will attempt to use it 2023-06-22T07:35:14.887 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks 2023-06-22T07:35:14.887 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-06-22T07:35:14.890 INFO:teuthology.task.internal:Checking packages... 2023-06-22T07:35:14.932 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '8627af3e0adcb765a3c249fcc209cba9f4873e1b' 2023-06-22T07:35:14.933 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-06-22T07:35:14.933 INFO:teuthology.packaging:ref: None 2023-06-22T07:35:14.933 INFO:teuthology.packaging:tag: None 2023-06-22T07:35:14.934 INFO:teuthology.packaging:branch: wip-rishabh-improvements-authmon 2023-06-22T07:35:14.934 INFO:teuthology.packaging:sha1: 8627af3e0adcb765a3c249fcc209cba9f4873e1b 2023-06-22T07:35:14.934 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-rishabh-improvements-authmon 2023-06-22T07:35:15.071 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4479.g8627af3e 2023-06-22T07:35:15.071 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-06-22T07:35:15.078 INFO:teuthology.task.internal:no buildpackages task found 2023-06-22T07:35:15.078 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-06-22T07:35:15.091 INFO:teuthology.task.internal:Saving configuration 2023-06-22T07:35:15.113 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-06-22T07:35:15.128 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T07:35:15.156 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 07:29:00.887135', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-06-22T07:35:15.181 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 07:29:00.888702', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-06-22T07:35:15.181 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-06-22T07:35:15.193 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-06-22T07:35:15.194 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-06-22T07:35:15.194 INFO:teuthology.run_tasks:Running task console_log... 2023-06-22T07:35:15.288 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2255f7f040>, signals=[15]) 2023-06-22T07:35:15.289 INFO:teuthology.run_tasks:Running task internal.connect... 2023-06-22T07:35:15.300 INFO:teuthology.task.internal:Opening connections... 2023-06-22T07:35:15.301 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T07:35:15.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T07:35:15.373 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T07:35:15.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T07:35:15.442 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-06-22T07:35:15.452 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2023-06-22T07:35:15.473 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2023-06-22T07:35:15.474 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2023-06-22T07:35:15.535 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="Red Hat Enterprise Linux" 2023-06-22T07:35:15.535 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="8.6 (Ootpa)" 2023-06-22T07:35:15.536 INFO:teuthology.orchestra.run.smithi023.stdout:ID="rhel" 2023-06-22T07:35:15.536 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="fedora" 2023-06-22T07:35:15.536 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="8.6" 2023-06-22T07:35:15.536 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el8" 2023-06-22T07:35:15.537 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-06-22T07:35:15.537 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2023-06-22T07:35:15.537 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-06-22T07:35:15.537 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://www.redhat.com/" 2023-06-22T07:35:15.537 INFO:teuthology.orchestra.run.smithi023.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-22T07:35:15.537 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T07:35:15.537 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:35:15.537 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T07:35:15.538 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-06-22T07:35:15.538 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-22T07:35:15.538 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-06-22T07:35:15.540 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2023-06-22T07:35:15.565 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2023-06-22T07:35:15.587 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2023-06-22T07:35:15.587 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2023-06-22T07:35:15.648 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Red Hat Enterprise Linux" 2023-06-22T07:35:15.648 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="8.6 (Ootpa)" 2023-06-22T07:35:15.649 INFO:teuthology.orchestra.run.smithi067.stdout:ID="rhel" 2023-06-22T07:35:15.649 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="fedora" 2023-06-22T07:35:15.649 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="8.6" 2023-06-22T07:35:15.649 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el8" 2023-06-22T07:35:15.649 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-06-22T07:35:15.649 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2023-06-22T07:35:15.649 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-06-22T07:35:15.650 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.redhat.com/" 2023-06-22T07:35:15.650 INFO:teuthology.orchestra.run.smithi067.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-22T07:35:15.650 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T07:35:15.650 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:35:15.650 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T07:35:15.650 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-06-22T07:35:15.651 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-22T07:35:15.651 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-06-22T07:35:15.651 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2023-06-22T07:35:15.673 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-06-22T07:35:15.690 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-06-22T07:35:15.750 INFO:teuthology.task.internal:Checking for old test directory... 2023-06-22T07:35:15.751 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2023-06-22T07:35:15.754 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2023-06-22T07:35:15.773 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-06-22T07:35:15.785 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-06-22T07:35:15.785 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2023-06-22T07:35:15.816 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2023-06-22T07:35:15.848 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-06-22T07:35:15.915 INFO:teuthology.run_tasks:Running task kernel... 2023-06-22T07:35:15.935 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1c622d92b6f18ce44a7c33802686ff61c3c273ab'} 2023-06-22T07:35:15.936 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-06-22T07:35:15.936 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-06-22T07:35:15.936 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-22T07:35:15.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T07:35:15.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-06-22T07:35:16.116 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1c622d92b6f18ce44a7c33802686ff61c3c273ab 2023-06-22T07:35:16.116 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1c622d92b6f18ce44a7c33802686ff61c3c273ab"... 2023-06-22T07:35:16.117 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2023-06-22T07:35:16.138 INFO:teuthology.orchestra.run.smithi023.stdout:4.18.0-372.9.1.el8.x86_64 2023-06-22T07:35:16.139 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs 1c622d92b6f18ce44a7c33802686ff61c3c273ab 2023-06-22T07:35:16.139 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-06-22T07:35:16.140 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum upgrade -y linux-firmware 2023-06-22T07:35:17.284 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2023-06-22T07:35:17.285 INFO:teuthology.orchestra.run.smithi023.stdout:Unable to read consumer identity 2023-06-22T07:35:21.161 INFO:teuthology.orchestra.run.smithi023.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2023-06-22T07:35:21.870 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2023-06-22T07:35:25.479 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 469 kB/s | 24 kB 00:00 2023-06-22T07:35:26.246 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-06-22T07:35:26.248 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-06-22T07:35:26.248 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-06-22T07:35:26.378 INFO:teuthology.task.kernel:Downloading kernel 1c622d92b6f18ce44a7c33802686ff61c3c273ab on client.0... 2023-06-22T07:35:26.379 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1c622d92b6f18ce44a7c33802686ff61c3c273ab 2023-06-22T07:35:26.521 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/1c622d92b6f18ce44a7c33802686ff61c3c273ab/centos/8/flavors/default/x86_64/ 2023-06-22T07:35:26.522 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.4.0_rc5_g1c622d92b6f1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/1c622d92b6f18ce44a7c33802686ff61c3c273ab/centos/8/flavors/default/x86_64/ --input-file=- 2023-06-22T07:35:26.524 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi023.front.sepia.ceph.com to complete... 2023-06-22T07:35:39.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22 07:35:39 URL:https://2.chacra.ceph.com/r/kernel/testing/1c622d92b6f18ce44a7c33802686ff61c3c273ab/centos/8/flavors/default/x86_64/kernel-6.4.0_rc5_g1c622d92b6f1-1.x86_64.rpm [379883408/379883408] -> "/tmp/kernel.x86_64.rpm" [1] 2023-06-22T07:35:39.667 INFO:teuthology.orchestra.run.smithi023.stderr:FINISHED --2023-06-22 07:35:39-- 2023-06-22T07:35:39.667 INFO:teuthology.orchestra.run.smithi023.stderr:Total wall clock time: 13s 2023-06-22T07:35:39.667 INFO:teuthology.orchestra.run.smithi023.stderr:Downloaded: 1 files, 362M in 13s (28.3 MB/s) 2023-06-22T07:35:39.669 INFO:teuthology.task.kernel:Installing kernel 1c622d92b6f18ce44a7c33802686ff61c3c273ab on client.0... 2023-06-22T07:35:39.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-06-22T07:35:41.301 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying... ######################################## 2023-06-22T07:35:41.351 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing... ######################################## 2023-06-22T07:35:41.437 INFO:teuthology.orchestra.run.smithi023.stdout:Updating / installing... 2023-06-22T07:36:21.788 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-6.4.0_rc5_g1c622d92b6f1-1 ######################################## 2023-06-22T07:36:49.202 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-06-22T07:36:49.203 DEBUG:teuthology.orchestra.run.smithi023:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-06-22T07:36:49.235 INFO:teuthology.orchestra.run.smithi023.stdout:/boot/System.map-6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:36:49.235 INFO:teuthology.orchestra.run.smithi023.stdout:/boot/config-6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:36:49.235 INFO:teuthology.orchestra.run.smithi023.stdout:/boot/vmlinuz-6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:36:49.235 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:36:49.235 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel 2023-06-22T07:36:49.236 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch 2023-06-22T07:36:49.236 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86 2023-06-22T07:36:49.236 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto 2023-06-22T07:36:49.236 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-06-22T07:36:49.236 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-06-22T07:36:49.236 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-06-22T07:36:49.236 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-06-22T07:36:49.237 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-06-22T07:36:49.237 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-06-22T07:36:49.237 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-06-22T07:36:49.237 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/crc32c-intel.ko 2023-06-22T07:36:49.237 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-06-22T07:36:49.237 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-06-22T07:36:49.237 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-06-22T07:36:49.237 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-06-22T07:36:49.238 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/serpent-avx2.ko 2023-06-22T07:36:49.238 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-06-22T07:36:49.238 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-06-22T07:36:49.238 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-06-22T07:36:49.238 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-06-22T07:36:49.238 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-06-22T07:36:49.238 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-06-22T07:36:49.239 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-06-22T07:36:49.239 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/kernel 2023-06-22T07:36:49.239 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/kernel/cpu 2023-06-22T07:36:49.239 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/kernel/cpu/mce 2023-06-22T07:36:49.239 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-06-22T07:36:49.239 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/kvm 2023-06-22T07:36:49.239 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/kvm/kvm-amd.ko 2023-06-22T07:36:49.239 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/kvm/kvm-intel.ko 2023-06-22T07:36:49.240 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/arch/x86/kvm/kvm.ko 2023-06-22T07:36:49.240 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto 2023-06-22T07:36:49.240 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/ansi_cprng.ko 2023-06-22T07:36:49.240 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/anubis.ko 2023-06-22T07:36:49.240 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/arc4.ko 2023-06-22T07:36:49.240 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/async_tx 2023-06-22T07:36:49.240 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/async_tx/async_memcpy.ko 2023-06-22T07:36:49.241 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/async_tx/async_pq.ko 2023-06-22T07:36:49.241 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/async_tx/async_raid6_recov.ko 2023-06-22T07:36:49.241 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/async_tx/async_tx.ko 2023-06-22T07:36:49.241 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/async_tx/async_xor.ko 2023-06-22T07:36:49.241 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/async_tx/raid6test.ko 2023-06-22T07:36:49.241 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/authenc.ko 2023-06-22T07:36:49.241 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/authencesn.ko 2023-06-22T07:36:49.241 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/blake2b_generic.ko 2023-06-22T07:36:49.242 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/blowfish_common.ko 2023-06-22T07:36:49.242 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/blowfish_generic.ko 2023-06-22T07:36:49.242 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/camellia_generic.ko 2023-06-22T07:36:49.242 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/cast5_generic.ko 2023-06-22T07:36:49.242 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/cast6_generic.ko 2023-06-22T07:36:49.242 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/cast_common.ko 2023-06-22T07:36:49.242 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/ccm.ko 2023-06-22T07:36:49.243 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/cmac.ko 2023-06-22T07:36:49.243 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/crc32_generic.ko 2023-06-22T07:36:49.243 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/crypto_engine.ko 2023-06-22T07:36:49.243 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/crypto_user.ko 2023-06-22T07:36:49.243 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/cts.ko 2023-06-22T07:36:49.243 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/des_generic.ko 2023-06-22T07:36:49.243 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/dh_generic.ko 2023-06-22T07:36:49.243 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/echainiv.ko 2023-06-22T07:36:49.244 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/essiv.ko 2023-06-22T07:36:49.244 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/fcrypt.ko 2023-06-22T07:36:49.244 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/khazad.ko 2023-06-22T07:36:49.244 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/lz4.ko 2023-06-22T07:36:49.244 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/lz4hc.ko 2023-06-22T07:36:49.244 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/md4.ko 2023-06-22T07:36:49.244 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/michael_mic.ko 2023-06-22T07:36:49.245 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/pcbc.ko 2023-06-22T07:36:49.245 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/pcrypt.ko 2023-06-22T07:36:49.245 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/rmd160.ko 2023-06-22T07:36:49.245 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/seed.ko 2023-06-22T07:36:49.245 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/serpent_generic.ko 2023-06-22T07:36:49.245 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/tcrypt.ko 2023-06-22T07:36:49.245 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/tea.ko 2023-06-22T07:36:49.245 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/twofish_common.ko 2023-06-22T07:36:49.246 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/twofish_generic.ko 2023-06-22T07:36:49.246 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/vmac.ko 2023-06-22T07:36:49.246 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/wp512.ko 2023-06-22T07:36:49.246 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/xcbc.ko 2023-06-22T07:36:49.246 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/xor.ko 2023-06-22T07:36:49.246 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/crypto/xxhash_generic.ko 2023-06-22T07:36:49.246 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers 2023-06-22T07:36:49.247 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/acpi 2023-06-22T07:36:49.247 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/acpi/acpi_ipmi.ko 2023-06-22T07:36:49.247 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/acpi/acpi_pad.ko 2023-06-22T07:36:49.247 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/acpi/custom_method.ko 2023-06-22T07:36:49.247 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/acpi/ec_sys.ko 2023-06-22T07:36:49.247 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/acpi/platform_profile.ko 2023-06-22T07:36:49.247 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/acpi/sbs.ko 2023-06-22T07:36:49.248 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/acpi/sbshc.ko 2023-06-22T07:36:49.248 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/acpi/video.ko 2023-06-22T07:36:49.248 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata 2023-06-22T07:36:49.248 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/acard-ahci.ko 2023-06-22T07:36:49.248 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/ahci_platform.ko 2023-06-22T07:36:49.248 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/ata_generic.ko 2023-06-22T07:36:49.248 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/libahci_platform.ko 2023-06-22T07:36:49.248 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_acpi.ko 2023-06-22T07:36:49.249 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_ali.ko 2023-06-22T07:36:49.249 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_amd.ko 2023-06-22T07:36:49.249 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_artop.ko 2023-06-22T07:36:49.249 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_atiixp.ko 2023-06-22T07:36:49.249 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_atp867x.ko 2023-06-22T07:36:49.249 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_cmd640.ko 2023-06-22T07:36:49.249 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_cmd64x.ko 2023-06-22T07:36:49.250 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_cypress.ko 2023-06-22T07:36:49.250 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_efar.ko 2023-06-22T07:36:49.250 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_hpt366.ko 2023-06-22T07:36:49.250 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_hpt37x.ko 2023-06-22T07:36:49.250 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_hpt3x2n.ko 2023-06-22T07:36:49.250 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_hpt3x3.ko 2023-06-22T07:36:49.250 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_it8213.ko 2023-06-22T07:36:49.250 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_it821x.ko 2023-06-22T07:36:49.251 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_jmicron.ko 2023-06-22T07:36:49.251 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_marvell.ko 2023-06-22T07:36:49.251 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_mpiix.ko 2023-06-22T07:36:49.251 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_netcell.ko 2023-06-22T07:36:49.251 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_ninja32.ko 2023-06-22T07:36:49.251 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_ns87410.ko 2023-06-22T07:36:49.251 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_ns87415.ko 2023-06-22T07:36:49.252 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_oldpiix.ko 2023-06-22T07:36:49.252 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_opti.ko 2023-06-22T07:36:49.252 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_optidma.ko 2023-06-22T07:36:49.252 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_pcmcia.ko 2023-06-22T07:36:49.252 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_pdc2027x.ko 2023-06-22T07:36:49.252 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-06-22T07:36:49.252 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_piccolo.ko 2023-06-22T07:36:49.252 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_rdc.ko 2023-06-22T07:36:49.253 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_sch.ko 2023-06-22T07:36:49.253 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_serverworks.ko 2023-06-22T07:36:49.253 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_sil680.ko 2023-06-22T07:36:49.253 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_sis.ko 2023-06-22T07:36:49.253 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_sl82c105.ko 2023-06-22T07:36:49.254 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_triflex.ko 2023-06-22T07:36:49.254 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pata_via.ko 2023-06-22T07:36:49.254 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/pdc_adma.ko 2023-06-22T07:36:49.254 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_inic162x.ko 2023-06-22T07:36:49.254 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_mv.ko 2023-06-22T07:36:49.254 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_nv.ko 2023-06-22T07:36:49.254 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_promise.ko 2023-06-22T07:36:49.254 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_qstor.ko 2023-06-22T07:36:49.255 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_sil.ko 2023-06-22T07:36:49.255 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_sil24.ko 2023-06-22T07:36:49.255 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_sis.ko 2023-06-22T07:36:49.255 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_svw.ko 2023-06-22T07:36:49.255 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_sx4.ko 2023-06-22T07:36:49.255 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_uli.ko 2023-06-22T07:36:49.255 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_via.ko 2023-06-22T07:36:49.256 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ata/sata_vsc.ko 2023-06-22T07:36:49.256 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/auxdisplay 2023-06-22T07:36:49.256 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/auxdisplay/ks0108.ko 2023-06-22T07:36:49.256 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/base 2023-06-22T07:36:49.256 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/base/regmap 2023-06-22T07:36:49.256 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/base/regmap/regmap-i2c.ko 2023-06-22T07:36:49.256 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/bcma 2023-06-22T07:36:49.257 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/bcma/bcma.ko 2023-06-22T07:36:49.257 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block 2023-06-22T07:36:49.257 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/aoe 2023-06-22T07:36:49.257 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/aoe/aoe.ko 2023-06-22T07:36:49.257 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/brd.ko 2023-06-22T07:36:49.257 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/drbd 2023-06-22T07:36:49.257 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/drbd/drbd.ko 2023-06-22T07:36:49.257 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/floppy.ko 2023-06-22T07:36:49.258 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/loop.ko 2023-06-22T07:36:49.258 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/mtip32xx 2023-06-22T07:36:49.258 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-06-22T07:36:49.258 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/nbd.ko 2023-06-22T07:36:49.258 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/null_blk 2023-06-22T07:36:49.258 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/null_blk/null_blk.ko 2023-06-22T07:36:49.258 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/pktcdvd.ko 2023-06-22T07:36:49.259 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/rbd.ko 2023-06-22T07:36:49.259 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/virtio_blk.ko 2023-06-22T07:36:49.259 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/xen-blkback 2023-06-22T07:36:49.259 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-06-22T07:36:49.259 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/xen-blkfront.ko 2023-06-22T07:36:49.259 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/zram 2023-06-22T07:36:49.259 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/block/zram/zram.ko 2023-06-22T07:36:49.259 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char 2023-06-22T07:36:49.260 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/hangcheck-timer.ko 2023-06-22T07:36:49.260 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/hw_random 2023-06-22T07:36:49.260 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/hw_random/amd-rng.ko 2023-06-22T07:36:49.260 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/hw_random/intel-rng.ko 2023-06-22T07:36:49.260 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-06-22T07:36:49.260 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/hw_random/via-rng.ko 2023-06-22T07:36:49.260 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/hw_random/virtio-rng.ko 2023-06-22T07:36:49.261 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/ipmi 2023-06-22T07:36:49.261 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-06-22T07:36:49.261 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-06-22T07:36:49.261 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-06-22T07:36:49.261 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/ipmi/ipmi_si.ko 2023-06-22T07:36:49.261 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-06-22T07:36:49.261 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-06-22T07:36:49.261 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/lp.ko 2023-06-22T07:36:49.262 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/mwave 2023-06-22T07:36:49.262 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/mwave/mwave.ko 2023-06-22T07:36:49.262 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/ppdev.ko 2023-06-22T07:36:49.262 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/tlclk.ko 2023-06-22T07:36:49.262 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/tpm 2023-06-22T07:36:49.262 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/tpm/tpm.ko 2023-06-22T07:36:49.262 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/tpm/tpm_atmel.ko 2023-06-22T07:36:49.263 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/tpm/tpm_infineon.ko 2023-06-22T07:36:49.263 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/tpm/tpm_nsc.ko 2023-06-22T07:36:49.263 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/tpm/tpm_tis.ko 2023-06-22T07:36:49.263 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-06-22T07:36:49.263 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/uv_mmtimer.ko 2023-06-22T07:36:49.263 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/char/virtio_console.ko 2023-06-22T07:36:49.263 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/cpufreq 2023-06-22T07:36:49.263 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-06-22T07:36:49.264 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-06-22T07:36:49.264 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/cpufreq/p4-clockmod.ko 2023-06-22T07:36:49.264 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-06-22T07:36:49.264 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/cpufreq/powernow-k8.ko 2023-06-22T07:36:49.264 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/cpufreq/speedstep-lib.ko 2023-06-22T07:36:49.264 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto 2023-06-22T07:36:49.264 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/ccp 2023-06-22T07:36:49.265 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-06-22T07:36:49.265 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/ccp/ccp.ko 2023-06-22T07:36:49.265 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/intel 2023-06-22T07:36:49.265 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/intel/qat 2023-06-22T07:36:49.265 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/intel/qat/qat_common 2023-06-22T07:36:49.265 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-06-22T07:36:49.265 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-06-22T07:36:49.266 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-06-22T07:36:49.266 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/padlock-aes.ko 2023-06-22T07:36:49.266 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/padlock-sha.ko 2023-06-22T07:36:49.266 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/virtio 2023-06-22T07:36:49.266 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-06-22T07:36:49.266 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/dca 2023-06-22T07:36:49.266 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/dca/dca.ko 2023-06-22T07:36:49.266 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/devfreq 2023-06-22T07:36:49.267 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-06-22T07:36:49.267 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/dma 2023-06-22T07:36:49.267 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/dma/dw 2023-06-22T07:36:49.267 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/dma/dw/dw_dmac.ko 2023-06-22T07:36:49.267 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/dma/ioat 2023-06-22T07:36:49.267 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/dma/ioat/ioatdma.ko 2023-06-22T07:36:49.267 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac 2023-06-22T07:36:49.268 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/amd64_edac.ko 2023-06-22T07:36:49.268 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/e752x_edac.ko 2023-06-22T07:36:49.268 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/edac_mce_amd.ko 2023-06-22T07:36:49.268 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/i3000_edac.ko 2023-06-22T07:36:49.268 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/i3200_edac.ko 2023-06-22T07:36:49.268 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/i5100_edac.ko 2023-06-22T07:36:49.268 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/i5400_edac.ko 2023-06-22T07:36:49.268 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/i7300_edac.ko 2023-06-22T07:36:49.269 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/i7core_edac.ko 2023-06-22T07:36:49.269 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/i82975x_edac.ko 2023-06-22T07:36:49.269 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/ie31200_edac.ko 2023-06-22T07:36:49.269 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/sb_edac.ko 2023-06-22T07:36:49.269 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/edac/x38_edac.ko 2023-06-22T07:36:49.269 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/firmware 2023-06-22T07:36:49.269 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/firmware/edd.ko 2023-06-22T07:36:49.270 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/firmware/iscsi_ibft.ko 2023-06-22T07:36:49.270 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/gpio 2023-06-22T07:36:49.270 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/gpio/gpio-ich.ko 2023-06-22T07:36:49.270 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/gpio/gpio-viperboard.ko 2023-06-22T07:36:49.270 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid 2023-06-22T07:36:49.270 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-appleir.ko 2023-06-22T07:36:49.270 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-aureal.ko 2023-06-22T07:36:49.270 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-axff.ko 2023-06-22T07:36:49.271 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-dr.ko 2023-06-22T07:36:49.271 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-elecom.ko 2023-06-22T07:36:49.271 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-elo.ko 2023-06-22T07:36:49.271 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-emsff.ko 2023-06-22T07:36:49.271 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-gaff.ko 2023-06-22T07:36:49.271 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-gt683r.ko 2023-06-22T07:36:49.271 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-gyration.ko 2023-06-22T07:36:49.272 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-holtek-kbd.ko 2023-06-22T07:36:49.272 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-holtek-mouse.ko 2023-06-22T07:36:49.272 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-holtekff.ko 2023-06-22T07:36:49.272 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-hyperv.ko 2023-06-22T07:36:49.272 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-icade.ko 2023-06-22T07:36:49.272 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-keytouch.ko 2023-06-22T07:36:49.272 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-kye.ko 2023-06-22T07:36:49.272 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-lcpower.ko 2023-06-22T07:36:49.273 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-led.ko 2023-06-22T07:36:49.273 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-lenovo.ko 2023-06-22T07:36:49.273 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-logitech-dj.ko 2023-06-22T07:36:49.273 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-06-22T07:36:49.273 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-multitouch.ko 2023-06-22T07:36:49.273 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-ortek.ko 2023-06-22T07:36:49.273 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-penmount.ko 2023-06-22T07:36:49.273 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-petalynx.ko 2023-06-22T07:36:49.274 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-picolcd.ko 2023-06-22T07:36:49.274 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-pl.ko 2023-06-22T07:36:49.274 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-plantronics.ko 2023-06-22T07:36:49.274 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-primax.ko 2023-06-22T07:36:49.274 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-rmi.ko 2023-06-22T07:36:49.274 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-arvo.ko 2023-06-22T07:36:49.274 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-common.ko 2023-06-22T07:36:49.274 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-isku.ko 2023-06-22T07:36:49.275 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-kone.ko 2023-06-22T07:36:49.275 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-06-22T07:36:49.275 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-konepure.ko 2023-06-22T07:36:49.275 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-06-22T07:36:49.275 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-lua.ko 2023-06-22T07:36:49.275 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-pyra.ko 2023-06-22T07:36:49.275 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-ryos.ko 2023-06-22T07:36:49.276 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat-savu.ko 2023-06-22T07:36:49.276 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-roccat.ko 2023-06-22T07:36:49.276 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-saitek.ko 2023-06-22T07:36:49.276 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-samsung.ko 2023-06-22T07:36:49.276 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-sensor-hub.ko 2023-06-22T07:36:49.276 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-sjoy.ko 2023-06-22T07:36:49.276 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-sony.ko 2023-06-22T07:36:49.277 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-speedlink.ko 2023-06-22T07:36:49.277 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-steelseries.ko 2023-06-22T07:36:49.277 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-sunplus.ko 2023-06-22T07:36:49.277 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-thrustmaster.ko 2023-06-22T07:36:49.292 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-tivo.ko 2023-06-22T07:36:49.293 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-tmff.ko 2023-06-22T07:36:49.293 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-topseed.ko 2023-06-22T07:36:49.293 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-twinhan.ko 2023-06-22T07:36:49.293 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-uclogic.ko 2023-06-22T07:36:49.293 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-waltop.ko 2023-06-22T07:36:49.293 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-wiimote.ko 2023-06-22T07:36:49.294 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-xinmo.ko 2023-06-22T07:36:49.294 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-zpff.ko 2023-06-22T07:36:49.294 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/hid-zydacron.ko 2023-06-22T07:36:49.294 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/uhid.ko 2023-06-22T07:36:49.294 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hid/wacom.ko 2023-06-22T07:36:49.294 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hv 2023-06-22T07:36:49.294 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hv/hv_balloon.ko 2023-06-22T07:36:49.295 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hv/hv_utils.ko 2023-06-22T07:36:49.295 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hv/hv_vmbus.ko 2023-06-22T07:36:49.295 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon 2023-06-22T07:36:49.295 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/abituguru.ko 2023-06-22T07:36:49.295 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/abituguru3.ko 2023-06-22T07:36:49.295 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/acpi_power_meter.ko 2023-06-22T07:36:49.295 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ad7414.ko 2023-06-22T07:36:49.295 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ad7418.ko 2023-06-22T07:36:49.296 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adc128d818.ko 2023-06-22T07:36:49.296 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adm1025.ko 2023-06-22T07:36:49.296 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adm1026.ko 2023-06-22T07:36:49.296 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adm1029.ko 2023-06-22T07:36:49.296 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adm1031.ko 2023-06-22T07:36:49.296 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adm9240.ko 2023-06-22T07:36:49.296 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ads7828.ko 2023-06-22T07:36:49.297 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adt7410.ko 2023-06-22T07:36:49.297 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adt7411.ko 2023-06-22T07:36:49.297 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adt7462.ko 2023-06-22T07:36:49.297 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adt7470.ko 2023-06-22T07:36:49.297 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adt7475.ko 2023-06-22T07:36:49.297 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/adt7x10.ko 2023-06-22T07:36:49.297 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/amc6821.ko 2023-06-22T07:36:49.297 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/applesmc.ko 2023-06-22T07:36:49.298 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/asb100.ko 2023-06-22T07:36:49.298 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/asc7621.ko 2023-06-22T07:36:49.298 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/asus_atk0110.ko 2023-06-22T07:36:49.298 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/atxp1.ko 2023-06-22T07:36:49.298 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/coretemp.ko 2023-06-22T07:36:49.298 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-06-22T07:36:49.298 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/dme1737.ko 2023-06-22T07:36:49.299 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ds1621.ko 2023-06-22T07:36:49.299 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ds620.ko 2023-06-22T07:36:49.299 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/emc1403.ko 2023-06-22T07:36:49.299 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/emc6w201.ko 2023-06-22T07:36:49.299 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/f71805f.ko 2023-06-22T07:36:49.299 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/f71882fg.ko 2023-06-22T07:36:49.299 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/f75375s.ko 2023-06-22T07:36:49.299 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/fam15h_power.ko 2023-06-22T07:36:49.300 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/fschmd.ko 2023-06-22T07:36:49.300 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/g760a.ko 2023-06-22T07:36:49.300 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/g762.ko 2023-06-22T07:36:49.300 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/gl518sm.ko 2023-06-22T07:36:49.300 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/gl520sm.ko 2023-06-22T07:36:49.300 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/hwmon-vid.ko 2023-06-22T07:36:49.300 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/i5500_temp.ko 2023-06-22T07:36:49.301 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/i5k_amb.ko 2023-06-22T07:36:49.301 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ibmaem.ko 2023-06-22T07:36:49.301 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ibmpex.ko 2023-06-22T07:36:49.301 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ina209.ko 2023-06-22T07:36:49.301 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ina2xx.ko 2023-06-22T07:36:49.301 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/it87.ko 2023-06-22T07:36:49.301 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/k10temp.ko 2023-06-22T07:36:49.301 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/k8temp.ko 2023-06-22T07:36:49.302 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lineage-pem.ko 2023-06-22T07:36:49.302 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm63.ko 2023-06-22T07:36:49.302 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm73.ko 2023-06-22T07:36:49.302 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm75.ko 2023-06-22T07:36:49.302 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm77.ko 2023-06-22T07:36:49.302 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm78.ko 2023-06-22T07:36:49.302 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm80.ko 2023-06-22T07:36:49.302 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm83.ko 2023-06-22T07:36:49.303 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm85.ko 2023-06-22T07:36:49.303 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm87.ko 2023-06-22T07:36:49.303 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm90.ko 2023-06-22T07:36:49.303 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm92.ko 2023-06-22T07:36:49.303 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm93.ko 2023-06-22T07:36:49.303 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm95234.ko 2023-06-22T07:36:49.303 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm95241.ko 2023-06-22T07:36:49.303 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/lm95245.ko 2023-06-22T07:36:49.304 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ltc2945.ko 2023-06-22T07:36:49.304 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ltc4151.ko 2023-06-22T07:36:49.304 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ltc4215.ko 2023-06-22T07:36:49.304 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ltc4222.ko 2023-06-22T07:36:49.304 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ltc4245.ko 2023-06-22T07:36:49.304 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ltc4260.ko 2023-06-22T07:36:49.304 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/ltc4261.ko 2023-06-22T07:36:49.305 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/max16065.ko 2023-06-22T07:36:49.305 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/max1619.ko 2023-06-22T07:36:49.305 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/max1668.ko 2023-06-22T07:36:49.305 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/max197.ko 2023-06-22T07:36:49.305 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/max6639.ko 2023-06-22T07:36:49.305 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/max6650.ko 2023-06-22T07:36:49.305 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/max6697.ko 2023-06-22T07:36:49.305 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/mcp3021.ko 2023-06-22T07:36:49.306 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/nct6683.ko 2023-06-22T07:36:49.306 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/nct6775-core.ko 2023-06-22T07:36:49.306 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/nct6775.ko 2023-06-22T07:36:49.306 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/nct7802.ko 2023-06-22T07:36:49.306 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pc87360.ko 2023-06-22T07:36:49.306 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pc87427.ko 2023-06-22T07:36:49.306 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pcf8591.ko 2023-06-22T07:36:49.306 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus 2023-06-22T07:36:49.307 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-06-22T07:36:49.307 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-06-22T07:36:49.307 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-06-22T07:36:49.307 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/max16064.ko 2023-06-22T07:36:49.307 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/max34440.ko 2023-06-22T07:36:49.307 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/max8688.ko 2023-06-22T07:36:49.307 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-06-22T07:36:49.307 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-06-22T07:36:49.308 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-06-22T07:36:49.308 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-06-22T07:36:49.308 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-06-22T07:36:49.308 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-06-22T07:36:49.308 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/powr1220.ko 2023-06-22T07:36:49.308 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/sch5627.ko 2023-06-22T07:36:49.308 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/sch5636.ko 2023-06-22T07:36:49.309 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/sch56xx-common.ko 2023-06-22T07:36:49.309 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/sht15.ko 2023-06-22T07:36:49.309 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/sht21.ko 2023-06-22T07:36:49.309 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/shtc1.ko 2023-06-22T07:36:49.309 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/sis5595.ko 2023-06-22T07:36:49.309 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/smsc47b397.ko 2023-06-22T07:36:49.309 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/smsc47m1.ko 2023-06-22T07:36:49.309 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/smsc47m192.ko 2023-06-22T07:36:49.310 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/thmc50.ko 2023-06-22T07:36:49.310 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/tmp102.ko 2023-06-22T07:36:49.310 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/tmp103.ko 2023-06-22T07:36:49.310 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/tmp401.ko 2023-06-22T07:36:49.310 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/tmp421.ko 2023-06-22T07:36:49.310 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/via-cputemp.ko 2023-06-22T07:36:49.310 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/via686a.ko 2023-06-22T07:36:49.310 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/vt1211.ko 2023-06-22T07:36:49.311 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/vt8231.ko 2023-06-22T07:36:49.311 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/w83627ehf.ko 2023-06-22T07:36:49.311 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/w83627hf.ko 2023-06-22T07:36:49.311 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/w83781d.ko 2023-06-22T07:36:49.311 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/w83791d.ko 2023-06-22T07:36:49.311 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/w83792d.ko 2023-06-22T07:36:49.311 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/w83793.ko 2023-06-22T07:36:49.311 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/w83795.ko 2023-06-22T07:36:49.312 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/w83l785ts.ko 2023-06-22T07:36:49.312 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/hwmon/w83l786ng.ko 2023-06-22T07:36:49.312 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c 2023-06-22T07:36:49.312 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/algos 2023-06-22T07:36:49.312 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-06-22T07:36:49.312 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-06-22T07:36:49.312 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses 2023-06-22T07:36:49.313 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-06-22T07:36:49.313 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-06-22T07:36:49.313 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-06-22T07:36:49.313 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-06-22T07:36:49.313 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-06-22T07:36:49.313 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-06-22T07:36:49.313 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-06-22T07:36:49.313 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-06-22T07:36:49.314 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-i801.ko 2023-06-22T07:36:49.314 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-isch.ko 2023-06-22T07:36:49.314 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-06-22T07:36:49.314 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-06-22T07:36:49.314 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-06-22T07:36:49.314 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-parport.ko 2023-06-22T07:36:49.314 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-06-22T07:36:49.314 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-06-22T07:36:49.315 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-06-22T07:36:49.315 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-06-22T07:36:49.315 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-06-22T07:36:49.315 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-06-22T07:36:49.315 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-via.ko 2023-06-22T07:36:49.315 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-06-22T07:36:49.315 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-06-22T07:36:49.315 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/i2c-dev.ko 2023-06-22T07:36:49.316 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/i2c-mux.ko 2023-06-22T07:36:49.316 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-06-22T07:36:49.316 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/i2c-smbus.ko 2023-06-22T07:36:49.316 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/i2c/i2c-stub.ko 2023-06-22T07:36:49.316 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband 2023-06-22T07:36:49.316 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/core 2023-06-22T07:36:49.316 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/core/ib_cm.ko 2023-06-22T07:36:49.317 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/core/ib_core.ko 2023-06-22T07:36:49.317 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/core/ib_umad.ko 2023-06-22T07:36:49.317 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-06-22T07:36:49.317 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/core/iw_cm.ko 2023-06-22T07:36:49.317 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/core/rdma_cm.ko 2023-06-22T07:36:49.317 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-06-22T07:36:49.317 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/hw 2023-06-22T07:36:49.318 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/hw/cxgb4 2023-06-22T07:36:49.318 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-06-22T07:36:49.318 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/hw/mlx4 2023-06-22T07:36:49.318 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-06-22T07:36:49.318 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/hw/mlx5 2023-06-22T07:36:49.318 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-06-22T07:36:49.318 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/hw/mthca 2023-06-22T07:36:49.318 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-06-22T07:36:49.319 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp 2023-06-22T07:36:49.319 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp/ipoib 2023-06-22T07:36:49.319 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-06-22T07:36:49.319 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp/iser 2023-06-22T07:36:49.319 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-06-22T07:36:49.319 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp/isert 2023-06-22T07:36:49.319 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-06-22T07:36:49.320 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp/srp 2023-06-22T07:36:49.320 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-06-22T07:36:49.320 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp/srpt 2023-06-22T07:36:49.320 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-06-22T07:36:49.320 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input 2023-06-22T07:36:49.320 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/gameport 2023-06-22T07:36:49.320 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-06-22T07:36:49.320 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/gameport/fm801-gp.ko 2023-06-22T07:36:49.321 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/gameport/gameport.ko 2023-06-22T07:36:49.321 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/gameport/lightning.ko 2023-06-22T07:36:49.321 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/gameport/ns558.ko 2023-06-22T07:36:49.321 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/joydev.ko 2023-06-22T07:36:49.321 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/keyboard 2023-06-22T07:36:49.321 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/keyboard/gpio_keys.ko 2023-06-22T07:36:49.321 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/mouse 2023-06-22T07:36:49.321 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/mouse/appletouch.ko 2023-06-22T07:36:49.322 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/mouse/bcm5974.ko 2023-06-22T07:36:49.322 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/mouse/cyapatp.ko 2023-06-22T07:36:49.322 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/mouse/elan_i2c.ko 2023-06-22T07:36:49.322 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/mouse/sermouse.ko 2023-06-22T07:36:49.322 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-06-22T07:36:49.322 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/mouse/synaptics_usb.ko 2023-06-22T07:36:49.322 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/mouse/vsxxxaa.ko 2023-06-22T07:36:49.323 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/rmi4 2023-06-22T07:36:49.323 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/rmi4/rmi_core.ko 2023-06-22T07:36:49.323 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/serio 2023-06-22T07:36:49.323 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/serio/altera_ps2.ko 2023-06-22T07:36:49.323 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/serio/arc_ps2.ko 2023-06-22T07:36:49.323 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-06-22T07:36:49.323 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/serio/serio_raw.ko 2023-06-22T07:36:49.323 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/input/sparse-keymap.ko 2023-06-22T07:36:49.324 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/iommu 2023-06-22T07:36:49.324 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/iommu/amd 2023-06-22T07:36:49.324 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/iommu/amd/iommu_v2.ko 2023-06-22T07:36:49.324 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds 2023-06-22T07:36:49.324 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/leds-blinkm.ko 2023-06-22T07:36:49.324 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/leds-lm3530.ko 2023-06-22T07:36:49.324 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/leds-lp3944.ko 2023-06-22T07:36:49.325 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/leds-ss4200.ko 2023-06-22T07:36:49.325 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/trigger 2023-06-22T07:36:49.325 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-06-22T07:36:49.325 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-06-22T07:36:49.325 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-06-22T07:36:49.325 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-06-22T07:36:49.325 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-06-22T07:36:49.325 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-06-22T07:36:49.326 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-06-22T07:36:49.326 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-06-22T07:36:49.326 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md 2023-06-22T07:36:49.326 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/bcache 2023-06-22T07:36:49.326 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/bcache/bcache.ko 2023-06-22T07:36:49.326 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-bio-prison.ko 2023-06-22T07:36:49.326 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-cache-smq.ko 2023-06-22T07:36:49.326 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-cache.ko 2023-06-22T07:36:49.327 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-crypt.ko 2023-06-22T07:36:49.327 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-delay.ko 2023-06-22T07:36:49.327 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-flakey.ko 2023-06-22T07:36:49.327 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-log-userspace.ko 2023-06-22T07:36:49.327 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-multipath.ko 2023-06-22T07:36:49.327 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-queue-length.ko 2023-06-22T07:36:49.327 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-raid.ko 2023-06-22T07:36:49.328 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-round-robin.ko 2023-06-22T07:36:49.328 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-service-time.ko 2023-06-22T07:36:49.328 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-switch.ko 2023-06-22T07:36:49.328 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-thin-pool.ko 2023-06-22T07:36:49.328 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/dm-verity.ko 2023-06-22T07:36:49.328 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/faulty.ko 2023-06-22T07:36:49.328 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/linear.ko 2023-06-22T07:36:49.328 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/multipath.ko 2023-06-22T07:36:49.329 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/persistent-data 2023-06-22T07:36:49.329 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-06-22T07:36:49.329 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/raid0.ko 2023-06-22T07:36:49.329 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/raid1.ko 2023-06-22T07:36:49.329 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/raid10.ko 2023-06-22T07:36:49.329 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/md/raid456.ko 2023-06-22T07:36:49.329 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/message 2023-06-22T07:36:49.329 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/message/fusion 2023-06-22T07:36:49.330 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/message/fusion/mptbase.ko 2023-06-22T07:36:49.330 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/message/fusion/mptctl.ko 2023-06-22T07:36:49.330 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/message/fusion/mptfc.ko 2023-06-22T07:36:49.330 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/message/fusion/mptlan.ko 2023-06-22T07:36:49.330 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/message/fusion/mptsas.ko 2023-06-22T07:36:49.330 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/message/fusion/mptscsih.ko 2023-06-22T07:36:49.330 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/message/fusion/mptspi.ko 2023-06-22T07:36:49.330 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/mfd 2023-06-22T07:36:49.331 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/mfd/lpc_ich.ko 2023-06-22T07:36:49.331 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/mfd/lpc_sch.ko 2023-06-22T07:36:49.331 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/mfd/mfd-core.ko 2023-06-22T07:36:49.331 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/mfd/sm501.ko 2023-06-22T07:36:49.331 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/mfd/viperboard.ko 2023-06-22T07:36:49.331 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/mfd/vx855.ko 2023-06-22T07:36:49.331 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/mfd/wl1273-core.ko 2023-06-22T07:36:49.332 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc 2023-06-22T07:36:49.332 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/altera-stapl 2023-06-22T07:36:49.332 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-06-22T07:36:49.332 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/apds9802als.ko 2023-06-22T07:36:49.332 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/apds990x.ko 2023-06-22T07:36:49.332 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/bh1770glc.ko 2023-06-22T07:36:49.332 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/cb710 2023-06-22T07:36:49.332 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/cb710/cb710.ko 2023-06-22T07:36:49.333 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/eeprom 2023-06-22T07:36:49.333 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/eeprom/at24.ko 2023-06-22T07:36:49.333 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/eeprom/eeprom.ko 2023-06-22T07:36:49.333 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-06-22T07:36:49.333 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/eeprom/max6875.ko 2023-06-22T07:36:49.334 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/enclosure.ko 2023-06-22T07:36:49.334 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/hpilo.ko 2023-06-22T07:36:49.334 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/isl29003.ko 2023-06-22T07:36:49.334 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/isl29020.ko 2023-06-22T07:36:49.334 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/lis3lv02d 2023-06-22T07:36:49.334 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-06-22T07:36:49.334 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-06-22T07:36:49.334 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/mei 2023-06-22T07:36:49.335 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/mei/mei-me.ko 2023-06-22T07:36:49.335 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/mei/mei-txe.ko 2023-06-22T07:36:49.335 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/mei/mei.ko 2023-06-22T07:36:49.335 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/sgi-gru 2023-06-22T07:36:49.335 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/sgi-gru/gru.ko 2023-06-22T07:36:49.335 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/sgi-xp 2023-06-22T07:36:49.335 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/sgi-xp/xp.ko 2023-06-22T07:36:49.336 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/sgi-xp/xpc.ko 2023-06-22T07:36:49.336 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-06-22T07:36:49.336 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/tifm_7xx1.ko 2023-06-22T07:36:49.336 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/tifm_core.ko 2023-06-22T07:36:49.336 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/tsl2550.ko 2023-06-22T07:36:49.336 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/vmw_balloon.ko 2023-06-22T07:36:49.336 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/vmw_vmci 2023-06-22T07:36:49.337 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-06-22T07:36:49.337 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net 2023-06-22T07:36:49.337 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/bonding 2023-06-22T07:36:49.337 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/bonding/bonding.ko 2023-06-22T07:36:49.337 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/dsa 2023-06-22T07:36:49.337 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/dsa/b53 2023-06-22T07:36:49.337 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/dsa/b53/b53_common.ko 2023-06-22T07:36:49.337 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/dsa/bcm-sf2.ko 2023-06-22T07:36:49.338 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/dsa/mv88e6060.ko 2023-06-22T07:36:49.338 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/dsa/mv88e6xxx 2023-06-22T07:36:49.338 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-06-22T07:36:49.338 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/dummy.ko 2023-06-22T07:36:49.338 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/eql.ko 2023-06-22T07:36:49.338 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet 2023-06-22T07:36:49.338 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/3com 2023-06-22T07:36:49.339 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-06-22T07:36:49.339 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-06-22T07:36:49.339 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-06-22T07:36:49.339 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-06-22T07:36:49.339 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/8390 2023-06-22T07:36:49.339 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/8390/8390.ko 2023-06-22T07:36:49.339 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-06-22T07:36:49.339 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-06-22T07:36:49.340 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-06-22T07:36:49.340 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/adaptec 2023-06-22T07:36:49.340 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-06-22T07:36:49.340 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/agere 2023-06-22T07:36:49.340 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/agere/et131x.ko 2023-06-22T07:36:49.340 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/alteon 2023-06-22T07:36:49.340 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-06-22T07:36:49.340 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/altera 2023-06-22T07:36:49.341 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-06-22T07:36:49.341 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/amd 2023-06-22T07:36:49.341 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-06-22T07:36:49.341 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-06-22T07:36:49.341 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-06-22T07:36:49.341 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/atheros 2023-06-22T07:36:49.341 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/atheros/alx 2023-06-22T07:36:49.342 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-06-22T07:36:49.342 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/atheros/atl1c 2023-06-22T07:36:49.342 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-06-22T07:36:49.342 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/atheros/atl1e 2023-06-22T07:36:49.342 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-06-22T07:36:49.342 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/atheros/atlx 2023-06-22T07:36:49.342 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-06-22T07:36:49.342 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-06-22T07:36:49.343 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/broadcom 2023-06-22T07:36:49.343 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-06-22T07:36:49.343 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-06-22T07:36:49.343 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-06-22T07:36:49.343 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-06-22T07:36:49.343 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-06-22T07:36:49.343 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/broadcom/genet 2023-06-22T07:36:49.343 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-06-22T07:36:49.344 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-06-22T07:36:49.344 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/brocade 2023-06-22T07:36:49.344 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/brocade/bna 2023-06-22T07:36:49.344 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-06-22T07:36:49.344 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio 2023-06-22T07:36:49.344 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio/cxgb 2023-06-22T07:36:49.344 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-06-22T07:36:49.344 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-06-22T07:36:49.345 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-06-22T07:36:49.345 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-06-22T07:36:49.345 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-06-22T07:36:49.345 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-06-22T07:36:49.345 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-06-22T07:36:49.345 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-06-22T07:36:49.345 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-06-22T07:36:49.345 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/cisco 2023-06-22T07:36:49.346 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/cisco/enic 2023-06-22T07:36:49.346 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-06-22T07:36:49.346 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dec 2023-06-22T07:36:49.346 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dec/tulip 2023-06-22T07:36:49.346 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-06-22T07:36:49.346 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-06-22T07:36:49.346 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-06-22T07:36:49.346 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-06-22T07:36:49.347 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-06-22T07:36:49.347 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-06-22T07:36:49.347 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dlink 2023-06-22T07:36:49.347 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-06-22T07:36:49.347 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-06-22T07:36:49.347 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/dnet.ko 2023-06-22T07:36:49.347 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/emulex 2023-06-22T07:36:49.348 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/emulex/benet 2023-06-22T07:36:49.348 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-06-22T07:36:49.348 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/ethoc.ko 2023-06-22T07:36:49.348 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/fealnx.ko 2023-06-22T07:36:49.348 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel 2023-06-22T07:36:49.348 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/e100.ko 2023-06-22T07:36:49.348 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/e1000 2023-06-22T07:36:49.349 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-06-22T07:36:49.349 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/e1000e 2023-06-22T07:36:49.349 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-06-22T07:36:49.349 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/fm10k 2023-06-22T07:36:49.349 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-06-22T07:36:49.349 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/i40e 2023-06-22T07:36:49.349 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-06-22T07:36:49.349 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/iavf 2023-06-22T07:36:49.350 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-06-22T07:36:49.350 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/igb 2023-06-22T07:36:49.350 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-06-22T07:36:49.350 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/igbvf 2023-06-22T07:36:49.350 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-06-22T07:36:49.350 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/ixgbe 2023-06-22T07:36:49.350 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-06-22T07:36:49.350 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/ixgbevf 2023-06-22T07:36:49.351 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-06-22T07:36:49.351 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/jme.ko 2023-06-22T07:36:49.351 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/marvell 2023-06-22T07:36:49.351 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-06-22T07:36:49.351 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/marvell/skge.ko 2023-06-22T07:36:49.351 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-06-22T07:36:49.351 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/mellanox 2023-06-22T07:36:49.351 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/mellanox/mlx4 2023-06-22T07:36:49.352 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-06-22T07:36:49.352 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-06-22T07:36:49.352 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/mellanox/mlx5 2023-06-22T07:36:49.352 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-06-22T07:36:49.352 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-06-22T07:36:49.352 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/micrel 2023-06-22T07:36:49.352 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-06-22T07:36:49.353 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/myricom 2023-06-22T07:36:49.353 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/myricom/myri10ge 2023-06-22T07:36:49.353 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-06-22T07:36:49.353 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/natsemi 2023-06-22T07:36:49.353 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-06-22T07:36:49.353 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-06-22T07:36:49.353 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/neterion 2023-06-22T07:36:49.353 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-06-22T07:36:49.354 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/nvidia 2023-06-22T07:36:49.354 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-06-22T07:36:49.354 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/packetengines 2023-06-22T07:36:49.354 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-06-22T07:36:49.354 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-06-22T07:36:49.354 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/qlogic 2023-06-22T07:36:49.354 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/qlogic/netxen 2023-06-22T07:36:49.354 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-06-22T07:36:49.355 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-06-22T07:36:49.355 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-06-22T07:36:49.355 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-06-22T07:36:49.355 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/rdc 2023-06-22T07:36:49.355 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-06-22T07:36:49.355 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/realtek 2023-06-22T07:36:49.355 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-06-22T07:36:49.356 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-06-22T07:36:49.356 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/realtek/atp.ko 2023-06-22T07:36:49.356 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-06-22T07:36:49.356 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/rocker 2023-06-22T07:36:49.356 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-06-22T07:36:49.356 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/sfc 2023-06-22T07:36:49.356 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-06-22T07:36:49.356 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/silan 2023-06-22T07:36:49.357 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-06-22T07:36:49.357 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/sis 2023-06-22T07:36:49.357 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/sis/sis190.ko 2023-06-22T07:36:49.357 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/sis/sis900.ko 2023-06-22T07:36:49.357 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/smsc 2023-06-22T07:36:49.357 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-06-22T07:36:49.357 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-06-22T07:36:49.357 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-06-22T07:36:49.358 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-06-22T07:36:49.358 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/stmicro 2023-06-22T07:36:49.358 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/stmicro/stmmac 2023-06-22T07:36:49.358 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-06-22T07:36:49.358 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-06-22T07:36:49.358 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/sun 2023-06-22T07:36:49.358 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/sun/cassini.ko 2023-06-22T07:36:49.358 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/sun/niu.ko 2023-06-22T07:36:49.359 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/sun/sungem.ko 2023-06-22T07:36:49.359 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-06-22T07:36:49.359 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/tehuti 2023-06-22T07:36:49.359 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-06-22T07:36:49.359 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/ti 2023-06-22T07:36:49.359 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/ti/tlan.ko 2023-06-22T07:36:49.359 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/via 2023-06-22T07:36:49.359 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-06-22T07:36:49.360 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-06-22T07:36:49.360 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/wiznet 2023-06-22T07:36:49.360 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-06-22T07:36:49.360 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-06-22T07:36:49.360 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/xircom 2023-06-22T07:36:49.360 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-06-22T07:36:49.360 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/geneve.ko 2023-06-22T07:36:49.361 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/hyperv 2023-06-22T07:36:49.361 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-06-22T07:36:49.361 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ieee802154 2023-06-22T07:36:49.361 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ieee802154/fakelb.ko 2023-06-22T07:36:49.361 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ifb.ko 2023-06-22T07:36:49.361 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ipvlan 2023-06-22T07:36:49.361 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ipvlan/ipvlan.ko 2023-06-22T07:36:49.361 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/macvlan.ko 2023-06-22T07:36:49.362 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/macvtap.ko 2023-06-22T07:36:49.362 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/mdio 2023-06-22T07:36:49.362 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/mdio.ko 2023-06-22T07:36:49.362 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-06-22T07:36:49.362 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-06-22T07:36:49.362 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/mii.ko 2023-06-22T07:36:49.362 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/net_failover.ko 2023-06-22T07:36:49.362 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/netconsole.ko 2023-06-22T07:36:49.363 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/nlmon.ko 2023-06-22T07:36:49.363 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/pcs 2023-06-22T07:36:49.363 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/pcs/pcs-altera-tse.ko 2023-06-22T07:36:49.363 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-06-22T07:36:49.363 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy 2023-06-22T07:36:49.363 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/amd.ko 2023-06-22T07:36:49.363 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-06-22T07:36:49.363 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-06-22T07:36:49.364 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/bcm7xxx.ko 2023-06-22T07:36:49.364 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/bcm87xx.ko 2023-06-22T07:36:49.364 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/broadcom.ko 2023-06-22T07:36:49.364 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/cicada.ko 2023-06-22T07:36:49.364 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/davicom.ko 2023-06-22T07:36:49.364 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/dp83640.ko 2023-06-22T07:36:49.364 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/et1011c.ko 2023-06-22T07:36:49.364 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/icplus.ko 2023-06-22T07:36:49.365 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/lxt.ko 2023-06-22T07:36:49.365 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/marvell.ko 2023-06-22T07:36:49.365 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/micrel.ko 2023-06-22T07:36:49.365 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/national.ko 2023-06-22T07:36:49.365 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/phylink.ko 2023-06-22T07:36:49.365 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/qsemi.ko 2023-06-22T07:36:49.365 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/realtek.ko 2023-06-22T07:36:49.366 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/smsc.ko 2023-06-22T07:36:49.366 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/ste10Xp.ko 2023-06-22T07:36:49.366 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/phy/vitesse.ko 2023-06-22T07:36:49.366 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ppp 2023-06-22T07:36:49.366 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ppp/bsd_comp.ko 2023-06-22T07:36:49.366 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ppp/ppp_async.ko 2023-06-22T07:36:49.366 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ppp/ppp_deflate.ko 2023-06-22T07:36:49.366 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ppp/ppp_generic.ko 2023-06-22T07:36:49.367 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ppp/ppp_mppe.ko 2023-06-22T07:36:49.367 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ppp/ppp_synctty.ko 2023-06-22T07:36:49.367 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ppp/pppoe.ko 2023-06-22T07:36:49.367 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ppp/pppox.ko 2023-06-22T07:36:49.367 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/ppp/pptp.ko 2023-06-22T07:36:49.367 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/slip 2023-06-22T07:36:49.367 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/slip/slhc.ko 2023-06-22T07:36:49.367 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/slip/slip.ko 2023-06-22T07:36:49.368 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/sungem_phy.ko 2023-06-22T07:36:49.368 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/tap.ko 2023-06-22T07:36:49.368 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/team 2023-06-22T07:36:49.368 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/team/team.ko 2023-06-22T07:36:49.368 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/team/team_mode_activebackup.ko 2023-06-22T07:36:49.368 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/team/team_mode_broadcast.ko 2023-06-22T07:36:49.368 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-06-22T07:36:49.369 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/team/team_mode_random.ko 2023-06-22T07:36:49.369 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-06-22T07:36:49.369 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/tun.ko 2023-06-22T07:36:49.369 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/veth.ko 2023-06-22T07:36:49.369 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/virtio_net.ko 2023-06-22T07:36:49.369 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/vmxnet3 2023-06-22T07:36:49.369 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-06-22T07:36:49.369 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/vxlan 2023-06-22T07:36:49.370 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/vxlan/vxlan.ko 2023-06-22T07:36:49.370 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/xen-netback 2023-06-22T07:36:49.370 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/xen-netback/xen-netback.ko 2023-06-22T07:36:49.370 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/net/xen-netfront.ko 2023-06-22T07:36:49.370 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ntb 2023-06-22T07:36:49.370 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ntb/ntb.ko 2023-06-22T07:36:49.370 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/nvme 2023-06-22T07:36:49.370 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/nvme/host 2023-06-22T07:36:49.371 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/nvme/host/nvme-core.ko 2023-06-22T07:36:49.371 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/nvme/host/nvme.ko 2023-06-22T07:36:49.371 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/parport 2023-06-22T07:36:49.371 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/parport/parport.ko 2023-06-22T07:36:49.371 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/parport/parport_cs.ko 2023-06-22T07:36:49.371 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/parport/parport_pc.ko 2023-06-22T07:36:49.371 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/parport/parport_serial.ko 2023-06-22T07:36:49.371 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pci 2023-06-22T07:36:49.372 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pci/hotplug 2023-06-22T07:36:49.372 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-06-22T07:36:49.372 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pci/pcie 2023-06-22T07:36:49.372 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pci/pcie/aer_inject.ko 2023-06-22T07:36:49.372 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pci/xen-pcifront.ko 2023-06-22T07:36:49.372 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pcmcia 2023-06-22T07:36:49.372 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pcmcia/i82092.ko 2023-06-22T07:36:49.373 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pcmcia/pd6729.ko 2023-06-22T07:36:49.373 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pcmcia/yenta_socket.ko 2023-06-22T07:36:49.373 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pinctrl 2023-06-22T07:36:49.373 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pinctrl/intel 2023-06-22T07:36:49.373 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-06-22T07:36:49.373 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform 2023-06-22T07:36:49.373 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86 2023-06-22T07:36:49.373 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/acer-wmi.ko 2023-06-22T07:36:49.374 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/acerhdf.ko 2023-06-22T07:36:49.374 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/apple-gmux.ko 2023-06-22T07:36:49.374 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/asus-laptop.ko 2023-06-22T07:36:49.374 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-06-22T07:36:49.374 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/asus-wmi.ko 2023-06-22T07:36:49.374 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/classmate-laptop.ko 2023-06-22T07:36:49.374 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-06-22T07:36:49.374 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-06-22T07:36:49.375 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-06-22T07:36:49.375 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-06-22T07:36:49.375 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/hdaps.ko 2023-06-22T07:36:49.375 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/intel 2023-06-22T07:36:49.375 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-06-22T07:36:49.375 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/intel_ips.ko 2023-06-22T07:36:49.375 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/msi-wmi.ko 2023-06-22T07:36:49.375 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/mxm-wmi.ko 2023-06-22T07:36:49.376 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-06-22T07:36:49.376 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/samsung-laptop.ko 2023-06-22T07:36:49.376 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/samsung-q10.ko 2023-06-22T07:36:49.376 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/topstar-laptop.ko 2023-06-22T07:36:49.376 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-06-22T07:36:49.376 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/toshiba_haps.ko 2023-06-22T07:36:49.376 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/wmi-bmof.ko 2023-06-22T07:36:49.376 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/platform/x86/wmi.ko 2023-06-22T07:36:49.377 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/powercap 2023-06-22T07:36:49.377 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/powercap/intel_rapl_common.ko 2023-06-22T07:36:49.377 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/powercap/intel_rapl_msr.ko 2023-06-22T07:36:49.377 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pps 2023-06-22T07:36:49.377 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pps/clients 2023-06-22T07:36:49.377 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pps/clients/pps-gpio.ko 2023-06-22T07:36:49.377 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pps/clients/pps-ldisc.ko 2023-06-22T07:36:49.377 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pps/clients/pps_parport.ko 2023-06-22T07:36:49.378 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/pps/pps_core.ko 2023-06-22T07:36:49.378 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ptp 2023-06-22T07:36:49.378 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ptp/ptp.ko 2023-06-22T07:36:49.378 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ptp/ptp_kvm.ko 2023-06-22T07:36:49.378 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc 2023-06-22T07:36:49.378 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-bq32k.ko 2023-06-22T07:36:49.378 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-bq4802.ko 2023-06-22T07:36:49.378 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-ds1286.ko 2023-06-22T07:36:49.379 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-ds1307.ko 2023-06-22T07:36:49.379 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-ds1374.ko 2023-06-22T07:36:49.379 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-ds1511.ko 2023-06-22T07:36:49.379 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-ds1553.ko 2023-06-22T07:36:49.379 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-ds1672.ko 2023-06-22T07:36:49.379 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-ds1742.ko 2023-06-22T07:36:49.379 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-ds2404.ko 2023-06-22T07:36:49.379 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-ds3232.ko 2023-06-22T07:36:49.380 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-em3027.ko 2023-06-22T07:36:49.380 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-fm3130.ko 2023-06-22T07:36:49.380 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-isl12022.ko 2023-06-22T07:36:49.380 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-isl1208.ko 2023-06-22T07:36:49.380 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-m41t80.ko 2023-06-22T07:36:49.380 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-m48t35.ko 2023-06-22T07:36:49.380 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-m48t59.ko 2023-06-22T07:36:49.381 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-max6900.ko 2023-06-22T07:36:49.381 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-msm6242.ko 2023-06-22T07:36:49.381 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-pcf2127.ko 2023-06-22T07:36:49.381 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-pcf85063.ko 2023-06-22T07:36:49.381 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-pcf8523.ko 2023-06-22T07:36:49.381 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-pcf8563.ko 2023-06-22T07:36:49.381 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-pcf8583.ko 2023-06-22T07:36:49.381 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-rp5c01.ko 2023-06-22T07:36:49.382 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-rs5c372.ko 2023-06-22T07:36:49.382 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-06-22T07:36:49.382 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-rx8025.ko 2023-06-22T07:36:49.382 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-rx8581.ko 2023-06-22T07:36:49.382 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-06-22T07:36:49.382 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/rtc/rtc-x1205.ko 2023-06-22T07:36:49.382 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi 2023-06-22T07:36:49.382 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/3w-9xxx.ko 2023-06-22T07:36:49.383 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/3w-sas.ko 2023-06-22T07:36:49.383 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/3w-xxxx.ko 2023-06-22T07:36:49.383 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/BusLogic.ko 2023-06-22T07:36:49.383 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/a100u2w.ko 2023-06-22T07:36:49.383 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/aacraid 2023-06-22T07:36:49.383 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/aacraid/aacraid.ko 2023-06-22T07:36:49.383 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/advansys.ko 2023-06-22T07:36:49.383 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/aic7xxx 2023-06-22T07:36:49.384 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-06-22T07:36:49.384 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-06-22T07:36:49.384 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/am53c974.ko 2023-06-22T07:36:49.384 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/arcmsr 2023-06-22T07:36:49.384 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-06-22T07:36:49.384 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/atp870u.ko 2023-06-22T07:36:49.384 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/be2iscsi 2023-06-22T07:36:49.384 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-06-22T07:36:49.385 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/bfa 2023-06-22T07:36:49.385 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/bfa/bfa.ko 2023-06-22T07:36:49.385 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/bnx2fc 2023-06-22T07:36:49.385 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-06-22T07:36:49.385 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/bnx2i 2023-06-22T07:36:49.385 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-06-22T07:36:49.385 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/ch.ko 2023-06-22T07:36:49.385 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/csiostor 2023-06-22T07:36:49.386 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/csiostor/csiostor.ko 2023-06-22T07:36:49.386 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/cxgbi 2023-06-22T07:36:49.386 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/cxgbi/cxgb3i 2023-06-22T07:36:49.386 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-06-22T07:36:49.386 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/cxgbi/cxgb4i 2023-06-22T07:36:49.386 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-06-22T07:36:49.386 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-06-22T07:36:49.386 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/dc395x.ko 2023-06-22T07:36:49.387 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/device_handler 2023-06-22T07:36:49.387 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-06-22T07:36:49.387 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-06-22T07:36:49.387 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-06-22T07:36:49.387 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-06-22T07:36:49.387 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/dmx3191d.ko 2023-06-22T07:36:49.387 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/esas2r 2023-06-22T07:36:49.387 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/esas2r/esas2r.ko 2023-06-22T07:36:49.388 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/esp_scsi.ko 2023-06-22T07:36:49.388 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/fcoe 2023-06-22T07:36:49.388 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/fcoe/fcoe.ko 2023-06-22T07:36:49.388 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-06-22T07:36:49.388 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/fnic 2023-06-22T07:36:49.388 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/fnic/fnic.ko 2023-06-22T07:36:49.388 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/hpsa.ko 2023-06-22T07:36:49.389 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/hptiop.ko 2023-06-22T07:36:49.389 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/hv_storvsc.ko 2023-06-22T07:36:49.389 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/initio.ko 2023-06-22T07:36:49.389 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/ipr.ko 2023-06-22T07:36:49.389 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/ips.ko 2023-06-22T07:36:49.389 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/isci 2023-06-22T07:36:49.389 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/isci/isci.ko 2023-06-22T07:36:49.389 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-06-22T07:36:49.390 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/iscsi_tcp.ko 2023-06-22T07:36:49.390 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/libfc 2023-06-22T07:36:49.390 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/libfc/libfc.ko 2023-06-22T07:36:49.390 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/libiscsi.ko 2023-06-22T07:36:49.390 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/libiscsi_tcp.ko 2023-06-22T07:36:49.390 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/libsas 2023-06-22T07:36:49.390 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/libsas/libsas.ko 2023-06-22T07:36:49.391 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/lpfc 2023-06-22T07:36:49.391 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/lpfc/lpfc.ko 2023-06-22T07:36:49.391 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/megaraid 2023-06-22T07:36:49.391 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/megaraid.ko 2023-06-22T07:36:49.391 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-06-22T07:36:49.391 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-06-22T07:36:49.391 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-06-22T07:36:49.392 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/mpt3sas 2023-06-22T07:36:49.392 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-06-22T07:36:49.392 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/mvsas 2023-06-22T07:36:49.392 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/mvsas/mvsas.ko 2023-06-22T07:36:49.392 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/mvumi.ko 2023-06-22T07:36:49.393 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/pm8001 2023-06-22T07:36:49.393 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-06-22T07:36:49.393 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/pmcraid.ko 2023-06-22T07:36:49.393 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/qla1280.ko 2023-06-22T07:36:49.393 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/qla2xxx 2023-06-22T07:36:49.393 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-06-22T07:36:49.393 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-06-22T07:36:49.393 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/qla4xxx 2023-06-22T07:36:49.394 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-06-22T07:36:49.394 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/raid_class.ko 2023-06-22T07:36:49.394 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/scsi_debug.ko 2023-06-22T07:36:49.394 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/scsi_transport_fc.ko 2023-06-22T07:36:49.394 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-06-22T07:36:49.394 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/scsi_transport_sas.ko 2023-06-22T07:36:49.394 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/scsi_transport_spi.ko 2023-06-22T07:36:49.394 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/scsi_transport_srp.ko 2023-06-22T07:36:49.395 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/ses.ko 2023-06-22T07:36:49.395 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/st.ko 2023-06-22T07:36:49.395 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/stex.ko 2023-06-22T07:36:49.395 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/sym53c8xx_2 2023-06-22T07:36:49.395 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-06-22T07:36:49.395 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/virtio_scsi.ko 2023-06-22T07:36:49.395 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/vmw_pvscsi.ko 2023-06-22T07:36:49.395 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/scsi/wd719x.ko 2023-06-22T07:36:49.396 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ssb 2023-06-22T07:36:49.396 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ssb/ssb.ko 2023-06-22T07:36:49.396 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target 2023-06-22T07:36:49.396 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/iscsi 2023-06-22T07:36:49.396 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-06-22T07:36:49.396 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/loopback 2023-06-22T07:36:49.396 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/loopback/tcm_loop.ko 2023-06-22T07:36:49.397 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/target_core_file.ko 2023-06-22T07:36:49.397 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/target_core_iblock.ko 2023-06-22T07:36:49.397 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/target_core_mod.ko 2023-06-22T07:36:49.397 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/target_core_pscsi.ko 2023-06-22T07:36:49.397 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/target_core_user.ko 2023-06-22T07:36:49.397 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/tcm_fc 2023-06-22T07:36:49.397 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-06-22T07:36:49.397 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal 2023-06-22T07:36:49.398 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel 2023-06-22T07:36:49.398 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal 2023-06-22T07:36:49.398 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-06-22T07:36:49.398 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-06-22T07:36:49.398 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-06-22T07:36:49.398 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-06-22T07:36:49.398 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-06-22T07:36:49.398 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-06-22T07:36:49.399 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-06-22T07:36:49.399 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-06-22T07:36:49.399 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-06-22T07:36:49.399 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-06-22T07:36:49.399 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-06-22T07:36:49.399 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-06-22T07:36:49.399 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-06-22T07:36:49.399 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-06-22T07:36:49.400 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-06-22T07:36:49.400 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty 2023-06-22T07:36:49.400 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/ipwireless 2023-06-22T07:36:49.400 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-06-22T07:36:49.400 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/n_gsm.ko 2023-06-22T07:36:49.400 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/n_hdlc.ko 2023-06-22T07:36:49.400 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/nozomi.ko 2023-06-22T07:36:49.400 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/serial 2023-06-22T07:36:49.401 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/serial/8250 2023-06-22T07:36:49.401 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-06-22T07:36:49.401 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/serial/arc_uart.ko 2023-06-22T07:36:49.401 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/serial/jsm 2023-06-22T07:36:49.401 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/serial/jsm/jsm.ko 2023-06-22T07:36:49.401 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/tty/synclink_gt.ko 2023-06-22T07:36:49.401 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ufs 2023-06-22T07:36:49.401 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ufs/core 2023-06-22T07:36:49.402 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ufs/core/ufshcd-core.ko 2023-06-22T07:36:49.402 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ufs/host 2023-06-22T07:36:49.402 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-06-22T07:36:49.402 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/uio 2023-06-22T07:36:49.402 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/uio/uio.ko 2023-06-22T07:36:49.402 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/uio/uio_aec.ko 2023-06-22T07:36:49.402 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/uio/uio_cif.ko 2023-06-22T07:36:49.402 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/uio/uio_pci_generic.ko 2023-06-22T07:36:49.403 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/uio/uio_sercos3.ko 2023-06-22T07:36:49.403 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb 2023-06-22T07:36:49.403 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/class 2023-06-22T07:36:49.403 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/class/cdc-acm.ko 2023-06-22T07:36:49.403 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/class/cdc-wdm.ko 2023-06-22T07:36:49.403 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/class/usblp.ko 2023-06-22T07:36:49.403 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/class/usbtmc.ko 2023-06-22T07:36:49.404 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/host 2023-06-22T07:36:49.404 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/host/sl811-hcd.ko 2023-06-22T07:36:49.404 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/image 2023-06-22T07:36:49.404 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/image/mdc800.ko 2023-06-22T07:36:49.404 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/image/microtek.ko 2023-06-22T07:36:49.404 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc 2023-06-22T07:36:49.404 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/adutux.ko 2023-06-22T07:36:49.404 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/appledisplay.ko 2023-06-22T07:36:49.405 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/emi26.ko 2023-06-22T07:36:49.405 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/emi62.ko 2023-06-22T07:36:49.405 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/ezusb.ko 2023-06-22T07:36:49.405 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/idmouse.ko 2023-06-22T07:36:49.405 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/iowarrior.ko 2023-06-22T07:36:49.405 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/isight_firmware.ko 2023-06-22T07:36:49.406 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/ldusb.ko 2023-06-22T07:36:49.406 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/legousbtower.ko 2023-06-22T07:36:49.406 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/sisusbvga 2023-06-22T07:36:49.406 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-06-22T07:36:49.406 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/trancevibrator.ko 2023-06-22T07:36:49.406 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/usb3503.ko 2023-06-22T07:36:49.406 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/usblcd.ko 2023-06-22T07:36:49.406 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/usbsevseg.ko 2023-06-22T07:36:49.407 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/uss720.ko 2023-06-22T07:36:49.407 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/misc/yurex.ko 2023-06-22T07:36:49.407 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/phy 2023-06-22T07:36:49.407 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/phy/phy-generic.ko 2023-06-22T07:36:49.407 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial 2023-06-22T07:36:49.407 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/aircable.ko 2023-06-22T07:36:49.407 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/ark3116.ko 2023-06-22T07:36:49.407 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/belkin_sa.ko 2023-06-22T07:36:49.408 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/ch341.ko 2023-06-22T07:36:49.408 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/cp210x.ko 2023-06-22T07:36:49.408 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/cyberjack.ko 2023-06-22T07:36:49.408 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/cypress_m8.ko 2023-06-22T07:36:49.408 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/digi_acceleport.ko 2023-06-22T07:36:49.408 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/empeg.ko 2023-06-22T07:36:49.408 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/ftdi_sio.ko 2023-06-22T07:36:49.409 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/garmin_gps.ko 2023-06-22T07:36:49.409 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/io_edgeport.ko 2023-06-22T07:36:49.409 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/io_ti.ko 2023-06-22T07:36:49.409 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/ipaq.ko 2023-06-22T07:36:49.409 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/ipw.ko 2023-06-22T07:36:49.409 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/ir-usb.ko 2023-06-22T07:36:49.409 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-06-22T07:36:49.409 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/keyspan.ko 2023-06-22T07:36:49.410 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/keyspan_pda.ko 2023-06-22T07:36:49.410 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/kl5kusb105.ko 2023-06-22T07:36:49.410 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/kobil_sct.ko 2023-06-22T07:36:49.410 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/mct_u232.ko 2023-06-22T07:36:49.410 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/mos7720.ko 2023-06-22T07:36:49.410 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/mos7840.ko 2023-06-22T07:36:49.410 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/navman.ko 2023-06-22T07:36:49.410 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/omninet.ko 2023-06-22T07:36:49.411 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/opticon.ko 2023-06-22T07:36:49.411 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/option.ko 2023-06-22T07:36:49.411 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/oti6858.ko 2023-06-22T07:36:49.411 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/pl2303.ko 2023-06-22T07:36:49.411 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/qcaux.ko 2023-06-22T07:36:49.411 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/qcserial.ko 2023-06-22T07:36:49.411 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/quatech2.ko 2023-06-22T07:36:49.411 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/safe_serial.ko 2023-06-22T07:36:49.412 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/sierra.ko 2023-06-22T07:36:49.412 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/spcp8x5.ko 2023-06-22T07:36:49.412 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/ssu100.ko 2023-06-22T07:36:49.412 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/symbolserial.ko 2023-06-22T07:36:49.412 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-06-22T07:36:49.412 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-06-22T07:36:49.412 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/usb_debug.ko 2023-06-22T07:36:49.413 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/usb_wwan.ko 2023-06-22T07:36:49.413 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/visor.ko 2023-06-22T07:36:49.413 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/whiteheat.ko 2023-06-22T07:36:49.413 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/serial/xsens_mt.ko 2023-06-22T07:36:49.413 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage 2023-06-22T07:36:49.413 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/uas.ko 2023-06-22T07:36:49.413 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-alauda.ko 2023-06-22T07:36:49.413 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-cypress.ko 2023-06-22T07:36:49.414 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-datafab.ko 2023-06-22T07:36:49.414 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-06-22T07:36:49.414 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-freecom.ko 2023-06-22T07:36:49.414 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-isd200.ko 2023-06-22T07:36:49.414 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-06-22T07:36:49.414 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-karma.ko 2023-06-22T07:36:49.414 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-onetouch.ko 2023-06-22T07:36:49.414 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-realtek.ko 2023-06-22T07:36:49.415 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-sddr09.ko 2023-06-22T07:36:49.415 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-sddr55.ko 2023-06-22T07:36:49.415 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/ums-usbat.ko 2023-06-22T07:36:49.415 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/storage/usb-storage.ko 2023-06-22T07:36:49.415 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/usbip 2023-06-22T07:36:49.415 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/usbip/usbip-core.ko 2023-06-22T07:36:49.415 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/usbip/usbip-host.ko 2023-06-22T07:36:49.415 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-06-22T07:36:49.416 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vfio 2023-06-22T07:36:49.416 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vfio/pci 2023-06-22T07:36:49.416 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-06-22T07:36:49.416 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vfio/pci/vfio-pci.ko 2023-06-22T07:36:49.416 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vfio/vfio.ko 2023-06-22T07:36:49.416 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-06-22T07:36:49.416 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vhost 2023-06-22T07:36:49.416 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vhost/vhost.ko 2023-06-22T07:36:49.417 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vhost/vhost_iotlb.ko 2023-06-22T07:36:49.417 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vhost/vhost_net.ko 2023-06-22T07:36:49.417 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/vhost/vhost_scsi.ko 2023-06-22T07:36:49.417 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/video 2023-06-22T07:36:49.417 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/video/backlight 2023-06-22T07:36:49.417 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/video/backlight/apple_bl.ko 2023-06-22T07:36:49.417 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/video/backlight/lcd.ko 2023-06-22T07:36:49.417 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/video/backlight/platform_lcd.ko 2023-06-22T07:36:49.418 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/virtio 2023-06-22T07:36:49.418 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/virtio/virtio.ko 2023-06-22T07:36:49.418 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/virtio/virtio_balloon.ko 2023-06-22T07:36:49.418 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/virtio/virtio_mmio.ko 2023-06-22T07:36:49.418 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/virtio/virtio_pci.ko 2023-06-22T07:36:49.418 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-06-22T07:36:49.418 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-06-22T07:36:49.418 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/virtio/virtio_ring.ko 2023-06-22T07:36:49.419 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1 2023-06-22T07:36:49.419 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/masters 2023-06-22T07:36:49.419 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/masters/ds2482.ko 2023-06-22T07:36:49.419 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/masters/ds2490.ko 2023-06-22T07:36:49.419 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves 2023-06-22T07:36:49.419 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-06-22T07:36:49.419 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-06-22T07:36:49.419 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-06-22T07:36:49.420 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-06-22T07:36:49.420 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-06-22T07:36:49.420 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-06-22T07:36:49.420 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-06-22T07:36:49.420 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-06-22T07:36:49.420 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-06-22T07:36:49.420 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_smem.ko 2023-06-22T07:36:49.420 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/slaves/w1_therm.ko 2023-06-22T07:36:49.421 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/w1/wire.ko 2023-06-22T07:36:49.421 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog 2023-06-22T07:36:49.421 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/alim1535_wdt.ko 2023-06-22T07:36:49.421 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/alim7101_wdt.ko 2023-06-22T07:36:49.421 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/f71808e_wdt.ko 2023-06-22T07:36:49.421 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/hpwdt.ko 2023-06-22T07:36:49.421 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/i6300esb.ko 2023-06-22T07:36:49.421 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-06-22T07:36:49.422 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/iTCO_wdt.ko 2023-06-22T07:36:49.422 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/ib700wdt.ko 2023-06-22T07:36:49.422 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/ibmasr.ko 2023-06-22T07:36:49.422 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-06-22T07:36:49.422 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/it8712f_wdt.ko 2023-06-22T07:36:49.422 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/it87_wdt.ko 2023-06-22T07:36:49.422 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/machzwd.ko 2023-06-22T07:36:49.422 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/nv_tco.ko 2023-06-22T07:36:49.423 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/pcwd_pci.ko 2023-06-22T07:36:49.423 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/pcwd_usb.ko 2023-06-22T07:36:49.423 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-06-22T07:36:49.423 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/sch311x_wdt.ko 2023-06-22T07:36:49.423 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/softdog.ko 2023-06-22T07:36:49.423 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/sp5100_tco.ko 2023-06-22T07:36:49.423 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/via_wdt.ko 2023-06-22T07:36:49.424 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-06-22T07:36:49.424 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/w83877f_wdt.ko 2023-06-22T07:36:49.424 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/w83977f_wdt.ko 2023-06-22T07:36:49.424 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/wdt_pci.ko 2023-06-22T07:36:49.424 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/watchdog/xen_wdt.ko 2023-06-22T07:36:49.424 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/xen 2023-06-22T07:36:49.424 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/xen/xen-acpi-processor.ko 2023-06-22T07:36:49.424 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/xen/xen-evtchn.ko 2023-06-22T07:36:49.425 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/xen/xen-gntalloc.ko 2023-06-22T07:36:49.425 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/xen/xen-gntdev.ko 2023-06-22T07:36:49.425 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/xen/xen-pciback 2023-06-22T07:36:49.425 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-06-22T07:36:49.425 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/xen/xen-privcmd.ko 2023-06-22T07:36:49.425 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/xen/xenfs 2023-06-22T07:36:49.425 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/drivers/xen/xenfs/xenfs.ko 2023-06-22T07:36:49.425 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs 2023-06-22T07:36:49.426 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/9p 2023-06-22T07:36:49.426 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/9p/9p.ko 2023-06-22T07:36:49.426 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/affs 2023-06-22T07:36:49.426 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/affs/affs.ko 2023-06-22T07:36:49.426 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/befs 2023-06-22T07:36:49.426 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/befs/befs.ko 2023-06-22T07:36:49.426 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/binfmt_misc.ko 2023-06-22T07:36:49.426 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/btrfs 2023-06-22T07:36:49.427 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/btrfs/btrfs.ko 2023-06-22T07:36:49.427 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/cachefiles 2023-06-22T07:36:49.427 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/cachefiles/cachefiles.ko 2023-06-22T07:36:49.427 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ceph 2023-06-22T07:36:49.427 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ceph/ceph.ko 2023-06-22T07:36:49.427 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/coda 2023-06-22T07:36:49.427 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/coda/coda.ko 2023-06-22T07:36:49.428 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/cramfs 2023-06-22T07:36:49.428 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/cramfs/cramfs.ko 2023-06-22T07:36:49.428 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/dlm 2023-06-22T07:36:49.428 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/dlm/dlm.ko 2023-06-22T07:36:49.428 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ecryptfs 2023-06-22T07:36:49.428 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ecryptfs/ecryptfs.ko 2023-06-22T07:36:49.428 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/f2fs 2023-06-22T07:36:49.428 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/f2fs/f2fs.ko 2023-06-22T07:36:49.429 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/fat 2023-06-22T07:36:49.429 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/fat/fat.ko 2023-06-22T07:36:49.429 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/fat/msdos.ko 2023-06-22T07:36:49.429 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/fat/vfat.ko 2023-06-22T07:36:49.429 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/fscache 2023-06-22T07:36:49.429 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/fscache/fscache.ko 2023-06-22T07:36:49.429 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/fuse 2023-06-22T07:36:49.429 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/fuse/cuse.ko 2023-06-22T07:36:49.430 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/fuse/fuse.ko 2023-06-22T07:36:49.430 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/gfs2 2023-06-22T07:36:49.430 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/gfs2/gfs2.ko 2023-06-22T07:36:49.430 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/hfs 2023-06-22T07:36:49.430 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/hfs/hfs.ko 2023-06-22T07:36:49.430 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/hfsplus 2023-06-22T07:36:49.430 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/hfsplus/hfsplus.ko 2023-06-22T07:36:49.430 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/isofs 2023-06-22T07:36:49.431 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/isofs/isofs.ko 2023-06-22T07:36:49.431 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/jfs 2023-06-22T07:36:49.431 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/jfs/jfs.ko 2023-06-22T07:36:49.431 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/lockd 2023-06-22T07:36:49.431 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/lockd/lockd.ko 2023-06-22T07:36:49.431 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/minix 2023-06-22T07:36:49.431 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/minix/minix.ko 2023-06-22T07:36:49.431 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/netfs 2023-06-22T07:36:49.432 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/netfs/netfs.ko 2023-06-22T07:36:49.432 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs 2023-06-22T07:36:49.432 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs/blocklayout 2023-06-22T07:36:49.432 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-06-22T07:36:49.432 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs/filelayout 2023-06-22T07:36:49.432 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-06-22T07:36:49.432 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs/flexfilelayout 2023-06-22T07:36:49.433 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-06-22T07:36:49.433 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs/nfs.ko 2023-06-22T07:36:49.433 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs/nfsv3.ko 2023-06-22T07:36:49.433 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs/nfsv4.ko 2023-06-22T07:36:49.433 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs_common 2023-06-22T07:36:49.433 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs_common/grace.ko 2023-06-22T07:36:49.433 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfs_common/nfs_acl.ko 2023-06-22T07:36:49.433 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfsd 2023-06-22T07:36:49.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nfsd/nfsd.ko 2023-06-22T07:36:49.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nilfs2 2023-06-22T07:36:49.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nilfs2/nilfs2.ko 2023-06-22T07:36:49.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls 2023-06-22T07:36:49.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-celtic.ko 2023-06-22T07:36:49.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-centeuro.ko 2023-06-22T07:36:49.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-croatian.ko 2023-06-22T07:36:49.434 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-cyrillic.ko 2023-06-22T07:36:49.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-gaelic.ko 2023-06-22T07:36:49.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-greek.ko 2023-06-22T07:36:49.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-iceland.ko 2023-06-22T07:36:49.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-inuit.ko 2023-06-22T07:36:49.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-roman.ko 2023-06-22T07:36:49.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-romanian.ko 2023-06-22T07:36:49.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/mac-turkish.ko 2023-06-22T07:36:49.435 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp1250.ko 2023-06-22T07:36:49.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp1251.ko 2023-06-22T07:36:49.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp1255.ko 2023-06-22T07:36:49.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp737.ko 2023-06-22T07:36:49.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp775.ko 2023-06-22T07:36:49.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp850.ko 2023-06-22T07:36:49.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp852.ko 2023-06-22T07:36:49.436 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp855.ko 2023-06-22T07:36:49.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp857.ko 2023-06-22T07:36:49.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp860.ko 2023-06-22T07:36:49.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp861.ko 2023-06-22T07:36:49.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp862.ko 2023-06-22T07:36:49.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp863.ko 2023-06-22T07:36:49.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp864.ko 2023-06-22T07:36:49.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp865.ko 2023-06-22T07:36:49.437 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp866.ko 2023-06-22T07:36:49.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp869.ko 2023-06-22T07:36:49.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp874.ko 2023-06-22T07:36:49.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp932.ko 2023-06-22T07:36:49.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp936.ko 2023-06-22T07:36:49.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp949.ko 2023-06-22T07:36:49.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_cp950.ko 2023-06-22T07:36:49.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_euc-jp.ko 2023-06-22T07:36:49.438 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-1.ko 2023-06-22T07:36:49.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-13.ko 2023-06-22T07:36:49.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-14.ko 2023-06-22T07:36:49.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-15.ko 2023-06-22T07:36:49.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-2.ko 2023-06-22T07:36:49.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-3.ko 2023-06-22T07:36:49.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-4.ko 2023-06-22T07:36:49.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-5.ko 2023-06-22T07:36:49.439 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-6.ko 2023-06-22T07:36:49.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-7.ko 2023-06-22T07:36:49.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_iso8859-9.ko 2023-06-22T07:36:49.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_koi8-r.ko 2023-06-22T07:36:49.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_koi8-ru.ko 2023-06-22T07:36:49.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_koi8-u.ko 2023-06-22T07:36:49.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/nls/nls_utf8.ko 2023-06-22T07:36:49.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2 2023-06-22T07:36:49.440 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2/cluster 2023-06-22T07:36:49.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-06-22T07:36:49.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2/dlm 2023-06-22T07:36:49.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-06-22T07:36:49.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2/dlmfs 2023-06-22T07:36:49.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-06-22T07:36:49.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2/ocfs2.ko 2023-06-22T07:36:49.441 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-06-22T07:36:49.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-06-22T07:36:49.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-06-22T07:36:49.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/overlayfs 2023-06-22T07:36:49.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/overlayfs/overlay.ko 2023-06-22T07:36:49.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/pstore 2023-06-22T07:36:49.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/pstore/ramoops.ko 2023-06-22T07:36:49.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/reiserfs 2023-06-22T07:36:49.442 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/reiserfs/reiserfs.ko 2023-06-22T07:36:49.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/romfs 2023-06-22T07:36:49.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/romfs/romfs.ko 2023-06-22T07:36:49.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/smb 2023-06-22T07:36:49.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/smb/client 2023-06-22T07:36:49.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/smb/client/cifs.ko 2023-06-22T07:36:49.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/smb/common 2023-06-22T07:36:49.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/smb/common/cifs_arc4.ko 2023-06-22T07:36:49.443 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/smb/common/cifs_md4.ko 2023-06-22T07:36:49.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/squashfs 2023-06-22T07:36:49.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/squashfs/squashfs.ko 2023-06-22T07:36:49.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/sysv 2023-06-22T07:36:49.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/sysv/sysv.ko 2023-06-22T07:36:49.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/udf 2023-06-22T07:36:49.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/udf/udf.ko 2023-06-22T07:36:49.444 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ufs 2023-06-22T07:36:49.445 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/ufs/ufs.ko 2023-06-22T07:36:49.445 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/xfs 2023-06-22T07:36:49.445 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/fs/xfs/xfs.ko 2023-06-22T07:36:49.445 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/kernel 2023-06-22T07:36:49.445 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/kernel/rcu 2023-06-22T07:36:49.445 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/kernel/rcu/rcutorture.ko 2023-06-22T07:36:49.445 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/kernel/torture.ko 2023-06-22T07:36:49.445 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/kernel/trace 2023-06-22T07:36:49.446 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-06-22T07:36:49.446 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib 2023-06-22T07:36:49.446 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/asn1_encoder.ko 2023-06-22T07:36:49.446 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/crc-itu-t.ko 2023-06-22T07:36:49.446 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/crc8.ko 2023-06-22T07:36:49.446 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/crypto 2023-06-22T07:36:49.446 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/crypto/libarc4.ko 2023-06-22T07:36:49.446 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/crypto/libdes.ko 2023-06-22T07:36:49.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/libcrc32c.ko 2023-06-22T07:36:49.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/lru_cache.ko 2023-06-22T07:36:49.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/lz4 2023-06-22T07:36:49.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/lz4/lz4_compress.ko 2023-06-22T07:36:49.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/lz4/lz4hc_compress.ko 2023-06-22T07:36:49.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/math 2023-06-22T07:36:49.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/math/cordic.ko 2023-06-22T07:36:49.447 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/raid6 2023-06-22T07:36:49.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/raid6/raid6_pq.ko 2023-06-22T07:36:49.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/reed_solomon 2023-06-22T07:36:49.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/reed_solomon/reed_solomon.ko 2023-06-22T07:36:49.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/ts_bm.ko 2023-06-22T07:36:49.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/ts_fsm.ko 2023-06-22T07:36:49.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/ts_kmp.ko 2023-06-22T07:36:49.448 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/zstd 2023-06-22T07:36:49.449 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/lib/zstd/zstd_compress.ko 2023-06-22T07:36:49.449 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/mm 2023-06-22T07:36:49.449 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/mm/hwpoison-inject.ko 2023-06-22T07:36:49.449 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net 2023-06-22T07:36:49.449 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/6lowpan 2023-06-22T07:36:49.449 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/6lowpan/6lowpan.ko 2023-06-22T07:36:49.449 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/6lowpan/nhc_dest.ko 2023-06-22T07:36:49.449 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/6lowpan/nhc_fragment.ko 2023-06-22T07:36:49.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/6lowpan/nhc_hop.ko 2023-06-22T07:36:49.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/6lowpan/nhc_ipv6.ko 2023-06-22T07:36:49.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/6lowpan/nhc_mobility.ko 2023-06-22T07:36:49.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/6lowpan/nhc_routing.ko 2023-06-22T07:36:49.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/6lowpan/nhc_udp.ko 2023-06-22T07:36:49.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/802 2023-06-22T07:36:49.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/802/garp.ko 2023-06-22T07:36:49.450 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/802/mrp.ko 2023-06-22T07:36:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/802/p8022.ko 2023-06-22T07:36:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/802/psnap.ko 2023-06-22T07:36:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/802/stp.ko 2023-06-22T07:36:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/8021q 2023-06-22T07:36:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/8021q/8021q.ko 2023-06-22T07:36:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/9p 2023-06-22T07:36:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/9p/9pnet.ko 2023-06-22T07:36:49.451 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/9p/9pnet_fd.ko 2023-06-22T07:36:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/9p/9pnet_rdma.ko 2023-06-22T07:36:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/9p/9pnet_virtio.ko 2023-06-22T07:36:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/batman-adv 2023-06-22T07:36:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/batman-adv/batman-adv.ko 2023-06-22T07:36:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge 2023-06-22T07:36:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/br_netfilter.ko 2023-06-22T07:36:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/bridge.ko 2023-06-22T07:36:49.452 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter 2023-06-22T07:36:49.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-06-22T07:36:49.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_among.ko 2023-06-22T07:36:49.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_arp.ko 2023-06-22T07:36:49.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-06-22T07:36:49.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-06-22T07:36:49.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_ip.ko 2023-06-22T07:36:49.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-06-22T07:36:49.453 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_limit.ko 2023-06-22T07:36:49.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_log.ko 2023-06-22T07:36:49.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_mark.ko 2023-06-22T07:36:49.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-06-22T07:36:49.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-06-22T07:36:49.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-06-22T07:36:49.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-06-22T07:36:49.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_snat.ko 2023-06-22T07:36:49.454 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_stp.ko 2023-06-22T07:36:49.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-06-22T07:36:49.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-06-22T07:36:49.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-06-22T07:36:49.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-06-22T07:36:49.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/ebtables.ko 2023-06-22T07:36:49.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-06-22T07:36:49.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-06-22T07:36:49.455 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-06-22T07:36:49.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ceph 2023-06-22T07:36:49.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ceph/libceph.ko 2023-06-22T07:36:49.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/core 2023-06-22T07:36:49.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/core/failover.ko 2023-06-22T07:36:49.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/core/pktgen.ko 2023-06-22T07:36:49.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dccp 2023-06-22T07:36:49.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dccp/dccp.ko 2023-06-22T07:36:49.456 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dccp/dccp_diag.ko 2023-06-22T07:36:49.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dccp/dccp_ipv4.ko 2023-06-22T07:36:49.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dccp/dccp_ipv6.ko 2023-06-22T07:36:49.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dns_resolver 2023-06-22T07:36:49.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dns_resolver/dns_resolver.ko 2023-06-22T07:36:49.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dsa 2023-06-22T07:36:49.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dsa/dsa_core.ko 2023-06-22T07:36:49.457 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dsa/tag_brcm.ko 2023-06-22T07:36:49.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dsa/tag_dsa.ko 2023-06-22T07:36:49.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dsa/tag_none.ko 2023-06-22T07:36:49.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/dsa/tag_trailer.ko 2023-06-22T07:36:49.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ieee802154 2023-06-22T07:36:49.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ieee802154/6lowpan 2023-06-22T07:36:49.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-06-22T07:36:49.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ieee802154/ieee802154.ko 2023-06-22T07:36:49.458 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ieee802154/ieee802154_socket.ko 2023-06-22T07:36:49.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4 2023-06-22T07:36:49.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/ah4.ko 2023-06-22T07:36:49.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/esp4.ko 2023-06-22T07:36:49.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/fou.ko 2023-06-22T07:36:49.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/gre.ko 2023-06-22T07:36:49.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/inet_diag.ko 2023-06-22T07:36:49.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/ip_gre.ko 2023-06-22T07:36:49.459 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/ip_tunnel.ko 2023-06-22T07:36:49.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/ip_vti.ko 2023-06-22T07:36:49.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/ipcomp.ko 2023-06-22T07:36:49.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/ipip.ko 2023-06-22T07:36:49.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter 2023-06-22T07:36:49.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/arp_tables.ko 2023-06-22T07:36:49.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-06-22T07:36:49.460 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-06-22T07:36:49.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-06-22T07:36:49.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-06-22T07:36:49.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-06-22T07:36:49.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-06-22T07:36:49.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-06-22T07:36:49.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-06-22T07:36:49.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-06-22T07:36:49.461 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/iptable_security.ko 2023-06-22T07:36:49.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-06-22T07:36:49.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-06-22T07:36:49.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-06-22T07:36:49.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-06-22T07:36:49.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-06-22T07:36:49.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-06-22T07:36:49.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-06-22T07:36:49.462 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-06-22T07:36:49.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-06-22T07:36:49.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-06-22T07:36:49.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_bic.ko 2023-06-22T07:36:49.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_dctcp.ko 2023-06-22T07:36:49.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_diag.ko 2023-06-22T07:36:49.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_highspeed.ko 2023-06-22T07:36:49.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_htcp.ko 2023-06-22T07:36:49.463 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_hybla.ko 2023-06-22T07:36:49.464 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_illinois.ko 2023-06-22T07:36:49.464 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_lp.ko 2023-06-22T07:36:49.464 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_scalable.ko 2023-06-22T07:36:49.464 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_vegas.ko 2023-06-22T07:36:49.464 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_veno.ko 2023-06-22T07:36:49.464 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_westwood.ko 2023-06-22T07:36:49.464 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tcp_yeah.ko 2023-06-22T07:36:49.465 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/tunnel4.ko 2023-06-22T07:36:49.465 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/udp_diag.ko 2023-06-22T07:36:49.465 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/udp_tunnel.ko 2023-06-22T07:36:49.465 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv4/xfrm4_tunnel.ko 2023-06-22T07:36:49.465 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6 2023-06-22T07:36:49.465 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/ah6.ko 2023-06-22T07:36:49.465 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/esp6.ko 2023-06-22T07:36:49.465 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/fou6.ko 2023-06-22T07:36:49.466 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/ip6_tunnel.ko 2023-06-22T07:36:49.466 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-06-22T07:36:49.466 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/ip6_vti.ko 2023-06-22T07:36:49.466 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/ipcomp6.ko 2023-06-22T07:36:49.466 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter 2023-06-22T07:36:49.466 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-06-22T07:36:49.466 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-06-22T07:36:49.466 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-06-22T07:36:49.467 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-06-22T07:36:49.467 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-06-22T07:36:49.467 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-06-22T07:36:49.467 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-06-22T07:36:49.467 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-06-22T07:36:49.467 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-06-22T07:36:49.467 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-06-22T07:36:49.467 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-06-22T07:36:49.468 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-06-22T07:36:49.468 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-06-22T07:36:49.468 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-06-22T07:36:49.468 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-06-22T07:36:49.468 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-06-22T07:36:49.468 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-06-22T07:36:49.468 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-06-22T07:36:49.469 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-06-22T07:36:49.469 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-06-22T07:36:49.469 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-06-22T07:36:49.469 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-06-22T07:36:49.469 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-06-22T07:36:49.469 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-06-22T07:36:49.469 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/sit.ko 2023-06-22T07:36:49.469 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/tunnel6.ko 2023-06-22T07:36:49.470 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/ipv6/xfrm6_tunnel.ko 2023-06-22T07:36:49.470 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/key 2023-06-22T07:36:49.470 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/key/af_key.ko 2023-06-22T07:36:49.470 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/l2tp 2023-06-22T07:36:49.470 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/l2tp/l2tp_core.ko 2023-06-22T07:36:49.470 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/l2tp/l2tp_debugfs.ko 2023-06-22T07:36:49.470 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/l2tp/l2tp_eth.ko 2023-06-22T07:36:49.470 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/l2tp/l2tp_ip.ko 2023-06-22T07:36:49.471 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/l2tp/l2tp_ip6.ko 2023-06-22T07:36:49.471 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/l2tp/l2tp_netlink.ko 2023-06-22T07:36:49.471 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/l2tp/l2tp_ppp.ko 2023-06-22T07:36:49.471 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/llc 2023-06-22T07:36:49.471 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/llc/llc.ko 2023-06-22T07:36:49.471 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/mac802154 2023-06-22T07:36:49.471 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/mac802154/mac802154.ko 2023-06-22T07:36:49.471 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/mpls 2023-06-22T07:36:49.472 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/mpls/mpls_gso.ko 2023-06-22T07:36:49.472 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter 2023-06-22T07:36:49.472 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset 2023-06-22T07:36:49.472 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set.ko 2023-06-22T07:36:49.472 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-06-22T07:36:49.472 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-06-22T07:36:49.472 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-06-22T07:36:49.473 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-06-22T07:36:49.473 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-06-22T07:36:49.473 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-06-22T07:36:49.473 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-06-22T07:36:49.473 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-06-22T07:36:49.473 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-06-22T07:36:49.473 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-06-22T07:36:49.474 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-06-22T07:36:49.474 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-06-22T07:36:49.474 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-06-22T07:36:49.474 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-06-22T07:36:49.474 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-06-22T07:36:49.474 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs 2023-06-22T07:36:49.474 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs.ko 2023-06-22T07:36:49.474 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-06-22T07:36:49.475 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-06-22T07:36:49.475 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-06-22T07:36:49.475 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-06-22T07:36:49.475 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-06-22T07:36:49.475 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-06-22T07:36:49.475 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-06-22T07:36:49.475 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-06-22T07:36:49.476 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-06-22T07:36:49.476 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-06-22T07:36:49.476 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-06-22T07:36:49.476 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-06-22T07:36:49.476 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-06-22T07:36:49.476 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conncount.ko 2023-06-22T07:36:49.476 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack.ko 2023-06-22T07:36:49.477 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-06-22T07:36:49.477 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-06-22T07:36:49.477 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-06-22T07:36:49.477 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_h323.ko 2023-06-22T07:36:49.477 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_irc.ko 2023-06-22T07:36:49.477 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-06-22T07:36:49.477 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-06-22T07:36:49.477 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-06-22T07:36:49.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_sane.ko 2023-06-22T07:36:49.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_sip.ko 2023-06-22T07:36:49.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-06-22T07:36:49.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-06-22T07:36:49.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_dup_netdev.ko 2023-06-22T07:36:49.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_flow_table.ko 2023-06-22T07:36:49.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_flow_table_inet.ko 2023-06-22T07:36:49.478 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_log_syslog.ko 2023-06-22T07:36:49.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_nat.ko 2023-06-22T07:36:49.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_nat_amanda.ko 2023-06-22T07:36:49.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_nat_ftp.ko 2023-06-22T07:36:49.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_nat_irc.ko 2023-06-22T07:36:49.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_nat_sip.ko 2023-06-22T07:36:49.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_nat_tftp.ko 2023-06-22T07:36:49.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_synproxy_core.ko 2023-06-22T07:36:49.479 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nf_tables.ko 2023-06-22T07:36:49.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nfnetlink.ko 2023-06-22T07:36:49.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nfnetlink_acct.ko 2023-06-22T07:36:49.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nfnetlink_log.ko 2023-06-22T07:36:49.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nfnetlink_osf.ko 2023-06-22T07:36:49.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nfnetlink_queue.ko 2023-06-22T07:36:49.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_chain_nat.ko 2023-06-22T07:36:49.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_compat.ko 2023-06-22T07:36:49.480 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_ct.ko 2023-06-22T07:36:49.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_dup_netdev.ko 2023-06-22T07:36:49.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_fib.ko 2023-06-22T07:36:49.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_fwd_netdev.ko 2023-06-22T07:36:49.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_hash.ko 2023-06-22T07:36:49.481 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_limit.ko 2023-06-22T07:36:49.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_log.ko 2023-06-22T07:36:49.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_masq.ko 2023-06-22T07:36:49.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_nat.ko 2023-06-22T07:36:49.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_numgen.ko 2023-06-22T07:36:49.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_queue.ko 2023-06-22T07:36:49.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_quota.ko 2023-06-22T07:36:49.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_redir.ko 2023-06-22T07:36:49.482 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_reject.ko 2023-06-22T07:36:49.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_reject_inet.ko 2023-06-22T07:36:49.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_socket.ko 2023-06-22T07:36:49.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/nft_tunnel.ko 2023-06-22T07:36:49.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_AUDIT.ko 2023-06-22T07:36:49.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_CHECKSUM.ko 2023-06-22T07:36:49.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_CLASSIFY.ko 2023-06-22T07:36:49.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-06-22T07:36:49.483 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_CT.ko 2023-06-22T07:36:49.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_DSCP.ko 2023-06-22T07:36:49.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_HL.ko 2023-06-22T07:36:49.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_HMARK.ko 2023-06-22T07:36:49.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_IDLETIMER.ko 2023-06-22T07:36:49.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_LED.ko 2023-06-22T07:36:49.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_LOG.ko 2023-06-22T07:36:49.484 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_MASQUERADE.ko 2023-06-22T07:36:49.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_NETMAP.ko 2023-06-22T07:36:49.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_NFLOG.ko 2023-06-22T07:36:49.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_NFQUEUE.ko 2023-06-22T07:36:49.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_RATEEST.ko 2023-06-22T07:36:49.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_REDIRECT.ko 2023-06-22T07:36:49.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_SECMARK.ko 2023-06-22T07:36:49.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_TCPMSS.ko 2023-06-22T07:36:49.485 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-06-22T07:36:49.486 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_TEE.ko 2023-06-22T07:36:49.486 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_TPROXY.ko 2023-06-22T07:36:49.486 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_TRACE.ko 2023-06-22T07:36:49.486 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_addrtype.ko 2023-06-22T07:36:49.486 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_bpf.ko 2023-06-22T07:36:49.486 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_cgroup.ko 2023-06-22T07:36:49.486 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_cluster.ko 2023-06-22T07:36:49.486 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_comment.ko 2023-06-22T07:36:49.487 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_connbytes.ko 2023-06-22T07:36:49.487 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_connlabel.ko 2023-06-22T07:36:49.487 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_connlimit.ko 2023-06-22T07:36:49.487 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_connmark.ko 2023-06-22T07:36:49.487 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_conntrack.ko 2023-06-22T07:36:49.487 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_cpu.ko 2023-06-22T07:36:49.487 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_dccp.ko 2023-06-22T07:36:49.487 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_devgroup.ko 2023-06-22T07:36:49.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_dscp.ko 2023-06-22T07:36:49.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_ecn.ko 2023-06-22T07:36:49.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_esp.ko 2023-06-22T07:36:49.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_hashlimit.ko 2023-06-22T07:36:49.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_helper.ko 2023-06-22T07:36:49.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_hl.ko 2023-06-22T07:36:49.488 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_ipcomp.ko 2023-06-22T07:36:49.489 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_iprange.ko 2023-06-22T07:36:49.489 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_ipvs.ko 2023-06-22T07:36:49.489 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_l2tp.ko 2023-06-22T07:36:49.489 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_length.ko 2023-06-22T07:36:49.489 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_limit.ko 2023-06-22T07:36:49.489 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_mac.ko 2023-06-22T07:36:49.489 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_mark.ko 2023-06-22T07:36:49.489 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_multiport.ko 2023-06-22T07:36:49.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_nat.ko 2023-06-22T07:36:49.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_nfacct.ko 2023-06-22T07:36:49.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_osf.ko 2023-06-22T07:36:49.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_owner.ko 2023-06-22T07:36:49.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_physdev.ko 2023-06-22T07:36:49.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_pkttype.ko 2023-06-22T07:36:49.490 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_policy.ko 2023-06-22T07:36:49.491 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_quota.ko 2023-06-22T07:36:49.491 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_rateest.ko 2023-06-22T07:36:49.491 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_realm.ko 2023-06-22T07:36:49.491 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_recent.ko 2023-06-22T07:36:49.491 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_sctp.ko 2023-06-22T07:36:49.491 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_set.ko 2023-06-22T07:36:49.491 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_socket.ko 2023-06-22T07:36:49.491 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_state.ko 2023-06-22T07:36:49.492 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_statistic.ko 2023-06-22T07:36:49.492 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_string.ko 2023-06-22T07:36:49.492 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_tcpmss.ko 2023-06-22T07:36:49.492 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_time.ko 2023-06-22T07:36:49.492 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netfilter/xt_u32.ko 2023-06-22T07:36:49.492 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netlink 2023-06-22T07:36:49.492 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/netlink/netlink_diag.ko 2023-06-22T07:36:49.492 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/nsh 2023-06-22T07:36:49.493 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/nsh/nsh.ko 2023-06-22T07:36:49.493 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/openvswitch 2023-06-22T07:36:49.493 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/openvswitch/openvswitch.ko 2023-06-22T07:36:49.493 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/openvswitch/vport-geneve.ko 2023-06-22T07:36:49.493 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/openvswitch/vport-gre.ko 2023-06-22T07:36:49.493 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/openvswitch/vport-vxlan.ko 2023-06-22T07:36:49.493 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/packet 2023-06-22T07:36:49.493 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/packet/af_packet_diag.ko 2023-06-22T07:36:49.494 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/rds 2023-06-22T07:36:49.494 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/rds/rds.ko 2023-06-22T07:36:49.494 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/rds/rds_rdma.ko 2023-06-22T07:36:49.494 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/rds/rds_tcp.ko 2023-06-22T07:36:49.494 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched 2023-06-22T07:36:49.494 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/act_csum.ko 2023-06-22T07:36:49.494 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/act_gact.ko 2023-06-22T07:36:49.494 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/act_ipt.ko 2023-06-22T07:36:49.495 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/act_mirred.ko 2023-06-22T07:36:49.495 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/act_nat.ko 2023-06-22T07:36:49.495 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/act_pedit.ko 2023-06-22T07:36:49.495 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/act_police.ko 2023-06-22T07:36:49.495 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/act_simple.ko 2023-06-22T07:36:49.495 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/act_skbedit.ko 2023-06-22T07:36:49.495 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/act_vlan.ko 2023-06-22T07:36:49.495 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/cls_basic.ko 2023-06-22T07:36:49.496 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/cls_bpf.ko 2023-06-22T07:36:49.496 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/cls_flow.ko 2023-06-22T07:36:49.496 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/cls_fw.ko 2023-06-22T07:36:49.496 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/cls_route.ko 2023-06-22T07:36:49.496 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/cls_u32.ko 2023-06-22T07:36:49.496 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/em_cmp.ko 2023-06-22T07:36:49.496 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/em_ipset.ko 2023-06-22T07:36:49.497 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/em_meta.ko 2023-06-22T07:36:49.497 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/em_nbyte.ko 2023-06-22T07:36:49.497 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/em_text.ko 2023-06-22T07:36:49.497 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/em_u32.ko 2023-06-22T07:36:49.497 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_choke.ko 2023-06-22T07:36:49.497 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_codel.ko 2023-06-22T07:36:49.497 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_drr.ko 2023-06-22T07:36:49.497 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_fq.ko 2023-06-22T07:36:49.498 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_gred.ko 2023-06-22T07:36:49.498 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_hfsc.ko 2023-06-22T07:36:49.498 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_hhf.ko 2023-06-22T07:36:49.498 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_htb.ko 2023-06-22T07:36:49.498 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_ingress.ko 2023-06-22T07:36:49.498 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_mqprio.ko 2023-06-22T07:36:49.498 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_mqprio_lib.ko 2023-06-22T07:36:49.498 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_multiq.ko 2023-06-22T07:36:49.499 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_netem.ko 2023-06-22T07:36:49.499 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_pie.ko 2023-06-22T07:36:49.499 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_plug.ko 2023-06-22T07:36:49.499 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_prio.ko 2023-06-22T07:36:49.499 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_qfq.ko 2023-06-22T07:36:49.499 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_red.ko 2023-06-22T07:36:49.499 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_sfb.ko 2023-06-22T07:36:49.499 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_sfq.ko 2023-06-22T07:36:49.500 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_tbf.ko 2023-06-22T07:36:49.500 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sched/sch_teql.ko 2023-06-22T07:36:49.500 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sctp 2023-06-22T07:36:49.500 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sctp/sctp.ko 2023-06-22T07:36:49.500 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sctp/sctp_diag.ko 2023-06-22T07:36:49.500 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sunrpc 2023-06-22T07:36:49.500 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sunrpc/auth_gss 2023-06-22T07:36:49.500 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-06-22T07:36:49.501 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-06-22T07:36:49.501 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sunrpc/sunrpc.ko 2023-06-22T07:36:49.501 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sunrpc/xprtrdma 2023-06-22T07:36:49.501 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-06-22T07:36:49.501 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/tipc 2023-06-22T07:36:49.501 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/tipc/diag.ko 2023-06-22T07:36:49.501 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/tipc/tipc.ko 2023-06-22T07:36:49.501 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/unix 2023-06-22T07:36:49.502 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/unix/unix_diag.ko 2023-06-22T07:36:49.502 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/vmw_vsock 2023-06-22T07:36:49.502 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-06-22T07:36:49.502 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-06-22T07:36:49.502 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/vmw_vsock/vsock.ko 2023-06-22T07:36:49.502 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/vmw_vsock/vsock_diag.ko 2023-06-22T07:36:49.502 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/vmw_vsock/vsock_loopback.ko 2023-06-22T07:36:49.503 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/xfrm 2023-06-22T07:36:49.503 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/net/xfrm/xfrm_ipcomp.ko 2023-06-22T07:36:49.503 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/security 2023-06-22T07:36:49.503 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/security/keys 2023-06-22T07:36:49.503 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/security/keys/encrypted-keys 2023-06-22T07:36:49.503 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-06-22T07:36:49.503 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/security/keys/trusted-keys 2023-06-22T07:36:49.503 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/security/keys/trusted-keys/trusted.ko 2023-06-22T07:36:49.504 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/virt 2023-06-22T07:36:49.504 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/virt/lib 2023-06-22T07:36:49.504 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/kernel/virt/lib/irqbypass.ko 2023-06-22T07:36:49.504 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.alias 2023-06-22T07:36:49.504 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.alias.bin 2023-06-22T07:36:49.504 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.builtin 2023-06-22T07:36:49.504 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.builtin.bin 2023-06-22T07:36:49.504 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.builtin.modinfo 2023-06-22T07:36:49.505 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.dep 2023-06-22T07:36:49.505 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.dep.bin 2023-06-22T07:36:49.505 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.devname 2023-06-22T07:36:49.505 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.order 2023-06-22T07:36:49.505 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.softdep 2023-06-22T07:36:49.505 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.symbols 2023-06-22T07:36:49.505 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.4.0-rc5-g1c622d92b6f1/modules.symbols.bin 2023-06-22T07:36:49.506 DEBUG:teuthology.task.kernel:get_image_version: 6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:36:49.506 DEBUG:teuthology.orchestra.run.smithi023:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-06-22T07:36:49.534 INFO:teuthology.orchestra.run.smithi023.stdout:postinstall scriptlet (using /bin/sh): 2023-06-22T07:36:49.535 INFO:teuthology.orchestra.run.smithi023.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.4.0-rc5-g1c622d92b6f1 -a -r /boot/System.map-6.4.0-rc5-g1c622d92b6f1 ]; then 2023-06-22T07:36:49.535 INFO:teuthology.orchestra.run.smithi023.stdout:cp /boot/vmlinuz-6.4.0-rc5-g1c622d92b6f1 /boot/.vmlinuz-6.4.0-rc5-g1c622d92b6f1-rpm 2023-06-22T07:36:49.535 INFO:teuthology.orchestra.run.smithi023.stdout:cp /boot/System.map-6.4.0-rc5-g1c622d92b6f1 /boot/.System.map-6.4.0-rc5-g1c622d92b6f1-rpm 2023-06-22T07:36:49.535 INFO:teuthology.orchestra.run.smithi023.stdout:rm -f /boot/vmlinuz-6.4.0-rc5-g1c622d92b6f1 /boot/System.map-6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:36:49.535 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/installkernel 6.4.0-rc5-g1c622d92b6f1 /boot/.vmlinuz-6.4.0-rc5-g1c622d92b6f1-rpm /boot/.System.map-6.4.0-rc5-g1c622d92b6f1-rpm 2023-06-22T07:36:49.536 INFO:teuthology.orchestra.run.smithi023.stdout:rm -f /boot/.vmlinuz-6.4.0-rc5-g1c622d92b6f1-rpm /boot/.System.map-6.4.0-rc5-g1c622d92b6f1-rpm 2023-06-22T07:36:49.536 INFO:teuthology.orchestra.run.smithi023.stdout:fi 2023-06-22T07:36:49.536 INFO:teuthology.orchestra.run.smithi023.stdout:preuninstall scriptlet (using /bin/sh): 2023-06-22T07:36:49.536 INFO:teuthology.orchestra.run.smithi023.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-06-22T07:36:49.536 INFO:teuthology.orchestra.run.smithi023.stdout:new-kernel-pkg --remove 6.4.0-rc5-g1c622d92b6f1 --rminitrd --initrdfile=/boot/initramfs-6.4.0-rc5-g1c622d92b6f1.img 2023-06-22T07:36:49.536 INFO:teuthology.orchestra.run.smithi023.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-06-22T07:36:49.536 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-install remove 6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:36:49.537 INFO:teuthology.orchestra.run.smithi023.stdout:fi 2023-06-22T07:36:49.537 INFO:teuthology.orchestra.run.smithi023.stdout:postuninstall scriptlet (using /bin/sh): 2023-06-22T07:36:49.537 INFO:teuthology.orchestra.run.smithi023.stdout:if [ -x /sbin/update-bootloader ]; then 2023-06-22T07:36:49.537 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/update-bootloader --remove 6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:36:49.537 INFO:teuthology.orchestra.run.smithi023.stdout:fi 2023-06-22T07:36:49.538 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -qi grub2-tools 2023-06-22T07:36:49.617 INFO:teuthology.orchestra.run.smithi023.stdout:Name : grub2-tools 2023-06-22T07:36:49.617 INFO:teuthology.orchestra.run.smithi023.stdout:Epoch : 1 2023-06-22T07:36:49.618 INFO:teuthology.orchestra.run.smithi023.stdout:Version : 2.02 2023-06-22T07:36:49.618 INFO:teuthology.orchestra.run.smithi023.stdout:Release : 123.el8 2023-06-22T07:36:49.618 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: x86_64 2023-06-22T07:36:49.618 INFO:teuthology.orchestra.run.smithi023.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2023-06-22T07:36:49.618 INFO:teuthology.orchestra.run.smithi023.stdout:Group : System Environment/Base 2023-06-22T07:36:49.618 INFO:teuthology.orchestra.run.smithi023.stdout:Size : 9265450 2023-06-22T07:36:49.618 INFO:teuthology.orchestra.run.smithi023.stdout:License : GPLv3+ 2023-06-22T07:36:49.619 INFO:teuthology.orchestra.run.smithi023.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2023-06-22T07:36:49.619 INFO:teuthology.orchestra.run.smithi023.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2023-06-22T07:36:49.619 INFO:teuthology.orchestra.run.smithi023.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2023-06-22T07:36:49.619 INFO:teuthology.orchestra.run.smithi023.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2023-06-22T07:36:49.619 INFO:teuthology.orchestra.run.smithi023.stdout:Relocations : (not relocatable) 2023-06-22T07:36:49.619 INFO:teuthology.orchestra.run.smithi023.stdout:Packager : Red Hat, Inc. 2023-06-22T07:36:49.619 INFO:teuthology.orchestra.run.smithi023.stdout:Vendor : Red Hat, Inc. 2023-06-22T07:36:49.620 INFO:teuthology.orchestra.run.smithi023.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-22T07:36:49.620 INFO:teuthology.orchestra.run.smithi023.stdout:Summary : Support tools for GRUB. 2023-06-22T07:36:49.620 INFO:teuthology.orchestra.run.smithi023.stdout:Description : 2023-06-22T07:36:49.620 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:36:49.620 INFO:teuthology.orchestra.run.smithi023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-22T07:36:49.620 INFO:teuthology.orchestra.run.smithi023.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-22T07:36:49.620 INFO:teuthology.orchestra.run.smithi023.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-22T07:36:49.621 INFO:teuthology.orchestra.run.smithi023.stdout:hardware devices. 2023-06-22T07:36:49.621 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:36:49.621 INFO:teuthology.orchestra.run.smithi023.stdout:This subpackage provides tools for support of all platforms. 2023-06-22T07:36:49.622 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-22T07:36:49.623 INFO:teuthology.task.kernel:Updating grub on smithi023 to boot 6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:36:49.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-22T07:36:51.655 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2023-06-22T07:36:53.663 INFO:teuthology.orchestra.run.smithi023.stderr:done 2023-06-22T07:36:53.665 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2023-06-22T07:36:53.682 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.pd1IeNMn2K 2023-06-22T07:36:53.683 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pd1IeNMn2K 2023-06-22T07:36:53.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.pd1IeNMn2K 2023-06-22T07:36:53.904 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.pd1IeNMn2K is 6KB 2023-06-22T07:36:53.952 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.pd1IeNMn2K 2023-06-22T07:36:53.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo /bin/ls /boot/loader/entries || true 2023-06-22T07:36:54.061 INFO:teuthology.orchestra.run.smithi023.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2023-06-22T07:36:54.062 INFO:teuthology.orchestra.run.smithi023.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2023-06-22T07:36:54.062 INFO:teuthology.orchestra.run.smithi023.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.4.0-rc5-g1c622d92b6f1.conf 2023-06-22T07:36:54.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:36:54.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2023-06-22T07:37:24.197 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-06-22T07:37:24.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-06-22T07:37:24.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T07:37:58.896 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2023-06-22T07:38:28.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-06-22T07:38:28.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T07:39:28.902 DEBUG:teuthology.orchestra.remote:timed out 2023-06-22T07:39:58.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-06-22T07:39:58.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T07:39:59.251 DEBUG:teuthology.orchestra.run.smithi023:> true 2023-06-22T07:39:59.656 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-06-22T07:39:59.657 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.4.0-rc5-g1c622d92b6f1"... 2023-06-22T07:39:59.657 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2023-06-22T07:39:59.699 INFO:teuthology.orchestra.run.smithi023.stdout:6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:39:59.700 DEBUG:teuthology.task.kernel:current kernel version is 6.4.0-rc5-g1c622d92b6f1 vs 6.4.0-rc5-g1c622d92b6f1 2023-06-22T07:39:59.700 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-22T07:39:59.700 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-22T07:40:00.702 DEBUG:teuthology.parallel:result is None 2023-06-22T07:40:00.703 INFO:teuthology.run_tasks:Running task internal.base... 2023-06-22T07:40:00.712 INFO:teuthology.task.internal:Creating test directory... 2023-06-22T07:40:00.713 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-22T07:40:00.718 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-22T07:40:00.760 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-06-22T07:40:00.773 INFO:teuthology.run_tasks:Running task internal.archive... 2023-06-22T07:40:00.785 INFO:teuthology.task.internal:Creating archive directory... 2023-06-22T07:40:00.786 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-22T07:40:00.808 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-22T07:40:00.864 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-06-22T07:40:00.873 INFO:teuthology.task.internal:Enabling coredump saving... 2023-06-22T07:40:00.874 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-22T07:40:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T07:40:00.900 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-22T07:40:00.905 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-22T07:40:00.994 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T07:40:01.013 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T07:40:01.032 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T07:40:01.064 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T07:40:01.067 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-06-22T07:40:01.074 INFO:teuthology.task.internal:Configuring sudo... 2023-06-22T07:40:01.074 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-22T07:40:01.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-22T07:40:01.143 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-06-22T07:40:01.153 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-06-22T07:40:01.154 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-22T07:40:01.159 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-22T07:40:01.197 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T07:40:01.311 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T07:40:01.411 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T07:40:01.482 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T07:40:01.564 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:40:01.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-22T07:40:01.632 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T07:40:01.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T07:40:01.739 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T07:40:01.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T07:40:01.872 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:40:01.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-22T07:40:01.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2023-06-22T07:40:01.963 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2023-06-22T07:40:02.067 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T07:40:02.080 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T07:40:02.309 INFO:teuthology.run_tasks:Running task internal.timer... 2023-06-22T07:40:02.320 INFO:teuthology.task.internal:Starting timer... 2023-06-22T07:40:02.321 INFO:teuthology.run_tasks:Running task pcp... 2023-06-22T07:40:02.353 INFO:teuthology.run_tasks:Running task selinux... 2023-06-22T07:40:02.370 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-06-22T07:40:02.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2023-06-22T07:40:02.495 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: 2023-06-22T07:40:02.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2023-06-22T07:40:02.572 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2023-06-22T07:40:02.575 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-06-22T07:40:02.575 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2023-06-22T07:40:02.627 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2023-06-22T07:40:02.628 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2023-06-22T07:40:02.651 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2023-06-22T07:40:02.652 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi067.front.sepia.ceph.com': 'permissive'} 2023-06-22T07:40:02.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T07:40:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T07:40:02.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T07:40:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T07:40:02.753 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-06-22T07:40:02.754 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2023-06-22T07:40:02.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2023-06-22T07:40:02.885 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-06-22T07:40:02.895 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-06-22T07:40:02.896 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-06-22T07:40:02.927 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-06-22T07:40:02.928 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi067.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-06-22T07:51:42.384 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi067.front.sepia.ceph.com')] 2023-06-22T07:51:42.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-06-22T07:51:42.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T07:51:42.503 DEBUG:teuthology.orchestra.run.smithi023:> true 2023-06-22T07:51:42.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-06-22T07:51:42.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T07:51:42.682 DEBUG:teuthology.orchestra.run.smithi067:> true 2023-06-22T07:51:42.757 INFO:teuthology.run_tasks:Running task clock... 2023-06-22T07:51:42.769 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-06-22T07:51:42.770 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-22T07:51:42.770 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T07:51:42.774 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-22T07:51:42.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T07:51:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-22T07:51:42.844 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-22T07:51:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-22T07:51:42.880 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-22T07:51:42.923 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2023-06-22T07:51:42.940 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2023-06-22T07:51:42.952 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-22T07:51:42.967 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-22T07:51:42.971 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2023-06-22T07:51:43.002 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2023-06-22T07:51:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-22T07:51:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:bash: ntpq: command not found 2023-06-22T07:51:43.053 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T07:51:43.054 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2023-06-22T07:51:43.054 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T07:51:43.054 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T07:51:43.054 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T07:51:43.054 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T07:51:43.070 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-22T07:51:43.171 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2023-06-22T07:51:43.178 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T07:51:43.178 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2023-06-22T07:51:43.178 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T07:51:43.179 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T07:51:43.179 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T07:51:43.179 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T07:51:43.180 INFO:teuthology.run_tasks:Running task install... 2023-06-22T07:51:43.188 DEBUG:teuthology.task.install:project ceph 2023-06-22T07:51:43.188 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8627af3e0adcb765a3c249fcc209cba9f4873e1b'}} 2023-06-22T07:51:43.189 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8627af3e0adcb765a3c249fcc209cba9f4873e1b'} 2023-06-22T07:51:43.189 INFO:teuthology.task.install:Using flavor: default 2023-06-22T07:51:43.198 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-06-22T07:51:43.198 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-06-22T07:51:43.199 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8627af3e0adcb765a3c249fcc209cba9f4873e1b', 'tag': None, 'wait_for_package': False} 2023-06-22T07:51:43.199 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8627af3e0adcb765a3c249fcc209cba9f4873e1b 2023-06-22T07:51:43.201 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8627af3e0adcb765a3c249fcc209cba9f4873e1b', 'tag': None, 'wait_for_package': False} 2023-06-22T07:51:43.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8627af3e0adcb765a3c249fcc209cba9f4873e1b 2023-06-22T07:51:43.355 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-improvements-authmon/8627af3e0adcb765a3c249fcc209cba9f4873e1b/centos/8/flavors/default/ 2023-06-22T07:51:43.355 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4479.g8627af3e 2023-06-22T07:51:43.502 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-improvements-authmon/8627af3e0adcb765a3c249fcc209cba9f4873e1b/centos/8/flavors/default/ 2023-06-22T07:51:43.503 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4479.g8627af3e 2023-06-22T07:51:43.605 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-improvements-authmon/8627af3e0adcb765a3c249fcc209cba9f4873e1b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-improvements-authmon/8627af3e0adcb765a3c249fcc209cba9f4873e1b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-improvements-authmon/8627af3e0adcb765a3c249fcc209cba9f4873e1b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-22T07:51:43.606 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:51:43.606 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-22T07:51:43.608 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-improvements-authmon/8627af3e0adcb765a3c249fcc209cba9f4873e1b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-improvements-authmon/8627af3e0adcb765a3c249fcc209cba9f4873e1b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-improvements-authmon/8627af3e0adcb765a3c249fcc209cba9f4873e1b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-22T07:51:43.609 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:51:43.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-22T07:51:43.641 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-06-22T07:51:43.641 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8627af3e0adcb765a3c249fcc209cba9f4873e1b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-22T07:51:43.678 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-06-22T07:51:43.679 DEBUG:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8627af3e0adcb765a3c249fcc209cba9f4873e1b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-22T07:51:43.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-22T07:51:43.788 DEBUG:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-22T07:51:43.791 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-22T07:51:43.858 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2023-06-22T07:51:43.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2023-06-22T07:51:43.883 DEBUG:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-22T07:51:43.950 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2023-06-22T07:51:43.953 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum clean all 2023-06-22T07:51:44.252 INFO:teuthology.orchestra.run.smithi067.stdout:Updating Subscription Management repositories. 2023-06-22T07:51:44.429 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2023-06-22T07:51:44.520 INFO:teuthology.orchestra.run.smithi067.stdout:44 files removed 2023-06-22T07:51:44.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-06-22T07:51:44.731 INFO:teuthology.orchestra.run.smithi023.stdout:44 files removed 2023-06-22T07:51:44.786 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-06-22T07:51:44.920 INFO:teuthology.orchestra.run.smithi067.stdout:Updating Subscription Management repositories. 2023-06-22T07:51:45.258 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2023-06-22T07:51:45.874 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 154 kB/s | 81 kB 00:00 2023-06-22T07:51:46.269 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 136 kB/s | 81 kB 00:00 2023-06-22T07:51:46.416 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 19 kB/s | 9.4 kB 00:00 2023-06-22T07:51:46.794 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 19 kB/s | 9.4 kB 00:00 2023-06-22T07:51:46.834 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 3.8 kB/s | 1.5 kB 00:00 2023-06-22T07:51:47.312 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 3.0 kB/s | 1.5 kB 00:00 2023-06-22T07:51:49.201 INFO:teuthology.orchestra.run.smithi067.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-06-22T07:51:49.937 INFO:teuthology.orchestra.run.smithi023.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2023-06-22T07:56:11.807 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 61 kB/s | 16 MB 04:21 2023-06-22T07:56:16.201 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 164 kB/s | 24 kB 00:00 2023-06-22T07:56:17.032 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 60 kB/s | 16 MB 04:27 2023-06-22T07:56:17.391 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 53 MB 00:01 2023-06-22T07:56:20.640 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 604 kB/s | 24 kB 00:00 2023-06-22T07:56:21.600 INFO:teuthology.orchestra.run.smithi067.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 MB/s | 53 MB 00:00 2023-06-22T07:56:26.624 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 47 MB/s | 47 MB 00:01 2023-06-22T07:56:30.276 INFO:teuthology.orchestra.run.smithi067.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 55 MB/s | 47 MB 00:00 2023-06-22T07:56:38.928 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T07:56:38.929 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T07:56:38.945 INFO:teuthology.orchestra.run.smithi023.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-06-22T07:56:38.946 INFO:teuthology.orchestra.run.smithi023.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-06-22T07:56:38.948 INFO:teuthology.orchestra.run.smithi023.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-06-22T07:56:38.949 INFO:teuthology.orchestra.run.smithi023.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-22T07:56:38.951 INFO:teuthology.orchestra.run.smithi023.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-22T07:56:38.955 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-06-22T07:56:38.956 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-06-22T07:56:39.106 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-06-22T07:56:39.119 INFO:teuthology.orchestra.run.smithi023.stdout:=========================================================================================================================== 2023-06-22T07:56:39.119 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2023-06-22T07:56:39.120 INFO:teuthology.orchestra.run.smithi023.stdout:=========================================================================================================================== 2023-06-22T07:56:39.120 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-06-22T07:56:39.120 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-06-22T07:56:39.120 INFO:teuthology.orchestra.run.smithi023.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-22T07:56:39.120 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 6.4 k 2023-06-22T07:56:39.120 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 5.4 M 2023-06-22T07:56:39.120 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 848 k 2023-06-22T07:56:39.121 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 165 k 2023-06-22T07:56:39.121 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 1.5 M 2023-06-22T07:56:39.121 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 128 k 2023-06-22T07:56:39.121 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 1.7 M 2023-06-22T07:56:39.121 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 7.4 M 2023-06-22T07:56:39.121 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 47 k 2023-06-22T07:56:39.121 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 13 M 2023-06-22T07:56:39.122 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 48 M 2023-06-22T07:56:39.122 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 258 k 2023-06-22T07:56:39.122 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 161 k 2023-06-22T07:56:39.122 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 258 k 2023-06-22T07:56:39.122 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 18 k 2023-06-22T07:56:39.122 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-06-22T07:56:39.122 INFO:teuthology.orchestra.run.smithi023.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-22T07:56:39.122 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-22T07:56:39.123 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 31 k 2023-06-22T07:56:39.123 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 771 k 2023-06-22T07:56:39.123 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 139 k 2023-06-22T07:56:39.123 INFO:teuthology.orchestra.run.smithi023.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-22T07:56:39.123 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-06-22T07:56:39.123 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-22T07:56:39.123 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-06-22T07:56:39.124 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-06-22T07:56:39.124 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-06-22T07:56:39.124 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 197 k 2023-06-22T07:56:39.124 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 370 k 2023-06-22T07:56:39.124 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 365 k 2023-06-22T07:56:39.124 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 107 k 2023-06-22T07:56:39.124 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 94 k 2023-06-22T07:56:39.125 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 3.4 M 2023-06-22T07:56:39.125 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 179 k 2023-06-22T07:56:39.125 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-06-22T07:56:39.125 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2023-06-22T07:56:39.125 INFO:teuthology.orchestra.run.smithi023.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-06-22T07:56:39.125 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 3.7 M 2023-06-22T07:56:39.125 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 3.6 M 2023-06-22T07:56:39.125 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-06-22T07:56:39.126 INFO:teuthology.orchestra.run.smithi023.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-22T07:56:39.126 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 22 M 2023-06-22T07:56:39.126 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 23 k 2023-06-22T07:56:39.126 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 2.3 M 2023-06-22T07:56:39.126 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 240 k 2023-06-22T07:56:39.126 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 4.6 M 2023-06-22T07:56:39.126 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 18 M 2023-06-22T07:56:39.127 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 15 k 2023-06-22T07:56:39.127 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 24 k 2023-06-22T07:56:39.127 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-22T07:56:39.127 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-22T07:56:39.127 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-06-22T07:56:39.127 INFO:teuthology.orchestra.run.smithi023.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-22T07:56:39.127 INFO:teuthology.orchestra.run.smithi023.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-06-22T07:56:39.127 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 137 k 2023-06-22T07:56:39.128 INFO:teuthology.orchestra.run.smithi023.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-06-22T07:56:39.128 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-06-22T07:56:39.128 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-22T07:56:39.128 INFO:teuthology.orchestra.run.smithi023.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-06-22T07:56:39.128 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-06-22T07:56:39.128 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-22T07:56:39.128 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 474 k 2023-06-22T07:56:39.129 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-06-22T07:56:39.129 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 6.7 M 2023-06-22T07:56:39.129 INFO:teuthology.orchestra.run.smithi023.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-22T07:56:39.129 INFO:teuthology.orchestra.run.smithi023.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-06-22T07:56:39.129 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-22T07:56:39.129 INFO:teuthology.orchestra.run.smithi023.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-22T07:56:39.129 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-22T07:56:39.129 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-22T07:56:39.130 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-22T07:56:39.130 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-06-22T07:56:39.130 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-22T07:56:39.130 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-22T07:56:39.130 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-22T07:56:39.130 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-22T07:56:39.130 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-22T07:56:39.130 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-22T07:56:39.131 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-22T07:56:39.131 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-22T07:56:39.131 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 45 k 2023-06-22T07:56:39.131 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 119 k 2023-06-22T07:56:39.131 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-22T07:56:39.131 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-06-22T07:56:39.131 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-22T07:56:39.132 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-22T07:56:39.132 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-22T07:56:39.132 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-22T07:56:39.132 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-22T07:56:39.132 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-22T07:56:39.132 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-22T07:56:39.132 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-22T07:56:39.132 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-22T07:56:39.133 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T07:56:39.133 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-22T07:56:39.133 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-22T07:56:39.133 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-22T07:56:39.133 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-22T07:56:39.133 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-22T07:56:39.133 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-22T07:56:39.133 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-22T07:56:39.134 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-22T07:56:39.134 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-22T07:56:39.134 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-22T07:56:39.134 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-22T07:56:39.134 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-22T07:56:39.134 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-06-22T07:56:39.134 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-22T07:56:39.135 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-22T07:56:39.135 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-06-22T07:56:39.135 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-22T07:56:39.135 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-22T07:56:39.135 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T07:56:39.135 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-22T07:56:39.135 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-22T07:56:39.135 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-06-22T07:56:39.136 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-22T07:56:39.136 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-22T07:56:39.136 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-22T07:56:39.136 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-22T07:56:39.136 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-22T07:56:39.136 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-22T07:56:39.136 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-22T07:56:39.136 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-22T07:56:39.137 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-22T07:56:39.137 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-22T07:56:39.137 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-06-22T07:56:39.137 INFO:teuthology.orchestra.run.smithi023.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-22T07:56:39.137 INFO:teuthology.orchestra.run.smithi023.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-06-22T07:56:39.137 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-22T07:56:39.137 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-22T07:56:39.137 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling module streams: 2023-06-22T07:56:39.138 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql 10 2023-06-22T07:56:39.138 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:56:39.138 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-06-22T07:56:39.138 INFO:teuthology.orchestra.run.smithi023.stdout:=========================================================================================================================== 2023-06-22T07:56:39.138 INFO:teuthology.orchestra.run.smithi023.stdout:Install 128 Packages 2023-06-22T07:56:39.138 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 3 Packages 2023-06-22T07:56:39.138 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:56:39.139 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 205 M 2023-06-22T07:56:39.139 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-06-22T07:56:39.649 INFO:teuthology.orchestra.run.smithi023.stdout:(1/131): ceph-18.0.0-4479.g8627af3e.el8.x86_64. 34 kB/s | 6.4 kB 00:00 2023-06-22T07:56:40.041 INFO:teuthology.orchestra.run.smithi023.stdout:(2/131): ceph-base-18.0.0-4479.g8627af3e.el8.x8 9.3 MB/s | 5.4 MB 00:00 2023-06-22T07:56:40.108 INFO:teuthology.orchestra.run.smithi023.stdout:(3/131): ceph-immutable-object-cache-18.0.0-447 2.4 MB/s | 165 kB 00:00 2023-06-22T07:56:40.267 INFO:teuthology.orchestra.run.smithi023.stdout:(4/131): ceph-fuse-18.0.0-4479.g8627af3e.el8.x8 1.3 MB/s | 848 kB 00:00 2023-06-22T07:56:40.350 INFO:teuthology.orchestra.run.smithi023.stdout:(5/131): ceph-mds-18.0.0-4479.g8627af3e.el8.x86 9.5 MB/s | 2.3 MB 00:00 2023-06-22T07:56:40.801 INFO:teuthology.orchestra.run.smithi023.stdout:(6/131): ceph-mon-18.0.0-4479.g8627af3e.el8.x86 10 MB/s | 4.6 MB 00:00 2023-06-22T07:56:41.251 INFO:teuthology.orchestra.run.smithi023.stdout:(7/131): ceph-common-18.0.0-4479.g8627af3e.el8. 12 MB/s | 22 MB 00:01 2023-06-22T07:56:41.284 INFO:teuthology.orchestra.run.smithi023.stdout:(8/131): ceph-mgr-18.0.0-4479.g8627af3e.el8.x86 1.5 MB/s | 1.5 MB 00:01 2023-06-22T07:56:41.360 INFO:teuthology.orchestra.run.smithi023.stdout:(9/131): ceph-selinux-18.0.0-4479.g8627af3e.el8 325 kB/s | 24 kB 00:00 2023-06-22T07:56:42.088 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T07:56:42.088 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T07:56:42.103 INFO:teuthology.orchestra.run.smithi067.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-06-22T07:56:42.104 INFO:teuthology.orchestra.run.smithi067.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-06-22T07:56:42.105 INFO:teuthology.orchestra.run.smithi067.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-06-22T07:56:42.106 INFO:teuthology.orchestra.run.smithi067.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-22T07:56:42.109 INFO:teuthology.orchestra.run.smithi067.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-22T07:56:42.112 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-06-22T07:56:42.113 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-06-22T07:56:42.257 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-06-22T07:56:42.270 INFO:teuthology.orchestra.run.smithi067.stdout:=========================================================================================================================== 2023-06-22T07:56:42.270 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-06-22T07:56:42.270 INFO:teuthology.orchestra.run.smithi067.stdout:=========================================================================================================================== 2023-06-22T07:56:42.271 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-06-22T07:56:42.271 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-06-22T07:56:42.271 INFO:teuthology.orchestra.run.smithi067.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-22T07:56:42.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 6.4 k 2023-06-22T07:56:42.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 5.4 M 2023-06-22T07:56:42.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 848 k 2023-06-22T07:56:42.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 165 k 2023-06-22T07:56:42.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 1.5 M 2023-06-22T07:56:42.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 128 k 2023-06-22T07:56:42.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 1.7 M 2023-06-22T07:56:42.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 7.4 M 2023-06-22T07:56:42.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 47 k 2023-06-22T07:56:42.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 13 M 2023-06-22T07:56:42.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 48 M 2023-06-22T07:56:42.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 258 k 2023-06-22T07:56:42.273 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 161 k 2023-06-22T07:56:42.273 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 258 k 2023-06-22T07:56:42.273 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 18 k 2023-06-22T07:56:42.273 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-06-22T07:56:42.273 INFO:teuthology.orchestra.run.smithi067.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-22T07:56:42.273 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-22T07:56:42.273 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 31 k 2023-06-22T07:56:42.274 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 771 k 2023-06-22T07:56:42.274 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 139 k 2023-06-22T07:56:42.274 INFO:teuthology.orchestra.run.smithi067.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-22T07:56:42.274 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-06-22T07:56:42.274 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-22T07:56:42.274 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-06-22T07:56:42.274 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-06-22T07:56:42.274 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-06-22T07:56:42.275 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 197 k 2023-06-22T07:56:42.275 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 370 k 2023-06-22T07:56:42.275 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 365 k 2023-06-22T07:56:42.275 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 107 k 2023-06-22T07:56:42.275 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 94 k 2023-06-22T07:56:42.275 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 3.4 M 2023-06-22T07:56:42.275 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 179 k 2023-06-22T07:56:42.276 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-06-22T07:56:42.276 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2023-06-22T07:56:42.276 INFO:teuthology.orchestra.run.smithi067.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-06-22T07:56:42.276 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 3.7 M 2023-06-22T07:56:42.276 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 3.6 M 2023-06-22T07:56:42.276 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-06-22T07:56:42.276 INFO:teuthology.orchestra.run.smithi067.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-22T07:56:42.276 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 22 M 2023-06-22T07:56:42.277 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 23 k 2023-06-22T07:56:42.277 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 2.3 M 2023-06-22T07:56:42.277 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 240 k 2023-06-22T07:56:42.277 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 4.6 M 2023-06-22T07:56:42.277 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 18 M 2023-06-22T07:56:42.277 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4479.g8627af3e.el8 ceph-noarch 15 k 2023-06-22T07:56:42.277 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 24 k 2023-06-22T07:56:42.278 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-22T07:56:42.278 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-22T07:56:42.278 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-06-22T07:56:42.278 INFO:teuthology.orchestra.run.smithi067.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-22T07:56:42.278 INFO:teuthology.orchestra.run.smithi067.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-06-22T07:56:42.278 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 137 k 2023-06-22T07:56:42.278 INFO:teuthology.orchestra.run.smithi067.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-06-22T07:56:42.278 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-06-22T07:56:42.279 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-22T07:56:42.279 INFO:teuthology.orchestra.run.smithi067.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-06-22T07:56:42.279 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-06-22T07:56:42.279 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-22T07:56:42.279 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 474 k 2023-06-22T07:56:42.279 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-06-22T07:56:42.279 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 6.7 M 2023-06-22T07:56:42.280 INFO:teuthology.orchestra.run.smithi067.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-22T07:56:42.280 INFO:teuthology.orchestra.run.smithi067.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-06-22T07:56:42.280 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-22T07:56:42.280 INFO:teuthology.orchestra.run.smithi067.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-22T07:56:42.280 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-22T07:56:42.280 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-22T07:56:42.280 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-22T07:56:42.280 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-06-22T07:56:42.281 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-22T07:56:42.281 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-22T07:56:42.281 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-22T07:56:42.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-22T07:56:42.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-22T07:56:42.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-22T07:56:42.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-22T07:56:42.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-22T07:56:42.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 45 k 2023-06-22T07:56:42.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:18.0.0-4479.g8627af3e.el8 ceph 119 k 2023-06-22T07:56:42.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-22T07:56:42.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-06-22T07:56:42.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-22T07:56:42.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-22T07:56:42.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-22T07:56:42.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-22T07:56:42.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-22T07:56:42.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-22T07:56:42.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-22T07:56:42.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-22T07:56:42.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-22T07:56:42.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T07:56:42.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-22T07:56:42.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-22T07:56:42.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-22T07:56:42.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-22T07:56:42.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-22T07:56:42.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-22T07:56:42.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-22T07:56:42.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-22T07:56:42.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-22T07:56:42.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-22T07:56:42.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-22T07:56:42.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-22T07:56:42.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-06-22T07:56:42.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-22T07:56:42.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-22T07:56:42.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-06-22T07:56:42.286 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-22T07:56:42.286 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-22T07:56:42.286 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T07:56:42.286 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-22T07:56:42.286 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-22T07:56:42.286 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-06-22T07:56:42.286 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-22T07:56:42.286 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-22T07:56:42.287 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-22T07:56:42.287 INFO:teuthology.orchestra.run.smithi067.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-22T07:56:42.287 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-22T07:56:42.287 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-22T07:56:42.287 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-22T07:56:42.287 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-22T07:56:42.287 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-22T07:56:42.287 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-22T07:56:42.288 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-06-22T07:56:42.288 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-22T07:56:42.288 INFO:teuthology.orchestra.run.smithi067.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-06-22T07:56:42.288 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-22T07:56:42.288 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-22T07:56:42.288 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling module streams: 2023-06-22T07:56:42.288 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql 10 2023-06-22T07:56:42.289 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:56:42.289 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-06-22T07:56:42.289 INFO:teuthology.orchestra.run.smithi067.stdout:=========================================================================================================================== 2023-06-22T07:56:42.289 INFO:teuthology.orchestra.run.smithi067.stdout:Install 128 Packages 2023-06-22T07:56:42.289 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 3 Packages 2023-06-22T07:56:42.289 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:56:42.290 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 205 M 2023-06-22T07:56:42.290 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-06-22T07:56:42.393 INFO:teuthology.orchestra.run.smithi023.stdout:(10/131): ceph-osd-18.0.0-4479.g8627af3e.el8.x8 11 MB/s | 18 MB 00:01 2023-06-22T07:56:42.543 INFO:teuthology.orchestra.run.smithi067.stdout:(1/131): ceph-18.0.0-4479.g8627af3e.el8.x86_64. 34 kB/s | 6.4 kB 00:00 2023-06-22T07:56:42.552 INFO:teuthology.orchestra.run.smithi023.stdout:(11/131): ceph-radosgw-18.0.0-4479.g8627af3e.el 10 MB/s | 13 MB 00:01 2023-06-22T07:56:42.577 INFO:teuthology.orchestra.run.smithi023.stdout:(12/131): cephfs-mirror-18.0.0-4479.g8627af3e.e 1.4 MB/s | 258 kB 00:00 2023-06-22T07:56:42.602 INFO:teuthology.orchestra.run.smithi023.stdout:(13/131): libcephfs-devel-18.0.0-4479.g8627af3e 624 kB/s | 31 kB 00:00 2023-06-22T07:56:42.669 INFO:teuthology.orchestra.run.smithi023.stdout:(14/131): libcephfs2-18.0.0-4479.g8627af3e.el8. 8.2 MB/s | 771 kB 00:00 2023-06-22T07:56:42.695 INFO:teuthology.orchestra.run.smithi023.stdout:(15/131): libcephsqlite-18.0.0-4479.g8627af3e.e 1.5 MB/s | 137 kB 00:00 2023-06-22T07:56:42.737 INFO:teuthology.orchestra.run.smithi023.stdout:(16/131): librados-devel-18.0.0-4479.g8627af3e. 2.0 MB/s | 139 kB 00:00 2023-06-22T07:56:42.821 INFO:teuthology.orchestra.run.smithi023.stdout:(17/131): libradosstriper1-18.0.0-4479.g8627af3 3.7 MB/s | 474 kB 00:00 2023-06-22T07:56:42.904 INFO:teuthology.orchestra.run.smithi023.stdout:(18/131): python3-ceph-argparse-18.0.0-4479.g86 538 kB/s | 45 kB 00:00 2023-06-22T07:56:43.004 INFO:teuthology.orchestra.run.smithi023.stdout:(19/131): python3-ceph-common-18.0.0-4479.g8627 1.2 MB/s | 119 kB 00:00 2023-06-22T07:56:43.104 INFO:teuthology.orchestra.run.smithi023.stdout:(20/131): python3-cephfs-18.0.0-4479.g8627af3e. 1.9 MB/s | 197 kB 00:00 2023-06-22T07:56:43.221 INFO:teuthology.orchestra.run.smithi023.stdout:(21/131): python3-rados-18.0.0-4479.g8627af3e.e 3.1 MB/s | 370 kB 00:00 2023-06-22T07:56:43.330 INFO:teuthology.orchestra.run.smithi023.stdout:(22/131): python3-rbd-18.0.0-4479.g8627af3e.el8 3.3 MB/s | 365 kB 00:00 2023-06-22T07:56:43.464 INFO:teuthology.orchestra.run.smithi023.stdout:(23/131): librgw2-18.0.0-4479.g8627af3e.el8.x86 9.3 MB/s | 6.7 MB 00:00 2023-06-22T07:56:43.489 INFO:teuthology.orchestra.run.smithi023.stdout:(24/131): python3-rgw-18.0.0-4479.g8627af3e.el8 672 kB/s | 107 kB 00:00 2023-06-22T07:56:43.510 INFO:teuthology.orchestra.run.smithi067.stdout:(2/131): ceph-fuse-18.0.0-4479.g8627af3e.el8.x8 877 kB/s | 848 kB 00:00 2023-06-22T07:56:43.523 INFO:teuthology.orchestra.run.smithi023.stdout:(25/131): rbd-fuse-18.0.0-4479.g8627af3e.el8.x8 1.6 MB/s | 94 kB 00:00 2023-06-22T07:56:43.623 INFO:teuthology.orchestra.run.smithi023.stdout:(26/131): rbd-nbd-18.0.0-4479.g8627af3e.el8.x86 1.8 MB/s | 179 kB 00:00 2023-06-22T07:56:43.685 INFO:teuthology.orchestra.run.smithi067.stdout:(3/131): ceph-immutable-object-cache-18.0.0-447 946 kB/s | 165 kB 00:00 2023-06-22T07:56:43.723 INFO:teuthology.orchestra.run.smithi023.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4479.g 231 kB/s | 23 kB 00:00 2023-06-22T07:56:43.849 INFO:teuthology.orchestra.run.smithi023.stdout:(28/131): rbd-mirror-18.0.0-4479.g8627af3e.el8. 9.6 MB/s | 3.4 MB 00:00 2023-06-22T07:56:43.874 INFO:teuthology.orchestra.run.smithi023.stdout:(29/131): ceph-mgr-cephadm-18.0.0-4479.g8627af3 851 kB/s | 128 kB 00:00 2023-06-22T07:56:43.952 INFO:teuthology.orchestra.run.smithi067.stdout:(4/131): ceph-base-18.0.0-4479.g8627af3e.el8.x8 3.4 MB/s | 5.4 MB 00:01 2023-06-22T07:56:44.024 INFO:teuthology.orchestra.run.smithi023.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4479.g8627a 9.8 MB/s | 1.7 MB 00:00 2023-06-22T07:56:44.124 INFO:teuthology.orchestra.run.smithi023.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4479.g86 2.4 MB/s | 240 kB 00:00 2023-06-22T07:56:44.208 INFO:teuthology.orchestra.run.smithi023.stdout:(32/131): ceph-mgr-rook-18.0.0-4479.g8627af3e.e 570 kB/s | 47 kB 00:00 2023-06-22T07:56:44.292 INFO:teuthology.orchestra.run.smithi023.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4479.g8 174 kB/s | 15 kB 00:00 2023-06-22T07:56:44.369 INFO:teuthology.orchestra.run.smithi067.stdout:(5/131): ceph-mgr-18.0.0-4479.g8627af3e.el8.x86 3.7 MB/s | 1.5 MB 00:00 2023-06-22T07:56:44.383 INFO:teuthology.orchestra.run.smithi023.stdout:(34/131): ceph-volume-18.0.0-4479.g8627af3e.el8 2.8 MB/s | 258 kB 00:00 2023-06-22T07:56:44.475 INFO:teuthology.orchestra.run.smithi023.stdout:(35/131): cephadm-18.0.0-4479.g8627af3e.el8.noa 1.7 MB/s | 161 kB 00:00 2023-06-22T07:56:44.559 INFO:teuthology.orchestra.run.smithi023.stdout:(36/131): cephfs-top-18.0.0-4479.g8627af3e.el8. 215 kB/s | 18 kB 00:00 2023-06-22T07:56:44.718 INFO:teuthology.orchestra.run.smithi023.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.9 MB/s | 306 kB 00:00 2023-06-22T07:56:44.835 INFO:teuthology.orchestra.run.smithi023.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 7.7 MB/s | 7.4 MB 00:00 2023-06-22T07:56:44.860 INFO:teuthology.orchestra.run.smithi023.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 414 kB/s | 59 kB 00:00 2023-06-22T07:56:44.903 INFO:teuthology.orchestra.run.smithi067.stdout:(6/131): ceph-mds-18.0.0-4479.g8627af3e.el8.x86 1.9 MB/s | 2.3 MB 00:01 2023-06-22T07:56:44.952 INFO:teuthology.orchestra.run.smithi023.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 643 kB/s | 75 kB 00:00 2023-06-22T07:56:45.003 INFO:teuthology.orchestra.run.smithi023.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 870 kB/s | 44 kB 00:00 2023-06-22T07:56:45.028 INFO:teuthology.orchestra.run.smithi023.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.8 MB/s | 476 kB 00:00 2023-06-22T07:56:45.080 INFO:teuthology.orchestra.run.smithi023.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 632 kB/s | 33 kB 00:00 2023-06-22T07:56:45.105 INFO:teuthology.orchestra.run.smithi023.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.8 MB/s | 185 kB 00:00 2023-06-22T07:56:45.131 INFO:teuthology.orchestra.run.smithi023.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 325 kB/s | 16 kB 00:00 2023-06-22T07:56:45.187 INFO:teuthology.orchestra.run.smithi067.stdout:(7/131): ceph-common-18.0.0-4479.g8627af3e.el8. 7.8 MB/s | 22 MB 00:02 2023-06-22T07:56:45.189 INFO:teuthology.orchestra.run.smithi023.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.0 MB/s | 173 kB 00:00 2023-06-22T07:56:45.240 INFO:teuthology.orchestra.run.smithi023.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2023-06-22T07:56:45.265 INFO:teuthology.orchestra.run.smithi023.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.8 MB/s | 384 kB 00:00 2023-06-22T07:56:45.290 INFO:teuthology.orchestra.run.smithi023.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-06-22T07:56:45.295 INFO:teuthology.orchestra.run.smithi067.stdout:(8/131): ceph-mon-18.0.0-4479.g8627af3e.el8.x86 5.0 MB/s | 4.6 MB 00:00 2023-06-22T07:56:45.307 INFO:teuthology.orchestra.run.smithi023.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 420 kB/s | 18 kB 00:00 2023-06-22T07:56:45.358 INFO:teuthology.orchestra.run.smithi023.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 977 kB/s | 49 kB 00:00 2023-06-22T07:56:45.364 INFO:teuthology.orchestra.run.smithi067.stdout:(9/131): ceph-selinux-18.0.0-4479.g8627af3e.el8 365 kB/s | 24 kB 00:00 2023-06-22T07:56:45.408 INFO:teuthology.orchestra.run.smithi023.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-06-22T07:56:45.458 INFO:teuthology.orchestra.run.smithi023.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-06-22T07:56:45.550 INFO:teuthology.orchestra.run.smithi023.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 4.5 MB/s | 1.2 MB 00:00 2023-06-22T07:56:45.575 INFO:teuthology.orchestra.run.smithi023.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2023-06-22T07:56:45.592 INFO:teuthology.orchestra.run.smithi023.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 372 kB/s | 16 kB 00:00 2023-06-22T07:56:45.618 INFO:teuthology.orchestra.run.smithi023.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 802 kB/s | 34 kB 00:00 2023-06-22T07:56:45.668 INFO:teuthology.orchestra.run.smithi023.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 2.6 MB/s | 196 kB 00:00 2023-06-22T07:56:45.685 INFO:teuthology.orchestra.run.smithi023.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 921 kB/s | 61 kB 00:00 2023-06-22T07:56:45.702 INFO:teuthology.orchestra.run.smithi023.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 558 kB/s | 19 kB 00:00 2023-06-22T07:56:45.727 INFO:teuthology.orchestra.run.smithi023.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 570 kB/s | 24 kB 00:00 2023-06-22T07:56:45.744 INFO:teuthology.orchestra.run.smithi023.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 698 kB/s | 29 kB 00:00 2023-06-22T07:56:45.770 INFO:teuthology.orchestra.run.smithi023.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 658 kB/s | 27 kB 00:00 2023-06-22T07:56:45.812 INFO:teuthology.orchestra.run.smithi023.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 3.7 MB/s | 255 kB 00:00 2023-06-22T07:56:45.854 INFO:teuthology.orchestra.run.smithi023.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 1.4 MB/s | 61 kB 00:00 2023-06-22T07:56:45.904 INFO:teuthology.orchestra.run.smithi023.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.8 MB/s | 251 kB 00:00 2023-06-22T07:56:45.929 INFO:teuthology.orchestra.run.smithi023.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2023-06-22T07:56:45.954 INFO:teuthology.orchestra.run.smithi023.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 456 kB/s | 23 kB 00:00 2023-06-22T07:56:45.988 INFO:teuthology.orchestra.run.smithi023.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2023-06-22T07:56:46.047 INFO:teuthology.orchestra.run.smithi023.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 804 kB/s | 47 kB 00:00 2023-06-22T07:56:46.089 INFO:teuthology.orchestra.run.smithi023.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-06-22T07:56:46.123 INFO:teuthology.orchestra.run.smithi023.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-06-22T07:56:46.165 INFO:teuthology.orchestra.run.smithi023.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-06-22T07:56:46.199 INFO:teuthology.orchestra.run.smithi023.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.4 MB/s | 48 kB 00:00 2023-06-22T07:56:46.240 INFO:teuthology.orchestra.run.smithi023.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-06-22T07:56:46.274 INFO:teuthology.orchestra.run.smithi023.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.3 MB/s | 43 kB 00:00 2023-06-22T07:56:46.316 INFO:teuthology.orchestra.run.smithi023.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 2.6 MB/s | 109 kB 00:00 2023-06-22T07:56:46.358 INFO:teuthology.orchestra.run.smithi023.stdout:(78/131): thrift-0.13.0-2.el8.x86_64.rpm 4.1 MB/s | 1.7 MB 00:00 2023-06-22T07:56:46.392 INFO:teuthology.orchestra.run.smithi023.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.0 MB/s | 229 kB 00:00 2023-06-22T07:56:46.442 INFO:teuthology.orchestra.run.smithi023.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 450 kB/s | 19 kB 00:00 2023-06-22T07:56:46.484 INFO:teuthology.orchestra.run.smithi023.stdout:(81/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.4 MB/s | 560 kB 00:00 2023-06-22T07:56:46.496 INFO:teuthology.orchestra.run.smithi067.stdout:(10/131): ceph-radosgw-18.0.0-4479.g8627af3e.el 10 MB/s | 13 MB 00:01 2023-06-22T07:56:46.509 INFO:teuthology.orchestra.run.smithi023.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 2.3 MB/s | 155 kB 00:00 2023-06-22T07:56:46.543 INFO:teuthology.orchestra.run.smithi023.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 9.3 MB/s | 559 kB 00:00 2023-06-22T07:56:46.563 INFO:teuthology.orchestra.run.smithi067.stdout:(11/131): cephfs-mirror-18.0.0-4479.g8627af3e.e 3.8 MB/s | 258 kB 00:00 2023-06-22T07:56:46.568 INFO:teuthology.orchestra.run.smithi023.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 304 kB/s | 18 kB 00:00 2023-06-22T07:56:46.620 INFO:teuthology.orchestra.run.smithi023.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 7.9 MB/s | 605 kB 00:00 2023-06-22T07:56:46.621 INFO:teuthology.orchestra.run.smithi067.stdout:(12/131): libcephfs-devel-18.0.0-4479.g8627af3e 534 kB/s | 31 kB 00:00 2023-06-22T07:56:46.652 INFO:teuthology.orchestra.run.smithi023.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2023-06-22T07:56:46.694 INFO:teuthology.orchestra.run.smithi023.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-06-22T07:56:46.761 INFO:teuthology.orchestra.run.smithi023.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2023-06-22T07:56:46.780 INFO:teuthology.orchestra.run.smithi067.stdout:(13/131): libcephfs2-18.0.0-4479.g8627af3e.el8. 4.8 MB/s | 771 kB 00:00 2023-06-22T07:56:46.819 INFO:teuthology.orchestra.run.smithi023.stdout:(89/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 12 MB/s | 2.3 MB 00:00 2023-06-22T07:56:46.847 INFO:teuthology.orchestra.run.smithi067.stdout:(14/131): libcephsqlite-18.0.0-4479.g8627af3e.e 2.0 MB/s | 137 kB 00:00 2023-06-22T07:56:46.922 INFO:teuthology.orchestra.run.smithi067.stdout:(15/131): librados-devel-18.0.0-4479.g8627af3e. 1.8 MB/s | 139 kB 00:00 2023-06-22T07:56:47.014 INFO:teuthology.orchestra.run.smithi067.stdout:(16/131): libradosstriper1-18.0.0-4479.g8627af3 5.0 MB/s | 474 kB 00:00 2023-06-22T07:56:47.128 INFO:teuthology.orchestra.run.smithi023.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 163 kB/s | 60 kB 00:00 2023-06-22T07:56:47.154 INFO:teuthology.orchestra.run.smithi023.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 294 kB/s | 98 kB 00:00 2023-06-22T07:56:47.179 INFO:teuthology.orchestra.run.smithi023.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-06-22T07:56:47.204 INFO:teuthology.orchestra.run.smithi023.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 770 kB/s | 39 kB 00:00 2023-06-22T07:56:47.223 INFO:teuthology.orchestra.run.smithi067.stdout:(17/131): ceph-osd-18.0.0-4479.g8627af3e.el8.x8 7.6 MB/s | 18 MB 00:02 2023-06-22T07:56:47.254 INFO:teuthology.orchestra.run.smithi023.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 8.4 MB/s | 644 kB 00:00 2023-06-22T07:56:47.288 INFO:teuthology.orchestra.run.smithi023.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.0 MB/s | 171 kB 00:00 2023-06-22T07:56:47.306 INFO:teuthology.orchestra.run.smithi067.stdout:(18/131): python3-ceph-argparse-18.0.0-4479.g86 539 kB/s | 45 kB 00:00 2023-06-22T07:56:47.322 INFO:teuthology.orchestra.run.smithi023.stdout:(96/131): NetworkManager-initscripts-updown-1.3 2.0 MB/s | 138 kB 00:00 2023-06-22T07:56:47.373 INFO:teuthology.orchestra.run.smithi023.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2023-06-22T07:56:47.390 INFO:teuthology.orchestra.run.smithi067.stdout:(19/131): python3-ceph-common-18.0.0-4479.g8627 1.4 MB/s | 119 kB 00:00 2023-06-22T07:56:47.406 INFO:teuthology.orchestra.run.smithi023.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2023-06-22T07:56:47.456 INFO:teuthology.orchestra.run.smithi023.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-06-22T07:56:47.473 INFO:teuthology.orchestra.run.smithi067.stdout:(20/131): python3-cephfs-18.0.0-4479.g8627af3e. 2.3 MB/s | 197 kB 00:00 2023-06-22T07:56:47.507 INFO:teuthology.orchestra.run.smithi023.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-06-22T07:56:47.549 INFO:teuthology.orchestra.run.smithi023.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 932 kB/s | 39 kB 00:00 2023-06-22T07:56:47.565 INFO:teuthology.orchestra.run.smithi067.stdout:(21/131): python3-rados-18.0.0-4479.g8627af3e.e 4.0 MB/s | 370 kB 00:00 2023-06-22T07:56:47.616 INFO:teuthology.orchestra.run.smithi023.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2023-06-22T07:56:47.674 INFO:teuthology.orchestra.run.smithi067.stdout:(22/131): librgw2-18.0.0-4479.g8627af3e.el8.x86 10 MB/s | 6.7 MB 00:00 2023-06-22T07:56:47.691 INFO:teuthology.orchestra.run.smithi023.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.0 MB/s | 538 kB 00:00 2023-06-22T07:56:47.699 INFO:teuthology.orchestra.run.smithi067.stdout:(23/131): python3-rbd-18.0.0-4479.g8627af3e.el8 2.7 MB/s | 365 kB 00:00 2023-06-22T07:56:47.716 INFO:teuthology.orchestra.run.smithi067.stdout:(24/131): python3-rgw-18.0.0-4479.g8627af3e.el8 2.5 MB/s | 107 kB 00:00 2023-06-22T07:56:47.733 INFO:teuthology.orchestra.run.smithi023.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-06-22T07:56:47.758 INFO:teuthology.orchestra.run.smithi067.stdout:(25/131): rbd-fuse-18.0.0-4479.g8627af3e.el8.x8 1.6 MB/s | 94 kB 00:00 2023-06-22T07:56:47.775 INFO:teuthology.orchestra.run.smithi023.stdout:(105/131): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-06-22T07:56:47.833 INFO:teuthology.orchestra.run.smithi023.stdout:(106/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-06-22T07:56:47.869 INFO:teuthology.orchestra.run.smithi067.stdout:(26/131): rbd-nbd-18.0.0-4479.g8627af3e.el8.x86 1.6 MB/s | 179 kB 00:00 2023-06-22T07:56:47.951 INFO:teuthology.orchestra.run.smithi067.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4479.g 283 kB/s | 23 kB 00:00 2023-06-22T07:56:47.984 INFO:teuthology.orchestra.run.smithi023.stdout:(107/131): python3-scipy-1.0.0-21.module+el8.5. 20 MB/s | 14 MB 00:00 2023-06-22T07:56:48.026 INFO:teuthology.orchestra.run.smithi023.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 640 kB/s | 27 kB 00:00 2023-06-22T07:56:48.076 INFO:teuthology.orchestra.run.smithi023.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2023-06-22T07:56:48.118 INFO:teuthology.orchestra.run.smithi067.stdout:(28/131): rbd-mirror-18.0.0-4479.g8627af3e.el8. 8.6 MB/s | 3.4 MB 00:00 2023-06-22T07:56:48.118 INFO:teuthology.orchestra.run.smithi023.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.0 MB/s | 44 kB 00:00 2023-06-22T07:56:48.143 INFO:teuthology.orchestra.run.smithi067.stdout:(29/131): ceph-mgr-cephadm-18.0.0-4479.g8627af3 667 kB/s | 128 kB 00:00 2023-06-22T07:56:48.202 INFO:teuthology.orchestra.run.smithi023.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2023-06-22T07:56:48.260 INFO:teuthology.orchestra.run.smithi023.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 7.7 MB/s | 457 kB 00:00 2023-06-22T07:56:48.310 INFO:teuthology.orchestra.run.smithi023.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 4.4 MB/s | 225 kB 00:00 2023-06-22T07:56:48.386 INFO:teuthology.orchestra.run.smithi023.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 14 MB/s | 3.7 MB 00:00 2023-06-22T07:56:48.411 INFO:teuthology.orchestra.run.smithi023.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 639 kB/s | 64 kB 00:00 2023-06-22T07:56:48.418 INFO:teuthology.orchestra.run.smithi067.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4479.g8627a 5.7 MB/s | 1.7 MB 00:00 2023-06-22T07:56:48.436 INFO:teuthology.orchestra.run.smithi023.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-06-22T07:56:48.461 INFO:teuthology.orchestra.run.smithi023.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 398 kB/s | 20 kB 00:00 2023-06-22T07:56:48.487 INFO:teuthology.orchestra.run.smithi023.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-06-22T07:56:48.529 INFO:teuthology.orchestra.run.smithi023.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-06-22T07:56:48.579 INFO:teuthology.orchestra.run.smithi023.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 3.9 MB/s | 198 kB 00:00 2023-06-22T07:56:48.713 INFO:teuthology.orchestra.run.smithi023.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 22 MB/s | 4.9 MB 00:00 2023-06-22T07:56:48.755 INFO:teuthology.orchestra.run.smithi023.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-06-22T07:56:48.760 INFO:teuthology.orchestra.run.smithi067.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4479.g86 704 kB/s | 240 kB 00:00 2023-06-22T07:56:48.839 INFO:teuthology.orchestra.run.smithi023.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-06-22T07:56:48.852 INFO:teuthology.orchestra.run.smithi067.stdout:(32/131): ceph-mgr-diskprediction-local-18.0.0- 10 MB/s | 7.4 MB 00:00 2023-06-22T07:56:48.877 INFO:teuthology.orchestra.run.smithi067.stdout:(33/131): ceph-mgr-rook-18.0.0-4479.g8627af3e.e 406 kB/s | 47 kB 00:00 2023-06-22T07:56:48.898 INFO:teuthology.orchestra.run.smithi023.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-06-22T07:56:48.911 INFO:teuthology.orchestra.run.smithi067.stdout:(34/131): ceph-prometheus-alerts-18.0.0-4479.g8 249 kB/s | 15 kB 00:00 2023-06-22T07:56:48.965 INFO:teuthology.orchestra.run.smithi023.stdout:(125/131): postgresql-contrib-10.21-2.module+el 12 MB/s | 808 kB 00:00 2023-06-22T07:56:48.969 INFO:teuthology.orchestra.run.smithi067.stdout:(35/131): cephadm-18.0.0-4479.g8627af3e.el8.noa 2.7 MB/s | 161 kB 00:00 2023-06-22T07:56:48.997 INFO:teuthology.orchestra.run.smithi067.stdout:(36/131): ceph-volume-18.0.0-4479.g8627af3e.el8 2.1 MB/s | 258 kB 00:00 2023-06-22T07:56:49.031 INFO:teuthology.orchestra.run.smithi067.stdout:(37/131): cephfs-top-18.0.0-4479.g8627af3e.el8. 295 kB/s | 18 kB 00:00 2023-06-22T07:56:49.056 INFO:teuthology.orchestra.run.smithi023.stdout:(126/131): postgresql-10.21-2.module+el8.6.0+15 16 MB/s | 1.5 MB 00:00 2023-06-22T07:56:49.132 INFO:teuthology.orchestra.run.smithi023.stdout:(127/131): postgresql-server-10.21-2.module+el8 17 MB/s | 5.1 MB 00:00 2023-06-22T07:56:49.189 INFO:teuthology.orchestra.run.smithi067.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 372 kB/s | 59 kB 00:00 2023-06-22T07:56:49.247 INFO:teuthology.orchestra.run.smithi067.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2023-06-22T07:56:49.281 INFO:teuthology.orchestra.run.smithi067.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 817 kB/s | 75 kB 00:00 2023-06-22T07:56:49.423 INFO:teuthology.orchestra.run.smithi067.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.6 MB/s | 476 kB 00:00 2023-06-22T07:56:49.448 INFO:teuthology.orchestra.run.smithi067.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 261 kB/s | 44 kB 00:00 2023-06-22T07:56:49.515 INFO:teuthology.orchestra.run.smithi023.stdout:(128/131): librados2-18.0.0-4479.g8627af3e.el8. 8.0 MB/s | 3.7 MB 00:00 2023-06-22T07:56:49.532 INFO:teuthology.orchestra.run.smithi067.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.7 MB/s | 185 kB 00:00 2023-06-22T07:56:49.557 INFO:teuthology.orchestra.run.smithi067.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 304 kB/s | 33 kB 00:00 2023-06-22T07:56:49.566 INFO:teuthology.orchestra.run.smithi023.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-06-22T07:56:49.599 INFO:teuthology.orchestra.run.smithi067.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 244 kB/s | 16 kB 00:00 2023-06-22T07:56:49.624 INFO:teuthology.orchestra.run.smithi023.stdout:(130/131): librbd1-18.0.0-4479.g8627af3e.el8.x8 7.2 MB/s | 3.6 MB 00:00 2023-06-22T07:56:49.775 INFO:teuthology.orchestra.run.smithi067.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 795 kB/s | 173 kB 00:00 2023-06-22T07:56:49.808 INFO:teuthology.orchestra.run.smithi067.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.8 MB/s | 384 kB 00:00 2023-06-22T07:56:49.858 INFO:teuthology.orchestra.run.smithi067.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 975 kB/s | 81 kB 00:00 2023-06-22T07:56:49.892 INFO:teuthology.orchestra.run.smithi067.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2023-06-22T07:56:49.967 INFO:teuthology.orchestra.run.smithi067.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 162 kB/s | 18 kB 00:00 2023-06-22T07:56:50.093 INFO:teuthology.orchestra.run.smithi067.stdout:(51/131): python3-kubernetes-11.0.0-6.el8.noarc 5.8 MB/s | 1.2 MB 00:00 2023-06-22T07:56:50.118 INFO:teuthology.orchestra.run.smithi067.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 324 kB/s | 49 kB 00:00 2023-06-22T07:56:50.194 INFO:teuthology.orchestra.run.smithi067.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 586 kB/s | 59 kB 00:00 2023-06-22T07:56:50.227 INFO:teuthology.orchestra.run.smithi067.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 555 kB/s | 60 kB 00:00 2023-06-22T07:56:50.294 INFO:teuthology.orchestra.run.smithi067.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2023-06-22T07:56:50.319 INFO:teuthology.orchestra.run.smithi067.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2023-06-22T07:56:50.361 INFO:teuthology.orchestra.run.smithi067.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 505 kB/s | 34 kB 00:00 2023-06-22T07:56:50.424 INFO:teuthology.orchestra.run.smithi023.stdout:(131/131): ceph-test-18.0.0-4479.g8627af3e.el8. 5.3 MB/s | 48 MB 00:09 2023-06-22T07:56:50.428 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-06-22T07:56:50.428 INFO:teuthology.orchestra.run.smithi023.stdout:Total 18 MB/s | 205 MB 00:11 2023-06-22T07:56:50.445 INFO:teuthology.orchestra.run.smithi067.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.5 MB/s | 196 kB 00:00 2023-06-22T07:56:50.470 INFO:teuthology.orchestra.run.smithi067.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 566 kB/s | 61 kB 00:00 2023-06-22T07:56:50.512 INFO:teuthology.orchestra.run.smithi067.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 280 kB/s | 19 kB 00:00 2023-06-22T07:56:50.546 INFO:teuthology.orchestra.run.smithi067.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 318 kB/s | 24 kB 00:00 2023-06-22T07:56:50.588 INFO:teuthology.orchestra.run.smithi067.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 388 kB/s | 29 kB 00:00 2023-06-22T07:56:50.613 INFO:teuthology.orchestra.run.smithi067.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 411 kB/s | 27 kB 00:00 2023-06-22T07:56:50.713 INFO:teuthology.orchestra.run.smithi067.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 2.0 MB/s | 255 kB 00:00 2023-06-22T07:56:50.764 INFO:teuthology.orchestra.run.smithi067.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.6 MB/s | 251 kB 00:00 2023-06-22T07:56:50.872 INFO:teuthology.orchestra.run.smithi067.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 386 kB/s | 61 kB 00:00 2023-06-22T07:56:50.898 INFO:teuthology.orchestra.run.smithi067.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 639 kB/s | 85 kB 00:00 2023-06-22T07:56:50.964 INFO:teuthology.orchestra.run.smithi067.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 249 kB/s | 23 kB 00:00 2023-06-22T07:56:50.990 INFO:teuthology.orchestra.run.smithi067.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-06-22T07:56:51.049 INFO:teuthology.orchestra.run.smithi067.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 804 kB/s | 47 kB 00:00 2023-06-22T07:56:51.090 INFO:teuthology.orchestra.run.smithi067.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-06-22T07:56:51.132 INFO:teuthology.orchestra.run.smithi067.stdout:(72/131): thrift-0.13.0-2.el8.x86_64.rpm 7.4 MB/s | 1.7 MB 00:00 2023-06-22T07:56:51.158 INFO:teuthology.orchestra.run.smithi067.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 636 kB/s | 43 kB 00:00 2023-06-22T07:56:51.200 INFO:teuthology.orchestra.run.smithi067.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.4 MB/s | 48 kB 00:00 2023-06-22T07:56:51.225 INFO:teuthology.orchestra.run.smithi067.stdout:(75/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.3 MB/s | 220 kB 00:00 2023-06-22T07:56:51.250 INFO:teuthology.orchestra.run.smithi067.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-06-22T07:56:51.275 INFO:teuthology.orchestra.run.smithi067.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 851 kB/s | 43 kB 00:00 2023-06-22T07:56:51.716 INFO:teuthology.orchestra.run.smithi067.stdout:(78/131): ceph-test-18.0.0-4479.g8627af3e.el8.x 7.5 MB/s | 48 MB 00:06 2023-06-22T07:56:51.742 INFO:teuthology.orchestra.run.smithi067.stdout:(79/131): python3-pycparser-2.14-14.el8.noarch. 222 kB/s | 109 kB 00:00 2023-06-22T07:56:51.776 INFO:teuthology.orchestra.run.smithi067.stdout:(80/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 457 kB/s | 229 kB 00:00 2023-06-22T07:56:51.802 INFO:teuthology.orchestra.run.smithi067.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 373 kB/s | 19 kB 00:00 2023-06-22T07:56:51.827 INFO:teuthology.orchestra.run.smithi067.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-06-22T07:56:51.860 INFO:teuthology.orchestra.run.smithi067.stdout:(83/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.8 MB/s | 560 kB 00:00 2023-06-22T07:56:51.886 INFO:teuthology.orchestra.run.smithi067.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 305 kB/s | 18 kB 00:00 2023-06-22T07:56:51.928 INFO:teuthology.orchestra.run.smithi067.stdout:(85/131): python3-cryptography-3.2.1-5.el8.x86_ 4.3 MB/s | 559 kB 00:00 2023-06-22T07:56:51.962 INFO:teuthology.orchestra.run.smithi067.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.1 MB/s | 238 kB 00:00 2023-06-22T07:56:51.995 INFO:teuthology.orchestra.run.smithi067.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.4 MB/s | 605 kB 00:00 2023-06-22T07:56:52.012 INFO:teuthology.orchestra.run.smithi067.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.4 MB/s | 72 kB 00:00 2023-06-22T07:56:52.037 INFO:teuthology.orchestra.run.smithi067.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-06-22T07:56:52.062 INFO:teuthology.orchestra.run.smithi067.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.2 MB/s | 60 kB 00:00 2023-06-22T07:56:52.079 INFO:teuthology.orchestra.run.smithi067.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 2.3 MB/s | 98 kB 00:00 2023-06-22T07:56:52.089 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-06-22T07:56:52.104 INFO:teuthology.orchestra.run.smithi067.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.3 MB/s | 58 kB 00:00 2023-06-22T07:56:52.130 INFO:teuthology.orchestra.run.smithi067.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 770 kB/s | 39 kB 00:00 2023-06-22T07:56:52.171 INFO:teuthology.orchestra.run.smithi067.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 9.4 MB/s | 644 kB 00:00 2023-06-22T07:56:52.206 INFO:teuthology.orchestra.run.smithi067.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.2 MB/s | 171 kB 00:00 2023-06-22T07:56:52.222 INFO:teuthology.orchestra.run.smithi067.stdout:(96/131): NetworkManager-initscripts-updown-1.3 2.7 MB/s | 138 kB 00:00 2023-06-22T07:56:52.235 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-06-22T07:56:52.236 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-06-22T07:56:52.289 INFO:teuthology.orchestra.run.smithi067.stdout:(97/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 6.4 MB/s | 2.3 MB 00:00 2023-06-22T07:56:52.314 INFO:teuthology.orchestra.run.smithi067.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.2 MB/s | 110 kB 00:00 2023-06-22T07:56:52.340 INFO:teuthology.orchestra.run.smithi067.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-06-22T07:56:52.374 INFO:teuthology.orchestra.run.smithi067.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.3 MB/s | 259 kB 00:00 2023-06-22T07:56:52.407 INFO:teuthology.orchestra.run.smithi067.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2023-06-22T07:56:52.424 INFO:teuthology.orchestra.run.smithi067.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 774 kB/s | 39 kB 00:00 2023-06-22T07:56:52.483 INFO:teuthology.orchestra.run.smithi067.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 8.9 MB/s | 688 kB 00:00 2023-06-22T07:56:52.516 INFO:teuthology.orchestra.run.smithi067.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 5.7 MB/s | 538 kB 00:00 2023-06-22T07:56:52.541 INFO:teuthology.orchestra.run.smithi067.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-06-22T07:56:52.567 INFO:teuthology.orchestra.run.smithi067.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-06-22T07:56:52.609 INFO:teuthology.orchestra.run.smithi067.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2023-06-22T07:56:52.642 INFO:teuthology.orchestra.run.smithi067.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 804 kB/s | 27 kB 00:00 2023-06-22T07:56:52.684 INFO:teuthology.orchestra.run.smithi067.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-06-22T07:56:52.718 INFO:teuthology.orchestra.run.smithi067.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-06-22T07:56:52.818 INFO:teuthology.orchestra.run.smithi067.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2023-06-22T07:56:52.868 INFO:teuthology.orchestra.run.smithi067.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-06-22T07:56:52.944 INFO:teuthology.orchestra.run.smithi067.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 17 MB/s | 3.7 MB 00:00 2023-06-22T07:56:52.969 INFO:teuthology.orchestra.run.smithi067.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 2.2 MB/s | 225 kB 00:00 2023-06-22T07:56:52.994 INFO:teuthology.orchestra.run.smithi067.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-06-22T07:56:53.019 INFO:teuthology.orchestra.run.smithi067.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-06-22T07:56:53.045 INFO:teuthology.orchestra.run.smithi067.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 398 kB/s | 20 kB 00:00 2023-06-22T07:56:53.070 INFO:teuthology.orchestra.run.smithi067.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-06-22T07:56:53.112 INFO:teuthology.orchestra.run.smithi067.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-06-22T07:56:53.154 INFO:teuthology.orchestra.run.smithi067.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.7 MB/s | 198 kB 00:00 2023-06-22T07:56:53.337 INFO:teuthology.orchestra.run.smithi067.stdout:(121/131): python3-scipy-1.0.0-21.module+el8.5. 12 MB/s | 14 MB 00:01 2023-06-22T07:56:53.371 INFO:teuthology.orchestra.run.smithi067.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 6.0 MB/s | 202 kB 00:00 2023-06-22T07:56:53.413 INFO:teuthology.orchestra.run.smithi067.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-06-22T07:56:53.489 INFO:teuthology.orchestra.run.smithi067.stdout:(124/131): openblas-threads-0.3.15-3.el8.x86_64 12 MB/s | 4.9 MB 00:00 2023-06-22T07:56:53.631 INFO:teuthology.orchestra.run.smithi067.stdout:(125/131): postgresql-contrib-10.21-2.module+el 5.6 MB/s | 808 kB 00:00 2023-06-22T07:56:53.697 INFO:teuthology.orchestra.run.smithi067.stdout:(126/131): openblas-0.3.15-3.el8.x86_64.rpm 8.7 MB/s | 4.7 MB 00:00 2023-06-22T07:56:53.756 INFO:teuthology.orchestra.run.smithi067.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 12 MB/s | 1.5 MB 00:00 2023-06-22T07:56:53.840 INFO:teuthology.orchestra.run.smithi067.stdout:(128/131): postgresql-server-10.21-2.module+el8 12 MB/s | 5.1 MB 00:00 2023-06-22T07:56:53.898 INFO:teuthology.orchestra.run.smithi067.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 13 MB/s | 798 kB 00:00 2023-06-22T07:56:54.007 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-06-22T07:56:54.012 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-06-22T07:56:55.382 INFO:teuthology.orchestra.run.smithi067.stdout:(130/131): librados2-18.0.0-4479.g8627af3e.el8. 2.2 MB/s | 3.7 MB 00:01 2023-06-22T07:56:55.757 INFO:teuthology.orchestra.run.smithi067.stdout:(131/131): librbd1-18.0.0-4479.g8627af3e.el8.x8 1.8 MB/s | 3.6 MB 00:02 2023-06-22T07:56:55.760 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-06-22T07:56:55.760 INFO:teuthology.orchestra.run.smithi067.stdout:Total 15 MB/s | 205 MB 00:13 2023-06-22T07:56:56.636 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-06-22T07:56:56.768 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-06-22T07:56:56.769 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-06-22T07:56:57.966 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-06-22T07:56:57.970 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-06-22T07:56:59.084 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-06-22T07:56:59.277 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-06-22T07:56:59.472 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-06-22T07:56:59.644 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-06-22T07:56:59.847 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-06-22T07:57:00.061 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-06-22T07:57:00.554 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-06-22T07:57:00.794 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:18.0.0-4479.g8627af3e.el8.x86_64 5/134 2023-06-22T07:57:01.281 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:18.0.0-4479.g8627af3e.el8.x86_64 5/134 2023-06-22T07:57:01.469 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:18.0.0-4479.g8627af3e.el8.x86_64 6/134 2023-06-22T07:57:01.611 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:18.0.0-4479.g8627af3e.el8.x86_64 6/134 2023-06-22T07:57:01.835 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:18.0.0-4479.g8627af3e.el8.x86_64 7/134 2023-06-22T07:57:01.953 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:18.0.0-4479.g8627af3e.el8.x86_64 7/134 2023-06-22T07:57:02.142 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:18.0.0-4479.g8627af3e.el8.x86_64 8/134 2023-06-22T07:57:02.423 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-06-22T07:57:02.691 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-06-22T07:57:02.900 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-06-22T07:57:03.251 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-06-22T07:57:03.359 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-06-22T07:57:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-06-22T07:57:03.607 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-06-22T07:57:03.608 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-06-22T07:57:03.766 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-06-22T07:57:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephsqlite-2:18.0.0-4479.g8627af3e.el8.x86_64 14/134 2023-06-22T07:57:03.898 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-06-22T07:57:03.920 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4479.g8627af3e.el8.x86_64 14/134 2023-06-22T07:57:04.077 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:18.0.0-4479.g8627af3e.el8.x86 15/134 2023-06-22T07:57:04.099 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-06-22T07:57:04.265 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4479.g8627af3e.el8.x86 15/134 2023-06-22T07:57:04.443 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-06-22T07:57:04.525 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-06-22T07:57:04.629 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-06-22T07:57:04.766 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:18.0.0-4479.g8627af3e.el8.x86_64 5/134 2023-06-22T07:57:04.802 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-06-22T07:57:04.935 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-06-22T07:57:05.088 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-06-22T07:57:05.214 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:18.0.0-4479.g8627af3e.el8.x86_64 5/134 2023-06-22T07:57:05.227 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-06-22T07:57:05.399 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:18.0.0-4479.g8627af3e.el8.x86_64 6/134 2023-06-22T07:57:05.528 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:18.0.0-4479.g8627af3e.el8.x86_64 6/134 2023-06-22T07:57:05.766 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:18.0.0-4479.g8627af3e.el8.x86_64 7/134 2023-06-22T07:57:05.864 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:18.0.0-4479.g8627af3e.el8.x86_64 7/134 2023-06-22T07:57:05.973 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-06-22T07:57:06.021 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:18.0.0-4479.g8627af3e.el8.x86_64 8/134 2023-06-22T07:57:06.144 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-06-22T07:57:06.184 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-06-22T07:57:06.431 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-06-22T07:57:06.628 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-06-22T07:57:06.863 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-06-22T07:57:07.051 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-06-22T07:57:07.057 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-06-22T07:57:07.353 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-06-22T07:57:07.532 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:18.0.0-4479.g8627af3e.el8.x86_64 14/134 2023-06-22T07:57:07.639 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4479.g8627af3e.el8.x86_64 14/134 2023-06-22T07:57:07.791 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:18.0.0-4479.g8627af3e.el8.x86 15/134 2023-06-22T07:57:07.945 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4479.g8627af3e.el8.x86 15/134 2023-06-22T07:57:07.948 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-06-22T07:57:08.128 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-06-22T07:57:08.202 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-06-22T07:57:08.298 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-06-22T07:57:08.433 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-06-22T07:57:08.464 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-06-22T07:57:08.582 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-06-22T07:57:08.675 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-06-22T07:57:08.715 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-06-22T07:57:08.826 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:18.0.0-4479.g8627af3e.el8. 26/134 2023-06-22T07:57:08.848 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-06-22T07:57:08.974 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:18.0.0-4479.g8627af3e.el 27/134 2023-06-22T07:57:09.233 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:18.0.0-4479.g8627af3e.el8.x86_6 28/134 2023-06-22T07:57:09.451 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-06-22T07:57:09.499 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-06-22T07:57:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-06-22T07:57:10.298 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-06-22T07:57:10.465 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-06-22T07:57:11.234 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-06-22T07:57:11.481 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-06-22T07:57:11.656 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-06-22T07:57:11.658 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-06-22T07:57:11.903 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-06-22T07:57:12.055 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:18.0.0-4479.g8627af3e.el8. 26/134 2023-06-22T07:57:12.185 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:18.0.0-4479.g8627af3e.el 27/134 2023-06-22T07:57:12.441 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:18.0.0-4479.g8627af3e.el8.x86_6 28/134 2023-06-22T07:57:12.640 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-06-22T07:57:12.872 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-06-22T07:57:13.223 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-06-22T07:57:13.463 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-06-22T07:57:13.685 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-06-22T07:57:13.907 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-06-22T07:57:14.118 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-06-22T07:57:14.324 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:18.0.0-4479.g8627af3e.el8.x86_64 36/134 2023-06-22T07:57:14.529 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-06-22T07:57:14.534 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:18.0.0-4479.g8627af3e.el8.x86_6 37/134 2023-06-22T07:57:14.702 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-06-22T07:57:14.874 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-06-22T07:57:15.067 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-06-22T07:57:15.157 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-06-22T07:57:15.424 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-06-22T07:57:15.614 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-06-22T07:57:15.632 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-06-22T07:57:15.792 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-06-22T07:57:15.877 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-06-22T07:57:15.930 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-06-22T07:57:16.105 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-06-22T07:57:16.188 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-06-22T07:57:16.307 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-06-22T07:57:16.496 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-06-22T07:57:16.687 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-06-22T07:57:16.800 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-06-22T07:57:16.862 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-06-22T07:57:17.007 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-06-22T07:57:17.022 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-06-22T07:57:17.223 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-06-22T07:57:17.228 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-06-22T07:57:17.413 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:18.0.0-4479.g8627af3e.el8.x86_64 36/134 2023-06-22T07:57:17.457 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-06-22T07:57:17.563 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:18.0.0-4479.g8627af3e.el8.x86_6 37/134 2023-06-22T07:57:17.698 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-06-22T07:57:17.842 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-06-22T07:57:18.047 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-06-22T07:57:18.122 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-06-22T07:57:18.175 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-06-22T07:57:18.359 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-06-22T07:57:18.484 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-06-22T07:57:18.577 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-06-22T07:57:18.708 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-06-22T07:57:18.756 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-06-22T07:57:18.841 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-06-22T07:57:18.975 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-06-22T07:57:19.039 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-06-22T07:57:19.193 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-06-22T07:57:19.195 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-06-22T07:57:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-06-22T07:57:19.541 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-06-22T07:57:19.552 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-06-22T07:57:19.708 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-06-22T07:57:19.785 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-06-22T07:57:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-06-22T07:57:19.980 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-06-22T07:57:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-06-22T07:57:20.211 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-06-22T07:57:20.250 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-06-22T07:57:20.410 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-06-22T07:57:20.564 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-06-22T07:57:20.591 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-06-22T07:57:20.859 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-06-22T07:57:21.047 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-06-22T07:57:21.102 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-06-22T07:57:21.345 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-06-22T07:57:21.350 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-06-22T07:57:21.562 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-06-22T07:57:21.602 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-06-22T07:57:21.853 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-06-22T07:57:21.890 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-06-22T07:57:22.018 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-06-22T07:57:22.106 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-06-22T07:57:22.227 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-06-22T07:57:22.305 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-06-22T07:57:22.473 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-06-22T07:57:22.539 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-06-22T07:57:22.716 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-06-22T07:57:22.825 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-06-22T07:57:22.886 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-06-22T07:57:22.972 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-06-22T07:57:23.113 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-06-22T07:57:23.218 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-06-22T07:57:23.316 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-06-22T07:57:23.522 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-06-22T07:57:23.525 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-06-22T07:57:23.732 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-06-22T07:57:23.747 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-06-22T07:57:23.895 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-06-22T07:57:23.957 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-06-22T07:57:24.032 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-06-22T07:57:24.139 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-06-22T07:57:24.276 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-06-22T07:57:24.318 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-06-22T07:57:24.481 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-06-22T07:57:24.586 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-06-22T07:57:24.656 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-06-22T07:57:24.837 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-06-22T07:57:24.945 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-06-22T07:57:25.001 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-06-22T07:57:25.150 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-06-22T07:57:25.158 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-06-22T07:57:25.328 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-06-22T07:57:25.342 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-06-22T07:57:25.510 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-06-22T07:57:25.535 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-06-22T07:57:25.717 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-06-22T07:57:25.850 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-06-22T07:57:25.961 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-06-22T07:57:26.198 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-06-22T07:57:26.419 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-06-22T07:57:26.612 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-06-22T07:57:26.869 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-06-22T07:57:26.935 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-06-22T07:57:27.172 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-06-22T07:57:27.188 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-06-22T07:57:27.348 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-06-22T07:57:27.422 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-06-22T07:57:27.566 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-06-22T07:57:27.740 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-06-22T07:57:28.044 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-06-22T07:57:28.251 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-06-22T07:57:28.707 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-06-22T07:57:28.953 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-06-22T07:57:29.127 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-06-22T07:57:29.190 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-06-22T07:57:29.331 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:18.0.0-4479.g8627af3e.el8.x86_64 92/134 2023-06-22T07:57:29.431 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:18.0.0-4479.g8627af3e.el8.x86_64 92/134 2023-06-22T07:57:29.499 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-06-22T07:57:29.607 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:18.0.0-4479.g8627af3e.el8.x86_64 93/134 2023-06-22T07:57:29.757 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-06-22T07:57:29.958 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-06-22T07:57:30.777 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-06-22T07:57:30.987 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:18.0.0-4479.g8627af3e.el8.x86_64 92/134 2023-06-22T07:57:31.063 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:18.0.0-4479.g8627af3e.el8.x86_64 92/134 2023-06-22T07:57:31.222 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:18.0.0-4479.g8627af3e.el8.x86_64 93/134 2023-06-22T07:57:34.727 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 94/134 2023-06-22T07:57:34.982 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 94/134 2023-06-22T07:57:35.053 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 94/134 2023-06-22T07:57:35.254 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 94/134 2023-06-22T07:57:35.725 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 94/134 2023-06-22T07:57:35.892 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 94/134 2023-06-22T07:57:36.013 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:18.0.0-4479.g8627af3e.el8.x86_64 95/134 2023-06-22T07:57:36.135 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:18.0.0-4479.g8627af3e.el8.x86_64 95/134 2023-06-22T07:57:36.155 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:18.0.0-4479.g8627af3e.el8.x86_64 95/134 2023-06-22T07:57:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:18.0.0-4479.g8627af3e.el8.x86_64 95/134 2023-06-22T07:57:36.256 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:18.0.0-4479.g8627af3e.el8.x86_64 96/134 2023-06-22T07:57:36.404 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:18.0.0-4479.g8627af3e.el8.x86_64 96/134 2023-06-22T07:57:55.841 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4479.g8627af3e.el8.x86_64 96/134 2023-06-22T07:57:55.841 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2023-06-22T07:57:55.842 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2023-06-22T07:57:55.842 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /dev 2023-06-22T07:57:55.842 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /run 2023-06-22T07:57:55.842 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2023-06-22T07:57:55.842 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2023-06-22T07:57:55.842 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2023-06-22T07:57:55.842 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2023-06-22T07:57:55.843 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /dev 2023-06-22T07:57:55.843 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:57:56.160 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:18.0.0-4479.g8627af3e.el8.noarch 97/134 2023-06-22T07:57:57.350 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4479.g8627af3e.el8.x86_64 96/134 2023-06-22T07:57:57.350 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2023-06-22T07:57:57.351 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2023-06-22T07:57:57.351 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2023-06-22T07:57:57.351 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /run 2023-06-22T07:57:57.351 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2023-06-22T07:57:57.351 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2023-06-22T07:57:57.352 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2023-06-22T07:57:57.352 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2023-06-22T07:57:57.352 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2023-06-22T07:57:57.352 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:57:57.728 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-volume-2:18.0.0-4479.g8627af3e.el8.noarch 97/134 2023-06-22T07:57:57.826 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:18.0.0-4479.g8627af3e.el8.noarch 97/134 2023-06-22T07:57:57.999 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:18.0.0-4479.g8627af3e.el8.x86_64 98/134 2023-06-22T07:57:58.231 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:18.0.0-4479.g8627af3e.el8.x86_64 98/134 2023-06-22T07:57:58.368 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:18.0.0-4479.g8627af3e.el8.x86_64 99/134 2023-06-22T07:57:58.859 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:18.0.0-4479.g8627af3e.el8.x86_64 99/134 2023-06-22T07:57:59.026 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:18.0.0-4479.g8627af3e.el8.x86_64 100/134 2023-06-22T07:57:59.044 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:18.0.0-4479.g8627af3e.el8.x86_64 100/134 2023-06-22T07:57:59.563 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-volume-2:18.0.0-4479.g8627af3e.el8.noarch 97/134 2023-06-22T07:57:59.743 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:18.0.0-4479.g8627af3e.el8.x86_64 98/134 2023-06-22T07:58:00.034 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:18.0.0-4479.g8627af3e.el8.x86_64 98/134 2023-06-22T07:58:00.178 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:18.0.0-4479.g8627af3e.el8.x86_64 99/134 2023-06-22T07:58:00.748 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:18.0.0-4479.g8627af3e.el8.x86_64 99/134 2023-06-22T07:58:00.815 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:18.0.0-4479.g8627af3e.el8.noarch 101/134 2023-06-22T07:58:00.969 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:18.0.0-4479.g8627af3e.el8.x86_64 100/134 2023-06-22T07:58:01.001 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:18.0.0-4479.g8627af3e.el8.noarch 101/134 2023-06-22T07:58:01.011 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:18.0.0-4479.g8627af3e.el8.x86_64 100/134 2023-06-22T07:58:01.146 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4479.g8627af3e.e 102/134 2023-06-22T07:58:01.317 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4479.g8627af3e. 103/134 2023-06-22T07:58:01.484 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4479.g8627af3e.el8.noa 104/134 2023-06-22T07:58:01.706 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4479.g8627af3e.el8.noa 104/134 2023-06-22T07:58:02.006 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4479.g8627af3e.el8.n 105/134 2023-06-22T07:58:03.032 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4479.g8627af3e.el8.n 105/134 2023-06-22T07:58:03.184 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4479.g862 106/134 2023-06-22T07:58:03.279 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4479.g862 106/134 2023-06-22T07:58:03.686 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4479.g8627af3e.el 107/134 2023-06-22T07:58:03.769 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:18.0.0-4479.g8627af3e.el8.noarch 101/134 2023-06-22T07:58:03.857 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:18.0.0-4479.g8627af3e.el8.x86_64 108/134 2023-06-22T07:58:03.893 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4479.g8627af3e.el8.x86_64 108/134 2023-06-22T07:58:03.946 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:18.0.0-4479.g8627af3e.el8.noarch 101/134 2023-06-22T07:58:04.050 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:18.0.0-4479.g8627af3e.el8.noarch 109/134 2023-06-22T07:58:04.062 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4479.g8627af3e.el8.noarch 109/134 2023-06-22T07:58:04.093 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4479.g8627af3e.e 102/134 2023-06-22T07:58:04.300 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4479.g8627af3e. 103/134 2023-06-22T07:58:04.451 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4479.g8627af3e.el8.noa 104/134 2023-06-22T07:58:04.724 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4479.g8627af3e.el8.noa 104/134 2023-06-22T07:58:04.967 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4479.g8627af3e.el8.n 105/134 2023-06-22T07:58:05.417 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:18.0.0-4479.g8627af3e.el8.x86_64 110/134 2023-06-22T07:58:05.626 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:18.0.0-4479.g8627af3e.el8.x86_64 111/134 2023-06-22T07:58:05.668 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4479.g8627af3e.el8.x86_64 111/134 2023-06-22T07:58:05.834 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4479.g8627a 112/134 2023-06-22T07:58:05.877 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4479.g8627a 112/134 2023-06-22T07:58:06.033 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 113/134 2023-06-22T07:58:06.156 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4479.g8627af3e.el8.n 105/134 2023-06-22T07:58:06.309 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4479.g862 106/134 2023-06-22T07:58:06.401 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 113/134 2023-06-22T07:58:06.447 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4479.g862 106/134 2023-06-22T07:58:06.584 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 114/134 2023-06-22T07:58:06.852 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4479.g8627af3e.el 107/134 2023-06-22T07:58:07.076 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:18.0.0-4479.g8627af3e.el8.x86_64 108/134 2023-06-22T07:58:07.144 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4479.g8627af3e.el8.x86_64 108/134 2023-06-22T07:58:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:18.0.0-4479.g8627af3e.el8.noarch 109/134 2023-06-22T07:58:07.397 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4479.g8627af3e.el8.noarch 109/134 2023-06-22T07:58:08.906 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:18.0.0-4479.g8627af3e.el8.x86_64 110/134 2023-06-22T07:58:09.144 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:18.0.0-4479.g8627af3e.el8.x86_64 111/134 2023-06-22T07:58:09.209 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4479.g8627af3e.el8.x86_64 111/134 2023-06-22T07:58:09.376 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4479.g8627a 112/134 2023-06-22T07:58:09.448 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4479.g8627a 112/134 2023-06-22T07:58:09.643 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 113/134 2023-06-22T07:58:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 113/134 2023-06-22T07:58:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 114/134 2023-06-22T07:58:11.404 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 114/134 2023-06-22T07:58:12.446 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:18.0.0-4479.g8627af3e.el8.x86_64 115/134 2023-06-22T07:58:12.802 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-06-22T07:58:12.986 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 117/134 2023-06-22T07:58:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-06-22T07:58:13.374 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-06-22T07:58:13.586 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-06-22T07:58:13.804 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-06-22T07:58:13.979 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:18.0.0-4479.g8627af3e.el8.x86_ 121/134 2023-06-22T07:58:14.134 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-06-22T07:58:15.560 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 114/134 2023-06-22T07:58:16.442 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-22T07:58:16.707 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-22T07:58:16.726 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-22T07:58:16.892 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:18.0.0-4479.g8627af3e.el8.x86_64 115/134 2023-06-22T07:58:16.942 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-top-2:18.0.0-4479.g8627af3e.el8.noarch 124/134 2023-06-22T07:58:17.183 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 125/134 2023-06-22T07:58:17.251 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-06-22T07:58:17.464 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 117/134 2023-06-22T07:58:17.492 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:18.0.0-4479.g8627af3e.el8.x86_64 126/134 2023-06-22T07:58:17.737 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-06-22T07:58:17.851 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-06-22T07:58:17.900 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-06-22T07:58:18.057 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-06-22T07:58:18.141 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-06-22T07:58:18.169 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-06-22T07:58:18.315 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-06-22T07:58:18.328 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-06-22T07:58:18.388 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-06-22T07:58:18.490 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-06-22T07:58:18.499 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-06-22T07:58:18.564 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:18.0.0-4479.g8627af3e.el8.x86_ 121/134 2023-06-22T07:58:18.634 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-06-22T07:58:18.635 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-22T07:58:18.717 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-06-22T07:58:18.835 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-22T07:58:18.836 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-06-22T07:58:19.063 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-06-22T07:58:19.223 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-06-22T07:58:19.517 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-06-22T07:58:19.518 INFO:teuthology.orchestra.run.smithi067.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-22T07:58:19.518 INFO:teuthology.orchestra.run.smithi067.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-22T07:58:19.518 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:19.748 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:18.0.0-4479.g8627af3e.el8.x86_64 1/134 2023-06-22T07:58:19.748 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:18.0.0-4479.g8627af3e.el8.x86_64 2/134 2023-06-22T07:58:19.748 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 3/134 2023-06-22T07:58:19.749 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 4/134 2023-06-22T07:58:19.749 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4479.g8627a 5/134 2023-06-22T07:58:19.749 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:18.0.0-4479.g8627af3e.el8.x86_64 6/134 2023-06-22T07:58:19.749 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:18.0.0-4479.g8627af3e.el8.x86_64 7/134 2023-06-22T07:58:19.749 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:18.0.0-4479.g8627af3e.el8.x86_64 8/134 2023-06-22T07:58:19.749 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:18.0.0-4479.g8627af3e.el8.x86_64 9/134 2023-06-22T07:58:19.749 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:18.0.0-4479.g8627af3e.el8.x86_64 10/134 2023-06-22T07:58:19.750 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:18.0.0-4479.g8627af3e.el8.x86_64 11/134 2023-06-22T07:58:19.750 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:18.0.0-4479.g8627af3e.el8.x86_64 12/134 2023-06-22T07:58:19.750 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 13/134 2023-06-22T07:58:19.750 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:18.0.0-4479.g8627af3e.el8.x86_ 14/134 2023-06-22T07:58:19.750 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:18.0.0-4479.g8627af3e.el8.x86_64 15/134 2023-06-22T07:58:19.750 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:18.0.0-4479.g8627af3e.el8.x86_64 16/134 2023-06-22T07:58:19.750 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:18.0.0-4479.g8627af3e.el8.x86_6 17/134 2023-06-22T07:58:19.750 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:18.0.0-4479.g8627af3e.el8.x86 18/134 2023-06-22T07:58:19.751 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:18.0.0-4479.g8627af3e.el8.x86_64 19/134 2023-06-22T07:58:19.751 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4479.g8627af3e.el 20/134 2023-06-22T07:58:19.751 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:18.0.0-4479.g8627af3e.el8. 21/134 2023-06-22T07:58:19.751 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:18.0.0-4479.g8627af3e.el8.x86_6 22/134 2023-06-22T07:58:19.751 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:18.0.0-4479.g8627af3e.el8.x86_64 23/134 2023-06-22T07:58:19.751 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:18.0.0-4479.g8627af3e.el8.x86_64 24/134 2023-06-22T07:58:19.751 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:18.0.0-4479.g8627af3e.el8.x86_64 25/134 2023-06-22T07:58:19.761 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 26/134 2023-06-22T07:58:19.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 27/134 2023-06-22T07:58:19.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:18.0.0-4479.g8627af3e.el8.x86_64 28/134 2023-06-22T07:58:19.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4479.g8627af3e. 29/134 2023-06-22T07:58:19.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4479.g8627af3e.el8.noa 30/134 2023-06-22T07:58:19.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4479.g8627af3e.el8.n 31/134 2023-06-22T07:58:19.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4479.g862 32/134 2023-06-22T07:58:19.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4479.g8627af3e.el 33/134 2023-06-22T07:58:19.763 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4479.g8627af3e.el8.noarch 34/134 2023-06-22T07:58:19.763 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4479.g8627af3e.e 35/134 2023-06-22T07:58:19.763 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:18.0.0-4479.g8627af3e.el8.noarch 36/134 2023-06-22T07:58:19.763 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:18.0.0-4479.g8627af3e.el8.noarch 37/134 2023-06-22T07:58:19.763 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-top-2:18.0.0-4479.g8627af3e.el8.noarch 38/134 2023-06-22T07:58:19.763 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-06-22T07:58:19.763 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-06-22T07:58:19.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-06-22T07:58:19.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-06-22T07:58:19.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-06-22T07:58:19.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-06-22T07:58:19.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-06-22T07:58:19.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-06-22T07:58:19.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-06-22T07:58:19.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-06-22T07:58:19.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-06-22T07:58:19.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-06-22T07:58:19.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-06-22T07:58:19.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-06-22T07:58:19.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-06-22T07:58:19.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-06-22T07:58:19.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-06-22T07:58:19.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-06-22T07:58:19.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-06-22T07:58:19.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-06-22T07:58:19.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-06-22T07:58:19.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-06-22T07:58:19.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-06-22T07:58:19.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-06-22T07:58:19.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-06-22T07:58:19.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-06-22T07:58:19.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-06-22T07:58:19.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-06-22T07:58:19.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-06-22T07:58:19.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-06-22T07:58:19.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-06-22T07:58:19.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-06-22T07:58:19.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-06-22T07:58:19.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-06-22T07:58:19.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-06-22T07:58:19.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-06-22T07:58:19.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-06-22T07:58:19.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-06-22T07:58:19.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-06-22T07:58:19.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-06-22T07:58:19.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-06-22T07:58:19.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-06-22T07:58:19.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-06-22T07:58:19.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-06-22T07:58:19.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-06-22T07:58:19.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-06-22T07:58:19.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-06-22T07:58:19.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-06-22T07:58:19.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-06-22T07:58:19.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-06-22T07:58:19.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-06-22T07:58:19.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-06-22T07:58:19.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-06-22T07:58:19.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-06-22T07:58:19.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-06-22T07:58:19.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-06-22T07:58:19.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-06-22T07:58:19.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-06-22T07:58:19.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-06-22T07:58:19.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-06-22T07:58:19.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-06-22T07:58:19.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-06-22T07:58:19.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-06-22T07:58:19.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-06-22T07:58:19.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-06-22T07:58:19.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-06-22T07:58:19.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-06-22T07:58:19.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-06-22T07:58:19.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-06-22T07:58:19.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-06-22T07:58:19.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-06-22T07:58:19.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-06-22T07:58:19.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-06-22T07:58:19.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-06-22T07:58:19.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-06-22T07:58:19.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-06-22T07:58:19.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-06-22T07:58:19.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-06-22T07:58:19.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-06-22T07:58:19.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-06-22T07:58:19.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-06-22T07:58:19.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-06-22T07:58:19.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-06-22T07:58:19.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-06-22T07:58:19.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-06-22T07:58:19.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-06-22T07:58:19.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-06-22T07:58:19.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-06-22T07:58:19.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-06-22T07:58:19.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-06-22T07:58:19.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:18.0.0-4479.g8627af3e.el8.x86_64 129/134 2023-06-22T07:58:19.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-06-22T07:58:19.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:18.0.0-4479.g8627af3e.el8.x86_64 131/134 2023-06-22T07:58:19.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-22T07:58:19.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-06-22T07:58:22.115 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-22T07:58:22.366 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-22T07:58:22.404 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-06-22T07:58:22.431 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-06-22T07:58:22.431 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:53 ago on Thu 22 Jun 2023 07:56:29 AM UTC. 2023-06-22T07:58:22.651 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-top-2:18.0.0-4479.g8627af3e.el8.noarch 124/134 2023-06-22T07:58:22.851 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 125/134 2023-06-22T07:58:23.189 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:18.0.0-4479.g8627af3e.el8.x86_64 126/134 2023-06-22T07:58:23.526 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-06-22T07:58:23.707 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-06-22T07:58:23.816 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-06-22T07:58:24.016 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-06-22T07:58:24.043 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-06-22T07:58:24.240 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-06-22T07:58:24.263 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-06-22T07:58:24.436 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-06-22T07:58:24.436 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-22T07:58:24.672 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-22T07:58:24.672 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-06-22T07:58:24.965 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-06-22T07:58:25.149 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-06-22T07:58:25.423 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-06-22T07:58:25.424 INFO:teuthology.orchestra.run.smithi023.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-06-22T07:58:25.424 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:25.746 INFO:teuthology.orchestra.run.smithi023.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-22T07:58:25.747 INFO:teuthology.orchestra.run.smithi023.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-22T07:58:25.747 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:25.976 INFO:teuthology.orchestra.run.smithi067.stdout:Installed products updated. 2023-06-22T07:58:26.038 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:18.0.0-4479.g8627af3e.el8.x86_64 1/134 2023-06-22T07:58:26.038 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:18.0.0-4479.g8627af3e.el8.x86_64 2/134 2023-06-22T07:58:26.039 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 3/134 2023-06-22T07:58:26.039 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 4/134 2023-06-22T07:58:26.039 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4479.g8627a 5/134 2023-06-22T07:58:26.039 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:18.0.0-4479.g8627af3e.el8.x86_64 6/134 2023-06-22T07:58:26.039 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:18.0.0-4479.g8627af3e.el8.x86_64 7/134 2023-06-22T07:58:26.039 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:18.0.0-4479.g8627af3e.el8.x86_64 8/134 2023-06-22T07:58:26.039 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:18.0.0-4479.g8627af3e.el8.x86_64 9/134 2023-06-22T07:58:26.039 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:18.0.0-4479.g8627af3e.el8.x86_64 10/134 2023-06-22T07:58:26.040 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:18.0.0-4479.g8627af3e.el8.x86_64 11/134 2023-06-22T07:58:26.040 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:18.0.0-4479.g8627af3e.el8.x86_64 12/134 2023-06-22T07:58:26.040 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 13/134 2023-06-22T07:58:26.040 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:18.0.0-4479.g8627af3e.el8.x86_ 14/134 2023-06-22T07:58:26.040 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:18.0.0-4479.g8627af3e.el8.x86_64 15/134 2023-06-22T07:58:26.040 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephsqlite-2:18.0.0-4479.g8627af3e.el8.x86_64 16/134 2023-06-22T07:58:26.040 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:18.0.0-4479.g8627af3e.el8.x86_6 17/134 2023-06-22T07:58:26.041 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:18.0.0-4479.g8627af3e.el8.x86 18/134 2023-06-22T07:58:26.041 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:18.0.0-4479.g8627af3e.el8.x86_64 19/134 2023-06-22T07:58:26.041 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4479.g8627af3e.el 20/134 2023-06-22T07:58:26.041 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:18.0.0-4479.g8627af3e.el8. 21/134 2023-06-22T07:58:26.041 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:18.0.0-4479.g8627af3e.el8.x86_6 22/134 2023-06-22T07:58:26.041 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:18.0.0-4479.g8627af3e.el8.x86_64 23/134 2023-06-22T07:58:26.041 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:18.0.0-4479.g8627af3e.el8.x86_64 24/134 2023-06-22T07:58:26.042 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:18.0.0-4479.g8627af3e.el8.x86_64 25/134 2023-06-22T07:58:26.049 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 26/134 2023-06-22T07:58:26.050 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 27/134 2023-06-22T07:58:26.050 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:18.0.0-4479.g8627af3e.el8.x86_64 28/134 2023-06-22T07:58:26.050 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4479.g8627af3e. 29/134 2023-06-22T07:58:26.050 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4479.g8627af3e.el8.noa 30/134 2023-06-22T07:58:26.050 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4479.g8627af3e.el8.n 31/134 2023-06-22T07:58:26.050 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4479.g862 32/134 2023-06-22T07:58:26.050 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4479.g8627af3e.el 33/134 2023-06-22T07:58:26.051 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4479.g8627af3e.el8.noarch 34/134 2023-06-22T07:58:26.051 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4479.g8627af3e.e 35/134 2023-06-22T07:58:26.051 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-volume-2:18.0.0-4479.g8627af3e.el8.noarch 36/134 2023-06-22T07:58:26.051 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:18.0.0-4479.g8627af3e.el8.noarch 37/134 2023-06-22T07:58:26.051 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-top-2:18.0.0-4479.g8627af3e.el8.noarch 38/134 2023-06-22T07:58:26.051 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-06-22T07:58:26.051 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-06-22T07:58:26.052 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-06-22T07:58:26.052 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-06-22T07:58:26.052 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-06-22T07:58:26.052 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-06-22T07:58:26.052 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-06-22T07:58:26.052 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-06-22T07:58:26.052 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-06-22T07:58:26.053 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-06-22T07:58:26.053 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-06-22T07:58:26.053 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-06-22T07:58:26.053 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-06-22T07:58:26.053 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-06-22T07:58:26.053 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-06-22T07:58:26.053 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-06-22T07:58:26.054 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-06-22T07:58:26.054 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-06-22T07:58:26.054 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-06-22T07:58:26.054 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-06-22T07:58:26.054 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-06-22T07:58:26.054 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-06-22T07:58:26.054 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-06-22T07:58:26.054 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-06-22T07:58:26.055 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-06-22T07:58:26.055 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-06-22T07:58:26.055 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-06-22T07:58:26.055 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-06-22T07:58:26.055 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-06-22T07:58:26.055 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-06-22T07:58:26.055 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-06-22T07:58:26.055 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-06-22T07:58:26.056 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-06-22T07:58:26.056 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-06-22T07:58:26.056 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-06-22T07:58:26.056 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-06-22T07:58:26.056 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-06-22T07:58:26.056 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-06-22T07:58:26.056 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-06-22T07:58:26.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-06-22T07:58:26.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-06-22T07:58:26.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-06-22T07:58:26.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-06-22T07:58:26.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-06-22T07:58:26.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-06-22T07:58:26.057 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-06-22T07:58:26.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-06-22T07:58:26.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-06-22T07:58:26.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-06-22T07:58:26.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-06-22T07:58:26.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-06-22T07:58:26.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-06-22T07:58:26.058 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-06-22T07:58:26.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-06-22T07:58:26.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-06-22T07:58:26.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-06-22T07:58:26.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-06-22T07:58:26.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-06-22T07:58:26.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-06-22T07:58:26.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-06-22T07:58:26.059 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-06-22T07:58:26.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-06-22T07:58:26.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-06-22T07:58:26.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-06-22T07:58:26.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-06-22T07:58:26.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-06-22T07:58:26.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-06-22T07:58:26.060 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-06-22T07:58:26.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-06-22T07:58:26.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-06-22T07:58:26.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-06-22T07:58:26.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-06-22T07:58:26.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-06-22T07:58:26.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-06-22T07:58:26.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-06-22T07:58:26.061 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-06-22T07:58:26.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-06-22T07:58:26.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-06-22T07:58:26.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-06-22T07:58:26.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-06-22T07:58:26.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-06-22T07:58:26.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-06-22T07:58:26.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-06-22T07:58:26.062 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-06-22T07:58:26.063 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-06-22T07:58:26.063 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-06-22T07:58:26.063 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-06-22T07:58:26.063 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-06-22T07:58:26.063 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-06-22T07:58:26.063 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-06-22T07:58:26.063 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:18.0.0-4479.g8627af3e.el8.x86_64 129/134 2023-06-22T07:58:26.064 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-06-22T07:58:26.064 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:18.0.0-4479.g8627af3e.el8.x86_64 131/134 2023-06-22T07:58:26.064 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-06-22T07:58:26.064 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-06-22T07:58:26.268 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:26.269 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2023-06-22T07:58:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-06-22T07:58:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.269 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-06-22T07:58:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-06-22T07:58:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: automake-1.16.1-7.el8.noarch 2023-06-22T07:58:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: bison-3.0.4-10.el8.x86_64 2023-06-22T07:58:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.270 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:26.271 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.272 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-06-22T07:58:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: flex-2.6.1-9.el8.x86_64 2023-06-22T07:58:26.273 INFO:teuthology.orchestra.run.smithi067.stdout: fuse-2.9.7-15.el8.x86_64 2023-06-22T07:58:26.274 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-22T07:58:26.274 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.6-3.el8.x86_64 2023-06-22T07:58:26.274 INFO:teuthology.orchestra.run.smithi067.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-06-22T07:58:26.274 INFO:teuthology.orchestra.run.smithi067.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-06-22T07:58:26.274 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-22T07:58:26.274 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.274 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.274 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.275 INFO:teuthology.orchestra.run.smithi067.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-06-22T07:58:26.275 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-06-22T07:58:26.275 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-22T07:58:26.275 INFO:teuthology.orchestra.run.smithi067.stdout: libpq-13.5-1.el8.x86_64 2023-06-22T07:58:26.275 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-06-22T07:58:26.275 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-22T07:58:26.275 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.276 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.276 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-06-22T07:58:26.276 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.276 INFO:teuthology.orchestra.run.smithi067.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-22T07:58:26.276 INFO:teuthology.orchestra.run.smithi067.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-06-22T07:58:26.276 INFO:teuthology.orchestra.run.smithi067.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-22T07:58:26.276 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-22T07:58:26.277 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-06-22T07:58:26.277 INFO:teuthology.orchestra.run.smithi067.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-22T07:58:26.277 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-22T07:58:26.277 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-22T07:58:26.277 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.15-3.el8.x86_64 2023-06-22T07:58:26.277 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-06-22T07:58:26.277 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-06-22T07:58:26.277 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-22T07:58:26.278 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-22T07:58:26.278 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-22T07:58:26.278 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-22T07:58:26.278 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-22T07:58:26.278 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-22T07:58:26.278 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-22T07:58:26.278 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.5.1-7.el8.noarch 2023-06-22T07:58:26.278 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-22T07:58:26.279 INFO:teuthology.orchestra.run.smithi067.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-22T07:58:26.279 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-22T07:58:26.279 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.279 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.279 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.279 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-22T07:58:26.279 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-06-22T07:58:26.279 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-22T07:58:26.280 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-22T07:58:26.280 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-06-22T07:58:26.280 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-22T07:58:26.280 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-22T07:58:26.280 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-22T07:58:26.280 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-06-22T07:58:26.280 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-22T07:58:26.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-22T07:58:26.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-22T07:58:26.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-22T07:58:26.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-22T07:58:26.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-22T07:58:26.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-22T07:58:26.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-22T07:58:26.281 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-22T07:58:26.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-06-22T07:58:26.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-06-22T07:58:26.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-22T07:58:26.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-22T07:58:26.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-2.6-1.el8.noarch 2023-06-22T07:58:26.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-22T07:58:26.282 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-06-22T07:58:26.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-22T07:58:26.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-22T07:58:26.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.14-14.el8.noarch 2023-06-22T07:58:26.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-22T07:58:26.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-22T07:58:26.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-22T07:58:26.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-22T07:58:26.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-22T07:58:26.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-06-22T07:58:26.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-22T07:58:26.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-22T07:58:26.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-22T07:58:26.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-22T07:58:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-22T07:58:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-22T07:58:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-22T07:58:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-22T07:58:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-22T07:58:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-22T07:58:26.285 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-1.el8.x86_64 2023-06-22T07:58:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-22T07:58:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: uuid-1.6.2-43.el8.x86_64 2023-06-22T07:58:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-06-22T07:58:26.286 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-22T07:58:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-06-22T07:58:26.287 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:26.287 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-06-22T07:58:26.488 DEBUG:teuthology.parallel:result is None 2023-06-22T07:58:28.726 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-06-22T07:58:28.726 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:02 ago on Thu 22 Jun 2023 07:56:26 AM UTC. 2023-06-22T07:58:32.700 INFO:teuthology.orchestra.run.smithi023.stdout:Installed products updated. 2023-06-22T07:58:33.013 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:33.014 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2023-06-22T07:58:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-06-22T07:58:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.014 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-06-22T07:58:33.014 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-06-22T07:58:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: automake-1.16.1-7.el8.noarch 2023-06-22T07:58:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: bison-3.0.4-10.el8.x86_64 2023-06-22T07:58:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:33.015 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:33.016 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.017 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top-2:18.0.0-4479.g8627af3e.el8.noarch 2023-06-22T07:58:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-06-22T07:58:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: flex-2.6.1-9.el8.x86_64 2023-06-22T07:58:33.018 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-2.9.7-15.el8.x86_64 2023-06-22T07:58:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-22T07:58:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: jq-1.6-3.el8.x86_64 2023-06-22T07:58:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-06-22T07:58:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-06-22T07:58:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-22T07:58:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.019 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-06-22T07:58:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-06-22T07:58:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-22T07:58:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: libpq-13.5-1.el8.x86_64 2023-06-22T07:58:33.020 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-06-22T07:58:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-22T07:58:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-06-22T07:58:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-22T07:58:33.021 INFO:teuthology.orchestra.run.smithi023.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-06-22T07:58:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-22T07:58:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-22T07:58:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-06-22T07:58:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-22T07:58:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-22T07:58:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-22T07:58:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.15-3.el8.x86_64 2023-06-22T07:58:33.022 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-06-22T07:58:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-06-22T07:58:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-22T07:58:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-22T07:58:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-22T07:58:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-22T07:58:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-22T07:58:33.023 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-06-22T07:58:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-22T07:58:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.5.1-7.el8.noarch 2023-06-22T07:58:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-22T07:58:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-22T07:58:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-22T07:58:33.024 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-22T07:58:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-06-22T07:58:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-22T07:58:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-22T07:58:33.025 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-06-22T07:58:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-22T07:58:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-22T07:58:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-22T07:58:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-06-22T07:58:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-22T07:58:33.026 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-22T07:58:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-22T07:58:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-22T07:58:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-22T07:58:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-22T07:58:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-22T07:58:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-22T07:58:33.027 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-22T07:58:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-06-22T07:58:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-06-22T07:58:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-22T07:58:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-22T07:58:33.028 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-2.6-1.el8.noarch 2023-06-22T07:58:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-22T07:58:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-06-22T07:58:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-22T07:58:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-22T07:58:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.14-14.el8.noarch 2023-06-22T07:58:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-22T07:58:33.029 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-22T07:58:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-22T07:58:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-22T07:58:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-22T07:58:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-06-22T07:58:33.030 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-22T07:58:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-22T07:58:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-22T07:58:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-22T07:58:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-22T07:58:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-22T07:58:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-22T07:58:33.031 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-22T07:58:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-22T07:58:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-22T07:58:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:18.0.0-4479.g8627af3e.el8.x86_64 2023-06-22T07:58:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.4.1-1.el8.x86_64 2023-06-22T07:58:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-22T07:58:33.032 INFO:teuthology.orchestra.run.smithi023.stdout: uuid-1.6.2-43.el8.x86_64 2023-06-22T07:58:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-06-22T07:58:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-22T07:58:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-06-22T07:58:33.033 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:33.034 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-06-22T07:58:33.356 DEBUG:teuthology.parallel:result is None 2023-06-22T07:58:33.357 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8627af3e0adcb765a3c249fcc209cba9f4873e1b 2023-06-22T07:58:33.496 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-22T07:58:33.552 INFO:teuthology.orchestra.run.smithi023.stdout:18.0.0-4479.g8627af3e.el8 2023-06-22T07:58:33.553 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4479.g8627af3e.el8 2023-06-22T07:58:33.553 INFO:teuthology.task.install:The correct ceph version 18.0.0-4479.g8627af3e is installed. 2023-06-22T07:58:33.553 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8627af3e0adcb765a3c249fcc209cba9f4873e1b 2023-06-22T07:58:33.692 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-22T07:58:33.716 INFO:teuthology.orchestra.run.smithi067.stdout:18.0.0-4479.g8627af3e.el8 2023-06-22T07:58:33.717 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4479.g8627af3e.el8 2023-06-22T07:58:33.717 INFO:teuthology.task.install:The correct ceph version 18.0.0-4479.g8627af3e is installed. 2023-06-22T07:58:33.718 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-06-22T07:58:33.718 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:33.719 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-22T07:58:33.809 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:58:33.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-22T07:58:33.853 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-06-22T07:58:33.853 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:33.854 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2023-06-22T07:58:33.923 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-22T07:58:34.037 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:58:34.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2023-06-22T07:58:34.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-22T07:58:34.133 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-06-22T07:58:34.134 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:34.134 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-22T07:58:34.203 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-22T07:58:34.312 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:58:34.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-22T07:58:34.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-22T07:58:34.408 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-06-22T07:58:34.409 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:34.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/stdin-killer 2023-06-22T07:58:34.480 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-22T07:58:34.589 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:58:34.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2023-06-22T07:58:34.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-22T07:58:34.686 INFO:teuthology.run_tasks:Running task ceph... 2023-06-22T07:58:34.762 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-06-22T07:58:34.763 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /var/log/ceph 2023-06-22T07:58:34.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2023-06-22T07:58:34.833 INFO:tasks.ceph:Disabling ceph logrotate... 2023-06-22T07:58:34.834 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-22T07:58:34.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-22T07:58:34.902 INFO:tasks.ceph:Creating extra log directories... 2023-06-22T07:58:34.902 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-22T07:58:34.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-22T07:58:35.000 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-06-22T07:58:35.001 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-06-22T07:58:35.001 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-06-22T07:58:35.002 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:35.002 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-22T07:58:35.045 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-22T07:58:35.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-22T07:58:35.330 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:58:35.331 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-22T07:58:35.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-22T07:58:35.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-22T07:58:35.516 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-06-22T07:58:35.517 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-06-22T07:58:35.517 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116', 'branch': 'wip-rishabh-improvements-authmon', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7311116', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1c622d92b6f18ce44a7c33802686ff61c3c273ab'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-rishabh-improvements-authmon'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '8627af3e0adcb765a3c249fcc209cba9f4873e1b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8627af3e0adcb765a3c249fcc209cba9f4873e1b'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rishabh-improvements-authmon', 'sha1': '8627af3e0adcb765a3c249fcc209cba9f4873e1b'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'sha1': '8627af3e0adcb765a3c249fcc209cba9f4873e1b', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-rishabh-improvements-authmon', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '8627af3e0adcb765a3c249fcc209cba9f4873e1b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi023.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=', 'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1c622d92b6f18ce44a7c33802686ff61c3c273ab'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8627af3e0adcb765a3c249fcc209cba9f4873e1b'}}, {'ceph': None}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '076bbebc42a14f7d568aaa78eabb0038327bcb23', 'timestamp': '2023-06-21_23:38:17', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-06-22T07:58:35.518 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-22T07:58:35.522 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-22T07:58:35.567 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m0777 -- /var/run/ceph 2023-06-22T07:58:35.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2023-06-22T07:58:35.670 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:35.670 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2023-06-22T07:58:35.713 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-22T07:58:35.713 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 2023-06-22T07:58:35.799 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-22T07:58:35.799 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T07:58:35.799 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 452 Links: 1 2023-06-22T07:58:35.800 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T07:58:35.800 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T07:58:35.800 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2023-06-22 07:58:25.545820100 +0000 2023-06-22T07:58:35.800 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2023-06-22 07:51:01.398560810 +0000 2023-06-22T07:58:35.800 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2023-06-22 07:51:01.398560810 +0000 2023-06-22T07:58:35.800 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2023-06-22 07:51:01.366561041 +0000 2023-06-22T07:58:35.801 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-22T07:58:35.906 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2023-06-22T07:58:35.906 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2023-06-22T07:58:35.907 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000399537 s, 1.3 MB/s 2023-06-22T07:58:35.909 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-22T07:58:35.955 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 2023-06-22T07:58:36.039 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-22T07:58:36.039 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T07:58:36.039 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 458 Links: 1 2023-06-22T07:58:36.040 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T07:58:36.040 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T07:58:36.040 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2023-06-22 07:58:25.546820092 +0000 2023-06-22T07:58:36.040 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2023-06-22 07:51:02.225554832 +0000 2023-06-22T07:58:36.040 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2023-06-22 07:51:02.225554832 +0000 2023-06-22T07:58:36.040 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2023-06-22 07:51:02.196555042 +0000 2023-06-22T07:58:36.041 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-22T07:58:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2023-06-22T07:58:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2023-06-22T07:58:36.148 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000345672 s, 1.5 MB/s 2023-06-22T07:58:36.150 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-22T07:58:36.199 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 2023-06-22T07:58:36.248 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-22T07:58:36.249 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T07:58:36.249 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 464 Links: 1 2023-06-22T07:58:36.250 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T07:58:36.250 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T07:58:36.250 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2023-06-22 07:58:25.546820092 +0000 2023-06-22T07:58:36.250 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2023-06-22 07:51:03.071548717 +0000 2023-06-22T07:58:36.251 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2023-06-22 07:51:03.071548717 +0000 2023-06-22T07:58:36.251 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2023-06-22 07:51:03.044548912 +0000 2023-06-22T07:58:36.252 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-22T07:58:36.359 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2023-06-22T07:58:36.359 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2023-06-22T07:58:36.359 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000402599 s, 1.3 MB/s 2023-06-22T07:58:36.361 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-22T07:58:36.409 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 2023-06-22T07:58:36.496 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-22T07:58:36.497 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T07:58:36.497 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 470 Links: 1 2023-06-22T07:58:36.497 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T07:58:36.497 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T07:58:36.497 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2023-06-22 07:58:25.546820092 +0000 2023-06-22T07:58:36.497 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2023-06-22 07:51:03.926542537 +0000 2023-06-22T07:58:36.498 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2023-06-22 07:51:03.926542537 +0000 2023-06-22T07:58:36.498 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2023-06-22 07:51:03.897542746 +0000 2023-06-22T07:58:36.498 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-22T07:58:36.607 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2023-06-22T07:58:36.607 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2023-06-22T07:58:36.607 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000372597 s, 1.4 MB/s 2023-06-22T07:58:36.609 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-22T07:58:36.657 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-22T07:58:36.657 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:58:36.657 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2023-06-22T07:58:36.674 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-22T07:58:36.675 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2023-06-22T07:58:36.733 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-22T07:58:36.733 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T07:58:36.733 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 375331 Links: 1 2023-06-22T07:58:36.734 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T07:58:36.734 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T07:58:36.734 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-06-22 07:58:19.430802987 +0000 2023-06-22T07:58:36.734 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-06-22 07:48:17.539816296 +0000 2023-06-22T07:58:36.734 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-06-22 07:48:17.539816296 +0000 2023-06-22T07:58:36.734 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2023-06-22T07:58:36.735 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-22T07:58:36.799 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-06-22T07:58:36.799 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-06-22T07:58:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000252522 s, 2.0 MB/s 2023-06-22T07:58:36.800 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-22T07:58:36.859 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2023-06-22T07:58:36.918 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-22T07:58:36.918 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T07:58:36.919 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 375418 Links: 1 2023-06-22T07:58:36.919 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T07:58:36.919 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T07:58:36.919 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-06-22 07:58:19.430802987 +0000 2023-06-22T07:58:36.920 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-06-22 07:48:18.061804638 +0000 2023-06-22T07:58:36.920 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-06-22 07:48:18.061804638 +0000 2023-06-22T07:58:36.920 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2023-06-22T07:58:36.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-22T07:58:36.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-06-22T07:58:36.986 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-06-22T07:58:36.986 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000201222 s, 2.5 MB/s 2023-06-22T07:58:36.987 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-22T07:58:37.046 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2023-06-22T07:58:37.103 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-22T07:58:37.104 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T07:58:37.104 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 375478 Links: 1 2023-06-22T07:58:37.104 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T07:58:37.104 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T07:58:37.104 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-06-22 07:58:19.430802987 +0000 2023-06-22T07:58:37.105 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-06-22 07:48:18.592792779 +0000 2023-06-22T07:58:37.105 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-06-22 07:48:18.592792779 +0000 2023-06-22T07:58:37.105 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2023-06-22T07:58:37.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-22T07:58:37.168 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-06-22T07:58:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-06-22T07:58:37.169 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000225525 s, 2.3 MB/s 2023-06-22T07:58:37.170 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-22T07:58:37.228 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2023-06-22T07:58:37.286 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-22T07:58:37.286 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T07:58:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 377467 Links: 1 2023-06-22T07:58:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T07:58:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T07:58:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-06-22 07:58:19.430802987 +0000 2023-06-22T07:58:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-06-22 07:48:19.158780138 +0000 2023-06-22T07:58:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-06-22 07:48:19.158780138 +0000 2023-06-22T07:58:37.287 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2023-06-22T07:58:37.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-22T07:58:37.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-06-22T07:58:37.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-06-22T07:58:37.352 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000177165 s, 2.9 MB/s 2023-06-22T07:58:37.353 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-22T07:58:37.411 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-22T07:58:37.412 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi023.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-22T07:58:37.412 INFO:tasks.ceph:Generating config... 2023-06-22T07:58:37.414 INFO:tasks.ceph:[client] client mount timeout = 600 2023-06-22T07:58:37.414 INFO:tasks.ceph:[client] debug client = 20 2023-06-22T07:58:37.415 INFO:tasks.ceph:[client] debug ms = 1 2023-06-22T07:58:37.415 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-06-22T07:58:37.415 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-06-22T07:58:37.415 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-06-22T07:58:37.416 INFO:tasks.ceph:[mds] debug mds = 20 2023-06-22T07:58:37.416 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-06-22T07:58:37.417 INFO:tasks.ceph:[mds] debug ms = 1 2023-06-22T07:58:37.417 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-06-22T07:58:37.418 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-06-22T07:58:37.418 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-06-22T07:58:37.418 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-06-22T07:58:37.419 INFO:tasks.ceph:[mds] mds debug frag = True 2023-06-22T07:58:37.419 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-06-22T07:58:37.420 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2023-06-22T07:58:37.420 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-06-22T07:58:37.420 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-06-22T07:58:37.421 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-06-22T07:58:37.421 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-06-22T07:58:37.422 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-06-22T07:58:37.422 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-06-22T07:58:37.422 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-06-22T07:58:37.423 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-06-22T07:58:37.423 INFO:tasks.ceph:[mgr] debug ms = 1 2023-06-22T07:58:37.423 INFO:tasks.ceph:[mon] debug mon = 20 2023-06-22T07:58:37.424 INFO:tasks.ceph:[mon] debug ms = 1 2023-06-22T07:58:37.424 INFO:tasks.ceph:[mon] debug paxos = 20 2023-06-22T07:58:37.425 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-06-22T07:58:37.425 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-06-22T07:58:37.425 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-06-22T07:58:37.426 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-06-22T07:58:37.426 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-06-22T07:58:37.427 INFO:tasks.ceph:[osd] debug ms = 1 2023-06-22T07:58:37.427 INFO:tasks.ceph:[osd] debug osd = 20 2023-06-22T07:58:37.427 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-06-22T07:58:37.428 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-06-22T07:58:37.428 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-06-22T07:58:37.428 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-06-22T07:58:37.429 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-06-22T07:58:37.429 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-06-22T07:58:37.429 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-06-22T07:58:37.429 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-06-22T07:58:37.429 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-06-22T07:58:37.430 INFO:tasks.ceph:Setting up mon.a... 2023-06-22T07:58:37.430 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-06-22T07:58:37.537 INFO:teuthology.orchestra.run.smithi023.stdout:creating /etc/ceph/ceph.keyring 2023-06-22T07:58:37.543 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-06-22T07:58:37.648 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-22T07:58:37.716 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.23'), ('mon.b', '172.21.15.67'), ('mon.c', '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]')] 2023-06-22T07:58:37.717 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.23,172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-06-22T07:58:37.718 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:37.719 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-06-22T07:58:37.762 DEBUG:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.23 --add b 172.21.15.67 --addv c '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:58:37.889 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:58:37.890 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool: generated fsid 2f053100-95a5-44db-a4cd-e7710a2fac9d 2023-06-22T07:58:37.890 INFO:teuthology.orchestra.run.smithi023.stdout:setting min_mon_release = pacific 2023-06-22T07:58:37.890 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 0 2023-06-22T07:58:37.890 INFO:teuthology.orchestra.run.smithi023.stdout:fsid 2f053100-95a5-44db-a4cd-e7710a2fac9d 2023-06-22T07:58:37.890 INFO:teuthology.orchestra.run.smithi023.stdout:last_changed 2023-06-22T07:58:37.887720+0000 2023-06-22T07:58:37.890 INFO:teuthology.orchestra.run.smithi023.stdout:created 2023-06-22T07:58:37.887720+0000 2023-06-22T07:58:37.890 INFO:teuthology.orchestra.run.smithi023.stdout:min_mon_release 16 (pacific) 2023-06-22T07:58:37.891 INFO:teuthology.orchestra.run.smithi023.stdout:election_strategy: 1 2023-06-22T07:58:37.891 INFO:teuthology.orchestra.run.smithi023.stdout:0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2023-06-22T07:58:37.891 INFO:teuthology.orchestra.run.smithi023.stdout:1: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2023-06-22T07:58:37.891 INFO:teuthology.orchestra.run.smithi023.stdout:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-06-22T07:58:37.891 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-06-22T07:58:37.892 DEBUG:teuthology.orchestra.run.smithi023:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-06-22T07:58:37.935 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2f053100-95a5-44db-a4cd-e7710a2fac9d... 2023-06-22T07:58:37.936 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-22T07:58:37.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-22T07:58:38.032 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2023-06-22T07:58:38.032 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2023-06-22T07:58:38.033 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-22T07:58:38.033 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2023-06-22T07:58:38.033 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.033 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2023-06-22T07:58:38.034 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.034 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2023-06-22T07:58:38.034 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.035 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2023-06-22T07:58:38.035 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2023-06-22T07:58:38.035 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.035 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2023-06-22T07:58:38.036 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2023-06-22T07:58:38.036 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.036 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-22T07:58:38.037 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2023-06-22T07:58:38.037 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.037 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2023-06-22T07:58:38.037 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-22T07:58:38.038 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.038 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2023-06-22T07:58:38.038 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.038 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2023-06-22T07:58:38.039 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2023-06-22T07:58:38.039 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2023-06-22T07:58:38.039 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2023-06-22T07:58:38.039 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2023-06-22T07:58:38.040 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2023-06-22T07:58:38.040 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2023-06-22T07:58:38.040 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-22T07:58:38.041 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-22T07:58:38.041 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2023-06-22T07:58:38.041 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.041 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-22T07:58:38.041 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.042 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2023-06-22T07:58:38.042 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.042 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2023-06-22T07:58:38.042 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.042 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2023-06-22T07:58:38.042 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2023-06-22T07:58:38.042 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2023-06-22T07:58:38.042 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.23,172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790]" 2023-06-22T07:58:38.043 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2023-06-22T07:58:38.043 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 2f053100-95a5-44db-a4cd-e7710a2fac9d 2023-06-22T07:58:38.043 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.043 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2023-06-22T07:58:38.043 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2023-06-22T07:58:38.043 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.043 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2023-06-22T07:58:38.044 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2023-06-22T07:58:38.044 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2023-06-22T07:58:38.044 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.044 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2023-06-22T07:58:38.044 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2023-06-22T07:58:38.044 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.044 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2023-06-22T07:58:38.044 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2023-06-22T07:58:38.045 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2023-06-22T07:58:38.045 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.045 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2023-06-22T07:58:38.045 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2023-06-22T07:58:38.045 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.045 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2023-06-22T07:58:38.045 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.045 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2023-06-22T07:58:38.046 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.046 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2023-06-22T07:58:38.046 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.046 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2023-06-22T07:58:38.046 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2023-06-22T07:58:38.046 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.046 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2023-06-22T07:58:38.047 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2023-06-22T07:58:38.047 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2023-06-22T07:58:38.047 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2023-06-22T07:58:38.047 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 20 2023-06-22T07:58:38.047 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 20 2023-06-22T07:58:38.047 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-06-22T07:58:38.048 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2023-06-22T07:58:38.048 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2023-06-22T07:58:38.048 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2023-06-22T07:58:38.048 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2023-06-22T07:58:38.048 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2023-06-22T07:58:38.049 INFO:teuthology.orchestra.run.smithi067.stdout: osd blocked scrub grace period = 3600 2023-06-22T07:58:38.049 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2023-06-22T07:58:38.049 INFO:teuthology.orchestra.run.smithi067.stdout: osd max markdown count = 1000 2023-06-22T07:58:38.049 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2023-06-22T07:58:38.050 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2023-06-22T07:58:38.050 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.050 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2023-06-22T07:58:38.050 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-06-22T07:58:38.051 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2023-06-22T07:58:38.051 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2023-06-22T07:58:38.051 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2023-06-22T07:58:38.052 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2023-06-22T07:58:38.052 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2023-06-22T07:58:38.052 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2023-06-22T07:58:38.052 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.053 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2023-06-22T07:58:38.053 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-06-22T07:58:38.053 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2023-06-22T07:58:38.053 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2023-06-22T07:58:38.053 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2023-06-22T07:58:38.054 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2023-06-22T07:58:38.054 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2023-06-22T07:58:38.054 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2023-06-22T07:58:38.054 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2023-06-22T07:58:38.055 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2023-06-22T07:58:38.055 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2023-06-22T07:58:38.055 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.055 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-22T07:58:38.056 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2023-06-22T07:58:38.056 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2023-06-22T07:58:38.056 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.056 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2023-06-22T07:58:38.056 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-22T07:58:38.056 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-22T07:58:38.056 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.057 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2023-06-22T07:58:38.057 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2023-06-22T07:58:38.057 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.057 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2023-06-22T07:58:38.058 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2023-06-22T07:58:38.058 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-22T07:58:38.058 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2023-06-22T07:58:38.058 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2023-06-22T07:58:38.058 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2023-06-22T07:58:38.058 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2023-06-22T07:58:38.058 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-22T07:58:38.058 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-22T07:58:38.059 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2023-06-22T07:58:38.059 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2023-06-22T07:58:38.059 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-06-22T07:58:38.059 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2023-06-22T07:58:38.059 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2023-06-22T07:58:38.059 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2023-06-22T07:58:38.059 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2023-06-22T07:58:38.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2023-06-22T07:58:38.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2023-06-22T07:58:38.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2023-06-22T07:58:38.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.e] 2023-06-22T07:58:38.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2023-06-22T07:58:38.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2023-06-22T07:58:38.060 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.f] 2023-06-22T07:58:38.061 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2023-06-22T07:58:38.061 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2023-06-22T07:58:38.061 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2023-06-22T07:58:38.061 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-06-22T07:58:38.061 INFO:teuthology.orchestra.run.smithi067.stdout: mds bal fragment size max = 10000 2023-06-22T07:58:38.061 INFO:teuthology.orchestra.run.smithi067.stdout: mds bal merge size = 5 2023-06-22T07:58:38.061 INFO:teuthology.orchestra.run.smithi067.stdout: mds bal split bits = 3 2023-06-22T07:58:38.062 INFO:teuthology.orchestra.run.smithi067.stdout: mds bal split size = 100 2023-06-22T07:58:38.062 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2023-06-22T07:58:38.062 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = True 2023-06-22T07:58:38.062 INFO:teuthology.orchestra.run.smithi067.stdout: mds oft prefetch dirfrags = True 2023-06-22T07:58:38.062 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2023-06-22T07:58:38.062 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = True 2023-06-22T07:58:38.062 INFO:teuthology.orchestra.run.smithi067.stdout: mds_dir_prefetch = False 2023-06-22T07:58:38.062 INFO:teuthology.orchestra.run.smithi067.stdout: mds_sleep_rank_change = 5000000.0 2023-06-22T07:58:38.063 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2023-06-22T07:58:38.063 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2023-06-22T07:58:38.063 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2023-06-22T07:58:38.099 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2023-06-22T07:58:38.100 INFO:teuthology.orchestra.run.smithi023.stdout: chdir = "" 2023-06-22T07:58:38.100 INFO:teuthology.orchestra.run.smithi023.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-22T07:58:38.101 INFO:teuthology.orchestra.run.smithi023.stdout: auth supported = cephx 2023-06-22T07:58:38.101 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.101 INFO:teuthology.orchestra.run.smithi023.stdout: filestore xattr use omap = true 2023-06-22T07:58:38.101 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.102 INFO:teuthology.orchestra.run.smithi023.stdout: mon clock drift allowed = 1.000 2023-06-22T07:58:38.102 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.102 INFO:teuthology.orchestra.run.smithi023.stdout: osd crush chooseleaf type = 0 2023-06-22T07:58:38.102 INFO:teuthology.orchestra.run.smithi023.stdout: auth debug = true 2023-06-22T07:58:38.103 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.103 INFO:teuthology.orchestra.run.smithi023.stdout: ms die on old message = true 2023-06-22T07:58:38.103 INFO:teuthology.orchestra.run.smithi023.stdout: ms die on bug = true 2023-06-22T07:58:38.103 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.104 INFO:teuthology.orchestra.run.smithi023.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-22T07:58:38.104 INFO:teuthology.orchestra.run.smithi023.stdout: mon pg warn max object skew = 0 2023-06-22T07:58:38.104 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.105 INFO:teuthology.orchestra.run.smithi023.stdout: # disable pg_autoscaler by default for new pools 2023-06-22T07:58:38.105 INFO:teuthology.orchestra.run.smithi023.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-22T07:58:38.105 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.105 INFO:teuthology.orchestra.run.smithi023.stdout: osd pool default size = 2 2023-06-22T07:58:38.106 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.106 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd allow primary affinity = true 2023-06-22T07:58:38.106 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd allow pg remap = true 2023-06-22T07:58:38.106 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on legacy crush tunables = false 2023-06-22T07:58:38.107 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on crush straw calc version zero = false 2023-06-22T07:58:38.107 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on no sortbitwise = false 2023-06-22T07:58:38.107 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on osd down out interval zero = false 2023-06-22T07:58:38.107 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on too few osds = false 2023-06-22T07:58:38.108 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-22T07:58:38.108 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-22T07:58:38.108 INFO:teuthology.orchestra.run.smithi023.stdout: mon_allow_pool_size_one = true 2023-06-22T07:58:38.109 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.109 INFO:teuthology.orchestra.run.smithi023.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-22T07:58:38.109 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.109 INFO:teuthology.orchestra.run.smithi023.stdout: osd default data pool replay window = 5 2023-06-22T07:58:38.110 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.110 INFO:teuthology.orchestra.run.smithi023.stdout: mon allow pool delete = true 2023-06-22T07:58:38.110 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.110 INFO:teuthology.orchestra.run.smithi023.stdout: mon cluster log file level = debug 2023-06-22T07:58:38.111 INFO:teuthology.orchestra.run.smithi023.stdout: debug asserts on shutdown = true 2023-06-22T07:58:38.111 INFO:teuthology.orchestra.run.smithi023.stdout: mon health detail to clog = false 2023-06-22T07:58:38.111 INFO:teuthology.orchestra.run.smithi023.stdout: mon host = "172.21.15.23,172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790]" 2023-06-22T07:58:38.111 INFO:teuthology.orchestra.run.smithi023.stdout: ms die on skipped message = False 2023-06-22T07:58:38.111 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = 2f053100-95a5-44db-a4cd-e7710a2fac9d 2023-06-22T07:58:38.112 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.112 INFO:teuthology.orchestra.run.smithi023.stdout:[osd] 2023-06-22T07:58:38.112 INFO:teuthology.orchestra.run.smithi023.stdout: osd journal size = 100 2023-06-22T07:58:38.112 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.112 INFO:teuthology.orchestra.run.smithi023.stdout: osd scrub load threshold = 5.0 2023-06-22T07:58:38.113 INFO:teuthology.orchestra.run.smithi023.stdout: osd scrub max interval = 600 2023-06-22T07:58:38.113 INFO:teuthology.orchestra.run.smithi023.stdout: osd mclock profile = high_recovery_ops 2023-06-22T07:58:38.113 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.113 INFO:teuthology.orchestra.run.smithi023.stdout: osd recover clone overlap = true 2023-06-22T07:58:38.114 INFO:teuthology.orchestra.run.smithi023.stdout: osd recovery max chunk = 1048576 2023-06-22T07:58:38.114 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.114 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug shutdown = true 2023-06-22T07:58:38.114 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug op order = true 2023-06-22T07:58:38.114 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug verify stray on activate = true 2023-06-22T07:58:38.114 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.115 INFO:teuthology.orchestra.run.smithi023.stdout: osd open classes on start = true 2023-06-22T07:58:38.115 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug pg log writeout = true 2023-06-22T07:58:38.115 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.115 INFO:teuthology.orchestra.run.smithi023.stdout: osd deep scrub update digest min age = 30 2023-06-22T07:58:38.115 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.116 INFO:teuthology.orchestra.run.smithi023.stdout: osd map max advance = 10 2023-06-22T07:58:38.116 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.116 INFO:teuthology.orchestra.run.smithi023.stdout: journal zero on create = true 2023-06-22T07:58:38.116 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.116 INFO:teuthology.orchestra.run.smithi023.stdout: filestore ondisk finisher threads = 3 2023-06-22T07:58:38.117 INFO:teuthology.orchestra.run.smithi023.stdout: filestore apply finisher threads = 3 2023-06-22T07:58:38.117 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.117 INFO:teuthology.orchestra.run.smithi023.stdout: bdev debug aio = true 2023-06-22T07:58:38.117 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug misdirected ops = true 2023-06-22T07:58:38.118 INFO:teuthology.orchestra.run.smithi023.stdout: bluestore block size = 96636764160 2023-06-22T07:58:38.118 INFO:teuthology.orchestra.run.smithi023.stdout: bluestore fsck on mount = True 2023-06-22T07:58:38.118 INFO:teuthology.orchestra.run.smithi023.stdout: debug bluefs = 20 2023-06-22T07:58:38.118 INFO:teuthology.orchestra.run.smithi023.stdout: debug bluestore = 20 2023-06-22T07:58:38.118 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2023-06-22T07:58:38.119 INFO:teuthology.orchestra.run.smithi023.stdout: debug osd = 20 2023-06-22T07:58:38.119 INFO:teuthology.orchestra.run.smithi023.stdout: debug rocksdb = 10 2023-06-22T07:58:38.119 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd backfillfull_ratio = 0.85 2023-06-22T07:58:38.119 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd full ratio = 0.9 2023-06-22T07:58:38.119 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd nearfull ratio = 0.8 2023-06-22T07:58:38.120 INFO:teuthology.orchestra.run.smithi023.stdout: osd blocked scrub grace period = 3600 2023-06-22T07:58:38.120 INFO:teuthology.orchestra.run.smithi023.stdout: osd failsafe full ratio = 0.95 2023-06-22T07:58:38.120 INFO:teuthology.orchestra.run.smithi023.stdout: osd max markdown count = 1000 2023-06-22T07:58:38.120 INFO:teuthology.orchestra.run.smithi023.stdout: osd objectstore = bluestore 2023-06-22T07:58:38.120 INFO:teuthology.orchestra.run.smithi023.stdout: osd op complaint time = 180 2023-06-22T07:58:38.121 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.121 INFO:teuthology.orchestra.run.smithi023.stdout:[mgr] 2023-06-22T07:58:38.121 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2023-06-22T07:58:38.121 INFO:teuthology.orchestra.run.smithi023.stdout: debug mgr = 20 2023-06-22T07:58:38.122 INFO:teuthology.orchestra.run.smithi023.stdout: debug mon = 20 2023-06-22T07:58:38.122 INFO:teuthology.orchestra.run.smithi023.stdout: debug auth = 20 2023-06-22T07:58:38.122 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min pgs per osd = 4 2023-06-22T07:58:38.122 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min bytes per osd = 10 2023-06-22T07:58:38.123 INFO:teuthology.orchestra.run.smithi023.stdout: mgr/telemetry/nag = false 2023-06-22T07:58:38.123 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.123 INFO:teuthology.orchestra.run.smithi023.stdout:[mon] 2023-06-22T07:58:38.123 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2023-06-22T07:58:38.124 INFO:teuthology.orchestra.run.smithi023.stdout: debug mon = 20 2023-06-22T07:58:38.124 INFO:teuthology.orchestra.run.smithi023.stdout: debug paxos = 20 2023-06-22T07:58:38.124 INFO:teuthology.orchestra.run.smithi023.stdout: debug auth = 20 2023-06-22T07:58:38.124 INFO:teuthology.orchestra.run.smithi023.stdout: mon data avail warn = 5 2023-06-22T07:58:38.124 INFO:teuthology.orchestra.run.smithi023.stdout: mon mgr mkfs grace = 240 2023-06-22T07:58:38.125 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min pgs per osd = 4 2023-06-22T07:58:38.125 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd reporter subtree level = osd 2023-06-22T07:58:38.125 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd prime pg temp = true 2023-06-22T07:58:38.125 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min bytes per osd = 10 2023-06-22T07:58:38.125 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.125 INFO:teuthology.orchestra.run.smithi023.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-22T07:58:38.126 INFO:teuthology.orchestra.run.smithi023.stdout: auth mon ticket ttl = 660 # 11m 2023-06-22T07:58:38.126 INFO:teuthology.orchestra.run.smithi023.stdout: auth service ticket ttl = 240 # 4m 2023-06-22T07:58:38.126 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.126 INFO:teuthology.orchestra.run.smithi023.stdout: # don't complain about insecure global_id in the test suite 2023-06-22T07:58:38.127 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-22T07:58:38.127 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-22T07:58:38.127 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.127 INFO:teuthology.orchestra.run.smithi023.stdout: # 1m isn't quite enough 2023-06-22T07:58:38.127 INFO:teuthology.orchestra.run.smithi023.stdout: mon_down_mkfs_grace = 2m 2023-06-22T07:58:38.127 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.127 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_filestore_osds = false 2023-06-22T07:58:38.127 INFO:teuthology.orchestra.run.smithi023.stdout: mon op complaint time = 120 2023-06-22T07:58:38.128 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:58:38.128 INFO:teuthology.orchestra.run.smithi023.stdout:[client] 2023-06-22T07:58:38.128 INFO:teuthology.orchestra.run.smithi023.stdout: rgw cache enabled = true 2023-06-22T07:58:38.128 INFO:teuthology.orchestra.run.smithi023.stdout: rgw enable ops log = true 2023-06-22T07:58:38.128 INFO:teuthology.orchestra.run.smithi023.stdout: rgw enable usage log = true 2023-06-22T07:58:38.129 INFO:teuthology.orchestra.run.smithi023.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-22T07:58:38.129 INFO:teuthology.orchestra.run.smithi023.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-22T07:58:38.129 INFO:teuthology.orchestra.run.smithi023.stdout: client mount timeout = 600 2023-06-22T07:58:38.129 INFO:teuthology.orchestra.run.smithi023.stdout: debug client = 20 2023-06-22T07:58:38.129 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2023-06-22T07:58:38.129 INFO:teuthology.orchestra.run.smithi023.stdout: rados mon op timeout = 900 2023-06-22T07:58:38.130 INFO:teuthology.orchestra.run.smithi023.stdout: rados osd op timeout = 900 2023-06-22T07:58:38.130 INFO:teuthology.orchestra.run.smithi023.stdout:[mon.a] 2023-06-22T07:58:38.130 INFO:teuthology.orchestra.run.smithi023.stdout:[mon.b] 2023-06-22T07:58:38.130 INFO:teuthology.orchestra.run.smithi023.stdout:[mon.c] 2023-06-22T07:58:38.130 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a] 2023-06-22T07:58:38.131 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.c] 2023-06-22T07:58:38.131 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.e] 2023-06-22T07:58:38.131 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.b] 2023-06-22T07:58:38.131 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.d] 2023-06-22T07:58:38.131 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.f] 2023-06-22T07:58:38.132 INFO:teuthology.orchestra.run.smithi023.stdout:[mds] 2023-06-22T07:58:38.132 INFO:teuthology.orchestra.run.smithi023.stdout: debug mds = 20 2023-06-22T07:58:38.132 INFO:teuthology.orchestra.run.smithi023.stdout: debug mds balancer = 20 2023-06-22T07:58:38.132 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2023-06-22T07:58:38.132 INFO:teuthology.orchestra.run.smithi023.stdout: mds bal fragment size max = 10000 2023-06-22T07:58:38.133 INFO:teuthology.orchestra.run.smithi023.stdout: mds bal merge size = 5 2023-06-22T07:58:38.133 INFO:teuthology.orchestra.run.smithi023.stdout: mds bal split bits = 3 2023-06-22T07:58:38.133 INFO:teuthology.orchestra.run.smithi023.stdout: mds bal split size = 100 2023-06-22T07:58:38.133 INFO:teuthology.orchestra.run.smithi023.stdout: mds debug frag = True 2023-06-22T07:58:38.133 INFO:teuthology.orchestra.run.smithi023.stdout: mds debug scatterstat = True 2023-06-22T07:58:38.134 INFO:teuthology.orchestra.run.smithi023.stdout: mds oft prefetch dirfrags = True 2023-06-22T07:58:38.134 INFO:teuthology.orchestra.run.smithi023.stdout: mds op complaint time = 180 2023-06-22T07:58:38.134 INFO:teuthology.orchestra.run.smithi023.stdout: mds verify scatter = True 2023-06-22T07:58:38.134 INFO:teuthology.orchestra.run.smithi023.stdout: mds_dir_prefetch = False 2023-06-22T07:58:38.134 INFO:teuthology.orchestra.run.smithi023.stdout: mds_sleep_rank_change = 5000000.0 2023-06-22T07:58:38.135 INFO:teuthology.orchestra.run.smithi023.stdout: osd op complaint time = 180 2023-06-22T07:58:38.135 INFO:teuthology.orchestra.run.smithi023.stdout: rados mon op timeout = 900 2023-06-22T07:58:38.135 INFO:teuthology.orchestra.run.smithi023.stdout: rados osd op timeout = 900 2023-06-22T07:58:38.136 INFO:tasks.ceph:Creating admin key on mon.a... 2023-06-22T07:58:38.137 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-06-22T07:58:38.253 INFO:tasks.ceph:Copying monmap to all nodes... 2023-06-22T07:58:38.254 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:38.254 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-06-22T07:58:38.297 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:38.298 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-06-22T07:58:38.381 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T07:58:38.382 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:38.382 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-22T07:58:38.382 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-22T07:58:38.517 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:58:38.518 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:58:38.600 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T07:58:38.601 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:58:38.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-22T07:58:38.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-22T07:58:38.641 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:58:38.641 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:58:38.699 INFO:tasks.ceph:Setting up mon nodes... 2023-06-22T07:58:38.699 INFO:tasks.ceph:Setting up mgr nodes... 2023-06-22T07:58:38.700 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-06-22T07:58:38.827 INFO:teuthology.orchestra.run.smithi023.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-06-22T07:58:38.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-06-22T07:58:38.970 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-06-22T07:58:38.972 INFO:tasks.ceph:Setting up mds nodes... 2023-06-22T07:58:38.973 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-06-22T07:58:39.103 INFO:teuthology.orchestra.run.smithi023.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-06-22T07:58:39.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-06-22T07:58:39.179 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-06-22T07:58:39.307 INFO:teuthology.orchestra.run.smithi023.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-06-22T07:58:39.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-06-22T07:58:39.384 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-06-22T07:58:39.511 INFO:teuthology.orchestra.run.smithi023.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-06-22T07:58:39.516 DEBUG:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-06-22T07:58:39.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-06-22T07:58:39.643 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-06-22T07:58:39.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-06-22T07:58:39.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-06-22T07:58:39.771 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-06-22T07:58:39.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-06-22T07:58:39.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-06-22T07:58:39.898 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-06-22T07:58:39.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-06-22T07:58:39.928 INFO:tasks.ceph_client:Setting up client nodes... 2023-06-22T07:58:39.929 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-06-22T07:58:40.028 INFO:teuthology.orchestra.run.smithi023.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-06-22T07:58:40.061 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-06-22T07:58:40.061 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi023.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-22T07:58:40.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-06-22T07:58:40.131 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-22T07:58:40.131 INFO:tasks.ceph:role: osd.0 2023-06-22T07:58:40.131 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T07:58:40.132 DEBUG:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-22T07:58:40.230 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T07:58:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T07:58:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T07:58:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-22T07:58:40.231 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T07:58:40.231 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2023-06-22T07:58:40.231 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T07:58:40.232 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T07:58:40.232 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T07:58:40.232 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T07:58:41.517 INFO:teuthology.orchestra.run.smithi023.stdout:Discarding blocks...Done. 2023-06-22T07:58:41.519 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2023-06-22T07:58:41.520 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-06-22T07:58:41.635 DEBUG:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-06-22T07:58:41.714 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-06-22T07:58:41.784 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-22T07:58:41.785 INFO:tasks.ceph:role: osd.1 2023-06-22T07:58:41.785 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T07:58:41.785 DEBUG:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-22T07:58:41.884 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T07:58:41.884 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T07:58:41.884 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T07:58:41.884 INFO:teuthology.orchestra.run.smithi023.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-22T07:58:41.885 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T07:58:41.885 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2023-06-22T07:58:41.885 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T07:58:41.885 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T07:58:41.885 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T07:58:41.885 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T07:58:43.173 INFO:teuthology.orchestra.run.smithi023.stdout:Discarding blocks...Done. 2023-06-22T07:58:43.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2023-06-22T07:58:43.177 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-06-22T07:58:43.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-06-22T07:58:43.367 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-06-22T07:58:43.438 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-22T07:58:43.438 INFO:tasks.ceph:role: osd.2 2023-06-22T07:58:43.438 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T07:58:43.439 DEBUG:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-22T07:58:43.538 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T07:58:43.539 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T07:58:43.539 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T07:58:43.539 INFO:teuthology.orchestra.run.smithi023.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-22T07:58:43.540 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T07:58:43.540 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2023-06-22T07:58:43.540 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T07:58:43.540 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T07:58:43.541 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T07:58:43.541 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T07:58:44.818 INFO:teuthology.orchestra.run.smithi023.stdout:Discarding blocks...Done. 2023-06-22T07:58:44.822 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2023-06-22T07:58:44.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-06-22T07:58:44.931 DEBUG:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-06-22T07:58:45.010 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-06-22T07:58:45.082 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-22T07:58:45.083 INFO:tasks.ceph:role: osd.3 2023-06-22T07:58:45.083 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T07:58:45.083 DEBUG:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-22T07:58:45.182 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T07:58:45.183 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T07:58:45.183 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T07:58:45.183 INFO:teuthology.orchestra.run.smithi023.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-22T07:58:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T07:58:45.183 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2023-06-22T07:58:45.183 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T07:58:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T07:58:45.184 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T07:58:45.184 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T07:58:46.467 INFO:teuthology.orchestra.run.smithi023.stdout:Discarding blocks...Done. 2023-06-22T07:58:46.472 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2023-06-22T07:58:46.472 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-06-22T07:58:46.582 DEBUG:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-06-22T07:58:46.658 DEBUG:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:58:46.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:46.779+0000 7facea676640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-06-22T07:58:46.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:46.779+0000 7facea676640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-06-22T07:58:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:46.779+0000 7facea676640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-22T07:58:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:46.779+0000 7facea676640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-22T07:58:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:46.779+0000 7facea676640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-22T07:58:46.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:46.781+0000 7facea676640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-06-22T07:58:49.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-06-22T07:58:49.832 DEBUG:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:58:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:49.936+0000 7f291756e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-06-22T07:58:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:49.937+0000 7f291756e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-06-22T07:58:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:49.937+0000 7f291756e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-22T07:58:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:49.937+0000 7f291756e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-22T07:58:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:49.937+0000 7f291756e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-22T07:58:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:49.939+0000 7f291756e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-06-22T07:58:52.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-06-22T07:58:52.990 DEBUG:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:58:53.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:53.093+0000 7fc228927640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-06-22T07:58:53.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:53.093+0000 7fc228927640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-06-22T07:58:53.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:53.093+0000 7fc228927640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-22T07:58:53.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:53.093+0000 7fc228927640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-22T07:58:53.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:53.093+0000 7fc228927640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-22T07:58:53.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:53.095+0000 7fc228927640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-06-22T07:58:56.052 DEBUG:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-06-22T07:58:56.122 DEBUG:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:58:56.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:56.226+0000 7fbaa19a1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-06-22T07:58:56.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:56.227+0000 7fbaa19a1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-06-22T07:58:56.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:56.227+0000 7fbaa19a1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-22T07:58:56.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:56.227+0000 7fbaa19a1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-22T07:58:56.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:56.227+0000 7fbaa19a1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-22T07:58:56.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:58:56.229+0000 7fbaa19a1640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-06-22T07:58:59.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-06-22T07:58:59.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-06-22T07:58:59.291 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-22T07:58:59.291 INFO:tasks.ceph:role: osd.4 2023-06-22T07:58:59.291 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T07:58:59.292 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-22T07:58:59.370 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T07:58:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T07:58:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T07:58:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-22T07:58:59.370 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T07:58:59.370 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-06-22T07:58:59.371 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T07:58:59.371 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T07:58:59.371 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T07:58:59.371 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T07:58:59.800 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-06-22T07:58:59.803 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-06-22T07:58:59.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-06-22T07:58:59.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-06-22T07:59:00.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-06-22T07:59:00.074 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-22T07:59:00.075 INFO:tasks.ceph:role: osd.5 2023-06-22T07:59:00.075 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T07:59:00.075 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-22T07:59:00.152 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T07:59:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T07:59:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T07:59:00.152 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-22T07:59:00.152 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T07:59:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-06-22T07:59:00.153 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T07:59:00.153 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T07:59:00.153 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T07:59:00.153 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T07:59:00.575 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-06-22T07:59:00.577 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-06-22T07:59:00.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-06-22T07:59:00.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-06-22T07:59:00.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-06-22T07:59:00.794 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-22T07:59:00.794 INFO:tasks.ceph:role: osd.6 2023-06-22T07:59:00.794 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T07:59:00.795 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-22T07:59:00.870 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T07:59:00.870 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T07:59:00.871 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T07:59:00.871 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-22T07:59:00.871 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T07:59:00.871 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-06-22T07:59:00.871 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T07:59:00.871 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T07:59:00.871 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T07:59:00.871 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T07:59:01.316 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-06-22T07:59:01.318 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-06-22T07:59:01.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-06-22T07:59:01.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-06-22T07:59:01.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-06-22T07:59:01.609 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-22T07:59:01.610 INFO:tasks.ceph:role: osd.7 2023-06-22T07:59:01.610 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T07:59:01.611 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-22T07:59:01.686 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T07:59:01.686 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T07:59:01.686 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T07:59:01.686 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-22T07:59:01.686 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T07:59:01.687 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-06-22T07:59:01.687 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T07:59:01.687 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T07:59:01.687 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T07:59:01.687 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T07:59:02.117 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-06-22T07:59:02.119 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-06-22T07:59:02.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-06-22T07:59:02.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-06-22T07:59:02.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:59:02.435 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:02.431+0000 7f114337b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-06-22T07:59:02.436 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:02.432+0000 7f114337b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-06-22T07:59:02.436 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:02.432+0000 7f114337b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-22T07:59:02.436 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:02.432+0000 7f114337b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-22T07:59:02.436 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:02.432+0000 7f114337b640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-22T07:59:02.437 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:02.432+0000 7f114337b640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-06-22T07:59:05.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-06-22T07:59:05.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:59:05.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:05.428+0000 7fa9760ab640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-06-22T07:59:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:05.428+0000 7fa9760ab640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-06-22T07:59:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:05.428+0000 7fa9760ab640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-22T07:59:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:05.428+0000 7fa9760ab640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-22T07:59:05.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:05.428+0000 7fa9760ab640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-22T07:59:05.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:05.429+0000 7fa9760ab640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-06-22T07:59:05.520 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T07:59:05.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T07:59:08.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-06-22T07:59:08.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:59:08.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:08.418+0000 7fe405643640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-06-22T07:59:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:08.418+0000 7fe405643640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-06-22T07:59:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:08.418+0000 7fe405643640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-22T07:59:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:08.418+0000 7fe405643640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-22T07:59:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:08.418+0000 7fe405643640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-22T07:59:08.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:08.419+0000 7fe405643640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-06-22T07:59:11.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-06-22T07:59:11.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:59:11.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:11.414+0000 7f8c91748640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-06-22T07:59:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:11.415+0000 7f8c91748640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-06-22T07:59:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:11.415+0000 7f8c91748640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-22T07:59:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:11.415+0000 7f8c91748640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-22T07:59:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:11.415+0000 7f8c91748640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-22T07:59:11.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:11.415+0000 7f8c91748640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-06-22T07:59:14.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-06-22T07:59:14.316 INFO:tasks.ceph:Reading keys from all nodes... 2023-06-22T07:59:14.316 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:14.317 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-06-22T07:59:14.385 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:14.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-06-22T07:59:14.495 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:14.496 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-06-22T07:59:14.604 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:14.604 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-06-22T07:59:14.714 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:14.714 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-06-22T07:59:14.823 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:14.824 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-06-22T07:59:14.932 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:14.933 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-06-22T07:59:15.042 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:15.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-06-22T07:59:15.154 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:15.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-06-22T07:59:15.187 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:15.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-06-22T07:59:15.258 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:15.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-06-22T07:59:15.327 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:15.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-06-22T07:59:15.396 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:15.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-06-22T07:59:15.464 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:15.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-06-22T07:59:15.533 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:15.533 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-06-22T07:59:15.601 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:15.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-06-22T07:59:15.669 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:15.670 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-06-22T07:59:15.712 INFO:tasks.ceph:Adding keys to all mons... 2023-06-22T07:59:15.712 DEBUG:teuthology.orchestra.run.smithi023:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-22T07:59:15.756 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-22T07:59:15.783 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2023-06-22T07:59:15.784 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAu/5NkMnE5MRAAbYz+R5SxvtNxDxvvVOeh6Q== 2023-06-22T07:59:15.784 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2023-06-22T07:59:15.784 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAv/5NkLWkKBhAA2tiTS0MtMscQh2VPRLeDxQ== 2023-06-22T07:59:15.784 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2023-06-22T07:59:15.785 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAv/5Nkr9gzEhAAG4dxeeu85zT+nSsod4BEUg== 2023-06-22T07:59:15.785 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.e] 2023-06-22T07:59:15.785 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAv/5NkUkJcHhAAa6TxyBlxumAlEqJkxlODEg== 2023-06-22T07:59:15.785 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2023-06-22T07:59:15.786 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA2/5NkXQ2ALhAA9HngU9DH+N8cBhRGpWXb0A== 2023-06-22T07:59:15.786 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2023-06-22T07:59:15.786 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA5/5NkzgjmNxAARikoYV4tZhF4D67bH2ASdA== 2023-06-22T07:59:15.786 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2023-06-22T07:59:15.786 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA9/5NkNtCaBRAAak6MeE8rjp4Ii+07oDV/MQ== 2023-06-22T07:59:15.786 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2023-06-22T07:59:15.786 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBA/5Nkzf6UDRAAzSGLdSobJeDTqwsWGubfGQ== 2023-06-22T07:59:15.786 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2023-06-22T07:59:15.787 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAu/5NksYS2ORAAlr6izfBeqHRw1+GFdqWy7g== 2023-06-22T07:59:15.787 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2023-06-22T07:59:15.787 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAv/5Nkqu86JhAAU7EcRhtZwAo7M86lXCxJbQ== 2023-06-22T07:59:15.787 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2023-06-22T07:59:15.787 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAv/5NkocXYLRAAsiv2J2v+cBs6U8lCHGTlUw== 2023-06-22T07:59:15.787 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.f] 2023-06-22T07:59:15.787 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAv/5NkP+dnNRAAAtdx++5PCEj2ALVnU7fsmg== 2023-06-22T07:59:15.788 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2023-06-22T07:59:15.788 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBG/5NkiqfPGRAAl1p5nSx1z7wjUMKpnI7CKA== 2023-06-22T07:59:15.788 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2023-06-22T07:59:15.788 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBJ/5NkY3ehGRAAFoddfGX0xgTsWRhEkotU+Q== 2023-06-22T07:59:15.788 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2023-06-22T07:59:15.788 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBM/5NkpoIDGRAADdsC66z3y6OszNZI55uWaw== 2023-06-22T07:59:15.788 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2023-06-22T07:59:15.789 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBP/5NkyzbSGBAAdSSxndDmn7g09YSGIa3fBQ== 2023-06-22T07:59:15.789 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2023-06-22T07:59:15.789 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAw/5Nk1TCTARAAC8c4jxtOiO6MAA2J7DuIHA== 2023-06-22T07:59:15.819 INFO:teuthology.orchestra.run.smithi023.stdout:[mgr.y] 2023-06-22T07:59:15.820 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAu/5NkMnE5MRAAbYz+R5SxvtNxDxvvVOeh6Q== 2023-06-22T07:59:15.820 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a] 2023-06-22T07:59:15.820 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAv/5NkLWkKBhAA2tiTS0MtMscQh2VPRLeDxQ== 2023-06-22T07:59:15.820 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.c] 2023-06-22T07:59:15.820 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAv/5Nkr9gzEhAAG4dxeeu85zT+nSsod4BEUg== 2023-06-22T07:59:15.820 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.e] 2023-06-22T07:59:15.821 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAv/5NkUkJcHhAAa6TxyBlxumAlEqJkxlODEg== 2023-06-22T07:59:15.821 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.0] 2023-06-22T07:59:15.821 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQA2/5NkXQ2ALhAA9HngU9DH+N8cBhRGpWXb0A== 2023-06-22T07:59:15.821 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.1] 2023-06-22T07:59:15.821 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQA5/5NkzgjmNxAARikoYV4tZhF4D67bH2ASdA== 2023-06-22T07:59:15.821 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.2] 2023-06-22T07:59:15.821 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQA9/5NkNtCaBRAAak6MeE8rjp4Ii+07oDV/MQ== 2023-06-22T07:59:15.821 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.3] 2023-06-22T07:59:15.822 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQBA/5Nkzf6UDRAAzSGLdSobJeDTqwsWGubfGQ== 2023-06-22T07:59:15.822 INFO:teuthology.orchestra.run.smithi023.stdout:[mgr.x] 2023-06-22T07:59:15.822 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAu/5NksYS2ORAAlr6izfBeqHRw1+GFdqWy7g== 2023-06-22T07:59:15.822 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.b] 2023-06-22T07:59:15.822 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAv/5Nkqu86JhAAU7EcRhtZwAo7M86lXCxJbQ== 2023-06-22T07:59:15.822 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.d] 2023-06-22T07:59:15.822 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAv/5NkocXYLRAAsiv2J2v+cBs6U8lCHGTlUw== 2023-06-22T07:59:15.822 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.f] 2023-06-22T07:59:15.823 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAv/5NkP+dnNRAAAtdx++5PCEj2ALVnU7fsmg== 2023-06-22T07:59:15.823 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.4] 2023-06-22T07:59:15.823 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQBG/5NkiqfPGRAAl1p5nSx1z7wjUMKpnI7CKA== 2023-06-22T07:59:15.823 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.5] 2023-06-22T07:59:15.823 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQBJ/5NkY3ehGRAAFoddfGX0xgTsWRhEkotU+Q== 2023-06-22T07:59:15.823 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.6] 2023-06-22T07:59:15.823 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQBM/5NkpoIDGRAADdsC66z3y6OszNZI55uWaw== 2023-06-22T07:59:15.824 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.7] 2023-06-22T07:59:15.824 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQBP/5NkyzbSGBAAdSSxndDmn7g09YSGIa3fBQ== 2023-06-22T07:59:15.824 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2023-06-22T07:59:15.824 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAw/5Nk1TCTARAAC8c4jxtOiO6MAA2J7DuIHA== 2023-06-22T07:59:15.825 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-22T07:59:15.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-22T07:59:15.976 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:15.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:16.090 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:16.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:16.204 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:16.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:16.319 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:16.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:16.432 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:16.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:16.549 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:16.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:16.658 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:16.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:16.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-22T07:59:16.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-22T07:59:16.874 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:16.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:16.986 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:16.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:17.098 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:17.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T07:59:17.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:17.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:17.318 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:17.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:17.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:17.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:17.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:17.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T07:59:17.645 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-22T07:59:17.650 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-22T07:59:17.757 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-06-22T07:59:17.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-06-22T07:59:17.827 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-22T07:59:17.954 DEBUG:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-06-22T07:59:18.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-06-22T07:59:18.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-22T07:59:18.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-06-22T07:59:18.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-06-22T07:59:18.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-22T07:59:18.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-06-22T07:59:18.371 DEBUG:teuthology.orchestra.run.smithi023:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:59:18.376 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-22T07:59:18.428 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-06-22T07:59:18.429 INFO:tasks.ceph.mon.a:Restarting daemon 2023-06-22T07:59:18.430 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-06-22T07:59:18.459 INFO:tasks.ceph.mon.a:Started 2023-06-22T07:59:18.460 INFO:tasks.ceph.mon.b:Restarting daemon 2023-06-22T07:59:18.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-06-22T07:59:18.471 INFO:tasks.ceph.mon.b:Started 2023-06-22T07:59:18.471 INFO:tasks.ceph.mon.c:Restarting daemon 2023-06-22T07:59:18.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-06-22T07:59:18.473 INFO:tasks.ceph.mon.c:Started 2023-06-22T07:59:18.474 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-06-22T07:59:18.474 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-06-22T07:59:18.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-06-22T07:59:18.479 INFO:tasks.ceph.mgr.y:Started 2023-06-22T07:59:18.479 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-06-22T07:59:18.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-06-22T07:59:18.481 INFO:tasks.ceph.mgr.x:Started 2023-06-22T07:59:18.481 INFO:tasks.ceph:Setting crush tunables to default 2023-06-22T07:59:18.482 DEBUG:teuthology.orchestra.run.smithi023:> sudo ceph --cluster ceph osd crush tunables default 2023-06-22T07:59:18.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.677+0000 7f94c09ff700 1 Processor -- start 2023-06-22T07:59:18.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.677+0000 7f94c09ff700 1 -- start start 2023-06-22T07:59:18.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.678+0000 7f94c09ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:18.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.678+0000 7f94c09ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:18.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.678+0000 7f94c09ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94bc0b1e50 con 0x7f94bc0b28b0 2023-06-22T07:59:18.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.678+0000 7f94c09ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc0b48d0 con 0x7f94bc0b19d0 2023-06-22T07:59:18.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.678+0000 7f94c09ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc0c92f0 con 0x7f94bc0b3d70 2023-06-22T07:59:18.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94b9d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:18.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94b9d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:45020/0 (socket says 172.21.15.23:45020) 2023-06-22T07:59:18.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 learned_addr learned my addr 172.21.15.23:0/2538868767 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:18.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:18.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f94bc159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:18.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f94bc159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:18.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:18.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T07:59:18.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94bad9d700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 msgr2=0x7f94bc0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-06-22T07:59:18.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94bad9d700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 msgr2=0x7f94bc0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:18.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:18.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.679+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T07:59:18.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.880+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:18.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.880+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:18.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.880+0000 7f94bad9d700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 msgr2=0x7f94bc0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:18.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.880+0000 7f94bad9d700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 msgr2=0x7f94bc0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:18.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.880+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:18.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.880+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T07:59:18.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.880+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f94bc159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-06-22T07:59:18.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.880+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f94bc159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:18.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.880+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:18.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:18.880+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T07:59:19.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:19.281+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:19.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:19.281+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:19.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:19.281+0000 7f94bad9d700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 msgr2=0x7f94bc0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:19.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:19.281+0000 7f94bad9d700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 msgr2=0x7f94bc0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:19.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:19.281+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:19.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:19.281+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T07:59:19.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:19.281+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f94bc159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-06-22T07:59:19.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:19.281+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f94bc159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:19.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:19.281+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:19.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:19.281+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T07:59:20.086 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:20.083+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:20.086 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:20.083+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:20.086 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:20.083+0000 7f94bad9d700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 msgr2=0x7f94bc0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:20.086 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:20.083+0000 7f94bad9d700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 msgr2=0x7f94bc0b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:20.083+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:20.083+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T07:59:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:20.083+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f94bc159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-06-22T07:59:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:20.083+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f94bc159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:20.083+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:20.083+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T07:59:21.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.679+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f94bc159530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T07:59:21.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.679+0000 7f94abfff700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f94bc159530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:21.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.679+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 msgr2=0x7f94bc0b4390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T07:59:21.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.679+0000 7f94abfff700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f94bc0b19d0 0x7f94bc0b4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:21.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.679+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 >> v1:172.21.15.23:6789/0 conn(0x7f94bc0b28b0 legacy=0x7f94bc15c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:21.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.679+0000 7f94abfff700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:21.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.679+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94bc0c95d0 con 0x7f949c003740 2023-06-22T07:59:21.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.679+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0b48d0 con 0x7f949c007360 2023-06-22T07:59:21.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.679+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:21.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.680+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 msgr2=0x7f949c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:21.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.680+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 msgr2=0x7f949c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:21.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.680+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:21.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.680+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T07:59:21.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.881+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:21.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.881+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 msgr2=0x7f949c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:21.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.881+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 msgr2=0x7f949c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:21.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.881+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:21.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:21.881+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T07:59:22.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:22.282+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:22.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:22.282+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 msgr2=0x7f949c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:22.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:22.282+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 msgr2=0x7f949c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:22.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:22.282+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:22.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:22.282+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T07:59:23.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:23.084+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:23.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:23.084+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 msgr2=0x7f949c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:23.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:23.084+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 msgr2=0x7f949c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:23.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:23.084+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:23.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:23.084+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T07:59:24.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 msgr2=0x7f949c0097f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T07:59:24.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94abfff700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c007360 0x7f949c0097f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:24.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 >> v1:172.21.15.67:6789/0 conn(0x7f949c003740 legacy=0x7f949c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:24.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94bad9d700 1 -- 172.21.15.23:0/2538868767 reap_dead start 2023-06-22T07:59:24.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94abfff700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:24.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94abfff700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:24.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949c00af90 con 0x7f949c009880 2023-06-22T07:59:24.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc0b48d0 con 0x7f949c00a5c0 2023-06-22T07:59:24.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc0c92f0 con 0x7f94bc0b3d70 2023-06-22T07:59:24.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:24.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:24.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f949c00a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-22T07:59:24.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f949c00a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:24.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:24.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T07:59:24.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 msgr2=0x7f949c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:24.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 msgr2=0x7f949c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:24.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:24.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.680+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T07:59:24.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.881+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:24.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.881+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:24.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.881+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f949c00a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-22T07:59:24.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.881+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f949c00a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:24.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.881+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:24.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.881+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T07:59:24.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.881+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 msgr2=0x7f949c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:24.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.881+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 msgr2=0x7f949c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:24.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.881+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:24.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:24.881+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T07:59:25.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:25.282+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:25.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:25.282+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:25.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:25.282+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f949c00a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-22T07:59:25.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:25.282+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f949c00a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:25.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:25.282+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:25.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:25.282+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T07:59:25.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:25.282+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 msgr2=0x7f949c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:25.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:25.282+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 msgr2=0x7f949c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:25.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:25.282+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:25.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:25.282+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T07:59:26.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:26.083+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:26.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:26.083+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:26.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:26.083+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f949c00a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-22T07:59:26.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:26.083+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f949c00a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:26.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:26.083+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:26.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:26.083+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 msgr2=0x7f949c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:26.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:26.083+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 msgr2=0x7f949c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:26.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:26.083+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:26.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:26.083+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T07:59:26.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:26.083+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T07:59:27.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 msgr2=0x7f949c00a080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T07:59:27.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94abfff700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.67:3300/0 conn(0x7f94bc0b3d70 0x7f949c00a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:27.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 msgr2=0x7f949c00a990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T07:59:27.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94abfff700 1 --2- 172.21.15.23:0/2538868767 >> v2:172.21.15.23:3300/0 conn(0x7f949c00a5c0 0x7f949c00a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:27.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 >> v1:172.21.15.67:6789/0 conn(0x7f949c009880 legacy=0x7f949c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:27.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94abfff700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:27.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949c006510 con 0x7f949c006800 2023-06-22T07:59:27.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94abfff700 1 -- 172.21.15.23:0/2538868767 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0b48d0 con 0x7f949c0113f0 2023-06-22T07:59:27.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:27.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 msgr2=0x7f949c013800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:27.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 msgr2=0x7f949c013800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:27.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:27.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.680+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T07:59:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.881+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:27.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.881+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 msgr2=0x7f949c013800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:27.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.881+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 msgr2=0x7f949c013800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:27.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.881+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:27.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:27.881+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T07:59:28.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.282+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:28.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.282+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 msgr2=0x7f949c013800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-22T07:59:28.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.282+0000 7f94b9d9b700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 msgr2=0x7f949c013800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T07:59:28.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.282+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-06-22T07:59:28.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.282+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T07:59:28.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173887865 0 0) 0x7f949c006510 con 0x7f949c006800 2023-06-22T07:59:28.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a40035b0 con 0x7f949c006800 2023-06-22T07:59:28.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3871085267 0 0) 0x7f94a40035b0 con 0x7f949c006800 2023-06-22T07:59:28.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 msgr2=0x7f949c013800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T07:59:28.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f949c013800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949c014f60 con 0x7f949c006800 2023-06-22T07:59:28.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7f94ac001230 con 0x7f949c006800 2023-06-22T07:59:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 >> v1:172.21.15.23:6789/0 conn(0x7f949c006800 legacy=0x7f949c006c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94bad9d700 1 -- 172.21.15.23:0/2538868767 reap_dead start 2023-06-22T07:59:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4003a30 0x7f94a4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f94a4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94a4004d90 0x7f94a4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0b48d0 con 0x7f94a4004d90 2023-06-22T07:59:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a400bc20 con 0x7f94a4003a30 2023-06-22T07:59:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.732+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a400bde0 con 0x7f949c0113f0 2023-06-22T07:59:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.733+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4003a30 0x7f94a4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.733+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94a4004d90 0x7f94a4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.733+0000 7f94b9d9b700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f94a4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.733+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 msgr2=0x7f94a4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.733+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f94a4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.733+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94a4004d90 msgr2=0x7f94a4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.733+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94a4004d90 0x7f94a4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.734+0000 7f94ba59c700 1 -- 172.21.15.23:0/2538868767 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a40035b0 con 0x7f94a4003a30 2023-06-22T07:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.734+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4003a30 0x7f94a4003e20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f94ac001b70 tx=0x7f94a4005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb0f5256c993d3c0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.734+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac0052d0 con 0x7f94a4003a30 2023-06-22T07:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.734+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ac005430 con 0x7f94a4003a30 2023-06-22T07:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.734+0000 7f94b959a700 1 -- 172.21.15.23:0/2538868767 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac012ab0 con 0x7f94a4003a30 2023-06-22T07:59:28.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.735+0000 7f94c09ff700 1 -- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4003a30 msgr2=0x7f94a4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:28.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.735+0000 7f94c09ff700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4003a30 0x7f94a4003e20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f94ac001b70 tx=0x7f94a4005880 comp rx=0 tx=0).stop 2023-06-22T07:59:28.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.735+0000 7f94c09ff700 1 -- 172.21.15.23:0/2538868767 shutdown_connections 2023-06-22T07:59:28.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.735+0000 7f94c09ff700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94a4004d90 0x7f94a4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.735+0000 7f94c09ff700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4003a30 0x7f94a4003e20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.735+0000 7f94c09ff700 1 --2- 172.21.15.23:0/2538868767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f94a4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.735+0000 7f94c09ff700 1 -- 172.21.15.23:0/2538868767 >> 172.21.15.23:0/2538868767 conn(0x7f94bc06ba40 msgr2=0x7f94bc0715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.735+0000 7f94c09ff700 1 -- 172.21.15.23:0/2538868767 shutdown_connections 2023-06-22T07:59:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.736+0000 7f94c09ff700 1 -- 172.21.15.23:0/2538868767 wait complete. 2023-06-22T07:59:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.738+0000 7f94c09ff700 1 Processor -- start 2023-06-22T07:59:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.738+0000 7f94c09ff700 1 -- start start 2023-06-22T07:59:28.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94c09ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f94bc1ede20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:28.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94c09ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94a4003a30 0x7f94bc1ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:28.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94c09ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4004d90 0x7f94bc0ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:28.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94c09ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0cf220 con 0x7f94a4003a30 2023-06-22T07:59:28.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94c09ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc1ee360 con 0x7f94a4004d90 2023-06-22T07:59:28.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94c09ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0ce600 con 0x7f949c0113f0 2023-06-22T07:59:28.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94b9d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94a4003a30 0x7f94bc1ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:28.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94ba59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f94bc1ede20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:28.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94ba59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f94bc1ede20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:34636/0 (socket says 172.21.15.23:34636) 2023-06-22T07:59:28.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94ba59c700 1 -- 172.21.15.23:0/2933166131 learned_addr learned my addr 172.21.15.23:0/2933166131 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:28.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94bad9d700 1 --2- 172.21.15.23:0/2933166131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4004d90 0x7f94bc0ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:28.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94ba59c700 1 -- 172.21.15.23:0/2933166131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4004d90 msgr2=0x7f94bc0ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:28.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2933166131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4004d90 0x7f94bc0ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94ba59c700 1 -- 172.21.15.23:0/2933166131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94a4003a30 msgr2=0x7f94bc1ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:28.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2933166131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94a4003a30 0x7f94bc1ee810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.740+0000 7f94ba59c700 1 -- 172.21.15.23:0/2933166131 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc0cd690 con 0x7f949c0113f0 2023-06-22T07:59:28.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.741+0000 7f94ba59c700 1 --2- 172.21.15.23:0/2933166131 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f94bc1ede20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f94ac001860 tx=0x7f94ac001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:28.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.741+0000 7f94aaffd700 1 -- 172.21.15.23:0/2933166131 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac006ea0 con 0x7f949c0113f0 2023-06-22T07:59:28.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.741+0000 7f94aaffd700 1 -- 172.21.15.23:0/2933166131 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ac001230 con 0x7f949c0113f0 2023-06-22T07:59:28.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.741+0000 7f94aaffd700 1 -- 172.21.15.23:0/2933166131 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ac012ad0 con 0x7f949c0113f0 2023-06-22T07:59:28.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.741+0000 7f94c09ff700 1 -- 172.21.15.23:0/2933166131 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94bc0cdcd0 con 0x7f949c0113f0 2023-06-22T07:59:28.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.741+0000 7f94c09ff700 1 -- 172.21.15.23:0/2933166131 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94bc0ce290 con 0x7f949c0113f0 2023-06-22T07:59:28.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.742+0000 7f94aaffd700 1 -- 172.21.15.23:0/2933166131 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f94ac007000 con 0x7f949c0113f0 2023-06-22T07:59:28.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.742+0000 7f94aaffd700 1 -- 172.21.15.23:0/2933166131 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f94ac003970 con 0x7f949c0113f0 2023-06-22T07:59:28.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.743+0000 7f94c09ff700 1 -- 172.21.15.23:0/2933166131 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94bc0ce450 con 0x7f949c0113f0 2023-06-22T07:59:28.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.748+0000 7f94aaffd700 1 -- 172.21.15.23:0/2933166131 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f94ac02f060 con 0x7f949c0113f0 2023-06-22T07:59:28.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.831+0000 7f94c09ff700 1 -- 172.21.15.23:0/2933166131 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f94bc0ce450 con 0x7f949c0113f0 2023-06-22T07:59:28.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.842+0000 7f94aaffd700 1 -- 172.21.15.23:0/2933166131 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f94ac038020 con 0x7f949c0113f0 2023-06-22T07:59:28.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.843+0000 7f94c09ff700 1 -- 172.21.15.23:0/2933166131 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 msgr2=0x7f94bc1ede20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:28.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.843+0000 7f94c09ff700 1 --2- 172.21.15.23:0/2933166131 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f94bc1ede20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f94ac001860 tx=0x7f94ac001120 comp rx=0 tx=0).stop 2023-06-22T07:59:28.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.843+0000 7f94c09ff700 1 -- 172.21.15.23:0/2933166131 shutdown_connections 2023-06-22T07:59:28.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.843+0000 7f94c09ff700 1 --2- 172.21.15.23:0/2933166131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94a4004d90 0x7f94bc0ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.843+0000 7f94c09ff700 1 --2- 172.21.15.23:0/2933166131 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f949c0113f0 0x7f94bc1ede20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.843+0000 7f94c09ff700 1 --2- 172.21.15.23:0/2933166131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94a4003a30 0x7f94bc1ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.843+0000 7f94c09ff700 1 -- 172.21.15.23:0/2933166131 >> 172.21.15.23:0/2933166131 conn(0x7f94bc06ba40 msgr2=0x7f94bc0715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.843+0000 7f94c09ff700 1 -- 172.21.15.23:0/2933166131 shutdown_connections 2023-06-22T07:59:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:28.843+0000 7f94c09ff700 1 -- 172.21.15.23:0/2933166131 wait complete. 2023-06-22T07:59:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2023-06-22T07:59:28.864 INFO:tasks.ceph:check_enable_crimson: False 2023-06-22T07:59:28.864 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-06-22T07:59:28.865 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:28.865 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-06-22T07:59:28.867 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:28.864+0000 7f95b7507200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-22T07:59:28.900 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:28.897+0000 7f84a80f0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-22T07:59:28.934 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:28.934 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-06-22T07:59:28.935 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:28.933+0000 7f95b7507200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-22T07:59:28.988 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:28.986+0000 7f84a80f0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-22T07:59:29.014 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:29.012+0000 7f95b7507200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-22T07:59:29.042 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:29.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-06-22T07:59:29.092 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:29.089+0000 7f84a80f0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-22T07:59:29.096 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:29.093+0000 7f95b7507200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-22T07:59:29.150 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T07:59:29.151 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-06-22T07:59:29.199 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:29.196+0000 7f84a80f0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-22T07:59:29.244 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:29.241+0000 7f95b7507200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-22T07:59:29.259 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:29.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-06-22T07:59:29.289 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:29.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-06-22T07:59:29.358 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:29.358 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-06-22T07:59:29.388 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:29.385+0000 7f84a80f0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-22T07:59:29.425 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-22T07:59:29.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-06-22T07:59:29.492 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new df8cf946-2149-44cf-9e40-d646b8aca813 0 2023-06-22T07:59:29.623 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:29.620+0000 7f95b7507200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-22T07:59:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.653+0000 7f18e2417700 1 Processor -- start 2023-06-22T07:59:29.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.653+0000 7f18e2417700 1 -- start start 2023-06-22T07:59:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e2417700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f18dc0cc850 0x7f18dc157230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e2417700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0ccc20 0x7f18dc1505d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e2417700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc0c7700 con 0x7f18dc0c88b0 2023-06-22T07:59:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e2417700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18dc0c3050 con 0x7f18dc0cc850 2023-06-22T07:59:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e2417700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc0c2720 con 0x7f18dc0ccc20 2023-06-22T07:59:29.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e09b4700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0ccc20 0x7f18dc1505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:29.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e09b4700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0ccc20 0x7f18dc1505d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48316/0 (socket says 172.21.15.67:48316) 2023-06-22T07:59:29.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e09b4700 1 -- 172.21.15.67:0/1922511492 learned_addr learned my addr 172.21.15.67:0/1922511492 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:29.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18dbfff700 1 --2- 172.21.15.67:0/1922511492 >> v2:172.21.15.23:3300/0 conn(0x7f18dc0cc850 0x7f18dc157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:29.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e09b4700 1 -- 172.21.15.67:0/1922511492 >> v2:172.21.15.23:3300/0 conn(0x7f18dc0cc850 msgr2=0x7f18dc157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:29.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e09b4700 1 --2- 172.21.15.67:0/1922511492 >> v2:172.21.15.23:3300/0 conn(0x7f18dc0cc850 0x7f18dc157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e09b4700 1 -- 172.21.15.67:0/1922511492 >> v1:172.21.15.23:6789/0 conn(0x7f18dc0c88b0 legacy=0x7f18dc0c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:29.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e09b4700 1 -- 172.21.15.67:0/1922511492 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc0c79e0 con 0x7f18dc0ccc20 2023-06-22T07:59:29.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18dbfff700 1 --2- 172.21.15.67:0/1922511492 >> v2:172.21.15.23:3300/0 conn(0x7f18dc0cc850 0x7f18dc157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T07:59:29.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.654+0000 7f18e09b4700 1 --2- 172.21.15.67:0/1922511492 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0ccc20 0x7f18dc1505d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f18d4009e50 tx=0x7f18d400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ee044392fbd8672 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:29.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18daffd700 1 -- 172.21.15.67:0/1922511492 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d400e070 con 0x7f18dc0ccc20 2023-06-22T07:59:29.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18daffd700 1 -- 172.21.15.67:0/1922511492 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d4003a20 con 0x7f18dc0ccc20 2023-06-22T07:59:29.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18daffd700 1 -- 172.21.15.67:0/1922511492 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4003dd0 con 0x7f18dc0ccc20 2023-06-22T07:59:29.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18e2417700 1 -- 172.21.15.67:0/1922511492 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0ccc20 msgr2=0x7f18dc1505d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18e2417700 1 --2- 172.21.15.67:0/1922511492 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0ccc20 0x7f18dc1505d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f18d4009e50 tx=0x7f18d400b330 comp rx=0 tx=0).stop 2023-06-22T07:59:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18e2417700 1 -- 172.21.15.67:0/1922511492 shutdown_connections 2023-06-22T07:59:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18e2417700 1 --2- 172.21.15.67:0/1922511492 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0ccc20 0x7f18dc1505d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18e2417700 1 --2- 172.21.15.67:0/1922511492 >> v2:172.21.15.23:3300/0 conn(0x7f18dc0cc850 0x7f18dc157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18e2417700 1 -- 172.21.15.67:0/1922511492 >> 172.21.15.67:0/1922511492 conn(0x7f18dc06fb70 msgr2=0x7f18dc06ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18e2417700 1 -- 172.21.15.67:0/1922511492 shutdown_connections 2023-06-22T07:59:29.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.655+0000 7f18e2417700 1 -- 172.21.15.67:0/1922511492 wait complete. 2023-06-22T07:59:29.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.656+0000 7f18e2417700 1 Processor -- start 2023-06-22T07:59:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.657+0000 7f18e2417700 1 -- start start 2023-06-22T07:59:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.657+0000 7f18e2417700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0c88b0 0x7f18dc0ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.657+0000 7f18e2417700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18dc0cc850 0x7f18dc1e3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.657+0000 7f18e2417700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18dc0ccc20 0x7f18dc1f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.657+0000 7f18e2417700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc1e26c0 con 0x7f18dc0cc850 2023-06-22T07:59:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.657+0000 7f18e2417700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc1e29a0 con 0x7f18dc0ccc20 2023-06-22T07:59:29.664 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.657+0000 7f18e2417700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc0aed10 con 0x7f18dc0c88b0 2023-06-22T07:59:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.657+0000 7f18dbfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0c88b0 0x7f18dc0ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18dbfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0c88b0 0x7f18dc0ad2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48318/0 (socket says 172.21.15.67:48318) 2023-06-22T07:59:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18dbfff700 1 -- 172.21.15.67:0/2152281595 learned_addr learned my addr 172.21.15.67:0/2152281595 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18e09b4700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18dc0ccc20 0x7f18dc1f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18db7fe700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18dc0cc850 0x7f18dc1e3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18dbfff700 1 -- 172.21.15.67:0/2152281595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18dc0ccc20 msgr2=0x7f18dc1f1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:29.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18dbfff700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18dc0ccc20 0x7f18dc1f1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18dbfff700 1 -- 172.21.15.67:0/2152281595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18dc0cc850 msgr2=0x7f18dc1e3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:29.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18dbfff700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18dc0cc850 0x7f18dc1e3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18dbfff700 1 -- 172.21.15.67:0/2152281595 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4009b30 con 0x7f18dc0c88b0 2023-06-22T07:59:29.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18e09b4700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18dc0ccc20 0x7f18dc1f1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:29.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18db7fe700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18dc0cc850 0x7f18dc1e3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T07:59:29.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18dbfff700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0c88b0 0x7f18dc0ad2f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00c7f0 tx=0x7f18cc00cb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:29.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18d8ff9700 1 -- 172.21.15.67:0/2152281595 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc005c20 con 0x7f18dc0c88b0 2023-06-22T07:59:29.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18d8ff9700 1 -- 172.21.15.67:0/2152281595 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18cc005d80 con 0x7f18dc0c88b0 2023-06-22T07:59:29.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18e2417700 1 -- 172.21.15.67:0/2152281595 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc0aeff0 con 0x7f18dc0c88b0 2023-06-22T07:59:29.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18d8ff9700 1 -- 172.21.15.67:0/2152281595 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc011820 con 0x7f18dc0c88b0 2023-06-22T07:59:29.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18e2417700 1 -- 172.21.15.67:0/2152281595 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc1e2360 con 0x7f18dc0c88b0 2023-06-22T07:59:29.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18d8ff9700 1 -- 172.21.15.67:0/2152281595 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f18cc011980 con 0x7f18dc0c88b0 2023-06-22T07:59:29.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.658+0000 7f18d8ff9700 1 -- 172.21.15.67:0/2152281595 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f18cc0193f0 con 0x7f18dc0c88b0 2023-06-22T07:59:29.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.659+0000 7f18e2417700 1 -- 172.21.15.67:0/2152281595 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc1f1980 con 0x7f18dc0c88b0 2023-06-22T07:59:29.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.661+0000 7f18d8ff9700 1 -- 172.21.15.67:0/2152281595 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f18cc038b00 con 0x7f18dc0c88b0 2023-06-22T07:59:29.704 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:29.701+0000 7f95b7507200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-22T07:59:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.749+0000 7f18e2417700 1 -- 172.21.15.67:0/2152281595 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd new", "uuid": "df8cf946-2149-44cf-9e40-d646b8aca813", "id": 0} v 0) v1 -- 0x7f18dc1f1c60 con 0x7f18dc0c88b0 2023-06-22T07:59:29.758 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.756+0000 7f18d8ff9700 1 -- 172.21.15.67:0/2152281595 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "df8cf946-2149-44cf-9e40-d646b8aca813", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f18cc027070 con 0x7f18dc0c88b0 2023-06-22T07:59:29.759 INFO:teuthology.orchestra.run.smithi067.stdout:0 2023-06-22T07:59:29.760 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.757+0000 7f18e2417700 1 -- 172.21.15.67:0/2152281595 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0c88b0 msgr2=0x7f18dc0ad2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:29.760 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.757+0000 7f18e2417700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0c88b0 0x7f18dc0ad2f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00c7f0 tx=0x7f18cc00cb00 comp rx=0 tx=0).stop 2023-06-22T07:59:29.760 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.757+0000 7f18e2417700 1 -- 172.21.15.67:0/2152281595 shutdown_connections 2023-06-22T07:59:29.760 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.757+0000 7f18e2417700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18dc0ccc20 0x7f18dc1f1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.760 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.757+0000 7f18e2417700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18dc0c88b0 0x7f18dc0ad2f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.757+0000 7f18e2417700 1 --2- 172.21.15.67:0/2152281595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18dc0cc850 0x7f18dc1e3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.757+0000 7f18e2417700 1 -- 172.21.15.67:0/2152281595 >> 172.21.15.67:0/2152281595 conn(0x7f18dc06fb70 msgr2=0x7f18dc0caa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.757+0000 7f18e2417700 1 -- 172.21.15.67:0/2152281595 shutdown_connections 2023-06-22T07:59:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.757+0000 7f18e2417700 1 -- 172.21.15.67:0/2152281595 wait complete. 2023-06-22T07:59:29.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 078652c0-df39-43b4-af0f-79560de2d996 1 2023-06-22T07:59:29.803 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:29.800+0000 7f84a80f0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-22T07:59:29.838 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:29.836+0000 7f95b7507200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-22T07:59:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.892+0000 7fe77388e700 1 Processor -- start 2023-06-22T07:59:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.893+0000 7fe77388e700 1 -- start start 2023-06-22T07:59:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.893+0000 7fe77388e700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c152ce0 0x7fe76c070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.893+0000 7fe77388e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe76c0c5f30 0x7fe76c15ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77388e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76c0c6360 con 0x7fe76c159350 2023-06-22T07:59:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77388e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76c0c7880 con 0x7fe76c0c5f30 2023-06-22T07:59:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77388e700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c15bf60 con 0x7fe76c152ce0 2023-06-22T07:59:29.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77162a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c152ce0 0x7fe76c070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77162a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c152ce0 0x7fe76c070e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48324/0 (socket says 172.21.15.67:48324) 2023-06-22T07:59:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77162a700 1 -- 172.21.15.67:0/4236651521 learned_addr learned my addr 172.21.15.67:0/4236651521 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe770e29700 1 --2- 172.21.15.67:0/4236651521 >> v2:172.21.15.23:3300/0 conn(0x7fe76c0c5f30 0x7fe76c15ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77162a700 1 -- 172.21.15.67:0/4236651521 >> v2:172.21.15.23:3300/0 conn(0x7fe76c0c5f30 msgr2=0x7fe76c15ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77162a700 1 --2- 172.21.15.67:0/4236651521 >> v2:172.21.15.23:3300/0 conn(0x7fe76c0c5f30 0x7fe76c15ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.898 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77162a700 1 -- 172.21.15.67:0/4236651521 >> v1:172.21.15.23:6789/0 conn(0x7fe76c159350 legacy=0x7fe76c0cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77162a700 1 -- 172.21.15.67:0/4236651521 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c06eac0 con 0x7fe76c152ce0 2023-06-22T07:59:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe77162a700 1 --2- 172.21.15.67:0/4236651521 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c152ce0 0x7fe76c070e30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe7680098f0 tx=0x7fe768009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbd4d27d333ee53b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:29.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe763fff700 1 -- 172.21.15.67:0/4236651521 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768004230 con 0x7fe76c152ce0 2023-06-22T07:59:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe763fff700 1 -- 172.21.15.67:0/4236651521 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768004390 con 0x7fe76c152ce0 2023-06-22T07:59:29.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.894+0000 7fe763fff700 1 -- 172.21.15.67:0/4236651521 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680036a0 con 0x7fe76c152ce0 2023-06-22T07:59:29.901 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:29.898+0000 7f84a80f0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-22T07:59:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.895+0000 7fe77388e700 1 -- 172.21.15.67:0/4236651521 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c152ce0 msgr2=0x7fe76c070e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.895+0000 7fe77388e700 1 --2- 172.21.15.67:0/4236651521 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c152ce0 0x7fe76c070e30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe7680098f0 tx=0x7fe768009c00 comp rx=0 tx=0).stop 2023-06-22T07:59:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.895+0000 7fe77388e700 1 -- 172.21.15.67:0/4236651521 shutdown_connections 2023-06-22T07:59:29.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.895+0000 7fe77388e700 1 --2- 172.21.15.67:0/4236651521 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c152ce0 0x7fe76c070e30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.895+0000 7fe77388e700 1 --2- 172.21.15.67:0/4236651521 >> v2:172.21.15.23:3300/0 conn(0x7fe76c0c5f30 0x7fe76c15ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.895+0000 7fe77388e700 1 -- 172.21.15.67:0/4236651521 >> 172.21.15.67:0/4236651521 conn(0x7fe76c065b90 msgr2=0x7fe76c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.895+0000 7fe77388e700 1 -- 172.21.15.67:0/4236651521 shutdown_connections 2023-06-22T07:59:29.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.895+0000 7fe77388e700 1 -- 172.21.15.67:0/4236651521 wait complete. 2023-06-22T07:59:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.896+0000 7fe77388e700 1 Processor -- start 2023-06-22T07:59:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.896+0000 7fe77388e700 1 -- start start 2023-06-22T07:59:29.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77388e700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c0c5f30 0x7fe76c155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:29.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77388e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe76c152ce0 0x7fe76c0c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:29.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77388e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c159350 0x7fe76c151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:29.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77388e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1542a0 con 0x7fe76c152ce0 2023-06-22T07:59:29.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77388e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c0c47e0 con 0x7fe76c159350 2023-06-22T07:59:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77388e700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c154ab0 con 0x7fe76c0c5f30 2023-06-22T07:59:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77162a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c0c5f30 0x7fe76c155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77162a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c0c5f30 0x7fe76c155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48330/0 (socket says 172.21.15.67:48330) 2023-06-22T07:59:29.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77162a700 1 -- 172.21.15.67:0/3287499604 learned_addr learned my addr 172.21.15.67:0/3287499604 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe771e2b700 1 --2- 172.21.15.67:0/3287499604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c159350 0x7fe76c151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe770e29700 1 --2- 172.21.15.67:0/3287499604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe76c152ce0 0x7fe76c0c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77162a700 1 -- 172.21.15.67:0/3287499604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c159350 msgr2=0x7fe76c151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:29.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77162a700 1 --2- 172.21.15.67:0/3287499604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c159350 0x7fe76c151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77162a700 1 -- 172.21.15.67:0/3287499604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe76c152ce0 msgr2=0x7fe76c0c4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77162a700 1 --2- 172.21.15.67:0/3287499604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe76c152ce0 0x7fe76c0c4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.897+0000 7fe77162a700 1 -- 172.21.15.67:0/3287499604 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768009550 con 0x7fe76c0c5f30 2023-06-22T07:59:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.898+0000 7fe77162a700 1 --2- 172.21.15.67:0/3287499604 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c0c5f30 0x7fe76c155820 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe768009130 tx=0x7fe768004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:29.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.898+0000 7fe761ffb700 1 -- 172.21.15.67:0/3287499604 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768003aa0 con 0x7fe76c0c5f30 2023-06-22T07:59:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.898+0000 7fe761ffb700 1 -- 172.21.15.67:0/3287499604 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76801f070 con 0x7fe76c0c5f30 2023-06-22T07:59:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.898+0000 7fe77388e700 1 -- 172.21.15.67:0/3287499604 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c0cb030 con 0x7fe76c0c5f30 2023-06-22T07:59:29.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.898+0000 7fe761ffb700 1 -- 172.21.15.67:0/3287499604 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768003e40 con 0x7fe76c0c5f30 2023-06-22T07:59:29.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.898+0000 7fe77388e700 1 -- 172.21.15.67:0/3287499604 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c06b5f0 con 0x7fe76c0c5f30 2023-06-22T07:59:29.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.898+0000 7fe761ffb700 1 -- 172.21.15.67:0/3287499604 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe768021030 con 0x7fe76c0c5f30 2023-06-22T07:59:29.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.898+0000 7fe761ffb700 1 -- 172.21.15.67:0/3287499604 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fe76801a7b0 con 0x7fe76c0c5f30 2023-06-22T07:59:29.914 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.899+0000 7fe77388e700 1 -- 172.21.15.67:0/3287499604 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76c04ea20 con 0x7fe76c0c5f30 2023-06-22T07:59:29.914 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.903+0000 7fe761ffb700 1 -- 172.21.15.67:0/3287499604 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fe76800baf0 con 0x7fe76c0c5f30 2023-06-22T07:59:29.926 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:29.924+0000 7f95b7507200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-22T07:59:29.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:29.988+0000 7fe77388e700 1 -- 172.21.15.67:0/3287499604 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd new", "uuid": "078652c0-df39-43b4-af0f-79560de2d996", "id": 1} v 0) v1 -- 0x7fe76c04ea20 con 0x7fe76c0c5f30 2023-06-22T07:59:30.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.000+0000 7fe761ffb700 1 -- 172.21.15.67:0/3287499604 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "078652c0-df39-43b4-af0f-79560de2d996", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe76800bc90 con 0x7fe76c0c5f30 2023-06-22T07:59:30.003 INFO:teuthology.orchestra.run.smithi067.stdout:1 2023-06-22T07:59:30.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.001+0000 7fe77388e700 1 -- 172.21.15.67:0/3287499604 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c0c5f30 msgr2=0x7fe76c155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.001+0000 7fe77388e700 1 --2- 172.21.15.67:0/3287499604 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c0c5f30 0x7fe76c155820 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe768009130 tx=0x7fe768004750 comp rx=0 tx=0).stop 2023-06-22T07:59:30.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.001+0000 7fe77388e700 1 -- 172.21.15.67:0/3287499604 shutdown_connections 2023-06-22T07:59:30.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.001+0000 7fe77388e700 1 --2- 172.21.15.67:0/3287499604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe76c159350 0x7fe76c151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.001+0000 7fe77388e700 1 --2- 172.21.15.67:0/3287499604 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe76c0c5f30 0x7fe76c155820 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.001+0000 7fe77388e700 1 --2- 172.21.15.67:0/3287499604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe76c152ce0 0x7fe76c0c4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.001+0000 7fe77388e700 1 -- 172.21.15.67:0/3287499604 >> 172.21.15.67:0/3287499604 conn(0x7fe76c065b90 msgr2=0x7fe76c15ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:30.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.001+0000 7fe77388e700 1 -- 172.21.15.67:0/3287499604 shutdown_connections 2023-06-22T07:59:30.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.001+0000 7fe77388e700 1 -- 172.21.15.67:0/3287499604 wait complete. 2023-06-22T07:59:30.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 16b7f36c-0516-41a2-ab01-0f32e4e263c6 2 2023-06-22T07:59:30.065 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:30.062+0000 7f84a80f0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-22T07:59:30.140 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.138+0000 7fe145049700 1 Processor -- start 2023-06-22T07:59:30.141 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.138+0000 7fe145049700 1 -- start start 2023-06-22T07:59:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.138+0000 7fe145049700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe140071e70 0x7fe140070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.138+0000 7fe145049700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe1400cb9d0 0x7fe1400cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe145049700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe140153090 con 0x7fe140152cc0 2023-06-22T07:59:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe145049700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe140158bf0 con 0x7fe140071e70 2023-06-22T07:59:30.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe145049700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1401588e0 con 0x7fe1400cb9d0 2023-06-22T07:59:30.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13f59e700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe1400cb9d0 0x7fe1400cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13f59e700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe1400cb9d0 0x7fe1400cbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48340/0 (socket says 172.21.15.67:48340) 2023-06-22T07:59:30.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13f59e700 1 -- 172.21.15.67:0/185068576 learned_addr learned my addr 172.21.15.67:0/185068576 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:30.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13e59c700 1 --2- 172.21.15.67:0/185068576 >> v2:172.21.15.23:3300/0 conn(0x7fe140071e70 0x7fe140070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13f59e700 1 -- 172.21.15.67:0/185068576 >> v2:172.21.15.23:3300/0 conn(0x7fe140071e70 msgr2=0x7fe140070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:30.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13f59e700 1 --2- 172.21.15.67:0/185068576 >> v2:172.21.15.23:3300/0 conn(0x7fe140071e70 0x7fe140070e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13f59e700 1 -- 172.21.15.67:0/185068576 >> v1:172.21.15.23:6789/0 conn(0x7fe140152cc0 legacy=0x7fe140159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13f59e700 1 -- 172.21.15.67:0/185068576 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14015ce60 con 0x7fe1400cb9d0 2023-06-22T07:59:30.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13f59e700 1 --2- 172.21.15.67:0/185068576 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe1400cb9d0 0x7fe1400cbe30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe138009e50 tx=0x7fe13800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc94319b583cbeab server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:30.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13dd9b700 1 -- 172.21.15.67:0/185068576 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13800e070 con 0x7fe1400cb9d0 2023-06-22T07:59:30.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13dd9b700 1 -- 172.21.15.67:0/185068576 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13800b910 con 0x7fe1400cb9d0 2023-06-22T07:59:30.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.139+0000 7fe13dd9b700 1 -- 172.21.15.67:0/185068576 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13800bcc0 con 0x7fe1400cb9d0 2023-06-22T07:59:30.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.140+0000 7fe145049700 1 -- 172.21.15.67:0/185068576 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe1400cb9d0 msgr2=0x7fe1400cbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.140+0000 7fe145049700 1 --2- 172.21.15.67:0/185068576 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe1400cb9d0 0x7fe1400cbe30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe138009e50 tx=0x7fe13800b330 comp rx=0 tx=0).stop 2023-06-22T07:59:30.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.140+0000 7fe145049700 1 -- 172.21.15.67:0/185068576 shutdown_connections 2023-06-22T07:59:30.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.140+0000 7fe145049700 1 --2- 172.21.15.67:0/185068576 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe1400cb9d0 0x7fe1400cbe30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.140+0000 7fe145049700 1 --2- 172.21.15.67:0/185068576 >> v2:172.21.15.23:3300/0 conn(0x7fe140071e70 0x7fe140070e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.140+0000 7fe145049700 1 -- 172.21.15.67:0/185068576 >> 172.21.15.67:0/185068576 conn(0x7fe140065b90 msgr2=0x7fe140065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.140+0000 7fe145049700 1 -- 172.21.15.67:0/185068576 shutdown_connections 2023-06-22T07:59:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.140+0000 7fe145049700 1 -- 172.21.15.67:0/185068576 wait complete. 2023-06-22T07:59:30.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe145049700 1 Processor -- start 2023-06-22T07:59:30.153 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe145049700 1 -- start start 2023-06-22T07:59:30.153 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe145049700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe140071e70 0x7fe14015b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe145049700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1400cb9d0 0x7fe1401f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe145049700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe140152cc0 0x7fe1400ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe145049700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14015bd70 con 0x7fe140152cc0 2023-06-22T07:59:30.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe145049700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1400c3f70 con 0x7fe1400cb9d0 2023-06-22T07:59:30.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe145049700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1401f28f0 con 0x7fe140071e70 2023-06-22T07:59:30.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe13ed9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe140071e70 0x7fe14015b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe13ed9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe140071e70 0x7fe14015b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48342/0 (socket says 172.21.15.67:48342) 2023-06-22T07:59:30.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe13ed9d700 1 -- 172.21.15.67:0/421353674 learned_addr learned my addr 172.21.15.67:0/421353674 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:30.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.142+0000 7fe13e59c700 1 --2- 172.21.15.67:0/421353674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1400cb9d0 0x7fe1401f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe13f59e700 1 --2- 172.21.15.67:0/421353674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe140152cc0 0x7fe1400ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe13ed9d700 1 -- 172.21.15.67:0/421353674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1400cb9d0 msgr2=0x7fe1401f23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe13ed9d700 1 --2- 172.21.15.67:0/421353674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1400cb9d0 0x7fe1401f23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe13ed9d700 1 -- 172.21.15.67:0/421353674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe140152cc0 msgr2=0x7fe1400ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe13ed9d700 1 --2- 172.21.15.67:0/421353674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe140152cc0 0x7fe1400ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe13ed9d700 1 -- 172.21.15.67:0/421353674 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe138009b30 con 0x7fe140071e70 2023-06-22T07:59:30.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe13ed9d700 1 --2- 172.21.15.67:0/421353674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe140071e70 0x7fe14015b6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe13000c450 tx=0x7fe13000c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:30.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe1277fe700 1 -- 172.21.15.67:0/421353674 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1300052d0 con 0x7fe140071e70 2023-06-22T07:59:30.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe1277fe700 1 -- 172.21.15.67:0/421353674 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe130005430 con 0x7fe140071e70 2023-06-22T07:59:30.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe145049700 1 -- 172.21.15.67:0/421353674 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1400c3360 con 0x7fe140071e70 2023-06-22T07:59:30.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe1277fe700 1 -- 172.21.15.67:0/421353674 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1300038c0 con 0x7fe140071e70 2023-06-22T07:59:30.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe145049700 1 -- 172.21.15.67:0/421353674 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1400c3920 con 0x7fe140071e70 2023-06-22T07:59:30.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe1277fe700 1 -- 172.21.15.67:0/421353674 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe130003d20 con 0x7fe140071e70 2023-06-22T07:59:30.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.143+0000 7fe1277fe700 1 -- 172.21.15.67:0/421353674 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe130011d10 con 0x7fe140071e70 2023-06-22T07:59:30.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.144+0000 7fe145049700 1 -- 172.21.15.67:0/421353674 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14015a1e0 con 0x7fe140071e70 2023-06-22T07:59:30.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.146+0000 7fe1277fe700 1 -- 172.21.15.67:0/421353674 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fe14015a1e0 con 0x7fe140071e70 2023-06-22T07:59:30.169 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:30.166+0000 7f84a80f0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-22T07:59:30.234 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.231+0000 7fe145049700 1 -- 172.21.15.67:0/421353674 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd new", "uuid": "16b7f36c-0516-41a2-ab01-0f32e4e263c6", "id": 2} v 0) v1 -- 0x7fe14004ea20 con 0x7fe140071e70 2023-06-22T07:59:30.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.239+0000 7fe1277fe700 1 -- 172.21.15.67:0/421353674 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "16b7f36c-0516-41a2-ab01-0f32e4e263c6", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe130017070 con 0x7fe140071e70 2023-06-22T07:59:30.242 INFO:teuthology.orchestra.run.smithi067.stdout:2 2023-06-22T07:59:30.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.240+0000 7fe145049700 1 -- 172.21.15.67:0/421353674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe140071e70 msgr2=0x7fe14015b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.240+0000 7fe145049700 1 --2- 172.21.15.67:0/421353674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe140071e70 0x7fe14015b6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe13000c450 tx=0x7fe13000c760 comp rx=0 tx=0).stop 2023-06-22T07:59:30.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.240+0000 7fe145049700 1 -- 172.21.15.67:0/421353674 shutdown_connections 2023-06-22T07:59:30.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.240+0000 7fe145049700 1 --2- 172.21.15.67:0/421353674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe140152cc0 0x7fe1400ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.240+0000 7fe145049700 1 --2- 172.21.15.67:0/421353674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe140071e70 0x7fe14015b6c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.240+0000 7fe145049700 1 --2- 172.21.15.67:0/421353674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1400cb9d0 0x7fe1401f23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.240+0000 7fe145049700 1 -- 172.21.15.67:0/421353674 >> 172.21.15.67:0/421353674 conn(0x7fe140065b90 msgr2=0x7fe140156730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:30.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.240+0000 7fe145049700 1 -- 172.21.15.67:0/421353674 shutdown_connections 2023-06-22T07:59:30.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.240+0000 7fe145049700 1 -- 172.21.15.67:0/421353674 wait complete. 2023-06-22T07:59:30.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 82c3c1dc-3259-4a66-984e-6696d109a377 3 2023-06-22T07:59:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.377+0000 7fb3a9c4a700 1 Processor -- start 2023-06-22T07:59:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.377+0000 7fb3a9c4a700 1 -- start start 2023-06-22T07:59:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a9c4a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb3a4150680 0x7fb3a4071cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.381 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a9c4a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a406f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a9c4a700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a40c4c60 con 0x7fb3a4156740 2023-06-22T07:59:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a9c4a700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a4156ba0 con 0x7fb3a4150680 2023-06-22T07:59:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a9c4a700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40722a0 con 0x7fb3a40714c0 2023-06-22T07:59:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a2ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a406f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a2ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a406f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48350/0 (socket says 172.21.15.67:48350) 2023-06-22T07:59:30.382 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a2ffd700 1 -- 172.21.15.67:0/4259887666 learned_addr learned my addr 172.21.15.67:0/4259887666 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a37fe700 1 --2- 172.21.15.67:0/4259887666 >> v2:172.21.15.23:3300/0 conn(0x7fb3a4150680 0x7fb3a4071cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a2ffd700 1 -- 172.21.15.67:0/4259887666 >> v2:172.21.15.23:3300/0 conn(0x7fb3a4150680 msgr2=0x7fb3a4071cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a2ffd700 1 --2- 172.21.15.67:0/4259887666 >> v2:172.21.15.23:3300/0 conn(0x7fb3a4150680 0x7fb3a4071cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a2ffd700 1 -- 172.21.15.67:0/4259887666 >> v1:172.21.15.67:6789/0 conn(0x7fb3a4156740 legacy=0x7fb3a40c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a2ffd700 1 -- 172.21.15.67:0/4259887666 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a40c4eb0 con 0x7fb3a40714c0 2023-06-22T07:59:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.378+0000 7fb3a27fc700 1 -- 172.21.15.67:0/4259887666 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469240272 0 0) 0x7fb3a40c4c60 con 0x7fb3a4156740 2023-06-22T07:59:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.379+0000 7fb3a2ffd700 1 --2- 172.21.15.67:0/4259887666 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a406f350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb3980098f0 tx=0x7fb398009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec4b37864b2b633f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.379+0000 7fb3a27fc700 1 -- 172.21.15.67:0/4259887666 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398011830 con 0x7fb3a40714c0 2023-06-22T07:59:30.384 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.379+0000 7fb3a27fc700 1 -- 172.21.15.67:0/4259887666 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb398011990 con 0x7fb3a40714c0 2023-06-22T07:59:30.384 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.379+0000 7fb3a27fc700 1 -- 172.21.15.67:0/4259887666 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398011d20 con 0x7fb3a40714c0 2023-06-22T07:59:30.386 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.379+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/4259887666 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 msgr2=0x7fb3a406f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.386 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.379+0000 7fb3a9c4a700 1 --2- 172.21.15.67:0/4259887666 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a406f350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb3980098f0 tx=0x7fb398009c00 comp rx=0 tx=0).stop 2023-06-22T07:59:30.386 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.379+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/4259887666 shutdown_connections 2023-06-22T07:59:30.386 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.379+0000 7fb3a9c4a700 1 --2- 172.21.15.67:0/4259887666 >> v2:172.21.15.23:3300/0 conn(0x7fb3a4150680 0x7fb3a4071cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.379+0000 7fb3a9c4a700 1 --2- 172.21.15.67:0/4259887666 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a406f350 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.379+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/4259887666 >> 172.21.15.67:0/4259887666 conn(0x7fb3a40cc2e0 msgr2=0x7fb3a40c80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.380+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/4259887666 shutdown_connections 2023-06-22T07:59:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.380+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/4259887666 wait complete. 2023-06-22T07:59:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.381+0000 7fb3a9c4a700 1 Processor -- start 2023-06-22T07:59:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.381+0000 7fb3a9c4a700 1 -- start start 2023-06-22T07:59:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a9c4a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a414f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a9c4a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4150680 0x7fb3a41eeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a9c4a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3a4156740 0x7fb3a41ef1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a9c4a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a4148dc0 con 0x7fb3a4150680 2023-06-22T07:59:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a9c4a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40cac70 con 0x7fb3a4156740 2023-06-22T07:59:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a9c4a700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40cb440 con 0x7fb3a40714c0 2023-06-22T07:59:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a37fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a414f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3a4156740 0x7fb3a41ef1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a37fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a414f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48354/0 (socket says 172.21.15.67:48354) 2023-06-22T07:59:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a37fe700 1 -- 172.21.15.67:0/231469906 learned_addr learned my addr 172.21.15.67:0/231469906 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.382+0000 7fb3a2ffd700 1 --2- 172.21.15.67:0/231469906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4150680 0x7fb3a41eeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb3a3fff700 1 -- 172.21.15.67:0/231469906 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 msgr2=0x7fb3a414f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb3a3fff700 1 --2- 172.21.15.67:0/231469906 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a414f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb3a3fff700 1 -- 172.21.15.67:0/231469906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4150680 msgr2=0x7fb3a41eeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb3a3fff700 1 --2- 172.21.15.67:0/231469906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4150680 0x7fb3a41eeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb3a3fff700 1 -- 172.21.15.67:0/231469906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb398009550 con 0x7fb3a4156740 2023-06-22T07:59:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb3a2ffd700 1 --2- 172.21.15.67:0/231469906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4150680 0x7fb3a41eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T07:59:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb3a37fe700 1 --2- 172.21.15.67:0/231469906 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a414f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb3a3fff700 1 --2- 172.21.15.67:0/231469906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3a4156740 0x7fb3a41ef1e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00c780 tx=0x7fb39c00ca90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb38bfff700 1 -- 172.21.15.67:0/231469906 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c003e30 con 0x7fb3a4156740 2023-06-22T07:59:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb38bfff700 1 -- 172.21.15.67:0/231469906 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c005070 con 0x7fb3a4156740 2023-06-22T07:59:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/231469906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a4153d90 con 0x7fb3a4156740 2023-06-22T07:59:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb38bfff700 1 -- 172.21.15.67:0/231469906 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c005420 con 0x7fb3a4156740 2023-06-22T07:59:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/231469906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a40cb160 con 0x7fb3a4156740 2023-06-22T07:59:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb38bfff700 1 -- 172.21.15.67:0/231469906 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb39c01c440 con 0x7fb3a4156740 2023-06-22T07:59:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.383+0000 7fb38bfff700 1 -- 172.21.15.67:0/231469906 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fb39c012890 con 0x7fb3a4156740 2023-06-22T07:59:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.384+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/231469906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a404ea20 con 0x7fb3a4156740 2023-06-22T07:59:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.387+0000 7fb38bfff700 1 -- 172.21.15.67:0/231469906 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fb39c017070 con 0x7fb3a4156740 2023-06-22T07:59:30.392 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:30.388+0000 7f95b7507200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-22T07:59:30.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.472+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/231469906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd new", "uuid": "82c3c1dc-3259-4a66-984e-6696d109a377", "id": 3} v 0) v1 -- 0x7fb3a414fea0 con 0x7fb3a4156740 2023-06-22T07:59:30.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.475+0000 7fb38bfff700 1 -- 172.21.15.67:0/231469906 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "82c3c1dc-3259-4a66-984e-6696d109a377", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb39c012530 con 0x7fb3a4156740 2023-06-22T07:59:30.478 INFO:teuthology.orchestra.run.smithi067.stdout:3 2023-06-22T07:59:30.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.476+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/231469906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3a4156740 msgr2=0x7fb3a41ef1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.476+0000 7fb3a9c4a700 1 --2- 172.21.15.67:0/231469906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3a4156740 0x7fb3a41ef1e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00c780 tx=0x7fb39c00ca90 comp rx=0 tx=0).stop 2023-06-22T07:59:30.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.476+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/231469906 shutdown_connections 2023-06-22T07:59:30.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.476+0000 7fb3a9c4a700 1 --2- 172.21.15.67:0/231469906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3a4156740 0x7fb3a41ef1e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.476+0000 7fb3a9c4a700 1 --2- 172.21.15.67:0/231469906 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3a40714c0 0x7fb3a414f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.476+0000 7fb3a9c4a700 1 --2- 172.21.15.67:0/231469906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4150680 0x7fb3a41eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.476+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/231469906 >> 172.21.15.67:0/231469906 conn(0x7fb3a40cc2e0 msgr2=0x7fb3a41543f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:30.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.476+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/231469906 shutdown_connections 2023-06-22T07:59:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.476+0000 7fb3a9c4a700 1 -- 172.21.15.67:0/231469906 wait complete. 2023-06-22T07:59:30.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 481e2a90-480b-459d-82b1-62b4f4758233 4 2023-06-22T07:59:30.613 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.611+0000 7f506aa93700 1 Processor -- start 2023-06-22T07:59:30.614 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.611+0000 7f506aa93700 1 -- start start 2023-06-22T07:59:30.616 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f506aa93700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5064152ce0 0x7f5064070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.616 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f506aa93700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50640c5f30 0x7f506415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f506aa93700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50640c6360 con 0x7f5064159350 2023-06-22T07:59:30.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f506aa93700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50640c7880 con 0x7f5064152ce0 2023-06-22T07:59:30.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f506aa93700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506415bf60 con 0x7f50640c5f30 2023-06-22T07:59:30.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f5063fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50640c5f30 0x7f506415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f5063fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50640c5f30 0x7f506415ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48360/0 (socket says 172.21.15.67:48360) 2023-06-22T07:59:30.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f5063fff700 1 -- 172.21.15.67:0/4058572290 learned_addr learned my addr 172.21.15.67:0/4058572290 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:30.618 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f506882f700 1 --2- 172.21.15.67:0/4058572290 >> v2:172.21.15.23:3300/0 conn(0x7f5064152ce0 0x7f5064070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.618 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f5063fff700 1 -- 172.21.15.67:0/4058572290 >> v2:172.21.15.23:3300/0 conn(0x7f5064152ce0 msgr2=0x7f5064070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:30.618 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f5063fff700 1 --2- 172.21.15.67:0/4058572290 >> v2:172.21.15.23:3300/0 conn(0x7f5064152ce0 0x7f5064070e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.618 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f5063fff700 1 -- 172.21.15.67:0/4058572290 >> v1:172.21.15.23:6789/0 conn(0x7f5064159350 legacy=0x7f50640cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:30.618 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f5063fff700 1 -- 172.21.15.67:0/4058572290 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506406eac0 con 0x7f50640c5f30 2023-06-22T07:59:30.618 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.612+0000 7f5063fff700 1 --2- 172.21.15.67:0/4058572290 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50640c5f30 0x7f506415ca00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f50500098f0 tx=0x7f5050009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd8af0f8037bf303 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:30.618 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f50637fe700 1 -- 172.21.15.67:0/4058572290 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5050004230 con 0x7f50640c5f30 2023-06-22T07:59:30.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f50637fe700 1 -- 172.21.15.67:0/4058572290 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5050004390 con 0x7f50640c5f30 2023-06-22T07:59:30.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f50637fe700 1 -- 172.21.15.67:0/4058572290 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50500036a0 con 0x7f50640c5f30 2023-06-22T07:59:30.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f506aa93700 1 -- 172.21.15.67:0/4058572290 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50640c5f30 msgr2=0x7f506415ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f506aa93700 1 --2- 172.21.15.67:0/4058572290 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50640c5f30 0x7f506415ca00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f50500098f0 tx=0x7f5050009c00 comp rx=0 tx=0).stop 2023-06-22T07:59:30.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f506aa93700 1 -- 172.21.15.67:0/4058572290 shutdown_connections 2023-06-22T07:59:30.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f506aa93700 1 --2- 172.21.15.67:0/4058572290 >> v2:172.21.15.23:3300/0 conn(0x7f5064152ce0 0x7f5064070e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f506aa93700 1 --2- 172.21.15.67:0/4058572290 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50640c5f30 0x7f506415ca00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f506aa93700 1 -- 172.21.15.67:0/4058572290 >> 172.21.15.67:0/4058572290 conn(0x7f5064065b90 msgr2=0x7f5064065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:30.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f506aa93700 1 -- 172.21.15.67:0/4058572290 shutdown_connections 2023-06-22T07:59:30.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.613+0000 7f506aa93700 1 -- 172.21.15.67:0/4058572290 wait complete. 2023-06-22T07:59:30.622 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.614+0000 7f506aa93700 1 Processor -- start 2023-06-22T07:59:30.622 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.614+0000 7f506aa93700 1 -- start start 2023-06-22T07:59:30.622 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f506aa93700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50640c5f30 0x7f50641f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.622 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f506aa93700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5064152ce0 0x7f506415ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.623 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f506aa93700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5064159350 0x7f50641f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.623 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f506aa93700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5064155850 con 0x7f50640c5f30 2023-06-22T07:59:30.623 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f506aa93700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50641f2c00 con 0x7f5064159350 2023-06-22T07:59:30.623 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f506aa93700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5064157890 con 0x7f5064152ce0 2023-06-22T07:59:30.623 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f5069030700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5064159350 0x7f50641f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.623 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f5063fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5064152ce0 0x7f506415ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.623 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f5069030700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5064159350 0x7f50641f05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55560/0 (socket says 172.21.15.67:55560) 2023-06-22T07:59:30.624 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f5063fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5064152ce0 0x7f506415ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48366/0 (socket says 172.21.15.67:48366) 2023-06-22T07:59:30.624 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.615+0000 7f5069030700 1 -- 172.21.15.67:0/110618283 learned_addr learned my addr 172.21.15.67:0/110618283 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:30.624 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f506882f700 1 --2- 172.21.15.67:0/110618283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50640c5f30 0x7f50641f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.624 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f5063fff700 1 -- 172.21.15.67:0/110618283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5064159350 msgr2=0x7f50641f05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.624 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f5063fff700 1 --2- 172.21.15.67:0/110618283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5064159350 0x7f50641f05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.624 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f5063fff700 1 -- 172.21.15.67:0/110618283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50640c5f30 msgr2=0x7f50641f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.624 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f5063fff700 1 --2- 172.21.15.67:0/110618283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50640c5f30 0x7f50641f26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.625 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f5063fff700 1 -- 172.21.15.67:0/110618283 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5050009550 con 0x7f5064152ce0 2023-06-22T07:59:30.625 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f506882f700 1 --2- 172.21.15.67:0/110618283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50640c5f30 0x7f50641f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T07:59:30.625 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f5063fff700 1 --2- 172.21.15.67:0/110618283 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5064152ce0 0x7f506415ac70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5050015000 tx=0x7f5050004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:30.625 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f50617fa700 1 -- 172.21.15.67:0/110618283 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5050003c10 con 0x7f5064152ce0 2023-06-22T07:59:30.625 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f50617fa700 1 -- 172.21.15.67:0/110618283 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505001f070 con 0x7f5064152ce0 2023-06-22T07:59:30.625 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f506aa93700 1 -- 172.21.15.67:0/110618283 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5064157b70 con 0x7f5064152ce0 2023-06-22T07:59:30.625 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f50617fa700 1 -- 172.21.15.67:0/110618283 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505000b660 con 0x7f5064152ce0 2023-06-22T07:59:30.626 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f506aa93700 1 -- 172.21.15.67:0/110618283 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50641efda0 con 0x7f5064152ce0 2023-06-22T07:59:30.626 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f50617fa700 1 -- 172.21.15.67:0/110618283 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5050021030 con 0x7f5064152ce0 2023-06-22T07:59:30.626 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.616+0000 7f50617fa700 1 -- 172.21.15.67:0/110618283 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f505001ac70 con 0x7f5064152ce0 2023-06-22T07:59:30.626 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.617+0000 7f506aa93700 1 -- 172.21.15.67:0/110618283 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5064158090 con 0x7f5064152ce0 2023-06-22T07:59:30.626 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.619+0000 7f50617fa700 1 -- 172.21.15.67:0/110618283 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f5064158090 con 0x7f5064152ce0 2023-06-22T07:59:30.700 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:30.697+0000 7f84a80f0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-22T07:59:30.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.704+0000 7f506aa93700 1 -- 172.21.15.67:0/110618283 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd new", "uuid": "481e2a90-480b-459d-82b1-62b4f4758233", "id": 4} v 0) v1 -- 0x7f50641efc00 con 0x7f5064152ce0 2023-06-22T07:59:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.712+0000 7f50617fa700 1 -- 172.21.15.67:0/110618283 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "481e2a90-480b-459d-82b1-62b4f4758233", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f50641efc00 con 0x7f5064152ce0 2023-06-22T07:59:30.714 INFO:teuthology.orchestra.run.smithi067.stdout:4 2023-06-22T07:59:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.713+0000 7f506aa93700 1 -- 172.21.15.67:0/110618283 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5064152ce0 msgr2=0x7f506415ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.715 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.713+0000 7f506aa93700 1 --2- 172.21.15.67:0/110618283 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5064152ce0 0x7f506415ac70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5050015000 tx=0x7f5050004750 comp rx=0 tx=0).stop 2023-06-22T07:59:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.713+0000 7f506aa93700 1 -- 172.21.15.67:0/110618283 shutdown_connections 2023-06-22T07:59:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.713+0000 7f506aa93700 1 --2- 172.21.15.67:0/110618283 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5064159350 0x7f50641f05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.713+0000 7f506aa93700 1 --2- 172.21.15.67:0/110618283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50640c5f30 0x7f50641f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.713+0000 7f506aa93700 1 --2- 172.21.15.67:0/110618283 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5064152ce0 0x7f506415ac70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.713+0000 7f506aa93700 1 -- 172.21.15.67:0/110618283 >> 172.21.15.67:0/110618283 conn(0x7f5064065b90 msgr2=0x7f50640ca380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.713+0000 7f506aa93700 1 -- 172.21.15.67:0/110618283 shutdown_connections 2023-06-22T07:59:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.713+0000 7f506aa93700 1 -- 172.21.15.67:0/110618283 wait complete. 2023-06-22T07:59:30.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new a845ba15-d7a0-4542-8367-caf4a9ac4bdd 5 2023-06-22T07:59:30.891 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.888+0000 7fc3268b3700 1 Processor -- start 2023-06-22T07:59:30.891 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.888+0000 7fc3268b3700 1 -- start start 2023-06-22T07:59:30.893 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc3268b3700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc320152ce0 0x7fc320070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.893 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc3268b3700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc320159350 0x7fc3200cba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.893 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc3268b3700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32006e9b0 con 0x7fc32015cb20 2023-06-22T07:59:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc3268b3700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc32015c690 con 0x7fc320159350 2023-06-22T07:59:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc3268b3700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3201597f0 con 0x7fc320152ce0 2023-06-22T07:59:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc31ffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc320152ce0 0x7fc320070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc31f7fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc32015cb20 0x7fc3200c5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39384/0 (socket says 172.21.15.67:39384) 2023-06-22T07:59:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc31f7fe700 1 -- 172.21.15.67:0/651347058 learned_addr learned my addr 172.21.15.67:0/651347058 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-22T07:59:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc324e50700 1 --2- 172.21.15.67:0/651347058 >> v2:172.21.15.23:3300/0 conn(0x7fc320159350 0x7fc3200cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc31ffff700 1 -- 172.21.15.67:0/651347058 >> v2:172.21.15.23:3300/0 conn(0x7fc320159350 msgr2=0x7fc3200cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:30.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc31ffff700 1 --2- 172.21.15.67:0/651347058 >> v2:172.21.15.23:3300/0 conn(0x7fc320159350 0x7fc3200cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc31ffff700 1 -- 172.21.15.67:0/651347058 >> v1:172.21.15.67:6789/0 conn(0x7fc32015cb20 legacy=0x7fc3200c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc31ffff700 1 -- 172.21.15.67:0/651347058 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32006ec00 con 0x7fc320152ce0 2023-06-22T07:59:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc31effd700 1 -- 172.21.15.67:0/651347058 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161312105 0 0) 0x7fc32006e9b0 con 0x7fc32015cb20 2023-06-22T07:59:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.889+0000 7fc324e50700 1 --2- 172.21.15.67:0/651347058 >> v2:172.21.15.23:3300/0 conn(0x7fc320159350 0x7fc3200cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T07:59:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc31ffff700 1 --2- 172.21.15.67:0/651347058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320152ce0 0x7fc320070e30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc3200c9020 tx=0x7fc310009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=691c7dacf68d8f12 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:30.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc31effd700 1 -- 172.21.15.67:0/651347058 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc310011830 con 0x7fc320152ce0 2023-06-22T07:59:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc31effd700 1 -- 172.21.15.67:0/651347058 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc310011990 con 0x7fc320152ce0 2023-06-22T07:59:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc31effd700 1 -- 172.21.15.67:0/651347058 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc310011d20 con 0x7fc320152ce0 2023-06-22T07:59:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc3268b3700 1 -- 172.21.15.67:0/651347058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320152ce0 msgr2=0x7fc320070e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc3268b3700 1 --2- 172.21.15.67:0/651347058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320152ce0 0x7fc320070e30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc3200c9020 tx=0x7fc310009b20 comp rx=0 tx=0).stop 2023-06-22T07:59:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc3268b3700 1 -- 172.21.15.67:0/651347058 shutdown_connections 2023-06-22T07:59:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc3268b3700 1 --2- 172.21.15.67:0/651347058 >> v2:172.21.15.23:3300/0 conn(0x7fc320159350 0x7fc3200cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc3268b3700 1 --2- 172.21.15.67:0/651347058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320152ce0 0x7fc320070e30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc3268b3700 1 -- 172.21.15.67:0/651347058 >> 172.21.15.67:0/651347058 conn(0x7fc320065b90 msgr2=0x7fc320065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc3268b3700 1 -- 172.21.15.67:0/651347058 shutdown_connections 2023-06-22T07:59:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.890+0000 7fc3268b3700 1 -- 172.21.15.67:0/651347058 wait complete. 2023-06-22T07:59:30.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.892+0000 7fc3268b3700 1 Processor -- start 2023-06-22T07:59:30.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.892+0000 7fc3268b3700 1 -- start start 2023-06-22T07:59:30.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.892+0000 7fc3268b3700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc320152ce0 0x7fc320158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.892+0000 7fc3268b3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320159350 0x7fc320155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.892+0000 7fc3268b3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc32015cb20 0x7fc3200c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:30.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.892+0000 7fc3268b3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3201533b0 con 0x7fc32015cb20 2023-06-22T07:59:30.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.892+0000 7fc3268b3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc320156740 con 0x7fc320159350 2023-06-22T07:59:30.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.892+0000 7fc3268b3700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc320153c80 con 0x7fc320152ce0 2023-06-22T07:59:30.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320159350 0x7fc320155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc320152ce0 0x7fc320158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320159350 0x7fc320155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55570/0 (socket says 172.21.15.67:55570) 2023-06-22T07:59:30.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31f7fe700 1 -- 172.21.15.67:0/2763375877 learned_addr learned my addr 172.21.15.67:0/2763375877 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:30.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc324e50700 1 --2- 172.21.15.67:0/2763375877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc32015cb20 0x7fc3200c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:30.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31f7fe700 1 -- 172.21.15.67:0/2763375877 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc320152ce0 msgr2=0x7fc320158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31f7fe700 1 --2- 172.21.15.67:0/2763375877 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc320152ce0 0x7fc320158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31f7fe700 1 -- 172.21.15.67:0/2763375877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc32015cb20 msgr2=0x7fc3200c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31f7fe700 1 --2- 172.21.15.67:0/2763375877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc32015cb20 0x7fc3200c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31f7fe700 1 -- 172.21.15.67:0/2763375877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3100094a0 con 0x7fc320159350 2023-06-22T07:59:30.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31ffff700 1 --2- 172.21.15.67:0/2763375877 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc320152ce0 0x7fc320158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:30.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31f7fe700 1 --2- 172.21.15.67:0/2763375877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320159350 0x7fc320155f40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc31400c550 tx=0x7fc31400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:30.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31cff9700 1 -- 172.21.15.67:0/2763375877 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc314004c80 con 0x7fc320159350 2023-06-22T07:59:30.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31cff9700 1 -- 172.21.15.67:0/2763375877 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc314004de0 con 0x7fc320159350 2023-06-22T07:59:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc3268b3700 1 -- 172.21.15.67:0/2763375877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc320156970 con 0x7fc320159350 2023-06-22T07:59:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31cff9700 1 -- 172.21.15.67:0/2763375877 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3140072d0 con 0x7fc320159350 2023-06-22T07:59:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc3268b3700 1 -- 172.21.15.67:0/2763375877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc320156f30 con 0x7fc320159350 2023-06-22T07:59:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.893+0000 7fc31cff9700 1 -- 172.21.15.67:0/2763375877 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc31401d3f0 con 0x7fc320159350 2023-06-22T07:59:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.894+0000 7fc31cff9700 1 -- 172.21.15.67:0/2763375877 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fc314013940 con 0x7fc320159350 2023-06-22T07:59:30.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.894+0000 7fc3268b3700 1 -- 172.21.15.67:0/2763375877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc32004ea20 con 0x7fc320159350 2023-06-22T07:59:30.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.897+0000 7fc31cff9700 1 -- 172.21.15.67:0/2763375877 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fc31400ce60 con 0x7fc320159350 2023-06-22T07:59:30.985 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.982+0000 7fc3268b3700 1 -- 172.21.15.67:0/2763375877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a845ba15-d7a0-4542-8367-caf4a9ac4bdd", "id": 5} v 0) v1 -- 0x7fc3200666a0 con 0x7fc320159350 2023-06-22T07:59:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.993+0000 7fc31cff9700 1 -- 172.21.15.67:0/2763375877 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a845ba15-d7a0-4542-8367-caf4a9ac4bdd", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc314013ce0 con 0x7fc320159350 2023-06-22T07:59:30.996 INFO:teuthology.orchestra.run.smithi067.stdout:5 2023-06-22T07:59:30.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.994+0000 7fc3268b3700 1 -- 172.21.15.67:0/2763375877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320159350 msgr2=0x7fc320155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:30.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.994+0000 7fc3268b3700 1 --2- 172.21.15.67:0/2763375877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320159350 0x7fc320155f40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc31400c550 tx=0x7fc31400c860 comp rx=0 tx=0).stop 2023-06-22T07:59:30.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.994+0000 7fc3268b3700 1 -- 172.21.15.67:0/2763375877 shutdown_connections 2023-06-22T07:59:30.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.994+0000 7fc3268b3700 1 --2- 172.21.15.67:0/2763375877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc32015cb20 0x7fc3200c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.994+0000 7fc3268b3700 1 --2- 172.21.15.67:0/2763375877 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc320152ce0 0x7fc320158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.994+0000 7fc3268b3700 1 --2- 172.21.15.67:0/2763375877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320159350 0x7fc320155f40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:30.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.994+0000 7fc3268b3700 1 -- 172.21.15.67:0/2763375877 >> 172.21.15.67:0/2763375877 conn(0x7fc320065b90 msgr2=0x7fc3200c50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:30.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.994+0000 7fc3268b3700 1 -- 172.21.15.67:0/2763375877 shutdown_connections 2023-06-22T07:59:30.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:30.994+0000 7fc3268b3700 1 -- 172.21.15.67:0/2763375877 wait complete. 2023-06-22T07:59:30.998 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:30.994+0000 7f95b7507200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-22T07:59:31.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 688057f7-a7ba-44bc-9913-0adcd507e06d 6 2023-06-22T07:59:31.135 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.133+0000 7f90e2faa700 1 Processor -- start 2023-06-22T07:59:31.136 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.133+0000 7f90e2faa700 1 -- start start 2023-06-22T07:59:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.133+0000 7f90e2faa700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f90dc152ce0 0x7f90dc070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e2faa700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc159350 0x7f90dc0cba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e2faa700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90dc06e9b0 con 0x7f90dc15cb20 2023-06-22T07:59:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e2faa700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90dc15c690 con 0x7f90dc152ce0 2023-06-22T07:59:31.138 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e2faa700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc1597f0 con 0x7f90dc159350 2023-06-22T07:59:31.138 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e0d46700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f90dc152ce0 0x7f90dc070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.138 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e1547700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc159350 0x7f90dc0cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.138 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e0d46700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f90dc152ce0 0x7f90dc070e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55574/0 (socket says 172.21.15.67:55574) 2023-06-22T07:59:31.138 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e1547700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc159350 0x7f90dc0cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48386/0 (socket says 172.21.15.67:48386) 2023-06-22T07:59:31.138 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e0d46700 1 -- 172.21.15.67:0/4280421026 learned_addr learned my addr 172.21.15.67:0/4280421026 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:31.138 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90dbfff700 1 --1- 172.21.15.67:0/4280421026 >> v1:172.21.15.67:6789/0 conn(0x7f90dc15cb20 0x7f90dc0c5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39396/0 (socket says 172.21.15.67:39396) 2023-06-22T07:59:31.139 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e1547700 1 -- 172.21.15.67:0/4280421026 >> v2:172.21.15.67:3300/0 conn(0x7f90dc152ce0 msgr2=0x7f90dc070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:31.139 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e1547700 1 --2- 172.21.15.67:0/4280421026 >> v2:172.21.15.67:3300/0 conn(0x7f90dc152ce0 0x7f90dc070e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.139 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e1547700 1 -- 172.21.15.67:0/4280421026 >> v1:172.21.15.67:6789/0 conn(0x7f90dc15cb20 legacy=0x7f90dc0c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.139 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e1547700 1 -- 172.21.15.67:0/4280421026 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc06ec00 con 0x7f90dc159350 2023-06-22T07:59:31.139 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90e1547700 1 --2- 172.21.15.67:0/4280421026 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc159350 0x7f90dc0cba90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f90d4009d90 tx=0x7f90d400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=900c0dbdce869239 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:31.139 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90db7fe700 1 -- 172.21.15.67:0/4280421026 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d400e070 con 0x7f90dc159350 2023-06-22T07:59:31.139 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90db7fe700 1 -- 172.21.15.67:0/4280421026 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d400bb20 con 0x7f90dc159350 2023-06-22T07:59:31.139 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.134+0000 7f90db7fe700 1 -- 172.21.15.67:0/4280421026 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d4012430 con 0x7f90dc159350 2023-06-22T07:59:31.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.135+0000 7f90e2faa700 1 -- 172.21.15.67:0/4280421026 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc159350 msgr2=0x7f90dc0cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.135+0000 7f90e2faa700 1 --2- 172.21.15.67:0/4280421026 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc159350 0x7f90dc0cba90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f90d4009d90 tx=0x7f90d400b330 comp rx=0 tx=0).stop 2023-06-22T07:59:31.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.135+0000 7f90e2faa700 1 -- 172.21.15.67:0/4280421026 shutdown_connections 2023-06-22T07:59:31.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.135+0000 7f90e2faa700 1 --2- 172.21.15.67:0/4280421026 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc159350 0x7f90dc0cba90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.135+0000 7f90e2faa700 1 --2- 172.21.15.67:0/4280421026 >> v2:172.21.15.67:3300/0 conn(0x7f90dc152ce0 0x7f90dc070e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.135+0000 7f90e2faa700 1 -- 172.21.15.67:0/4280421026 >> 172.21.15.67:0/4280421026 conn(0x7f90dc065b90 msgr2=0x7f90dc065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:31.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.135+0000 7f90e2faa700 1 -- 172.21.15.67:0/4280421026 shutdown_connections 2023-06-22T07:59:31.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.135+0000 7f90e2faa700 1 -- 172.21.15.67:0/4280421026 wait complete. 2023-06-22T07:59:31.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.136+0000 7f90e2faa700 1 Processor -- start 2023-06-22T07:59:31.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.136+0000 7f90e2faa700 1 -- start start 2023-06-22T07:59:31.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90e2faa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc152ce0 0x7f90dc1e4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90e2faa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc159350 0x7f90dc0c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90e2faa700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc15cb20 0x7f90dc0c5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90e2faa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0c4690 con 0x7f90dc159350 2023-06-22T07:59:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90e2faa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc06bdd0 con 0x7f90dc152ce0 2023-06-22T07:59:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90e2faa700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc1e5ef0 con 0x7f90dc15cb20 2023-06-22T07:59:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90e0d46700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc152ce0 0x7f90dc1e4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90e0d46700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc152ce0 0x7f90dc1e4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55580/0 (socket says 172.21.15.67:55580) 2023-06-22T07:59:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90e1547700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc15cb20 0x7f90dc0c5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90e0d46700 1 -- 172.21.15.67:0/3036243027 learned_addr learned my addr 172.21.15.67:0/3036243027 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:31.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.137+0000 7f90dbfff700 1 --2- 172.21.15.67:0/3036243027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc159350 0x7f90dc0c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90e0d46700 1 -- 172.21.15.67:0/3036243027 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc15cb20 msgr2=0x7f90dc0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90e0d46700 1 --2- 172.21.15.67:0/3036243027 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc15cb20 0x7f90dc0c5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90e0d46700 1 -- 172.21.15.67:0/3036243027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc159350 msgr2=0x7f90dc0c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90e0d46700 1 --2- 172.21.15.67:0/3036243027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc159350 0x7f90dc0c4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90e0d46700 1 -- 172.21.15.67:0/3036243027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4009a70 con 0x7f90dc152ce0 2023-06-22T07:59:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90dbfff700 1 --2- 172.21.15.67:0/3036243027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc159350 0x7f90dc0c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T07:59:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90e0d46700 1 --2- 172.21.15.67:0/3036243027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc152ce0 0x7f90dc1e4240 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f90d000e8b0 tx=0x7f90d000ebc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90d97fa700 1 -- 172.21.15.67:0/3036243027 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d0009600 con 0x7f90dc152ce0 2023-06-22T07:59:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90d97fa700 1 -- 172.21.15.67:0/3036243027 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d0013680 con 0x7f90dc152ce0 2023-06-22T07:59:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90e2faa700 1 -- 172.21.15.67:0/3036243027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90dc06b530 con 0x7f90dc152ce0 2023-06-22T07:59:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90d97fa700 1 -- 172.21.15.67:0/3036243027 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d0013a30 con 0x7f90dc152ce0 2023-06-22T07:59:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90e2faa700 1 -- 172.21.15.67:0/3036243027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc06baf0 con 0x7f90dc152ce0 2023-06-22T07:59:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90d97fa700 1 -- 172.21.15.67:0/3036243027 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f90d001c450 con 0x7f90dc152ce0 2023-06-22T07:59:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.138+0000 7f90d97fa700 1 -- 172.21.15.67:0/3036243027 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f90d000cb60 con 0x7f90dc152ce0 2023-06-22T07:59:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.139+0000 7f90e2faa700 1 -- 172.21.15.67:0/3036243027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90dc0c44b0 con 0x7f90dc152ce0 2023-06-22T07:59:31.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.141+0000 7f90d97fa700 1 -- 172.21.15.67:0/3036243027 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f90d0021060 con 0x7f90dc152ce0 2023-06-22T07:59:31.174 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:31.172+0000 7f95b7507200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-22T07:59:31.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.240+0000 7f90e2faa700 1 -- 172.21.15.67:0/3036243027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd new", "uuid": "688057f7-a7ba-44bc-9913-0adcd507e06d", "id": 6} v 0) v1 -- 0x7f90dc04ea20 con 0x7f90dc152ce0 2023-06-22T07:59:31.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.260+0000 7f90d97fa700 1 -- 172.21.15.67:0/3036243027 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "688057f7-a7ba-44bc-9913-0adcd507e06d", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f90d002a0b0 con 0x7f90dc152ce0 2023-06-22T07:59:31.263 INFO:teuthology.orchestra.run.smithi067.stdout:6 2023-06-22T07:59:31.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.261+0000 7f90e2faa700 1 -- 172.21.15.67:0/3036243027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc152ce0 msgr2=0x7f90dc1e4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.261+0000 7f90e2faa700 1 --2- 172.21.15.67:0/3036243027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc152ce0 0x7f90dc1e4240 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f90d000e8b0 tx=0x7f90d000ebc0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.261+0000 7f90e2faa700 1 -- 172.21.15.67:0/3036243027 shutdown_connections 2023-06-22T07:59:31.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.261+0000 7f90e2faa700 1 --2- 172.21.15.67:0/3036243027 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc15cb20 0x7f90dc0c5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.261+0000 7f90e2faa700 1 --2- 172.21.15.67:0/3036243027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc152ce0 0x7f90dc1e4240 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.261+0000 7f90e2faa700 1 --2- 172.21.15.67:0/3036243027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc159350 0x7f90dc0c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.261+0000 7f90e2faa700 1 -- 172.21.15.67:0/3036243027 >> 172.21.15.67:0/3036243027 conn(0x7f90dc065b90 msgr2=0x7f90dc0cadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:31.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.261+0000 7f90e2faa700 1 -- 172.21.15.67:0/3036243027 shutdown_connections 2023-06-22T07:59:31.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.261+0000 7f90e2faa700 1 -- 172.21.15.67:0/3036243027 wait complete. 2023-06-22T07:59:31.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new ee654631-ffe8-4952-9160-21a40babb1f6 7 2023-06-22T07:59:31.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.397+0000 7fb5828a2700 1 Processor -- start 2023-06-22T07:59:31.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.397+0000 7fb5828a2700 1 -- start start 2023-06-22T07:59:31.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.397+0000 7fb5828a2700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c14f360 0x7fb57c0c7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.397+0000 7fb5828a2700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb57c070f30 0x7fb57c0713b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.397+0000 7fb5828a2700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c155bf0 con 0x7fb57c0705b0 2023-06-22T07:59:31.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb5828a2700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb57c0c0400 con 0x7fb57c070f30 2023-06-22T07:59:31.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb5828a2700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c071b80 con 0x7fb57c14f360 2023-06-22T07:59:31.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb57bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c14f360 0x7fb57c0c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb580e3f700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb57c070f30 0x7fb57c0713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb57bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c14f360 0x7fb57c0c7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48394/0 (socket says 172.21.15.67:48394) 2023-06-22T07:59:31.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb580e3f700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb57c070f30 0x7fb57c0713b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55588/0 (socket says 172.21.15.67:55588) 2023-06-22T07:59:31.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb57bfff700 1 -- 172.21.15.67:0/2006192105 learned_addr learned my addr 172.21.15.67:0/2006192105 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:31.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb580e3f700 1 -- 172.21.15.67:0/2006192105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c14f360 msgr2=0x7fb57c0c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:31.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb580e3f700 1 --2- 172.21.15.67:0/2006192105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c14f360 0x7fb57c0c7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb580e3f700 1 -- 172.21.15.67:0/2006192105 >> v1:172.21.15.23:6789/0 conn(0x7fb57c0705b0 legacy=0x7fb57c0c0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb580e3f700 1 -- 172.21.15.67:0/2006192105 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c14f730 con 0x7fb57c070f30 2023-06-22T07:59:31.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb580e3f700 1 --2- 172.21.15.67:0/2006192105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c070f30 0x7fb57c0713b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb57400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c3664125cefb99f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:31.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb57affd700 1 -- 172.21.15.67:0/2006192105 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57400e070 con 0x7fb57c070f30 2023-06-22T07:59:31.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb57affd700 1 -- 172.21.15.67:0/2006192105 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57400bb20 con 0x7fb57c070f30 2023-06-22T07:59:31.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.398+0000 7fb57affd700 1 -- 172.21.15.67:0/2006192105 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb574012430 con 0x7fb57c070f30 2023-06-22T07:59:31.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.399+0000 7fb5828a2700 1 -- 172.21.15.67:0/2006192105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c070f30 msgr2=0x7fb57c0713b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.399+0000 7fb5828a2700 1 --2- 172.21.15.67:0/2006192105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c070f30 0x7fb57c0713b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb57400b330 comp rx=0 tx=0).stop 2023-06-22T07:59:31.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.399+0000 7fb5828a2700 1 -- 172.21.15.67:0/2006192105 shutdown_connections 2023-06-22T07:59:31.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.399+0000 7fb5828a2700 1 --2- 172.21.15.67:0/2006192105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c070f30 0x7fb57c0713b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.399+0000 7fb5828a2700 1 --2- 172.21.15.67:0/2006192105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c14f360 0x7fb57c0c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.399+0000 7fb5828a2700 1 -- 172.21.15.67:0/2006192105 >> 172.21.15.67:0/2006192105 conn(0x7fb57c0c77b0 msgr2=0x7fb57c0c7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:31.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.399+0000 7fb5828a2700 1 -- 172.21.15.67:0/2006192105 shutdown_connections 2023-06-22T07:59:31.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.399+0000 7fb5828a2700 1 -- 172.21.15.67:0/2006192105 wait complete. 2023-06-22T07:59:31.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.400+0000 7fb5828a2700 1 Processor -- start 2023-06-22T07:59:31.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.400+0000 7fb5828a2700 1 -- start start 2023-06-22T07:59:31.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb5828a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0705b0 0x7fb57c150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb5828a2700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c070f30 0x7fb57c14b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb5828a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb57c14f360 0x7fb57c1512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb5828a2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c14bb20 con 0x7fb57c14f360 2023-06-22T07:59:31.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb5828a2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c14c480 con 0x7fb57c0705b0 2023-06-22T07:59:31.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb5828a2700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c1517e0 con 0x7fb57c070f30 2023-06-22T07:59:31.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb57b7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c070f30 0x7fb57c14b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb57bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0705b0 0x7fb57c150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb57b7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c070f30 0x7fb57c14b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:48398/0 (socket says 172.21.15.67:48398) 2023-06-22T07:59:31.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb57b7fe700 1 -- 172.21.15.67:0/2899156607 learned_addr learned my addr 172.21.15.67:0/2899156607 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-22T07:59:31.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.401+0000 7fb580e3f700 1 --2- 172.21.15.67:0/2899156607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb57c14f360 0x7fb57c1512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb57b7fe700 1 -- 172.21.15.67:0/2899156607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0705b0 msgr2=0x7fb57c150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb57b7fe700 1 --2- 172.21.15.67:0/2899156607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0705b0 0x7fb57c150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb57b7fe700 1 -- 172.21.15.67:0/2899156607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb57c14f360 msgr2=0x7fb57c1512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb57b7fe700 1 --2- 172.21.15.67:0/2899156607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb57c14f360 0x7fb57c1512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb57b7fe700 1 -- 172.21.15.67:0/2899156607 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb574009a70 con 0x7fb57c070f30 2023-06-22T07:59:31.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb580e3f700 1 --2- 172.21.15.67:0/2899156607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb57c14f360 0x7fb57c1512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T07:59:31.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb57b7fe700 1 --2- 172.21.15.67:0/2899156607 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c070f30 0x7fb57c14b5e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb57000c510 tx=0x7fb57000c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:31.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb578ff9700 1 -- 172.21.15.67:0/2899156607 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700052d0 con 0x7fb57c070f30 2023-06-22T07:59:31.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb578ff9700 1 -- 172.21.15.67:0/2899156607 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb570005430 con 0x7fb57c070f30 2023-06-22T07:59:31.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb578ff9700 1 -- 172.21.15.67:0/2899156607 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700038c0 con 0x7fb57c070f30 2023-06-22T07:59:31.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb5828a2700 1 -- 172.21.15.67:0/2899156607 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c0c95e0 con 0x7fb57c070f30 2023-06-22T07:59:31.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb5828a2700 1 -- 172.21.15.67:0/2899156607 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c14c1a0 con 0x7fb57c070f30 2023-06-22T07:59:31.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb578ff9700 1 -- 172.21.15.67:0/2899156607 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb570003d20 con 0x7fb57c070f30 2023-06-22T07:59:31.411 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.402+0000 7fb578ff9700 1 -- 172.21.15.67:0/2899156607 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fb570011e00 con 0x7fb57c070f30 2023-06-22T07:59:31.411 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.403+0000 7fb5828a2700 1 -- 172.21.15.67:0/2899156607 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57c04ea20 con 0x7fb57c070f30 2023-06-22T07:59:31.411 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.405+0000 7fb578ff9700 1 -- 172.21.15.67:0/2899156607 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fb570017070 con 0x7fb57c070f30 2023-06-22T07:59:31.428 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:31.426+0000 7f84a80f0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-22T07:59:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.490+0000 7fb5828a2700 1 -- 172.21.15.67:0/2899156607 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ee654631-ffe8-4952-9160-21a40babb1f6", "id": 7} v 0) v1 -- 0x7fb57c0c60f0 con 0x7fb57c070f30 2023-06-22T07:59:31.500 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.498+0000 7fb578ff9700 1 -- 172.21.15.67:0/2899156607 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ee654631-ffe8-4952-9160-21a40babb1f6", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb570012ad0 con 0x7fb57c070f30 2023-06-22T07:59:31.500 INFO:teuthology.orchestra.run.smithi067.stdout:7 2023-06-22T07:59:31.501 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.499+0000 7fb5828a2700 1 -- 172.21.15.67:0/2899156607 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c070f30 msgr2=0x7fb57c14b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.501 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.499+0000 7fb5828a2700 1 --2- 172.21.15.67:0/2899156607 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c070f30 0x7fb57c14b5e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb57000c510 tx=0x7fb57000c820 comp rx=0 tx=0).stop 2023-06-22T07:59:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.499+0000 7fb5828a2700 1 -- 172.21.15.67:0/2899156607 shutdown_connections 2023-06-22T07:59:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.499+0000 7fb5828a2700 1 --2- 172.21.15.67:0/2899156607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb57c14f360 0x7fb57c1512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.499+0000 7fb5828a2700 1 --2- 172.21.15.67:0/2899156607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb57c0705b0 0x7fb57c150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.499+0000 7fb5828a2700 1 --2- 172.21.15.67:0/2899156607 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb57c070f30 0x7fb57c14b5e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.499+0000 7fb5828a2700 1 -- 172.21.15.67:0/2899156607 >> 172.21.15.67:0/2899156607 conn(0x7fb57c0c77b0 msgr2=0x7fb57c14e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.499+0000 7fb5828a2700 1 -- 172.21.15.67:0/2899156607 shutdown_connections 2023-06-22T07:59:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-22T07:59:31.499+0000 7fb5828a2700 1 -- 172.21.15.67:0/2899156607 wait complete. 2023-06-22T07:59:31.513 INFO:tasks.ceph.osd.0:Restarting daemon 2023-06-22T07:59:31.513 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-06-22T07:59:31.518 INFO:tasks.ceph.osd.0:Started 2023-06-22T07:59:31.518 INFO:tasks.ceph.osd.1:Restarting daemon 2023-06-22T07:59:31.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-06-22T07:59:31.522 INFO:tasks.ceph.osd.1:Started 2023-06-22T07:59:31.523 INFO:tasks.ceph.osd.2:Restarting daemon 2023-06-22T07:59:31.523 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-06-22T07:59:31.529 INFO:tasks.ceph.osd.2:Started 2023-06-22T07:59:31.530 INFO:tasks.ceph.osd.3:Restarting daemon 2023-06-22T07:59:31.530 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-06-22T07:59:31.535 INFO:tasks.ceph.osd.3:Started 2023-06-22T07:59:31.535 INFO:tasks.ceph.osd.4:Restarting daemon 2023-06-22T07:59:31.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-06-22T07:59:31.537 INFO:tasks.ceph.osd.4:Started 2023-06-22T07:59:31.538 INFO:tasks.ceph.osd.5:Restarting daemon 2023-06-22T07:59:31.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-06-22T07:59:31.540 INFO:tasks.ceph.osd.5:Started 2023-06-22T07:59:31.540 INFO:tasks.ceph.osd.6:Restarting daemon 2023-06-22T07:59:31.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-06-22T07:59:31.542 INFO:tasks.ceph.osd.6:Started 2023-06-22T07:59:31.542 INFO:tasks.ceph.osd.7:Restarting daemon 2023-06-22T07:59:31.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-06-22T07:59:31.545 INFO:tasks.ceph.osd.7:Started 2023-06-22T07:59:31.546 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T07:59:31.670 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:31.667+0000 7f84a80f0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-22T07:59:31.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.818+0000 7fc047bff700 1 Processor -- start 2023-06-22T07:59:31.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.819+0000 7fc047bff700 1 -- start start 2023-06-22T07:59:31.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc047bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0400cb6b0 0x7fc040069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc047bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc0400c5740 0x7fc0401593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc047bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cba80 0x7fc040159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc047bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04006bd20 con 0x7fc0400c5740 2023-06-22T07:59:31.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc047bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc040158f60 con 0x7fc0400cb6b0 2023-06-22T07:59:31.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc047bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0400cc450 con 0x7fc0400cba80 2023-06-22T07:59:31.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc04519a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc0400c5740 0x7fc0401593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc04599b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0400cb6b0 0x7fc040069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc04619c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cba80 0x7fc040159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc04599b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0400cb6b0 0x7fc040069eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:37904/0 (socket says 172.21.15.23:37904) 2023-06-22T07:59:31.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc04619c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cba80 0x7fc040159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:34670/0 (socket says 172.21.15.23:34670) 2023-06-22T07:59:31.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.821+0000 7fc04599b700 1 -- 172.21.15.23:0/1963521667 learned_addr learned my addr 172.21.15.23:0/1963521667 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:31.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.822+0000 7fc04619c700 1 -- 172.21.15.23:0/1963521667 >> v2:172.21.15.67:3300/0 conn(0x7fc0400cb6b0 msgr2=0x7fc040069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:31.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.822+0000 7fc04619c700 1 --2- 172.21.15.23:0/1963521667 >> v2:172.21.15.67:3300/0 conn(0x7fc0400cb6b0 0x7fc040069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.822+0000 7fc04619c700 1 -- 172.21.15.23:0/1963521667 >> v2:172.21.15.23:3300/0 conn(0x7fc0400c5740 msgr2=0x7fc0401593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:31.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.822+0000 7fc04619c700 1 --2- 172.21.15.23:0/1963521667 >> v2:172.21.15.23:3300/0 conn(0x7fc0400c5740 0x7fc0401593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.822+0000 7fc04619c700 1 -- 172.21.15.23:0/1963521667 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc040158980 con 0x7fc0400cba80 2023-06-22T07:59:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.822+0000 7fc04599b700 1 --2- 172.21.15.23:0/1963521667 >> v2:172.21.15.67:3300/0 conn(0x7fc0400cb6b0 0x7fc040069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.822+0000 7fc04619c700 1 --2- 172.21.15.23:0/1963521667 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cba80 0x7fc040159ce0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0400cc230 tx=0x7fc03800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59ebbf962130c7d7 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.822+0000 7fc044999700 1 -- 172.21.15.23:0/1963521667 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03800e070 con 0x7fc0400cba80 2023-06-22T07:59:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.822+0000 7fc044999700 1 -- 172.21.15.23:0/1963521667 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03800bb20 con 0x7fc0400cba80 2023-06-22T07:59:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.822+0000 7fc044999700 1 -- 172.21.15.23:0/1963521667 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038012430 con 0x7fc0400cba80 2023-06-22T07:59:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.823+0000 7fc047bff700 1 -- 172.21.15.23:0/1963521667 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cba80 msgr2=0x7fc040159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.823+0000 7fc047bff700 1 --2- 172.21.15.23:0/1963521667 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cba80 0x7fc040159ce0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0400cc230 tx=0x7fc03800b330 comp rx=0 tx=0).stop 2023-06-22T07:59:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.823+0000 7fc047bff700 1 -- 172.21.15.23:0/1963521667 shutdown_connections 2023-06-22T07:59:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.823+0000 7fc047bff700 1 --2- 172.21.15.23:0/1963521667 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cba80 0x7fc040159ce0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.823+0000 7fc047bff700 1 --2- 172.21.15.23:0/1963521667 >> v2:172.21.15.67:3300/0 conn(0x7fc0400cb6b0 0x7fc040069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.823+0000 7fc047bff700 1 --2- 172.21.15.23:0/1963521667 >> v2:172.21.15.23:3300/0 conn(0x7fc0400c5740 0x7fc0401593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.823+0000 7fc047bff700 1 -- 172.21.15.23:0/1963521667 >> 172.21.15.23:0/1963521667 conn(0x7fc040070d80 msgr2=0x7fc040071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.823+0000 7fc047bff700 1 -- 172.21.15.23:0/1963521667 shutdown_connections 2023-06-22T07:59:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.823+0000 7fc047bff700 1 -- 172.21.15.23:0/1963521667 wait complete. 2023-06-22T07:59:31.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.826+0000 7fc047bff700 1 Processor -- start 2023-06-22T07:59:31.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.826+0000 7fc047bff700 1 -- start start 2023-06-22T07:59:31.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc047bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cb6b0 0x7fc04014c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc047bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0400cba80 0x7fc040157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc047bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc04014fa50 0x7fc0401489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:31.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc047bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0401518c0 con 0x7fc04014fa50 2023-06-22T07:59:31.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc047bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc040151b10 con 0x7fc0400cba80 2023-06-22T07:59:31.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc047bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc040151df0 con 0x7fc0400cb6b0 2023-06-22T07:59:31.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc04599b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cb6b0 0x7fc04014c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc04519a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0400cba80 0x7fc040157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc04599b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cb6b0 0x7fc04014c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:34680/0 (socket says 172.21.15.23:34680) 2023-06-22T07:59:31.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc04599b700 1 -- 172.21.15.23:0/3261609340 learned_addr learned my addr 172.21.15.23:0/3261609340 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:31.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.828+0000 7fc04619c700 1 --2- 172.21.15.23:0/3261609340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc04014fa50 0x7fc0401489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:31.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc04519a700 1 -- 172.21.15.23:0/3261609340 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cb6b0 msgr2=0x7fc04014c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc04519a700 1 --2- 172.21.15.23:0/3261609340 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cb6b0 0x7fc04014c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc04519a700 1 -- 172.21.15.23:0/3261609340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc04014fa50 msgr2=0x7fc0401489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc04519a700 1 --2- 172.21.15.23:0/3261609340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc04014fa50 0x7fc0401489b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc04519a700 1 -- 172.21.15.23:0/3261609340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c009550 con 0x7fc0400cba80 2023-06-22T07:59:31.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc04599b700 1 --2- 172.21.15.23:0/3261609340 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cb6b0 0x7fc04014c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:31.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc04519a700 1 --2- 172.21.15.23:0/3261609340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0400cba80 0x7fc040157e40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc0400c9380 tx=0x7fc03000ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc0367fc700 1 -- 172.21.15.23:0/3261609340 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc030013680 con 0x7fc0400cba80 2023-06-22T07:59:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc0367fc700 1 -- 172.21.15.23:0/3261609340 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0300137e0 con 0x7fc0400cba80 2023-06-22T07:59:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc0367fc700 1 -- 172.21.15.23:0/3261609340 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc030013b50 con 0x7fc0400cba80 2023-06-22T07:59:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc047bff700 1 -- 172.21.15.23:0/3261609340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc038009a70 con 0x7fc0400cba80 2023-06-22T07:59:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.829+0000 7fc047bff700 1 -- 172.21.15.23:0/3261609340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0401f1aa0 con 0x7fc0400cba80 2023-06-22T07:59:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.830+0000 7fc0367fc700 1 -- 172.21.15.23:0/3261609340 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc03001c450 con 0x7fc0400cba80 2023-06-22T07:59:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.830+0000 7fc0367fc700 1 -- 172.21.15.23:0/3261609340 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc03001cdd0 con 0x7fc0400cba80 2023-06-22T07:59:31.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.830+0000 7fc047bff700 1 -- 172.21.15.23:0/3261609340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04006ffc0 con 0x7fc0400cba80 2023-06-22T07:59:31.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.833+0000 7fc0367fc700 1 -- 172.21.15.23:0/3261609340 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fc030021060 con 0x7fc0400cba80 2023-06-22T07:59:31.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.908+0000 7fc047bff700 1 -- 172.21.15.23:0/3261609340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc040070300 con 0x7fc0400cba80 2023-06-22T07:59:31.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.909+0000 7fc0367fc700 1 -- 172.21.15.23:0/3261609340 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fc030017070 con 0x7fc0400cba80 2023-06-22T07:59:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:59:31.912 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":10,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:31.491476+0000","last_up_change":"0.000000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T07:59:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.909+0000 7fc047bff700 1 -- 172.21.15.23:0/3261609340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0400cba80 msgr2=0x7fc040157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:31.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.910+0000 7fc047bff700 1 --2- 172.21.15.23:0/3261609340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0400cba80 0x7fc040157e40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc0400c9380 tx=0x7fc03000ebb0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.910+0000 7fc047bff700 1 -- 172.21.15.23:0/3261609340 shutdown_connections 2023-06-22T07:59:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.910+0000 7fc047bff700 1 --2- 172.21.15.23:0/3261609340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc04014fa50 0x7fc0401489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.910+0000 7fc047bff700 1 --2- 172.21.15.23:0/3261609340 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0400cb6b0 0x7fc04014c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.910+0000 7fc047bff700 1 --2- 172.21.15.23:0/3261609340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0400cba80 0x7fc040157e40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.910+0000 7fc047bff700 1 -- 172.21.15.23:0/3261609340 >> 172.21.15.23:0/3261609340 conn(0x7fc040070d80 msgr2=0x7fc0400c9980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.910+0000 7fc047bff700 1 -- 172.21.15.23:0/3261609340 shutdown_connections 2023-06-22T07:59:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:31.910+0000 7fc047bff700 1 -- 172.21.15.23:0/3261609340 wait complete. 2023-06-22T07:59:31.930 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-06-22T07:59:31.931 INFO:tasks.ceph:Waiting for OSDs to come up 2023-06-22T07:59:31.946 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:31.943+0000 7f95b7507200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-22T07:59:32.026 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:32.024+0000 7f95b7507200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-22T07:59:32.163 INFO:tasks.ceph.osd.7.smithi067.stderr:2023-06-22T07:59:32.160+0000 7f591b4d7640 -1 Falling back to public interface 2023-06-22T07:59:32.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2023-06-22T07:59:32.161+0000 7ffbeaad1640 -1 Falling back to public interface 2023-06-22T07:59:32.164 INFO:tasks.ceph.osd.5.smithi067.stderr:2023-06-22T07:59:32.161+0000 7f2e397c3640 -1 Falling back to public interface 2023-06-22T07:59:32.164 INFO:tasks.ceph.osd.4.smithi067.stderr:2023-06-22T07:59:32.162+0000 7ffbe1e8f700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2023-06-22T07:59:32.173 INFO:tasks.ceph.osd.6.smithi067.stderr:2023-06-22T07:59:32.170+0000 7f4a2ed38640 -1 Falling back to public interface 2023-06-22T07:59:32.174 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:32.172+0000 7f95b7507200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-22T07:59:32.231 INFO:tasks.ceph.osd.0.smithi023.stderr:2023-06-22T07:59:32.228+0000 7f7d070fb640 -1 Falling back to public interface 2023-06-22T07:59:32.247 INFO:tasks.ceph.osd.2.smithi023.stderr:2023-06-22T07:59:32.245+0000 7fd0f53b5640 -1 Falling back to public interface 2023-06-22T07:59:32.258 INFO:tasks.ceph.osd.1.smithi023.stderr:2023-06-22T07:59:32.255+0000 7f3e3da78640 -1 Falling back to public interface 2023-06-22T07:59:32.273 INFO:tasks.ceph.osd.3.smithi023.stderr:2023-06-22T07:59:32.270+0000 7f3f33b4d640 -1 Falling back to public interface 2023-06-22T07:59:32.417 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:32.414+0000 7f95b7507200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-22T07:59:32.495 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:32.493+0000 7f95b7507200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-22T07:59:32.563 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:32.560+0000 7f84a80f0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-22T07:59:32.664 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:32.661+0000 7f84a80f0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-22T07:59:32.735 DEBUG:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-22T07:59:32.779 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:32.777+0000 7f95b7507200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-22T07:59:32.848 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:32.846+0000 7f84a80f0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-22T07:59:32.911 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.908+0000 7f9b42bff700 1 Processor -- start 2023-06-22T07:59:32.911 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.908+0000 7f9b42bff700 1 -- start start 2023-06-22T07:59:32.913 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.909+0000 7f9b42bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c1528f0 0x7f9b3c0cb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:32.913 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.909+0000 7f9b42bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9b3c15c9b0 0x7f9b3c0cb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:32.913 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.909+0000 7f9b42bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c0ca190 con 0x7f9b3c152cc0 2023-06-22T07:59:32.913 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b42bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b3c0ca3c0 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.913 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b42bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0ca6a0 con 0x7f9b3c1528f0 2023-06-22T07:59:32.913 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b4099b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c1528f0 0x7f9b3c0cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:32.913 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b4119c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9b3c15c9b0 0x7f9b3c0cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:32.914 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b4099b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c1528f0 0x7f9b3c0cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:34696/0 (socket says 172.21.15.23:34696) 2023-06-22T07:59:32.914 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b4099b700 1 -- 172.21.15.23:0/662530250 learned_addr learned my addr 172.21.15.23:0/662530250 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:32.914 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b4119c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9b3c15c9b0 0x7f9b3c0cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60882/0 (socket says 172.21.15.23:60882) 2023-06-22T07:59:32.915 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b4099b700 1 -- 172.21.15.23:0/662530250 >> v2:172.21.15.23:3300/0 conn(0x7f9b3c15c9b0 msgr2=0x7f9b3c0cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:32.915 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b4099b700 1 --2- 172.21.15.23:0/662530250 >> v2:172.21.15.23:3300/0 conn(0x7f9b3c15c9b0 0x7f9b3c0cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:32.915 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b4099b700 1 -- 172.21.15.23:0/662530250 >> v1:172.21.15.23:6789/0 conn(0x7f9b3c152cc0 legacy=0x7f9b3c15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:32.916 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b4099b700 1 -- 172.21.15.23:0/662530250 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c0ca980 con 0x7f9b3c1528f0 2023-06-22T07:59:32.916 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.910+0000 7f9b4099b700 1 --2- 172.21.15.23:0/662530250 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c1528f0 0x7f9b3c0cb200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9b300098f0 tx=0x7f9b30009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbe0132478950a8c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:32.916 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.911+0000 7f9b3b7fe700 1 -- 172.21.15.23:0/662530250 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30011830 con 0x7f9b3c1528f0 2023-06-22T07:59:32.916 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.911+0000 7f9b3b7fe700 1 -- 172.21.15.23:0/662530250 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b30011990 con 0x7f9b3c1528f0 2023-06-22T07:59:32.916 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.911+0000 7f9b3b7fe700 1 -- 172.21.15.23:0/662530250 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b30011d20 con 0x7f9b3c1528f0 2023-06-22T07:59:32.916 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.911+0000 7f9b42bff700 1 -- 172.21.15.23:0/662530250 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c1528f0 msgr2=0x7f9b3c0cb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:32.916 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.911+0000 7f9b42bff700 1 --2- 172.21.15.23:0/662530250 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c1528f0 0x7f9b3c0cb200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9b300098f0 tx=0x7f9b30009c00 comp rx=0 tx=0).stop 2023-06-22T07:59:32.917 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.911+0000 7f9b42bff700 1 -- 172.21.15.23:0/662530250 shutdown_connections 2023-06-22T07:59:32.917 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.911+0000 7f9b42bff700 1 --2- 172.21.15.23:0/662530250 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c1528f0 0x7f9b3c0cb200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:32.917 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.911+0000 7f9b42bff700 1 --2- 172.21.15.23:0/662530250 >> v2:172.21.15.23:3300/0 conn(0x7f9b3c15c9b0 0x7f9b3c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:32.917 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.911+0000 7f9b42bff700 1 -- 172.21.15.23:0/662530250 >> 172.21.15.23:0/662530250 conn(0x7f9b3c065c20 msgr2=0x7f9b3c066020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:32.917 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.911+0000 7f9b42bff700 1 -- 172.21.15.23:0/662530250 shutdown_connections 2023-06-22T07:59:32.917 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.912+0000 7f9b42bff700 1 -- 172.21.15.23:0/662530250 wait complete. 2023-06-22T07:59:32.919 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.913+0000 7f9b42bff700 1 Processor -- start 2023-06-22T07:59:32.919 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.913+0000 7f9b42bff700 1 -- start start 2023-06-22T07:59:32.919 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.914+0000 7f9b42bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b3c1528f0 0x7f9b3c1ef390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:32.919 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.914+0000 7f9b42bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b3c152cc0 0x7f9b3c0b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:32.920 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.914+0000 7f9b42bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c15c9b0 0x7f9b3c157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:32.920 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.914+0000 7f9b42bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0b3130 con 0x7f9b3c152cc0 2023-06-22T07:59:32.920 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.914+0000 7f9b42bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c1579a0 con 0x7f9b3c1528f0 2023-06-22T07:59:32.920 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.914+0000 7f9b42bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c1580e0 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.920 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.914+0000 7f9b4119c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c15c9b0 0x7f9b3c157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:32.920 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.914+0000 7f9b3bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b3c152cc0 0x7f9b3c0b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:32.920 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.914+0000 7f9b4119c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c15c9b0 0x7f9b3c157460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:34702/0 (socket says 172.21.15.23:34702) 2023-06-22T07:59:32.921 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b4119c700 1 -- 172.21.15.23:0/3298003401 learned_addr learned my addr 172.21.15.23:0/3298003401 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:32.921 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b4119c700 1 -- 172.21.15.23:0/3298003401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b3c1528f0 msgr2=0x7f9b3c1ef390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T07:59:32.921 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b4119c700 1 --2- 172.21.15.23:0/3298003401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b3c1528f0 0x7f9b3c1ef390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:32.921 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b4119c700 1 -- 172.21.15.23:0/3298003401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b3c152cc0 msgr2=0x7f9b3c0b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:32.921 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b4119c700 1 --2- 172.21.15.23:0/3298003401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b3c152cc0 0x7f9b3c0b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:32.921 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b4119c700 1 -- 172.21.15.23:0/3298003401 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b30009550 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.921 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b3bfff700 1 --2- 172.21.15.23:0/3298003401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b3c152cc0 0x7f9b3c0b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:32.922 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b4119c700 1 --2- 172.21.15.23:0/3298003401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c15c9b0 0x7f9b3c157460 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400efc0 tx=0x7f9b34009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:32.922 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b397fa700 1 -- 172.21.15.23:0/3298003401 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34009b20 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.922 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b397fa700 1 -- 172.21.15.23:0/3298003401 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3400f040 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.922 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b397fa700 1 -- 172.21.15.23:0/3298003401 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3400cd20 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.922 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.915+0000 7f9b42bff700 1 -- 172.21.15.23:0/3298003401 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3c1583c0 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.922 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.916+0000 7f9b42bff700 1 -- 172.21.15.23:0/3298003401 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3c0b3620 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.922 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.916+0000 7f9b397fa700 1 -- 172.21.15.23:0/3298003401 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9b34014780 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.922 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.916+0000 7f9b397fa700 1 -- 172.21.15.23:0/3298003401 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9b3400ce80 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.923 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.917+0000 7f9b42bff700 1 -- 172.21.15.23:0/3298003401 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3c0b3840 con 0x7f9b3c15c9b0 2023-06-22T07:59:32.923 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:32.919+0000 7f9b397fa700 1 -- 172.21.15.23:0/3298003401 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f9b3403bb00 con 0x7f9b3c15c9b0 2023-06-22T07:59:33.005 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:33.003+0000 7f95b7507200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-22T07:59:33.017 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.014+0000 7f9b42bff700 1 -- 172.21.15.23:0/3298003401 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b3c1585b0 con 0x7f9b3c15c9b0 2023-06-22T07:59:33.017 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.014+0000 7f9b397fa700 1 -- 172.21.15.23:0/3298003401 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9b3402a070 con 0x7f9b3c15c9b0 2023-06-22T07:59:33.018 INFO:teuthology.misc.health.smithi023.stdout: 2023-06-22T07:59:33.018 INFO:teuthology.misc.health.smithi023.stdout:{"epoch":10,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:31.491476+0000","last_up_change":"0.000000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T07:59:33.019 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.015+0000 7f9b42bff700 1 -- 172.21.15.23:0/3298003401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c15c9b0 msgr2=0x7f9b3c157460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:33.019 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.015+0000 7f9b42bff700 1 --2- 172.21.15.23:0/3298003401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c15c9b0 0x7f9b3c157460 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400efc0 tx=0x7f9b34009f60 comp rx=0 tx=0).stop 2023-06-22T07:59:33.019 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.016+0000 7f9b42bff700 1 -- 172.21.15.23:0/3298003401 shutdown_connections 2023-06-22T07:59:33.020 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.016+0000 7f9b42bff700 1 --2- 172.21.15.23:0/3298003401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9b3c15c9b0 0x7f9b3c157460 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:33.020 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.016+0000 7f9b42bff700 1 --2- 172.21.15.23:0/3298003401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9b3c1528f0 0x7f9b3c1ef390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:33.020 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.016+0000 7f9b42bff700 1 --2- 172.21.15.23:0/3298003401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b3c152cc0 0x7f9b3c0b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:33.020 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.016+0000 7f9b42bff700 1 -- 172.21.15.23:0/3298003401 >> 172.21.15.23:0/3298003401 conn(0x7f9b3c065c20 msgr2=0x7f9b3c14d860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:33.020 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.016+0000 7f9b42bff700 1 -- 172.21.15.23:0/3298003401 shutdown_connections 2023-06-22T07:59:33.020 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:33.016+0000 7f9b42bff700 1 -- 172.21.15.23:0/3298003401 wait complete. 2023-06-22T07:59:33.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-22T07:59:33.166 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:33.164+0000 7f95b7507200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-22T07:59:33.172 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:33.169+0000 7f84a80f0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-22T07:59:33.271 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:33.268+0000 7f84a80f0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-22T07:59:33.287 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:33.284+0000 7f95b7507200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-22T07:59:33.441 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:33.438+0000 7f95b7507200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-22T07:59:33.604 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:33.601+0000 7f84a80f0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-22T07:59:33.727 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:33.724+0000 7f95b7507200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-22T07:59:33.888 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:33.885+0000 7f84a80f0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-22T07:59:33.917 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:33.915+0000 7f95b7507200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-22T07:59:34.092 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:34.090+0000 7f84a80f0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-22T07:59:34.243 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:34.241+0000 7f84a80f0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-22T07:59:34.436 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:34.433+0000 7f84a80f0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-22T07:59:34.530 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:34.528+0000 7f95b7507200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-22T07:59:34.796 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:34.793+0000 7f84a80f0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-22T07:59:35.055 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:35.052+0000 7f84a80f0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-22T07:59:35.557 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:35.555+0000 7f956f733700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-06-22T07:59:35.558 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T07:59:35.555+0000 7f956f733700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-06-22T07:59:35.640 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T07:59:35.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T07:59:35.648 INFO:tasks.ceph.osd.5.smithi067.stderr:2023-06-22T07:59:35.645+0000 7f2e397c3640 -1 osd.5 0 log_to_monitors true 2023-06-22T07:59:35.666 INFO:tasks.ceph.osd.7.smithi067.stderr:2023-06-22T07:59:35.663+0000 7f591b4d7640 -1 osd.7 0 log_to_monitors true 2023-06-22T07:59:35.681 INFO:tasks.ceph.osd.4.smithi067.stderr:2023-06-22T07:59:35.678+0000 7ffbeaad1640 -1 osd.4 0 log_to_monitors true 2023-06-22T07:59:35.683 INFO:tasks.ceph.osd.6.smithi067.stderr:2023-06-22T07:59:35.681+0000 7f4a2ed38640 -1 osd.6 0 log_to_monitors true 2023-06-22T07:59:35.835 INFO:tasks.ceph.osd.0.smithi023.stderr:2023-06-22T07:59:35.832+0000 7f7d070fb640 -1 osd.0 0 log_to_monitors true 2023-06-22T07:59:35.856 INFO:tasks.ceph.osd.1.smithi023.stderr:2023-06-22T07:59:35.854+0000 7f3e3da78640 -1 osd.1 0 log_to_monitors true 2023-06-22T07:59:35.862 INFO:tasks.ceph.osd.3.smithi023.stderr:2023-06-22T07:59:35.859+0000 7f3f33b4d640 -1 osd.3 0 log_to_monitors true 2023-06-22T07:59:35.965 INFO:tasks.ceph.mgr.y.smithi023.stderr:2023-06-22T07:59:35.963+0000 7f84a80f0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-22T07:59:36.081 INFO:tasks.ceph.osd.2.smithi023.stderr:2023-06-22T07:59:36.078+0000 7fd0f53b5640 -1 osd.2 0 log_to_monitors true 2023-06-22T07:59:39.167 INFO:tasks.ceph.osd.1.smithi023.stderr:2023-06-22T07:59:39.164+0000 7f3e2c9f7700 -1 osd.1 0 waiting for initial osdmap 2023-06-22T07:59:39.243 INFO:tasks.ceph.osd.1.smithi023.stderr:2023-06-22T07:59:39.240+0000 7f3e241fa700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T07:59:39.517 INFO:tasks.ceph.osd.6.smithi067.stderr:2023-06-22T07:59:39.513+0000 7f4a230f0700 -1 osd.6 0 waiting for initial osdmap 2023-06-22T07:59:39.534 INFO:tasks.ceph.osd.4.smithi067.stderr:2023-06-22T07:59:39.530+0000 7ffbdee89700 -1 osd.4 0 waiting for initial osdmap 2023-06-22T07:59:39.541 INFO:tasks.ceph.osd.6.smithi067.stderr:2023-06-22T07:59:39.537+0000 7f4a1acca700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T07:59:39.551 INFO:tasks.ceph.osd.4.smithi067.stderr:2023-06-22T07:59:39.548+0000 7ffbd6a63700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T07:59:39.688 INFO:tasks.ceph.osd.7.smithi067.stderr:2023-06-22T07:59:39.684+0000 7f590f88f700 -1 osd.7 0 waiting for initial osdmap 2023-06-22T07:59:39.701 INFO:tasks.ceph.osd.7.smithi067.stderr:2023-06-22T07:59:39.698+0000 7f5907469700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T07:59:39.719 INFO:tasks.ceph.osd.5.smithi067.stderr:2023-06-22T07:59:39.716+0000 7f2e2db7b700 -1 osd.5 0 waiting for initial osdmap 2023-06-22T07:59:39.729 INFO:tasks.ceph.osd.5.smithi067.stderr:2023-06-22T07:59:39.726+0000 7f2e25755700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T07:59:39.838 DEBUG:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-22T07:59:40.084 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.082+0000 7fe84bdff700 1 Processor -- start 2023-06-22T07:59:40.085 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.082+0000 7fe84bdff700 1 -- start start 2023-06-22T07:59:40.086 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.083+0000 7fe84bdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8440cb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:40.087 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe84bdff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe844152cc0 0x7fe8440cb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:40.087 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe84bdff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe8440c9170 0x7fe84415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:40.087 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe84bdff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8440ca190 con 0x7fe8440c9170 2023-06-22T07:59:40.087 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe84bdff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8440ca3c0 con 0x7fe844152cc0 2023-06-22T07:59:40.088 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe84bdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8440ca6a0 con 0x7fe8441528f0 2023-06-22T07:59:40.088 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe849b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8440cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:40.088 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe84939a700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe844152cc0 0x7fe8440cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:40.088 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe849b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8440cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:52270/0 (socket says 172.21.15.23:52270) 2023-06-22T07:59:40.088 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe849b9b700 1 -- 172.21.15.23:0/2336895449 learned_addr learned my addr 172.21.15.23:0/2336895449 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:40.089 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe84a39c700 1 --2- 172.21.15.23:0/2336895449 >> v2:172.21.15.23:3300/0 conn(0x7fe8440c9170 0x7fe84415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:40.090 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe849b9b700 1 -- 172.21.15.23:0/2336895449 >> v2:172.21.15.67:3300/0 conn(0x7fe844152cc0 msgr2=0x7fe8440cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:40.090 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe849b9b700 1 --2- 172.21.15.23:0/2336895449 >> v2:172.21.15.67:3300/0 conn(0x7fe844152cc0 0x7fe8440cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.090 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe849b9b700 1 -- 172.21.15.23:0/2336895449 >> v2:172.21.15.23:3300/0 conn(0x7fe8440c9170 msgr2=0x7fe84415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:40.090 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.084+0000 7fe849b9b700 1 --2- 172.21.15.23:0/2336895449 >> v2:172.21.15.23:3300/0 conn(0x7fe8440c9170 0x7fe84415c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.090 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.085+0000 7fe849b9b700 1 -- 172.21.15.23:0/2336895449 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8440ca980 con 0x7fe8441528f0 2023-06-22T07:59:40.090 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.085+0000 7fe84939a700 1 --2- 172.21.15.23:0/2336895449 >> v2:172.21.15.67:3300/0 conn(0x7fe844152cc0 0x7fe8440cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T07:59:40.090 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.085+0000 7fe849b9b700 1 --2- 172.21.15.23:0/2336895449 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8440cb200 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe8440674d0 tx=0x7fe838009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ee773617ef3f8b0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:40.091 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.085+0000 7fe848b99700 1 -- 172.21.15.23:0/2336895449 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe838014e90 con 0x7fe8441528f0 2023-06-22T07:59:40.091 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.085+0000 7fe848b99700 1 -- 172.21.15.23:0/2336895449 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83801c440 con 0x7fe8441528f0 2023-06-22T07:59:40.091 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.085+0000 7fe848b99700 1 -- 172.21.15.23:0/2336895449 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83801c7d0 con 0x7fe8441528f0 2023-06-22T07:59:40.091 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.086+0000 7fe84bdff700 1 -- 172.21.15.23:0/2336895449 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 msgr2=0x7fe8440cb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:40.091 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.086+0000 7fe84bdff700 1 --2- 172.21.15.23:0/2336895449 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8440cb200 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe8440674d0 tx=0x7fe838009bd0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.091 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.086+0000 7fe84bdff700 1 -- 172.21.15.23:0/2336895449 shutdown_connections 2023-06-22T07:59:40.092 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.086+0000 7fe84bdff700 1 --2- 172.21.15.23:0/2336895449 >> v2:172.21.15.23:3300/0 conn(0x7fe8440c9170 0x7fe84415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.092 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.086+0000 7fe84bdff700 1 --2- 172.21.15.23:0/2336895449 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8440cb200 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.092 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.086+0000 7fe84bdff700 1 --2- 172.21.15.23:0/2336895449 >> v2:172.21.15.67:3300/0 conn(0x7fe844152cc0 0x7fe8440cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.092 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.086+0000 7fe84bdff700 1 -- 172.21.15.23:0/2336895449 >> 172.21.15.23:0/2336895449 conn(0x7fe844065c20 msgr2=0x7fe844066020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:40.092 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.087+0000 7fe84bdff700 1 -- 172.21.15.23:0/2336895449 shutdown_connections 2023-06-22T07:59:40.092 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.087+0000 7fe84bdff700 1 -- 172.21.15.23:0/2336895449 wait complete. 2023-06-22T07:59:40.093 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.089+0000 7fe84bdff700 1 Processor -- start 2023-06-22T07:59:40.093 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.090+0000 7fe84bdff700 1 -- start start 2023-06-22T07:59:40.095 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.091+0000 7fe84bdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8440c9170 0x7fe844157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:40.095 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.091+0000 7fe84bdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8441e9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:40.095 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.091+0000 7fe84bdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe844152cc0 0x7fe84414d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:40.095 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.091+0000 7fe84bdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8441edd40 con 0x7fe8440c9170 2023-06-22T07:59:40.095 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.091+0000 7fe84bdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8441544e0 con 0x7fe844152cc0 2023-06-22T07:59:40.095 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.091+0000 7fe84bdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe844157cc0 con 0x7fe8441528f0 2023-06-22T07:59:40.095 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe849b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8440c9170 0x7fe844157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:40.096 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe849b9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8440c9170 0x7fe844157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33640/0 (socket says 172.21.15.23:33640) 2023-06-22T07:59:40.096 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe849b9b700 1 -- 172.21.15.23:0/2360680102 learned_addr learned my addr 172.21.15.23:0/2360680102 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:40.096 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe84a39c700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe844152cc0 0x7fe84414d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:40.096 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe84939a700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8441e9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:40.097 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe84a39c700 1 -- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 msgr2=0x7fe8441e9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:40.097 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe84a39c700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8441e9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.097 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe84a39c700 1 -- 172.21.15.23:0/2360680102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8440c9170 msgr2=0x7fe844157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:40.098 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe84a39c700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8440c9170 0x7fe844157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.098 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe84a39c700 1 -- 172.21.15.23:0/2360680102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe838009550 con 0x7fe844152cc0 2023-06-22T07:59:40.098 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe84939a700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8441e9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:40.098 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.092+0000 7fe849b9b700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8440c9170 0x7fe844157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:40.098 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.093+0000 7fe84a39c700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe844152cc0 0x7fe84414d830 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe84406c3d0 tx=0x7fe83c00e0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:40.098 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.093+0000 7fe8367fc700 1 -- 172.21.15.23:0/2360680102 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c003c20 con 0x7fe844152cc0 2023-06-22T07:59:40.098 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.093+0000 7fe8367fc700 1 -- 172.21.15.23:0/2360680102 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83c003da0 con 0x7fe844152cc0 2023-06-22T07:59:40.098 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.093+0000 7fe8367fc700 1 -- 172.21.15.23:0/2360680102 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c00ebb0 con 0x7fe844152cc0 2023-06-22T07:59:40.099 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.093+0000 7fe84bdff700 1 -- 172.21.15.23:0/2360680102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8441ed6c0 con 0x7fe844152cc0 2023-06-22T07:59:40.099 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.093+0000 7fe84bdff700 1 -- 172.21.15.23:0/2360680102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe844157700 con 0x7fe844152cc0 2023-06-22T07:59:40.100 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.095+0000 7fe84bdff700 1 -- 172.21.15.23:0/2360680102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82c0052f0 con 0x7fe844152cc0 2023-06-22T07:59:40.100 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.096+0000 7fe8367fc700 1 -- 172.21.15.23:0/2360680102 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 88981+0+0 (secure 0 0 0) 0x7fe83c00ed10 con 0x7fe844152cc0 2023-06-22T07:59:40.100 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.097+0000 7fe8367fc700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe82806ac80 0x7fe82806d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:40.100 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.097+0000 7fe8367fc700 1 -- 172.21.15.23:0/2360680102 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fe83c087fa0 con 0x7fe844152cc0 2023-06-22T07:59:40.100 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.097+0000 7fe849b9b700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe82806ac80 0x7fe82806d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:40.101 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.098+0000 7fe849b9b700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe82806ac80 0x7fe82806d130 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe838009c10 tx=0x7fe838018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:40.105 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.102+0000 7fe8367fc700 1 -- 172.21.15.23:0/2360680102 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe83c054010 con 0x7fe844152cc0 2023-06-22T07:59:40.381 INFO:tasks.ceph.osd.0.smithi023.stderr:2023-06-22T07:59:40.377+0000 7f7cf67f8700 -1 osd.0 0 waiting for initial osdmap 2023-06-22T07:59:40.422 INFO:tasks.ceph.osd.0.smithi023.stderr:2023-06-22T07:59:40.419+0000 7f7ced7fa700 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T07:59:40.485 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.482+0000 7fe84bdff700 1 -- 172.21.15.23:0/2360680102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe82c0055a0 con 0x7fe844152cc0 2023-06-22T07:59:40.485 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.483+0000 7fe8367fc700 1 -- 172.21.15.23:0/2360680102 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+9438 (secure 0 0 0) 0x7fe83c019090 con 0x7fe844152cc0 2023-06-22T07:59:40.486 INFO:teuthology.misc.health.smithi023.stdout: 2023-06-22T07:59:40.486 INFO:teuthology.misc.health.smithi023.stdout:{"epoch":13,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:40.240688+0000","last_up_change":"2023-06-22T07:59:40.240688+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T07:59:40.489 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.485+0000 7fe84bdff700 1 -- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe82806ac80 msgr2=0x7fe82806d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:40.489 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.485+0000 7fe84bdff700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe82806ac80 0x7fe82806d130 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe838009c10 tx=0x7fe838018000 comp rx=0 tx=0).stop 2023-06-22T07:59:40.489 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.485+0000 7fe84bdff700 1 -- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe844152cc0 msgr2=0x7fe84414d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:40.489 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.485+0000 7fe84bdff700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe844152cc0 0x7fe84414d830 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe84406c3d0 tx=0x7fe83c00e0e0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.489 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.486+0000 7fe84bdff700 1 -- 172.21.15.23:0/2360680102 shutdown_connections 2023-06-22T07:59:40.489 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.486+0000 7fe84bdff700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe82806ac80 0x7fe82806d130 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.490 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.486+0000 7fe84bdff700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe844152cc0 0x7fe84414d830 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.490 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.486+0000 7fe84bdff700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8440c9170 0x7fe844157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.490 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.486+0000 7fe84bdff700 1 --2- 172.21.15.23:0/2360680102 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8441528f0 0x7fe8441e9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:40.490 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.486+0000 7fe84bdff700 1 -- 172.21.15.23:0/2360680102 >> 172.21.15.23:0/2360680102 conn(0x7fe844065c20 msgr2=0x7fe84414ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:40.490 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.486+0000 7fe84bdff700 1 -- 172.21.15.23:0/2360680102 shutdown_connections 2023-06-22T07:59:40.490 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:40.486+0000 7fe84bdff700 1 -- 172.21.15.23:0/2360680102 wait complete. 2023-06-22T07:59:40.504 DEBUG:teuthology.misc:5 of 8 OSDs are up 2023-06-22T07:59:40.553 INFO:tasks.ceph.osd.3.smithi023.stderr:2023-06-22T07:59:40.549+0000 7f3f229f7700 -1 osd.3 0 waiting for initial osdmap 2023-06-22T07:59:40.591 INFO:tasks.ceph.osd.3.smithi023.stderr:2023-06-22T07:59:40.588+0000 7f3f1a1fa700 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T07:59:42.766 INFO:tasks.ceph.osd.2.smithi023.stderr:2023-06-22T07:59:42.764+0000 7fd0e41f7700 -1 osd.2 0 waiting for initial osdmap 2023-06-22T07:59:42.793 INFO:tasks.ceph.osd.2.smithi023.stderr:2023-06-22T07:59:42.790+0000 7fd0db9fa700 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T07:59:47.309 DEBUG:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-22T07:59:47.494 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.491+0000 7f6e773ff700 1 Processor -- start 2023-06-22T07:59:47.494 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.491+0000 7f6e773ff700 1 -- start start 2023-06-22T07:59:47.495 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.492+0000 7f6e773ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6e701528f0 0x7f6e700cb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.495 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.492+0000 7f6e773ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e7015c9b0 0x7f6e700cb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.496 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.492+0000 7f6e773ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e700ca190 con 0x7f6e70152cc0 2023-06-22T07:59:47.496 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.492+0000 7f6e773ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e700ca3c0 con 0x7f6e701528f0 2023-06-22T07:59:47.496 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.492+0000 7f6e773ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700ca6a0 con 0x7f6e7015c9b0 2023-06-22T07:59:47.497 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.493+0000 7f6e7519b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6e701528f0 0x7f6e700cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.498 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.493+0000 7f6e7519b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6e701528f0 0x7f6e700cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44072/0 (socket says 172.21.15.23:44072) 2023-06-22T07:59:47.498 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.493+0000 7f6e7519b700 1 -- 172.21.15.23:0/3407840613 learned_addr learned my addr 172.21.15.23:0/3407840613 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:47.498 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.493+0000 7f6e7599c700 1 --2- 172.21.15.23:0/3407840613 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e7015c9b0 0x7f6e700cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.498 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.493+0000 7f6e7519b700 1 -- 172.21.15.23:0/3407840613 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e7015c9b0 msgr2=0x7f6e700cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:47.498 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.493+0000 7f6e7519b700 1 --2- 172.21.15.23:0/3407840613 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e7015c9b0 0x7f6e700cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.498 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.493+0000 7f6e7519b700 1 -- 172.21.15.23:0/3407840613 >> v1:172.21.15.23:6789/0 conn(0x7f6e70152cc0 legacy=0x7f6e7015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.498 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.493+0000 7f6e67fff700 1 -- 172.21.15.23:0/3407840613 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746080250 0 0) 0x7f6e700ca190 con 0x7f6e70152cc0 2023-06-22T07:59:47.499 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.493+0000 7f6e7519b700 1 -- 172.21.15.23:0/3407840613 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e700ca980 con 0x7f6e701528f0 2023-06-22T07:59:47.499 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.493+0000 7f6e7519b700 1 --2- 172.21.15.23:0/3407840613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e701528f0 0x7f6e700cb200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0098f0 tx=0x7f6e6c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2fcb0438b7d4b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:47.499 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e67fff700 1 -- 172.21.15.23:0/3407840613 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c011830 con 0x7f6e701528f0 2023-06-22T07:59:47.499 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e67fff700 1 -- 172.21.15.23:0/3407840613 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c011990 con 0x7f6e701528f0 2023-06-22T07:59:47.499 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e67fff700 1 -- 172.21.15.23:0/3407840613 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c011d20 con 0x7f6e701528f0 2023-06-22T07:59:47.499 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e773ff700 1 -- 172.21.15.23:0/3407840613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e701528f0 msgr2=0x7f6e700cb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.499 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e773ff700 1 --2- 172.21.15.23:0/3407840613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e701528f0 0x7f6e700cb200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0098f0 tx=0x7f6e6c009c00 comp rx=0 tx=0).stop 2023-06-22T07:59:47.500 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e773ff700 1 -- 172.21.15.23:0/3407840613 shutdown_connections 2023-06-22T07:59:47.500 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e773ff700 1 --2- 172.21.15.23:0/3407840613 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e7015c9b0 0x7f6e700cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.500 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e773ff700 1 --2- 172.21.15.23:0/3407840613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e701528f0 0x7f6e700cb200 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.500 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e773ff700 1 -- 172.21.15.23:0/3407840613 >> 172.21.15.23:0/3407840613 conn(0x7f6e70065c20 msgr2=0x7f6e70066020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:47.500 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e773ff700 1 -- 172.21.15.23:0/3407840613 shutdown_connections 2023-06-22T07:59:47.500 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.494+0000 7f6e773ff700 1 -- 172.21.15.23:0/3407840613 wait complete. 2023-06-22T07:59:47.503 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.496+0000 7f6e773ff700 1 Processor -- start 2023-06-22T07:59:47.503 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.496+0000 7f6e773ff700 1 -- start start 2023-06-22T07:59:47.503 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e773ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e701528f0 0x7f6e701f1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.503 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e773ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e70152cc0 0x7f6e701f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.504 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e773ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e7015c9b0 0x7f6e700b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.504 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e773ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e701574b0 con 0x7f6e7015c9b0 2023-06-22T07:59:47.504 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e773ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e70153660 con 0x7f6e70152cc0 2023-06-22T07:59:47.504 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e773ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e701f2240 con 0x7f6e701528f0 2023-06-22T07:59:47.504 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e7519b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e701528f0 0x7f6e701f1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.507 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e7599c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e7015c9b0 0x7f6e700b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.508 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e7519b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e701528f0 0x7f6e701f1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:50330/0 (socket says 172.21.15.23:50330) 2023-06-22T07:59:47.508 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e7519b700 1 -- 172.21.15.23:0/184570398 learned_addr learned my addr 172.21.15.23:0/184570398 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:47.509 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.497+0000 7f6e7499a700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e70152cc0 0x7f6e701f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.509 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e7519b700 1 -- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e70152cc0 msgr2=0x7f6e701f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.509 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e7519b700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e70152cc0 0x7f6e701f1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.510 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e7519b700 1 -- 172.21.15.23:0/184570398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e7015c9b0 msgr2=0x7f6e700b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.510 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e7519b700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e7015c9b0 0x7f6e700b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.510 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e7519b700 1 -- 172.21.15.23:0/184570398 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c009550 con 0x7f6e701528f0 2023-06-22T07:59:47.511 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e7599c700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e7015c9b0 0x7f6e700b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:47.511 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e7519b700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e701528f0 0x7f6e701f1600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6e70072560 tx=0x7f6e6c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:47.511 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e65ffb700 1 -- 172.21.15.23:0/184570398 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c004300 con 0x7f6e701528f0 2023-06-22T07:59:47.511 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e65ffb700 1 -- 172.21.15.23:0/184570398 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c01f070 con 0x7f6e701528f0 2023-06-22T07:59:47.512 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e773ff700 1 -- 172.21.15.23:0/184570398 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e70158830 con 0x7f6e701528f0 2023-06-22T07:59:47.512 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.498+0000 7f6e65ffb700 1 -- 172.21.15.23:0/184570398 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c0036a0 con 0x7f6e701528f0 2023-06-22T07:59:47.512 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.499+0000 7f6e773ff700 1 -- 172.21.15.23:0/184570398 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e70153280 con 0x7f6e701528f0 2023-06-22T07:59:47.513 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.500+0000 7f6e773ff700 1 -- 172.21.15.23:0/184570398 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e70158d50 con 0x7f6e701528f0 2023-06-22T07:59:47.514 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.500+0000 7f6e65ffb700 1 -- 172.21.15.23:0/184570398 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6e6c021030 con 0x7f6e701528f0 2023-06-22T07:59:47.514 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.501+0000 7f6e65ffb700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ace0 0x7f6e5406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.514 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.502+0000 7f6e65ffb700 1 -- 172.21.15.23:0/184570398 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6e6c08a380 con 0x7f6e701528f0 2023-06-22T07:59:47.515 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.502+0000 7f6e7499a700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ace0 0x7f6e5406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.515 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.502+0000 7f6e7499a700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ace0 0x7f6e5406d190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6e701562f0 tx=0x7f6e60009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:47.515 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.507+0000 7f6e65ffb700 1 -- 172.21.15.23:0/184570398 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6e70158d50 con 0x7f6e701528f0 2023-06-22T07:59:47.717 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.714+0000 7f6e773ff700 1 -- 172.21.15.23:0/184570398 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e701530e0 con 0x7f6e701528f0 2023-06-22T07:59:47.718 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.716+0000 7f6e65ffb700 1 -- 172.21.15.23:0/184570398 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12778 (secure 0 0 0) 0x7f6e701530e0 con 0x7f6e701528f0 2023-06-22T07:59:47.718 INFO:teuthology.misc.health.smithi023.stdout: 2023-06-22T07:59:47.719 INFO:teuthology.misc.health.smithi023.stdout:{"epoch":17,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:44.397847+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T07:59:47.720 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.717+0000 7f6e773ff700 1 -- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ace0 msgr2=0x7f6e5406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.721 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.717+0000 7f6e773ff700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ace0 0x7f6e5406d190 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6e701562f0 tx=0x7f6e60009040 comp rx=0 tx=0).stop 2023-06-22T07:59:47.721 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.717+0000 7f6e773ff700 1 -- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e701528f0 msgr2=0x7f6e701f1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.721 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.717+0000 7f6e773ff700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e701528f0 0x7f6e701f1600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6e70072560 tx=0x7f6e6c004750 comp rx=0 tx=0).stop 2023-06-22T07:59:47.721 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.717+0000 7f6e773ff700 1 -- 172.21.15.23:0/184570398 shutdown_connections 2023-06-22T07:59:47.721 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.717+0000 7f6e773ff700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e7015c9b0 0x7f6e700b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.721 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.717+0000 7f6e773ff700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e701528f0 0x7f6e701f1600 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.722 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.717+0000 7f6e773ff700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ace0 0x7f6e5406d190 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.722 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.717+0000 7f6e773ff700 1 --2- 172.21.15.23:0/184570398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e70152cc0 0x7f6e701f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.722 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.717+0000 7f6e773ff700 1 -- 172.21.15.23:0/184570398 >> 172.21.15.23:0/184570398 conn(0x7f6e70065c20 msgr2=0x7f6e7014d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:47.722 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.718+0000 7f6e773ff700 1 -- 172.21.15.23:0/184570398 shutdown_connections 2023-06-22T07:59:47.722 INFO:teuthology.misc.health.smithi023.stderr:2023-06-22T07:59:47.718+0000 7f6e773ff700 1 -- 172.21.15.23:0/184570398 wait complete. 2023-06-22T07:59:47.735 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-06-22T07:59:47.736 INFO:tasks.ceph:Creating RBD pool 2023-06-22T07:59:47.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo ceph --cluster ceph osd pool create rbd 8 2023-06-22T07:59:47.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.924+0000 7fa2b9bff700 1 Processor -- start 2023-06-22T07:59:47.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.925+0000 7fa2b9bff700 1 -- start start 2023-06-22T07:59:47.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.926+0000 7fa2b9bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa2b40c9ee0 0x7fa2b40ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.926+0000 7fa2b9bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b40c9aa0 con 0x7fa2b4152ba0 2023-06-22T07:59:47.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.926+0000 7fa2b9bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b406aed0 con 0x7fa2b41593f0 2023-06-22T07:59:47.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.926+0000 7fa2b9bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b406b1b0 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.926+0000 7fa2b37fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa2b4152ba0 0x7fa2b40c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:42298/0 (socket says 172.21.15.23:42298) 2023-06-22T07:59:47.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.926+0000 7fa2b37fe700 1 -- 172.21.15.23:0/3858745548 learned_addr learned my addr 172.21.15.23:0/3858745548 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T07:59:47.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.926+0000 7fa2b3fff700 1 --2- 172.21.15.23:0/3858745548 >> v2:172.21.15.67:3300/0 conn(0x7fa2b40c9ee0 0x7fa2b40ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.927+0000 7fa2b27fc700 1 -- 172.21.15.23:0/3858745548 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193494777 0 0) 0x7fa2b406aed0 con 0x7fa2b41593f0 2023-06-22T07:59:47.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.927+0000 7fa2b3fff700 1 -- 172.21.15.23:0/3858745548 >> v1:172.21.15.67:6789/0 conn(0x7fa2b41593f0 legacy=0x7fa2b40c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.927+0000 7fa2b3fff700 1 -- 172.21.15.23:0/3858745548 >> v1:172.21.15.23:6789/0 conn(0x7fa2b4152ba0 legacy=0x7fa2b40c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.927+0000 7fa2b3fff700 1 -- 172.21.15.23:0/3858745548 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b406b490 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.927+0000 7fa2b3fff700 1 --2- 172.21.15.23:0/3858745548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b40c9ee0 0x7fa2b40ca2f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009d90 tx=0x7fa2ac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25caaee5a416a8e7 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:47.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.927+0000 7fa2b27fc700 1 -- 172.21.15.23:0/3858745548 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac00e070 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.927+0000 7fa2b27fc700 1 -- 172.21.15.23:0/3858745548 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac00bad0 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.927+0000 7fa2b27fc700 1 -- 172.21.15.23:0/3858745548 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac00be80 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.928+0000 7fa2b9bff700 1 -- 172.21.15.23:0/3858745548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b40c9ee0 msgr2=0x7fa2b40ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.928+0000 7fa2b9bff700 1 --2- 172.21.15.23:0/3858745548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b40c9ee0 0x7fa2b40ca2f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009d90 tx=0x7fa2ac00b330 comp rx=0 tx=0).stop 2023-06-22T07:59:47.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.928+0000 7fa2b9bff700 1 -- 172.21.15.23:0/3858745548 shutdown_connections 2023-06-22T07:59:47.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.928+0000 7fa2b9bff700 1 --2- 172.21.15.23:0/3858745548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b40c9ee0 0x7fa2b40ca2f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.928+0000 7fa2b9bff700 1 -- 172.21.15.23:0/3858745548 >> 172.21.15.23:0/3858745548 conn(0x7fa2b4065b90 msgr2=0x7fa2b4065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:47.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.928+0000 7fa2b9bff700 1 -- 172.21.15.23:0/3858745548 shutdown_connections 2023-06-22T07:59:47.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.928+0000 7fa2b9bff700 1 -- 172.21.15.23:0/3858745548 wait complete. 2023-06-22T07:59:47.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.929+0000 7fa2b9bff700 1 Processor -- start 2023-06-22T07:59:47.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.930+0000 7fa2b9bff700 1 -- start start 2023-06-22T07:59:47.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.930+0000 7fa2b9bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2b40c9ee0 0x7fa2b414b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.930+0000 7fa2b9bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b4152ba0 0x7fa2b414b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.930+0000 7fa2b9bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa2b41593f0 0x7fa2b40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.930+0000 7fa2b9bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b41557d0 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.930+0000 7fa2b9bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b41561a0 con 0x7fa2b4152ba0 2023-06-22T07:59:47.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.930+0000 7fa2b9bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40c6210 con 0x7fa2b41593f0 2023-06-22T07:59:47.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2b40c9ee0 0x7fa2b414b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2b40c9ee0 0x7fa2b414b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44086/0 (socket says 172.21.15.23:44086) 2023-06-22T07:59:47.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b37fe700 1 -- 172.21.15.23:0/4050509159 learned_addr learned my addr 172.21.15.23:0/4050509159 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:47.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b2ffd700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b4152ba0 0x7fa2b414b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b3fff700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa2b41593f0 0x7fa2b40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b37fe700 1 -- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa2b41593f0 msgr2=0x7fa2b40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b37fe700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa2b41593f0 0x7fa2b40cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b37fe700 1 -- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b4152ba0 msgr2=0x7fa2b414b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:47.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b37fe700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b4152ba0 0x7fa2b414b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:47.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b37fe700 1 -- 172.21.15.23:0/4050509159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac009a70 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.931+0000 7fa2b37fe700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2b40c9ee0 0x7fa2b414b2e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400c7b0 tx=0x7fa2a400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:47.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.932+0000 7fa29bfff700 1 -- 172.21.15.23:0/4050509159 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4004c80 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.932+0000 7fa29bfff700 1 -- 172.21.15.23:0/4050509159 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a4004de0 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.932+0000 7fa29bfff700 1 -- 172.21.15.23:0/4050509159 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4007310 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.932+0000 7fa2b9bff700 1 -- 172.21.15.23:0/4050509159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b4153a90 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.932+0000 7fa2b9bff700 1 -- 172.21.15.23:0/4050509159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b4155110 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.933+0000 7fa2b9bff700 1 -- 172.21.15.23:0/4050509159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b40cbed0 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.934+0000 7fa29bfff700 1 -- 172.21.15.23:0/4050509159 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa2a4007470 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.935+0000 7fa29bfff700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa2a006ace0 0x7fa2a006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:47.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.935+0000 7fa29bfff700 1 -- 172.21.15.23:0/4050509159 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa2a4088d40 con 0x7fa2b40c9ee0 2023-06-22T07:59:47.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.935+0000 7fa2b2ffd700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa2a006ace0 0x7fa2a006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:47.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.936+0000 7fa2b2ffd700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa2a006ace0 0x7fa2a006d190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4156020 tx=0x7fa2a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:47.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:47.940+0000 7fa29bfff700 1 -- 172.21.15.23:0/4050509159 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa2a4054290 con 0x7fa2b40c9ee0 2023-06-22T07:59:48.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.150+0000 7fa2b9bff700 1 -- 172.21.15.23:0/4050509159 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fa2b4152f70 con 0x7fa2b40c9ee0 2023-06-22T07:59:48.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.430+0000 7fa29bfff700 1 -- 172.21.15.23:0/4050509159 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v18) v1 ==== 109+0+0 (secure 0 0 0) 0x7fa2a405d020 con 0x7fa2b40c9ee0 2023-06-22T07:59:48.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 -- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa2a006ace0 msgr2=0x7fa2a006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:48.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa2a006ace0 0x7fa2a006d190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4156020 tx=0x7fa2a8009040 comp rx=0 tx=0).stop 2023-06-22T07:59:48.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 -- 172.21.15.23:0/4050509159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2b40c9ee0 msgr2=0x7fa2b414b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:48.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2b40c9ee0 0x7fa2b414b2e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400c7b0 tx=0x7fa2a400cac0 comp rx=0 tx=0).stop 2023-06-22T07:59:48.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 -- 172.21.15.23:0/4050509159 shutdown_connections 2023-06-22T07:59:48.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa2b41593f0 0x7fa2b40cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:48.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2b40c9ee0 0x7fa2b414b2e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:48.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa2a006ace0 0x7fa2a006d190 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:48.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 --2- 172.21.15.23:0/4050509159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2b4152ba0 0x7fa2b414b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:48.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 -- 172.21.15.23:0/4050509159 >> 172.21.15.23:0/4050509159 conn(0x7fa2b4065b90 msgr2=0x7fa2b4151da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:48.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 -- 172.21.15.23:0/4050509159 shutdown_connections 2023-06-22T07:59:48.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.433+0000 7fa2b9bff700 1 -- 172.21.15.23:0/4050509159 wait complete. 2023-06-22T07:59:48.439 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'rbd' created 2023-06-22T07:59:48.455 DEBUG:teuthology.orchestra.run.smithi023:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-06-22T07:59:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.646+0000 7f12f91ff700 1 Processor -- start 2023-06-22T07:59:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.647+0000 7f12f91ff700 1 -- start start 2023-06-22T07:59:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.648+0000 7f12f91ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f12f4152ba0 0x7f12f41593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.648+0000 7f12f91ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f4159930 con 0x7f12f40c9260 2023-06-22T07:59:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.648+0000 7f12f91ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f40c9aa0 con 0x7f12f40c9ee0 2023-06-22T07:59:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.648+0000 7f12f91ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f406aed0 con 0x7f12f4152ba0 2023-06-22T07:59:48.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.648+0000 7f12f2d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f12f4152ba0 0x7f12f41593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:48.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.648+0000 7f12f2d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f12f4152ba0 0x7f12f41593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44092/0 (socket says 172.21.15.23:44092) 2023-06-22T07:59:48.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.648+0000 7f12f2d9d700 1 -- 172.21.15.23:0/2634270079 learned_addr learned my addr 172.21.15.23:0/2634270079 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:48.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.648+0000 7f12f259c700 1 --1- 172.21.15.23:0/2634270079 >> v1:172.21.15.23:6789/0 conn(0x7f12f40c9260 0x7f12f40c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:42310/0 (socket says 172.21.15.23:42310) 2023-06-22T07:59:48.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.649+0000 7f12f2d9d700 1 -- 172.21.15.23:0/2634270079 >> v1:172.21.15.67:6789/0 conn(0x7f12f40c9ee0 legacy=0x7f12f40ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:48.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.649+0000 7f12f2d9d700 1 -- 172.21.15.23:0/2634270079 >> v1:172.21.15.23:6789/0 conn(0x7f12f40c9260 legacy=0x7f12f40c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:48.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.649+0000 7f12f2d9d700 1 -- 172.21.15.23:0/2634270079 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f406b1b0 con 0x7f12f4152ba0 2023-06-22T07:59:48.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.649+0000 7f12f2d9d700 1 --2- 172.21.15.23:0/2634270079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 0x7f12f41593f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f12e40098f0 tx=0x7f12e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea7349c0c43c96f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:48.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.649+0000 7f12f1d9b700 1 -- 172.21.15.23:0/2634270079 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e400b8a0 con 0x7f12f4152ba0 2023-06-22T07:59:48.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.649+0000 7f12f1d9b700 1 -- 172.21.15.23:0/2634270079 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e400ba00 con 0x7f12f4152ba0 2023-06-22T07:59:48.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.649+0000 7f12f1d9b700 1 -- 172.21.15.23:0/2634270079 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e400bd90 con 0x7f12f4152ba0 2023-06-22T07:59:48.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.650+0000 7f12f91ff700 1 -- 172.21.15.23:0/2634270079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 msgr2=0x7f12f41593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:48.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.650+0000 7f12f91ff700 1 --2- 172.21.15.23:0/2634270079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 0x7f12f41593f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f12e40098f0 tx=0x7f12e4009c00 comp rx=0 tx=0).stop 2023-06-22T07:59:48.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.650+0000 7f12f91ff700 1 -- 172.21.15.23:0/2634270079 shutdown_connections 2023-06-22T07:59:48.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.650+0000 7f12f91ff700 1 --2- 172.21.15.23:0/2634270079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 0x7f12f41593f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:48.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.650+0000 7f12f91ff700 1 -- 172.21.15.23:0/2634270079 >> 172.21.15.23:0/2634270079 conn(0x7f12f4065b90 msgr2=0x7f12f4065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:48.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.650+0000 7f12f91ff700 1 -- 172.21.15.23:0/2634270079 shutdown_connections 2023-06-22T07:59:48.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.650+0000 7f12f91ff700 1 -- 172.21.15.23:0/2634270079 wait complete. 2023-06-22T07:59:48.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.651+0000 7f12f91ff700 1 Processor -- start 2023-06-22T07:59:48.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.652+0000 7f12f91ff700 1 -- start start 2023-06-22T07:59:48.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.652+0000 7f12f91ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f40c9260 0x7f12f4151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.652+0000 7f12f91ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f12f40c9ee0 0x7f12f40b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f91ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 0x7f12f4148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f91ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f4159740 con 0x7f12f4152ba0 2023-06-22T07:59:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f91ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f41e1130 con 0x7f12f40c9260 2023-06-22T07:59:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f91ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f41e1750 con 0x7f12f40c9ee0 2023-06-22T07:59:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 0x7f12f4148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f2d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f40c9260 0x7f12f4151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f12f40c9ee0 0x7f12f40b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 0x7f12f4148180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44106/0 (socket says 172.21.15.23:44106) 2023-06-22T07:59:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f359e700 1 -- 172.21.15.23:0/2701316903 learned_addr learned my addr 172.21.15.23:0/2701316903 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f359e700 1 -- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f12f40c9ee0 msgr2=0x7f12f40b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f359e700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f12f40c9ee0 0x7f12f40b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f359e700 1 -- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f40c9260 msgr2=0x7f12f4151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f359e700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f40c9260 0x7f12f4151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:48.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f359e700 1 -- 172.21.15.23:0/2701316903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e4009550 con 0x7f12f4152ba0 2023-06-22T07:59:48.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.653+0000 7f12f359e700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 0x7f12f4148180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00cc50 tx=0x7f12ec00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:48.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.654+0000 7f12db7fe700 1 -- 172.21.15.23:0/2701316903 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec003c00 con 0x7f12f4152ba0 2023-06-22T07:59:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.654+0000 7f12db7fe700 1 -- 172.21.15.23:0/2701316903 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec003d60 con 0x7f12f4152ba0 2023-06-22T07:59:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.654+0000 7f12db7fe700 1 -- 172.21.15.23:0/2701316903 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec00aaf0 con 0x7f12f4152ba0 2023-06-22T07:59:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.654+0000 7f12f91ff700 1 -- 172.21.15.23:0/2701316903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f41f2400 con 0x7f12f4152ba0 2023-06-22T07:59:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.654+0000 7f12f91ff700 1 -- 172.21.15.23:0/2701316903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f40b23d0 con 0x7f12f4152ba0 2023-06-22T07:59:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.655+0000 7f12f91ff700 1 -- 172.21.15.23:0/2701316903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f40cbed0 con 0x7f12f4152ba0 2023-06-22T07:59:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.656+0000 7f12db7fe700 1 -- 172.21.15.23:0/2701316903 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f12ec00ac50 con 0x7f12f4152ba0 2023-06-22T07:59:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.658+0000 7f12db7fe700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12e006ace0 0x7f12e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.658+0000 7f12db7fe700 1 -- 172.21.15.23:0/2701316903 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f12ec089210 con 0x7f12f4152ba0 2023-06-22T07:59:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.658+0000 7f12f2d9d700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12e006ace0 0x7f12e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.659+0000 7f12f2d9d700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12e006ace0 0x7f12e006d190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12e4014040 tx=0x7f12e4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:48.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.663+0000 7f12db7fe700 1 -- 172.21.15.23:0/2701316903 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f12ec054580 con 0x7f12f4152ba0 2023-06-22T07:59:48.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:48.871+0000 7f12f91ff700 1 -- 172.21.15.23:0/2701316903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f12f41f2920 con 0x7f12f4152ba0 2023-06-22T07:59:49.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.429+0000 7f12db7fe700 1 -- 172.21.15.23:0/2701316903 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v19) v1 ==== 173+0+0 (secure 0 0 0) 0x7f12ec057ba0 con 0x7f12f4152ba0 2023-06-22T07:59:49.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.431+0000 7f12f91ff700 1 -- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12e006ace0 msgr2=0x7f12e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:49.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.431+0000 7f12f91ff700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12e006ace0 0x7f12e006d190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12e4014040 tx=0x7f12e4008030 comp rx=0 tx=0).stop 2023-06-22T07:59:49.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.431+0000 7f12f91ff700 1 -- 172.21.15.23:0/2701316903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 msgr2=0x7f12f4148180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:49.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.431+0000 7f12f91ff700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 0x7f12f4148180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00cc50 tx=0x7f12ec00cf60 comp rx=0 tx=0).stop 2023-06-22T07:59:49.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.432+0000 7f12f91ff700 1 -- 172.21.15.23:0/2701316903 shutdown_connections 2023-06-22T07:59:49.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.432+0000 7f12f91ff700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f4152ba0 0x7f12f4148180 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.432+0000 7f12f91ff700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12e006ace0 0x7f12e006d190 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.432+0000 7f12f91ff700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f40c9260 0x7f12f4151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.432+0000 7f12f91ff700 1 --2- 172.21.15.23:0/2701316903 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f12f40c9ee0 0x7f12f40b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.432+0000 7f12f91ff700 1 -- 172.21.15.23:0/2701316903 >> 172.21.15.23:0/2701316903 conn(0x7f12f4065b90 msgr2=0x7f12f41577b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:49.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.432+0000 7f12f91ff700 1 -- 172.21.15.23:0/2701316903 shutdown_connections 2023-06-22T07:59:49.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.432+0000 7f12f91ff700 1 -- 172.21.15.23:0/2701316903 wait complete. 2023-06-22T07:59:49.436 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rbd' on pool 'rbd' 2023-06-22T07:59:49.458 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-06-22T07:59:49.458 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-22T07:59:49.459 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-22T07:59:49.463 INFO:tasks.ceph.mds.a:Started 2023-06-22T07:59:49.463 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-22T07:59:49.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-22T07:59:49.467 INFO:tasks.ceph.mds.c:Started 2023-06-22T07:59:49.468 INFO:tasks.ceph.mds.e:Restarting daemon 2023-06-22T07:59:49.468 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-06-22T07:59:49.472 INFO:tasks.ceph.mds.e:Started 2023-06-22T07:59:49.473 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-22T07:59:49.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-22T07:59:49.475 INFO:tasks.ceph.mds.b:Started 2023-06-22T07:59:49.475 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-22T07:59:49.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-22T07:59:49.477 INFO:tasks.ceph.mds.d:Started 2023-06-22T07:59:49.478 INFO:tasks.ceph.mds.f:Restarting daemon 2023-06-22T07:59:49.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-06-22T07:59:49.480 INFO:tasks.ceph.mds.f:Started 2023-06-22T07:59:49.480 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-06-22T07:59:49.481 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T07:59:49.570 INFO:tasks.ceph.mds.d.smithi067.stdout:starting mds.d at 2023-06-22T07:59:49.571 INFO:tasks.ceph.mds.b.smithi067.stdout:starting mds.b at 2023-06-22T07:59:49.571 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T07:59:49.567+0000 7fc6c0d01700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-06-22T07:59:49.572 INFO:tasks.ceph.mds.f.smithi067.stdout:starting mds.f at 2023-06-22T07:59:49.629 INFO:tasks.ceph.mds.a.smithi023.stdout:starting mds.a at 2023-06-22T07:59:49.629 INFO:tasks.ceph.mds.c.smithi023.stdout:starting mds.c at 2023-06-22T07:59:49.644 INFO:tasks.ceph.mds.e.smithi023.stdout:starting mds.e at 2023-06-22T07:59:49.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.704+0000 7f6bc83ff700 1 Processor -- start 2023-06-22T07:59:49.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.705+0000 7f6bc83ff700 1 -- start start 2023-06-22T07:59:49.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.706+0000 7f6bc83ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 0x7f6bc00cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:49.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.706+0000 7f6bc83ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6bc00c9fd0 0x7f6bc015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:49.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.706+0000 7f6bc83ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc015c370 con 0x7f6bc0158880 2023-06-22T07:59:49.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.706+0000 7f6bc83ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc01530b0 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.706+0000 7f6bc83ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc00cf6f0 con 0x7f6bc0152ce0 2023-06-22T07:59:49.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.706+0000 7f6bc699c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f6bc0158880 0x7f6bc0158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:42338/0 (socket says 172.21.15.23:42338) 2023-06-22T07:59:49.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.706+0000 7f6bc699c700 1 -- 172.21.15.23:0/1670599949 learned_addr learned my addr 172.21.15.23:0/1670599949 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T07:59:49.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.706+0000 7f6bc619b700 1 --2- 172.21.15.23:0/1670599949 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 0x7f6bc00cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:49.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.706+0000 7f6bc599a700 1 --2- 172.21.15.23:0/1670599949 >> v2:172.21.15.67:3300/0 conn(0x7f6bc00c9fd0 0x7f6bc015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:49.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.706+0000 7f6bc5199700 1 -- 172.21.15.23:0/1670599949 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133033194 0 0) 0x7f6bc015c370 con 0x7f6bc0158880 2023-06-22T07:59:49.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc5199700 1 -- 172.21.15.23:0/1670599949 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bac0035b0 con 0x7f6bc0158880 2023-06-22T07:59:49.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc619b700 1 -- 172.21.15.23:0/1670599949 >> v2:172.21.15.67:3300/0 conn(0x7f6bc00c9fd0 msgr2=0x7f6bc015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:49.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc619b700 1 --2- 172.21.15.23:0/1670599949 >> v2:172.21.15.67:3300/0 conn(0x7f6bc00c9fd0 0x7f6bc015c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc619b700 1 -- 172.21.15.23:0/1670599949 >> v1:172.21.15.23:6789/0 conn(0x7f6bc0158880 legacy=0x7f6bc0158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:49.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc619b700 1 -- 172.21.15.23:0/1670599949 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc015bda0 con 0x7f6bc0152ce0 2023-06-22T07:59:49.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc619b700 1 --2- 172.21.15.23:0/1670599949 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 0x7f6bc00cf1b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc009840 tx=0x7f6bbc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9b6da08c0706d4a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:49.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc5199700 1 -- 172.21.15.23:0/1670599949 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc011830 con 0x7f6bc0152ce0 2023-06-22T07:59:49.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc5199700 1 -- 172.21.15.23:0/1670599949 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc011990 con 0x7f6bc0152ce0 2023-06-22T07:59:49.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc5199700 1 -- 172.21.15.23:0/1670599949 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc011d20 con 0x7f6bc0152ce0 2023-06-22T07:59:49.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1670599949 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 msgr2=0x7f6bc00cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:49.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.707+0000 7f6bc83ff700 1 --2- 172.21.15.23:0/1670599949 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 0x7f6bc00cf1b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc009840 tx=0x7f6bbc009b50 comp rx=0 tx=0).stop 2023-06-22T07:59:49.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.708+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1670599949 shutdown_connections 2023-06-22T07:59:49.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.708+0000 7f6bc83ff700 1 --2- 172.21.15.23:0/1670599949 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 0x7f6bc00cf1b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.708+0000 7f6bc83ff700 1 --2- 172.21.15.23:0/1670599949 >> v2:172.21.15.67:3300/0 conn(0x7f6bc00c9fd0 0x7f6bc015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.708+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1670599949 >> 172.21.15.23:0/1670599949 conn(0x7f6bc006ae30 msgr2=0x7f6bc006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:49.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.708+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1670599949 shutdown_connections 2023-06-22T07:59:49.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.708+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1670599949 wait complete. 2023-06-22T07:59:49.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.709+0000 7f6bc83ff700 1 Processor -- start 2023-06-22T07:59:49.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.710+0000 7f6bc83ff700 1 -- start start 2023-06-22T07:59:49.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc83ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bc00c9fd0 0x7f6bc00b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:49.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc83ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 0x7f6bc0148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:49.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc83ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bc0158880 0x7f6bc01f1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:49.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc83ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc01f2340 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc83ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc014ec40 con 0x7f6bc0158880 2023-06-22T07:59:49.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc83ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc00b3d30 con 0x7f6bc0152ce0 2023-06-22T07:59:49.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bc00c9fd0 0x7f6bc00b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:49.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bc00c9fd0 0x7f6bc00b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44168/0 (socket says 172.21.15.23:44168) 2023-06-22T07:59:49.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc619b700 1 -- 172.21.15.23:0/1694796631 learned_addr learned my addr 172.21.15.23:0/1694796631 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:49.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc599a700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 0x7f6bc0148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:49.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc699c700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bc0158880 0x7f6bc01f1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:49.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc619b700 1 -- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 msgr2=0x7f6bc0148430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:49.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc619b700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 0x7f6bc0148430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc619b700 1 -- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bc0158880 msgr2=0x7f6bc01f1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:49.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc619b700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bc0158880 0x7f6bc01f1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc619b700 1 -- 172.21.15.23:0/1694796631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc0094a0 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.711+0000 7f6bc619b700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bc00c9fd0 0x7f6bc00b37f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0090b0 tx=0x7f6bbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:49.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.712+0000 7f6bb6ffd700 1 -- 172.21.15.23:0/1694796631 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc004590 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.712+0000 7f6bb6ffd700 1 -- 172.21.15.23:0/1694796631 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc01f070 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.712+0000 7f6bb6ffd700 1 -- 172.21.15.23:0/1694796631 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc0038e0 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.712+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1694796631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc00b2d10 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.712+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1694796631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc00b32d0 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.713+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1694796631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc004ea20 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.714+0000 7f6bb6ffd700 1 -- 172.21.15.23:0/1694796631 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6bbc003a40 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.715+0000 7f6bb6ffd700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6ba406ad10 0x7f6ba406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:49.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.716+0000 7f6bb6ffd700 1 -- 172.21.15.23:0/1694796631 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6bbc021070 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.716+0000 7f6bc599a700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6ba406ad10 0x7f6ba406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:49.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.716+0000 7f6bc599a700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6ba406ad10 0x7f6ba406d1c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00ca7d0 tx=0x7f6bb0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:49.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.721+0000 7f6bb6ffd700 1 -- 172.21.15.23:0/1694796631 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6bbc055c80 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.930+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1694796631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bc00c9db0 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.931+0000 7f6bb6ffd700 1 -- 172.21.15.23:0/1694796631 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14603 (secure 0 0 0) 0x7f6bbc0592a0 con 0x7f6bc00c9fd0 2023-06-22T07:59:49.934 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:59:49.934 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":19,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:49.426193+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T07:59:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6ba406ad10 msgr2=0x7f6ba406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6ba406ad10 0x7f6ba406d1c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00ca7d0 tx=0x7f6bb0009210 comp rx=0 tx=0).stop 2023-06-22T07:59:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1694796631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bc00c9fd0 msgr2=0x7f6bc00b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bc00c9fd0 0x7f6bc00b37f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc0090b0 tx=0x7f6bbc004750 comp rx=0 tx=0).stop 2023-06-22T07:59:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1694796631 shutdown_connections 2023-06-22T07:59:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6ba406ad10 0x7f6ba406d1c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bc0158880 0x7f6bc01f1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bc00c9fd0 0x7f6bc00b37f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 --2- 172.21.15.23:0/1694796631 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6bc0152ce0 0x7f6bc0148430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1694796631 >> 172.21.15.23:0/1694796631 conn(0x7f6bc006ae30 msgr2=0x7f6bc0156de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1694796631 shutdown_connections 2023-06-22T07:59:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:49.934+0000 7f6bc83ff700 1 -- 172.21.15.23:0/1694796631 wait complete. 2023-06-22T07:59:49.955 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T07:59:40.617747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-22T07:59:48.190528+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-22T07:59:49.955 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T07:59:50.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.153+0000 7f584a7ff700 1 Processor -- start 2023-06-22T07:59:50.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.153+0000 7f584a7ff700 1 -- start start 2023-06-22T07:59:50.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.155+0000 7f584a7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5844152ba0 0x7f58441593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.155+0000 7f584a7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f58440c9ee0 0x7f58440ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.155+0000 7f584a7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5844159930 con 0x7f58440c9260 2023-06-22T07:59:50.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.155+0000 7f584a7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58440c9aa0 con 0x7f5844152ba0 2023-06-22T07:59:50.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.155+0000 7f584a7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f584406aed0 con 0x7f58440c9ee0 2023-06-22T07:59:50.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.155+0000 7f5843fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5844152ba0 0x7f58441593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.155+0000 7f58437fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f58440c9260 0x7f58440c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:42348/0 (socket says 172.21.15.23:42348) 2023-06-22T07:59:50.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.155+0000 7f58437fe700 1 -- 172.21.15.23:0/2374846179 learned_addr learned my addr 172.21.15.23:0/2374846179 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T07:59:50.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.155+0000 7f5848d9c700 1 --2- 172.21.15.23:0/2374846179 >> v2:172.21.15.67:3300/0 conn(0x7f58440c9ee0 0x7f58440ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.156+0000 7f5843fff700 1 -- 172.21.15.23:0/2374846179 >> v2:172.21.15.67:3300/0 conn(0x7f58440c9ee0 msgr2=0x7f58440ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:50.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.156+0000 7f5843fff700 1 --2- 172.21.15.23:0/2374846179 >> v2:172.21.15.67:3300/0 conn(0x7f58440c9ee0 0x7f58440ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.156+0000 7f5843fff700 1 -- 172.21.15.23:0/2374846179 >> v1:172.21.15.23:6789/0 conn(0x7f58440c9260 legacy=0x7f58440c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.156+0000 7f5842ffd700 1 -- 172.21.15.23:0/2374846179 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074259520 0 0) 0x7f582c001110 con 0x7f58440c9260 2023-06-22T07:59:50.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.156+0000 7f5843fff700 1 -- 172.21.15.23:0/2374846179 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584406b1b0 con 0x7f5844152ba0 2023-06-22T07:59:50.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.156+0000 7f5848d9c700 1 --2- 172.21.15.23:0/2374846179 >> v2:172.21.15.67:3300/0 conn(0x7f58440c9ee0 0x7f58440ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:50.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.156+0000 7f5843fff700 1 --2- 172.21.15.23:0/2374846179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5844152ba0 0x7f58441593f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f584408b390 tx=0x7f5834009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e599e7d12020b310 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:50.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.156+0000 7f5842ffd700 1 -- 172.21.15.23:0/2374846179 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834011830 con 0x7f5844152ba0 2023-06-22T07:59:50.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.156+0000 7f5842ffd700 1 -- 172.21.15.23:0/2374846179 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5834011990 con 0x7f5844152ba0 2023-06-22T07:59:50.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.156+0000 7f5842ffd700 1 -- 172.21.15.23:0/2374846179 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834011d20 con 0x7f5844152ba0 2023-06-22T07:59:50.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.157+0000 7f584a7ff700 1 -- 172.21.15.23:0/2374846179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5844152ba0 msgr2=0x7f58441593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.157+0000 7f584a7ff700 1 --2- 172.21.15.23:0/2374846179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5844152ba0 0x7f58441593f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f584408b390 tx=0x7f5834009b20 comp rx=0 tx=0).stop 2023-06-22T07:59:50.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.157+0000 7f584a7ff700 1 -- 172.21.15.23:0/2374846179 shutdown_connections 2023-06-22T07:59:50.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.157+0000 7f584a7ff700 1 --2- 172.21.15.23:0/2374846179 >> v2:172.21.15.67:3300/0 conn(0x7f58440c9ee0 0x7f58440ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.157+0000 7f584a7ff700 1 --2- 172.21.15.23:0/2374846179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5844152ba0 0x7f58441593f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.157+0000 7f584a7ff700 1 -- 172.21.15.23:0/2374846179 >> 172.21.15.23:0/2374846179 conn(0x7f5844065b90 msgr2=0x7f5844065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:50.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.157+0000 7f584a7ff700 1 -- 172.21.15.23:0/2374846179 shutdown_connections 2023-06-22T07:59:50.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.157+0000 7f584a7ff700 1 -- 172.21.15.23:0/2374846179 wait complete. 2023-06-22T07:59:50.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.159+0000 7f584a7ff700 1 Processor -- start 2023-06-22T07:59:50.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.160+0000 7f584a7ff700 1 -- start start 2023-06-22T07:59:50.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f584a7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58440c9260 0x7f58440ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f584a7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58440c9ee0 0x7f584414d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f584a7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5844152ba0 0x7f584414daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f584a7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441f2180 con 0x7f58440c9260 2023-06-22T07:59:50.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f584a7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441ef9c0 con 0x7f58440c9ee0 2023-06-22T07:59:50.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f584a7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58440b35d0 con 0x7f5844152ba0 2023-06-22T07:59:50.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f5843fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58440c9260 0x7f58440ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f5843fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58440c9260 0x7f58440ced70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44182/0 (socket says 172.21.15.23:44182) 2023-06-22T07:59:50.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f5843fff700 1 -- 172.21.15.23:0/2104797235 learned_addr learned my addr 172.21.15.23:0/2104797235 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:50.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f5848d9c700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5844152ba0 0x7f584414daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f58437fe700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58440c9ee0 0x7f584414d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f5843fff700 1 -- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5844152ba0 msgr2=0x7f584414daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f5843fff700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5844152ba0 0x7f584414daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f5843fff700 1 -- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58440c9ee0 msgr2=0x7f584414d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f5843fff700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58440c9ee0 0x7f584414d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f5843fff700 1 -- 172.21.15.23:0/2104797235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c009a70 con 0x7f58440c9260 2023-06-22T07:59:50.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f5848d9c700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5844152ba0 0x7f584414daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T07:59:50.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.161+0000 7f58437fe700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58440c9ee0 0x7f584414d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T07:59:50.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.162+0000 7f5843fff700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58440c9260 0x7f58440ced70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f58340090b0 tx=0x7f5834004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:50.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.162+0000 7f5840ff9700 1 -- 172.21.15.23:0/2104797235 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58340038a0 con 0x7f58440c9260 2023-06-22T07:59:50.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.162+0000 7f5840ff9700 1 -- 172.21.15.23:0/2104797235 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583401e070 con 0x7f58440c9260 2023-06-22T07:59:50.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.162+0000 7f584a7ff700 1 -- 172.21.15.23:0/2104797235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58340094a0 con 0x7f58440c9260 2023-06-22T07:59:50.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.162+0000 7f5840ff9700 1 -- 172.21.15.23:0/2104797235 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5834003c20 con 0x7f58440c9260 2023-06-22T07:59:50.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.162+0000 7f584a7ff700 1 -- 172.21.15.23:0/2104797235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58441ef650 con 0x7f58440c9260 2023-06-22T07:59:50.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.163+0000 7f584a7ff700 1 -- 172.21.15.23:0/2104797235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58440cbed0 con 0x7f58440c9260 2023-06-22T07:59:50.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.164+0000 7f5840ff9700 1 -- 172.21.15.23:0/2104797235 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5834024400 con 0x7f58440c9260 2023-06-22T07:59:50.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.165+0000 7f5840ff9700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f582406ad10 0x7f582406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.165+0000 7f5840ff9700 1 -- 172.21.15.23:0/2104797235 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f583408a1a0 con 0x7f58440c9260 2023-06-22T07:59:50.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.165+0000 7f58437fe700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f582406ad10 0x7f582406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.166+0000 7f58437fe700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f582406ad10 0x7f582406d1c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f582c007fd0 tx=0x7f582c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:50.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.170+0000 7f5840ff9700 1 -- 172.21.15.23:0/2104797235 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5834055590 con 0x7f58440c9260 2023-06-22T07:59:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.374+0000 7f584a7ff700 1 -- 172.21.15.23:0/2104797235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f584406b460 con 0x7f58440c9260 2023-06-22T07:59:50.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.375+0000 7f5840ff9700 1 -- 172.21.15.23:0/2104797235 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5834058bb0 con 0x7f58440c9260 2023-06-22T07:59:50.378 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-06-22T07:59:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 -- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f582406ad10 msgr2=0x7f582406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f582406ad10 0x7f582406d1c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f582c007fd0 tx=0x7f582c009040 comp rx=0 tx=0).stop 2023-06-22T07:59:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 -- 172.21.15.23:0/2104797235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58440c9260 msgr2=0x7f58440ced70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58440c9260 0x7f58440ced70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f58340090b0 tx=0x7f5834004750 comp rx=0 tx=0).stop 2023-06-22T07:59:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 -- 172.21.15.23:0/2104797235 shutdown_connections 2023-06-22T07:59:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5844152ba0 0x7f584414daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58440c9260 0x7f58440ced70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f582406ad10 0x7f582406d1c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 --2- 172.21.15.23:0/2104797235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f58440c9ee0 0x7f584414d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 -- 172.21.15.23:0/2104797235 >> 172.21.15.23:0/2104797235 conn(0x7f5844065b90 msgr2=0x7f58440cb560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 -- 172.21.15.23:0/2104797235 shutdown_connections 2023-06-22T07:59:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.377+0000 7f584a7ff700 1 -- 172.21.15.23:0/2104797235 wait complete. 2023-06-22T07:59:50.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T07:59:50.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.600+0000 7fc8febff700 1 Processor -- start 2023-06-22T07:59:50.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.600+0000 7fc8febff700 1 -- start start 2023-06-22T07:59:50.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8febff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc8f806aac0 0x7fc8f80c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8febff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8f815bae0 0x7fc8f815bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8febff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f80717f0 con 0x7fc8f81524f0 2023-06-22T07:59:50.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8febff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f8071d70 con 0x7fc8f806aac0 2023-06-22T07:59:50.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8febff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f8069eb0 con 0x7fc8f815bae0 2023-06-22T07:59:50.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8f7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc8f806aac0 0x7fc8f80c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8f7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc8f806aac0 0x7fc8f80c8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44194/0 (socket says 172.21.15.23:44194) 2023-06-22T07:59:50.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8f7fff700 1 -- 172.21.15.23:0/347987194 learned_addr learned my addr 172.21.15.23:0/347987194 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:50.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8fd19c700 1 --2- 172.21.15.23:0/347987194 >> v2:172.21.15.67:3300/0 conn(0x7fc8f815bae0 0x7fc8f815bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8f7fff700 1 -- 172.21.15.23:0/347987194 >> v2:172.21.15.67:3300/0 conn(0x7fc8f815bae0 msgr2=0x7fc8f815bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:50.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8f7fff700 1 --2- 172.21.15.23:0/347987194 >> v2:172.21.15.67:3300/0 conn(0x7fc8f815bae0 0x7fc8f815bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8f7fff700 1 -- 172.21.15.23:0/347987194 >> v1:172.21.15.23:6789/0 conn(0x7fc8f81524f0 legacy=0x7fc8f806a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.602+0000 7fc8f7fff700 1 -- 172.21.15.23:0/347987194 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f8071a90 con 0x7fc8f806aac0 2023-06-22T07:59:50.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.603+0000 7fc8f7fff700 1 --2- 172.21.15.23:0/347987194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f806aac0 0x7fc8f80c8aa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8f806de50 tx=0x7fc8e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d391aec8b4b2a44 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:50.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.603+0000 7fc8f77fe700 1 -- 172.21.15.23:0/347987194 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4011830 con 0x7fc8f806aac0 2023-06-22T07:59:50.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.603+0000 7fc8f77fe700 1 -- 172.21.15.23:0/347987194 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e4011990 con 0x7fc8f806aac0 2023-06-22T07:59:50.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.603+0000 7fc8f77fe700 1 -- 172.21.15.23:0/347987194 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4011d20 con 0x7fc8f806aac0 2023-06-22T07:59:50.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.603+0000 7fc8febff700 1 -- 172.21.15.23:0/347987194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f806aac0 msgr2=0x7fc8f80c8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.603+0000 7fc8febff700 1 --2- 172.21.15.23:0/347987194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f806aac0 0x7fc8f80c8aa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc8f806de50 tx=0x7fc8e4009bd0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.604+0000 7fc8febff700 1 -- 172.21.15.23:0/347987194 shutdown_connections 2023-06-22T07:59:50.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.604+0000 7fc8febff700 1 --2- 172.21.15.23:0/347987194 >> v2:172.21.15.67:3300/0 conn(0x7fc8f815bae0 0x7fc8f815bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.604+0000 7fc8febff700 1 --2- 172.21.15.23:0/347987194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f806aac0 0x7fc8f80c8aa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.604+0000 7fc8febff700 1 -- 172.21.15.23:0/347987194 >> 172.21.15.23:0/347987194 conn(0x7fc8f806e9f0 msgr2=0x7fc8f806edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:50.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.604+0000 7fc8febff700 1 -- 172.21.15.23:0/347987194 shutdown_connections 2023-06-22T07:59:50.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.604+0000 7fc8febff700 1 -- 172.21.15.23:0/347987194 wait complete. 2023-06-22T07:59:50.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.605+0000 7fc8febff700 1 Processor -- start 2023-06-22T07:59:50.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.605+0000 7fc8febff700 1 -- start start 2023-06-22T07:59:50.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.606+0000 7fc8febff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f806aac0 0x7fc8f80c3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.606+0000 7fc8febff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f81524f0 0x7fc8f80c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.606+0000 7fc8febff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f815bae0 0x7fc8f8144d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.606+0000 7fc8febff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f80c55e0 con 0x7fc8f81524f0 2023-06-22T07:59:50.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.606+0000 7fc8febff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f80c2de0 con 0x7fc8f806aac0 2023-06-22T07:59:50.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.606+0000 7fc8febff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f80c30c0 con 0x7fc8f815bae0 2023-06-22T07:59:50.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8f7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f81524f0 0x7fc8f80c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8f7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f81524f0 0x7fc8f80c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44210/0 (socket says 172.21.15.23:44210) 2023-06-22T07:59:50.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8f7fff700 1 -- 172.21.15.23:0/3877473218 learned_addr learned my addr 172.21.15.23:0/3877473218 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:50.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8fc99b700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f806aac0 0x7fc8f80c3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8fd19c700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f815bae0 0x7fc8f8144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8f7fff700 1 -- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f815bae0 msgr2=0x7fc8f8144d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8f7fff700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f815bae0 0x7fc8f8144d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8f7fff700 1 -- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f806aac0 msgr2=0x7fc8f80c3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8f7fff700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f806aac0 0x7fc8f80c3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8f7fff700 1 -- 172.21.15.23:0/3877473218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4009550 con 0x7fc8f81524f0 2023-06-22T07:59:50.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.607+0000 7fc8f7fff700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f81524f0 0x7fc8f80c43c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4000c00 tx=0x7fc8e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:50.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.608+0000 7fc8f57fa700 1 -- 172.21.15.23:0/3877473218 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4004380 con 0x7fc8f81524f0 2023-06-22T07:59:50.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.608+0000 7fc8f57fa700 1 -- 172.21.15.23:0/3877473218 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e401f070 con 0x7fc8f81524f0 2023-06-22T07:59:50.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.608+0000 7fc8f57fa700 1 -- 172.21.15.23:0/3877473218 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e40036a0 con 0x7fc8f81524f0 2023-06-22T07:59:50.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.608+0000 7fc8febff700 1 -- 172.21.15.23:0/3877473218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f80c33a0 con 0x7fc8f81524f0 2023-06-22T07:59:50.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.608+0000 7fc8febff700 1 -- 172.21.15.23:0/3877473218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f80c3960 con 0x7fc8f81524f0 2023-06-22T07:59:50.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.609+0000 7fc8febff700 1 -- 172.21.15.23:0/3877473218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f804ea20 con 0x7fc8f81524f0 2023-06-22T07:59:50.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.610+0000 7fc8f57fa700 1 -- 172.21.15.23:0/3877473218 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc8e4021030 con 0x7fc8f81524f0 2023-06-22T07:59:50.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.612+0000 7fc8f57fa700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8dc06ace0 0x7fc8dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:50.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.612+0000 7fc8f57fa700 1 -- 172.21.15.23:0/3877473218 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc8e408a450 con 0x7fc8f81524f0 2023-06-22T07:59:50.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.612+0000 7fc8fc99b700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8dc06ace0 0x7fc8dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:50.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.613+0000 7fc8fc99b700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8dc06ace0 0x7fc8dc06d190 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0012e0 tx=0x7fc8ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:50.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.616+0000 7fc8f57fa700 1 -- 172.21.15.23:0/3877473218 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc8e4055860 con 0x7fc8f81524f0 2023-06-22T07:59:50.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.825+0000 7fc8febff700 1 -- 172.21.15.23:0/3877473218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc8f80c2690 con 0x7fc8f81524f0 2023-06-22T07:59:50.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.826+0000 7fc8f57fa700 1 -- 172.21.15.23:0/3877473218 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc8e4058e80 con 0x7fc8f81524f0 2023-06-22T07:59:50.829 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2023-06-22T07:59:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 -- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8dc06ace0 msgr2=0x7fc8dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8dc06ace0 0x7fc8dc06d190 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0012e0 tx=0x7fc8ec009040 comp rx=0 tx=0).stop 2023-06-22T07:59:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 -- 172.21.15.23:0/3877473218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f81524f0 msgr2=0x7fc8f80c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f81524f0 0x7fc8f80c43c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4000c00 tx=0x7fc8e4004750 comp rx=0 tx=0).stop 2023-06-22T07:59:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 -- 172.21.15.23:0/3877473218 shutdown_connections 2023-06-22T07:59:50.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f815bae0 0x7fc8f8144d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8dc06ace0 0x7fc8dc06d190 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f806aac0 0x7fc8f80c3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 --2- 172.21.15.23:0/3877473218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f81524f0 0x7fc8f80c43c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:50.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 -- 172.21.15.23:0/3877473218 >> 172.21.15.23:0/3877473218 conn(0x7fc8f806e9f0 msgr2=0x7fc8f814fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:50.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 -- 172.21.15.23:0/3877473218 shutdown_connections 2023-06-22T07:59:50.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:50.828+0000 7fc8febff700 1 -- 172.21.15.23:0/3877473218 wait complete. 2023-06-22T07:59:50.849 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T07:59:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.042+0000 7f2bb5bff700 1 Processor -- start 2023-06-22T07:59:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.042+0000 7f2bb5bff700 1 -- start start 2023-06-22T07:59:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.043+0000 7f2bb5bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bb0152ce0 0x7f2bb00cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.043+0000 7f2bb5bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2bb00c9fd0 0x7f2bb015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.043+0000 7f2bb5bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2bb0158880 0x7f2bb0158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.043+0000 7f2bb5bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb01530b0 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2bb5bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb00cf6f0 con 0x7f2bb0158880 2023-06-22T07:59:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2bb5bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb015c370 con 0x7f2bb0152ce0 2023-06-22T07:59:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2baeffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2bb00c9fd0 0x7f2bb015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2baeffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2bb00c9fd0 0x7f2bb015c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44220/0 (socket says 172.21.15.23:44220) 2023-06-22T07:59:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2baeffd700 1 -- 172.21.15.23:0/2572109812 learned_addr learned my addr 172.21.15.23:0/2572109812 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2baf7fe700 1 --2- 172.21.15.23:0/2572109812 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bb0152ce0 0x7f2bb00cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2baffff700 1 --2- 172.21.15.23:0/2572109812 >> v2:172.21.15.67:3300/0 conn(0x7f2bb0158880 0x7f2bb0158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2baeffd700 1 -- 172.21.15.23:0/2572109812 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bb0152ce0 msgr2=0x7f2bb00cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2baeffd700 1 --2- 172.21.15.23:0/2572109812 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bb0152ce0 0x7f2bb00cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2baeffd700 1 -- 172.21.15.23:0/2572109812 >> v2:172.21.15.67:3300/0 conn(0x7f2bb0158880 msgr2=0x7f2bb0158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2baeffd700 1 --2- 172.21.15.23:0/2572109812 >> v2:172.21.15.67:3300/0 conn(0x7f2bb0158880 0x7f2bb0158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.044+0000 7f2baeffd700 1 -- 172.21.15.23:0/2572109812 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb015bda0 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2baeffd700 1 --2- 172.21.15.23:0/2572109812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb00c9fd0 0x7f2bb015c800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2bb008b9e0 tx=0x7f2ba4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c83390fe42d561 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bae7fc700 1 -- 172.21.15.23:0/2572109812 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4011830 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bae7fc700 1 -- 172.21.15.23:0/2572109812 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4011990 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bae7fc700 1 -- 172.21.15.23:0/2572109812 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4011d20 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2572109812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb00c9fd0 msgr2=0x7f2bb015c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bb5bff700 1 --2- 172.21.15.23:0/2572109812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb00c9fd0 0x7f2bb015c800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2bb008b9e0 tx=0x7f2ba4009bd0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2572109812 shutdown_connections 2023-06-22T07:59:51.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bb5bff700 1 --2- 172.21.15.23:0/2572109812 >> v2:172.21.15.67:3300/0 conn(0x7f2bb0158880 0x7f2bb0158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bb5bff700 1 --2- 172.21.15.23:0/2572109812 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bb0152ce0 0x7f2bb00cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bb5bff700 1 --2- 172.21.15.23:0/2572109812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb00c9fd0 0x7f2bb015c800 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2572109812 >> 172.21.15.23:0/2572109812 conn(0x7f2bb006ae30 msgr2=0x7f2bb006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:51.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.045+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2572109812 shutdown_connections 2023-06-22T07:59:51.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.046+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2572109812 wait complete. 2023-06-22T07:59:51.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.048+0000 7f2bb5bff700 1 Processor -- start 2023-06-22T07:59:51.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.048+0000 7f2bb5bff700 1 -- start start 2023-06-22T07:59:51.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.049+0000 7f2bb5bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bb00c9fd0 0x7f2bb01ee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.049+0000 7f2bb5bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bb0152ce0 0x7f2bb014e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.049+0000 7f2bb5bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0158880 0x7f2bb014a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.049+0000 7f2bb5bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb0157100 con 0x7f2bb0158880 2023-06-22T07:59:51.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.049+0000 7f2bb5bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb01f1410 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.049+0000 7f2bb5bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb01f2270 con 0x7f2bb0152ce0 2023-06-22T07:59:51.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bb00c9fd0 0x7f2bb01ee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bb00c9fd0 0x7f2bb01ee8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:41540/0 (socket says 172.21.15.23:41540) 2023-06-22T07:59:51.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baf7fe700 1 -- 172.21.15.23:0/2041076639 learned_addr learned my addr 172.21.15.23:0/2041076639 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:51.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baeffd700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bb0152ce0 0x7f2bb014e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baffff700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0158880 0x7f2bb014a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baf7fe700 1 -- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bb0152ce0 msgr2=0x7f2bb014e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baf7fe700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bb0152ce0 0x7f2bb014e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baf7fe700 1 -- 172.21.15.23:0/2041076639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0158880 msgr2=0x7f2bb014a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baf7fe700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0158880 0x7f2bb014a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baf7fe700 1 -- 172.21.15.23:0/2041076639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba4009550 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.050+0000 7f2baf7fe700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bb00c9fd0 0x7f2bb01ee8b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000e9e0 tx=0x7f2ba000ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:51.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.051+0000 7f2b97fff700 1 -- 172.21.15.23:0/2041076639 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0009710 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.051+0000 7f2b97fff700 1 -- 172.21.15.23:0/2041076639 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba000ee80 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.051+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2041076639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb01eedf0 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.051+0000 7f2b97fff700 1 -- 172.21.15.23:0/2041076639 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba000ca50 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.051+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2041076639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb01f1940 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.052+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2041076639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb004ea20 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.053+0000 7f2b97fff700 1 -- 172.21.15.23:0/2041076639 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2ba000cbb0 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.054+0000 7f2b97fff700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b9c06ace0 0x7f2b9c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.054+0000 7f2b97fff700 1 -- 172.21.15.23:0/2041076639 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2ba0089920 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.054+0000 7f2baeffd700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b9c06ace0 0x7f2b9c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.055+0000 7f2baeffd700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b9c06ace0 0x7f2b9c06d190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009c10 tx=0x7f2ba4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:51.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.058+0000 7f2b97fff700 1 -- 172.21.15.23:0/2041076639 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2ba0058790 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.309+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2041076639 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bb00c9db0 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.310+0000 7f2b97fff700 1 -- 172.21.15.23:0/2041076639 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5033 (secure 0 0 0) 0x7f2ba0058320 con 0x7f2bb00c9fd0 2023-06-22T07:59:51.313 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:59:51.313 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4268,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4285,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-06-22T07:59:51.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b9c06ace0 msgr2=0x7f2b9c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b9c06ace0 0x7f2b9c06d190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009c10 tx=0x7f2ba4008030 comp rx=0 tx=0).stop 2023-06-22T07:59:51.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bb00c9fd0 msgr2=0x7f2bb01ee8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bb00c9fd0 0x7f2bb01ee8b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000e9e0 tx=0x7f2ba000ecf0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2041076639 shutdown_connections 2023-06-22T07:59:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bb0158880 0x7f2bb014a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b9c06ace0 0x7f2b9c06d190 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bb00c9fd0 0x7f2bb01ee8b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 --2- 172.21.15.23:0/2041076639 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bb0152ce0 0x7f2bb014e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2041076639 >> 172.21.15.23:0/2041076639 conn(0x7f2bb006ae30 msgr2=0x7f2bb01566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:51.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2041076639 shutdown_connections 2023-06-22T07:59:51.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.312+0000 7f2bb5bff700 1 -- 172.21.15.23:0/2041076639 wait complete. 2023-06-22T07:59:51.317 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2023-06-22T07:59:51.333 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T07:59:51.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.529+0000 7f40fb5ff700 1 Processor -- start 2023-06-22T07:59:51.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.529+0000 7f40fb5ff700 1 -- start start 2023-06-22T07:59:51.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.530+0000 7f40fb5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40f4152a10 0x7f40f40c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.530+0000 7f40fb5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 0x7f40f40cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.530+0000 7f40fb5ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f4159730 con 0x7f40f40cb7a0 2023-06-22T07:59:51.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.530+0000 7f40fb5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f40c5470 con 0x7f40f4152a10 2023-06-22T07:59:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.530+0000 7f40fb5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f40cbbd0 con 0x7f40f40c9200 2023-06-22T07:59:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.530+0000 7f40f9b9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f40f40cb7a0 0x7f40f415c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:42370/0 (socket says 172.21.15.23:42370) 2023-06-22T07:59:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.530+0000 7f40f9b9c700 1 -- 172.21.15.23:0/2353308930 learned_addr learned my addr 172.21.15.23:0/2353308930 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T07:59:51.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.530+0000 7f40f939b700 1 --2- 172.21.15.23:0/2353308930 >> v2:172.21.15.67:3300/0 conn(0x7f40f4152a10 0x7f40f40c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.530+0000 7f40f8b9a700 1 --2- 172.21.15.23:0/2353308930 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 0x7f40f40cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40ebfff700 1 -- 172.21.15.23:0/2353308930 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52167491 0 0) 0x7f40f4159730 con 0x7f40f40cb7a0 2023-06-22T07:59:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40ebfff700 1 -- 172.21.15.23:0/2353308930 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e00035b0 con 0x7f40f40cb7a0 2023-06-22T07:59:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40f939b700 1 -- 172.21.15.23:0/2353308930 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 msgr2=0x7f40f40cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40f939b700 1 --2- 172.21.15.23:0/2353308930 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 0x7f40f40cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40f939b700 1 -- 172.21.15.23:0/2353308930 >> v1:172.21.15.23:6789/0 conn(0x7f40f40cb7a0 legacy=0x7f40f415c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40f939b700 1 -- 172.21.15.23:0/2353308930 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4158910 con 0x7f40f4152a10 2023-06-22T07:59:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40f8b9a700 1 --2- 172.21.15.23:0/2353308930 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 0x7f40f40cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40f939b700 1 --2- 172.21.15.23:0/2353308930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f4152a10 0x7f40f40c4a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009840 tx=0x7f40f0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a6b2a4c19a0d35f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40ebfff700 1 -- 172.21.15.23:0/2353308930 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0011830 con 0x7f40f4152a10 2023-06-22T07:59:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40ebfff700 1 -- 172.21.15.23:0/2353308930 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f0011990 con 0x7f40f4152a10 2023-06-22T07:59:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.531+0000 7f40ebfff700 1 -- 172.21.15.23:0/2353308930 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0011d20 con 0x7f40f4152a10 2023-06-22T07:59:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.532+0000 7f40fb5ff700 1 -- 172.21.15.23:0/2353308930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f4152a10 msgr2=0x7f40f40c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.532+0000 7f40fb5ff700 1 --2- 172.21.15.23:0/2353308930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f4152a10 0x7f40f40c4a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009840 tx=0x7f40f0009b50 comp rx=0 tx=0).stop 2023-06-22T07:59:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.532+0000 7f40fb5ff700 1 -- 172.21.15.23:0/2353308930 shutdown_connections 2023-06-22T07:59:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.532+0000 7f40fb5ff700 1 --2- 172.21.15.23:0/2353308930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f4152a10 0x7f40f40c4a30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.532+0000 7f40fb5ff700 1 --2- 172.21.15.23:0/2353308930 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 0x7f40f40cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.532+0000 7f40fb5ff700 1 -- 172.21.15.23:0/2353308930 >> 172.21.15.23:0/2353308930 conn(0x7f40f4065b90 msgr2=0x7f40f4065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:51.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.532+0000 7f40fb5ff700 1 -- 172.21.15.23:0/2353308930 shutdown_connections 2023-06-22T07:59:51.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.532+0000 7f40fb5ff700 1 -- 172.21.15.23:0/2353308930 wait complete. 2023-06-22T07:59:51.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.534+0000 7f40fb5ff700 1 Processor -- start 2023-06-22T07:59:51.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.534+0000 7f40fb5ff700 1 -- start start 2023-06-22T07:59:51.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40fb5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 0x7f40f4153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40fb5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f40cb7a0 0x7f40f40b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40fb5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40f4152a10 0x7f40f414ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40fb5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f40c5eb0 con 0x7f40f4152a10 2023-06-22T07:59:51.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40fb5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f4154300 con 0x7f40f40cb7a0 2023-06-22T07:59:51.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40fb5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f40ccf10 con 0x7f40f40c9200 2023-06-22T07:59:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40f9b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40f4152a10 0x7f40f414ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40f9b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40f4152a10 0x7f40f414ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44224/0 (socket says 172.21.15.23:44224) 2023-06-22T07:59:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40f9b9c700 1 -- 172.21.15.23:0/4268900376 learned_addr learned my addr 172.21.15.23:0/4268900376 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40f8b9a700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f40cb7a0 0x7f40f40b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.535+0000 7f40f939b700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 0x7f40f4153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40f9b9c700 1 -- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 msgr2=0x7f40f4153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40f9b9c700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 0x7f40f4153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40f9b9c700 1 -- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f40cb7a0 msgr2=0x7f40f40b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40f9b9c700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f40cb7a0 0x7f40f40b36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40f9b9c700 1 -- 172.21.15.23:0/4268900376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e40094a0 con 0x7f40f4152a10 2023-06-22T07:59:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40f9b9c700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40f4152a10 0x7f40f414ec20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00cb40 tx=0x7f40ec00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40e9ffb700 1 -- 172.21.15.23:0/4268900376 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec00a9e0 con 0x7f40f4152a10 2023-06-22T07:59:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40e9ffb700 1 -- 172.21.15.23:0/4268900376 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec00ab40 con 0x7f40f4152a10 2023-06-22T07:59:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40fb5ff700 1 -- 172.21.15.23:0/4268900376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f00094a0 con 0x7f40f4152a10 2023-06-22T07:59:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40e9ffb700 1 -- 172.21.15.23:0/4268900376 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec012400 con 0x7f40f4152a10 2023-06-22T07:59:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.536+0000 7f40fb5ff700 1 -- 172.21.15.23:0/4268900376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f41550d0 con 0x7f40f4152a10 2023-06-22T07:59:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.537+0000 7f40fb5ff700 1 -- 172.21.15.23:0/4268900376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f4073550 con 0x7f40f4152a10 2023-06-22T07:59:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.538+0000 7f40e9ffb700 1 -- 172.21.15.23:0/4268900376 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f40ec01b3b0 con 0x7f40f4152a10 2023-06-22T07:59:51.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.539+0000 7f40e9ffb700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f40d806ace0 0x7f40d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.539+0000 7f40e9ffb700 1 -- 172.21.15.23:0/4268900376 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f40ec088d20 con 0x7f40f4152a10 2023-06-22T07:59:51.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.539+0000 7f40f939b700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f40d806ace0 0x7f40d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.540+0000 7f40f939b700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f40d806ace0 0x7f40d806d190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40f0000c00 tx=0x7f40f001c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:51.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.543+0000 7f40e9ffb700 1 -- 172.21.15.23:0/4268900376 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f40ec054130 con 0x7f40f4152a10 2023-06-22T07:59:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.755+0000 7f40fb5ff700 1 -- 172.21.15.23:0/4268900376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40f404ea20 con 0x7f40f4152a10 2023-06-22T07:59:51.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.756+0000 7f40e9ffb700 1 -- 172.21.15.23:0/4268900376 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+14590 (secure 0 0 0) 0x7f40ec057750 con 0x7f40f4152a10 2023-06-22T07:59:51.759 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:59:51.760 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":20,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:50.446099+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T07:59:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 -- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f40d806ace0 msgr2=0x7f40d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f40d806ace0 0x7f40d806d190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40f0000c00 tx=0x7f40f001c040 comp rx=0 tx=0).stop 2023-06-22T07:59:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 -- 172.21.15.23:0/4268900376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40f4152a10 msgr2=0x7f40f414ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40f4152a10 0x7f40f414ec20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00cb40 tx=0x7f40ec00ce50 comp rx=0 tx=0).stop 2023-06-22T07:59:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 -- 172.21.15.23:0/4268900376 shutdown_connections 2023-06-22T07:59:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40f4152a10 0x7f40f414ec20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40f40c9200 0x7f40f4153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f40d806ace0 0x7f40d806d190 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 --2- 172.21.15.23:0/4268900376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40f40cb7a0 0x7f40f40b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 -- 172.21.15.23:0/4268900376 >> 172.21.15.23:0/4268900376 conn(0x7f40f4065b90 msgr2=0x7f40f4157fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 -- 172.21.15.23:0/4268900376 shutdown_connections 2023-06-22T07:59:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.759+0000 7f40fb5ff700 1 -- 172.21.15.23:0/4268900376 wait complete. 2023-06-22T07:59:51.779 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T07:59:40.617747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-22T07:59:48.190528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-06-22T07:59:51.782 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T07:59:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.980+0000 7f99791ff700 1 Processor -- start 2023-06-22T07:59:51.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.981+0000 7f99791ff700 1 -- start start 2023-06-22T07:59:51.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f99791ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f99741593f0 0x7f99740c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f99791ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f99740c4650 0x7f99740c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f99791ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99740c9aa0 con 0x7f9974152ba0 2023-06-22T07:59:51.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f99791ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f997406aed0 con 0x7f99740c4650 2023-06-22T07:59:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f99791ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f997406b1b0 con 0x7f99741593f0 2023-06-22T07:59:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f997359e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f99740c4650 0x7f99740c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f997359e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f99740c4650 0x7f99740c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44226/0 (socket says 172.21.15.23:44226) 2023-06-22T07:59:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f997359e700 1 -- 172.21.15.23:0/3369769317 learned_addr learned my addr 172.21.15.23:0/3369769317 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f997259c700 1 --2- 172.21.15.23:0/3369769317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f99741593f0 0x7f99740c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f997359e700 1 -- 172.21.15.23:0/3369769317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f99741593f0 msgr2=0x7f99740c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:51.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f997359e700 1 --2- 172.21.15.23:0/3369769317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f99741593f0 0x7f99740c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f997359e700 1 -- 172.21.15.23:0/3369769317 >> v1:172.21.15.67:6789/0 conn(0x7f9974152ba0 legacy=0x7f99740c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.982+0000 7f997359e700 1 -- 172.21.15.23:0/3369769317 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997406b490 con 0x7f99740c4650 2023-06-22T07:59:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.983+0000 7f997359e700 1 --2- 172.21.15.23:0/3369769317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 0x7f99740c9ee0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f996c009e50 tx=0x7f996c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd988e85df0a9467 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.983+0000 7f9971d9b700 1 -- 172.21.15.23:0/3369769317 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c00e070 con 0x7f99740c4650 2023-06-22T07:59:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.983+0000 7f9971d9b700 1 -- 172.21.15.23:0/3369769317 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996c00b910 con 0x7f99740c4650 2023-06-22T07:59:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.983+0000 7f9971d9b700 1 -- 172.21.15.23:0/3369769317 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c00bcc0 con 0x7f99740c4650 2023-06-22T07:59:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.983+0000 7f99791ff700 1 -- 172.21.15.23:0/3369769317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 msgr2=0x7f99740c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.983+0000 7f99791ff700 1 --2- 172.21.15.23:0/3369769317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 0x7f99740c9ee0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f996c009e50 tx=0x7f996c00b330 comp rx=0 tx=0).stop 2023-06-22T07:59:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.984+0000 7f99791ff700 1 -- 172.21.15.23:0/3369769317 shutdown_connections 2023-06-22T07:59:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.984+0000 7f99791ff700 1 --2- 172.21.15.23:0/3369769317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 0x7f99740c9ee0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.984+0000 7f99791ff700 1 --2- 172.21.15.23:0/3369769317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f99741593f0 0x7f99740c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.984+0000 7f99791ff700 1 -- 172.21.15.23:0/3369769317 >> 172.21.15.23:0/3369769317 conn(0x7f9974065b90 msgr2=0x7f9974065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.984+0000 7f99791ff700 1 -- 172.21.15.23:0/3369769317 shutdown_connections 2023-06-22T07:59:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.984+0000 7f99791ff700 1 -- 172.21.15.23:0/3369769317 wait complete. 2023-06-22T07:59:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.985+0000 7f99791ff700 1 Processor -- start 2023-06-22T07:59:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.986+0000 7f99791ff700 1 -- start start 2023-06-22T07:59:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.986+0000 7f99791ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 0x7f99741f0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f99791ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9974152ba0 0x7f997414d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f99791ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f99741593f0 0x7f997414dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f99791ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99740c8cc0 con 0x7f99740c4650 2023-06-22T07:59:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f99791ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997414e2d0 con 0x7f9974152ba0 2023-06-22T07:59:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f99791ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99741f0970 con 0x7f99741593f0 2023-06-22T07:59:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f9972d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 0x7f99741f0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f9972d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 0x7f99741f0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44230/0 (socket says 172.21.15.23:44230) 2023-06-22T07:59:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f9972d9d700 1 -- 172.21.15.23:0/292230893 learned_addr learned my addr 172.21.15.23:0/292230893 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f997259c700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9974152ba0 0x7f997414d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f997359e700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f99741593f0 0x7f997414dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f9972d9d700 1 -- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f99741593f0 msgr2=0x7f997414dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f9972d9d700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f99741593f0 0x7f997414dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f9972d9d700 1 -- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9974152ba0 msgr2=0x7f997414d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f9972d9d700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9974152ba0 0x7f997414d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.987+0000 7f9972d9d700 1 -- 172.21.15.23:0/292230893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c009b30 con 0x7f99740c4650 2023-06-22T07:59:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.988+0000 7f9972d9d700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 0x7f99741f0e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f996400c510 tx=0x7f996400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.988+0000 7f995b7fe700 1 -- 172.21.15.23:0/292230893 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99640052d0 con 0x7f99740c4650 2023-06-22T07:59:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.988+0000 7f995b7fe700 1 -- 172.21.15.23:0/292230893 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9964005430 con 0x7f99740c4650 2023-06-22T07:59:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.988+0000 7f995b7fe700 1 -- 172.21.15.23:0/292230893 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99640038c0 con 0x7f99740c4650 2023-06-22T07:59:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.988+0000 7f99791ff700 1 -- 172.21.15.23:0/292230893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99741f1340 con 0x7f99740c4650 2023-06-22T07:59:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.988+0000 7f99791ff700 1 -- 172.21.15.23:0/292230893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99741f0040 con 0x7f99740c4650 2023-06-22T07:59:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.989+0000 7f99791ff700 1 -- 172.21.15.23:0/292230893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99740cbed0 con 0x7f99740c4650 2023-06-22T07:59:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.990+0000 7f995b7fe700 1 -- 172.21.15.23:0/292230893 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9964003a20 con 0x7f99740c4650 2023-06-22T07:59:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.991+0000 7f995b7fe700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f996006ace0 0x7f996006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.991+0000 7f995b7fe700 1 -- 172.21.15.23:0/292230893 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f99640884d0 con 0x7f99740c4650 2023-06-22T07:59:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.991+0000 7f997259c700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f996006ace0 0x7f996006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.992+0000 7f997259c700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f996006ace0 0x7f996006d190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f99740cbe00 tx=0x7f9968009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:51.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:51.996+0000 7f995b7fe700 1 -- 172.21.15.23:0/292230893 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9964053830 con 0x7f99740c4650 2023-06-22T07:59:52.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.200+0000 7f99791ff700 1 -- 172.21.15.23:0/292230893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f997414fef0 con 0x7f99740c4650 2023-06-22T07:59:52.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.200+0000 7f995b7fe700 1 -- 172.21.15.23:0/292230893 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9964056e50 con 0x7f99740c4650 2023-06-22T07:59:52.203 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-06-22T07:59:52.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 -- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f996006ace0 msgr2=0x7f996006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f996006ace0 0x7f996006d190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f99740cbe00 tx=0x7f9968009210 comp rx=0 tx=0).stop 2023-06-22T07:59:52.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 -- 172.21.15.23:0/292230893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 msgr2=0x7f99741f0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 0x7f99741f0e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f996400c510 tx=0x7f996400c820 comp rx=0 tx=0).stop 2023-06-22T07:59:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 -- 172.21.15.23:0/292230893 shutdown_connections 2023-06-22T07:59:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f99741593f0 0x7f997414dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740c4650 0x7f99741f0e00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f996006ace0 0x7f996006d190 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 --2- 172.21.15.23:0/292230893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9974152ba0 0x7f997414d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 -- 172.21.15.23:0/292230893 >> 172.21.15.23:0/292230893 conn(0x7f9974065b90 msgr2=0x7f99740b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:52.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 -- 172.21.15.23:0/292230893 shutdown_connections 2023-06-22T07:59:52.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.202+0000 7f99791ff700 1 -- 172.21.15.23:0/292230893 wait complete. 2023-06-22T07:59:52.223 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T07:59:52.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.421+0000 7fefb8bff700 1 Processor -- start 2023-06-22T07:59:52.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.421+0000 7fefb8bff700 1 -- start start 2023-06-22T07:59:52.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb8bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fefb4152ba0 0x7fefb41593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb8bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefb40c9260 0x7fefb40c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb8bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fefb40c4650 0x7fefb40c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb8bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb4159930 con 0x7fefb40c4650 2023-06-22T07:59:52.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb8bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb40c9aa0 con 0x7fefb4152ba0 2023-06-22T07:59:52.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb8bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb406aed0 con 0x7fefb40c9260 2023-06-22T07:59:52.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb2d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fefb40c4650 0x7fefb40c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb2d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fefb40c4650 0x7fefb40c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44232/0 (socket says 172.21.15.23:44232) 2023-06-22T07:59:52.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb2d9d700 1 -- 172.21.15.23:0/2080389374 learned_addr learned my addr 172.21.15.23:0/2080389374 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:52.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb259c700 1 --2- 172.21.15.23:0/2080389374 >> v2:172.21.15.67:3300/0 conn(0x7fefb4152ba0 0x7fefb41593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb1d9b700 1 --2- 172.21.15.23:0/2080389374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefb40c9260 0x7fefb40c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb2d9d700 1 -- 172.21.15.23:0/2080389374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefb40c9260 msgr2=0x7fefb40c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:52.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb2d9d700 1 --2- 172.21.15.23:0/2080389374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefb40c9260 0x7fefb40c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb2d9d700 1 -- 172.21.15.23:0/2080389374 >> v2:172.21.15.67:3300/0 conn(0x7fefb4152ba0 msgr2=0x7fefb41593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:52.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb2d9d700 1 --2- 172.21.15.23:0/2080389374 >> v2:172.21.15.67:3300/0 conn(0x7fefb4152ba0 0x7fefb41593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.423+0000 7fefb2d9d700 1 -- 172.21.15.23:0/2080389374 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb406b1b0 con 0x7fefb40c4650 2023-06-22T07:59:52.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb2d9d700 1 --2- 172.21.15.23:0/2080389374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 0x7fefb40c9ee0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fefb408c670 tx=0x7fefac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f916801f0414b799 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:52.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb159a700 1 -- 172.21.15.23:0/2080389374 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac00e070 con 0x7fefb40c4650 2023-06-22T07:59:52.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb159a700 1 -- 172.21.15.23:0/2080389374 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac00b910 con 0x7fefb40c4650 2023-06-22T07:59:52.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb159a700 1 -- 172.21.15.23:0/2080389374 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac00bcc0 con 0x7fefb40c4650 2023-06-22T07:59:52.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb8bff700 1 -- 172.21.15.23:0/2080389374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 msgr2=0x7fefb40c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb8bff700 1 --2- 172.21.15.23:0/2080389374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 0x7fefb40c9ee0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fefb408c670 tx=0x7fefac00b330 comp rx=0 tx=0).stop 2023-06-22T07:59:52.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb8bff700 1 -- 172.21.15.23:0/2080389374 shutdown_connections 2023-06-22T07:59:52.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb8bff700 1 --2- 172.21.15.23:0/2080389374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 0x7fefb40c9ee0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb8bff700 1 --2- 172.21.15.23:0/2080389374 >> v2:172.21.15.67:3300/0 conn(0x7fefb4152ba0 0x7fefb41593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb8bff700 1 --2- 172.21.15.23:0/2080389374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefb40c9260 0x7fefb40c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.424+0000 7fefb8bff700 1 -- 172.21.15.23:0/2080389374 >> 172.21.15.23:0/2080389374 conn(0x7fefb4065b90 msgr2=0x7fefb4065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:52.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.425+0000 7fefb8bff700 1 -- 172.21.15.23:0/2080389374 shutdown_connections 2023-06-22T07:59:52.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.425+0000 7fefb8bff700 1 -- 172.21.15.23:0/2080389374 wait complete. 2023-06-22T07:59:52.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.426+0000 7fefb8bff700 1 Processor -- start 2023-06-22T07:59:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.426+0000 7fefb8bff700 1 -- start start 2023-06-22T07:59:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.427+0000 7fefb8bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 0x7fefb41ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.427+0000 7fefb8bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefb40c9260 0x7fefb40cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.427+0000 7fefb8bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefb4152ba0 0x7fefb414b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.427+0000 7fefb8bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb41f17a0 con 0x7fefb40c4650 2023-06-22T07:59:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.427+0000 7fefb8bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb4156c20 con 0x7fefb40c9260 2023-06-22T07:59:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.427+0000 7fefb8bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb4153670 con 0x7fefb4152ba0 2023-06-22T07:59:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 0x7fefb41ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 0x7fefb41ee230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44236/0 (socket says 172.21.15.23:44236) 2023-06-22T07:59:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb259c700 1 -- 172.21.15.23:0/1404280310 learned_addr learned my addr 172.21.15.23:0/1404280310 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb1d9b700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefb40c9260 0x7fefb40cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb2d9d700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefb4152ba0 0x7fefb414b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb259c700 1 -- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefb4152ba0 msgr2=0x7fefb414b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb259c700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefb4152ba0 0x7fefb414b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb259c700 1 -- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefb40c9260 msgr2=0x7fefb40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb259c700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefb40c9260 0x7fefb40cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb259c700 1 -- 172.21.15.23:0/1404280310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac009b30 con 0x7fefb40c4650 2023-06-22T07:59:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.428+0000 7fefb259c700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 0x7fefb41ee230 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fefa400e9b0 tx=0x7fefa400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.429+0000 7fefa2ffd700 1 -- 172.21.15.23:0/1404280310 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4009710 con 0x7fefb40c4650 2023-06-22T07:59:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.429+0000 7fefa2ffd700 1 -- 172.21.15.23:0/1404280310 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa400ee50 con 0x7fefb40c4650 2023-06-22T07:59:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.429+0000 7fefa2ffd700 1 -- 172.21.15.23:0/1404280310 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa400ca50 con 0x7fefb40c4650 2023-06-22T07:59:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.429+0000 7fefb8bff700 1 -- 172.21.15.23:0/1404280310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb4153950 con 0x7fefb40c4650 2023-06-22T07:59:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.429+0000 7fefb8bff700 1 -- 172.21.15.23:0/1404280310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb41f1c90 con 0x7fefb40c4650 2023-06-22T07:59:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.430+0000 7fefb8bff700 1 -- 172.21.15.23:0/1404280310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb40cbed0 con 0x7fefb40c4650 2023-06-22T07:59:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.431+0000 7fefa2ffd700 1 -- 172.21.15.23:0/1404280310 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fefa400cbb0 con 0x7fefb40c4650 2023-06-22T07:59:52.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.432+0000 7fefa2ffd700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef9406ace0 0x7fef9406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.432+0000 7fefa2ffd700 1 -- 172.21.15.23:0/1404280310 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fefa4089540 con 0x7fefb40c4650 2023-06-22T07:59:52.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.432+0000 7fefb1d9b700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef9406ace0 0x7fef9406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.433+0000 7fefb1d9b700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef9406ace0 0x7fef9406d190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fefb40c8c20 tx=0x7fefa8009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:52.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.436+0000 7fefa2ffd700 1 -- 172.21.15.23:0/1404280310 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fefa4058340 con 0x7fefb40c4650 2023-06-22T07:59:52.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.646+0000 7fefb8bff700 1 -- 172.21.15.23:0/1404280310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fefb404ea20 con 0x7fefb40c4650 2023-06-22T07:59:52.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.646+0000 7fefa2ffd700 1 -- 172.21.15.23:0/1404280310 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fefa40170a0 con 0x7fefb40c4650 2023-06-22T07:59:52.650 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2023-06-22T07:59:52.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 -- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef9406ace0 msgr2=0x7fef9406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef9406ace0 0x7fef9406d190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fefb40c8c20 tx=0x7fefa8009210 comp rx=0 tx=0).stop 2023-06-22T07:59:52.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 -- 172.21.15.23:0/1404280310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 msgr2=0x7fefb41ee230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 0x7fefb41ee230 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fefa400e9b0 tx=0x7fefa400ecc0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 -- 172.21.15.23:0/1404280310 shutdown_connections 2023-06-22T07:59:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefb4152ba0 0x7fefb414b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb40c4650 0x7fefb41ee230 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef9406ace0 0x7fef9406d190 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 --2- 172.21.15.23:0/1404280310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefb40c9260 0x7fefb40cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 -- 172.21.15.23:0/1404280310 >> 172.21.15.23:0/1404280310 conn(0x7fefb4065b90 msgr2=0x7fefb4156510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 -- 172.21.15.23:0/1404280310 shutdown_connections 2023-06-22T07:59:52.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.649+0000 7fefb8bff700 1 -- 172.21.15.23:0/1404280310 wait complete. 2023-06-22T07:59:52.670 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-22T07:59:52.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.865+0000 7fef23fff700 1 Processor -- start 2023-06-22T07:59:52.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.865+0000 7fef23fff700 1 -- start start 2023-06-22T07:59:52.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.866+0000 7fef23fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 0x7fef1c0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.866+0000 7fef23fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fef1c0c8bd0 0x7fef1c06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.866+0000 7fef23fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c0c7400 con 0x7fef1c06b500 2023-06-22T07:59:52.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.866+0000 7fef23fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef1c0c81c0 con 0x7fef1c0c8bd0 2023-06-22T07:59:52.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.866+0000 7fef23fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c0c84a0 con 0x7fef1c152dd0 2023-06-22T07:59:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.867+0000 7fef21d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 0x7fef1c0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.867+0000 7fef2159a700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fef1c0c8bd0 0x7fef1c06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.867+0000 7fef2259c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fef1c06b500 0x7fef1c0cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:38538/0 (socket says 172.21.15.23:38538) 2023-06-22T07:59:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.867+0000 7fef2259c700 1 -- 172.21.15.23:0/669064750 learned_addr learned my addr 172.21.15.23:0/669064750 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T07:59:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.867+0000 7fef2159a700 1 -- 172.21.15.23:0/669064750 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 msgr2=0x7fef1c0aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.867+0000 7fef2159a700 1 --2- 172.21.15.23:0/669064750 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 0x7fef1c0aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.867+0000 7fef2159a700 1 -- 172.21.15.23:0/669064750 >> v1:172.21.15.67:6789/0 conn(0x7fef1c06b500 legacy=0x7fef1c0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.867+0000 7fef2159a700 1 -- 172.21.15.23:0/669064750 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c0cd370 con 0x7fef1c0c8bd0 2023-06-22T07:59:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.867+0000 7fef21d9b700 1 --2- 172.21.15.23:0/669064750 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 0x7fef1c0aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.867+0000 7fef2159a700 1 --2- 172.21.15.23:0/669064750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef1c0c8bd0 0x7fef1c06afc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009840 tx=0x7fef0c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f600305d34d1a200 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.868+0000 7fef20d99700 1 -- 172.21.15.23:0/669064750 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c011830 con 0x7fef1c0c8bd0 2023-06-22T07:59:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.868+0000 7fef20d99700 1 -- 172.21.15.23:0/669064750 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c011990 con 0x7fef1c0c8bd0 2023-06-22T07:59:52.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.868+0000 7fef20d99700 1 -- 172.21.15.23:0/669064750 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c011d20 con 0x7fef1c0c8bd0 2023-06-22T07:59:52.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.868+0000 7fef23fff700 1 -- 172.21.15.23:0/669064750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef1c0c8bd0 msgr2=0x7fef1c06afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.868+0000 7fef23fff700 1 --2- 172.21.15.23:0/669064750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef1c0c8bd0 0x7fef1c06afc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009840 tx=0x7fef0c009b50 comp rx=0 tx=0).stop 2023-06-22T07:59:52.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.868+0000 7fef23fff700 1 -- 172.21.15.23:0/669064750 shutdown_connections 2023-06-22T07:59:52.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.868+0000 7fef23fff700 1 --2- 172.21.15.23:0/669064750 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 0x7fef1c0aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.868+0000 7fef23fff700 1 --2- 172.21.15.23:0/669064750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef1c0c8bd0 0x7fef1c06afc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.868+0000 7fef23fff700 1 -- 172.21.15.23:0/669064750 >> 172.21.15.23:0/669064750 conn(0x7fef1c06fb70 msgr2=0x7fef1c06ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:52.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.868+0000 7fef23fff700 1 -- 172.21.15.23:0/669064750 shutdown_connections 2023-06-22T07:59:52.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.869+0000 7fef23fff700 1 -- 172.21.15.23:0/669064750 wait complete. 2023-06-22T07:59:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.870+0000 7fef23fff700 1 Processor -- start 2023-06-22T07:59:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.870+0000 7fef23fff700 1 -- start start 2023-06-22T07:59:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef23fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef1c06b500 0x7fef1c15a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef23fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1c0c8bd0 0x7fef1c1ef400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef23fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 0x7fef1c1ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef23fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c1581c0 con 0x7fef1c0c8bd0 2023-06-22T07:59:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef23fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c150510 con 0x7fef1c06b500 2023-06-22T07:59:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef23fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c157140 con 0x7fef1c152dd0 2023-06-22T07:59:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef2159a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1c0c8bd0 0x7fef1c1ef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef21d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef1c06b500 0x7fef1c15a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef2159a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1c0c8bd0 0x7fef1c1ef400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44238/0 (socket says 172.21.15.23:44238) 2023-06-22T07:59:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef2259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 0x7fef1c1ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.871+0000 7fef2159a700 1 -- 172.21.15.23:0/509993088 learned_addr learned my addr 172.21.15.23:0/509993088 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef21d9b700 1 -- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 msgr2=0x7fef1c1ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef21d9b700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 0x7fef1c1ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef21d9b700 1 -- 172.21.15.23:0/509993088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1c0c8bd0 msgr2=0x7fef1c1ef400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef21d9b700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1c0c8bd0 0x7fef1c1ef400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef21d9b700 1 -- 172.21.15.23:0/509993088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef18009430 con 0x7fef1c06b500 2023-06-22T07:59:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef2159a700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1c0c8bd0 0x7fef1c1ef400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef21d9b700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef1c06b500 0x7fef1c15a740 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef1800e910 tx=0x7fef1800ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef127fc700 1 -- 172.21.15.23:0/509993088 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18005a20 con 0x7fef1c06b500 2023-06-22T07:59:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef127fc700 1 -- 172.21.15.23:0/509993088 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef18005b80 con 0x7fef1c06b500 2023-06-22T07:59:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef23fff700 1 -- 172.21.15.23:0/509993088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0c0094a0 con 0x7fef1c06b500 2023-06-22T07:59:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.872+0000 7fef127fc700 1 -- 172.21.15.23:0/509993088 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18013680 con 0x7fef1c06b500 2023-06-22T07:59:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.873+0000 7fef23fff700 1 -- 172.21.15.23:0/509993088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1c150760 con 0x7fef1c06b500 2023-06-22T07:59:52.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.873+0000 7fef23fff700 1 -- 172.21.15.23:0/509993088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1c04ea20 con 0x7fef1c06b500 2023-06-22T07:59:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.875+0000 7fef127fc700 1 -- 172.21.15.23:0/509993088 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fef180137e0 con 0x7fef1c06b500 2023-06-22T07:59:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.875+0000 7fef127fc700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef0006ace0 0x7fef0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.876+0000 7fef127fc700 1 -- 172.21.15.23:0/509993088 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fef18089480 con 0x7fef1c06b500 2023-06-22T07:59:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.876+0000 7fef2159a700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef0006ace0 0x7fef0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.876+0000 7fef2159a700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef0006ace0 0x7fef0006d190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fef1c1566e0 tx=0x7fef0c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:52.879+0000 7fef127fc700 1 -- 172.21.15.23:0/509993088 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fef18054850 con 0x7fef1c06b500 2023-06-22T07:59:53.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.091+0000 7fef23fff700 1 -- 172.21.15.23:0/509993088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fef1c0c3830 con 0x7fef1c06b500 2023-06-22T07:59:53.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.091+0000 7fef127fc700 1 -- 172.21.15.23:0/509993088 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v20) v1 ==== 84+0+126 (secure 0 0 0) 0x7fef18020020 con 0x7fef1c06b500 2023-06-22T07:59:53.094 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T07:59:53.095 INFO:teuthology.orchestra.run.smithi023.stdout:[ 2023-06-22T07:59:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-06-22T07:59:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: "poolnum": 1, 2023-06-22T07:59:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: "poolname": ".mgr" 2023-06-22T07:59:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-06-22T07:59:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: { 2023-06-22T07:59:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: "poolnum": 2, 2023-06-22T07:59:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: "poolname": "rbd" 2023-06-22T07:59:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-06-22T07:59:53.096 INFO:teuthology.orchestra.run.smithi023.stdout:] 2023-06-22T07:59:53.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 -- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef0006ace0 msgr2=0x7fef0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef0006ace0 0x7fef0006d190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fef1c1566e0 tx=0x7fef0c008030 comp rx=0 tx=0).stop 2023-06-22T07:59:53.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 -- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef1c06b500 msgr2=0x7fef1c15a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef1c06b500 0x7fef1c15a740 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef1800e910 tx=0x7fef1800ec20 comp rx=0 tx=0).stop 2023-06-22T07:59:53.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 -- 172.21.15.23:0/509993088 shutdown_connections 2023-06-22T07:59:53.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef1c152dd0 0x7fef1c1ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef0006ace0 0x7fef0006d190 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef1c06b500 0x7fef1c15a740 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 --2- 172.21.15.23:0/509993088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef1c0c8bd0 0x7fef1c1ef400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 -- 172.21.15.23:0/509993088 >> 172.21.15.23:0/509993088 conn(0x7fef1c06fb70 msgr2=0x7fef1c15b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:53.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 -- 172.21.15.23:0/509993088 shutdown_connections 2023-06-22T07:59:53.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.094+0000 7fef23fff700 1 -- 172.21.15.23:0/509993088 wait complete. 2023-06-22T07:59:53.114 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-22T07:59:53.115 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-22T07:59:53.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.314+0000 7efd9c1ff700 1 Processor -- start 2023-06-22T07:59:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.314+0000 7efd9c1ff700 1 -- start start 2023-06-22T07:59:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd9c1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd94152ba0 0x7efd941593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd9c1ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd94159930 con 0x7efd940c9ee0 2023-06-22T07:59:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd9c1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd940c9aa0 con 0x7efd940c9260 2023-06-22T07:59:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd9c1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9406aed0 con 0x7efd94152ba0 2023-06-22T07:59:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd9a79c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7efd940c9ee0 0x7efd940ca340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:42374/0 (socket says 172.21.15.23:42374) 2023-06-22T07:59:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd9a79c700 1 -- 172.21.15.23:0/3306271450 learned_addr learned my addr 172.21.15.23:0/3306271450 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T07:59:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd99f9b700 1 --2- 172.21.15.23:0/3306271450 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd94152ba0 0x7efd941593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd98f99700 1 -- 172.21.15.23:0/3306271450 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307504909 0 0) 0x7efd8c001490 con 0x7efd940c9ee0 2023-06-22T07:59:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd98f99700 1 -- 172.21.15.23:0/3306271450 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd800035b0 con 0x7efd940c9ee0 2023-06-22T07:59:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd98f99700 1 -- 172.21.15.23:0/3306271450 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678299449 0 0) 0x7efd940c9aa0 con 0x7efd940c9260 2023-06-22T07:59:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.316+0000 7efd98f99700 1 -- 172.21.15.23:0/3306271450 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd8c001490 con 0x7efd940c9260 2023-06-22T07:59:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.317+0000 7efd99f9b700 1 -- 172.21.15.23:0/3306271450 >> v1:172.21.15.67:6789/0 conn(0x7efd940c9260 legacy=0x7efd940c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.317+0000 7efd99f9b700 1 -- 172.21.15.23:0/3306271450 >> v1:172.21.15.23:6789/0 conn(0x7efd940c9ee0 legacy=0x7efd940ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.317+0000 7efd99f9b700 1 -- 172.21.15.23:0/3306271450 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9406b1b0 con 0x7efd94152ba0 2023-06-22T07:59:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.317+0000 7efd99f9b700 1 --2- 172.21.15.23:0/3306271450 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd94152ba0 0x7efd941593f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efd90009870 tx=0x7efd90009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff3df205fac4dfaf server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.317+0000 7efd98f99700 1 -- 172.21.15.23:0/3306271450 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90011830 con 0x7efd94152ba0 2023-06-22T07:59:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.317+0000 7efd98f99700 1 -- 172.21.15.23:0/3306271450 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90011990 con 0x7efd94152ba0 2023-06-22T07:59:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.317+0000 7efd98f99700 1 -- 172.21.15.23:0/3306271450 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90011d20 con 0x7efd94152ba0 2023-06-22T07:59:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.318+0000 7efd9c1ff700 1 -- 172.21.15.23:0/3306271450 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd94152ba0 msgr2=0x7efd941593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.318+0000 7efd9c1ff700 1 --2- 172.21.15.23:0/3306271450 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd94152ba0 0x7efd941593f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efd90009870 tx=0x7efd90009b80 comp rx=0 tx=0).stop 2023-06-22T07:59:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.318+0000 7efd9c1ff700 1 -- 172.21.15.23:0/3306271450 shutdown_connections 2023-06-22T07:59:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.318+0000 7efd9c1ff700 1 --2- 172.21.15.23:0/3306271450 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd94152ba0 0x7efd941593f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.318+0000 7efd9c1ff700 1 -- 172.21.15.23:0/3306271450 >> 172.21.15.23:0/3306271450 conn(0x7efd94065b90 msgr2=0x7efd94065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:53.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.318+0000 7efd9c1ff700 1 -- 172.21.15.23:0/3306271450 shutdown_connections 2023-06-22T07:59:53.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.318+0000 7efd9c1ff700 1 -- 172.21.15.23:0/3306271450 wait complete. 2023-06-22T07:59:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.319+0000 7efd9c1ff700 1 Processor -- start 2023-06-22T07:59:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.320+0000 7efd9c1ff700 1 -- start start 2023-06-22T07:59:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.320+0000 7efd9c1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd940c9260 0x7efd94155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.320+0000 7efd9c1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd940c9ee0 0x7efd940ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.320+0000 7efd9c1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd94152ba0 0x7efd940cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.320+0000 7efd9c1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd941547b0 con 0x7efd94152ba0 2023-06-22T07:59:53.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.320+0000 7efd9c1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd940ce300 con 0x7efd940c9260 2023-06-22T07:59:53.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.320+0000 7efd9c1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd940b47c0 con 0x7efd940c9ee0 2023-06-22T07:59:53.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd99f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd940c9260 0x7efd94155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd9a79c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd94152ba0 0x7efd940cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd99f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd940c9260 0x7efd94155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:41610/0 (socket says 172.21.15.23:41610) 2023-06-22T07:59:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd99f9b700 1 -- 172.21.15.23:0/1948647842 learned_addr learned my addr 172.21.15.23:0/1948647842 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd9979a700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd940c9ee0 0x7efd940ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd99f9b700 1 -- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd940c9ee0 msgr2=0x7efd940ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd99f9b700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd940c9ee0 0x7efd940ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd99f9b700 1 -- 172.21.15.23:0/1948647842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd94152ba0 msgr2=0x7efd940cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd99f9b700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd94152ba0 0x7efd940cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd99f9b700 1 -- 172.21.15.23:0/1948647842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd900094a0 con 0x7efd940c9260 2023-06-22T07:59:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd9a79c700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd94152ba0 0x7efd940cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.321+0000 7efd99f9b700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd940c9260 0x7efd94155ba0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efd90014040 tx=0x7efd90004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.322+0000 7efd8a7fc700 1 -- 172.21.15.23:0/1948647842 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd900044d0 con 0x7efd940c9260 2023-06-22T07:59:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.322+0000 7efd8a7fc700 1 -- 172.21.15.23:0/1948647842 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9001f070 con 0x7efd940c9260 2023-06-22T07:59:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.322+0000 7efd8a7fc700 1 -- 172.21.15.23:0/1948647842 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd900038e0 con 0x7efd940c9260 2023-06-22T07:59:53.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.322+0000 7efd9c1ff700 1 -- 172.21.15.23:0/1948647842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd941535f0 con 0x7efd940c9260 2023-06-22T07:59:53.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.322+0000 7efd9c1ff700 1 -- 172.21.15.23:0/1948647842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd94153bb0 con 0x7efd940c9260 2023-06-22T07:59:53.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.323+0000 7efd9c1ff700 1 -- 172.21.15.23:0/1948647842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd940cbed0 con 0x7efd940c9260 2023-06-22T07:59:53.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.324+0000 7efd8a7fc700 1 -- 172.21.15.23:0/1948647842 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efd90003a40 con 0x7efd940c9260 2023-06-22T07:59:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.325+0000 7efd8a7fc700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd7806ace0 0x7efd7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.325+0000 7efd8a7fc700 1 -- 172.21.15.23:0/1948647842 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efd90021070 con 0x7efd940c9260 2023-06-22T07:59:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.325+0000 7efd9979a700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd7806ace0 0x7efd7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.326+0000 7efd9979a700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd7806ace0 0x7efd7806d190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd941e4b00 tx=0x7efd84009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.330+0000 7efd8a7fc700 1 -- 172.21.15.23:0/1948647842 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efd90055c60 con 0x7efd940c9260 2023-06-22T07:59:53.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.535+0000 7efd9c1ff700 1 -- 172.21.15.23:0/1948647842 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7efd9404ea20 con 0x7efd940c9260 2023-06-22T07:59:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.612+0000 7efd8a7fc700 1 -- 172.21.15.23:0/1948647842 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v21) v1 ==== 138+0+0 (secure 0 0 0) 0x7efd90059280 con 0x7efd940c9260 2023-06-22T07:59:53.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 -- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd7806ace0 msgr2=0x7efd7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd7806ace0 0x7efd7806d190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd941e4b00 tx=0x7efd84009040 comp rx=0 tx=0).stop 2023-06-22T07:59:53.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 -- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd940c9260 msgr2=0x7efd94155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd940c9260 0x7efd94155ba0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efd90014040 tx=0x7efd90004750 comp rx=0 tx=0).stop 2023-06-22T07:59:53.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 -- 172.21.15.23:0/1948647842 shutdown_connections 2023-06-22T07:59:53.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd94152ba0 0x7efd940cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd7806ace0 0x7efd7806d190 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd941e4b00 tx=0x7efd84009040 comp rx=0 tx=0).stop 2023-06-22T07:59:53.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd940c9260 0x7efd94155ba0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 --2- 172.21.15.23:0/1948647842 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd940c9ee0 0x7efd940ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 -- 172.21.15.23:0/1948647842 >> 172.21.15.23:0/1948647842 conn(0x7efd94065b90 msgr2=0x7efd94150a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:53.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 -- 172.21.15.23:0/1948647842 shutdown_connections 2023-06-22T07:59:53.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.616+0000 7efd9c1ff700 1 -- 172.21.15.23:0/1948647842 wait complete. 2023-06-22T07:59:53.622 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' created 2023-06-22T07:59:53.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-06-22T07:59:53.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.843+0000 7fafdd7ff700 1 Processor -- start 2023-06-22T07:59:53.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.843+0000 7fafdd7ff700 1 -- start start 2023-06-22T07:59:53.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.844+0000 7fafdd7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fafd8152c60 0x7fafd80c9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.844+0000 7fafdd7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80c9760 0x7fafd80c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.844+0000 7fafdd7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd81597e0 con 0x7fafd80ca370 2023-06-22T07:59:53.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.844+0000 7fafdd7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd8153030 con 0x7fafd8152c60 2023-06-22T07:59:53.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.844+0000 7fafdd7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8158c60 con 0x7fafd80c9760 2023-06-22T07:59:53.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.844+0000 7fafd6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fafd8152c60 0x7fafd80c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.844+0000 7fafd67fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80c9760 0x7fafd80c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.844+0000 7fafd77fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fafd80ca370 0x7fafd815c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:38546/0 (socket says 172.21.15.23:38546) 2023-06-22T07:59:53.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.844+0000 7fafd77fe700 1 -- 172.21.15.23:0/449076861 learned_addr learned my addr 172.21.15.23:0/449076861 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T07:59:53.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.845+0000 7fafd5ffb700 1 -- 172.21.15.23:0/449076861 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598079093 0 0) 0x7fafd81597e0 con 0x7fafd80ca370 2023-06-22T07:59:53.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.845+0000 7fafd5ffb700 1 -- 172.21.15.23:0/449076861 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc00035b0 con 0x7fafd80ca370 2023-06-22T07:59:53.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.845+0000 7fafd67fc700 1 -- 172.21.15.23:0/449076861 >> v2:172.21.15.23:3300/0 conn(0x7fafd8152c60 msgr2=0x7fafd80c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:53.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.845+0000 7fafd67fc700 1 --2- 172.21.15.23:0/449076861 >> v2:172.21.15.23:3300/0 conn(0x7fafd8152c60 0x7fafd80c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.845+0000 7fafd67fc700 1 -- 172.21.15.23:0/449076861 >> v1:172.21.15.67:6789/0 conn(0x7fafd80ca370 legacy=0x7fafd815c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.845+0000 7fafd67fc700 1 -- 172.21.15.23:0/449076861 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd8158f40 con 0x7fafd80c9760 2023-06-22T07:59:53.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.845+0000 7fafd67fc700 1 --2- 172.21.15.23:0/449076861 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80c9760 0x7fafd80c9e30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009840 tx=0x7fafcc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ade8c69762ebaddf server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:53.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.845+0000 7fafd5ffb700 1 -- 172.21.15.23:0/449076861 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc011830 con 0x7fafd80c9760 2023-06-22T07:59:53.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.845+0000 7fafd5ffb700 1 -- 172.21.15.23:0/449076861 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafcc011990 con 0x7fafd80c9760 2023-06-22T07:59:53.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.845+0000 7fafd5ffb700 1 -- 172.21.15.23:0/449076861 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc011d20 con 0x7fafd80c9760 2023-06-22T07:59:53.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.846+0000 7fafdd7ff700 1 -- 172.21.15.23:0/449076861 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80c9760 msgr2=0x7fafd80c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.846+0000 7fafdd7ff700 1 --2- 172.21.15.23:0/449076861 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80c9760 0x7fafd80c9e30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009840 tx=0x7fafcc009b50 comp rx=0 tx=0).stop 2023-06-22T07:59:53.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.846+0000 7fafdd7ff700 1 -- 172.21.15.23:0/449076861 shutdown_connections 2023-06-22T07:59:53.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.846+0000 7fafdd7ff700 1 --2- 172.21.15.23:0/449076861 >> v2:172.21.15.23:3300/0 conn(0x7fafd8152c60 0x7fafd80c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.846+0000 7fafdd7ff700 1 --2- 172.21.15.23:0/449076861 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80c9760 0x7fafd80c9e30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.846+0000 7fafdd7ff700 1 -- 172.21.15.23:0/449076861 >> 172.21.15.23:0/449076861 conn(0x7fafd8065b90 msgr2=0x7fafd8065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:53.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.846+0000 7fafdd7ff700 1 -- 172.21.15.23:0/449076861 shutdown_connections 2023-06-22T07:59:53.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.846+0000 7fafdd7ff700 1 -- 172.21.15.23:0/449076861 wait complete. 2023-06-22T07:59:53.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.848+0000 7fafdd7ff700 1 Processor -- start 2023-06-22T07:59:53.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.848+0000 7fafdd7ff700 1 -- start start 2023-06-22T07:59:53.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafdd7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafd80c9760 0x7fafd814a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafdd7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80ca370 0x7fafd814b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafdd7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd8152c60 0x7fafd814b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafdd7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd81eeed0 con 0x7fafd80c9760 2023-06-22T07:59:53.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafdd7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8151940 con 0x7fafd8152c60 2023-06-22T07:59:53.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafdd7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd80b4360 con 0x7fafd80ca370 2023-06-22T07:59:53.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafd6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafd80c9760 0x7fafd814a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafd6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafd80c9760 0x7fafd814a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44268/0 (socket says 172.21.15.23:44268) 2023-06-22T07:59:53.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafd6ffd700 1 -- 172.21.15.23:0/3725945358 learned_addr learned my addr 172.21.15.23:0/3725945358 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:53.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafd67fc700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80ca370 0x7fafd814b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafd77fe700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd8152c60 0x7fafd814b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.849+0000 7fafd6ffd700 1 -- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80ca370 msgr2=0x7fafd814b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.850+0000 7fafd6ffd700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80ca370 0x7fafd814b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.850+0000 7fafd6ffd700 1 -- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd8152c60 msgr2=0x7fafd814b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:53.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.850+0000 7fafd6ffd700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd8152c60 0x7fafd814b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:53.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.850+0000 7fafd6ffd700 1 -- 172.21.15.23:0/3725945358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc0094a0 con 0x7fafd80c9760 2023-06-22T07:59:53.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.850+0000 7fafd6ffd700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafd80c9760 0x7fafd814a840 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fafc800e9c0 tx=0x7fafc800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:53.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.850+0000 7fafbf7fe700 1 -- 172.21.15.23:0/3725945358 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc80096f0 con 0x7fafd80c9760 2023-06-22T07:59:53.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.850+0000 7fafbf7fe700 1 -- 172.21.15.23:0/3725945358 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc800ee60 con 0x7fafd80c9760 2023-06-22T07:59:53.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.850+0000 7fafdd7ff700 1 -- 172.21.15.23:0/3725945358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd81ee250 con 0x7fafd80c9760 2023-06-22T07:59:53.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.850+0000 7fafbf7fe700 1 -- 172.21.15.23:0/3725945358 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc800ca50 con 0x7fafd80c9760 2023-06-22T07:59:53.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.850+0000 7fafdd7ff700 1 -- 172.21.15.23:0/3725945358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd81ee810 con 0x7fafd80c9760 2023-06-22T07:59:53.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.851+0000 7fafdd7ff700 1 -- 172.21.15.23:0/3725945358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd814ad80 con 0x7fafd80c9760 2023-06-22T07:59:53.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.852+0000 7fafbf7fe700 1 -- 172.21.15.23:0/3725945358 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fafc800cbb0 con 0x7fafd80c9760 2023-06-22T07:59:53.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.853+0000 7fafbf7fe700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fafc406ad10 0x7fafc406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:53.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.853+0000 7fafbf7fe700 1 -- 172.21.15.23:0/3725945358 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5580+0+0 (secure 0 0 0) 0x7fafc8089550 con 0x7fafd80c9760 2023-06-22T07:59:53.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.854+0000 7fafd67fc700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fafc406ad10 0x7fafc406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:53.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.854+0000 7fafd67fc700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fafc406ad10 0x7fafc406d1c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009840 tx=0x7fafcc01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:53.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:53.858+0000 7fafbf7fe700 1 -- 172.21.15.23:0/3725945358 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fafc8058270 con 0x7fafd80c9760 2023-06-22T07:59:54.067 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.065+0000 7fafdd7ff700 1 -- 172.21.15.23:0/3725945358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fafd80742e0 con 0x7fafd80c9760 2023-06-22T07:59:54.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.624+0000 7fafbf7fe700 1 -- 172.21.15.23:0/3725945358 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v22) v1 ==== 171+0+0 (secure 0 0 0) 0x7fafc8057e00 con 0x7fafd80c9760 2023-06-22T07:59:54.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.628+0000 7fafdd7ff700 1 -- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fafc406ad10 msgr2=0x7fafc406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:54.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.628+0000 7fafdd7ff700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fafc406ad10 0x7fafc406d1c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009840 tx=0x7fafcc01c040 comp rx=0 tx=0).stop 2023-06-22T07:59:54.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.628+0000 7fafdd7ff700 1 -- 172.21.15.23:0/3725945358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafd80c9760 msgr2=0x7fafd814a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:54.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.628+0000 7fafdd7ff700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafd80c9760 0x7fafd814a840 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fafc800e9c0 tx=0x7fafc800ecd0 comp rx=0 tx=0).stop 2023-06-22T07:59:54.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.629+0000 7fafdd7ff700 1 -- 172.21.15.23:0/3725945358 shutdown_connections 2023-06-22T07:59:54.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.629+0000 7fafdd7ff700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fafc406ad10 0x7fafc406d1c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:54.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.629+0000 7fafdd7ff700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd8152c60 0x7fafd814b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:54.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.629+0000 7fafdd7ff700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fafd80c9760 0x7fafd814a840 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:54.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.629+0000 7fafdd7ff700 1 --2- 172.21.15.23:0/3725945358 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fafd80ca370 0x7fafd814b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:54.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.629+0000 7fafdd7ff700 1 -- 172.21.15.23:0/3725945358 >> 172.21.15.23:0/3725945358 conn(0x7fafd8065b90 msgr2=0x7fafd8065f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:54.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.629+0000 7fafdd7ff700 1 -- 172.21.15.23:0/3725945358 shutdown_connections 2023-06-22T07:59:54.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.632+0000 7fafdd7ff700 1 -- 172.21.15.23:0/3725945358 wait complete. 2023-06-22T07:59:54.649 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' created 2023-06-22T07:59:54.667 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-22T07:59:54.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.866+0000 7f7a3bbff700 1 Processor -- start 2023-06-22T07:59:54.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.866+0000 7f7a3bbff700 1 -- start start 2023-06-22T07:59:54.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.867+0000 7f7a3bbff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a341593f0 0x7f7a340c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:54.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.867+0000 7f7a3bbff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a340c9aa0 con 0x7f7a34152ba0 2023-06-22T07:59:54.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.867+0000 7f7a3bbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3406aed0 con 0x7f7a340c4650 2023-06-22T07:59:54.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.867+0000 7f7a3bbff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a340c4a60 con 0x7f7a341593f0 2023-06-22T07:59:54.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.867+0000 7f7a3999b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7a34152ba0 0x7f7a340c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:42384/0 (socket says 172.21.15.23:42384) 2023-06-22T07:59:54.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.867+0000 7f7a3999b700 1 -- 172.21.15.23:0/2382978427 learned_addr learned my addr 172.21.15.23:0/2382978427 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T07:59:54.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.867+0000 7f7a3919a700 1 --2- 172.21.15.23:0/2382978427 >> v2:172.21.15.67:3300/0 conn(0x7f7a341593f0 0x7f7a340c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:54.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.868+0000 7f7a38999700 1 -- 172.21.15.23:0/2382978427 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397202177 0 0) 0x7f7a340c9aa0 con 0x7f7a34152ba0 2023-06-22T07:59:54.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.868+0000 7f7a38999700 1 -- 172.21.15.23:0/2382978427 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a200035b0 con 0x7f7a34152ba0 2023-06-22T07:59:54.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.868+0000 7f7a3919a700 1 -- 172.21.15.23:0/2382978427 >> v1:172.21.15.67:6789/0 conn(0x7f7a340c4650 legacy=0x7f7a340c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:54.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.868+0000 7f7a3919a700 1 -- 172.21.15.23:0/2382978427 >> v1:172.21.15.23:6789/0 conn(0x7f7a34152ba0 legacy=0x7f7a340c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.868+0000 7f7a3919a700 1 -- 172.21.15.23:0/2382978427 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3406b1b0 con 0x7f7a341593f0 2023-06-22T07:59:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.868+0000 7f7a3919a700 1 --2- 172.21.15.23:0/2382978427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a341593f0 0x7f7a340c4110 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a24009840 tx=0x7f7a24009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bad61ee613b76d52 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.869+0000 7f7a38999700 1 -- 172.21.15.23:0/2382978427 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24011830 con 0x7f7a341593f0 2023-06-22T07:59:54.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.869+0000 7f7a38999700 1 -- 172.21.15.23:0/2382978427 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a24011990 con 0x7f7a341593f0 2023-06-22T07:59:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.869+0000 7f7a38999700 1 -- 172.21.15.23:0/2382978427 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24011d20 con 0x7f7a341593f0 2023-06-22T07:59:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.869+0000 7f7a3bbff700 1 -- 172.21.15.23:0/2382978427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a341593f0 msgr2=0x7f7a340c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:54.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.869+0000 7f7a3bbff700 1 --2- 172.21.15.23:0/2382978427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a341593f0 0x7f7a340c4110 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7a24009840 tx=0x7f7a24009b50 comp rx=0 tx=0).stop 2023-06-22T07:59:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.869+0000 7f7a3bbff700 1 -- 172.21.15.23:0/2382978427 shutdown_connections 2023-06-22T07:59:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.869+0000 7f7a3bbff700 1 --2- 172.21.15.23:0/2382978427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a341593f0 0x7f7a340c4110 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.869+0000 7f7a3bbff700 1 -- 172.21.15.23:0/2382978427 >> 172.21.15.23:0/2382978427 conn(0x7f7a34065b90 msgr2=0x7f7a34065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:54.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.869+0000 7f7a3bbff700 1 -- 172.21.15.23:0/2382978427 shutdown_connections 2023-06-22T07:59:54.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.870+0000 7f7a3bbff700 1 -- 172.21.15.23:0/2382978427 wait complete. 2023-06-22T07:59:54.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.871+0000 7f7a3bbff700 1 Processor -- start 2023-06-22T07:59:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.871+0000 7f7a3bbff700 1 -- start start 2023-06-22T07:59:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.872+0000 7f7a3bbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a340c4650 0x7f7a341f13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.872+0000 7f7a3bbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a34152ba0 0x7f7a341f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.872+0000 7f7a3bbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a341593f0 0x7f7a340c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.872+0000 7f7a3bbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340b3ec0 con 0x7f7a341593f0 2023-06-22T07:59:54.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.872+0000 7f7a3bbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340cf730 con 0x7f7a340c4650 2023-06-22T07:59:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.872+0000 7f7a3bbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340b3990 con 0x7f7a34152ba0 2023-06-22T07:59:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.872+0000 7f7a3a19c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a341593f0 0x7f7a340c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.872+0000 7f7a3999b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a340c4650 0x7f7a341f13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.872+0000 7f7a3a19c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a341593f0 0x7f7a340c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44270/0 (socket says 172.21.15.23:44270) 2023-06-22T07:59:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a3a19c700 1 -- 172.21.15.23:0/3400575876 learned_addr learned my addr 172.21.15.23:0/3400575876 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.872+0000 7f7a3919a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a34152ba0 0x7f7a341f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a3a19c700 1 -- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a34152ba0 msgr2=0x7f7a341f1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:54.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a3a19c700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a34152ba0 0x7f7a341f1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:54.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a3a19c700 1 -- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a340c4650 msgr2=0x7f7a341f13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:54.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a3a19c700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a340c4650 0x7f7a341f13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:54.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a3a19c700 1 -- 172.21.15.23:0/3400575876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a240094a0 con 0x7f7a341593f0 2023-06-22T07:59:54.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a3a19c700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a341593f0 0x7f7a340c6740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00cbd0 tx=0x7f7a2c00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:54.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a2a7fc700 1 -- 172.21.15.23:0/3400575876 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c003c00 con 0x7f7a341593f0 2023-06-22T07:59:54.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a2a7fc700 1 -- 172.21.15.23:0/3400575876 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c003d60 con 0x7f7a341593f0 2023-06-22T07:59:54.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a2a7fc700 1 -- 172.21.15.23:0/3400575876 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c00aa20 con 0x7f7a341593f0 2023-06-22T07:59:54.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.873+0000 7f7a3bbff700 1 -- 172.21.15.23:0/3400575876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a340c62b0 con 0x7f7a341593f0 2023-06-22T07:59:54.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.874+0000 7f7a3bbff700 1 -- 172.21.15.23:0/3400575876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a340c5c60 con 0x7f7a341593f0 2023-06-22T07:59:54.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.875+0000 7f7a3bbff700 1 -- 172.21.15.23:0/3400575876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a340cbed0 con 0x7f7a341593f0 2023-06-22T07:59:54.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.876+0000 7f7a2a7fc700 1 -- 172.21.15.23:0/3400575876 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7a2c00ab80 con 0x7f7a341593f0 2023-06-22T07:59:54.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.877+0000 7f7a2a7fc700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1806ace0 0x7f7a1806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:54.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.877+0000 7f7a2a7fc700 1 -- 172.21.15.23:0/3400575876 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f7a2c089340 con 0x7f7a341593f0 2023-06-22T07:59:54.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.877+0000 7f7a3999b700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1806ace0 0x7f7a1806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:54.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.878+0000 7f7a3999b700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1806ace0 0x7f7a1806d190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7a30007fd0 tx=0x7f7a30007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:54.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:54.882+0000 7f7a2a7fc700 1 -- 172.21.15.23:0/3400575876 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7a2c054440 con 0x7f7a341593f0 2023-06-22T07:59:55.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.137+0000 7f7a3bbff700 1 -- 172.21.15.23:0/3400575876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7a340c5ac0 con 0x7f7a341593f0 2023-06-22T07:59:55.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.147+0000 7f7a2a7fc700 1 -- 172.21.15.23:0/3400575876 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7a340c5ac0 con 0x7f7a341593f0 2023-06-22T07:59:55.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.158+0000 7f7a3bbff700 1 -- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1806ace0 msgr2=0x7f7a1806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:55.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.158+0000 7f7a3bbff700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1806ace0 0x7f7a1806d190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7a30007fd0 tx=0x7f7a30007dc0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.158+0000 7f7a3bbff700 1 -- 172.21.15.23:0/3400575876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a341593f0 msgr2=0x7f7a340c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:55.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.158+0000 7f7a3bbff700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a341593f0 0x7f7a340c6740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00cbd0 tx=0x7f7a2c00cee0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.168+0000 7f7a3bbff700 1 -- 172.21.15.23:0/3400575876 shutdown_connections 2023-06-22T07:59:55.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.168+0000 7f7a3bbff700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a341593f0 0x7f7a340c6740 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.168+0000 7f7a3bbff700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1806ace0 0x7f7a1806d190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.168+0000 7f7a3bbff700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a340c4650 0x7f7a341f13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.168+0000 7f7a3bbff700 1 --2- 172.21.15.23:0/3400575876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a34152ba0 0x7f7a341f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.168+0000 7f7a3bbff700 1 -- 172.21.15.23:0/3400575876 >> 172.21.15.23:0/3400575876 conn(0x7f7a34065b90 msgr2=0x7f7a340c9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:55.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.169+0000 7f7a3bbff700 1 -- 172.21.15.23:0/3400575876 shutdown_connections 2023-06-22T07:59:55.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.169+0000 7f7a3bbff700 1 -- 172.21.15.23:0/3400575876 wait complete. 2023-06-22T07:59:55.180 INFO:teuthology.orchestra.run.smithi023.stderr:new fs with metadata pool 3 and data pool 4 2023-06-22T07:59:55.194 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-06-22T07:59:55.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-06-22T07:59:55.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.395+0000 7fbf97fff700 1 Processor -- start 2023-06-22T07:59:55.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.395+0000 7fbf97fff700 1 -- start start 2023-06-22T07:59:55.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.396+0000 7fbf97fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf90152ba0 0x7fbf901593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:55.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.396+0000 7fbf97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf900c9260 0x7fbf900c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:55.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.396+0000 7fbf97fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf90159930 con 0x7fbf900c4650 2023-06-22T07:59:55.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.396+0000 7fbf97fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf900c9aa0 con 0x7fbf900c9260 2023-06-22T07:59:55.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.396+0000 7fbf97fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9006aed0 con 0x7fbf90152ba0 2023-06-22T07:59:55.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.396+0000 7fbf95d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf90152ba0 0x7fbf901593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:55.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.396+0000 7fbf95d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf90152ba0 0x7fbf901593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:50544/0 (socket says 172.21.15.23:50544) 2023-06-22T07:59:55.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.396+0000 7fbf95d9b700 1 -- 172.21.15.23:0/771768924 learned_addr learned my addr 172.21.15.23:0/771768924 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:55.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.397+0000 7fbf9559a700 1 --2- 172.21.15.23:0/771768924 >> v2:172.21.15.67:3300/0 conn(0x7fbf900c9260 0x7fbf900c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:55.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.397+0000 7fbf95d9b700 1 -- 172.21.15.23:0/771768924 >> v2:172.21.15.67:3300/0 conn(0x7fbf900c9260 msgr2=0x7fbf900c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:55.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.397+0000 7fbf95d9b700 1 --2- 172.21.15.23:0/771768924 >> v2:172.21.15.67:3300/0 conn(0x7fbf900c9260 0x7fbf900c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.397+0000 7fbf95d9b700 1 -- 172.21.15.23:0/771768924 >> v1:172.21.15.67:6789/0 conn(0x7fbf900c4650 legacy=0x7fbf900c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:55.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.397+0000 7fbf94d99700 1 -- 172.21.15.23:0/771768924 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522033930 0 0) 0x7fbf88001490 con 0x7fbf900c4650 2023-06-22T07:59:55.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.397+0000 7fbf95d9b700 1 -- 172.21.15.23:0/771768924 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9006b1b0 con 0x7fbf90152ba0 2023-06-22T07:59:55.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.397+0000 7fbf95d9b700 1 --2- 172.21.15.23:0/771768924 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf90152ba0 0x7fbf901593f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0098f0 tx=0x7fbf8c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c1b4303d32d2c1c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:55.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf94d99700 1 -- 172.21.15.23:0/771768924 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c011830 con 0x7fbf90152ba0 2023-06-22T07:59:55.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf94d99700 1 -- 172.21.15.23:0/771768924 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8c011990 con 0x7fbf90152ba0 2023-06-22T07:59:55.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf94d99700 1 -- 172.21.15.23:0/771768924 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c011d20 con 0x7fbf90152ba0 2023-06-22T07:59:55.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf97fff700 1 -- 172.21.15.23:0/771768924 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf90152ba0 msgr2=0x7fbf901593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:55.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf97fff700 1 --2- 172.21.15.23:0/771768924 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf90152ba0 0x7fbf901593f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0098f0 tx=0x7fbf8c009c00 comp rx=0 tx=0).stop 2023-06-22T07:59:55.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf97fff700 1 -- 172.21.15.23:0/771768924 shutdown_connections 2023-06-22T07:59:55.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf97fff700 1 --2- 172.21.15.23:0/771768924 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf90152ba0 0x7fbf901593f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf97fff700 1 --2- 172.21.15.23:0/771768924 >> v2:172.21.15.67:3300/0 conn(0x7fbf900c9260 0x7fbf900c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf97fff700 1 -- 172.21.15.23:0/771768924 >> 172.21.15.23:0/771768924 conn(0x7fbf90065b90 msgr2=0x7fbf90065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf97fff700 1 -- 172.21.15.23:0/771768924 shutdown_connections 2023-06-22T07:59:55.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.398+0000 7fbf97fff700 1 -- 172.21.15.23:0/771768924 wait complete. 2023-06-22T07:59:55.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.400+0000 7fbf97fff700 1 Processor -- start 2023-06-22T07:59:55.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.400+0000 7fbf97fff700 1 -- start start 2023-06-22T07:59:55.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.401+0000 7fbf97fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf900c4650 0x7fbf900b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:55.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.401+0000 7fbf97fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf900c9260 0x7fbf9014de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:55.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.401+0000 7fbf97fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf90152ba0 0x7fbf900cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:55.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.401+0000 7fbf97fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900cf4e0 con 0x7fbf90152ba0 2023-06-22T07:59:55.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.401+0000 7fbf97fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900cdf40 con 0x7fbf900c9260 2023-06-22T07:59:55.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.401+0000 7fbf97fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900ce150 con 0x7fbf900c4650 2023-06-22T07:59:55.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf95d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf900c4650 0x7fbf900b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf9559a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf900c9260 0x7fbf9014de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf9559a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf900c9260 0x7fbf9014de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:41660/0 (socket says 172.21.15.23:41660) 2023-06-22T07:59:55.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf9559a700 1 -- 172.21.15.23:0/2078599829 learned_addr learned my addr 172.21.15.23:0/2078599829 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:55.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf9659c700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf90152ba0 0x7fbf900cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:55.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf9559a700 1 -- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf900c4650 msgr2=0x7fbf900b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:55.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf9559a700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf900c4650 0x7fbf900b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf9559a700 1 -- 172.21.15.23:0/2078599829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf90152ba0 msgr2=0x7fbf900cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:55.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf9559a700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf90152ba0 0x7fbf900cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:55.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf9559a700 1 -- 172.21.15.23:0/2078599829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c009550 con 0x7fbf900c9260 2023-06-22T07:59:55.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf95d9b700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf900c4650 0x7fbf900b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:55.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf9559a700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf900c9260 0x7fbf9014de70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000e9b0 tx=0x7fbf8000ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:55.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.402+0000 7fbf867fc700 1 -- 172.21.15.23:0/2078599829 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80009710 con 0x7fbf900c9260 2023-06-22T07:59:55.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.403+0000 7fbf867fc700 1 -- 172.21.15.23:0/2078599829 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8000ee50 con 0x7fbf900c9260 2023-06-22T07:59:55.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.403+0000 7fbf97fff700 1 -- 172.21.15.23:0/2078599829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf900ce430 con 0x7fbf900c9260 2023-06-22T07:59:55.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.403+0000 7fbf867fc700 1 -- 172.21.15.23:0/2078599829 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8000ca50 con 0x7fbf900c9260 2023-06-22T07:59:55.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.403+0000 7fbf97fff700 1 -- 172.21.15.23:0/2078599829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf900ce9f0 con 0x7fbf900c9260 2023-06-22T07:59:55.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.404+0000 7fbf97fff700 1 -- 172.21.15.23:0/2078599829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf900cbed0 con 0x7fbf900c9260 2023-06-22T07:59:55.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.405+0000 7fbf867fc700 1 -- 172.21.15.23:0/2078599829 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbf8000cbb0 con 0x7fbf900c9260 2023-06-22T07:59:55.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.406+0000 7fbf867fc700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbf7406ad10 0x7fbf7406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:55.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.406+0000 7fbf867fc700 1 -- 172.21.15.23:0/2078599829 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6062+0+0 (secure 0 0 0) 0x7fbf80089500 con 0x7fbf900c9260 2023-06-22T07:59:55.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.406+0000 7fbf95d9b700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbf7406ad10 0x7fbf7406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:55.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.407+0000 7fbf95d9b700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbf7406ad10 0x7fbf7406d1c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c014040 tx=0x7fbf8c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:55.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.411+0000 7fbf867fc700 1 -- 172.21.15.23:0/2078599829 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbf80057f10 con 0x7fbf900c9260 2023-06-22T07:59:55.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:55.629+0000 7fbf97fff700 1 -- 172.21.15.23:0/2078599829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fbf900b2250 con 0x7fbf900c9260 2023-06-22T07:59:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.149+0000 7fbf867fc700 1 -- 172.21.15.23:0/2078599829 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v24) v1 ==== 157+0+0 (secure 0 0 0) 0x7fbf900b2250 con 0x7fbf900c9260 2023-06-22T07:59:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 -- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbf7406ad10 msgr2=0x7fbf7406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbf7406ad10 0x7fbf7406d1c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c014040 tx=0x7fbf8c008030 comp rx=0 tx=0).stop 2023-06-22T07:59:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 -- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf900c9260 msgr2=0x7fbf9014de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf900c9260 0x7fbf9014de70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000e9b0 tx=0x7fbf8000ecc0 comp rx=0 tx=0).stop 2023-06-22T07:59:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 -- 172.21.15.23:0/2078599829 shutdown_connections 2023-06-22T07:59:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf90152ba0 0x7fbf900cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf900c4650 0x7fbf900b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbf7406ad10 0x7fbf7406d1c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 --2- 172.21.15.23:0/2078599829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf900c9260 0x7fbf9014de70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 -- 172.21.15.23:0/2078599829 >> 172.21.15.23:0/2078599829 conn(0x7fbf90065b90 msgr2=0x7fbf900ca280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 -- 172.21.15.23:0/2078599829 shutdown_connections 2023-06-22T07:59:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.152+0000 7fbf97fff700 1 -- 172.21.15.23:0/2078599829 wait complete. 2023-06-22T07:59:56.174 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-06-22T07:59:56.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.387+0000 7f840e5ff700 1 Processor -- start 2023-06-22T07:59:56.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.388+0000 7f840e5ff700 1 -- start start 2023-06-22T07:59:56.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.389+0000 7f840e5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 0x7f84080715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.389+0000 7f840e5ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8408152fe0 con 0x7f8408071b20 2023-06-22T07:59:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.389+0000 7f840e5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84080ce940 con 0x7f840815ca50 2023-06-22T07:59:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.389+0000 7f840e5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840815bf20 con 0x7f8408152c10 2023-06-22T07:59:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.389+0000 7f84077fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f8408071b20 0x7f840815c340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60232/0 (socket says 172.21.15.23:60232) 2023-06-22T07:59:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.389+0000 7f84077fe700 1 -- 172.21.15.23:0/1559640590 learned_addr learned my addr 172.21.15.23:0/1559640590 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T07:59:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.389+0000 7f8407fff700 1 --2- 172.21.15.23:0/1559640590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 0x7f84080715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.390+0000 7f8406ffd700 1 -- 172.21.15.23:0/1559640590 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054255544 0 0) 0x7f8408152fe0 con 0x7f8408071b20 2023-06-22T07:59:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.390+0000 7f8406ffd700 1 -- 172.21.15.23:0/1559640590 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f00035b0 con 0x7f8408071b20 2023-06-22T07:59:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.390+0000 7f8406ffd700 1 -- 172.21.15.23:0/1559640590 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375567954 0 0) 0x7f84080ce940 con 0x7f840815ca50 2023-06-22T07:59:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.390+0000 7f8406ffd700 1 -- 172.21.15.23:0/1559640590 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8408152fe0 con 0x7f840815ca50 2023-06-22T07:59:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.390+0000 7f8407fff700 1 -- 172.21.15.23:0/1559640590 >> v1:172.21.15.67:6789/0 conn(0x7f840815ca50 legacy=0x7f84080c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.390+0000 7f8407fff700 1 -- 172.21.15.23:0/1559640590 >> v1:172.21.15.23:6789/0 conn(0x7f8408071b20 legacy=0x7f840815c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:56.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.390+0000 7f8407fff700 1 -- 172.21.15.23:0/1559640590 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84080cec20 con 0x7f8408152c10 2023-06-22T07:59:56.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.390+0000 7f8407fff700 1 --2- 172.21.15.23:0/1559640590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 0x7f84080715e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009870 tx=0x7f83f8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=508f26d10a3f84a5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:56.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.391+0000 7f8406ffd700 1 -- 172.21.15.23:0/1559640590 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8011830 con 0x7f8408152c10 2023-06-22T07:59:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.391+0000 7f8406ffd700 1 -- 172.21.15.23:0/1559640590 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f8011990 con 0x7f8408152c10 2023-06-22T07:59:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.391+0000 7f8406ffd700 1 -- 172.21.15.23:0/1559640590 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8011d20 con 0x7f8408152c10 2023-06-22T07:59:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.391+0000 7f840e5ff700 1 -- 172.21.15.23:0/1559640590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 msgr2=0x7f84080715e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.391+0000 7f840e5ff700 1 --2- 172.21.15.23:0/1559640590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 0x7f84080715e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009870 tx=0x7f83f8009b80 comp rx=0 tx=0).stop 2023-06-22T07:59:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.391+0000 7f840e5ff700 1 -- 172.21.15.23:0/1559640590 shutdown_connections 2023-06-22T07:59:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.391+0000 7f840e5ff700 1 --2- 172.21.15.23:0/1559640590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 0x7f84080715e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.391+0000 7f840e5ff700 1 -- 172.21.15.23:0/1559640590 >> 172.21.15.23:0/1559640590 conn(0x7f840806b890 msgr2=0x7f840806bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.391+0000 7f840e5ff700 1 -- 172.21.15.23:0/1559640590 shutdown_connections 2023-06-22T07:59:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.391+0000 7f840e5ff700 1 -- 172.21.15.23:0/1559640590 wait complete. 2023-06-22T07:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.393+0000 7f840e5ff700 1 Processor -- start 2023-06-22T07:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.393+0000 7f840e5ff700 1 -- start start 2023-06-22T07:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f840e5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8408071b20 0x7f84080b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f840e5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 0x7f84081e2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f840e5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840815ca50 0x7f84080b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f840e5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84081e4700 con 0x7f8408071b20 2023-06-22T07:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f840e5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84081e4b20 con 0x7f840815ca50 2023-06-22T07:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f840e5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84081eeb40 con 0x7f8408152c10 2023-06-22T07:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f8407fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8408071b20 0x7f84080b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f8407fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8408071b20 0x7f84080b38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59988/0 (socket says 172.21.15.23:59988) 2023-06-22T07:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f8407fff700 1 -- 172.21.15.23:0/4207668868 learned_addr learned my addr 172.21.15.23:0/4207668868 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f84077fe700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 0x7f84081e2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.394+0000 7f840cb9c700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840815ca50 0x7f84080b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f8407fff700 1 -- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 msgr2=0x7f84081e2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f8407fff700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 0x7f84081e2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f8407fff700 1 -- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840815ca50 msgr2=0x7f84080b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f8407fff700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840815ca50 0x7f84080b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f8407fff700 1 -- 172.21.15.23:0/4207668868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f80094a0 con 0x7f8408071b20 2023-06-22T07:59:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f840cb9c700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840815ca50 0x7f84080b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T07:59:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f8407fff700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8408071b20 0x7f84080b38e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f83f80097c0 tx=0x7f83f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f8404ff9700 1 -- 172.21.15.23:0/4207668868 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f801aa70 con 0x7f8408071b20 2023-06-22T07:59:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f8404ff9700 1 -- 172.21.15.23:0/4207668868 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f801abd0 con 0x7f8408071b20 2023-06-22T07:59:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f840e5ff700 1 -- 172.21.15.23:0/4207668868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84080b44e0 con 0x7f8408071b20 2023-06-22T07:59:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f8404ff9700 1 -- 172.21.15.23:0/4207668868 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f80243f0 con 0x7f8408071b20 2023-06-22T07:59:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.395+0000 7f840e5ff700 1 -- 172.21.15.23:0/4207668868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84080b3f20 con 0x7f8408071b20 2023-06-22T07:59:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.396+0000 7f840e5ff700 1 -- 172.21.15.23:0/4207668868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84081ee390 con 0x7f8408071b20 2023-06-22T07:59:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.397+0000 7f8404ff9700 1 -- 172.21.15.23:0/4207668868 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f83f8024550 con 0x7f8408071b20 2023-06-22T07:59:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.398+0000 7f8404ff9700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f83f406ad10 0x7f83f406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.398+0000 7f8404ff9700 1 -- 172.21.15.23:0/4207668868 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f83f808aad0 con 0x7f8408071b20 2023-06-22T07:59:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.398+0000 7f84077fe700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f83f406ad10 0x7f83f406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.399+0000 7f84077fe700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f83f406ad10 0x7f83f406d1c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f84080ccbd0 tx=0x7f83fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.403+0000 7f8404ff9700 1 -- 172.21.15.23:0/4207668868 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f84081ee390 con 0x7f8408071b20 2023-06-22T07:59:56.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:56.613+0000 7f840e5ff700 1 -- 172.21.15.23:0/4207668868 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f84081ee390 con 0x7f8408071b20 2023-06-22T07:59:58.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.162+0000 7f8404ff9700 1 -- 172.21.15.23:0/4207668868 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v26) v1 ==== 228+0+0 (secure 0 0 0) 0x7f84081ee390 con 0x7f8408071b20 2023-06-22T07:59:58.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.168+0000 7f840e5ff700 1 -- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f83f406ad10 msgr2=0x7f83f406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:58.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.168+0000 7f840e5ff700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f83f406ad10 0x7f83f406d1c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f84080ccbd0 tx=0x7f83fc009040 comp rx=0 tx=0).stop 2023-06-22T07:59:58.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.168+0000 7f840e5ff700 1 -- 172.21.15.23:0/4207668868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8408071b20 msgr2=0x7f84080b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:58.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.168+0000 7f840e5ff700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8408071b20 0x7f84080b38e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f83f80097c0 tx=0x7f83f8004750 comp rx=0 tx=0).stop 2023-06-22T07:59:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.170+0000 7f840e5ff700 1 -- 172.21.15.23:0/4207668868 shutdown_connections 2023-06-22T07:59:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.170+0000 7f840e5ff700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f83f406ad10 0x7f83f406d1c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.170+0000 7f840e5ff700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f840815ca50 0x7f84080b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.170+0000 7f840e5ff700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8408071b20 0x7f84080b38e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.170+0000 7f840e5ff700 1 --2- 172.21.15.23:0/4207668868 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8408152c10 0x7f84081e2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.170+0000 7f840e5ff700 1 -- 172.21.15.23:0/4207668868 >> 172.21.15.23:0/4207668868 conn(0x7f840806b890 msgr2=0x7f84080ca310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:58.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.170+0000 7f840e5ff700 1 -- 172.21.15.23:0/4207668868 shutdown_connections 2023-06-22T07:59:58.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.170+0000 7f840e5ff700 1 -- 172.21.15.23:0/4207668868 wait complete. 2023-06-22T07:59:58.174 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' created 2023-06-22T07:59:58.194 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-06-22T07:59:58.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.397+0000 7f05cd9ff700 1 Processor -- start 2023-06-22T07:59:58.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.397+0000 7f05cd9ff700 1 -- start start 2023-06-22T07:59:58.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.398+0000 7f05cd9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05c8152ba0 0x7f05c81593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:58.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.398+0000 7f05cd9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05c80c9ee0 0x7f05c80ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:58.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.398+0000 7f05cd9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c8159930 con 0x7f05c80c9260 2023-06-22T07:59:58.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.398+0000 7f05cd9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c80c9aa0 con 0x7f05c8152ba0 2023-06-22T07:59:58.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.398+0000 7f05cd9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c806aed0 con 0x7f05c80c9ee0 2023-06-22T07:59:58.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.398+0000 7f05c6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05c8152ba0 0x7f05c81593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:58.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.398+0000 7f05c6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05c8152ba0 0x7f05c81593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59998/0 (socket says 172.21.15.23:59998) 2023-06-22T07:59:58.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.398+0000 7f05c6ffd700 1 -- 172.21.15.23:0/2912571187 learned_addr learned my addr 172.21.15.23:0/2912571187 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:58.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.398+0000 7f05c77fe700 1 --2- 172.21.15.23:0/2912571187 >> v2:172.21.15.67:3300/0 conn(0x7f05c80c9ee0 0x7f05c80ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:58.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.399+0000 7f05c6ffd700 1 -- 172.21.15.23:0/2912571187 >> v2:172.21.15.67:3300/0 conn(0x7f05c80c9ee0 msgr2=0x7f05c80ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:58.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.399+0000 7f05c6ffd700 1 --2- 172.21.15.23:0/2912571187 >> v2:172.21.15.67:3300/0 conn(0x7f05c80c9ee0 0x7f05c80ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:58.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.399+0000 7f05c6ffd700 1 -- 172.21.15.23:0/2912571187 >> v1:172.21.15.67:6789/0 conn(0x7f05c80c9260 legacy=0x7f05c80c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:58.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.399+0000 7f05c6ffd700 1 -- 172.21.15.23:0/2912571187 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c806b1b0 con 0x7f05c8152ba0 2023-06-22T07:59:58.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.399+0000 7f05c6ffd700 1 --2- 172.21.15.23:0/2912571187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c8152ba0 0x7f05c81593f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f05c808b390 tx=0x7f05b8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502236da2bf6c1cf server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:58.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.399+0000 7f05c5ffb700 1 -- 172.21.15.23:0/2912571187 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8011830 con 0x7f05c8152ba0 2023-06-22T07:59:58.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.399+0000 7f05c5ffb700 1 -- 172.21.15.23:0/2912571187 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b8011990 con 0x7f05c8152ba0 2023-06-22T07:59:58.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.399+0000 7f05c5ffb700 1 -- 172.21.15.23:0/2912571187 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8011d20 con 0x7f05c8152ba0 2023-06-22T07:59:58.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.400+0000 7f05cd9ff700 1 -- 172.21.15.23:0/2912571187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c8152ba0 msgr2=0x7f05c81593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:58.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.400+0000 7f05cd9ff700 1 --2- 172.21.15.23:0/2912571187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c8152ba0 0x7f05c81593f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f05c808b390 tx=0x7f05b8009bd0 comp rx=0 tx=0).stop 2023-06-22T07:59:58.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.400+0000 7f05cd9ff700 1 -- 172.21.15.23:0/2912571187 shutdown_connections 2023-06-22T07:59:58.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.400+0000 7f05cd9ff700 1 --2- 172.21.15.23:0/2912571187 >> v2:172.21.15.67:3300/0 conn(0x7f05c80c9ee0 0x7f05c80ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:58.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.400+0000 7f05cd9ff700 1 --2- 172.21.15.23:0/2912571187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c8152ba0 0x7f05c81593f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:58.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.400+0000 7f05cd9ff700 1 -- 172.21.15.23:0/2912571187 >> 172.21.15.23:0/2912571187 conn(0x7f05c8065b90 msgr2=0x7f05c8065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:58.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.400+0000 7f05cd9ff700 1 -- 172.21.15.23:0/2912571187 shutdown_connections 2023-06-22T07:59:58.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.400+0000 7f05cd9ff700 1 -- 172.21.15.23:0/2912571187 wait complete. 2023-06-22T07:59:58.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.402+0000 7f05cd9ff700 1 Processor -- start 2023-06-22T07:59:58.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.402+0000 7f05cd9ff700 1 -- start start 2023-06-22T07:59:58.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05cd9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05c80c9260 0x7f05c814e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:58.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05cd9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c80c9ee0 0x7f05c81f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:58.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05cd9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c8152ba0 0x7f05c80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:58.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05cd9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80b43d0 con 0x7f05c80c9ee0 2023-06-22T07:59:58.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05cd9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80cd7c0 con 0x7f05c8152ba0 2023-06-22T07:59:58.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05cd9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80ccca0 con 0x7f05c80c9260 2023-06-22T07:59:58.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05c67fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c80c9ee0 0x7f05c81f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:58.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05c67fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c80c9ee0 0x7f05c81f2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60008/0 (socket says 172.21.15.23:60008) 2023-06-22T07:59:58.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05c67fc700 1 -- 172.21.15.23:0/1268370130 learned_addr learned my addr 172.21.15.23:0/1268370130 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:58.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05c6ffd700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05c80c9260 0x7f05c814e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:58.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05c77fe700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c8152ba0 0x7f05c80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:58.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05c67fc700 1 -- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05c80c9260 msgr2=0x7f05c814e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:58.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05c67fc700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05c80c9260 0x7f05c814e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:58.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05c67fc700 1 -- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c8152ba0 msgr2=0x7f05c80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:58.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05c67fc700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c8152ba0 0x7f05c80b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:58.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.403+0000 7f05c67fc700 1 -- 172.21.15.23:0/1268370130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b8009550 con 0x7f05c80c9ee0 2023-06-22T07:59:58.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.404+0000 7f05c67fc700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c80c9ee0 0x7f05c81f2170 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00c4e0 tx=0x7f05bc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:58.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.404+0000 7f05affff700 1 -- 172.21.15.23:0/1268370130 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05bc0052d0 con 0x7f05c80c9ee0 2023-06-22T07:59:58.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.404+0000 7f05affff700 1 -- 172.21.15.23:0/1268370130 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05bc005430 con 0x7f05c80c9ee0 2023-06-22T07:59:58.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.404+0000 7f05cd9ff700 1 -- 172.21.15.23:0/1268370130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c80ccf20 con 0x7f05c80c9ee0 2023-06-22T07:59:58.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.404+0000 7f05affff700 1 -- 172.21.15.23:0/1268370130 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05bc0038c0 con 0x7f05c80c9ee0 2023-06-22T07:59:58.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.404+0000 7f05cd9ff700 1 -- 172.21.15.23:0/1268370130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c80cd4e0 con 0x7f05c80c9ee0 2023-06-22T07:59:58.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.405+0000 7f05cd9ff700 1 -- 172.21.15.23:0/1268370130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c80cbed0 con 0x7f05c80c9ee0 2023-06-22T07:59:58.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.406+0000 7f05affff700 1 -- 172.21.15.23:0/1268370130 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f05bc003a20 con 0x7f05c80c9ee0 2023-06-22T07:59:58.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.407+0000 7f05affff700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05b406ace0 0x7f05b406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:58.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.407+0000 7f05affff700 1 -- 172.21.15.23:0/1268370130 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6781+0+0 (secure 0 0 0) 0x7f05bc0889c0 con 0x7f05c80c9ee0 2023-06-22T07:59:58.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.408+0000 7f05c6ffd700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05b406ace0 0x7f05b406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:58.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.408+0000 7f05c6ffd700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05b406ace0 0x7f05b406d190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f05b8014040 tx=0x7f05b8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:58.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.412+0000 7f05affff700 1 -- 172.21.15.23:0/1268370130 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f05bc053700 con 0x7f05c80c9ee0 2023-06-22T07:59:58.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:58.619+0000 7f05cd9ff700 1 -- 172.21.15.23:0/1268370130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f05c8159930 con 0x7f05c80c9ee0 2023-06-22T07:59:59.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.174+0000 7f05affff700 1 -- 172.21.15.23:0/1268370130 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 5 allow_ec_overwrites to true v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7f05c8159930 con 0x7f05c80c9ee0 2023-06-22T07:59:59.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.185+0000 7f05cd9ff700 1 -- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05b406ace0 msgr2=0x7f05b406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.185+0000 7f05cd9ff700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05b406ace0 0x7f05b406d190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f05b8014040 tx=0x7f05b8008030 comp rx=0 tx=0).stop 2023-06-22T07:59:59.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.185+0000 7f05cd9ff700 1 -- 172.21.15.23:0/1268370130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c80c9ee0 msgr2=0x7f05c81f2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.185+0000 7f05cd9ff700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c80c9ee0 0x7f05c81f2170 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00c4e0 tx=0x7f05bc00c7f0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.186+0000 7f05cd9ff700 1 -- 172.21.15.23:0/1268370130 shutdown_connections 2023-06-22T07:59:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.186+0000 7f05cd9ff700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05b406ace0 0x7f05b406d190 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.186+0000 7f05cd9ff700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05c8152ba0 0x7f05c80b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.186+0000 7f05cd9ff700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05c80c9260 0x7f05c814e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.186+0000 7f05cd9ff700 1 --2- 172.21.15.23:0/1268370130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05c80c9ee0 0x7f05c81f2170 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.187+0000 7f05cd9ff700 1 -- 172.21.15.23:0/1268370130 >> 172.21.15.23:0/1268370130 conn(0x7f05c8065b90 msgr2=0x7f05c8157700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:59.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.192+0000 7f05cd9ff700 1 -- 172.21.15.23:0/1268370130 shutdown_connections 2023-06-22T07:59:59.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.192+0000 7f05cd9ff700 1 -- 172.21.15.23:0/1268370130 wait complete. 2023-06-22T07:59:59.207 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 5 allow_ec_overwrites to true 2023-06-22T07:59:59.227 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-06-22T07:59:59.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.426+0000 7fca941ff700 1 Processor -- start 2023-06-22T07:59:59.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.426+0000 7fca941ff700 1 -- start start 2023-06-22T07:59:59.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.427+0000 7fca941ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fca8c0b28b0 0x7fca8c159530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.427+0000 7fca941ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b3d70 0x7fca8c0b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.427+0000 7fca941ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca8c0b4390 0x7fca8c15c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.427+0000 7fca941ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c15cdd0 con 0x7fca8c0b28b0 2023-06-22T07:59:59.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.427+0000 7fca941ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c0b4830 con 0x7fca8c0b4390 2023-06-22T07:59:59.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.427+0000 7fca941ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0c92f0 con 0x7fca8c0b3d70 2023-06-22T07:59:59.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca91f9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fca8c0b28b0 0x7fca8c159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9279c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca8c0b4390 0x7fca8c15c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9179a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b3d70 0x7fca8c0b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9279c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fca8c0b4390 0x7fca8c15c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:34244/0 (socket says 172.21.15.23:34244) 2023-06-22T07:59:59.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9179a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b3d70 0x7fca8c0b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:58908/0 (socket says 172.21.15.23:58908) 2023-06-22T07:59:59.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9279c700 1 -- 172.21.15.23:0/3555221771 learned_addr learned my addr 172.21.15.23:0/3555221771 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:59.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9179a700 1 -- 172.21.15.23:0/3555221771 >> v2:172.21.15.67:3300/0 conn(0x7fca8c0b4390 msgr2=0x7fca8c15c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:59.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9179a700 1 --2- 172.21.15.23:0/3555221771 >> v2:172.21.15.67:3300/0 conn(0x7fca8c0b4390 0x7fca8c15c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9179a700 1 -- 172.21.15.23:0/3555221771 >> v2:172.21.15.23:3300/0 conn(0x7fca8c0b28b0 msgr2=0x7fca8c159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T07:59:59.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9179a700 1 --2- 172.21.15.23:0/3555221771 >> v2:172.21.15.23:3300/0 conn(0x7fca8c0b28b0 0x7fca8c159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9179a700 1 -- 172.21.15.23:0/3555221771 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c0c95d0 con 0x7fca8c0b3d70 2023-06-22T07:59:59.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.428+0000 7fca9179a700 1 --2- 172.21.15.23:0/3555221771 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b3d70 0x7fca8c0b19d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0c89c0 tx=0x7fca7c009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfa2ff72edd915b7 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:59.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca90f99700 1 -- 172.21.15.23:0/3555221771 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c011830 con 0x7fca8c0b3d70 2023-06-22T07:59:59.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca90f99700 1 -- 172.21.15.23:0/3555221771 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c011990 con 0x7fca8c0b3d70 2023-06-22T07:59:59.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca90f99700 1 -- 172.21.15.23:0/3555221771 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c011d20 con 0x7fca8c0b3d70 2023-06-22T07:59:59.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca941ff700 1 -- 172.21.15.23:0/3555221771 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b3d70 msgr2=0x7fca8c0b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca941ff700 1 --2- 172.21.15.23:0/3555221771 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b3d70 0x7fca8c0b19d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0c89c0 tx=0x7fca7c009b20 comp rx=0 tx=0).stop 2023-06-22T07:59:59.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca941ff700 1 -- 172.21.15.23:0/3555221771 shutdown_connections 2023-06-22T07:59:59.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca941ff700 1 --2- 172.21.15.23:0/3555221771 >> v2:172.21.15.67:3300/0 conn(0x7fca8c0b4390 0x7fca8c15c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca941ff700 1 --2- 172.21.15.23:0/3555221771 >> v2:172.21.15.23:3300/0 conn(0x7fca8c0b28b0 0x7fca8c159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca941ff700 1 --2- 172.21.15.23:0/3555221771 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b3d70 0x7fca8c0b19d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca941ff700 1 -- 172.21.15.23:0/3555221771 >> 172.21.15.23:0/3555221771 conn(0x7fca8c06ba40 msgr2=0x7fca8c0715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:59.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca941ff700 1 -- 172.21.15.23:0/3555221771 shutdown_connections 2023-06-22T07:59:59.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.429+0000 7fca941ff700 1 -- 172.21.15.23:0/3555221771 wait complete. 2023-06-22T07:59:59.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.431+0000 7fca941ff700 1 Processor -- start 2023-06-22T07:59:59.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.431+0000 7fca941ff700 1 -- start start 2023-06-22T07:59:59.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca941ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca8c0b28b0 0x7fca8c0c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca941ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca8c0b3d70 0x7fca8c0c7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca941ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b4390 0x7fca8c14ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca941ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c1ee730 con 0x7fca8c0b28b0 2023-06-22T07:59:59.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca941ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c158340 con 0x7fca8c0b3d70 2023-06-22T07:59:59.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca941ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c1ed840 con 0x7fca8c0b4390 2023-06-22T07:59:59.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca91f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca8c0b28b0 0x7fca8c0c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca9279c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b4390 0x7fca8c14ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca9179a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca8c0b3d70 0x7fca8c0c7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca91f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca8c0b28b0 0x7fca8c0c6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60024/0 (socket says 172.21.15.23:60024) 2023-06-22T07:59:59.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.432+0000 7fca91f9b700 1 -- 172.21.15.23:0/1517001906 learned_addr learned my addr 172.21.15.23:0/1517001906 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:59.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca9179a700 1 -- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b4390 msgr2=0x7fca8c14ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca9179a700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b4390 0x7fca8c14ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca9179a700 1 -- 172.21.15.23:0/1517001906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca8c0b28b0 msgr2=0x7fca8c0c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca9179a700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca8c0b28b0 0x7fca8c0c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca9179a700 1 -- 172.21.15.23:0/1517001906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0094a0 con 0x7fca8c0b3d70 2023-06-22T07:59:59.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca91f9b700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca8c0b28b0 0x7fca8c0c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T07:59:59.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca9179a700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca8c0b3d70 0x7fca8c0c7530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0b4b20 tx=0x7fca7c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:59.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca827fc700 1 -- 172.21.15.23:0/1517001906 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c004450 con 0x7fca8c0b3d70 2023-06-22T07:59:59.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca827fc700 1 -- 172.21.15.23:0/1517001906 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c01f070 con 0x7fca8c0b3d70 2023-06-22T07:59:59.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca941ff700 1 -- 172.21.15.23:0/1517001906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8c1edb20 con 0x7fca8c0b3d70 2023-06-22T07:59:59.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca827fc700 1 -- 172.21.15.23:0/1517001906 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c0038e0 con 0x7fca8c0b3d70 2023-06-22T07:59:59.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.433+0000 7fca941ff700 1 -- 172.21.15.23:0/1517001906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c1ee0e0 con 0x7fca8c0b3d70 2023-06-22T07:59:59.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.434+0000 7fca941ff700 1 -- 172.21.15.23:0/1517001906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c04ea20 con 0x7fca8c0b3d70 2023-06-22T07:59:59.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.436+0000 7fca827fc700 1 -- 172.21.15.23:0/1517001906 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fca7c003a40 con 0x7fca8c0b3d70 2023-06-22T07:59:59.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.437+0000 7fca827fc700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fca7006ace0 0x7fca7006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.437+0000 7fca827fc700 1 -- 172.21.15.23:0/1517001906 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6781+0+0 (secure 0 0 0) 0x7fca7c021070 con 0x7fca8c0b3d70 2023-06-22T07:59:59.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.437+0000 7fca91f9b700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fca7006ace0 0x7fca7006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.437+0000 7fca91f9b700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fca7006ace0 0x7fca7006d190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fca88011310 tx=0x7fca88009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:59.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.442+0000 7fca827fc700 1 -- 172.21.15.23:0/1517001906 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fca7c055b30 con 0x7fca8c0b3d70 2023-06-22T07:59:59.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.697+0000 7fca941ff700 1 -- 172.21.15.23:0/1517001906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fca8c0c9af0 con 0x7fca8c0b3d70 2023-06-22T07:59:59.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.708+0000 7fca827fc700 1 -- 172.21.15.23:0/1517001906 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 5 to fsmap v5) v1 ==== 137+0+0 (secure 0 0 0) 0x7fca7c0150b0 con 0x7fca8c0b3d70 2023-06-22T07:59:59.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 -- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fca7006ace0 msgr2=0x7fca7006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fca7006ace0 0x7fca7006d190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fca88011310 tx=0x7fca88009210 comp rx=0 tx=0).stop 2023-06-22T07:59:59.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 -- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca8c0b3d70 msgr2=0x7fca8c0c7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca8c0b3d70 0x7fca8c0c7530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0b4b20 tx=0x7fca7c004750 comp rx=0 tx=0).stop 2023-06-22T07:59:59.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 -- 172.21.15.23:0/1517001906 shutdown_connections 2023-06-22T07:59:59.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fca8c0b4390 0x7fca8c14ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca8c0b28b0 0x7fca8c0c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fca7006ace0 0x7fca7006d190 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 --2- 172.21.15.23:0/1517001906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca8c0b3d70 0x7fca8c0c7530 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 -- 172.21.15.23:0/1517001906 >> 172.21.15.23:0/1517001906 conn(0x7fca8c06ba40 msgr2=0x7fca8c15a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:59.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 -- 172.21.15.23:0/1517001906 shutdown_connections 2023-06-22T07:59:59.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.711+0000 7fca941ff700 1 -- 172.21.15.23:0/1517001906 wait complete. 2023-06-22T07:59:59.718 INFO:teuthology.orchestra.run.smithi023.stderr:added data pool 5 to fsmap 2023-06-22T07:59:59.731 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T07:59:59.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.928+0000 7fabb0dff700 1 Processor -- start 2023-06-22T07:59:59.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.928+0000 7fabb0dff700 1 -- start start 2023-06-22T07:59:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.930+0000 7fabb0dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fabac152ce0 0x7fabac0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.930+0000 7fabb0dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabac15c370 con 0x7fabac0c40b0 2023-06-22T07:59:59.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.930+0000 7fabb0dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabac15bda0 con 0x7fabac0c9fd0 2023-06-22T07:59:59.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.930+0000 7fabb0dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabac1530b0 con 0x7fabac152ce0 2023-06-22T07:59:59.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.930+0000 7fabaa59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fabac152ce0 0x7fabac0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.930+0000 7fabaa59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fabac152ce0 0x7fabac0cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60034/0 (socket says 172.21.15.23:60034) 2023-06-22T07:59:59.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.930+0000 7fabaa59c700 1 -- 172.21.15.23:0/2061225281 learned_addr learned my addr 172.21.15.23:0/2061225281 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:59.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.930+0000 7fabaa59c700 1 -- 172.21.15.23:0/2061225281 >> v1:172.21.15.67:6789/0 conn(0x7fabac0c9fd0 legacy=0x7fabac15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.930+0000 7fabaa59c700 1 -- 172.21.15.23:0/2061225281 >> v1:172.21.15.23:6789/0 conn(0x7fabac0c40b0 legacy=0x7fabac0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.930+0000 7fabaa59c700 1 -- 172.21.15.23:0/2061225281 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac0c9460 con 0x7fabac152ce0 2023-06-22T07:59:59.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.931+0000 7fabaa59c700 1 --2- 172.21.15.23:0/2061225281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac152ce0 0x7fabac0cf1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0098f0 tx=0x7fab9c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a841915a3079a57 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:59.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.931+0000 7faba9d9b700 1 -- 172.21.15.23:0/2061225281 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00b8a0 con 0x7fabac152ce0 2023-06-22T07:59:59.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.931+0000 7faba9d9b700 1 -- 172.21.15.23:0/2061225281 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c00ba00 con 0x7fabac152ce0 2023-06-22T07:59:59.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.931+0000 7faba9d9b700 1 -- 172.21.15.23:0/2061225281 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00bd90 con 0x7fabac152ce0 2023-06-22T07:59:59.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.931+0000 7fabb0dff700 1 -- 172.21.15.23:0/2061225281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac152ce0 msgr2=0x7fabac0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.931+0000 7fabb0dff700 1 --2- 172.21.15.23:0/2061225281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac152ce0 0x7fabac0cf1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0098f0 tx=0x7fab9c009c00 comp rx=0 tx=0).stop 2023-06-22T07:59:59.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.931+0000 7fabb0dff700 1 -- 172.21.15.23:0/2061225281 shutdown_connections 2023-06-22T07:59:59.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.931+0000 7fabb0dff700 1 --2- 172.21.15.23:0/2061225281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac152ce0 0x7fabac0cf1b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.931+0000 7fabb0dff700 1 -- 172.21.15.23:0/2061225281 >> 172.21.15.23:0/2061225281 conn(0x7fabac06ae30 msgr2=0x7fabac06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T07:59:59.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.932+0000 7fabb0dff700 1 -- 172.21.15.23:0/2061225281 shutdown_connections 2023-06-22T07:59:59.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.932+0000 7fabb0dff700 1 -- 172.21.15.23:0/2061225281 wait complete. 2023-06-22T07:59:59.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.933+0000 7fabb0dff700 1 Processor -- start 2023-06-22T07:59:59.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.933+0000 7fabb0dff700 1 -- start start 2023-06-22T07:59:59.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.934+0000 7fabb0dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabac0c40b0 0x7fabac1e1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.934+0000 7fabb0dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac0c9fd0 0x7fabac1e1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.934+0000 7fabb0dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fabac152ce0 0x7fabac153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.934+0000 7fabb0dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac155890 con 0x7fabac0c9fd0 2023-06-22T07:59:59.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.934+0000 7fabb0dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac154c80 con 0x7fabac0c40b0 2023-06-22T07:59:59.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.934+0000 7fabb0dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac154f60 con 0x7fabac152ce0 2023-06-22T07:59:59.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.934+0000 7faba3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac0c9fd0 0x7fabac1e1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.934+0000 7faba3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac0c9fd0 0x7fabac1e1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60050/0 (socket says 172.21.15.23:60050) 2023-06-22T07:59:59.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.934+0000 7fabaa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabac0c40b0 0x7fabac1e1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7faba3fff700 1 -- 172.21.15.23:0/22547901 learned_addr learned my addr 172.21.15.23:0/22547901 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T07:59:59.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7fabaad9d700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fabac152ce0 0x7fabac153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7faba3fff700 1 -- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fabac152ce0 msgr2=0x7fabac153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7faba3fff700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fabac152ce0 0x7fabac153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7faba3fff700 1 -- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabac0c40b0 msgr2=0x7fabac1e1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T07:59:59.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7faba3fff700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabac0c40b0 0x7fabac1e1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T07:59:59.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7faba3fff700 1 -- 172.21.15.23:0/22547901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c009550 con 0x7fabac0c9fd0 2023-06-22T07:59:59.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7faba3fff700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac0c9fd0 0x7fabac1e1a50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab9400c4e0 tx=0x7fab9400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:59.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7faba2ffd700 1 -- 172.21.15.23:0/22547901 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab940052d0 con 0x7fabac0c9fd0 2023-06-22T07:59:59.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7faba2ffd700 1 -- 172.21.15.23:0/22547901 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab94005430 con 0x7fabac0c9fd0 2023-06-22T07:59:59.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7faba2ffd700 1 -- 172.21.15.23:0/22547901 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab940038c0 con 0x7fabac0c9fd0 2023-06-22T07:59:59.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.935+0000 7fabb0dff700 1 -- 172.21.15.23:0/22547901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabac155240 con 0x7fabac0c9fd0 2023-06-22T07:59:59.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.936+0000 7fabb0dff700 1 -- 172.21.15.23:0/22547901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabac14aae0 con 0x7fabac0c9fd0 2023-06-22T07:59:59.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.936+0000 7fabb0dff700 1 -- 172.21.15.23:0/22547901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabac0cca20 con 0x7fabac0c9fd0 2023-06-22T07:59:59.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.938+0000 7faba2ffd700 1 -- 172.21.15.23:0/22547901 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fab94003a20 con 0x7fabac0c9fd0 2023-06-22T07:59:59.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.938+0000 7faba2ffd700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab8c06ace0 0x7fab8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T07:59:59.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.939+0000 7faba2ffd700 1 -- 172.21.15.23:0/22547901 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fab94088b70 con 0x7fabac0c9fd0 2023-06-22T07:59:59.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.939+0000 7fabaa59c700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab8c06ace0 0x7fab8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T07:59:59.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.939+0000 7fabaa59c700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab8c06ace0 0x7fab8c06d190 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fab9c014040 tx=0x7fab9c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T07:59:59.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T07:59:59.943+0000 7faba2ffd700 1 -- 172.21.15.23:0/22547901 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fabac0cca20 con 0x7fabac0c9fd0 2023-06-22T08:00:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.204+0000 7fabb0dff700 1 -- 172.21.15.23:0/22547901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabac155a40 con 0x7fabac0c9fd0 2023-06-22T08:00:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.204+0000 7faba2ffd700 1 -- 172.21.15.23:0/22547901 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+6178 (secure 0 0 0) 0x7fabac155a40 con 0x7fabac0c9fd0 2023-06-22T08:00:00.207 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:00.208 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4268,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T07:59:59.700375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:00:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.206+0000 7fabb0dff700 1 -- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab8c06ace0 msgr2=0x7fab8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.206+0000 7fabb0dff700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab8c06ace0 0x7fab8c06d190 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fab9c014040 tx=0x7fab9c008030 comp rx=0 tx=0).stop 2023-06-22T08:00:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.206+0000 7fabb0dff700 1 -- 172.21.15.23:0/22547901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac0c9fd0 msgr2=0x7fabac1e1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.206+0000 7fabb0dff700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac0c9fd0 0x7fabac1e1a50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab9400c4e0 tx=0x7fab9400c7f0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.207+0000 7fabb0dff700 1 -- 172.21.15.23:0/22547901 shutdown_connections 2023-06-22T08:00:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.207+0000 7fabb0dff700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fabac152ce0 0x7fabac153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.207+0000 7fabb0dff700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab8c06ace0 0x7fab8c06d190 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.207+0000 7fabb0dff700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabac0c40b0 0x7fabac1e1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.207+0000 7fabb0dff700 1 --2- 172.21.15.23:0/22547901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabac0c9fd0 0x7fabac1e1a50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.207+0000 7fabb0dff700 1 -- 172.21.15.23:0/22547901 >> 172.21.15.23:0/22547901 conn(0x7fabac06ae30 msgr2=0x7fabac159bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.207+0000 7fabb0dff700 1 -- 172.21.15.23:0/22547901 shutdown_connections 2023-06-22T08:00:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.207+0000 7fabb0dff700 1 -- 172.21.15.23:0/22547901 wait complete. 2023-06-22T08:00:00.212 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 5 2023-06-22T08:00:00.227 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:00.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.430+0000 7f97633ff700 1 Processor -- start 2023-06-22T08:00:00.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.430+0000 7f97633ff700 1 -- start start 2023-06-22T08:00:00.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.431+0000 7f97633ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f975c14f850 0x7f975c0c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.431+0000 7f97633ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c0c69e0 0x7f975c0c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.431+0000 7f97633ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c0c51a0 con 0x7f975c071070 2023-06-22T08:00:00.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.431+0000 7f97633ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f975c0c44e0 con 0x7f975c14f850 2023-06-22T08:00:00.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.431+0000 7f97633ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975c0c5440 con 0x7f975c0c69e0 2023-06-22T08:00:00.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.431+0000 7f976099a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c0c69e0 0x7f975c0c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.431+0000 7f976119b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f975c14f850 0x7f975c0c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.432+0000 7f976099a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c0c69e0 0x7f975c0c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:58924/0 (socket says 172.21.15.23:58924) 2023-06-22T08:00:00.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.432+0000 7f976099a700 1 -- 172.21.15.23:0/1694621886 learned_addr learned my addr 172.21.15.23:0/1694621886 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:00.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.432+0000 7f976099a700 1 -- 172.21.15.23:0/1694621886 >> v2:172.21.15.67:3300/0 conn(0x7f975c14f850 msgr2=0x7f975c0c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:00.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.432+0000 7f976099a700 1 --2- 172.21.15.23:0/1694621886 >> v2:172.21.15.67:3300/0 conn(0x7f975c14f850 0x7f975c0c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.432+0000 7f976099a700 1 -- 172.21.15.23:0/1694621886 >> v1:172.21.15.67:6789/0 conn(0x7f975c071070 legacy=0x7f975c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.432+0000 7f976099a700 1 -- 172.21.15.23:0/1694621886 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c0c5720 con 0x7f975c0c69e0 2023-06-22T08:00:00.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.432+0000 7f9753fff700 1 -- 172.21.15.23:0/1694621886 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293977854 0 0) 0x7f975c0c51a0 con 0x7f975c071070 2023-06-22T08:00:00.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.432+0000 7f976099a700 1 --2- 172.21.15.23:0/1694621886 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c0c69e0 0x7f975c0c35c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f974c0098f0 tx=0x7f974c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35ec782297d51c15 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:00.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.432+0000 7f9753fff700 1 -- 172.21.15.23:0/1694621886 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c011830 con 0x7f975c0c69e0 2023-06-22T08:00:00.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.432+0000 7f9753fff700 1 -- 172.21.15.23:0/1694621886 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974c011990 con 0x7f975c0c69e0 2023-06-22T08:00:00.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.433+0000 7f9753fff700 1 -- 172.21.15.23:0/1694621886 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c011d20 con 0x7f975c0c69e0 2023-06-22T08:00:00.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.433+0000 7f97633ff700 1 -- 172.21.15.23:0/1694621886 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c0c69e0 msgr2=0x7f975c0c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.433+0000 7f97633ff700 1 --2- 172.21.15.23:0/1694621886 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c0c69e0 0x7f975c0c35c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f974c0098f0 tx=0x7f974c009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:00.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.433+0000 7f97633ff700 1 -- 172.21.15.23:0/1694621886 shutdown_connections 2023-06-22T08:00:00.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.433+0000 7f97633ff700 1 --2- 172.21.15.23:0/1694621886 >> v2:172.21.15.67:3300/0 conn(0x7f975c14f850 0x7f975c0c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.433+0000 7f97633ff700 1 --2- 172.21.15.23:0/1694621886 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c0c69e0 0x7f975c0c35c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.433+0000 7f97633ff700 1 -- 172.21.15.23:0/1694621886 >> 172.21.15.23:0/1694621886 conn(0x7f975c0ca3f0 msgr2=0x7f975c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:00.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.433+0000 7f97633ff700 1 -- 172.21.15.23:0/1694621886 shutdown_connections 2023-06-22T08:00:00.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.433+0000 7f97633ff700 1 -- 172.21.15.23:0/1694621886 wait complete. 2023-06-22T08:00:00.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.435+0000 7f97633ff700 1 Processor -- start 2023-06-22T08:00:00.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.435+0000 7f97633ff700 1 -- start start 2023-06-22T08:00:00.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f97633ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f975c071070 0x7f975c0c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f97633ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f975c0c69e0 0x7f975c14a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f97633ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c14f850 0x7f975c14ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f97633ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c14a4f0 con 0x7f975c071070 2023-06-22T08:00:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f97633ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975c146340 con 0x7f975c0c69e0 2023-06-22T08:00:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f97633ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975c146590 con 0x7f975c14f850 2023-06-22T08:00:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f976119b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f975c071070 0x7f975c0c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f976119b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f975c071070 0x7f975c0c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60056/0 (socket says 172.21.15.23:60056) 2023-06-22T08:00:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f976119b700 1 -- 172.21.15.23:0/4086594598 learned_addr learned my addr 172.21.15.23:0/4086594598 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f976099a700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f975c0c69e0 0x7f975c14a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.436+0000 7f976199c700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c14f850 0x7f975c14ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f976119b700 1 -- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c14f850 msgr2=0x7f975c14ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f976119b700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c14f850 0x7f975c14ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f976119b700 1 -- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f975c0c69e0 msgr2=0x7f975c14a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f976119b700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f975c0c69e0 0x7f975c14a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f976119b700 1 -- 172.21.15.23:0/4086594598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c009550 con 0x7f975c071070 2023-06-22T08:00:00.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f976099a700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f975c0c69e0 0x7f975c14a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f976119b700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f975c071070 0x7f975c0c9490 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f975800e870 tx=0x7f975800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f9751ffb700 1 -- 172.21.15.23:0/4086594598 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758009590 con 0x7f975c071070 2023-06-22T08:00:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f9751ffb700 1 -- 172.21.15.23:0/4086594598 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9758013680 con 0x7f975c071070 2023-06-22T08:00:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f9751ffb700 1 -- 172.21.15.23:0/4086594598 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97580139f0 con 0x7f975c071070 2023-06-22T08:00:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f97633ff700 1 -- 172.21.15.23:0/4086594598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975c146870 con 0x7f975c071070 2023-06-22T08:00:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.437+0000 7f97633ff700 1 -- 172.21.15.23:0/4086594598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975c149e30 con 0x7f975c071070 2023-06-22T08:00:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.438+0000 7f97633ff700 1 -- 172.21.15.23:0/4086594598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975c0c2950 con 0x7f975c071070 2023-06-22T08:00:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.439+0000 7f9751ffb700 1 -- 172.21.15.23:0/4086594598 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9758013b50 con 0x7f975c071070 2023-06-22T08:00:00.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.440+0000 7f9751ffb700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f974006ace0 0x7f974006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.440+0000 7f9751ffb700 1 -- 172.21.15.23:0/4086594598 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9758089680 con 0x7f975c071070 2023-06-22T08:00:00.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.440+0000 7f976099a700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f974006ace0 0x7f974006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.441+0000 7f976099a700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f974006ace0 0x7f974006d190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f975c0c6db0 tx=0x7f974c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:00.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.444+0000 7f9751ffb700 1 -- 172.21.15.23:0/4086594598 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9758057e30 con 0x7f975c071070 2023-06-22T08:00:00.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.652+0000 7f97633ff700 1 -- 172.21.15.23:0/4086594598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f975c14a760 con 0x7f975c071070 2023-06-22T08:00:00.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.653+0000 7f9751ffb700 1 -- 172.21.15.23:0/4086594598 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f97580579c0 con 0x7f975c071070 2023-06-22T08:00:00.657 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:00.657 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 -- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f974006ace0 msgr2=0x7f974006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f974006ace0 0x7f974006d190 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f975c0c6db0 tx=0x7f974c008030 comp rx=0 tx=0).stop 2023-06-22T08:00:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 -- 172.21.15.23:0/4086594598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f975c071070 msgr2=0x7f975c0c9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f975c071070 0x7f975c0c9490 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f975800e870 tx=0x7f975800eb80 comp rx=0 tx=0).stop 2023-06-22T08:00:00.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 -- 172.21.15.23:0/4086594598 shutdown_connections 2023-06-22T08:00:00.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f975c14f850 0x7f975c14ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f975c071070 0x7f975c0c9490 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f974006ace0 0x7f974006d190 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 --2- 172.21.15.23:0/4086594598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f975c0c69e0 0x7f975c14a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 -- 172.21.15.23:0/4086594598 >> 172.21.15.23:0/4086594598 conn(0x7f975c0ca3f0 msgr2=0x7f975c06d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:00.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 -- 172.21.15.23:0/4086594598 shutdown_connections 2023-06-22T08:00:00.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.656+0000 7f97633ff700 1 -- 172.21.15.23:0/4086594598 wait complete. 2023-06-22T08:00:00.678 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:00.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.874+0000 7f26931ff700 1 Processor -- start 2023-06-22T08:00:00.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.874+0000 7f26931ff700 1 -- start start 2023-06-22T08:00:00.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.875+0000 7f26931ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f268c152ce0 0x7f268c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.875+0000 7f26931ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f268c0c9fd0 0x7f268c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.875+0000 7f26931ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f268c158880 0x7f268c158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.875+0000 7f26931ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f268c1530b0 con 0x7f268c152ce0 2023-06-22T08:00:00.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.875+0000 7f26931ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f268c0cf6f0 con 0x7f268c158880 2023-06-22T08:00:00.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.875+0000 7f26931ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268c15c370 con 0x7f268c0c9fd0 2023-06-22T08:00:00.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f2690f9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f268c152ce0 0x7f268c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f2690f9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f268c152ce0 0x7f268c0cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60062/0 (socket says 172.21.15.23:60062) 2023-06-22T08:00:00.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f2690f9b700 1 -- 172.21.15.23:0/3070160083 learned_addr learned my addr 172.21.15.23:0/3070160083 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:00.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f269179c700 1 --2- 172.21.15.23:0/3070160083 >> v2:172.21.15.67:3300/0 conn(0x7f268c158880 0x7f268c158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f268bfff700 1 --2- 172.21.15.23:0/3070160083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f268c0c9fd0 0x7f268c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f2690f9b700 1 -- 172.21.15.23:0/3070160083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f268c0c9fd0 msgr2=0x7f268c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:00.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f2690f9b700 1 --2- 172.21.15.23:0/3070160083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f268c0c9fd0 0x7f268c15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f2690f9b700 1 -- 172.21.15.23:0/3070160083 >> v2:172.21.15.67:3300/0 conn(0x7f268c158880 msgr2=0x7f268c158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:00.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f2690f9b700 1 --2- 172.21.15.23:0/3070160083 >> v2:172.21.15.67:3300/0 conn(0x7f268c158880 0x7f268c158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f2690f9b700 1 -- 172.21.15.23:0/3070160083 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c15bda0 con 0x7f268c152ce0 2023-06-22T08:00:00.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.876+0000 7f2690f9b700 1 --2- 172.21.15.23:0/3070160083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c152ce0 0x7f268c0cf1b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f268c08b9e0 tx=0x7f2680009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499e2c39385dd62 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:00.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f268b7fe700 1 -- 172.21.15.23:0/3070160083 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680011830 con 0x7f268c152ce0 2023-06-22T08:00:00.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f268b7fe700 1 -- 172.21.15.23:0/3070160083 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2680011990 con 0x7f268c152ce0 2023-06-22T08:00:00.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f268b7fe700 1 -- 172.21.15.23:0/3070160083 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680011d20 con 0x7f268c152ce0 2023-06-22T08:00:00.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f26931ff700 1 -- 172.21.15.23:0/3070160083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c152ce0 msgr2=0x7f268c0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f26931ff700 1 --2- 172.21.15.23:0/3070160083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c152ce0 0x7f268c0cf1b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f268c08b9e0 tx=0x7f2680009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f26931ff700 1 -- 172.21.15.23:0/3070160083 shutdown_connections 2023-06-22T08:00:00.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f26931ff700 1 --2- 172.21.15.23:0/3070160083 >> v2:172.21.15.67:3300/0 conn(0x7f268c158880 0x7f268c158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f26931ff700 1 --2- 172.21.15.23:0/3070160083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c152ce0 0x7f268c0cf1b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f26931ff700 1 --2- 172.21.15.23:0/3070160083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f268c0c9fd0 0x7f268c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f26931ff700 1 -- 172.21.15.23:0/3070160083 >> 172.21.15.23:0/3070160083 conn(0x7f268c06ae30 msgr2=0x7f268c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:00.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f26931ff700 1 -- 172.21.15.23:0/3070160083 shutdown_connections 2023-06-22T08:00:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.877+0000 7f26931ff700 1 -- 172.21.15.23:0/3070160083 wait complete. 2023-06-22T08:00:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.879+0000 7f26931ff700 1 Processor -- start 2023-06-22T08:00:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.879+0000 7f26931ff700 1 -- start start 2023-06-22T08:00:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.880+0000 7f26931ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c0c9fd0 0x7f268c1514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.880+0000 7f26931ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f268c152ce0 0x7f268c1ef740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.880+0000 7f26931ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f268c158880 0x7f268c1efc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.880+0000 7f26931ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c0ce950 con 0x7f268c0c9fd0 2023-06-22T08:00:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.880+0000 7f26931ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268c1547b0 con 0x7f268c152ce0 2023-06-22T08:00:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.880+0000 7f26931ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268c1ef2b0 con 0x7f268c158880 2023-06-22T08:00:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.880+0000 7f2690f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c0c9fd0 0x7f268c1514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.880+0000 7f2690f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c0c9fd0 0x7f268c1514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60066/0 (socket says 172.21.15.23:60066) 2023-06-22T08:00:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.880+0000 7f2690f9b700 1 -- 172.21.15.23:0/504066922 learned_addr learned my addr 172.21.15.23:0/504066922 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f269179c700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f268c158880 0x7f268c1efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f268bfff700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f268c152ce0 0x7f268c1ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f2690f9b700 1 -- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f268c158880 msgr2=0x7f268c1efc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f2690f9b700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f268c158880 0x7f268c1efc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f2690f9b700 1 -- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f268c152ce0 msgr2=0x7f268c1ef740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:00.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f2690f9b700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f268c152ce0 0x7f268c1ef740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:00.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f2690f9b700 1 -- 172.21.15.23:0/504066922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680009550 con 0x7f268c0c9fd0 2023-06-22T08:00:00.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f268bfff700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f268c152ce0 0x7f268c1ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:00.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f2690f9b700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c0c9fd0 0x7f268c1514f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2680000c00 tx=0x7f2680004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:00.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f26897fa700 1 -- 172.21.15.23:0/504066922 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680004300 con 0x7f268c0c9fd0 2023-06-22T08:00:00.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f26897fa700 1 -- 172.21.15.23:0/504066922 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268001f070 con 0x7f268c0c9fd0 2023-06-22T08:00:00.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f26931ff700 1 -- 172.21.15.23:0/504066922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268c1549e0 con 0x7f268c0c9fd0 2023-06-22T08:00:00.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.881+0000 7f26897fa700 1 -- 172.21.15.23:0/504066922 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26800036a0 con 0x7f268c0c9fd0 2023-06-22T08:00:00.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.882+0000 7f26931ff700 1 -- 172.21.15.23:0/504066922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268c154fa0 con 0x7f268c0c9fd0 2023-06-22T08:00:00.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.883+0000 7f26931ff700 1 -- 172.21.15.23:0/504066922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268c1ef590 con 0x7f268c0c9fd0 2023-06-22T08:00:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.883+0000 7f26897fa700 1 -- 172.21.15.23:0/504066922 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2680021030 con 0x7f268c0c9fd0 2023-06-22T08:00:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.885+0000 7f26897fa700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f267006ace0 0x7f267006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.885+0000 7f26897fa700 1 -- 172.21.15.23:0/504066922 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f268008a950 con 0x7f268c0c9fd0 2023-06-22T08:00:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.885+0000 7f268bfff700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f267006ace0 0x7f267006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.886+0000 7f268bfff700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f267006ace0 0x7f267006d190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f268c0c4750 tx=0x7f2678008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:00.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:00.889+0000 7f26897fa700 1 -- 172.21.15.23:0/504066922 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f26800558c0 con 0x7f268c0c9fd0 2023-06-22T08:00:01.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.098+0000 7f26931ff700 1 -- 172.21.15.23:0/504066922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f268c04ea20 con 0x7f268c0c9fd0 2023-06-22T08:00:01.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.099+0000 7f26897fa700 1 -- 172.21.15.23:0/504066922 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f2680058ee0 con 0x7f268c0c9fd0 2023-06-22T08:00:01.103 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:01.103 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:01.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.101+0000 7f26931ff700 1 -- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f267006ace0 msgr2=0x7f267006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.101+0000 7f26931ff700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f267006ace0 0x7f267006d190 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f268c0c4750 tx=0x7f2678008040 comp rx=0 tx=0).stop 2023-06-22T08:00:01.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.101+0000 7f26931ff700 1 -- 172.21.15.23:0/504066922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c0c9fd0 msgr2=0x7f268c1514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.101+0000 7f26931ff700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c0c9fd0 0x7f268c1514f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2680000c00 tx=0x7f2680004750 comp rx=0 tx=0).stop 2023-06-22T08:00:01.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.102+0000 7f26931ff700 1 -- 172.21.15.23:0/504066922 shutdown_connections 2023-06-22T08:00:01.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.102+0000 7f26931ff700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f268c158880 0x7f268c1efc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.102+0000 7f26931ff700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f268c0c9fd0 0x7f268c1514f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.102+0000 7f26931ff700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f267006ace0 0x7f267006d190 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.102+0000 7f26931ff700 1 --2- 172.21.15.23:0/504066922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f268c152ce0 0x7f268c1ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.102+0000 7f26931ff700 1 -- 172.21.15.23:0/504066922 >> 172.21.15.23:0/504066922 conn(0x7f268c06ae30 msgr2=0x7f268c0c49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.102+0000 7f26931ff700 1 -- 172.21.15.23:0/504066922 shutdown_connections 2023-06-22T08:00:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.102+0000 7f26931ff700 1 -- 172.21.15.23:0/504066922 wait complete. 2023-06-22T08:00:01.136 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:00:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.333+0000 7fd97cbff700 1 Processor -- start 2023-06-22T08:00:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.333+0000 7fd97cbff700 1 -- start start 2023-06-22T08:00:01.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.334+0000 7fd97cbff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd978152ce0 0x7fd9780cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.334+0000 7fd97cbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 0x7fd97815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.334+0000 7fd97cbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97815c370 con 0x7fd978158880 2023-06-22T08:00:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.334+0000 7fd97cbff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9781530b0 con 0x7fd978152ce0 2023-06-22T08:00:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.334+0000 7fd97cbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9780cf6f0 con 0x7fd9780c9fd0 2023-06-22T08:00:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.334+0000 7fd97659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd978152ce0 0x7fd9780cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.334+0000 7fd97659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd978152ce0 0x7fd9780cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60080/0 (socket says 172.21.15.23:60080) 2023-06-22T08:00:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.334+0000 7fd976d9d700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd978158880 0x7fd978158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:33790/0 (socket says 172.21.15.23:33790) 2023-06-22T08:00:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.334+0000 7fd976d9d700 1 -- 172.21.15.23:0/4076873736 learned_addr learned my addr 172.21.15.23:0/4076873736 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.334+0000 7fd975d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 0x7fd97815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd97559a700 1 -- 172.21.15.23:0/4076873736 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230283493 0 0) 0x7fd97815c370 con 0x7fd978158880 2023-06-22T08:00:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd97559a700 1 -- 172.21.15.23:0/4076873736 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9600035b0 con 0x7fd978158880 2023-06-22T08:00:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd97659c700 1 -- 172.21.15.23:0/4076873736 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 msgr2=0x7fd97815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd97659c700 1 --2- 172.21.15.23:0/4076873736 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 0x7fd97815c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd97659c700 1 -- 172.21.15.23:0/4076873736 >> v1:172.21.15.67:6789/0 conn(0x7fd978158880 legacy=0x7fd978158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd97659c700 1 -- 172.21.15.23:0/4076873736 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97815bda0 con 0x7fd978152ce0 2023-06-22T08:00:01.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd975d9b700 1 --2- 172.21.15.23:0/4076873736 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 0x7fd97815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:01.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd97659c700 1 --2- 172.21.15.23:0/4076873736 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 0x7fd9780cf1b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd968009840 tx=0x7fd968009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc03088d6643af72 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:01.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd97559a700 1 -- 172.21.15.23:0/4076873736 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968011830 con 0x7fd978152ce0 2023-06-22T08:00:01.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd97559a700 1 -- 172.21.15.23:0/4076873736 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd968011990 con 0x7fd978152ce0 2023-06-22T08:00:01.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.335+0000 7fd97559a700 1 -- 172.21.15.23:0/4076873736 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968011d20 con 0x7fd978152ce0 2023-06-22T08:00:01.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.336+0000 7fd97cbff700 1 -- 172.21.15.23:0/4076873736 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 msgr2=0x7fd9780cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.336+0000 7fd97cbff700 1 --2- 172.21.15.23:0/4076873736 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 0x7fd9780cf1b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd968009840 tx=0x7fd968009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.336+0000 7fd97cbff700 1 -- 172.21.15.23:0/4076873736 shutdown_connections 2023-06-22T08:00:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.336+0000 7fd97cbff700 1 --2- 172.21.15.23:0/4076873736 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 0x7fd9780cf1b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.336+0000 7fd97cbff700 1 --2- 172.21.15.23:0/4076873736 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 0x7fd97815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.336+0000 7fd97cbff700 1 -- 172.21.15.23:0/4076873736 >> 172.21.15.23:0/4076873736 conn(0x7fd97806ae30 msgr2=0x7fd97806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.336+0000 7fd97cbff700 1 -- 172.21.15.23:0/4076873736 shutdown_connections 2023-06-22T08:00:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.336+0000 7fd97cbff700 1 -- 172.21.15.23:0/4076873736 wait complete. 2023-06-22T08:00:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.338+0000 7fd97cbff700 1 Processor -- start 2023-06-22T08:00:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.338+0000 7fd97cbff700 1 -- start start 2023-06-22T08:00:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd97cbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 0x7fd9781477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd97cbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 0x7fd978150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd97cbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd978158880 0x7fd978151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd97cbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9781eeec0 con 0x7fd978152ce0 2023-06-22T08:00:01.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd97cbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9781ee010 con 0x7fd978158880 2023-06-22T08:00:01.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd97cbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9780ccac0 con 0x7fd9780c9fd0 2023-06-22T08:00:01.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd975d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 0x7fd978150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd975d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 0x7fd978150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60090/0 (socket says 172.21.15.23:60090) 2023-06-22T08:00:01.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd975d9b700 1 -- 172.21.15.23:0/3231252087 learned_addr learned my addr 172.21.15.23:0/3231252087 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:01.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd97659c700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 0x7fd9781477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd976d9d700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd978158880 0x7fd978151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd975d9b700 1 -- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 msgr2=0x7fd9781477c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd975d9b700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 0x7fd9781477c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd975d9b700 1 -- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd978158880 msgr2=0x7fd978151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd975d9b700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd978158880 0x7fd978151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd975d9b700 1 -- 172.21.15.23:0/3231252087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c0094a0 con 0x7fd978152ce0 2023-06-22T08:00:01.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.339+0000 7fd976d9d700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd978158880 0x7fd978151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:01.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.340+0000 7fd975d9b700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 0x7fd978150bd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00ebe0 tx=0x7fd96c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:01.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.340+0000 7fd966ffd700 1 -- 172.21.15.23:0/3231252087 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c00c850 con 0x7fd978152ce0 2023-06-22T08:00:01.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.340+0000 7fd966ffd700 1 -- 172.21.15.23:0/3231252087 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c00c9b0 con 0x7fd978152ce0 2023-06-22T08:00:01.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.340+0000 7fd966ffd700 1 -- 172.21.15.23:0/3231252087 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c00cd40 con 0x7fd978152ce0 2023-06-22T08:00:01.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.340+0000 7fd97cbff700 1 -- 172.21.15.23:0/3231252087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9680094a0 con 0x7fd978152ce0 2023-06-22T08:00:01.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.340+0000 7fd97cbff700 1 -- 172.21.15.23:0/3231252087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9781ee2b0 con 0x7fd978152ce0 2023-06-22T08:00:01.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.341+0000 7fd97cbff700 1 -- 172.21.15.23:0/3231252087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd978158600 con 0x7fd978152ce0 2023-06-22T08:00:01.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.342+0000 7fd966ffd700 1 -- 172.21.15.23:0/3231252087 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd96c013470 con 0x7fd978152ce0 2023-06-22T08:00:01.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.343+0000 7fd966ffd700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd95806ace0 0x7fd95806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.343+0000 7fd966ffd700 1 -- 172.21.15.23:0/3231252087 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd96c0898e0 con 0x7fd978152ce0 2023-06-22T08:00:01.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.343+0000 7fd97659c700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd95806ace0 0x7fd95806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.344+0000 7fd97659c700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd95806ace0 0x7fd95806d190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd968000c00 tx=0x7fd968008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:01.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.347+0000 7fd966ffd700 1 -- 172.21.15.23:0/3231252087 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd96c054610 con 0x7fd978152ce0 2023-06-22T08:00:01.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.596+0000 7fd97cbff700 1 -- 172.21.15.23:0/3231252087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd97804ea20 con 0x7fd978152ce0 2023-06-22T08:00:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.597+0000 7fd966ffd700 1 -- 172.21.15.23:0/3231252087 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6176 (secure 0 0 0) 0x7fd96c017090 con 0x7fd978152ce0 2023-06-22T08:00:01.600 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:01.600 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4268,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:00:00.703306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-06-22T08:00:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 -- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd95806ace0 msgr2=0x7fd95806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd95806ace0 0x7fd95806d190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd968000c00 tx=0x7fd968008030 comp rx=0 tx=0).stop 2023-06-22T08:00:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 -- 172.21.15.23:0/3231252087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 msgr2=0x7fd978150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 0x7fd978150bd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00ebe0 tx=0x7fd96c00eef0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 -- 172.21.15.23:0/3231252087 shutdown_connections 2023-06-22T08:00:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd95806ace0 0x7fd95806d190 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd978158880 0x7fd978151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd9780c9fd0 0x7fd9781477c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 --2- 172.21.15.23:0/3231252087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd978152ce0 0x7fd978150bd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 -- 172.21.15.23:0/3231252087 >> 172.21.15.23:0/3231252087 conn(0x7fd97806ae30 msgr2=0x7fd9780c4620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 -- 172.21.15.23:0/3231252087 shutdown_connections 2023-06-22T08:00:01.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.599+0000 7fd97cbff700 1 -- 172.21.15.23:0/3231252087 wait complete. 2023-06-22T08:00:01.605 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2023-06-22T08:00:01.621 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:01.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.816+0000 7f8225dff700 1 Processor -- start 2023-06-22T08:00:01.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.816+0000 7f8225dff700 1 -- start start 2023-06-22T08:00:01.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f8225dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82200cf1b0 0x7f82200c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f8225dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822015c370 con 0x7f8220152ce0 2023-06-22T08:00:01.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f8225dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822015bda0 con 0x7f82200c9fd0 2023-06-22T08:00:01.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f8225dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82201530b0 con 0x7f82200cf1b0 2023-06-22T08:00:01.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f821f7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f8220152ce0 0x7f822015c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60258/0 (socket says 172.21.15.23:60258) 2023-06-22T08:00:01.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f821f7fe700 1 -- 172.21.15.23:0/2924946234 learned_addr learned my addr 172.21.15.23:0/2924946234 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:01.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f821ffff700 1 --2- 172.21.15.23:0/2924946234 >> v2:172.21.15.67:3300/0 conn(0x7f82200cf1b0 0x7f82200c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f821e7fc700 1 -- 172.21.15.23:0/2924946234 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015251178 0 0) 0x7f822015c370 con 0x7f8220152ce0 2023-06-22T08:00:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f821e7fc700 1 -- 172.21.15.23:0/2924946234 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82080035b0 con 0x7f8220152ce0 2023-06-22T08:00:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f821e7fc700 1 -- 172.21.15.23:0/2924946234 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137058790 0 0) 0x7f822015bda0 con 0x7f82200c9fd0 2023-06-22T08:00:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f821e7fc700 1 -- 172.21.15.23:0/2924946234 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822015c370 con 0x7f82200c9fd0 2023-06-22T08:00:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.818+0000 7f821ffff700 1 -- 172.21.15.23:0/2924946234 >> v1:172.21.15.67:6789/0 conn(0x7f82200c9fd0 legacy=0x7f82200c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.819+0000 7f821ffff700 1 -- 172.21.15.23:0/2924946234 >> v1:172.21.15.23:6789/0 conn(0x7f8220152ce0 legacy=0x7f822015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.819+0000 7f821ffff700 1 -- 172.21.15.23:0/2924946234 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82200cf630 con 0x7f82200cf1b0 2023-06-22T08:00:01.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.819+0000 7f821ffff700 1 --2- 172.21.15.23:0/2924946234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 0x7f82200c47c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8218009d90 tx=0x7f821800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f26ed8c7ccfccca0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:01.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.819+0000 7f821e7fc700 1 -- 172.21.15.23:0/2924946234 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821800e070 con 0x7f82200cf1b0 2023-06-22T08:00:01.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.819+0000 7f821e7fc700 1 -- 172.21.15.23:0/2924946234 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f821800bb20 con 0x7f82200cf1b0 2023-06-22T08:00:01.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.819+0000 7f821e7fc700 1 -- 172.21.15.23:0/2924946234 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8218012430 con 0x7f82200cf1b0 2023-06-22T08:00:01.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.820+0000 7f8225dff700 1 -- 172.21.15.23:0/2924946234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 msgr2=0x7f82200c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.820+0000 7f8225dff700 1 --2- 172.21.15.23:0/2924946234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 0x7f82200c47c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8218009d90 tx=0x7f821800b330 comp rx=0 tx=0).stop 2023-06-22T08:00:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.820+0000 7f8225dff700 1 -- 172.21.15.23:0/2924946234 shutdown_connections 2023-06-22T08:00:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.820+0000 7f8225dff700 1 --2- 172.21.15.23:0/2924946234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 0x7f82200c47c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.820+0000 7f8225dff700 1 -- 172.21.15.23:0/2924946234 >> 172.21.15.23:0/2924946234 conn(0x7f822006ae30 msgr2=0x7f822006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.820+0000 7f8225dff700 1 -- 172.21.15.23:0/2924946234 shutdown_connections 2023-06-22T08:00:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.820+0000 7f8225dff700 1 -- 172.21.15.23:0/2924946234 wait complete. 2023-06-22T08:00:01.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.822+0000 7f8225dff700 1 Processor -- start 2023-06-22T08:00:01.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.822+0000 7f8225dff700 1 -- start start 2023-06-22T08:00:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f8225dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82200c9fd0 0x7f822014c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f8225dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 0x7f82201e3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f8225dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8220152ce0 0x7f82201ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f8225dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822014cad0 con 0x7f8220152ce0 2023-06-22T08:00:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f8225dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82201f2330 con 0x7f82200cf1b0 2023-06-22T08:00:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f8225dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82201f1720 con 0x7f82200c9fd0 2023-06-22T08:00:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f821effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 0x7f82201e3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f821f7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82200c9fd0 0x7f822014c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f821effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 0x7f82201e3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:34318/0 (socket says 172.21.15.23:34318) 2023-06-22T08:00:01.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f821effd700 1 -- 172.21.15.23:0/2647977291 learned_addr learned my addr 172.21.15.23:0/2647977291 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:01.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f821ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8220152ce0 0x7f82201ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f821effd700 1 -- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82200c9fd0 msgr2=0x7f822014c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f821effd700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82200c9fd0 0x7f822014c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f821effd700 1 -- 172.21.15.23:0/2647977291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8220152ce0 msgr2=0x7f82201ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f821effd700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8220152ce0 0x7f82201ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.823+0000 7f821effd700 1 -- 172.21.15.23:0/2647977291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8218009a70 con 0x7f82200cf1b0 2023-06-22T08:00:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.824+0000 7f821effd700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 0x7f82201e3a60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f821400c550 tx=0x7f821400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.824+0000 7f8207fff700 1 -- 172.21.15.23:0/2647977291 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82140052d0 con 0x7f82200cf1b0 2023-06-22T08:00:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.824+0000 7f8207fff700 1 -- 172.21.15.23:0/2647977291 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8214005430 con 0x7f82200cf1b0 2023-06-22T08:00:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.824+0000 7f8207fff700 1 -- 172.21.15.23:0/2647977291 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82140038c0 con 0x7f82200cf1b0 2023-06-22T08:00:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.824+0000 7f8225dff700 1 -- 172.21.15.23:0/2647977291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82201f1a00 con 0x7f82200cf1b0 2023-06-22T08:00:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.824+0000 7f8225dff700 1 -- 172.21.15.23:0/2647977291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82201f1fc0 con 0x7f82200cf1b0 2023-06-22T08:00:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.825+0000 7f8225dff700 1 -- 172.21.15.23:0/2647977291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82201f24e0 con 0x7f82200cf1b0 2023-06-22T08:00:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.826+0000 7f8207fff700 1 -- 172.21.15.23:0/2647977291 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8214003a20 con 0x7f82200cf1b0 2023-06-22T08:00:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.827+0000 7f8207fff700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f820c06ace0 0x7f820c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.827+0000 7f8207fff700 1 -- 172.21.15.23:0/2647977291 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f8214088d40 con 0x7f82200cf1b0 2023-06-22T08:00:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.827+0000 7f821f7fe700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f820c06ace0 0x7f820c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.828+0000 7f821f7fe700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f820c06ace0 0x7f820c06d190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8210007fd0 tx=0x7f8210007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:01.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:01.832+0000 7f8207fff700 1 -- 172.21.15.23:0/2647977291 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8214053950 con 0x7f82200cf1b0 2023-06-22T08:00:02.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.040+0000 7f8225dff700 1 -- 172.21.15.23:0/2647977291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f822004ea20 con 0x7f82200cf1b0 2023-06-22T08:00:02.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.042+0000 7f8207fff700 1 -- 172.21.15.23:0/2647977291 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f8214056f70 con 0x7f82200cf1b0 2023-06-22T08:00:02.045 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:02.045 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:02.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.043+0000 7f8225dff700 1 -- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f820c06ace0 msgr2=0x7f820c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:02.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.043+0000 7f8225dff700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f820c06ace0 0x7f820c06d190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8210007fd0 tx=0x7f8210007de0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.044+0000 7f8225dff700 1 -- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 msgr2=0x7f82201e3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:02.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.044+0000 7f8225dff700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 0x7f82201e3a60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f821400c550 tx=0x7f821400c860 comp rx=0 tx=0).stop 2023-06-22T08:00:02.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.044+0000 7f8225dff700 1 -- 172.21.15.23:0/2647977291 shutdown_connections 2023-06-22T08:00:02.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.044+0000 7f8225dff700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8220152ce0 0x7f82201ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.044+0000 7f8225dff700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82200c9fd0 0x7f822014c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.044+0000 7f8225dff700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f820c06ace0 0x7f820c06d190 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.044+0000 7f8225dff700 1 --2- 172.21.15.23:0/2647977291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82200cf1b0 0x7f82201e3a60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.044+0000 7f8225dff700 1 -- 172.21.15.23:0/2647977291 >> 172.21.15.23:0/2647977291 conn(0x7f822006ae30 msgr2=0x7f82200b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:02.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.044+0000 7f8225dff700 1 -- 172.21.15.23:0/2647977291 shutdown_connections 2023-06-22T08:00:02.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.044+0000 7f8225dff700 1 -- 172.21.15.23:0/2647977291 wait complete. 2023-06-22T08:00:02.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-22T07:59:55.139239+0000', 'modified': '2023-06-22T08:00:00.703306+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4285': {'gid': 4285, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6839/567663545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6838', 'nonce': 567663545}, {'type': 'v1', 'addr': '172.21.15.67:6839', 'nonce': 567663545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-06-22T08:00:02.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-22T08:00:02.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-06-22T08:00:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.866+0000 7f7b825ff700 1 Processor -- start 2023-06-22T08:00:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.866+0000 7f7b825ff700 1 -- start start 2023-06-22T08:00:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.868+0000 7f7b825ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7b7c0cf1b0 0x7f7b7c0c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.868+0000 7f7b825ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c15c370 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.868+0000 7f7b825ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c15bda0 con 0x7f7b7c152ce0 2023-06-22T08:00:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.868+0000 7f7b825ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7c1530b0 con 0x7f7b7c0cf1b0 2023-06-22T08:00:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.868+0000 7f7b7b7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7b7c0c9fd0 0x7f7b7c0c40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60270/0 (socket says 172.21.15.23:60270) 2023-06-22T08:00:02.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.868+0000 7f7b7b7fe700 1 -- 172.21.15.23:0/1350281144 learned_addr learned my addr 172.21.15.23:0/1350281144 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:02.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.868+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/1350281144 >> v2:172.21.15.23:3300/0 conn(0x7f7b7c0cf1b0 0x7f7b7c0c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.868+0000 7f7b7affd700 1 -- 172.21.15.23:0/1350281144 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768937256 0 0) 0x7f7b7c15c370 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.868+0000 7f7b7affd700 1 -- 172.21.15.23:0/1350281144 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b640035b0 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.869+0000 7f7b7affd700 1 -- 172.21.15.23:0/1350281144 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334466009 0 0) 0x7f7b7c15bda0 con 0x7f7b7c152ce0 2023-06-22T08:00:02.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.869+0000 7f7b80b9c700 1 -- 172.21.15.23:0/1350281144 >> v1:172.21.15.67:6789/0 conn(0x7f7b7c152ce0 legacy=0x7f7b7c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.869+0000 7f7b80b9c700 1 -- 172.21.15.23:0/1350281144 >> v1:172.21.15.23:6789/0 conn(0x7f7b7c0c9fd0 legacy=0x7f7b7c0c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.869+0000 7f7b80b9c700 1 -- 172.21.15.23:0/1350281144 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c0cf630 con 0x7f7b7c0cf1b0 2023-06-22T08:00:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.869+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/1350281144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0cf1b0 0x7f7b7c0c47c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009d90 tx=0x7f7b7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7282e85016f511a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.869+0000 7f7b7affd700 1 -- 172.21.15.23:0/1350281144 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7400e070 con 0x7f7b7c0cf1b0 2023-06-22T08:00:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.869+0000 7f7b7affd700 1 -- 172.21.15.23:0/1350281144 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7400bb20 con 0x7f7b7c0cf1b0 2023-06-22T08:00:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.869+0000 7f7b7affd700 1 -- 172.21.15.23:0/1350281144 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7400bed0 con 0x7f7b7c0cf1b0 2023-06-22T08:00:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.870+0000 7f7b825ff700 1 -- 172.21.15.23:0/1350281144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0cf1b0 msgr2=0x7f7b7c0c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.870+0000 7f7b825ff700 1 --2- 172.21.15.23:0/1350281144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0cf1b0 0x7f7b7c0c47c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009d90 tx=0x7f7b7400b330 comp rx=0 tx=0).stop 2023-06-22T08:00:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.870+0000 7f7b825ff700 1 -- 172.21.15.23:0/1350281144 shutdown_connections 2023-06-22T08:00:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.870+0000 7f7b825ff700 1 --2- 172.21.15.23:0/1350281144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0cf1b0 0x7f7b7c0c47c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.870+0000 7f7b825ff700 1 -- 172.21.15.23:0/1350281144 >> 172.21.15.23:0/1350281144 conn(0x7f7b7c06ae30 msgr2=0x7f7b7c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.870+0000 7f7b825ff700 1 -- 172.21.15.23:0/1350281144 shutdown_connections 2023-06-22T08:00:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.870+0000 7f7b825ff700 1 -- 172.21.15.23:0/1350281144 wait complete. 2023-06-22T08:00:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.871+0000 7f7b825ff700 1 Processor -- start 2023-06-22T08:00:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.872+0000 7f7b825ff700 1 -- start start 2023-06-22T08:00:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.872+0000 7f7b825ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0c9fd0 0x7f7b7c14a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b825ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7b7c0cf1b0 0x7f7b7c15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b825ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7c152ce0 0x7f7b7c15b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b825ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c155170 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b825ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c153780 con 0x7f7b7c152ce0 2023-06-22T08:00:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b825ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c14aac0 con 0x7f7b7c0cf1b0 2023-06-22T08:00:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b7bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0c9fd0 0x7f7b7c14a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b7bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0c9fd0 0x7f7b7c14a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60102/0 (socket says 172.21.15.23:60102) 2023-06-22T08:00:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b7bfff700 1 -- 172.21.15.23:0/140767892 learned_addr learned my addr 172.21.15.23:0/140767892 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7c152ce0 0x7f7b7c15b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b7b7fe700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7b7c0cf1b0 0x7f7b7c15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b7bfff700 1 -- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7b7c0cf1b0 msgr2=0x7f7b7c15ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7b7c0cf1b0 0x7f7b7c15ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b7bfff700 1 -- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7c152ce0 msgr2=0x7f7b7c15b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7c152ce0 0x7f7b7c15b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b7bfff700 1 -- 172.21.15.23:0/140767892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b74009a70 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.873+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0c9fd0 0x7f7b7c14a580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00c580 tx=0x7f7b6c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.874+0000 7f7b78ff9700 1 -- 172.21.15.23:0/140767892 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0054e0 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.874+0000 7f7b78ff9700 1 -- 172.21.15.23:0/140767892 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c003680 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.874+0000 7f7b825ff700 1 -- 172.21.15.23:0/140767892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c1534a0 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.874+0000 7f7b78ff9700 1 -- 172.21.15.23:0/140767892 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c003a30 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.874+0000 7f7b825ff700 1 -- 172.21.15.23:0/140767892 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c0c4650 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.876+0000 7f7b78ff9700 1 -- 172.21.15.23:0/140767892 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7b6c003b90 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.877+0000 7f7b78ff9700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7b6806ace0 0x7f7b6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.877+0000 7f7b78ff9700 1 -- 172.21.15.23:0/140767892 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f7b6c088b50 con 0x7f7b7c0c9fd0 2023-06-22T08:00:02.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.877+0000 7f7b7b7fe700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7b6806ace0 0x7f7b6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.878+0000 7f7b7b7fe700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7b6806ace0 0x7f7b6806d190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0ce780 tx=0x7f7b70009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:02.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.879+0000 7f7b95e83b80 1 Processor -- start 2023-06-22T08:00:02.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.880+0000 7f7b95e83b80 1 -- start start 2023-06-22T08:00:02.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f509256150 0x55f509210f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256520 0x55f5092114c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5092137c0 0x55f50923e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f50911bc60 con 0x55f5092137c0 2023-06-22T08:00:02.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f50916e9f0 con 0x55f509256150 2023-06-22T08:00:02.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f509167e80 con 0x55f509256520 2023-06-22T08:00:02.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b80b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5092137c0 0x55f50923e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b80b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5092137c0 0x55f50923e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60116/0 (socket says 172.21.15.23:60116) 2023-06-22T08:00:02.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b7b7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256520 0x55f5092114c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b80b9c700 1 -- 172.21.15.23:0/3892013587 learned_addr learned my addr 172.21.15.23:0/3892013587 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:02.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/3892013587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f509256150 0x55f509210f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b80b9c700 1 -- 172.21.15.23:0/3892013587 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256520 msgr2=0x55f5092114c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:02.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/3892013587 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256520 0x55f5092114c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b80b9c700 1 -- 172.21.15.23:0/3892013587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f509256150 msgr2=0x55f509210f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/3892013587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f509256150 0x55f509210f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.881+0000 7f7b80b9c700 1 -- 172.21.15.23:0/3892013587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f508fe95e0 con 0x55f5092137c0 2023-06-22T08:00:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/3892013587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5092137c0 0x55f50923e880 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b740124e0 tx=0x7f7b74011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cade19784b5846d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b61ffb700 1 -- 172.21.15.23:0/3892013587 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74012780 con 0x55f5092137c0 2023-06-22T08:00:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b61ffb700 1 -- 172.21.15.23:0/3892013587 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b740128e0 con 0x55f5092137c0 2023-06-22T08:00:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b61ffb700 1 -- 172.21.15.23:0/3892013587 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74012c90 con 0x55f5092137c0 2023-06-22T08:00:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3892013587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5092137c0 msgr2=0x55f50923e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3892013587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5092137c0 0x55f50923e880 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b740124e0 tx=0x7f7b74011000 comp rx=0 tx=0).stop 2023-06-22T08:00:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3892013587 shutdown_connections 2023-06-22T08:00:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3892013587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f5092137c0 0x55f50923e880 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3892013587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f509256150 0x55f509210f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3892013587 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256520 0x55f5092114c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.882+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3892013587 >> 172.21.15.23:0/3892013587 conn(0x55f50920bfc0 msgr2=0x55f5092146a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.883+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3892013587 shutdown_connections 2023-06-22T08:00:02.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.883+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3892013587 wait complete. 2023-06-22T08:00:02.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.883+0000 7f7b95e83b80 1 -- start start 2023-06-22T08:00:02.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.884+0000 7f7b95e83b80 1 Processor -- start 2023-06-22T08:00:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 0x55f509258e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256150 0x55f50923d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f50923d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f509107a70 con 0x55f509256520 2023-06-22T08:00:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f509107770 con 0x55f5092137c0 2023-06-22T08:00:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f508feb8d0 con 0x55f509256150 2023-06-22T08:00:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b95e83b80 10 client.0 _ll_register_callbacks cb 0x55f50910d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b80b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f50923d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b80b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f50923d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60124/0 (socket says 172.21.15.23:60124) 2023-06-22T08:00:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b80b9c700 1 -- 172.21.15.23:0/2894248350 learned_addr learned my addr 172.21.15.23:0/2894248350 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 0x55f509258e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b7b7fe700 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256150 0x55f50923d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b80b9c700 1 -- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256150 msgr2=0x55f50923d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256150 0x55f50923d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b80b9c700 1 -- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 msgr2=0x55f509258e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 0x55f509258e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.885+0000 7f7b80b9c700 1 -- 172.21.15.23:0/2894248350 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c0cf630 con 0x55f509256520 2023-06-22T08:00:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.886+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f50923d560 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b74029040 tx=0x7f7b74004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0da2d29c4315555 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.886+0000 7f7b5affd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-06-22T08:00:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.886+0000 7f7b5affd700 1 -- 172.21.15.23:0/2894248350 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74003c30 con 0x55f509256520 2023-06-22T08:00:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.886+0000 7f7b5affd700 1 -- 172.21.15.23:0/2894248350 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f7b74003d90 con 0x55f509256520 2023-06-22T08:00:02.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.886+0000 7f7b95e83b80 1 -- 172.21.15.23:0/2894248350 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f509209780 con 0x55f509256520 2023-06-22T08:00:02.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.886+0000 7f7b5affd700 1 -- 172.21.15.23:0/2894248350 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b74004050 con 0x55f509256520 2023-06-22T08:00:02.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.886+0000 7f7b95e83b80 10 client.4413 fetch_fsmap learned FSMap version 7 2023-06-22T08:00:02.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.886+0000 7f7b95e83b80 1 -- 172.21.15.23:0/2894248350 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55f508f95d90 con 0x55f509256520 2023-06-22T08:00:02.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.886+0000 7f7b5affd700 1 -- 172.21.15.23:0/2894248350 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f509209780 con 0x55f509256520 2023-06-22T08:00:02.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.886+0000 7f7b5affd700 1 -- 172.21.15.23:0/2894248350 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 7) v1 ==== 4212+0+0 (secure 0 0 0) 0x7f7b740041b0 con 0x55f509256520 2023-06-22T08:00:02.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.887+0000 7f7b95e83b80 10 client.4413 fetch_fsmap finished waiting for FSMap version 7 2023-06-22T08:00:02.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.887+0000 7f7b95e83b80 10 client.4413 resolve_mds: resolved name 'f' to daemon mds.f 2023-06-22T08:00:02.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.887+0000 7f7b95e83b80 20 client.4413 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:02.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.887+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55f5092547e0 0x55f509267140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:02.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.887+0000 7f7b95e83b80 4 client.4413 mds_command: new command op to 4285 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-22T08:00:02.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.887+0000 7f7b95e83b80 1 -- 172.21.15.23:0/2894248350 --> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f509160e60 con 0x55f5092547e0 2023-06-22T08:00:02.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.887+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55f5092547e0 0x55f509267140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:02.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.887+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55f5092547e0 0x55f509267140 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18938b689afd2091 server_cookie=76229c1f5be70107 in_seq=0 out_seq=0 2023-06-22T08:00:02.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:02.887+0000 7f7b5affd700 10 client.4413 ms_handle_connect on v2:172.21.15.67:6838/567663545 2023-06-22T08:00:05.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.707+0000 7f7b5affd700 1 -- 172.21.15.23:0/2894248350 <== mds.0 v2:172.21.15.67:6838/567663545 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f509160e60 con 0x55f5092547e0 2023-06-22T08:00:05.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.707+0000 7f7b5affd700 10 client.4413 handle_command_reply: tid=0 2023-06-22T08:00:05.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.707+0000 7f7b95e83b80 1 client.4413 shutdown 2023-06-22T08:00:05.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.707+0000 7f7b95e83b80 1 -- 172.21.15.23:0/2894248350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 msgr2=0x55f50923d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:05.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.707+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f50923d560 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b74029040 tx=0x7f7b74004b40 comp rx=0 tx=0).stop 2023-06-22T08:00:05.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.708+0000 7f7b95e83b80 1 -- 172.21.15.23:0/2894248350 shutdown_connections 2023-06-22T08:00:05.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.708+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55f5092547e0 0x55f509267140 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.708+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f50923d560 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.708+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 0x55f509258e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.708+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/2894248350 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256150 0x55f50923d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.708+0000 7f7b5affd700 0 client.4413 ms_handle_reset on v2:172.21.15.67:6838/567663545 2023-06-22T08:00:05.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.708+0000 7f7b95e83b80 1 -- 172.21.15.23:0/2894248350 >> 172.21.15.23:0/2894248350 conn(0x55f50920bfc0 msgr2=0x55f50923e650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:05.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.708+0000 7f7b95e83b80 1 -- 172.21.15.23:0/2894248350 shutdown_connections 2023-06-22T08:00:05.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.708+0000 7f7b95e83b80 1 -- 172.21.15.23:0/2894248350 wait complete. 2023-06-22T08:00:05.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.708+0000 7f7b95e83b80 20 client.4413 trim_cache size 0 max 16384 2023-06-22T08:00:05.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.727+0000 7f7b95e83b80 1 Processor -- start 2023-06-22T08:00:05.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.727+0000 7f7b95e83b80 1 -- start start 2023-06-22T08:00:05.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 0x55f50926a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:05.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256150 0x55f509257ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:05.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f509217930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:05.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5091f3440 con 0x55f509256520 2023-06-22T08:00:05.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f50923edd0 con 0x55f5092137c0 2023-06-22T08:00:05.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f5092184d0 con 0x55f509256150 2023-06-22T08:00:05.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b80b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f509217930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:05.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b80b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f509217930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60132/0 (socket says 172.21.15.23:60132) 2023-06-22T08:00:05.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b80b9c700 1 -- 172.21.15.23:0/3083182624 learned_addr learned my addr 172.21.15.23:0/3083182624 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:05.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b7b7fe700 1 --2- 172.21.15.23:0/3083182624 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256150 0x55f509257ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:05.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/3083182624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 0x55f50926a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:05.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b80b9c700 1 -- 172.21.15.23:0/3083182624 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256150 msgr2=0x55f509257ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:05.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/3083182624 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256150 0x55f509257ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b80b9c700 1 -- 172.21.15.23:0/3083182624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 msgr2=0x55f50926a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:05.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/3083182624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 0x55f50926a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b80b9c700 1 -- 172.21.15.23:0/3083182624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f508f7ec10 con 0x55f509256520 2023-06-22T08:00:05.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.728+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/3083182624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 0x55f50926a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:05.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.729+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/3083182624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f509217930 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7b74000f80 tx=0x7f7b7402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac75fe363b3d171 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:05.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.729+0000 7f7b627fc700 1 -- 172.21.15.23:0/3083182624 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74004900 con 0x55f509256520 2023-06-22T08:00:05.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.729+0000 7f7b627fc700 1 -- 172.21.15.23:0/3083182624 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b740046c0 con 0x55f509256520 2023-06-22T08:00:05.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.729+0000 7f7b627fc700 1 -- 172.21.15.23:0/3083182624 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b74039c30 con 0x55f509256520 2023-06-22T08:00:05.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.730+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3083182624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 msgr2=0x55f509217930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:05.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.730+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3083182624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f509217930 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7b74000f80 tx=0x7f7b7402e000 comp rx=0 tx=0).stop 2023-06-22T08:00:05.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.730+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3083182624 shutdown_connections 2023-06-22T08:00:05.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.730+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3083182624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256520 0x55f509217930 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.730+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3083182624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f5092137c0 0x55f50926a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.730+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3083182624 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f509256150 0x55f509257ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.730+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3083182624 >> 172.21.15.23:0/3083182624 conn(0x55f50920bfc0 msgr2=0x55f509254c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:05.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.730+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3083182624 shutdown_connections 2023-06-22T08:00:05.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.730+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3083182624 wait complete. 2023-06-22T08:00:05.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.730+0000 7f7b95e83b80 1 -- start start 2023-06-22T08:00:05.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.731+0000 7f7b95e83b80 1 Processor -- start 2023-06-22T08:00:05.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f5092137c0 0x55f509256f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:05.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256150 0x55f5092b74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:05.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b95e83b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f509256520 0x55f50926a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:05.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5092bd650 con 0x55f509256150 2023-06-22T08:00:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f509258bc0 con 0x55f509256520 2023-06-22T08:00:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b95e83b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f509258e10 con 0x55f5092137c0 2023-06-22T08:00:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b95e83b80 10 client.0 _ll_register_callbacks cb 0x55f50910d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b7b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256150 0x55f5092b74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:05.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b7b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256150 0x55f5092b74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60138/0 (socket says 172.21.15.23:60138) 2023-06-22T08:00:05.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b7b7fe700 1 -- 172.21.15.23:0/3612044830 learned_addr learned my addr 172.21.15.23:0/3612044830 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:05.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b80b9c700 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f509256520 0x55f50926a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:05.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f5092137c0 0x55f509256f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:05.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b7b7fe700 1 -- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f5092137c0 msgr2=0x55f509256f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:05.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b7b7fe700 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f5092137c0 0x55f509256f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b7b7fe700 1 -- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f509256520 msgr2=0x55f50926a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:05.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b7b7fe700 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f509256520 0x55f50926a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.732+0000 7f7b7b7fe700 1 -- 172.21.15.23:0/3612044830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f508f95d90 con 0x55f509256150 2023-06-22T08:00:05.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.733+0000 7f7b7b7fe700 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256150 0x55f5092b74b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b70005910 tx=0x7f7b70000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d357991ceaaf82a8 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:05.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.733+0000 7f7b5affd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-06-22T08:00:05.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.733+0000 7f7b5affd700 1 -- 172.21.15.23:0/3612044830 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70015b20 con 0x55f509256150 2023-06-22T08:00:05.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.733+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3612044830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b740128e0 con 0x55f509256150 2023-06-22T08:00:05.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.733+0000 7f7b5affd700 1 -- 172.21.15.23:0/3612044830 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f7b70015c80 con 0x55f509256150 2023-06-22T08:00:05.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.733+0000 7f7b5affd700 1 -- 172.21.15.23:0/3612044830 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b70015e40 con 0x55f509256150 2023-06-22T08:00:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.733+0000 7f7b95e83b80 10 client.4419 fetch_fsmap learned FSMap version 7 2023-06-22T08:00:05.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.733+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3612044830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55f508fbc200 con 0x55f509256150 2023-06-22T08:00:05.742 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:00:05.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.733+0000 7f7b5affd700 1 -- 172.21.15.23:0/3612044830 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b740128e0 con 0x55f509256150 2023-06-22T08:00:05.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.734+0000 7f7b5affd700 1 -- 172.21.15.23:0/3612044830 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 7) v1 ==== 4212+0+0 (secure 0 0 0) 0x7f7b7002d470 con 0x55f509256150 2023-06-22T08:00:05.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.734+0000 7f7b95e83b80 10 client.4419 fetch_fsmap finished waiting for FSMap version 7 2023-06-22T08:00:05.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.734+0000 7f7b95e83b80 10 client.4419 resolve_mds: resolved name 'f' to daemon mds.f 2023-06-22T08:00:05.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.734+0000 7f7b95e83b80 20 client.4419 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.734+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55f509236e60 0x55f509237230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.734+0000 7f7b95e83b80 4 client.4419 mds_command: new command op to 4285 tid=0[{"prefix": "status"}] 2023-06-22T08:00:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.734+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3612044830 --> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f509123000 con 0x55f509236e60 2023-06-22T08:00:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.734+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55f509236e60 0x55f509237230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.734+0000 7f7b5affd700 10 client.4419 ms_handle_connect on v2:172.21.15.67:6838/567663545 2023-06-22T08:00:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.734+0000 7f7b7bfff700 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55f509236e60 0x55f509237230 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dd3eb7010be06dd server_cookie=19ceaaed37354d47 in_seq=0 out_seq=0 2023-06-22T08:00:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b5affd700 1 -- 172.21.15.23:0/3612044830 <== mds.0 v2:172.21.15.67:6838/567663545 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55f509123000 con 0x55f509236e60 2023-06-22T08:00:05.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b5affd700 10 client.4419 handle_command_reply: tid=0 2023-06-22T08:00:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 client.4419 shutdown 2023-06-22T08:00:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3612044830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256150 msgr2=0x55f5092b74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256150 0x55f5092b74b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b70005910 tx=0x7f7b70000bc0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3612044830 shutdown_connections 2023-06-22T08:00:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55f509236e60 0x55f509237230 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f509256520 0x55f50926a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55f5092137c0 0x55f509256f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 --2- 172.21.15.23:0/3612044830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f509256150 0x55f5092b74b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b5affd700 0 client.4419 ms_handle_reset on v2:172.21.15.67:6838/567663545 2023-06-22T08:00:05.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3612044830 >> 172.21.15.23:0/3612044830 conn(0x55f50920bfc0 msgr2=0x55f509254080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:05.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3612044830 shutdown_connections 2023-06-22T08:00:05.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 1 -- 172.21.15.23:0/3612044830 wait complete. 2023-06-22T08:00:05.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.735+0000 7f7b95e83b80 20 client.4419 trim_cache size 0 max 16384 2023-06-22T08:00:05.747 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-06-22T08:00:05.748 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "2f053100-95a5-44db-a4cd-e7710a2fac9d", 2023-06-22T08:00:05.748 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2023-06-22T08:00:05.748 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 4285, 2023-06-22T08:00:05.748 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-06-22T08:00:05.748 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-06-22T08:00:05.748 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-06-22T08:00:05.749 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 10.593775328, 2023-06-22T08:00:05.749 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 6, 2023-06-22T08:00:05.749 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 28, 2023-06-22T08:00:05.749 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 28, 2023-06-22T08:00:05.749 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 16.163258171999999 2023-06-22T08:00:05.749 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-06-22T08:00:05.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 -- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7b6806ace0 msgr2=0x7f7b6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:05.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7b6806ace0 0x7f7b6806d190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0ce780 tx=0x7f7b70009040 comp rx=0 tx=0).stop 2023-06-22T08:00:05.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 -- 172.21.15.23:0/140767892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0c9fd0 msgr2=0x7f7b7c14a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:05.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0c9fd0 0x7f7b7c14a580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00c580 tx=0x7f7b6c00c890 comp rx=0 tx=0).stop 2023-06-22T08:00:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 -- 172.21.15.23:0/140767892 shutdown_connections 2023-06-22T08:00:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7b6806ace0 0x7f7b6806d190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b7c152ce0 0x7f7b7c15b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b7c0c9fd0 0x7f7b7c14a580 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 --2- 172.21.15.23:0/140767892 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7b7c0cf1b0 0x7f7b7c15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 -- 172.21.15.23:0/140767892 >> 172.21.15.23:0/140767892 conn(0x7f7b7c06ae30 msgr2=0x7f7b7c0c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 -- 172.21.15.23:0/140767892 shutdown_connections 2023-06-22T08:00:05.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.736+0000 7f7b825ff700 1 -- 172.21.15.23:0/140767892 wait complete. 2023-06-22T08:00:05.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:00:05.757 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-06-22T08:00:05.758 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-06-22T08:00:05.758 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-22T08:00:05.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.996+0000 7f08acbff700 1 Processor -- start 2023-06-22T08:00:05.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.996+0000 7f08acbff700 1 -- start start 2023-06-22T08:00:06.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.997+0000 7f08acbff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08a8152d70 0x7f08a80c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:06.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.997+0000 7f08acbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08a8159330 0x7f08a80cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:06.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.997+0000 7f08acbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a815bea0 con 0x7f08a80cb800 2023-06-22T08:00:06.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.997+0000 7f08acbff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a80c95e0 con 0x7f08a8152d70 2023-06-22T08:00:06.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.997+0000 7f08acbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a815c160 con 0x7f08a8159330 2023-06-22T08:00:06.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.997+0000 7f08a659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08a8152d70 0x7f08a80c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:06.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.997+0000 7f08a659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08a8152d70 0x7f08a80c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44002/0 (socket says 172.21.15.23:44002) 2023-06-22T08:00:06.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.997+0000 7f08a659c700 1 -- 172.21.15.23:0/1063639848 learned_addr learned my addr 172.21.15.23:0/1063639848 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:06.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.997+0000 7f08a5d9b700 1 --2- 172.21.15.23:0/1063639848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08a8159330 0x7f08a80cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:06.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.998+0000 7f08a659c700 1 -- 172.21.15.23:0/1063639848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08a8159330 msgr2=0x7f08a80cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:06.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.998+0000 7f08a659c700 1 --2- 172.21.15.23:0/1063639848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08a8159330 0x7f08a80cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:06.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.998+0000 7f08a659c700 1 -- 172.21.15.23:0/1063639848 >> v1:172.21.15.67:6789/0 conn(0x7f08a80cb800 legacy=0x7f08a80c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:06.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.998+0000 7f08a659c700 1 -- 172.21.15.23:0/1063639848 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a815c440 con 0x7f08a8152d70 2023-06-22T08:00:06.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.998+0000 7f08a659c700 1 --2- 172.21.15.23:0/1063639848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8152d70 0x7f08a80c97d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f08980098f0 tx=0x7f0898009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2450cc2b57d5adfa server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:06.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.998+0000 7f08a559a700 1 -- 172.21.15.23:0/1063639848 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0898011830 con 0x7f08a8152d70 2023-06-22T08:00:06.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.998+0000 7f08a559a700 1 -- 172.21.15.23:0/1063639848 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0898011990 con 0x7f08a8152d70 2023-06-22T08:00:06.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.998+0000 7f08a559a700 1 -- 172.21.15.23:0/1063639848 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0898011d20 con 0x7f08a8152d70 2023-06-22T08:00:06.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.999+0000 7f08acbff700 1 -- 172.21.15.23:0/1063639848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8152d70 msgr2=0x7f08a80c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:06.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.999+0000 7f08acbff700 1 --2- 172.21.15.23:0/1063639848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8152d70 0x7f08a80c97d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f08980098f0 tx=0x7f0898009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:06.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.999+0000 7f08acbff700 1 -- 172.21.15.23:0/1063639848 shutdown_connections 2023-06-22T08:00:06.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.999+0000 7f08acbff700 1 --2- 172.21.15.23:0/1063639848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8152d70 0x7f08a80c97d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:06.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.999+0000 7f08acbff700 1 --2- 172.21.15.23:0/1063639848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08a8159330 0x7f08a80cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:06.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.999+0000 7f08acbff700 1 -- 172.21.15.23:0/1063639848 >> 172.21.15.23:0/1063639848 conn(0x7f08a806ae30 msgr2=0x7f08a806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:06.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.999+0000 7f08acbff700 1 -- 172.21.15.23:0/1063639848 shutdown_connections 2023-06-22T08:00:06.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:05.999+0000 7f08acbff700 1 -- 172.21.15.23:0/1063639848 wait complete. 2023-06-22T08:00:06.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.001+0000 7f08acbff700 1 Processor -- start 2023-06-22T08:00:06.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.001+0000 7f08acbff700 1 -- start start 2023-06-22T08:00:06.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08acbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a80cb800 0x7f08a80ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:06.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08acbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8152d70 0x7f08a8158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:06.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08acbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08a8159330 0x7f08a8148760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:06.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08acbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a8150fc0 con 0x7f08a8152d70 2023-06-22T08:00:06.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08acbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80c89f0 con 0x7f08a80cb800 2023-06-22T08:00:06.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08acbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a81eeef0 con 0x7f08a8159330 2023-06-22T08:00:06.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a80cb800 0x7f08a80ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:06.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8152d70 0x7f08a8158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:06.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a80cb800 0x7f08a80ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:46622/0 (socket says 172.21.15.23:46622) 2023-06-22T08:00:06.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a659c700 1 -- 172.21.15.23:0/1061899533 learned_addr learned my addr 172.21.15.23:0/1061899533 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:06.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a6d9d700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08a8159330 0x7f08a8148760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:06.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a5d9b700 1 -- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08a8159330 msgr2=0x7f08a8148760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:06.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a5d9b700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08a8159330 0x7f08a8148760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:06.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a5d9b700 1 -- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a80cb800 msgr2=0x7f08a80ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:06.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a5d9b700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a80cb800 0x7f08a80ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:06.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a5d9b700 1 -- 172.21.15.23:0/1061899533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0898009550 con 0x7f08a8152d70 2023-06-22T08:00:06.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.002+0000 7f08a659c700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a80cb800 0x7f08a80ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:06.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.003+0000 7f08a5d9b700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8152d70 0x7f08a8158460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f089c00e870 tx=0x7f089c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:06.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.003+0000 7f0896ffd700 1 -- 172.21.15.23:0/1061899533 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c013680 con 0x7f08a8152d70 2023-06-22T08:00:06.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.003+0000 7f0896ffd700 1 -- 172.21.15.23:0/1061899533 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c0137e0 con 0x7f08a8152d70 2023-06-22T08:00:06.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.003+0000 7f08acbff700 1 -- 172.21.15.23:0/1061899533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a81ef1d0 con 0x7f08a8152d70 2023-06-22T08:00:06.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.003+0000 7f0896ffd700 1 -- 172.21.15.23:0/1061899533 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c013b90 con 0x7f08a8152d70 2023-06-22T08:00:06.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.003+0000 7f08acbff700 1 -- 172.21.15.23:0/1061899533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a81514f0 con 0x7f08a8152d70 2023-06-22T08:00:06.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.004+0000 7f08acbff700 1 -- 172.21.15.23:0/1061899533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a80cbd50 con 0x7f08a8152d70 2023-06-22T08:00:06.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.005+0000 7f0896ffd700 1 -- 172.21.15.23:0/1061899533 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f089c013cf0 con 0x7f08a8152d70 2023-06-22T08:00:06.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.006+0000 7f0896ffd700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f088806ad10 0x7f088806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:06.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.006+0000 7f0896ffd700 1 -- 172.21.15.23:0/1061899533 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f089c089590 con 0x7f08a8152d70 2023-06-22T08:00:06.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.006+0000 7f08a659c700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f088806ad10 0x7f088806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:06.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.007+0000 7f08a659c700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f088806ad10 0x7f088806d1c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0898009130 tx=0x7f0898008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:06.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.011+0000 7f0896ffd700 1 -- 172.21.15.23:0/1061899533 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f089c0542b0 con 0x7f08a8152d70 2023-06-22T08:00:06.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.267+0000 7f08acbff700 1 -- 172.21.15.23:0/1061899533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f08a804ea20 con 0x7f08a8152d70 2023-06-22T08:00:06.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.267+0000 7f0896ffd700 1 -- 172.21.15.23:0/1061899533 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f089c0578d0 con 0x7f08a8152d70 2023-06-22T08:00:06.271 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-06-22T08:00:06.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 -- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f088806ad10 msgr2=0x7f088806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:06.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f088806ad10 0x7f088806d1c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0898009130 tx=0x7f0898008030 comp rx=0 tx=0).stop 2023-06-22T08:00:06.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 -- 172.21.15.23:0/1061899533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8152d70 msgr2=0x7f08a8158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:06.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8152d70 0x7f08a8158460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f089c00e870 tx=0x7f089c00eb80 comp rx=0 tx=0).stop 2023-06-22T08:00:06.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 -- 172.21.15.23:0/1061899533 shutdown_connections 2023-06-22T08:00:06.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08a8159330 0x7f08a8148760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:06.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f088806ad10 0x7f088806d1c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:06.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08a80cb800 0x7f08a80ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:06.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 --2- 172.21.15.23:0/1061899533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8152d70 0x7f08a8158460 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:06.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 -- 172.21.15.23:0/1061899533 >> 172.21.15.23:0/1061899533 conn(0x7f08a806ae30 msgr2=0x7f08a814e650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:06.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 -- 172.21.15.23:0/1061899533 shutdown_connections 2023-06-22T08:00:06.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:06.270+0000 7f08acbff700 1 -- 172.21.15.23:0/1061899533 wait complete. 2023-06-22T08:00:06.291 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-06-22T08:00:06.291 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-22T08:00:06.291 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2023-06-22T08:00:06.291 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-06-22T08:00:06.292 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-06-22T08:00:06.292 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-06-22T08:00:06.292 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-22T08:00:06.292 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-22T08:00:06.293 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-06-22T08:00:06.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:06.293 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2023-06-22T08:00:06.336 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-22T08:00:06.337 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-22T08:00:06.337 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2023-06-22T08:00:06.337 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-06-22T08:00:06.338 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2023-06-22T08:00:06.338 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-06-22T08:00:06.338 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-22T08:00:06.338 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2023-06-22T08:00:06.338 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2023-06-22T08:00:06.338 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-22T08:00:06.338 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2023-06-22T08:00:06.339 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2023-06-22T08:00:06.339 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-22T08:00:06.339 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2023-06-22T08:00:06.339 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2023-06-22T08:00:06.339 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-06-22T08:00:06.340 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 41944sec preferred_lft 41944sec 2023-06-22T08:00:06.340 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2023-06-22T08:00:06.340 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-06-22T08:00:06.340 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-22T08:00:06.340 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2023-06-22T08:00:06.340 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2023-06-22T08:00:06.341 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-06-22T08:00:06.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:06.342 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-06-22T08:00:06.342 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-06-22T08:00:06.343 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2023-06-22T08:00:06.343 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2023-06-22T08:00:06.343 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2023-06-22T08:00:06.343 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-06-22T08:00:06.344 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-06-22T08:00:06.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:06.807 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-22T08:00:06.881 INFO:teuthology.orchestra.run.smithi023.stdout:1 2023-06-22T08:00:06.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:06.885 DEBUG:teuthology.orchestra.run.smithi023:> route 2023-06-22T08:00:06.941 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2023-06-22T08:00:06.941 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-22T08:00:06.941 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-06-22T08:00:06.941 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-06-22T08:00:06.942 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-22T08:00:06.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:06.943 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-06-22T08:00:06.943 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-06-22T08:00:06.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-06-22T08:00:06.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-06-22T08:00:06.944 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-06-22T08:00:06.945 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-06-22T08:00:07.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:07.625 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2023-06-22T08:00:07.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:07.669 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2023-06-22T08:00:07.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:07.751 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-06-22T08:00:07.751 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-06-22T08:00:07.751 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-06-22T08:00:07.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-06-22T08:00:07.752 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-06-22T08:00:07.933 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-06-22T08:00:07.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:07.934 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-06-22T08:00:07.934 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-06-22T08:00:07.934 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-06-22T08:00:07.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-06-22T08:00:07.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-06-22T08:00:07.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-06-22T08:00:07.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-06-22T08:00:07.936 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-06-22T08:00:08.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:08.230 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-06-22T08:00:08.230 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-06-22T08:00:08.230 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2023-06-22T08:00:08.231 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2023-06-22T08:00:08.231 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-06-22T08:00:08.410 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-06-22T08:00:08.411 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-22T08:00:08.411 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-06-22T08:00:08.454 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-06-22T08:00:08.454 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-22T08:00:08.455 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-06-22T08:00:08.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2023-06-22T08:00:08.645 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-22T08:00:08.646 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-22T08:00:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-22T08:00:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-22T08:00:08.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:08.717 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2023-06-22T08:00:08.759 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-22T08:00:08.760 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-06-22T08:00:08.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2023-06-22T08:00:08.873 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-22T08:00:08.873 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-22T08:00:08.941 INFO:teuthology.orchestra.run.smithi023.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-22T08:00:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-22T08:00:08.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:08.944 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2023-06-22T08:00:08.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:2023-06-22T08:00:08.982+0000 7f82bdcd75c0 -1 init, newargv = 0x557ba3ee5290 newargc=15 2023-06-22T08:00:08.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[137150]: starting ceph client 2023-06-22T08:00:08.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[137150]: starting fuse 2023-06-22T08:00:10.090 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2023-06-22T08:00:10.166 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-22T08:00:10.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-22T08:00:10.235 INFO:teuthology.orchestra.run.smithi023.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-22T08:00:10.237 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-22T08:00:10.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:10.238 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2023-06-22T08:00:10.281 INFO:teuthology.orchestra.run.smithi023.stdout:43 2023-06-22T08:00:10.282 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2023-06-22T08:00:10.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-22T08:00:10.282 DEBUG:teuthology.orchestra.run.smithi023:> import glob 2023-06-22T08:00:10.283 DEBUG:teuthology.orchestra.run.smithi023:> import re 2023-06-22T08:00:10.283 DEBUG:teuthology.orchestra.run.smithi023:> import os 2023-06-22T08:00:10.283 DEBUG:teuthology.orchestra.run.smithi023:> import subprocess 2023-06-22T08:00:10.283 DEBUG:teuthology.orchestra.run.smithi023:> 2023-06-22T08:00:10.284 DEBUG:teuthology.orchestra.run.smithi023:> def _find_admin_socket(client_name): 2023-06-22T08:00:10.284 DEBUG:teuthology.orchestra.run.smithi023:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-06-22T08:00:10.284 DEBUG:teuthology.orchestra.run.smithi023:> files = glob.glob(asok_path) 2023-06-22T08:00:10.284 DEBUG:teuthology.orchestra.run.smithi023:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-06-22T08:00:10.285 DEBUG:teuthology.orchestra.run.smithi023:> 2023-06-22T08:00:10.285 DEBUG:teuthology.orchestra.run.smithi023:> # Given a non-glob path, it better be there 2023-06-22T08:00:10.285 DEBUG:teuthology.orchestra.run.smithi023:> if "*" not in asok_path: 2023-06-22T08:00:10.285 DEBUG:teuthology.orchestra.run.smithi023:> assert(len(files) == 1) 2023-06-22T08:00:10.286 DEBUG:teuthology.orchestra.run.smithi023:> return files[0] 2023-06-22T08:00:10.286 DEBUG:teuthology.orchestra.run.smithi023:> 2023-06-22T08:00:10.286 DEBUG:teuthology.orchestra.run.smithi023:> for f in files: 2023-06-22T08:00:10.286 DEBUG:teuthology.orchestra.run.smithi023:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-22T08:00:10.287 DEBUG:teuthology.orchestra.run.smithi023:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-22T08:00:10.287 DEBUG:teuthology.orchestra.run.smithi023:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-22T08:00:10.287 DEBUG:teuthology.orchestra.run.smithi023:> contents = proc_f.read() 2023-06-22T08:00:10.288 DEBUG:teuthology.orchestra.run.smithi023:> if mountpoint in contents: 2023-06-22T08:00:10.288 DEBUG:teuthology.orchestra.run.smithi023:> return f 2023-06-22T08:00:10.288 DEBUG:teuthology.orchestra.run.smithi023:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-22T08:00:10.288 DEBUG:teuthology.orchestra.run.smithi023:> 2023-06-22T08:00:10.289 DEBUG:teuthology.orchestra.run.smithi023:> print(_find_admin_socket("client.admin")) 2023-06-22T08:00:10.289 DEBUG:teuthology.orchestra.run.smithi023:> ' 2023-06-22T08:00:10.465 INFO:teuthology.orchestra.run.smithi023.stdout:/var/run/ceph/ceph-client.admin.137150.asok 2023-06-22T08:00:10.596 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:00:10.594+0000 7fa5334b2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-22T08:00:10.643 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.137150.asok 2023-06-22T08:00:10.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:10.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.137150.asok status 2023-06-22T08:00:10.870 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-06-22T08:00:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "metadata": { 2023-06-22T08:00:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_sha1": "8627af3e0adcb765a3c249fcc209cba9f4873e1b", 2023-06-22T08:00:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 18.0.0-4479-g8627af3e (8627af3e0adcb765a3c249fcc209cba9f4873e1b) reef (dev)", 2023-06-22T08:00:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "entity_id": "admin", 2023-06-22T08:00:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": "smithi023", 2023-06-22T08:00:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-06-22T08:00:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "pid": "137150", 2023-06-22T08:00:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "root": "/" 2023-06-22T08:00:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-06-22T08:00:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_count": 0, 2023-06-22T08:00:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_pinned_count": 0, 2023-06-22T08:00:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 4431, 2023-06-22T08:00:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "inst": { 2023-06-22T08:00:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "name": { 2023-06-22T08:00:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "client", 2023-06-22T08:00:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4431 2023-06-22T08:00:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-06-22T08:00:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2023-06-22T08:00:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2023-06-22T08:00:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "192.168.0.1:0", 2023-06-22T08:00:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 384862148 2023-06-22T08:00:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: } 2023-06-22T08:00:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-06-22T08:00:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2023-06-22T08:00:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2023-06-22T08:00:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "192.168.0.1:0", 2023-06-22T08:00:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 384862148 2023-06-22T08:00:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2023-06-22T08:00:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "inst_str": "client.4431 192.168.0.1:0/384862148", 2023-06-22T08:00:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "addr_str": "192.168.0.1:0/384862148", 2023-06-22T08:00:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "inode_count": 1, 2023-06-22T08:00:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_epoch": 6, 2023-06-22T08:00:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch": 28, 2023-06-22T08:00:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch_barrier": 0, 2023-06-22T08:00:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklisted": false, 2023-06-22T08:00:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs" 2023-06-22T08:00:10.876 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-06-22T08:00:10.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:10.885 DEBUG:teuthology.orchestra.run.smithi023:> stat --file-system '--printf=%T 2023-06-22T08:00:10.885 DEBUG:teuthology.orchestra.run.smithi023:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-06-22T08:00:10.928 INFO:teuthology.orchestra.run.smithi023.stdout:fuseblk 2023-06-22T08:00:10.928 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-06-22T08:00:10.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:10.929 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-06-22T08:00:11.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:11.038 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-06-22T08:00:11.160 INFO:teuthology.orchestra.run.smithi023.stdout:# file: . 2023-06-22T08:00:11.161 INFO:teuthology.orchestra.run.smithi023.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-06-22T08:00:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:11.162 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T08:00:11.162 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-22T08:00:11.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:11.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.440+0000 7fd8119ff700 1 Processor -- start 2023-06-22T08:00:11.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.441+0000 7fd8119ff700 1 -- start start 2023-06-22T08:00:11.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.442+0000 7fd8119ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c152d70 0x7fd80c0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:11.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.442+0000 7fd8119ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c15bea0 con 0x7fd80c159330 2023-06-22T08:00:11.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.442+0000 7fd8119ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80c15c160 con 0x7fd80c0c45f0 2023-06-22T08:00:11.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.442+0000 7fd8119ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c0c95e0 con 0x7fd80c152d70 2023-06-22T08:00:11.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.442+0000 7fd80a7fc700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fd80c159330 0x7fd80c0cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:49206/0 (socket says 172.21.15.23:49206) 2023-06-22T08:00:11.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.442+0000 7fd80a7fc700 1 -- 172.21.15.23:0/3588733629 learned_addr learned my addr 172.21.15.23:0/3588733629 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.442+0000 7fd80affd700 1 --2- 172.21.15.23:0/3588733629 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c152d70 0x7fd80c0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd809ffb700 1 -- 172.21.15.23:0/3588733629 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778427492 0 0) 0x7fd80c15bea0 con 0x7fd80c159330 2023-06-22T08:00:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd809ffb700 1 -- 172.21.15.23:0/3588733629 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f40035b0 con 0x7fd80c159330 2023-06-22T08:00:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd809ffb700 1 -- 172.21.15.23:0/3588733629 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299627436 0 0) 0x7fd80c15c160 con 0x7fd80c0c45f0 2023-06-22T08:00:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd809ffb700 1 -- 172.21.15.23:0/3588733629 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd80c15bea0 con 0x7fd80c0c45f0 2023-06-22T08:00:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd80affd700 1 -- 172.21.15.23:0/3588733629 >> v1:172.21.15.67:6789/0 conn(0x7fd80c0c45f0 legacy=0x7fd80c0c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd80affd700 1 -- 172.21.15.23:0/3588733629 >> v1:172.21.15.23:6789/0 conn(0x7fd80c159330 legacy=0x7fd80c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd80affd700 1 -- 172.21.15.23:0/3588733629 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c15c440 con 0x7fd80c152d70 2023-06-22T08:00:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd80affd700 1 --2- 172.21.15.23:0/3588733629 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c152d70 0x7fd80c0c97d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009870 tx=0x7fd7fc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bbf0cd028ca7f03 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd809ffb700 1 -- 172.21.15.23:0/3588733629 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc011830 con 0x7fd80c152d70 2023-06-22T08:00:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd809ffb700 1 -- 172.21.15.23:0/3588733629 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc011990 con 0x7fd80c152d70 2023-06-22T08:00:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.443+0000 7fd809ffb700 1 -- 172.21.15.23:0/3588733629 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc011d20 con 0x7fd80c152d70 2023-06-22T08:00:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.444+0000 7fd8119ff700 1 -- 172.21.15.23:0/3588733629 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c152d70 msgr2=0x7fd80c0c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.444+0000 7fd8119ff700 1 --2- 172.21.15.23:0/3588733629 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c152d70 0x7fd80c0c97d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009870 tx=0x7fd7fc009b80 comp rx=0 tx=0).stop 2023-06-22T08:00:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.444+0000 7fd8119ff700 1 -- 172.21.15.23:0/3588733629 shutdown_connections 2023-06-22T08:00:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.444+0000 7fd8119ff700 1 --2- 172.21.15.23:0/3588733629 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c152d70 0x7fd80c0c97d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.444+0000 7fd8119ff700 1 -- 172.21.15.23:0/3588733629 >> 172.21.15.23:0/3588733629 conn(0x7fd80c06ae30 msgr2=0x7fd80c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.444+0000 7fd8119ff700 1 -- 172.21.15.23:0/3588733629 shutdown_connections 2023-06-22T08:00:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.444+0000 7fd8119ff700 1 -- 172.21.15.23:0/3588733629 wait complete. 2023-06-22T08:00:11.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.446+0000 7fd8119ff700 1 Processor -- start 2023-06-22T08:00:11.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.446+0000 7fd8119ff700 1 -- start start 2023-06-22T08:00:11.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd8119ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd80c0c45f0 0x7fd80c1f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:11.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd8119ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd80c152d70 0x7fd80c0ce750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:11.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd8119ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c159330 0x7fd80c0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:11.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd8119ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c158b30 con 0x7fd80c152d70 2023-06-22T08:00:11.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd8119ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c0b4670 con 0x7fd80c0c45f0 2023-06-22T08:00:11.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd8119ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80c0b40b0 con 0x7fd80c159330 2023-06-22T08:00:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd80a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd80c152d70 0x7fd80c0ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd80a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd80c152d70 0x7fd80c0ce750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44020/0 (socket says 172.21.15.23:44020) 2023-06-22T08:00:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd80a7fc700 1 -- 172.21.15.23:0/3701099128 learned_addr learned my addr 172.21.15.23:0/3701099128 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd80b7fe700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c159330 0x7fd80c0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.447+0000 7fd80affd700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd80c0c45f0 0x7fd80c1f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd80a7fc700 1 -- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c159330 msgr2=0x7fd80c0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd80a7fc700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c159330 0x7fd80c0ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd80a7fc700 1 -- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd80c0c45f0 msgr2=0x7fd80c1f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd80a7fc700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd80c0c45f0 0x7fd80c1f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd80a7fc700 1 -- 172.21.15.23:0/3701099128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7fc0094a0 con 0x7fd80c152d70 2023-06-22T08:00:11.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd80a7fc700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd80c152d70 0x7fd80c0ce750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd800001eb0 tx=0x7fd80000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:11.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd7f3fff700 1 -- 172.21.15.23:0/3701099128 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800005d70 con 0x7fd80c152d70 2023-06-22T08:00:11.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd7f3fff700 1 -- 172.21.15.23:0/3701099128 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8000070d0 con 0x7fd80c152d70 2023-06-22T08:00:11.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd7f3fff700 1 -- 172.21.15.23:0/3701099128 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800007460 con 0x7fd80c152d70 2023-06-22T08:00:11.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd8119ff700 1 -- 172.21.15.23:0/3701099128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80c0b4390 con 0x7fd80c152d70 2023-06-22T08:00:11.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.448+0000 7fd8119ff700 1 -- 172.21.15.23:0/3701099128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80c158850 con 0x7fd80c152d70 2023-06-22T08:00:11.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.449+0000 7fd8119ff700 1 -- 172.21.15.23:0/3701099128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80c04ea20 con 0x7fd80c152d70 2023-06-22T08:00:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.450+0000 7fd7f3fff700 1 -- 172.21.15.23:0/3701099128 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd8000075c0 con 0x7fd80c152d70 2023-06-22T08:00:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.451+0000 7fd7f3fff700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd7f806ace0 0x7fd7f806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.451+0000 7fd7f3fff700 1 -- 172.21.15.23:0/3701099128 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd800088c70 con 0x7fd80c152d70 2023-06-22T08:00:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.451+0000 7fd80affd700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd7f806ace0 0x7fd7f806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.452+0000 7fd80affd700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd7f806ace0 0x7fd7f806d190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0097c0 tx=0x7fd7fc01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:11.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.456+0000 7fd7f3fff700 1 -- 172.21.15.23:0/3701099128 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd800053990 con 0x7fd80c152d70 2023-06-22T08:00:11.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.667+0000 7fd8119ff700 1 -- 172.21.15.23:0/3701099128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd80c0c9e00 con 0x7fd80c152d70 2023-06-22T08:00:11.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.668+0000 7fd7f3fff700 1 -- 172.21.15.23:0/3701099128 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7fd800056fb0 con 0x7fd80c152d70 2023-06-22T08:00:11.673 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:11.673 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:11.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 -- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd7f806ace0 msgr2=0x7fd7f806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd7f806ace0 0x7fd7f806d190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0097c0 tx=0x7fd7fc01c040 comp rx=0 tx=0).stop 2023-06-22T08:00:11.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 -- 172.21.15.23:0/3701099128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd80c152d70 msgr2=0x7fd80c0ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd80c152d70 0x7fd80c0ce750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd800001eb0 tx=0x7fd80000c7b0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 -- 172.21.15.23:0/3701099128 shutdown_connections 2023-06-22T08:00:11.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd80c159330 0x7fd80c0ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd7f806ace0 0x7fd7f806d190 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd80c0c45f0 0x7fd80c1f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 --2- 172.21.15.23:0/3701099128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd80c152d70 0x7fd80c0ce750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 -- 172.21.15.23:0/3701099128 >> 172.21.15.23:0/3701099128 conn(0x7fd80c06ae30 msgr2=0x7fd80c14e430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:11.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 -- 172.21.15.23:0/3701099128 shutdown_connections 2023-06-22T08:00:11.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.671+0000 7fd8119ff700 1 -- 172.21.15.23:0/3701099128 wait complete. 2023-06-22T08:00:11.693 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T07:59:40.617747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-22T07:59:48.190528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-22T07:59:53.576494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-22T07:59:54.105472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-22T07:59:57.198896+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-22T08:00:11.694 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T08:00:11.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.891+0000 7fbdd17ff700 1 Processor -- start 2023-06-22T08:00:11.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.891+0000 7fbdd17ff700 1 -- start start 2023-06-22T08:00:11.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.892+0000 7fbdd17ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 0x7fbdcc0ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:11.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.892+0000 7fbdd17ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdcc0c9aa0 con 0x7fbdcc1593f0 2023-06-22T08:00:11.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.892+0000 7fbdd17ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdcc0ca880 con 0x7fbdcc152ba0 2023-06-22T08:00:11.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.892+0000 7fbdd17ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc06aed0 con 0x7fbdcc0c9ee0 2023-06-22T08:00:11.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.892+0000 7fbdca7fc700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbdcc1593f0 0x7fbdcc0c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:49212/0 (socket says 172.21.15.23:49212) 2023-06-22T08:00:11.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.892+0000 7fbdca7fc700 1 -- 172.21.15.23:0/3173257309 learned_addr learned my addr 172.21.15.23:0/3173257309 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:11.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.892+0000 7fbdcb7fe700 1 --2- 172.21.15.23:0/3173257309 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 0x7fbdcc0ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:11.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdc9ffb700 1 -- 172.21.15.23:0/3173257309 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602196922 0 0) 0x7fbdcc0c9aa0 con 0x7fbdcc1593f0 2023-06-22T08:00:11.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdc9ffb700 1 -- 172.21.15.23:0/3173257309 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb40035b0 con 0x7fbdcc1593f0 2023-06-22T08:00:11.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdc9ffb700 1 -- 172.21.15.23:0/3173257309 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015532634 0 0) 0x7fbdcc0ca880 con 0x7fbdcc152ba0 2023-06-22T08:00:11.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdc9ffb700 1 -- 172.21.15.23:0/3173257309 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdcc0c9aa0 con 0x7fbdcc152ba0 2023-06-22T08:00:11.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdcb7fe700 1 -- 172.21.15.23:0/3173257309 >> v1:172.21.15.67:6789/0 conn(0x7fbdcc152ba0 legacy=0x7fbdcc0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdcb7fe700 1 -- 172.21.15.23:0/3173257309 >> v1:172.21.15.23:6789/0 conn(0x7fbdcc1593f0 legacy=0x7fbdcc0c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdcb7fe700 1 -- 172.21.15.23:0/3173257309 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdcc06b1b0 con 0x7fbdcc0c9ee0 2023-06-22T08:00:11.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdcb7fe700 1 --2- 172.21.15.23:0/3173257309 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 0x7fbdcc0ca340 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009dc0 tx=0x7fbdc400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbadf5edc524ae80 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:11.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdc9ffb700 1 -- 172.21.15.23:0/3173257309 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc400e070 con 0x7fbdcc0c9ee0 2023-06-22T08:00:11.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdc9ffb700 1 -- 172.21.15.23:0/3173257309 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc400bb20 con 0x7fbdcc0c9ee0 2023-06-22T08:00:11.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.893+0000 7fbdc9ffb700 1 -- 172.21.15.23:0/3173257309 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4012430 con 0x7fbdcc0c9ee0 2023-06-22T08:00:11.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.894+0000 7fbdd17ff700 1 -- 172.21.15.23:0/3173257309 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 msgr2=0x7fbdcc0ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.894+0000 7fbdd17ff700 1 --2- 172.21.15.23:0/3173257309 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 0x7fbdcc0ca340 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbdc4009dc0 tx=0x7fbdc400b330 comp rx=0 tx=0).stop 2023-06-22T08:00:11.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.894+0000 7fbdd17ff700 1 -- 172.21.15.23:0/3173257309 shutdown_connections 2023-06-22T08:00:11.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.894+0000 7fbdd17ff700 1 --2- 172.21.15.23:0/3173257309 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 0x7fbdcc0ca340 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.894+0000 7fbdd17ff700 1 -- 172.21.15.23:0/3173257309 >> 172.21.15.23:0/3173257309 conn(0x7fbdcc065b90 msgr2=0x7fbdcc065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:11.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.894+0000 7fbdd17ff700 1 -- 172.21.15.23:0/3173257309 shutdown_connections 2023-06-22T08:00:11.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.894+0000 7fbdd17ff700 1 -- 172.21.15.23:0/3173257309 wait complete. 2023-06-22T08:00:11.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.896+0000 7fbdd17ff700 1 Processor -- start 2023-06-22T08:00:11.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.896+0000 7fbdd17ff700 1 -- start start 2023-06-22T08:00:11.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdd17ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 0x7fbdcc0c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:11.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdd17ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdcc152ba0 0x7fbdcc14af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:11.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdd17ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc1593f0 0x7fbdcc14b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:11.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdd17ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc0ce220 con 0x7fbdcc152ba0 2023-06-22T08:00:11.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdd17ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc0c6610 con 0x7fbdcc1593f0 2023-06-22T08:00:11.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdd17ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc0c5d90 con 0x7fbdcc0c9ee0 2023-06-22T08:00:11.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdca7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdcc152ba0 0x7fbdcc14af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:11.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdca7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdcc152ba0 0x7fbdcc14af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44034/0 (socket says 172.21.15.23:44034) 2023-06-22T08:00:11.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdca7fc700 1 -- 172.21.15.23:0/2379139472 learned_addr learned my addr 172.21.15.23:0/2379139472 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:11.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdcb7fe700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc1593f0 0x7fbdcc14b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:11.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdcaffd700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 0x7fbdcc0c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:11.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdca7fc700 1 -- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 msgr2=0x7fbdcc0c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.897+0000 7fbdca7fc700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 0x7fbdcc0c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.898+0000 7fbdca7fc700 1 -- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc1593f0 msgr2=0x7fbdcc14b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:11.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.898+0000 7fbdca7fc700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc1593f0 0x7fbdcc14b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:11.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.898+0000 7fbdca7fc700 1 -- 172.21.15.23:0/2379139472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc4009a70 con 0x7fbdcc152ba0 2023-06-22T08:00:11.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.898+0000 7fbdca7fc700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdcc152ba0 0x7fbdcc14af20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000c700 tx=0x7fbdc000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:11.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.898+0000 7fbdb37fe700 1 -- 172.21.15.23:0/2379139472 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0005d70 con 0x7fbdcc152ba0 2023-06-22T08:00:11.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.898+0000 7fbdb37fe700 1 -- 172.21.15.23:0/2379139472 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc00070d0 con 0x7fbdcc152ba0 2023-06-22T08:00:11.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.898+0000 7fbdb37fe700 1 -- 172.21.15.23:0/2379139472 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0007440 con 0x7fbdcc152ba0 2023-06-22T08:00:11.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.898+0000 7fbdd17ff700 1 -- 172.21.15.23:0/2379139472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdcc1ee980 con 0x7fbdcc152ba0 2023-06-22T08:00:11.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.898+0000 7fbdd17ff700 1 -- 172.21.15.23:0/2379139472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdcc0c6050 con 0x7fbdcc152ba0 2023-06-22T08:00:11.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.899+0000 7fbdd17ff700 1 -- 172.21.15.23:0/2379139472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdcc0cbed0 con 0x7fbdcc152ba0 2023-06-22T08:00:11.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.900+0000 7fbdb37fe700 1 -- 172.21.15.23:0/2379139472 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbdc00075a0 con 0x7fbdcc152ba0 2023-06-22T08:00:11.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.901+0000 7fbdb37fe700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdb806ace0 0x7fbdb806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:11.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.901+0000 7fbdb37fe700 1 -- 172.21.15.23:0/2379139472 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fbdc0088b30 con 0x7fbdcc152ba0 2023-06-22T08:00:11.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.901+0000 7fbdcaffd700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdb806ace0 0x7fbdb806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:11.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.902+0000 7fbdcaffd700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdb806ace0 0x7fbdb806d190 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc0011e0 tx=0x7fbdbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:11.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:11.906+0000 7fbdb37fe700 1 -- 172.21.15.23:0/2379139472 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbdc0053900 con 0x7fbdcc152ba0 2023-06-22T08:00:12.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.113+0000 7fbdd17ff700 1 -- 172.21.15.23:0/2379139472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbdcc0c6870 con 0x7fbdcc152ba0 2023-06-22T08:00:12.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.114+0000 7fbdb37fe700 1 -- 172.21.15.23:0/2379139472 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbdcc0c6870 con 0x7fbdcc152ba0 2023-06-22T08:00:12.118 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-06-22T08:00:12.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 -- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdb806ace0 msgr2=0x7fbdb806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdb806ace0 0x7fbdb806d190 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc0011e0 tx=0x7fbdbc009040 comp rx=0 tx=0).stop 2023-06-22T08:00:12.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 -- 172.21.15.23:0/2379139472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdcc152ba0 msgr2=0x7fbdcc14af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdcc152ba0 0x7fbdcc14af20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000c700 tx=0x7fbdc000ca10 comp rx=0 tx=0).stop 2023-06-22T08:00:12.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 -- 172.21.15.23:0/2379139472 shutdown_connections 2023-06-22T08:00:12.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdb806ace0 0x7fbdb806d190 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdcc1593f0 0x7fbdcc14b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdcc0c9ee0 0x7fbdcc0c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 --2- 172.21.15.23:0/2379139472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdcc152ba0 0x7fbdcc14af20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 -- 172.21.15.23:0/2379139472 >> 172.21.15.23:0/2379139472 conn(0x7fbdcc065b90 msgr2=0x7fbdcc0c3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:12.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 -- 172.21.15.23:0/2379139472 shutdown_connections 2023-06-22T08:00:12.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.116+0000 7fbdd17ff700 1 -- 172.21.15.23:0/2379139472 wait complete. 2023-06-22T08:00:12.138 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T08:00:12.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.336+0000 7fa5b7bff700 1 Processor -- start 2023-06-22T08:00:12.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.337+0000 7fa5b7bff700 1 -- start start 2023-06-22T08:00:12.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.338+0000 7fa5b7bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa5b01593f0 0x7fa5b00c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.338+0000 7fa5b7bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa5b00c4650 0x7fa5b00c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.338+0000 7fa5b7bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b00c9aa0 con 0x7fa5b0152ba0 2023-06-22T08:00:12.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.338+0000 7fa5b7bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b006aed0 con 0x7fa5b01593f0 2023-06-22T08:00:12.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.338+0000 7fa5b7bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b006b1b0 con 0x7fa5b00c4650 2023-06-22T08:00:12.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.338+0000 7fa5b519a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa5b01593f0 0x7fa5b00c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.338+0000 7fa5b599b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa5b0152ba0 0x7fa5b00c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:49218/0 (socket says 172.21.15.23:49218) 2023-06-22T08:00:12.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.338+0000 7fa5b599b700 1 -- 172.21.15.23:0/2488886201 learned_addr learned my addr 172.21.15.23:0/2488886201 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:12.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.338+0000 7fa5b619c700 1 --2- 172.21.15.23:0/2488886201 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa5b00c4650 0x7fa5b00c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.339+0000 7fa5b519a700 1 -- 172.21.15.23:0/2488886201 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa5b00c4650 msgr2=0x7fa5b00c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:12.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.339+0000 7fa5b519a700 1 --2- 172.21.15.23:0/2488886201 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa5b00c4650 0x7fa5b00c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.339+0000 7fa5b519a700 1 -- 172.21.15.23:0/2488886201 >> v1:172.21.15.23:6789/0 conn(0x7fa5b0152ba0 legacy=0x7fa5b00c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.339+0000 7fa5b519a700 1 -- 172.21.15.23:0/2488886201 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b006b490 con 0x7fa5b01593f0 2023-06-22T08:00:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.339+0000 7fa5b519a700 1 --2- 172.21.15.23:0/2488886201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b01593f0 0x7fa5b00c4110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009840 tx=0x7fa5a0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ddd6434cc487362 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.339+0000 7fa5b4999700 1 -- 172.21.15.23:0/2488886201 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0011830 con 0x7fa5b01593f0 2023-06-22T08:00:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.339+0000 7fa5b4999700 1 -- 172.21.15.23:0/2488886201 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0011990 con 0x7fa5b01593f0 2023-06-22T08:00:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.339+0000 7fa5b4999700 1 -- 172.21.15.23:0/2488886201 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0011d20 con 0x7fa5b01593f0 2023-06-22T08:00:12.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.340+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2488886201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b01593f0 msgr2=0x7fa5b00c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.340+0000 7fa5b7bff700 1 --2- 172.21.15.23:0/2488886201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b01593f0 0x7fa5b00c4110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009840 tx=0x7fa5a0009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.340+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2488886201 shutdown_connections 2023-06-22T08:00:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.340+0000 7fa5b7bff700 1 --2- 172.21.15.23:0/2488886201 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa5b00c4650 0x7fa5b00c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.340+0000 7fa5b7bff700 1 --2- 172.21.15.23:0/2488886201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b01593f0 0x7fa5b00c4110 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.340+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2488886201 >> 172.21.15.23:0/2488886201 conn(0x7fa5b0065b90 msgr2=0x7fa5b0065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:12.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.340+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2488886201 shutdown_connections 2023-06-22T08:00:12.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.340+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2488886201 wait complete. 2023-06-22T08:00:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.342+0000 7fa5b7bff700 1 Processor -- start 2023-06-22T08:00:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.342+0000 7fa5b7bff700 1 -- start start 2023-06-22T08:00:12.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b7bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b00c4650 0x7fa5b01f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b7bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b0152ba0 0x7fa5b01e4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b7bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa5b01593f0 0x7fa5b00b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b7bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b014c0c0 con 0x7fa5b0152ba0 2023-06-22T08:00:12.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b7bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b01f2470 con 0x7fa5b00c4650 2023-06-22T08:00:12.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b7bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00c62f0 con 0x7fa5b01593f0 2023-06-22T08:00:12.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b519a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b0152ba0 0x7fa5b01e4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b519a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b0152ba0 0x7fa5b01e4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44040/0 (socket says 172.21.15.23:44040) 2023-06-22T08:00:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b519a700 1 -- 172.21.15.23:0/2700082756 learned_addr learned my addr 172.21.15.23:0/2700082756 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b599b700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b00c4650 0x7fa5b01f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b619c700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa5b01593f0 0x7fa5b00b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b519a700 1 -- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa5b01593f0 msgr2=0x7fa5b00b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b519a700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa5b01593f0 0x7fa5b00b35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b519a700 1 -- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b00c4650 msgr2=0x7fa5b01f1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b519a700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b00c4650 0x7fa5b01f1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b519a700 1 -- 172.21.15.23:0/2700082756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a00094a0 con 0x7fa5b0152ba0 2023-06-22T08:00:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b599b700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b00c4650 0x7fa5b01f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.343+0000 7fa5b519a700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b0152ba0 0x7fa5b01e4e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009b90 tx=0x7fa5a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.344+0000 7fa5a67fc700 1 -- 172.21.15.23:0/2700082756 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a00036a0 con 0x7fa5b0152ba0 2023-06-22T08:00:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.344+0000 7fa5a67fc700 1 -- 172.21.15.23:0/2700082756 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0003800 con 0x7fa5b0152ba0 2023-06-22T08:00:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.344+0000 7fa5a67fc700 1 -- 172.21.15.23:0/2700082756 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0003b90 con 0x7fa5b0152ba0 2023-06-22T08:00:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.344+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2700082756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b00c6590 con 0x7fa5b0152ba0 2023-06-22T08:00:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.344+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2700082756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b01ee5e0 con 0x7fa5b0152ba0 2023-06-22T08:00:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.345+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2700082756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b00cbed0 con 0x7fa5b0152ba0 2023-06-22T08:00:12.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.346+0000 7fa5a67fc700 1 -- 172.21.15.23:0/2700082756 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa5a0003cf0 con 0x7fa5b0152ba0 2023-06-22T08:00:12.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.347+0000 7fa5a67fc700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa59406ace0 0x7fa59406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.347+0000 7fa5a67fc700 1 -- 172.21.15.23:0/2700082756 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa5a008abf0 con 0x7fa5b0152ba0 2023-06-22T08:00:12.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.347+0000 7fa5b599b700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa59406ace0 0x7fa59406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.348+0000 7fa5b599b700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa59406ace0 0x7fa59406d190 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac007fd0 tx=0x7fa5ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:12.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.351+0000 7fa5a67fc700 1 -- 172.21.15.23:0/2700082756 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa5a00559a0 con 0x7fa5b0152ba0 2023-06-22T08:00:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.558+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2700082756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa5b004ea20 con 0x7fa5b0152ba0 2023-06-22T08:00:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.558+0000 7fa5a67fc700 1 -- 172.21.15.23:0/2700082756 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa5a00150a0 con 0x7fa5b0152ba0 2023-06-22T08:00:12.563 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2023-06-22T08:00:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa59406ace0 msgr2=0x7fa59406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa59406ace0 0x7fa59406d190 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac007fd0 tx=0x7fa5ac009040 comp rx=0 tx=0).stop 2023-06-22T08:00:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2700082756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b0152ba0 msgr2=0x7fa5b01e4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b0152ba0 0x7fa5b01e4e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009b90 tx=0x7fa5a0004750 comp rx=0 tx=0).stop 2023-06-22T08:00:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2700082756 shutdown_connections 2023-06-22T08:00:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa5b01593f0 0x7fa5b00b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa59406ace0 0x7fa59406d190 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5b00c4650 0x7fa5b01f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 --2- 172.21.15.23:0/2700082756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b0152ba0 0x7fa5b01e4e30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2700082756 >> 172.21.15.23:0/2700082756 conn(0x7fa5b0065b90 msgr2=0x7fa5b00cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:12.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2700082756 shutdown_connections 2023-06-22T08:00:12.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.561+0000 7fa5b7bff700 1 -- 172.21.15.23:0/2700082756 wait complete. 2023-06-22T08:00:12.584 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-22T08:00:12.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.784+0000 7fe298bff700 1 Processor -- start 2023-06-22T08:00:12.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.784+0000 7fe298bff700 1 -- start start 2023-06-22T08:00:12.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.785+0000 7fe298bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe294152ba0 0x7fe2941593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.785+0000 7fe298bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe2940c9ee0 0x7fe2940ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.785+0000 7fe298bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe294159930 con 0x7fe2940c9260 2023-06-22T08:00:12.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.785+0000 7fe298bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2940c9aa0 con 0x7fe2940c9ee0 2023-06-22T08:00:12.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.785+0000 7fe298bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2940ca880 con 0x7fe294152ba0 2023-06-22T08:00:12.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.785+0000 7fe291d9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe2940c9260 0x7fe2940c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:49224/0 (socket says 172.21.15.23:49224) 2023-06-22T08:00:12.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.785+0000 7fe291d9b700 1 -- 172.21.15.23:0/59687281 learned_addr learned my addr 172.21.15.23:0/59687281 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:12.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.785+0000 7fe292d9d700 1 --2- 172.21.15.23:0/59687281 >> v2:172.21.15.67:3300/0 conn(0x7fe2940c9ee0 0x7fe2940ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.786+0000 7fe29259c700 1 --2- 172.21.15.23:0/59687281 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe294152ba0 0x7fe2941593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.786+0000 7fe29159a700 1 -- 172.21.15.23:0/59687281 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417278605 0 0) 0x7fe288001110 con 0x7fe2940c9260 2023-06-22T08:00:12.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.786+0000 7fe29159a700 1 -- 172.21.15.23:0/59687281 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe27c0035b0 con 0x7fe2940c9260 2023-06-22T08:00:12.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.786+0000 7fe292d9d700 1 -- 172.21.15.23:0/59687281 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe294152ba0 msgr2=0x7fe2941593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:12.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.786+0000 7fe292d9d700 1 --2- 172.21.15.23:0/59687281 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe294152ba0 0x7fe2941593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.786+0000 7fe292d9d700 1 -- 172.21.15.23:0/59687281 >> v1:172.21.15.23:6789/0 conn(0x7fe2940c9260 legacy=0x7fe2940c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.786+0000 7fe292d9d700 1 -- 172.21.15.23:0/59687281 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29406aed0 con 0x7fe2940c9ee0 2023-06-22T08:00:12.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.786+0000 7fe29259c700 1 --2- 172.21.15.23:0/59687281 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe294152ba0 0x7fe2941593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:12.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.786+0000 7fe292d9d700 1 --2- 172.21.15.23:0/59687281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2940c9ee0 0x7fe2940ca340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009d90 tx=0x7fe28c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25643c1599b9085e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:12.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe29159a700 1 -- 172.21.15.23:0/59687281 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c00e070 con 0x7fe2940c9ee0 2023-06-22T08:00:12.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe29159a700 1 -- 172.21.15.23:0/59687281 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28c00bb20 con 0x7fe2940c9ee0 2023-06-22T08:00:12.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe29159a700 1 -- 172.21.15.23:0/59687281 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c012430 con 0x7fe2940c9ee0 2023-06-22T08:00:12.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe298bff700 1 -- 172.21.15.23:0/59687281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2940c9ee0 msgr2=0x7fe2940ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe298bff700 1 --2- 172.21.15.23:0/59687281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2940c9ee0 0x7fe2940ca340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009d90 tx=0x7fe28c00b330 comp rx=0 tx=0).stop 2023-06-22T08:00:12.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe298bff700 1 -- 172.21.15.23:0/59687281 shutdown_connections 2023-06-22T08:00:12.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe298bff700 1 --2- 172.21.15.23:0/59687281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2940c9ee0 0x7fe2940ca340 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe298bff700 1 --2- 172.21.15.23:0/59687281 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe294152ba0 0x7fe2941593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe298bff700 1 -- 172.21.15.23:0/59687281 >> 172.21.15.23:0/59687281 conn(0x7fe294065b90 msgr2=0x7fe294065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:12.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe298bff700 1 -- 172.21.15.23:0/59687281 shutdown_connections 2023-06-22T08:00:12.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.787+0000 7fe298bff700 1 -- 172.21.15.23:0/59687281 wait complete. 2023-06-22T08:00:12.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.790+0000 7fe298bff700 1 Processor -- start 2023-06-22T08:00:12.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.790+0000 7fe298bff700 1 -- start start 2023-06-22T08:00:12.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe298bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe2940c9260 0x7fe2941eef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe298bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2940c9ee0 0x7fe2940b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe298bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe294152ba0 0x7fe29414e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe298bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2941ef450 con 0x7fe2940c9ee0 2023-06-22T08:00:12.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe298bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2940cf070 con 0x7fe294152ba0 2023-06-22T08:00:12.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe298bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2940c88f0 con 0x7fe2940c9260 2023-06-22T08:00:12.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe291d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2940c9ee0 0x7fe2940b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe291d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2940c9ee0 0x7fe2940b28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44056/0 (socket says 172.21.15.23:44056) 2023-06-22T08:00:12.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe291d9b700 1 -- 172.21.15.23:0/3082380272 learned_addr learned my addr 172.21.15.23:0/3082380272 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:12.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe29259c700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe2940c9260 0x7fe2941eef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.791+0000 7fe292d9d700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe294152ba0 0x7fe29414e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.792+0000 7fe291d9b700 1 -- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe2940c9260 msgr2=0x7fe2941eef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.792+0000 7fe291d9b700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe2940c9260 0x7fe2941eef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.792+0000 7fe291d9b700 1 -- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe294152ba0 msgr2=0x7fe29414e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:12.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.792+0000 7fe291d9b700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe294152ba0 0x7fe29414e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:12.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.792+0000 7fe291d9b700 1 -- 172.21.15.23:0/3082380272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2840094a0 con 0x7fe2940c9ee0 2023-06-22T08:00:12.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.792+0000 7fe291d9b700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2940c9ee0 0x7fe2940b28d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe288007fd0 tx=0x7fe28800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:12.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.792+0000 7fe282ffd700 1 -- 172.21.15.23:0/3082380272 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe288005d70 con 0x7fe2940c9ee0 2023-06-22T08:00:12.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.792+0000 7fe282ffd700 1 -- 172.21.15.23:0/3082380272 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2880070d0 con 0x7fe2940c9ee0 2023-06-22T08:00:12.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.792+0000 7fe298bff700 1 -- 172.21.15.23:0/3082380272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c009a70 con 0x7fe2940c9ee0 2023-06-22T08:00:12.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.792+0000 7fe282ffd700 1 -- 172.21.15.23:0/3082380272 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe288007460 con 0x7fe2940c9ee0 2023-06-22T08:00:12.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.793+0000 7fe298bff700 1 -- 172.21.15.23:0/3082380272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2940c6060 con 0x7fe2940c9ee0 2023-06-22T08:00:12.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.794+0000 7fe298bff700 1 -- 172.21.15.23:0/3082380272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2940cbed0 con 0x7fe2940c9ee0 2023-06-22T08:00:12.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.794+0000 7fe282ffd700 1 -- 172.21.15.23:0/3082380272 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe2880075c0 con 0x7fe2940c9ee0 2023-06-22T08:00:12.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.795+0000 7fe282ffd700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe27406ace0 0x7fe27406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:12.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.795+0000 7fe282ffd700 1 -- 172.21.15.23:0/3082380272 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe288088aa0 con 0x7fe2940c9ee0 2023-06-22T08:00:12.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.796+0000 7fe29259c700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe27406ace0 0x7fe27406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:12.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.796+0000 7fe29259c700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe27406ace0 0x7fe27406d190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe284009cd0 tx=0x7fe284009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:12.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:12.800+0000 7fe282ffd700 1 -- 172.21.15.23:0/3082380272 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe288053860 con 0x7fe2940c9ee0 2023-06-22T08:00:13.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.005+0000 7fe298bff700 1 -- 172.21.15.23:0/3082380272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe294159930 con 0x7fe2940c9ee0 2023-06-22T08:00:13.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.005+0000 7fe282ffd700 1 -- 172.21.15.23:0/3082380272 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe294159930 con 0x7fe2940c9ee0 2023-06-22T08:00:13.013 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:13.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 -- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe27406ace0 msgr2=0x7fe27406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe27406ace0 0x7fe27406d190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe284009cd0 tx=0x7fe284009210 comp rx=0 tx=0).stop 2023-06-22T08:00:13.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 -- 172.21.15.23:0/3082380272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2940c9ee0 msgr2=0x7fe2940b28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2940c9ee0 0x7fe2940b28d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe288007fd0 tx=0x7fe28800c9e0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 -- 172.21.15.23:0/3082380272 shutdown_connections 2023-06-22T08:00:13.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe27406ace0 0x7fe27406d190 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe294152ba0 0x7fe29414e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe2940c9260 0x7fe2941eef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 --2- 172.21.15.23:0/3082380272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2940c9ee0 0x7fe2940b28d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 -- 172.21.15.23:0/3082380272 >> 172.21.15.23:0/3082380272 conn(0x7fe294065b90 msgr2=0x7fe2941552f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 -- 172.21.15.23:0/3082380272 shutdown_connections 2023-06-22T08:00:13.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.008+0000 7fe298bff700 1 -- 172.21.15.23:0/3082380272 wait complete. 2023-06-22T08:00:13.029 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-22T08:00:13.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.227+0000 7f0c4a7ff700 1 Processor -- start 2023-06-22T08:00:13.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.227+0000 7f0c4a7ff700 1 -- start start 2023-06-22T08:00:13.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.228+0000 7f0c4a7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c44152ba0 0x7f0c441593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.228+0000 7f0c4a7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c440c9ee0 0x7f0c440ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.228+0000 7f0c4a7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c44159930 con 0x7f0c440c9260 2023-06-22T08:00:13.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.228+0000 7f0c4a7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c440c9aa0 con 0x7f0c440c9ee0 2023-06-22T08:00:13.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.228+0000 7f0c4a7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c440ca880 con 0x7f0c44152ba0 2023-06-22T08:00:13.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.228+0000 7f0c43fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c44152ba0 0x7f0c441593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.228+0000 7f0c437fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f0c440c9260 0x7f0c440c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:49240/0 (socket says 172.21.15.23:49240) 2023-06-22T08:00:13.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.228+0000 7f0c437fe700 1 -- 172.21.15.23:0/369164118 learned_addr learned my addr 172.21.15.23:0/369164118 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:13.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.228+0000 7f0c48d9c700 1 --2- 172.21.15.23:0/369164118 >> v2:172.21.15.67:3300/0 conn(0x7f0c440c9ee0 0x7f0c440ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.229+0000 7f0c43fff700 1 -- 172.21.15.23:0/369164118 >> v2:172.21.15.67:3300/0 conn(0x7f0c440c9ee0 msgr2=0x7f0c440ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:13.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.229+0000 7f0c43fff700 1 --2- 172.21.15.23:0/369164118 >> v2:172.21.15.67:3300/0 conn(0x7f0c440c9ee0 0x7f0c440ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.229+0000 7f0c43fff700 1 -- 172.21.15.23:0/369164118 >> v1:172.21.15.23:6789/0 conn(0x7f0c440c9260 legacy=0x7f0c440c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.229+0000 7f0c43fff700 1 -- 172.21.15.23:0/369164118 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4406aed0 con 0x7f0c44152ba0 2023-06-22T08:00:13.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.229+0000 7f0c42ffd700 1 -- 172.21.15.23:0/369164118 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008401016 0 0) 0x7f0c38001110 con 0x7f0c440c9260 2023-06-22T08:00:13.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.229+0000 7f0c43fff700 1 --2- 172.21.15.23:0/369164118 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c44152ba0 0x7f0c441593f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009840 tx=0x7f0c34009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71110027e2003fc9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:13.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.229+0000 7f0c42ffd700 1 -- 172.21.15.23:0/369164118 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34011830 con 0x7f0c44152ba0 2023-06-22T08:00:13.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.229+0000 7f0c42ffd700 1 -- 172.21.15.23:0/369164118 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c34011990 con 0x7f0c44152ba0 2023-06-22T08:00:13.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.229+0000 7f0c42ffd700 1 -- 172.21.15.23:0/369164118 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34011d20 con 0x7f0c44152ba0 2023-06-22T08:00:13.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.230+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/369164118 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c44152ba0 msgr2=0x7f0c441593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.230+0000 7f0c4a7ff700 1 --2- 172.21.15.23:0/369164118 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c44152ba0 0x7f0c441593f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009840 tx=0x7f0c34009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:13.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.230+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/369164118 shutdown_connections 2023-06-22T08:00:13.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.230+0000 7f0c4a7ff700 1 --2- 172.21.15.23:0/369164118 >> v2:172.21.15.67:3300/0 conn(0x7f0c440c9ee0 0x7f0c440ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.230+0000 7f0c4a7ff700 1 --2- 172.21.15.23:0/369164118 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c44152ba0 0x7f0c441593f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.230+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/369164118 >> 172.21.15.23:0/369164118 conn(0x7f0c44065b90 msgr2=0x7f0c44065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:13.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.230+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/369164118 shutdown_connections 2023-06-22T08:00:13.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.230+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/369164118 wait complete. 2023-06-22T08:00:13.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.233+0000 7f0c4a7ff700 1 Processor -- start 2023-06-22T08:00:13.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.233+0000 7f0c4a7ff700 1 -- start start 2023-06-22T08:00:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c4a7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c440c9260 0x7f0c4414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c4a7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c440c9ee0 0x7f0c441ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c4a7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c44152ba0 0x7f0c441ee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c4a7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441eee20 con 0x7f0c440c9260 2023-06-22T08:00:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c4a7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441f2520 con 0x7f0c44152ba0 2023-06-22T08:00:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c4a7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c440cd470 con 0x7f0c440c9ee0 2023-06-22T08:00:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c440c9260 0x7f0c4414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c437fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c440c9ee0 0x7f0c441ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c48d9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c44152ba0 0x7f0c441ee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c437fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c440c9ee0 0x7f0c441ee3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49096/0 (socket says 172.21.15.23:49096) 2023-06-22T08:00:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c440c9260 0x7f0c4414d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44058/0 (socket says 172.21.15.23:44058) 2023-06-22T08:00:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.234+0000 7f0c437fe700 1 -- 172.21.15.23:0/1745957088 learned_addr learned my addr 172.21.15.23:0/1745957088 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:13.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.235+0000 7f0c43fff700 1 -- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c440c9ee0 msgr2=0x7f0c441ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.235+0000 7f0c43fff700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c440c9ee0 0x7f0c441ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.235+0000 7f0c43fff700 1 -- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c44152ba0 msgr2=0x7f0c441ee8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.235+0000 7f0c43fff700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c44152ba0 0x7f0c441ee8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.235+0000 7f0c43fff700 1 -- 172.21.15.23:0/1745957088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c340094a0 con 0x7f0c440c9260 2023-06-22T08:00:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.235+0000 7f0c43fff700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c440c9260 0x7f0c4414d5a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0c34016000 tx=0x7f0c34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:13.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.235+0000 7f0c40ff9700 1 -- 172.21.15.23:0/1745957088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c340036a0 con 0x7f0c440c9260 2023-06-22T08:00:13.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.235+0000 7f0c40ff9700 1 -- 172.21.15.23:0/1745957088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3401f070 con 0x7f0c440c9260 2023-06-22T08:00:13.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.235+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/1745957088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c441f1bf0 con 0x7f0c440c9260 2023-06-22T08:00:13.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.235+0000 7f0c40ff9700 1 -- 172.21.15.23:0/1745957088 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34003a80 con 0x7f0c440c9260 2023-06-22T08:00:13.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.236+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/1745957088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c441f21b0 con 0x7f0c440c9260 2023-06-22T08:00:13.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.236+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/1745957088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c440cbed0 con 0x7f0c440c9260 2023-06-22T08:00:13.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.238+0000 7f0c40ff9700 1 -- 172.21.15.23:0/1745957088 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0c34003be0 con 0x7f0c440c9260 2023-06-22T08:00:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.238+0000 7f0c40ff9700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c3006ace0 0x7f0c3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.238+0000 7f0c40ff9700 1 -- 172.21.15.23:0/1745957088 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0c34021070 con 0x7f0c440c9260 2023-06-22T08:00:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.239+0000 7f0c437fe700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c3006ace0 0x7f0c3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.239+0000 7f0c437fe700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c3006ace0 0x7f0c3006d190 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0c38007fd0 tx=0x7f0c38007ce0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:13.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.243+0000 7f0c40ff9700 1 -- 172.21.15.23:0/1745957088 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0c34055c20 con 0x7f0c440c9260 2023-06-22T08:00:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.449+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/1745957088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0c4404ea20 con 0x7f0c440c9260 2023-06-22T08:00:13.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.450+0000 7f0c40ff9700 1 -- 172.21.15.23:0/1745957088 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0c340150a0 con 0x7f0c440c9260 2023-06-22T08:00:13.454 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-06-22T08:00:13.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c3006ace0 msgr2=0x7f0c3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c3006ace0 0x7f0c3006d190 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0c38007fd0 tx=0x7f0c38007ce0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/1745957088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c440c9260 msgr2=0x7f0c4414d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c440c9260 0x7f0c4414d5a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0c34016000 tx=0x7f0c34004750 comp rx=0 tx=0).stop 2023-06-22T08:00:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/1745957088 shutdown_connections 2023-06-22T08:00:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c3006ace0 0x7f0c3006d190 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c44152ba0 0x7f0c441ee8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c440c9260 0x7f0c4414d5a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 --2- 172.21.15.23:0/1745957088 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c440c9ee0 0x7f0c441ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/1745957088 >> 172.21.15.23:0/1745957088 conn(0x7f0c44065b90 msgr2=0x7f0c44151b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:13.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/1745957088 shutdown_connections 2023-06-22T08:00:13.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.452+0000 7f0c4a7ff700 1 -- 172.21.15.23:0/1745957088 wait complete. 2023-06-22T08:00:13.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-22T08:00:13.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.674+0000 7fa06f7ff700 1 Processor -- start 2023-06-22T08:00:13.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.674+0000 7fa06f7ff700 1 -- start start 2023-06-22T08:00:13.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.675+0000 7fa06f7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa0681593f0 0x7fa0680c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.675+0000 7fa06f7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0680c4650 0x7fa0680c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06f7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0680c9aa0 con 0x7fa068152ba0 2023-06-22T08:00:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06f7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0680c4a60 con 0x7fa0681593f0 2023-06-22T08:00:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06f7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06806aed0 con 0x7fa0680c4650 2023-06-22T08:00:13.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06cd9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa0681593f0 0x7fa0680c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06cd9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa0681593f0 0x7fa0680c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44074/0 (socket says 172.21.15.23:44074) 2023-06-22T08:00:13.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06cd9a700 1 -- 172.21.15.23:0/1930060560 learned_addr learned my addr 172.21.15.23:0/1930060560 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:13.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06dd9c700 1 --2- 172.21.15.23:0/1930060560 >> v2:172.21.15.67:3300/0 conn(0x7fa0680c4650 0x7fa0680c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06cd9a700 1 -- 172.21.15.23:0/1930060560 >> v2:172.21.15.67:3300/0 conn(0x7fa0680c4650 msgr2=0x7fa0680c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:13.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06cd9a700 1 --2- 172.21.15.23:0/1930060560 >> v2:172.21.15.67:3300/0 conn(0x7fa0680c4650 0x7fa0680c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06cd9a700 1 -- 172.21.15.23:0/1930060560 >> v1:172.21.15.67:6789/0 conn(0x7fa068152ba0 legacy=0x7fa0680c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06cd9a700 1 -- 172.21.15.23:0/1930060560 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06806b1b0 con 0x7fa0681593f0 2023-06-22T08:00:13.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06dd9c700 1 --2- 172.21.15.23:0/1930060560 >> v2:172.21.15.67:3300/0 conn(0x7fa0680c4650 0x7fa0680c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:13.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.676+0000 7fa06cd9a700 1 --2- 172.21.15.23:0/1930060560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0681593f0 0x7fa0680c4110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa06808b390 tx=0x7fa058009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ef149f6eab55d2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:13.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa05ffff700 1 -- 172.21.15.23:0/1930060560 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05800b870 con 0x7fa0681593f0 2023-06-22T08:00:13.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa05ffff700 1 -- 172.21.15.23:0/1930060560 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05800b9d0 con 0x7fa0681593f0 2023-06-22T08:00:13.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa05ffff700 1 -- 172.21.15.23:0/1930060560 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05800bd60 con 0x7fa0681593f0 2023-06-22T08:00:13.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa06f7ff700 1 -- 172.21.15.23:0/1930060560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0681593f0 msgr2=0x7fa0680c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa06f7ff700 1 --2- 172.21.15.23:0/1930060560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0681593f0 0x7fa0680c4110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa06808b390 tx=0x7fa058009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa06f7ff700 1 -- 172.21.15.23:0/1930060560 shutdown_connections 2023-06-22T08:00:13.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa06f7ff700 1 --2- 172.21.15.23:0/1930060560 >> v2:172.21.15.67:3300/0 conn(0x7fa0680c4650 0x7fa0680c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa06f7ff700 1 --2- 172.21.15.23:0/1930060560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0681593f0 0x7fa0680c4110 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa06f7ff700 1 -- 172.21.15.23:0/1930060560 >> 172.21.15.23:0/1930060560 conn(0x7fa068065b90 msgr2=0x7fa068065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:13.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa06f7ff700 1 -- 172.21.15.23:0/1930060560 shutdown_connections 2023-06-22T08:00:13.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.677+0000 7fa06f7ff700 1 -- 172.21.15.23:0/1930060560 wait complete. 2023-06-22T08:00:13.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.679+0000 7fa06f7ff700 1 Processor -- start 2023-06-22T08:00:13.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.679+0000 7fa06f7ff700 1 -- start start 2023-06-22T08:00:13.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.680+0000 7fa06f7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0680c4650 0x7fa06814df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.680+0000 7fa06f7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa068152ba0 0x7fa0680cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.680+0000 7fa06f7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0681593f0 0x7fa0680caec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.680+0000 7fa06f7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0681ef970 con 0x7fa068152ba0 2023-06-22T08:00:13.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.680+0000 7fa06f7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0680c8cf0 con 0x7fa0681593f0 2023-06-22T08:00:13.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.680+0000 7fa06f7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa068153250 con 0x7fa0680c4650 2023-06-22T08:00:13.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.680+0000 7fa06cd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa068152ba0 0x7fa0680cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.680+0000 7fa06cd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa068152ba0 0x7fa0680cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44082/0 (socket says 172.21.15.23:44082) 2023-06-22T08:00:13.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.680+0000 7fa06cd9a700 1 -- 172.21.15.23:0/2159271197 learned_addr learned my addr 172.21.15.23:0/2159271197 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:13.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa06dd9c700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0681593f0 0x7fa0680caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa06d59b700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0680c4650 0x7fa06814df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa06cd9a700 1 -- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0680c4650 msgr2=0x7fa06814df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa06cd9a700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0680c4650 0x7fa06814df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa06cd9a700 1 -- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0681593f0 msgr2=0x7fa0680caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa06cd9a700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0681593f0 0x7fa0680caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa06cd9a700 1 -- 172.21.15.23:0/2159271197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa058009550 con 0x7fa068152ba0 2023-06-22T08:00:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa06cd9a700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa068152ba0 0x7fa0680cd1f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa058009fd0 tx=0x7fa058004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa05dffb700 1 -- 172.21.15.23:0/2159271197 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa058004480 con 0x7fa068152ba0 2023-06-22T08:00:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa05dffb700 1 -- 172.21.15.23:0/2159271197 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0580045e0 con 0x7fa068152ba0 2023-06-22T08:00:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa05dffb700 1 -- 172.21.15.23:0/2159271197 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0580038c0 con 0x7fa068152ba0 2023-06-22T08:00:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.681+0000 7fa06f7ff700 1 -- 172.21.15.23:0/2159271197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0680ce420 con 0x7fa068152ba0 2023-06-22T08:00:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.682+0000 7fa06f7ff700 1 -- 172.21.15.23:0/2159271197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0680ce670 con 0x7fa068152ba0 2023-06-22T08:00:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.683+0000 7fa06f7ff700 1 -- 172.21.15.23:0/2159271197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0680cbed0 con 0x7fa068152ba0 2023-06-22T08:00:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.683+0000 7fa05dffb700 1 -- 172.21.15.23:0/2159271197 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa058021070 con 0x7fa068152ba0 2023-06-22T08:00:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.684+0000 7fa05dffb700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa04c06ace0 0x7fa04c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.684+0000 7fa05dffb700 1 -- 172.21.15.23:0/2159271197 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa05808ad70 con 0x7fa068152ba0 2023-06-22T08:00:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.685+0000 7fa06d59b700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa04c06ace0 0x7fa04c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.685+0000 7fa06d59b700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa04c06ace0 0x7fa04c06d190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa064007fd0 tx=0x7fa064009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.689+0000 7fa05dffb700 1 -- 172.21.15.23:0/2159271197 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa058055b20 con 0x7fa068152ba0 2023-06-22T08:00:13.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.896+0000 7fa06f7ff700 1 -- 172.21.15.23:0/2159271197 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa0681efab0 con 0x7fa068152ba0 2023-06-22T08:00:13.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.896+0000 7fa05dffb700 1 -- 172.21.15.23:0/2159271197 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa0681efab0 con 0x7fa068152ba0 2023-06-22T08:00:13.901 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:13.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 -- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa04c06ace0 msgr2=0x7fa04c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa04c06ace0 0x7fa04c06d190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa064007fd0 tx=0x7fa064009040 comp rx=0 tx=0).stop 2023-06-22T08:00:13.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 -- 172.21.15.23:0/2159271197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa068152ba0 msgr2=0x7fa0680cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:13.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa068152ba0 0x7fa0680cd1f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa058009fd0 tx=0x7fa058004750 comp rx=0 tx=0).stop 2023-06-22T08:00:13.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 -- 172.21.15.23:0/2159271197 shutdown_connections 2023-06-22T08:00:13.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa04c06ace0 0x7fa04c06d190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0681593f0 0x7fa0680caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0680c4650 0x7fa06814df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 --2- 172.21.15.23:0/2159271197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa068152ba0 0x7fa0680cd1f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:13.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 -- 172.21.15.23:0/2159271197 >> 172.21.15.23:0/2159271197 conn(0x7fa068065b90 msgr2=0x7fa0681517d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:13.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 -- 172.21.15.23:0/2159271197 shutdown_connections 2023-06-22T08:00:13.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:13.899+0000 7fa06f7ff700 1 -- 172.21.15.23:0/2159271197 wait complete. 2023-06-22T08:00:13.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-22T08:00:14.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.116+0000 7fc87c3ff700 1 Processor -- start 2023-06-22T08:00:14.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.116+0000 7fc87c3ff700 1 -- start start 2023-06-22T08:00:14.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87c3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc874152ce0 0x7fc8740cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87c3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc8740c40b0 0x7fc8740c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87c3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87415c370 con 0x7fc8740c9fd0 2023-06-22T08:00:14.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87c3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8741530b0 con 0x7fc8740c40b0 2023-06-22T08:00:14.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87c3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740cf6f0 con 0x7fc874152ce0 2023-06-22T08:00:14.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87a99c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc8740c40b0 0x7fc8740c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87a19b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc874152ce0 0x7fc8740cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87a99c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc8740c40b0 0x7fc8740c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44094/0 (socket says 172.21.15.23:44094) 2023-06-22T08:00:14.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87a99c700 1 -- 172.21.15.23:0/3162978382 learned_addr learned my addr 172.21.15.23:0/3162978382 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:14.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87a99c700 1 -- 172.21.15.23:0/3162978382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc874152ce0 msgr2=0x7fc8740cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:14.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87a99c700 1 --2- 172.21.15.23:0/3162978382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc874152ce0 0x7fc8740cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87a99c700 1 -- 172.21.15.23:0/3162978382 >> v1:172.21.15.67:6789/0 conn(0x7fc8740c9fd0 legacy=0x7fc87415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.118+0000 7fc87a99c700 1 -- 172.21.15.23:0/3162978382 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87415bda0 con 0x7fc8740c40b0 2023-06-22T08:00:14.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.119+0000 7fc87a99c700 1 --2- 172.21.15.23:0/3162978382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8740c40b0 0x7fc8740c4530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009e50 tx=0x7fc86c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a28ea907fde28f1 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:14.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.119+0000 7fc879199700 1 -- 172.21.15.23:0/3162978382 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c00e070 con 0x7fc8740c40b0 2023-06-22T08:00:14.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.119+0000 7fc879199700 1 -- 172.21.15.23:0/3162978382 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86c00b910 con 0x7fc8740c40b0 2023-06-22T08:00:14.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.119+0000 7fc879199700 1 -- 172.21.15.23:0/3162978382 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c00bcc0 con 0x7fc8740c40b0 2023-06-22T08:00:14.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.119+0000 7fc87c3ff700 1 -- 172.21.15.23:0/3162978382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8740c40b0 msgr2=0x7fc8740c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.119+0000 7fc87c3ff700 1 --2- 172.21.15.23:0/3162978382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8740c40b0 0x7fc8740c4530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009e50 tx=0x7fc86c00b330 comp rx=0 tx=0).stop 2023-06-22T08:00:14.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.119+0000 7fc87c3ff700 1 -- 172.21.15.23:0/3162978382 shutdown_connections 2023-06-22T08:00:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.119+0000 7fc87c3ff700 1 --2- 172.21.15.23:0/3162978382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8740c40b0 0x7fc8740c4530 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.119+0000 7fc87c3ff700 1 --2- 172.21.15.23:0/3162978382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc874152ce0 0x7fc8740cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.119+0000 7fc87c3ff700 1 -- 172.21.15.23:0/3162978382 >> 172.21.15.23:0/3162978382 conn(0x7fc87406ae30 msgr2=0x7fc87406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.120+0000 7fc87c3ff700 1 -- 172.21.15.23:0/3162978382 shutdown_connections 2023-06-22T08:00:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.120+0000 7fc87c3ff700 1 -- 172.21.15.23:0/3162978382 wait complete. 2023-06-22T08:00:14.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.121+0000 7fc87c3ff700 1 Processor -- start 2023-06-22T08:00:14.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.121+0000 7fc87c3ff700 1 -- start start 2023-06-22T08:00:14.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.122+0000 7fc87c3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8740c40b0 0x7fc87414d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.122+0000 7fc87c3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8740c9fd0 0x7fc8741ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.122+0000 7fc87c3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc874152ce0 0x7fc8741eea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.122+0000 7fc87c3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8741f14a0 con 0x7fc8740c9fd0 2023-06-22T08:00:14.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.122+0000 7fc87c3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8741f1740 con 0x7fc874152ce0 2023-06-22T08:00:14.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.122+0000 7fc87c3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8741f1a00 con 0x7fc8740c40b0 2023-06-22T08:00:14.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.122+0000 7fc87a19b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8740c40b0 0x7fc87414d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.122+0000 7fc87999a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8740c9fd0 0x7fc8741ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.122+0000 7fc87a19b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8740c40b0 0x7fc87414d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49134/0 (socket says 172.21.15.23:49134) 2023-06-22T08:00:14.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc87a19b700 1 -- 172.21.15.23:0/2814094728 learned_addr learned my addr 172.21.15.23:0/2814094728 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:14.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc87a99c700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc874152ce0 0x7fc8741eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc87a19b700 1 -- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc874152ce0 msgr2=0x7fc8741eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc87a19b700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc874152ce0 0x7fc8741eea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc87a19b700 1 -- 172.21.15.23:0/2814094728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8740c9fd0 msgr2=0x7fc8741ee4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc87a19b700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8740c9fd0 0x7fc8741ee4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc87a19b700 1 -- 172.21.15.23:0/2814094728 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c009b30 con 0x7fc8740c40b0 2023-06-22T08:00:14.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc87a19b700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8740c40b0 0x7fc87414d9b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc87000e9b0 tx=0x7fc87000ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:14.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc86affd700 1 -- 172.21.15.23:0/2814094728 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870009710 con 0x7fc8740c40b0 2023-06-22T08:00:14.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc86affd700 1 -- 172.21.15.23:0/2814094728 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87000ee50 con 0x7fc8740c40b0 2023-06-22T08:00:14.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc87c3ff700 1 -- 172.21.15.23:0/2814094728 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8741f1ce0 con 0x7fc8740c40b0 2023-06-22T08:00:14.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.123+0000 7fc86affd700 1 -- 172.21.15.23:0/2814094728 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87000ca50 con 0x7fc8740c40b0 2023-06-22T08:00:14.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.124+0000 7fc87c3ff700 1 -- 172.21.15.23:0/2814094728 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8741f22a0 con 0x7fc8740c40b0 2023-06-22T08:00:14.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.124+0000 7fc87c3ff700 1 -- 172.21.15.23:0/2814094728 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87404ea20 con 0x7fc8740c40b0 2023-06-22T08:00:14.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.126+0000 7fc86affd700 1 -- 172.21.15.23:0/2814094728 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc87000cbb0 con 0x7fc8740c40b0 2023-06-22T08:00:14.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.127+0000 7fc86affd700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc85806ad10 0x7fc85806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.127+0000 7fc86affd700 1 -- 172.21.15.23:0/2814094728 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc870089970 con 0x7fc8740c40b0 2023-06-22T08:00:14.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.127+0000 7fc87999a700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc85806ad10 0x7fc85806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.128+0000 7fc87999a700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc85806ad10 0x7fc85806d1c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc8741ebb60 tx=0x7fc864009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:14.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.131+0000 7fc86affd700 1 -- 172.21.15.23:0/2814094728 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc870058110 con 0x7fc8740c40b0 2023-06-22T08:00:14.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.355+0000 7fc87c3ff700 1 -- 172.21.15.23:0/2814094728 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc8740c9db0 con 0x7fc8740c40b0 2023-06-22T08:00:14.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.356+0000 7fc86affd700 1 -- 172.21.15.23:0/2814094728 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc870021070 con 0x7fc8740c40b0 2023-06-22T08:00:14.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 -- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc85806ad10 msgr2=0x7fc85806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc85806ad10 0x7fc85806d1c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc8741ebb60 tx=0x7fc864009040 comp rx=0 tx=0).stop 2023-06-22T08:00:14.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 -- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8740c40b0 msgr2=0x7fc87414d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8740c40b0 0x7fc87414d9b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc87000e9b0 tx=0x7fc87000ecc0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 -- 172.21.15.23:0/2814094728 shutdown_connections 2023-06-22T08:00:14.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc85806ad10 0x7fc85806d1c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc874152ce0 0x7fc8741eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8740c40b0 0x7fc87414d9b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 --2- 172.21.15.23:0/2814094728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8740c9fd0 0x7fc8741ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 -- 172.21.15.23:0/2814094728 >> 172.21.15.23:0/2814094728 conn(0x7fc87406ae30 msgr2=0x7fc874152010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:14.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 -- 172.21.15.23:0/2814094728 shutdown_connections 2023-06-22T08:00:14.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.358+0000 7fc87c3ff700 1 -- 172.21.15.23:0/2814094728 wait complete. 2023-06-22T08:00:14.364 INFO:teuthology.orchestra.run.smithi023.stderr:listed 0 entries 2023-06-22T08:00:14.380 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T08:00:14.381 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-22T08:00:14.423 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:14.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.659+0000 7fe7305ff700 1 Processor -- start 2023-06-22T08:00:14.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.659+0000 7fe7305ff700 1 -- start start 2023-06-22T08:00:14.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.661+0000 7fe7305ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe72815c550 0x7fe7280c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.661+0000 7fe7305ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7280cbd60 con 0x7fe7280cb260 2023-06-22T08:00:14.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.661+0000 7fe7305ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe728159550 con 0x7fe7280c9200 2023-06-22T08:00:14.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.661+0000 7fe7305ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe728158980 con 0x7fe72815c550 2023-06-22T08:00:14.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72e39b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe7280cb260 0x7fe72815be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:49256/0 (socket says 172.21.15.23:49256) 2023-06-22T08:00:14.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72e39b700 1 -- 172.21.15.23:0/885663607 learned_addr learned my addr 172.21.15.23:0/885663607 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:14.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72eb9c700 1 --2- 172.21.15.23:0/885663607 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe72815c550 0x7fe7280c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72d399700 1 -- 172.21.15.23:0/885663607 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450200217 0 0) 0x7fe728159550 con 0x7fe7280c9200 2023-06-22T08:00:14.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72d399700 1 -- 172.21.15.23:0/885663607 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7140035b0 con 0x7fe7280c9200 2023-06-22T08:00:14.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72d399700 1 -- 172.21.15.23:0/885663607 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337028744 0 0) 0x7fe7280cbd60 con 0x7fe7280cb260 2023-06-22T08:00:14.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72d399700 1 -- 172.21.15.23:0/885663607 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe728159550 con 0x7fe7280cb260 2023-06-22T08:00:14.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72eb9c700 1 -- 172.21.15.23:0/885663607 >> v1:172.21.15.67:6789/0 conn(0x7fe7280c9200 legacy=0x7fe7281611c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72eb9c700 1 -- 172.21.15.23:0/885663607 >> v1:172.21.15.23:6789/0 conn(0x7fe7280cb260 legacy=0x7fe72815be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72eb9c700 1 -- 172.21.15.23:0/885663607 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe728158c20 con 0x7fe72815c550 2023-06-22T08:00:14.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.662+0000 7fe72eb9c700 1 --2- 172.21.15.23:0/885663607 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe72815c550 0x7fe7280c4a50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe720009dc0 tx=0x7fe72000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9c7535393459292 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:14.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.663+0000 7fe72d399700 1 -- 172.21.15.23:0/885663607 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72000e070 con 0x7fe72815c550 2023-06-22T08:00:14.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.663+0000 7fe72d399700 1 -- 172.21.15.23:0/885663607 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72000bb20 con 0x7fe72815c550 2023-06-22T08:00:14.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.663+0000 7fe72d399700 1 -- 172.21.15.23:0/885663607 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe720012430 con 0x7fe72815c550 2023-06-22T08:00:14.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.663+0000 7fe7305ff700 1 -- 172.21.15.23:0/885663607 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe72815c550 msgr2=0x7fe7280c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.663+0000 7fe7305ff700 1 --2- 172.21.15.23:0/885663607 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe72815c550 0x7fe7280c4a50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe720009dc0 tx=0x7fe72000b330 comp rx=0 tx=0).stop 2023-06-22T08:00:14.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.663+0000 7fe7305ff700 1 -- 172.21.15.23:0/885663607 shutdown_connections 2023-06-22T08:00:14.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.663+0000 7fe7305ff700 1 --2- 172.21.15.23:0/885663607 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe72815c550 0x7fe7280c4a50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.663+0000 7fe7305ff700 1 -- 172.21.15.23:0/885663607 >> 172.21.15.23:0/885663607 conn(0x7fe728065b90 msgr2=0x7fe728065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:14.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.663+0000 7fe7305ff700 1 -- 172.21.15.23:0/885663607 shutdown_connections 2023-06-22T08:00:14.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.664+0000 7fe7305ff700 1 -- 172.21.15.23:0/885663607 wait complete. 2023-06-22T08:00:14.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.666+0000 7fe7305ff700 1 Processor -- start 2023-06-22T08:00:14.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.666+0000 7fe7305ff700 1 -- start start 2023-06-22T08:00:14.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.667+0000 7fe7305ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7280c9200 0x7fe7280cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.667+0000 7fe7305ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7280cb260 0x7fe7280cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.667+0000 7fe7305ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72815c550 0x7fe72814bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.667+0000 7fe7305ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7280b3550 con 0x7fe7280cb260 2023-06-22T08:00:14.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.667+0000 7fe7305ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7281ef0a0 con 0x7fe72815c550 2023-06-22T08:00:14.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.667+0000 7fe7305ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7280b3a80 con 0x7fe7280c9200 2023-06-22T08:00:14.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72db9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7280cb260 0x7fe7280cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72e39b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7280c9200 0x7fe7280cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72db9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7280cb260 0x7fe7280cd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44114/0 (socket says 172.21.15.23:44114) 2023-06-22T08:00:14.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72db9a700 1 -- 172.21.15.23:0/3318239715 learned_addr learned my addr 172.21.15.23:0/3318239715 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:14.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72eb9c700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72815c550 0x7fe72814bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72db9a700 1 -- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7280c9200 msgr2=0x7fe7280cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72db9a700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7280c9200 0x7fe7280cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72db9a700 1 -- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72815c550 msgr2=0x7fe72814bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72db9a700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72815c550 0x7fe72814bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72db9a700 1 -- 172.21.15.23:0/3318239715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe720009a70 con 0x7fe7280cb260 2023-06-22T08:00:14.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.668+0000 7fe72db9a700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7280cb260 0x7fe7280cd540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe71800c4b0 tx=0x7fe71800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:14.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.669+0000 7fe71effd700 1 -- 172.21.15.23:0/3318239715 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7180052d0 con 0x7fe7280cb260 2023-06-22T08:00:14.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.669+0000 7fe7305ff700 1 -- 172.21.15.23:0/3318239715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7280b37a0 con 0x7fe7280cb260 2023-06-22T08:00:14.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.669+0000 7fe71effd700 1 -- 172.21.15.23:0/3318239715 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe718005430 con 0x7fe7280cb260 2023-06-22T08:00:14.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.669+0000 7fe71effd700 1 -- 172.21.15.23:0/3318239715 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7180038c0 con 0x7fe7280cb260 2023-06-22T08:00:14.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.669+0000 7fe7305ff700 1 -- 172.21.15.23:0/3318239715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7280b4120 con 0x7fe7280cb260 2023-06-22T08:00:14.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.670+0000 7fe7305ff700 1 -- 172.21.15.23:0/3318239715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7280c5600 con 0x7fe7280cb260 2023-06-22T08:00:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.671+0000 7fe71effd700 1 -- 172.21.15.23:0/3318239715 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe718003a20 con 0x7fe7280cb260 2023-06-22T08:00:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.672+0000 7fe71effd700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe70c06ad10 0x7fe70c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.672+0000 7fe71effd700 1 -- 172.21.15.23:0/3318239715 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe718088c50 con 0x7fe7280cb260 2023-06-22T08:00:14.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.672+0000 7fe72e39b700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe70c06ad10 0x7fe70c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:14.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.673+0000 7fe72e39b700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe70c06ad10 0x7fe70c06d1c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe724007fd0 tx=0x7fe724007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:14.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.677+0000 7fe71effd700 1 -- 172.21.15.23:0/3318239715 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe7180538f0 con 0x7fe7280cb260 2023-06-22T08:00:14.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.885+0000 7fe7305ff700 1 -- 172.21.15.23:0/3318239715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7280c3a00 con 0x7fe7280cb260 2023-06-22T08:00:14.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.887+0000 7fe71effd700 1 -- 172.21.15.23:0/3318239715 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7fe718056f10 con 0x7fe7280cb260 2023-06-22T08:00:14.891 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:14.891 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:14.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 -- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe70c06ad10 msgr2=0x7fe70c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe70c06ad10 0x7fe70c06d1c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe724007fd0 tx=0x7fe724007dc0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 -- 172.21.15.23:0/3318239715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7280cb260 msgr2=0x7fe7280cd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:14.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7280cb260 0x7fe7280cd540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe71800c4b0 tx=0x7fe71800c7c0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 -- 172.21.15.23:0/3318239715 shutdown_connections 2023-06-22T08:00:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe70c06ad10 0x7fe70c06d1c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe72815c550 0x7fe72814bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7280c9200 0x7fe7280cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 --2- 172.21.15.23:0/3318239715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7280cb260 0x7fe7280cd540 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 -- 172.21.15.23:0/3318239715 >> 172.21.15.23:0/3318239715 conn(0x7fe728065b90 msgr2=0x7fe7280cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 -- 172.21.15.23:0/3318239715 shutdown_connections 2023-06-22T08:00:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:14.889+0000 7fe7305ff700 1 -- 172.21.15.23:0/3318239715 wait complete. 2023-06-22T08:00:14.911 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T07:59:40.617747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-22T07:59:48.190528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-22T07:59:53.576494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-22T07:59:54.105472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-22T07:59:57.198896+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-22T08:00:14.912 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T08:00:15.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.109+0000 7efd1f9ff700 1 Processor -- start 2023-06-22T08:00:15.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.109+0000 7efd1f9ff700 1 -- start start 2023-06-22T08:00:15.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.110+0000 7efd1f9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd181593f0 0x7efd180c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1f9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd180c4650 0x7efd180c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1f9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd180c9aa0 con 0x7efd18152ba0 2023-06-22T08:00:15.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1f9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd1806aed0 con 0x7efd180c4650 2023-06-22T08:00:15.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1f9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1806b1b0 con 0x7efd181593f0 2023-06-22T08:00:15.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1d79b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7efd18152ba0 0x7efd180c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:49268/0 (socket says 172.21.15.23:49268) 2023-06-22T08:00:15.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1d79b700 1 -- 172.21.15.23:0/2847660362 learned_addr learned my addr 172.21.15.23:0/2847660362 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:15.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1df9c700 1 --2- 172.21.15.23:0/2847660362 >> v2:172.21.15.67:3300/0 conn(0x7efd180c4650 0x7efd180c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1cf9a700 1 --2- 172.21.15.23:0/2847660362 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd181593f0 0x7efd180c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1cf9a700 1 -- 172.21.15.23:0/2847660362 >> v2:172.21.15.67:3300/0 conn(0x7efd180c4650 msgr2=0x7efd180c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:15.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1cf9a700 1 --2- 172.21.15.23:0/2847660362 >> v2:172.21.15.67:3300/0 conn(0x7efd180c4650 0x7efd180c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1cf9a700 1 -- 172.21.15.23:0/2847660362 >> v1:172.21.15.23:6789/0 conn(0x7efd18152ba0 legacy=0x7efd180c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1cf9a700 1 -- 172.21.15.23:0/2847660362 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1806b490 con 0x7efd181593f0 2023-06-22T08:00:15.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd0ffff700 1 -- 172.21.15.23:0/2847660362 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424894149 0 0) 0x7efd180c9aa0 con 0x7efd18152ba0 2023-06-22T08:00:15.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.111+0000 7efd1df9c700 1 --2- 172.21.15.23:0/2847660362 >> v2:172.21.15.67:3300/0 conn(0x7efd180c4650 0x7efd180c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:15.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.112+0000 7efd1cf9a700 1 --2- 172.21.15.23:0/2847660362 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd181593f0 0x7efd180c4110 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd08009840 tx=0x7efd08009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=799efb318280b5ce server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:15.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.112+0000 7efd0ffff700 1 -- 172.21.15.23:0/2847660362 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08011830 con 0x7efd181593f0 2023-06-22T08:00:15.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.112+0000 7efd0ffff700 1 -- 172.21.15.23:0/2847660362 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd08011990 con 0x7efd181593f0 2023-06-22T08:00:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.112+0000 7efd0ffff700 1 -- 172.21.15.23:0/2847660362 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08011d20 con 0x7efd181593f0 2023-06-22T08:00:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.112+0000 7efd1f9ff700 1 -- 172.21.15.23:0/2847660362 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd181593f0 msgr2=0x7efd180c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.112+0000 7efd1f9ff700 1 --2- 172.21.15.23:0/2847660362 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd181593f0 0x7efd180c4110 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efd08009840 tx=0x7efd08009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.112+0000 7efd1f9ff700 1 -- 172.21.15.23:0/2847660362 shutdown_connections 2023-06-22T08:00:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.112+0000 7efd1f9ff700 1 --2- 172.21.15.23:0/2847660362 >> v2:172.21.15.67:3300/0 conn(0x7efd180c4650 0x7efd180c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.112+0000 7efd1f9ff700 1 --2- 172.21.15.23:0/2847660362 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd181593f0 0x7efd180c4110 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.113+0000 7efd1f9ff700 1 -- 172.21.15.23:0/2847660362 >> 172.21.15.23:0/2847660362 conn(0x7efd18065b90 msgr2=0x7efd18065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.113+0000 7efd1f9ff700 1 -- 172.21.15.23:0/2847660362 shutdown_connections 2023-06-22T08:00:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.113+0000 7efd1f9ff700 1 -- 172.21.15.23:0/2847660362 wait complete. 2023-06-22T08:00:15.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.114+0000 7efd1f9ff700 1 Processor -- start 2023-06-22T08:00:15.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.115+0000 7efd1f9ff700 1 -- start start 2023-06-22T08:00:15.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.115+0000 7efd1f9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd180c4650 0x7efd1814d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.115+0000 7efd1f9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd18152ba0 0x7efd181ee960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.115+0000 7efd1f9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd181593f0 0x7efd181f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.115+0000 7efd1f9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd180ccba0 con 0x7efd181593f0 2023-06-22T08:00:15.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.115+0000 7efd1f9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd181ee4d0 con 0x7efd180c4650 2023-06-22T08:00:15.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1f9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd180b2250 con 0x7efd18152ba0 2023-06-22T08:00:15.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1d79b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd180c4650 0x7efd1814d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1cf9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd18152ba0 0x7efd181ee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1df9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd181593f0 0x7efd181f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1d79b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd180c4650 0x7efd1814d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:46736/0 (socket says 172.21.15.23:46736) 2023-06-22T08:00:15.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1d79b700 1 -- 172.21.15.23:0/3811744259 learned_addr learned my addr 172.21.15.23:0/3811744259 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:15.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1d79b700 1 -- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd18152ba0 msgr2=0x7efd181ee960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1d79b700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd18152ba0 0x7efd181ee960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1d79b700 1 -- 172.21.15.23:0/3811744259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd181593f0 msgr2=0x7efd181f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1d79b700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd181593f0 0x7efd181f1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1d79b700 1 -- 172.21.15.23:0/3811744259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd10009a70 con 0x7efd180c4650 2023-06-22T08:00:15.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1cf9a700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd18152ba0 0x7efd181ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:15.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1df9c700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd181593f0 0x7efd181f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:15.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.116+0000 7efd1d79b700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd180c4650 0x7efd1814d580 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c780 tx=0x7efd1400ca90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:15.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.117+0000 7efd0dffb700 1 -- 172.21.15.23:0/3811744259 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14004c80 con 0x7efd180c4650 2023-06-22T08:00:15.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.117+0000 7efd0dffb700 1 -- 172.21.15.23:0/3811744259 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd14004e00 con 0x7efd180c4650 2023-06-22T08:00:15.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.117+0000 7efd0dffb700 1 -- 172.21.15.23:0/3811744259 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14007310 con 0x7efd180c4650 2023-06-22T08:00:15.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.117+0000 7efd1f9ff700 1 -- 172.21.15.23:0/3811744259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd080094a0 con 0x7efd180c4650 2023-06-22T08:00:15.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.117+0000 7efd1f9ff700 1 -- 172.21.15.23:0/3811744259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd180ccdd0 con 0x7efd180c4650 2023-06-22T08:00:15.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.118+0000 7efd1f9ff700 1 -- 172.21.15.23:0/3811744259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd180cbed0 con 0x7efd180c4650 2023-06-22T08:00:15.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.119+0000 7efd0dffb700 1 -- 172.21.15.23:0/3811744259 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efd14007470 con 0x7efd180c4650 2023-06-22T08:00:15.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.120+0000 7efd0dffb700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcfc06ad10 0x7efcfc06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.120+0000 7efd0dffb700 1 -- 172.21.15.23:0/3811744259 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7efd14089410 con 0x7efd180c4650 2023-06-22T08:00:15.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.120+0000 7efd1cf9a700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcfc06ad10 0x7efcfc06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.121+0000 7efd1cf9a700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcfc06ad10 0x7efcfc06d1c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efd080090b0 tx=0x7efd08008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:15.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.125+0000 7efd0dffb700 1 -- 172.21.15.23:0/3811744259 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efd140541c0 con 0x7efd180c4650 2023-06-22T08:00:15.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.338+0000 7efd1f9ff700 1 -- 172.21.15.23:0/3811744259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efd1804ea20 con 0x7efd180c4650 2023-06-22T08:00:15.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.339+0000 7efd0dffb700 1 -- 172.21.15.23:0/3811744259 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7efd140577e0 con 0x7efd180c4650 2023-06-22T08:00:15.343 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-06-22T08:00:15.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 -- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcfc06ad10 msgr2=0x7efcfc06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcfc06ad10 0x7efcfc06d1c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efd080090b0 tx=0x7efd08008030 comp rx=0 tx=0).stop 2023-06-22T08:00:15.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 -- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd180c4650 msgr2=0x7efd1814d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd180c4650 0x7efd1814d580 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c780 tx=0x7efd1400ca90 comp rx=0 tx=0).stop 2023-06-22T08:00:15.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 -- 172.21.15.23:0/3811744259 shutdown_connections 2023-06-22T08:00:15.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd181593f0 0x7efd181f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcfc06ad10 0x7efcfc06d1c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd180c4650 0x7efd1814d580 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 --2- 172.21.15.23:0/3811744259 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd18152ba0 0x7efd181ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 -- 172.21.15.23:0/3811744259 >> 172.21.15.23:0/3811744259 conn(0x7efd18065b90 msgr2=0x7efd180c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:15.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 -- 172.21.15.23:0/3811744259 shutdown_connections 2023-06-22T08:00:15.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.341+0000 7efd1f9ff700 1 -- 172.21.15.23:0/3811744259 wait complete. 2023-06-22T08:00:15.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T08:00:15.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.562+0000 7fe93c1ff700 1 Processor -- start 2023-06-22T08:00:15.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.562+0000 7fe93c1ff700 1 -- start start 2023-06-22T08:00:15.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.563+0000 7fe93c1ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe934152ba0 0x7fe9341593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.563+0000 7fe93c1ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9340c9260 0x7fe9340c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.563+0000 7fe93c1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 0x7fe9340c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe93c1ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe934159930 con 0x7fe934152ba0 2023-06-22T08:00:15.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe93c1ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9340c9aa0 con 0x7fe9340c9260 2023-06-22T08:00:15.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe93c1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93406aed0 con 0x7fe9340c4650 2023-06-22T08:00:15.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe939f9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe934152ba0 0x7fe9341593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe939f9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe934152ba0 0x7fe9341593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44124/0 (socket says 172.21.15.23:44124) 2023-06-22T08:00:15.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe939f9b700 1 -- 172.21.15.23:0/3976049789 learned_addr learned my addr 172.21.15.23:0/3976049789 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:15.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe93979a700 1 --2- 172.21.15.23:0/3976049789 >> v2:172.21.15.67:3300/0 conn(0x7fe9340c9260 0x7fe9340c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe93a79c700 1 --2- 172.21.15.23:0/3976049789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 0x7fe9340c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe939f9b700 1 -- 172.21.15.23:0/3976049789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 msgr2=0x7fe9340c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:15.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe939f9b700 1 --2- 172.21.15.23:0/3976049789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 0x7fe9340c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe939f9b700 1 -- 172.21.15.23:0/3976049789 >> v2:172.21.15.67:3300/0 conn(0x7fe9340c9260 msgr2=0x7fe9340c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:15.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe939f9b700 1 --2- 172.21.15.23:0/3976049789 >> v2:172.21.15.67:3300/0 conn(0x7fe9340c9260 0x7fe9340c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.564+0000 7fe939f9b700 1 -- 172.21.15.23:0/3976049789 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93406b1b0 con 0x7fe934152ba0 2023-06-22T08:00:15.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe939f9b700 1 --2- 172.21.15.23:0/3976049789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe934152ba0 0x7fe9341593f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe9340b1670 tx=0x7fe930009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e498b9f23b505292 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:15.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe938f99700 1 -- 172.21.15.23:0/3976049789 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930011830 con 0x7fe934152ba0 2023-06-22T08:00:15.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe938f99700 1 -- 172.21.15.23:0/3976049789 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe930011990 con 0x7fe934152ba0 2023-06-22T08:00:15.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe938f99700 1 -- 172.21.15.23:0/3976049789 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930011d20 con 0x7fe934152ba0 2023-06-22T08:00:15.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe93c1ff700 1 -- 172.21.15.23:0/3976049789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe934152ba0 msgr2=0x7fe9341593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe93c1ff700 1 --2- 172.21.15.23:0/3976049789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe934152ba0 0x7fe9341593f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe9340b1670 tx=0x7fe930009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe93c1ff700 1 -- 172.21.15.23:0/3976049789 shutdown_connections 2023-06-22T08:00:15.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe93c1ff700 1 --2- 172.21.15.23:0/3976049789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 0x7fe9340c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe93c1ff700 1 --2- 172.21.15.23:0/3976049789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe934152ba0 0x7fe9341593f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe93c1ff700 1 --2- 172.21.15.23:0/3976049789 >> v2:172.21.15.67:3300/0 conn(0x7fe9340c9260 0x7fe9340c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.565+0000 7fe93c1ff700 1 -- 172.21.15.23:0/3976049789 >> 172.21.15.23:0/3976049789 conn(0x7fe934065b90 msgr2=0x7fe934065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:15.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.566+0000 7fe93c1ff700 1 -- 172.21.15.23:0/3976049789 shutdown_connections 2023-06-22T08:00:15.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.566+0000 7fe93c1ff700 1 -- 172.21.15.23:0/3976049789 wait complete. 2023-06-22T08:00:15.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.567+0000 7fe93c1ff700 1 Processor -- start 2023-06-22T08:00:15.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.568+0000 7fe93c1ff700 1 -- start start 2023-06-22T08:00:15.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.568+0000 7fe93c1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 0x7fe9340c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.568+0000 7fe93c1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9340c9260 0x7fe93414dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe93c1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe934152ba0 0x7fe9340b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe93c1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9341551c0 con 0x7fe9340c9260 2023-06-22T08:00:15.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe93c1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9340c8c90 con 0x7fe934152ba0 2023-06-22T08:00:15.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe93c1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9341f1940 con 0x7fe9340c4650 2023-06-22T08:00:15.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe93979a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9340c9260 0x7fe93414dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe939f9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 0x7fe9340c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe93979a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9340c9260 0x7fe93414dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44136/0 (socket says 172.21.15.23:44136) 2023-06-22T08:00:15.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe93979a700 1 -- 172.21.15.23:0/1843437742 learned_addr learned my addr 172.21.15.23:0/1843437742 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:15.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe93a79c700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe934152ba0 0x7fe9340b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe939f9b700 1 -- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe934152ba0 msgr2=0x7fe9340b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe939f9b700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe934152ba0 0x7fe9340b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe939f9b700 1 -- 172.21.15.23:0/1843437742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9340c9260 msgr2=0x7fe93414dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe939f9b700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9340c9260 0x7fe93414dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.569+0000 7fe939f9b700 1 -- 172.21.15.23:0/1843437742 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930009550 con 0x7fe9340c4650 2023-06-22T08:00:15.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.570+0000 7fe939f9b700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 0x7fe9340c8470 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe930016000 tx=0x7fe930004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:15.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.570+0000 7fe92a7fc700 1 -- 172.21.15.23:0/1843437742 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930004240 con 0x7fe9340c4650 2023-06-22T08:00:15.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.570+0000 7fe92a7fc700 1 -- 172.21.15.23:0/1843437742 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93001f070 con 0x7fe9340c4650 2023-06-22T08:00:15.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.570+0000 7fe93c1ff700 1 -- 172.21.15.23:0/1843437742 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9341f1c20 con 0x7fe9340c4650 2023-06-22T08:00:15.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.570+0000 7fe92a7fc700 1 -- 172.21.15.23:0/1843437742 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9300045e0 con 0x7fe9340c4650 2023-06-22T08:00:15.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.570+0000 7fe93c1ff700 1 -- 172.21.15.23:0/1843437742 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9341f21e0 con 0x7fe9340c4650 2023-06-22T08:00:15.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.571+0000 7fe93c1ff700 1 -- 172.21.15.23:0/1843437742 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9340cbed0 con 0x7fe9340c4650 2023-06-22T08:00:15.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.572+0000 7fe92a7fc700 1 -- 172.21.15.23:0/1843437742 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe930021030 con 0x7fe9340c4650 2023-06-22T08:00:15.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.573+0000 7fe92a7fc700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe91806ace0 0x7fe91806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:15.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.573+0000 7fe92a7fc700 1 -- 172.21.15.23:0/1843437742 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe93008a8e0 con 0x7fe9340c4650 2023-06-22T08:00:15.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.573+0000 7fe93979a700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe91806ace0 0x7fe91806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:15.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.574+0000 7fe93979a700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe91806ace0 0x7fe91806d190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe9340b3d80 tx=0x7fe924009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:15.581 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.578+0000 7fe92a7fc700 1 -- 172.21.15.23:0/1843437742 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe930055690 con 0x7fe9340c4650 2023-06-22T08:00:15.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.787+0000 7fe93c1ff700 1 -- 172.21.15.23:0/1843437742 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe9340cf840 con 0x7fe9340c4650 2023-06-22T08:00:15.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.789+0000 7fe92a7fc700 1 -- 172.21.15.23:0/1843437742 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe930015020 con 0x7fe9340c4650 2023-06-22T08:00:15.792 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2023-06-22T08:00:15.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 -- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe91806ace0 msgr2=0x7fe91806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe91806ace0 0x7fe91806d190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe9340b3d80 tx=0x7fe924009210 comp rx=0 tx=0).stop 2023-06-22T08:00:15.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 -- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 msgr2=0x7fe9340c8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:15.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 0x7fe9340c8470 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe930016000 tx=0x7fe930004750 comp rx=0 tx=0).stop 2023-06-22T08:00:15.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 -- 172.21.15.23:0/1843437742 shutdown_connections 2023-06-22T08:00:15.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe91806ace0 0x7fe91806d190 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe934152ba0 0x7fe9340b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9340c4650 0x7fe9340c8470 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 --2- 172.21.15.23:0/1843437742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9340c9260 0x7fe93414dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:15.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 -- 172.21.15.23:0/1843437742 >> 172.21.15.23:0/1843437742 conn(0x7fe934065b90 msgr2=0x7fe934156c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:15.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 -- 172.21.15.23:0/1843437742 shutdown_connections 2023-06-22T08:00:15.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:15.790+0000 7fe93c1ff700 1 -- 172.21.15.23:0/1843437742 wait complete. 2023-06-22T08:00:15.812 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-22T08:00:16.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.010+0000 7fa0fa5ff700 1 Processor -- start 2023-06-22T08:00:16.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.010+0000 7fa0fa5ff700 1 -- start start 2023-06-22T08:00:16.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.011+0000 7fa0fa5ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa0f4152ba0 0x7fa0f41593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0fa5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0f40c9260 0x7fa0f40c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0fa5ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f4159930 con 0x7fa0f40c4650 2023-06-22T08:00:16.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0fa5ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f40c4a60 con 0x7fa0f4152ba0 2023-06-22T08:00:16.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0fa5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f40c9aa0 con 0x7fa0f40c9260 2023-06-22T08:00:16.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0f3fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa0f4152ba0 0x7fa0f41593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0f8b9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa0f40c4650 0x7fa0f40c9ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40018/0 (socket says 172.21.15.23:40018) 2023-06-22T08:00:16.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0f8b9c700 1 -- 172.21.15.23:0/1476445693 learned_addr learned my addr 172.21.15.23:0/1476445693 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:16.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0f37fe700 1 --2- 172.21.15.23:0/1476445693 >> v2:172.21.15.67:3300/0 conn(0x7fa0f40c9260 0x7fa0f40c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0f3fff700 1 -- 172.21.15.23:0/1476445693 >> v2:172.21.15.67:3300/0 conn(0x7fa0f40c9260 msgr2=0x7fa0f40c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:16.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0f3fff700 1 --2- 172.21.15.23:0/1476445693 >> v2:172.21.15.67:3300/0 conn(0x7fa0f40c9260 0x7fa0f40c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0f3fff700 1 -- 172.21.15.23:0/1476445693 >> v1:172.21.15.23:6789/0 conn(0x7fa0f40c4650 legacy=0x7fa0f40c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.012+0000 7fa0f3fff700 1 -- 172.21.15.23:0/1476445693 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f406aed0 con 0x7fa0f4152ba0 2023-06-22T08:00:16.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.013+0000 7fa0f3fff700 1 --2- 172.21.15.23:0/1476445693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f4152ba0 0x7fa0f41593f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40b1670 tx=0x7fa0e4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1479585503475e39 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:16.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.013+0000 7fa0f2ffd700 1 -- 172.21.15.23:0/1476445693 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4011830 con 0x7fa0f4152ba0 2023-06-22T08:00:16.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.013+0000 7fa0f2ffd700 1 -- 172.21.15.23:0/1476445693 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e4011990 con 0x7fa0f4152ba0 2023-06-22T08:00:16.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.013+0000 7fa0f2ffd700 1 -- 172.21.15.23:0/1476445693 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4011d20 con 0x7fa0f4152ba0 2023-06-22T08:00:16.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.013+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/1476445693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f4152ba0 msgr2=0x7fa0f41593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.013+0000 7fa0fa5ff700 1 --2- 172.21.15.23:0/1476445693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f4152ba0 0x7fa0f41593f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40b1670 tx=0x7fa0e4009b20 comp rx=0 tx=0).stop 2023-06-22T08:00:16.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.014+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/1476445693 shutdown_connections 2023-06-22T08:00:16.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.014+0000 7fa0fa5ff700 1 --2- 172.21.15.23:0/1476445693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f4152ba0 0x7fa0f41593f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.014+0000 7fa0fa5ff700 1 --2- 172.21.15.23:0/1476445693 >> v2:172.21.15.67:3300/0 conn(0x7fa0f40c9260 0x7fa0f40c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.014+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/1476445693 >> 172.21.15.23:0/1476445693 conn(0x7fa0f4065b90 msgr2=0x7fa0f4065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:16.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.014+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/1476445693 shutdown_connections 2023-06-22T08:00:16.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.014+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/1476445693 wait complete. 2023-06-22T08:00:16.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.015+0000 7fa0fa5ff700 1 Processor -- start 2023-06-22T08:00:16.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.016+0000 7fa0fa5ff700 1 -- start start 2023-06-22T08:00:16.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.016+0000 7fa0fa5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f40c4650 0x7fa0f40b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.016+0000 7fa0fa5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0f40c9260 0x7fa0f414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0fa5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f4152ba0 0x7fa0f40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0fa5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f414b510 con 0x7fa0f40c4650 2023-06-22T08:00:16.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0fa5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f40b38a0 con 0x7fa0f4152ba0 2023-06-22T08:00:16.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0fa5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f41f1cb0 con 0x7fa0f40c9260 2023-06-22T08:00:16.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0f3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f40c4650 0x7fa0f40b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0f3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f40c4650 0x7fa0f40b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38602/0 (socket says 172.21.15.23:38602) 2023-06-22T08:00:16.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0f3fff700 1 -- 172.21.15.23:0/2686664344 learned_addr learned my addr 172.21.15.23:0/2686664344 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:16.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0f8b9c700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f4152ba0 0x7fa0f40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0f37fe700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0f40c9260 0x7fa0f414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0f3fff700 1 -- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0f40c9260 msgr2=0x7fa0f414d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0f3fff700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0f40c9260 0x7fa0f414d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0f3fff700 1 -- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f4152ba0 msgr2=0x7fa0f40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0f3fff700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f4152ba0 0x7fa0f40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.017+0000 7fa0f3fff700 1 -- 172.21.15.23:0/2686664344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e40094a0 con 0x7fa0f40c4650 2023-06-22T08:00:16.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.018+0000 7fa0f3fff700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f40c4650 0x7fa0f40b3b80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009b60 tx=0x7fa0e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:16.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.018+0000 7fa0f0ff9700 1 -- 172.21.15.23:0/2686664344 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e40036a0 con 0x7fa0f40c4650 2023-06-22T08:00:16.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.018+0000 7fa0f0ff9700 1 -- 172.21.15.23:0/2686664344 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e401e070 con 0x7fa0f40c4650 2023-06-22T08:00:16.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.018+0000 7fa0f0ff9700 1 -- 172.21.15.23:0/2686664344 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4003a60 con 0x7fa0f40c4650 2023-06-22T08:00:16.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.018+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/2686664344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f414b270 con 0x7fa0f40c4650 2023-06-22T08:00:16.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.018+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/2686664344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f41f1ea0 con 0x7fa0f40c4650 2023-06-22T08:00:16.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.019+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/2686664344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f40cbed0 con 0x7fa0f40c4650 2023-06-22T08:00:16.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.020+0000 7fa0f0ff9700 1 -- 172.21.15.23:0/2686664344 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa0e4003bc0 con 0x7fa0f40c4650 2023-06-22T08:00:16.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.021+0000 7fa0f0ff9700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa0e006ace0 0x7fa0e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.021+0000 7fa0f0ff9700 1 -- 172.21.15.23:0/2686664344 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa0e408a670 con 0x7fa0f40c4650 2023-06-22T08:00:16.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.021+0000 7fa0f37fe700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa0e006ace0 0x7fa0e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.022+0000 7fa0f37fe700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa0e006ace0 0x7fa0e006d190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa0f414b890 tx=0x7fa0e8009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:16.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.025+0000 7fa0f0ff9700 1 -- 172.21.15.23:0/2686664344 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa0e4055420 con 0x7fa0f40c4650 2023-06-22T08:00:16.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.232+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/2686664344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa0f40b39e0 con 0x7fa0f40c4650 2023-06-22T08:00:16.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.233+0000 7fa0f0ff9700 1 -- 172.21.15.23:0/2686664344 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa0f40b39e0 con 0x7fa0f40c4650 2023-06-22T08:00:16.237 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:16.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa0e006ace0 msgr2=0x7fa0e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa0e006ace0 0x7fa0e006d190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa0f414b890 tx=0x7fa0e8009210 comp rx=0 tx=0).stop 2023-06-22T08:00:16.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/2686664344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f40c4650 msgr2=0x7fa0f40b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f40c4650 0x7fa0f40b3b80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009b60 tx=0x7fa0e4004750 comp rx=0 tx=0).stop 2023-06-22T08:00:16.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/2686664344 shutdown_connections 2023-06-22T08:00:16.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa0e006ace0 0x7fa0e006d190 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0f4152ba0 0x7fa0f40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0f40c4650 0x7fa0f40b3b80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 --2- 172.21.15.23:0/2686664344 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0f40c9260 0x7fa0f414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/2686664344 >> 172.21.15.23:0/2686664344 conn(0x7fa0f4065b90 msgr2=0x7fa0f4151e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:16.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/2686664344 shutdown_connections 2023-06-22T08:00:16.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.235+0000 7fa0fa5ff700 1 -- 172.21.15.23:0/2686664344 wait complete. 2023-06-22T08:00:16.257 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-22T08:00:16.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.455+0000 7f36029ff700 1 Processor -- start 2023-06-22T08:00:16.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.455+0000 7f36029ff700 1 -- start start 2023-06-22T08:00:16.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.456+0000 7f36029ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 0x7f35fc0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.456+0000 7f36029ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35fc0c9aa0 con 0x7f35fc0c4650 2023-06-22T08:00:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.456+0000 7f36029ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35fc06aed0 con 0x7f35fc152ba0 2023-06-22T08:00:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.456+0000 7f36029ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc06b1b0 con 0x7f35fc1593f0 2023-06-22T08:00:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.457+0000 7f35fbfff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f35fc152ba0 0x7f35fc0c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39158/0 (socket says 172.21.15.23:39158) 2023-06-22T08:00:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.457+0000 7f35fb7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 0x7f35fc0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.457+0000 7f35fbfff700 1 -- 172.21.15.23:0/1281706543 learned_addr learned my addr 172.21.15.23:0/1281706543 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.457+0000 7f35faffd700 1 -- 172.21.15.23:0/1281706543 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190845561 0 0) 0x7f35fc06aed0 con 0x7f35fc152ba0 2023-06-22T08:00:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.457+0000 7f35faffd700 1 -- 172.21.15.23:0/1281706543 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e40035b0 con 0x7f35fc152ba0 2023-06-22T08:00:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.457+0000 7f35fb7fe700 1 -- 172.21.15.23:0/1281706543 >> v1:172.21.15.67:6789/0 conn(0x7f35fc152ba0 legacy=0x7f35fc0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.457+0000 7f35fb7fe700 1 -- 172.21.15.23:0/1281706543 >> v1:172.21.15.23:6789/0 conn(0x7f35fc0c4650 legacy=0x7f35fc0c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.457+0000 7f35fb7fe700 1 -- 172.21.15.23:0/1281706543 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc06b490 con 0x7f35fc1593f0 2023-06-22T08:00:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.457+0000 7f35fb7fe700 1 --2- 172.21.15.23:0/1281706543 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 0x7f35fc0c4110 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009870 tx=0x7f35f0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1997b0f8056c446 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.458+0000 7f35faffd700 1 -- 172.21.15.23:0/1281706543 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0011830 con 0x7f35fc1593f0 2023-06-22T08:00:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.458+0000 7f35faffd700 1 -- 172.21.15.23:0/1281706543 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f0011990 con 0x7f35fc1593f0 2023-06-22T08:00:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.458+0000 7f35faffd700 1 -- 172.21.15.23:0/1281706543 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0011d20 con 0x7f35fc1593f0 2023-06-22T08:00:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.458+0000 7f36029ff700 1 -- 172.21.15.23:0/1281706543 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 msgr2=0x7f35fc0c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.458+0000 7f36029ff700 1 --2- 172.21.15.23:0/1281706543 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 0x7f35fc0c4110 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009870 tx=0x7f35f0009b80 comp rx=0 tx=0).stop 2023-06-22T08:00:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.458+0000 7f36029ff700 1 -- 172.21.15.23:0/1281706543 shutdown_connections 2023-06-22T08:00:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.458+0000 7f36029ff700 1 --2- 172.21.15.23:0/1281706543 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 0x7f35fc0c4110 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.458+0000 7f36029ff700 1 -- 172.21.15.23:0/1281706543 >> 172.21.15.23:0/1281706543 conn(0x7f35fc065b90 msgr2=0x7f35fc065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.458+0000 7f36029ff700 1 -- 172.21.15.23:0/1281706543 shutdown_connections 2023-06-22T08:00:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.458+0000 7f36029ff700 1 -- 172.21.15.23:0/1281706543 wait complete. 2023-06-22T08:00:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.460+0000 7f36029ff700 1 Processor -- start 2023-06-22T08:00:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.460+0000 7f36029ff700 1 -- start start 2023-06-22T08:00:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f36029ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0c4650 0x7f35fc0c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f36029ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc152ba0 0x7f35fc1f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f36029ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 0x7f35fc14b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f36029ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0c8020 con 0x7f35fc152ba0 2023-06-22T08:00:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f36029ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0b3100 con 0x7f35fc0c4650 2023-06-22T08:00:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f36029ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc1f16c0 con 0x7f35fc1593f0 2023-06-22T08:00:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f35fb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc152ba0 0x7f35fc1f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f3600f9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 0x7f35fc14b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f35fb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc152ba0 0x7f35fc1f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38604/0 (socket says 172.21.15.23:38604) 2023-06-22T08:00:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f35fb7fe700 1 -- 172.21.15.23:0/2396980973 learned_addr learned my addr 172.21.15.23:0/2396980973 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.461+0000 7f35fbfff700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0c4650 0x7f35fc0c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f35fb7fe700 1 -- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 msgr2=0x7f35fc14b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f35fb7fe700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 0x7f35fc14b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f35fb7fe700 1 -- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0c4650 msgr2=0x7f35fc0c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f35fb7fe700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0c4650 0x7f35fc0c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f35fb7fe700 1 -- 172.21.15.23:0/2396980973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00094a0 con 0x7f35fc152ba0 2023-06-22T08:00:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f3600f9c700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 0x7f35fc14b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f35fb7fe700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc152ba0 0x7f35fc1f1e50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f35f00090b0 tx=0x7f35f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f35f8ff9700 1 -- 172.21.15.23:0/2396980973 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f00036a0 con 0x7f35fc152ba0 2023-06-22T08:00:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f35f8ff9700 1 -- 172.21.15.23:0/2396980973 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f001f070 con 0x7f35fc152ba0 2023-06-22T08:00:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f36029ff700 1 -- 172.21.15.23:0/2396980973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35fc0cc960 con 0x7f35fc152ba0 2023-06-22T08:00:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.462+0000 7f35f8ff9700 1 -- 172.21.15.23:0/2396980973 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0003a60 con 0x7f35fc152ba0 2023-06-22T08:00:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.463+0000 7f36029ff700 1 -- 172.21.15.23:0/2396980973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35fc0b3610 con 0x7f35fc152ba0 2023-06-22T08:00:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.464+0000 7f36029ff700 1 -- 172.21.15.23:0/2396980973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35fc0cbed0 con 0x7f35fc152ba0 2023-06-22T08:00:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.464+0000 7f35f8ff9700 1 -- 172.21.15.23:0/2396980973 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f35f0003bc0 con 0x7f35fc152ba0 2023-06-22T08:00:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.465+0000 7f35f8ff9700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f35e806ace0 0x7f35e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.465+0000 7f35f8ff9700 1 -- 172.21.15.23:0/2396980973 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f35f0021070 con 0x7f35fc152ba0 2023-06-22T08:00:16.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.466+0000 7f35fbfff700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f35e806ace0 0x7f35e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.466+0000 7f35fbfff700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f35e806ace0 0x7f35e806d190 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f35ec007fd0 tx=0x7f35ec007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:16.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.470+0000 7f35f8ff9700 1 -- 172.21.15.23:0/2396980973 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f35f0055d30 con 0x7f35fc152ba0 2023-06-22T08:00:16.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.676+0000 7f36029ff700 1 -- 172.21.15.23:0/2396980973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f35fc04ea20 con 0x7f35fc152ba0 2023-06-22T08:00:16.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.677+0000 7f35f8ff9700 1 -- 172.21.15.23:0/2396980973 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f35f0015020 con 0x7f35fc152ba0 2023-06-22T08:00:16.681 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-06-22T08:00:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 -- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f35e806ace0 msgr2=0x7f35e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f35e806ace0 0x7f35e806d190 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f35ec007fd0 tx=0x7f35ec007dc0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 -- 172.21.15.23:0/2396980973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc152ba0 msgr2=0x7f35fc1f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc152ba0 0x7f35fc1f1e50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f35f00090b0 tx=0x7f35f0004750 comp rx=0 tx=0).stop 2023-06-22T08:00:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 -- 172.21.15.23:0/2396980973 shutdown_connections 2023-06-22T08:00:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35fc1593f0 0x7f35fc14b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f35e806ace0 0x7f35e806d190 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35fc0c4650 0x7f35fc0c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 --2- 172.21.15.23:0/2396980973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc152ba0 0x7f35fc1f1e50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 -- 172.21.15.23:0/2396980973 >> 172.21.15.23:0/2396980973 conn(0x7f35fc065b90 msgr2=0x7f35fc151c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 -- 172.21.15.23:0/2396980973 shutdown_connections 2023-06-22T08:00:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.679+0000 7f36029ff700 1 -- 172.21.15.23:0/2396980973 wait complete. 2023-06-22T08:00:16.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-22T08:00:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.901+0000 7f0010bff700 1 Processor -- start 2023-06-22T08:00:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.901+0000 7f0010bff700 1 -- start start 2023-06-22T08:00:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.902+0000 7f0010bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f000c1593f0 0x7f000c0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.902+0000 7f0010bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c0c4650 0x7f000c0c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.902+0000 7f0010bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c0c9aa0 con 0x7f000c152ba0 2023-06-22T08:00:16.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.902+0000 7f0010bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f000c06aed0 con 0x7f000c1593f0 2023-06-22T08:00:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.902+0000 7f0010bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000c06b1b0 con 0x7f000c0c4650 2023-06-22T08:00:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.902+0000 7f0009d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f000c1593f0 0x7f000c0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f000a59c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f000c152ba0 0x7f000c0c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39170/0 (socket says 172.21.15.23:39170) 2023-06-22T08:00:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f000a59c700 1 -- 172.21.15.23:0/3895530062 learned_addr learned my addr 172.21.15.23:0/3895530062 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f000ad9d700 1 --2- 172.21.15.23:0/3895530062 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c0c4650 0x7f000c0c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f000959a700 1 -- 172.21.15.23:0/3895530062 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841032923 0 0) 0x7f000c0c9aa0 con 0x7f000c152ba0 2023-06-22T08:00:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f000959a700 1 -- 172.21.15.23:0/3895530062 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff40035b0 con 0x7f000c152ba0 2023-06-22T08:00:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f0009d9b700 1 -- 172.21.15.23:0/3895530062 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c0c4650 msgr2=0x7f000c0c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f0009d9b700 1 --2- 172.21.15.23:0/3895530062 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c0c4650 0x7f000c0c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f0009d9b700 1 -- 172.21.15.23:0/3895530062 >> v1:172.21.15.67:6789/0 conn(0x7f000c152ba0 legacy=0x7f000c0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f0009d9b700 1 -- 172.21.15.23:0/3895530062 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c06b490 con 0x7f000c1593f0 2023-06-22T08:00:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f000ad9d700 1 --2- 172.21.15.23:0/3895530062 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c0c4650 0x7f000c0c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.903+0000 7f0009d9b700 1 --2- 172.21.15.23:0/3895530062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c1593f0 0x7f000c0c4110 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0000009840 tx=0x7f0000009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77b17e156cd535bc server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f000959a700 1 -- 172.21.15.23:0/3895530062 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000011830 con 0x7f000c1593f0 2023-06-22T08:00:16.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f000959a700 1 -- 172.21.15.23:0/3895530062 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0000011990 con 0x7f000c1593f0 2023-06-22T08:00:16.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f000959a700 1 -- 172.21.15.23:0/3895530062 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000011d20 con 0x7f000c1593f0 2023-06-22T08:00:16.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f0010bff700 1 -- 172.21.15.23:0/3895530062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c1593f0 msgr2=0x7f000c0c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f0010bff700 1 --2- 172.21.15.23:0/3895530062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c1593f0 0x7f000c0c4110 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0000009840 tx=0x7f0000009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:16.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f0010bff700 1 -- 172.21.15.23:0/3895530062 shutdown_connections 2023-06-22T08:00:16.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f0010bff700 1 --2- 172.21.15.23:0/3895530062 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c0c4650 0x7f000c0c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f0010bff700 1 --2- 172.21.15.23:0/3895530062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c1593f0 0x7f000c0c4110 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f0010bff700 1 -- 172.21.15.23:0/3895530062 >> 172.21.15.23:0/3895530062 conn(0x7f000c065b90 msgr2=0x7f000c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:16.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f0010bff700 1 -- 172.21.15.23:0/3895530062 shutdown_connections 2023-06-22T08:00:16.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.904+0000 7f0010bff700 1 -- 172.21.15.23:0/3895530062 wait complete. 2023-06-22T08:00:16.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.906+0000 7f0010bff700 1 Processor -- start 2023-06-22T08:00:16.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.906+0000 7f0010bff700 1 -- start start 2023-06-22T08:00:16.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.907+0000 7f0010bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c0c4650 0x7f000c1f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.907+0000 7f0010bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c152ba0 0x7f000c157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.907+0000 7f0010bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f000c1593f0 0x7f000c1ef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.907+0000 7f0010bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c1ef550 con 0x7f000c1593f0 2023-06-22T08:00:16.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.907+0000 7f0010bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c0c7de0 con 0x7f000c0c4650 2023-06-22T08:00:16.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.907+0000 7f0010bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000c0c80c0 con 0x7f000c152ba0 2023-06-22T08:00:16.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.907+0000 7f000a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c0c4650 0x7f000c1f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.907+0000 7f0009d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c152ba0 0x7f000c157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.907+0000 7f000a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c0c4650 0x7f000c1f2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:48050/0 (socket says 172.21.15.23:48050) 2023-06-22T08:00:16.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.908+0000 7f000a59c700 1 -- 172.21.15.23:0/1278883379 learned_addr learned my addr 172.21.15.23:0/1278883379 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:16.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.908+0000 7f000ad9d700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f000c1593f0 0x7f000c1ef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.908+0000 7f000a59c700 1 -- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c152ba0 msgr2=0x7f000c157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.908+0000 7f000a59c700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c152ba0 0x7f000c157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.908+0000 7f000a59c700 1 -- 172.21.15.23:0/1278883379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f000c1593f0 msgr2=0x7f000c1ef9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:16.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.908+0000 7f000a59c700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f000c1593f0 0x7f000c1ef9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:16.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.908+0000 7f000a59c700 1 -- 172.21.15.23:0/1278883379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004009a70 con 0x7f000c0c4650 2023-06-22T08:00:16.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.908+0000 7f0009d9b700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c152ba0 0x7f000c157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:16.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.908+0000 7f000a59c700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c0c4650 0x7f000c1f2060 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efffc00c700 tx=0x7efffc00ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:16.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.908+0000 7efffaffd700 1 -- 172.21.15.23:0/1278883379 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc005d70 con 0x7f000c0c4650 2023-06-22T08:00:16.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.909+0000 7efffaffd700 1 -- 172.21.15.23:0/1278883379 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc0070d0 con 0x7f000c0c4650 2023-06-22T08:00:16.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.909+0000 7efffaffd700 1 -- 172.21.15.23:0/1278883379 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc007460 con 0x7f000c0c4650 2023-06-22T08:00:16.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.909+0000 7f0010bff700 1 -- 172.21.15.23:0/1278883379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00000094a0 con 0x7f000c0c4650 2023-06-22T08:00:16.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.909+0000 7f0010bff700 1 -- 172.21.15.23:0/1278883379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000c0c8680 con 0x7f000c0c4650 2023-06-22T08:00:16.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.910+0000 7f0010bff700 1 -- 172.21.15.23:0/1278883379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000c0cbed0 con 0x7f000c0c4650 2023-06-22T08:00:16.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.911+0000 7efffaffd700 1 -- 172.21.15.23:0/1278883379 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efffc0075c0 con 0x7f000c0c4650 2023-06-22T08:00:16.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.912+0000 7efffaffd700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effec06ace0 0x7effec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:16.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.912+0000 7efffaffd700 1 -- 172.21.15.23:0/1278883379 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7efffc088fe0 con 0x7f000c0c4650 2023-06-22T08:00:16.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.912+0000 7f0009d9b700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effec06ace0 0x7effec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:16.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.913+0000 7f0009d9b700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effec06ace0 0x7effec06d190 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0000000c00 tx=0x7f000001d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:16.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:16.916+0000 7efffaffd700 1 -- 172.21.15.23:0/1278883379 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efffc053cf0 con 0x7f000c0c4650 2023-06-22T08:00:17.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.122+0000 7f0010bff700 1 -- 172.21.15.23:0/1278883379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f000c1ef830 con 0x7f000c0c4650 2023-06-22T08:00:17.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.123+0000 7efffaffd700 1 -- 172.21.15.23:0/1278883379 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7efffc017020 con 0x7f000c0c4650 2023-06-22T08:00:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:17.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 -- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effec06ace0 msgr2=0x7effec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:17.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effec06ace0 0x7effec06d190 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0000000c00 tx=0x7f000001d040 comp rx=0 tx=0).stop 2023-06-22T08:00:17.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 -- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c0c4650 msgr2=0x7f000c1f2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:17.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c0c4650 0x7f000c1f2060 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efffc00c700 tx=0x7efffc00ca10 comp rx=0 tx=0).stop 2023-06-22T08:00:17.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 -- 172.21.15.23:0/1278883379 shutdown_connections 2023-06-22T08:00:17.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f000c1593f0 0x7f000c1ef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effec06ace0 0x7effec06d190 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f000c0c4650 0x7f000c1f2060 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 --2- 172.21.15.23:0/1278883379 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f000c152ba0 0x7f000c157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 -- 172.21.15.23:0/1278883379 >> 172.21.15.23:0/1278883379 conn(0x7f000c065b90 msgr2=0x7f000c0c3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:17.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 -- 172.21.15.23:0/1278883379 shutdown_connections 2023-06-22T08:00:17.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.126+0000 7f0010bff700 1 -- 172.21.15.23:0/1278883379 wait complete. 2023-06-22T08:00:17.148 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-22T08:00:17.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.344+0000 7f00acfff700 1 Processor -- start 2023-06-22T08:00:17.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.344+0000 7f00acfff700 1 -- start start 2023-06-22T08:00:17.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.345+0000 7f00acfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00a80c97d0 0x7f00a8159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:17.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.345+0000 7f00acfff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a80c9c30 con 0x7f00a8152d70 2023-06-22T08:00:17.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.345+0000 7f00acfff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a815bea0 con 0x7f00a80c45f0 2023-06-22T08:00:17.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.345+0000 7f00acfff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a80c95e0 con 0x7f00a80c97d0 2023-06-22T08:00:17.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.345+0000 7f00a659c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f00a8152d70 0x7f00a80cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40024/0 (socket says 172.21.15.23:40024) 2023-06-22T08:00:17.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.345+0000 7f00a659c700 1 -- 172.21.15.23:0/2089808049 learned_addr learned my addr 172.21.15.23:0/2089808049 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:17.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.346+0000 7f00a5d9b700 1 --2- 172.21.15.23:0/2089808049 >> v2:172.21.15.67:3300/0 conn(0x7f00a80c97d0 0x7f00a8159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:17.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.346+0000 7f00a559a700 1 -- 172.21.15.23:0/2089808049 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192532370 0 0) 0x7f00a80c9c30 con 0x7f00a8152d70 2023-06-22T08:00:17.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.346+0000 7f00a559a700 1 -- 172.21.15.23:0/2089808049 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00900035b0 con 0x7f00a8152d70 2023-06-22T08:00:17.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.346+0000 7f00a5d9b700 1 -- 172.21.15.23:0/2089808049 >> v1:172.21.15.67:6789/0 conn(0x7f00a80c45f0 legacy=0x7f00a80c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:17.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.346+0000 7f00a5d9b700 1 -- 172.21.15.23:0/2089808049 >> v1:172.21.15.23:6789/0 conn(0x7f00a8152d70 legacy=0x7f00a80cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:17.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.346+0000 7f00a5d9b700 1 -- 172.21.15.23:0/2089808049 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a815c160 con 0x7f00a80c97d0 2023-06-22T08:00:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.346+0000 7f00a5d9b700 1 --2- 172.21.15.23:0/2089808049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 0x7f00a8159330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f009c009840 tx=0x7f009c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=695505fdf4a669b6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.347+0000 7f00a559a700 1 -- 172.21.15.23:0/2089808049 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c011830 con 0x7f00a80c97d0 2023-06-22T08:00:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.347+0000 7f00a559a700 1 -- 172.21.15.23:0/2089808049 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c011990 con 0x7f00a80c97d0 2023-06-22T08:00:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.347+0000 7f00a559a700 1 -- 172.21.15.23:0/2089808049 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c011d20 con 0x7f00a80c97d0 2023-06-22T08:00:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.347+0000 7f00acfff700 1 -- 172.21.15.23:0/2089808049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 msgr2=0x7f00a8159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.347+0000 7f00acfff700 1 --2- 172.21.15.23:0/2089808049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 0x7f00a8159330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f009c009840 tx=0x7f009c009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.347+0000 7f00acfff700 1 -- 172.21.15.23:0/2089808049 shutdown_connections 2023-06-22T08:00:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.347+0000 7f00acfff700 1 --2- 172.21.15.23:0/2089808049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 0x7f00a8159330 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.347+0000 7f00acfff700 1 -- 172.21.15.23:0/2089808049 >> 172.21.15.23:0/2089808049 conn(0x7f00a806ae30 msgr2=0x7f00a806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:17.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.347+0000 7f00acfff700 1 -- 172.21.15.23:0/2089808049 shutdown_connections 2023-06-22T08:00:17.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.347+0000 7f00acfff700 1 -- 172.21.15.23:0/2089808049 wait complete. 2023-06-22T08:00:17.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.350+0000 7f00acfff700 1 Processor -- start 2023-06-22T08:00:17.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.350+0000 7f00acfff700 1 -- start start 2023-06-22T08:00:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00acfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00a80c45f0 0x7f00a814c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00acfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 0x7f00a80b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00acfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f00a8152d70 0x7f00a80b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00acfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a81f1840 con 0x7f00a80c45f0 2023-06-22T08:00:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00acfff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a81f1a90 con 0x7f00a80c97d0 2023-06-22T08:00:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00acfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a81f1d70 con 0x7f00a8152d70 2023-06-22T08:00:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00a659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00a80c45f0 0x7f00a814c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00a5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 0x7f00a80b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00a5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 0x7f00a80b3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:48062/0 (socket says 172.21.15.23:48062) 2023-06-22T08:00:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00a5d9b700 1 -- 172.21.15.23:0/3509773337 learned_addr learned my addr 172.21.15.23:0/3509773337 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00a659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00a80c45f0 0x7f00a814c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38614/0 (socket says 172.21.15.23:38614) 2023-06-22T08:00:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.351+0000 7f00a6d9d700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f00a8152d70 0x7f00a80b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.352+0000 7f00a5d9b700 1 -- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f00a8152d70 msgr2=0x7f00a80b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:17.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.352+0000 7f00a5d9b700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f00a8152d70 0x7f00a80b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.352+0000 7f00a5d9b700 1 -- 172.21.15.23:0/3509773337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00a80c45f0 msgr2=0x7f00a814c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:17.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.352+0000 7f00a5d9b700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00a80c45f0 0x7f00a814c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.352+0000 7f00a5d9b700 1 -- 172.21.15.23:0/3509773337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c0094a0 con 0x7f00a80c97d0 2023-06-22T08:00:17.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.352+0000 7f00a5d9b700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 0x7f00a80b3b10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f009c009fd0 tx=0x7f009c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:17.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.352+0000 7f0096ffd700 1 -- 172.21.15.23:0/3509773337 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c0036a0 con 0x7f00a80c97d0 2023-06-22T08:00:17.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.352+0000 7f0096ffd700 1 -- 172.21.15.23:0/3509773337 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c01f070 con 0x7f00a80c97d0 2023-06-22T08:00:17.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.353+0000 7f00acfff700 1 -- 172.21.15.23:0/3509773337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a81f2050 con 0x7f00a80c97d0 2023-06-22T08:00:17.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.353+0000 7f0096ffd700 1 -- 172.21.15.23:0/3509773337 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c003a60 con 0x7f00a80c97d0 2023-06-22T08:00:17.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.353+0000 7f00acfff700 1 -- 172.21.15.23:0/3509773337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a81f2610 con 0x7f00a80c97d0 2023-06-22T08:00:17.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.353+0000 7f00acfff700 1 -- 172.21.15.23:0/3509773337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a804ea20 con 0x7f00a80c97d0 2023-06-22T08:00:17.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.355+0000 7f0096ffd700 1 -- 172.21.15.23:0/3509773337 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f009c003bc0 con 0x7f00a80c97d0 2023-06-22T08:00:17.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.356+0000 7f0096ffd700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f008806ace0 0x7f008806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:17.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.356+0000 7f0096ffd700 1 -- 172.21.15.23:0/3509773337 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f009c021070 con 0x7f00a80c97d0 2023-06-22T08:00:17.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.356+0000 7f00a659c700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f008806ace0 0x7f008806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:17.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.357+0000 7f00a659c700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f008806ace0 0x7f008806d190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0098007fd0 tx=0x7f0098007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:17.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.359+0000 7f0096ffd700 1 -- 172.21.15.23:0/3509773337 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f009c0563e0 con 0x7f00a80c97d0 2023-06-22T08:00:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.583+0000 7f00acfff700 1 -- 172.21.15.23:0/3509773337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f00a80c9e00 con 0x7f00a80c97d0 2023-06-22T08:00:17.588 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.584+0000 7f0096ffd700 1 -- 172.21.15.23:0/3509773337 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7f009c015080 con 0x7f00a80c97d0 2023-06-22T08:00:17.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 -- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f008806ace0 msgr2=0x7f008806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f008806ace0 0x7f008806d190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0098007fd0 tx=0x7f0098007dc0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 -- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 msgr2=0x7f00a80b3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 0x7f00a80b3b10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f009c009fd0 tx=0x7f009c004750 comp rx=0 tx=0).stop 2023-06-22T08:00:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 -- 172.21.15.23:0/3509773337 shutdown_connections 2023-06-22T08:00:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f00a8152d70 0x7f00a80b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00a80c45f0 0x7f00a814c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f008806ace0 0x7f008806d190 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 --2- 172.21.15.23:0/3509773337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00a80c97d0 0x7f00a80b3b10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 -- 172.21.15.23:0/3509773337 >> 172.21.15.23:0/3509773337 conn(0x7f00a806ae30 msgr2=0x7f00a80ca4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 -- 172.21.15.23:0/3509773337 shutdown_connections 2023-06-22T08:00:17.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:17.587+0000 7f00acfff700 1 -- 172.21.15.23:0/3509773337 wait complete. 2023-06-22T08:00:17.594 INFO:teuthology.orchestra.run.smithi023.stderr:listed 0 entries 2023-06-22T08:00:17.609 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi023.front.sepia.ceph.com... 2023-06-22T08:00:17.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:17.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-06-22T08:00:17.678 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[137150]: fuse finished with error 0 and tester_r 0 2023-06-22T08:00:17.682 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:00:17.774 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T08:00:17.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:17.775 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-06-22T08:00:17.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:18.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.024+0000 7f8b4e3ff700 1 Processor -- start 2023-06-22T08:00:18.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.024+0000 7f8b4e3ff700 1 -- start start 2023-06-22T08:00:18.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b4e3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8b480c9fd0 0x7f8b480cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b4e3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c40b0 0x7f8b480c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b4e3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4815c370 con 0x7f8b48152ce0 2023-06-22T08:00:18.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b4e3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b481530b0 con 0x7f8b480c9fd0 2023-06-22T08:00:18.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b4e3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480cf6f0 con 0x7f8b480c40b0 2023-06-22T08:00:18.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b477fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8b480c9fd0 0x7f8b480cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b477fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8b480c9fd0 0x7f8b480cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38620/0 (socket says 172.21.15.23:38620) 2023-06-22T08:00:18.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b477fe700 1 -- 172.21.15.23:0/3058040789 learned_addr learned my addr 172.21.15.23:0/3058040789 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:18.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b4c99c700 1 --2- 172.21.15.23:0/3058040789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c40b0 0x7f8b480c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b477fe700 1 -- 172.21.15.23:0/3058040789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c40b0 msgr2=0x7f8b480c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:18.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b477fe700 1 --2- 172.21.15.23:0/3058040789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c40b0 0x7f8b480c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b477fe700 1 -- 172.21.15.23:0/3058040789 >> v1:172.21.15.67:6789/0 conn(0x7f8b48152ce0 legacy=0x7f8b4815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.027+0000 7f8b477fe700 1 -- 172.21.15.23:0/3058040789 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4815bda0 con 0x7f8b480c9fd0 2023-06-22T08:00:18.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.027+0000 7f8b4c99c700 1 --2- 172.21.15.23:0/3058040789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c40b0 0x7f8b480c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:18.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.026+0000 7f8b46ffd700 1 -- 172.21.15.23:0/3058040789 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677926145 0 0) 0x7f8b4815c370 con 0x7f8b48152ce0 2023-06-22T08:00:18.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.027+0000 7f8b477fe700 1 --2- 172.21.15.23:0/3058040789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c9fd0 0x7f8b480cf1b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c0098f0 tx=0x7f8b3c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5336498ad592ce1 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:18.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.027+0000 7f8b46ffd700 1 -- 172.21.15.23:0/3058040789 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c00b8a0 con 0x7f8b480c9fd0 2023-06-22T08:00:18.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.027+0000 7f8b46ffd700 1 -- 172.21.15.23:0/3058040789 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c00ba00 con 0x7f8b480c9fd0 2023-06-22T08:00:18.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.027+0000 7f8b46ffd700 1 -- 172.21.15.23:0/3058040789 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c00bd90 con 0x7f8b480c9fd0 2023-06-22T08:00:18.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.027+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/3058040789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c9fd0 msgr2=0x7f8b480cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.027+0000 7f8b4e3ff700 1 --2- 172.21.15.23:0/3058040789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c9fd0 0x7f8b480cf1b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c0098f0 tx=0x7f8b3c009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:18.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.028+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/3058040789 shutdown_connections 2023-06-22T08:00:18.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.028+0000 7f8b4e3ff700 1 --2- 172.21.15.23:0/3058040789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c40b0 0x7f8b480c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.028+0000 7f8b4e3ff700 1 --2- 172.21.15.23:0/3058040789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c9fd0 0x7f8b480cf1b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.028+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/3058040789 >> 172.21.15.23:0/3058040789 conn(0x7f8b4806ae30 msgr2=0x7f8b4806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:18.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.028+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/3058040789 shutdown_connections 2023-06-22T08:00:18.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.028+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/3058040789 wait complete. 2023-06-22T08:00:18.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.029+0000 7f8b4e3ff700 1 Processor -- start 2023-06-22T08:00:18.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.030+0000 7f8b4e3ff700 1 -- start start 2023-06-22T08:00:18.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.030+0000 7f8b4e3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c40b0 0x7f8b481eef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b4e3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c9fd0 0x7f8b4814f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b4e3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b48152ce0 0x7f8b480b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b4e3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480cef00 con 0x7f8b480c40b0 2023-06-22T08:00:18.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b4e3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b481ef450 con 0x7f8b48152ce0 2023-06-22T08:00:18.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b4e3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4815b7c0 con 0x7f8b480c9fd0 2023-06-22T08:00:18.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b47fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c40b0 0x7f8b481eef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b47fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c40b0 0x7f8b481eef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38636/0 (socket says 172.21.15.23:38636) 2023-06-22T08:00:18.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b47fff700 1 -- 172.21.15.23:0/743544918 learned_addr learned my addr 172.21.15.23:0/743544918 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:18.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b477fe700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c9fd0 0x7f8b4814f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b4c99c700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b48152ce0 0x7f8b480b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b47fff700 1 -- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c9fd0 msgr2=0x7f8b4814f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b47fff700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c9fd0 0x7f8b4814f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b47fff700 1 -- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b48152ce0 msgr2=0x7f8b480b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b47fff700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b48152ce0 0x7f8b480b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.031+0000 7f8b47fff700 1 -- 172.21.15.23:0/743544918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c009550 con 0x7f8b480c40b0 2023-06-22T08:00:18.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.032+0000 7f8b47fff700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c40b0 0x7f8b481eef10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b3800c580 tx=0x7f8b3800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:18.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.032+0000 7f8b44ff9700 1 -- 172.21.15.23:0/743544918 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b380054e0 con 0x7f8b480c40b0 2023-06-22T08:00:18.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.032+0000 7f8b44ff9700 1 -- 172.21.15.23:0/743544918 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3800ce90 con 0x7f8b480c40b0 2023-06-22T08:00:18.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.032+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/743544918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b480b4330 con 0x7f8b480c40b0 2023-06-22T08:00:18.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.032+0000 7f8b44ff9700 1 -- 172.21.15.23:0/743544918 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b380038c0 con 0x7f8b480c40b0 2023-06-22T08:00:18.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.032+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/743544918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b480b2360 con 0x7f8b480c40b0 2023-06-22T08:00:18.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.033+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/743544918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4804ea20 con 0x7f8b480c40b0 2023-06-22T08:00:18.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.034+0000 7f8b44ff9700 1 -- 172.21.15.23:0/743544918 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8b38003a20 con 0x7f8b480c40b0 2023-06-22T08:00:18.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.035+0000 7f8b44ff9700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b3406ad10 0x7f8b3406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.035+0000 7f8b44ff9700 1 -- 172.21.15.23:0/743544918 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f8b38088af0 con 0x7f8b480c40b0 2023-06-22T08:00:18.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.035+0000 7f8b477fe700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b3406ad10 0x7f8b3406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.036+0000 7f8b477fe700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b3406ad10 0x7f8b3406d1c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c009c40 tx=0x7f8b3c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:18.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.039+0000 7f8b44ff9700 1 -- 172.21.15.23:0/743544918 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8b38053810 con 0x7f8b480c40b0 2023-06-22T08:00:18.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.250+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/743544918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b480b13d0 con 0x7f8b480c40b0 2023-06-22T08:00:18.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.251+0000 7f8b44ff9700 1 -- 172.21.15.23:0/743544918 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f8b38017070 con 0x7f8b480c40b0 2023-06-22T08:00:18.255 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:18.256 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:18.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b3406ad10 msgr2=0x7f8b3406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b3406ad10 0x7f8b3406d1c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c009c40 tx=0x7f8b3c008030 comp rx=0 tx=0).stop 2023-06-22T08:00:18.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/743544918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c40b0 msgr2=0x7f8b481eef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c40b0 0x7f8b481eef10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b3800c580 tx=0x7f8b3800c890 comp rx=0 tx=0).stop 2023-06-22T08:00:18.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/743544918 shutdown_connections 2023-06-22T08:00:18.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b3406ad10 0x7f8b3406d1c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b48152ce0 0x7f8b480b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b480c40b0 0x7f8b481eef10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 --2- 172.21.15.23:0/743544918 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b480c9fd0 0x7f8b4814f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/743544918 >> 172.21.15.23:0/743544918 conn(0x7f8b4806ae30 msgr2=0x7f8b480ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:18.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/743544918 shutdown_connections 2023-06-22T08:00:18.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.254+0000 7f8b4e3ff700 1 -- 172.21.15.23:0/743544918 wait complete. 2023-06-22T08:00:18.276 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.473+0000 7f9f88bff700 1 Processor -- start 2023-06-22T08:00:18.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.474+0000 7f9f88bff700 1 -- start start 2023-06-22T08:00:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.475+0000 7f9f88bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 0x7f9f840cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.475+0000 7f9f88bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f840c40b0 0x7f9f840c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.475+0000 7f9f88bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8415c370 con 0x7f9f840c9fd0 2023-06-22T08:00:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.475+0000 7f9f88bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f841530b0 con 0x7f9f840c40b0 2023-06-22T08:00:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.475+0000 7f9f88bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f840cf6f0 con 0x7f9f84152ce0 2023-06-22T08:00:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.475+0000 7f9f82d9d700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f840c40b0 0x7f9f840c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.475+0000 7f9f8259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 0x7f9f840cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.475+0000 7f9f81d9b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9f840c9fd0 0x7f9f8415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39194/0 (socket says 172.21.15.23:39194) 2023-06-22T08:00:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.475+0000 7f9f81d9b700 1 -- 172.21.15.23:0/1231946998 learned_addr learned my addr 172.21.15.23:0/1231946998 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:18.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.476+0000 7f9f8159a700 1 -- 172.21.15.23:0/1231946998 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130860008 0 0) 0x7f9f8415c370 con 0x7f9f840c9fd0 2023-06-22T08:00:18.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.476+0000 7f9f8159a700 1 -- 172.21.15.23:0/1231946998 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f6c0035b0 con 0x7f9f840c9fd0 2023-06-22T08:00:18.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.476+0000 7f9f8259c700 1 -- 172.21.15.23:0/1231946998 >> v2:172.21.15.67:3300/0 conn(0x7f9f840c40b0 msgr2=0x7f9f840c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:18.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.476+0000 7f9f8259c700 1 --2- 172.21.15.23:0/1231946998 >> v2:172.21.15.67:3300/0 conn(0x7f9f840c40b0 0x7f9f840c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.476+0000 7f9f8259c700 1 -- 172.21.15.23:0/1231946998 >> v1:172.21.15.67:6789/0 conn(0x7f9f840c9fd0 legacy=0x7f9f8415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.476+0000 7f9f8259c700 1 -- 172.21.15.23:0/1231946998 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8415bda0 con 0x7f9f84152ce0 2023-06-22T08:00:18.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.476+0000 7f9f8259c700 1 --2- 172.21.15.23:0/1231946998 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 0x7f9f840cf1b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009840 tx=0x7f9f74009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a66852307c394b67 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:18.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.476+0000 7f9f8159a700 1 -- 172.21.15.23:0/1231946998 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f74011830 con 0x7f9f84152ce0 2023-06-22T08:00:18.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.476+0000 7f9f8159a700 1 -- 172.21.15.23:0/1231946998 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f74011990 con 0x7f9f84152ce0 2023-06-22T08:00:18.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.476+0000 7f9f8159a700 1 -- 172.21.15.23:0/1231946998 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f74011d20 con 0x7f9f84152ce0 2023-06-22T08:00:18.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.477+0000 7f9f88bff700 1 -- 172.21.15.23:0/1231946998 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 msgr2=0x7f9f840cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.477+0000 7f9f88bff700 1 --2- 172.21.15.23:0/1231946998 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 0x7f9f840cf1b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009840 tx=0x7f9f74009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:18.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.477+0000 7f9f88bff700 1 -- 172.21.15.23:0/1231946998 shutdown_connections 2023-06-22T08:00:18.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.477+0000 7f9f88bff700 1 --2- 172.21.15.23:0/1231946998 >> v2:172.21.15.67:3300/0 conn(0x7f9f840c40b0 0x7f9f840c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.477+0000 7f9f88bff700 1 --2- 172.21.15.23:0/1231946998 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 0x7f9f840cf1b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.477+0000 7f9f88bff700 1 -- 172.21.15.23:0/1231946998 >> 172.21.15.23:0/1231946998 conn(0x7f9f8406ae30 msgr2=0x7f9f8406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:18.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.477+0000 7f9f88bff700 1 -- 172.21.15.23:0/1231946998 shutdown_connections 2023-06-22T08:00:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.477+0000 7f9f88bff700 1 -- 172.21.15.23:0/1231946998 wait complete. 2023-06-22T08:00:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.479+0000 7f9f88bff700 1 Processor -- start 2023-06-22T08:00:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.479+0000 7f9f88bff700 1 -- start start 2023-06-22T08:00:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f88bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f840c40b0 0x7f9f840ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f88bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f840c9fd0 0x7f9f841eec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f88bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 0x7f9f841e1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f88bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f840cef60 con 0x7f9f840c9fd0 2023-06-22T08:00:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f88bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f840cd100 con 0x7f9f840c40b0 2023-06-22T08:00:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f88bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f840b3010 con 0x7f9f84152ce0 2023-06-22T08:00:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f81d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f840c9fd0 0x7f9f841eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f8259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f840c40b0 0x7f9f840ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f82d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 0x7f9f841e1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f8259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f840c40b0 0x7f9f840ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:48096/0 (socket says 172.21.15.23:48096) 2023-06-22T08:00:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f8259c700 1 -- 172.21.15.23:0/3806456536 learned_addr learned my addr 172.21.15.23:0/3806456536 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f8259c700 1 -- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 msgr2=0x7f9f841e1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f8259c700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 0x7f9f841e1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f8259c700 1 -- 172.21.15.23:0/3806456536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f840c9fd0 msgr2=0x7f9f841eec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.480+0000 7f9f8259c700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f840c9fd0 0x7f9f841eec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.481+0000 7f9f8259c700 1 -- 172.21.15.23:0/3806456536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f740094a0 con 0x7f9f840c40b0 2023-06-22T08:00:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.481+0000 7f9f82d9d700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 0x7f9f841e1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.481+0000 7f9f8259c700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f840c40b0 0x7f9f840ccbc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009b90 tx=0x7f9f74004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.481+0000 7f9f72ffd700 1 -- 172.21.15.23:0/3806456536 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f74004510 con 0x7f9f840c40b0 2023-06-22T08:00:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.481+0000 7f9f72ffd700 1 -- 172.21.15.23:0/3806456536 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f740036a0 con 0x7f9f840c40b0 2023-06-22T08:00:18.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.481+0000 7f9f72ffd700 1 -- 172.21.15.23:0/3806456536 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f74003a50 con 0x7f9f840c40b0 2023-06-22T08:00:18.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.481+0000 7f9f88bff700 1 -- 172.21.15.23:0/3806456536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f841ef140 con 0x7f9f840c40b0 2023-06-22T08:00:18.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.481+0000 7f9f88bff700 1 -- 172.21.15.23:0/3806456536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f840b2400 con 0x7f9f840c40b0 2023-06-22T08:00:18.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.482+0000 7f9f88bff700 1 -- 172.21.15.23:0/3806456536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8404ea20 con 0x7f9f840c40b0 2023-06-22T08:00:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.483+0000 7f9f72ffd700 1 -- 172.21.15.23:0/3806456536 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9f74003bb0 con 0x7f9f840c40b0 2023-06-22T08:00:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.484+0000 7f9f72ffd700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6406ace0 0x7f9f6406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.484+0000 7f9f72ffd700 1 -- 172.21.15.23:0/3806456536 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9f7408ac50 con 0x7f9f840c40b0 2023-06-22T08:00:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.484+0000 7f9f81d9b700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6406ace0 0x7f9f6406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.485+0000 7f9f81d9b700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6406ace0 0x7f9f6406d190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9f8414fab0 tx=0x7f9f78007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:18.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.489+0000 7f9f72ffd700 1 -- 172.21.15.23:0/3806456536 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9f74055aa0 con 0x7f9f840c40b0 2023-06-22T08:00:18.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.700+0000 7f9f88bff700 1 -- 172.21.15.23:0/3806456536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f841ef140 con 0x7f9f840c40b0 2023-06-22T08:00:18.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.703+0000 7f9f72ffd700 1 -- 172.21.15.23:0/3806456536 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f9f841ef140 con 0x7f9f840c40b0 2023-06-22T08:00:18.706 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:18.707 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.704+0000 7f9f88bff700 1 -- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6406ace0 msgr2=0x7f9f6406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.704+0000 7f9f88bff700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6406ace0 0x7f9f6406d190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9f8414fab0 tx=0x7f9f78007dc0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.704+0000 7f9f88bff700 1 -- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f840c40b0 msgr2=0x7f9f840ccbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.704+0000 7f9f88bff700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f840c40b0 0x7f9f840ccbc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009b90 tx=0x7f9f74004750 comp rx=0 tx=0).stop 2023-06-22T08:00:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.704+0000 7f9f88bff700 1 -- 172.21.15.23:0/3806456536 shutdown_connections 2023-06-22T08:00:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.704+0000 7f9f88bff700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f84152ce0 0x7f9f841e1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.704+0000 7f9f88bff700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6406ace0 0x7f9f6406d190 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.704+0000 7f9f88bff700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f840c40b0 0x7f9f840ccbc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.704+0000 7f9f88bff700 1 --2- 172.21.15.23:0/3806456536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f840c9fd0 0x7f9f841eec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.704+0000 7f9f88bff700 1 -- 172.21.15.23:0/3806456536 >> 172.21.15.23:0/3806456536 conn(0x7f9f8406ae30 msgr2=0x7f9f8406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.705+0000 7f9f88bff700 1 -- 172.21.15.23:0/3806456536 shutdown_connections 2023-06-22T08:00:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.705+0000 7f9f88bff700 1 -- 172.21.15.23:0/3806456536 wait complete. 2023-06-22T08:00:18.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-22T08:00:18.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.925+0000 7f0835dff700 1 Processor -- start 2023-06-22T08:00:18.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.925+0000 7f0835dff700 1 -- start start 2023-06-22T08:00:18.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f0835dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0830152ba0 0x7f08301593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f0835dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08300c9ee0 0x7f08300ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f0835dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0830159930 con 0x7f08300c9260 2023-06-22T08:00:18.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f0835dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08300c9aa0 con 0x7f0830152ba0 2023-06-22T08:00:18.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f0835dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083006aed0 con 0x7f08300c9ee0 2023-06-22T08:00:18.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f082f7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0830152ba0 0x7f08301593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f082f7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0830152ba0 0x7f08301593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38654/0 (socket says 172.21.15.23:38654) 2023-06-22T08:00:18.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f082f7fe700 1 -- 172.21.15.23:0/1842085634 learned_addr learned my addr 172.21.15.23:0/1842085634 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f082effd700 1 --1- 172.21.15.23:0/1842085634 >> v1:172.21.15.23:6789/0 conn(0x7f08300c9260 0x7f08300c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40028/0 (socket says 172.21.15.23:40028) 2023-06-22T08:00:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f082ffff700 1 --2- 172.21.15.23:0/1842085634 >> v2:172.21.15.67:3300/0 conn(0x7f08300c9ee0 0x7f08300ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.927+0000 7f082f7fe700 1 -- 172.21.15.23:0/1842085634 >> v2:172.21.15.67:3300/0 conn(0x7f08300c9ee0 msgr2=0x7f08300ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.928+0000 7f082f7fe700 1 --2- 172.21.15.23:0/1842085634 >> v2:172.21.15.67:3300/0 conn(0x7f08300c9ee0 0x7f08300ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.928+0000 7f082f7fe700 1 -- 172.21.15.23:0/1842085634 >> v1:172.21.15.23:6789/0 conn(0x7f08300c9260 legacy=0x7f08300c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.928+0000 7f082f7fe700 1 -- 172.21.15.23:0/1842085634 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083006b1b0 con 0x7f0830152ba0 2023-06-22T08:00:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.928+0000 7f082f7fe700 1 --2- 172.21.15.23:0/1842085634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0830152ba0 0x7f08301593f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f083008b390 tx=0x7f0820009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d298f59525e5a494 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.928+0000 7f082e7fc700 1 -- 172.21.15.23:0/1842085634 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0820011830 con 0x7f0830152ba0 2023-06-22T08:00:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.928+0000 7f082e7fc700 1 -- 172.21.15.23:0/1842085634 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0820011990 con 0x7f0830152ba0 2023-06-22T08:00:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.928+0000 7f082e7fc700 1 -- 172.21.15.23:0/1842085634 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0820011d20 con 0x7f0830152ba0 2023-06-22T08:00:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.929+0000 7f0835dff700 1 -- 172.21.15.23:0/1842085634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0830152ba0 msgr2=0x7f08301593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.929+0000 7f0835dff700 1 --2- 172.21.15.23:0/1842085634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0830152ba0 0x7f08301593f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f083008b390 tx=0x7f0820009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.929+0000 7f0835dff700 1 -- 172.21.15.23:0/1842085634 shutdown_connections 2023-06-22T08:00:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.929+0000 7f0835dff700 1 --2- 172.21.15.23:0/1842085634 >> v2:172.21.15.67:3300/0 conn(0x7f08300c9ee0 0x7f08300ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.929+0000 7f0835dff700 1 --2- 172.21.15.23:0/1842085634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0830152ba0 0x7f08301593f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.929+0000 7f0835dff700 1 -- 172.21.15.23:0/1842085634 >> 172.21.15.23:0/1842085634 conn(0x7f0830065b90 msgr2=0x7f0830065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:18.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.929+0000 7f0835dff700 1 -- 172.21.15.23:0/1842085634 shutdown_connections 2023-06-22T08:00:18.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.929+0000 7f0835dff700 1 -- 172.21.15.23:0/1842085634 wait complete. 2023-06-22T08:00:18.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.931+0000 7f0835dff700 1 Processor -- start 2023-06-22T08:00:18.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.931+0000 7f0835dff700 1 -- start start 2023-06-22T08:00:18.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.932+0000 7f0835dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08300c9260 0x7f08301eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f0835dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08300c9ee0 0x7f08300c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f0835dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0830152ba0 0x7f08300c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f0835dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08301537e0 con 0x7f08300c9260 2023-06-22T08:00:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f0835dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08301519a0 con 0x7f0830152ba0 2023-06-22T08:00:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f0835dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0830151c60 con 0x7f08300c9ee0 2023-06-22T08:00:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08300c9260 0x7f08301eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08300c9260 0x7f08301eeed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38666/0 (socket says 172.21.15.23:38666) 2023-06-22T08:00:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082f7fe700 1 -- 172.21.15.23:0/3699246939 learned_addr learned my addr 172.21.15.23:0/3699246939 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:18.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082effd700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08300c9ee0 0x7f08300c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082ffff700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0830152ba0 0x7f08300c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082f7fe700 1 -- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08300c9ee0 msgr2=0x7f08300c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082f7fe700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08300c9ee0 0x7f08300c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082f7fe700 1 -- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0830152ba0 msgr2=0x7f08300c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:18.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082f7fe700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0830152ba0 0x7f08300c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082f7fe700 1 -- 172.21.15.23:0/3699246939 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0820009550 con 0x7f08300c9260 2023-06-22T08:00:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082effd700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08300c9ee0 0x7f08300c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.933+0000 7f082ffff700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0830152ba0 0x7f08300c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.934+0000 7f082f7fe700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08300c9260 0x7f08301eeed0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0820009c10 tx=0x7f0820004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.934+0000 7f0817fff700 1 -- 172.21.15.23:0/3699246939 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08200036a0 con 0x7f08300c9260 2023-06-22T08:00:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.934+0000 7f0817fff700 1 -- 172.21.15.23:0/3699246939 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082001e070 con 0x7f08300c9260 2023-06-22T08:00:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.934+0000 7f0835dff700 1 -- 172.21.15.23:0/3699246939 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08300c73d0 con 0x7f08300c9260 2023-06-22T08:00:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.934+0000 7f0817fff700 1 -- 172.21.15.23:0/3699246939 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0820003a60 con 0x7f08300c9260 2023-06-22T08:00:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.934+0000 7f0835dff700 1 -- 172.21.15.23:0/3699246939 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0830151f40 con 0x7f08300c9260 2023-06-22T08:00:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.935+0000 7f0835dff700 1 -- 172.21.15.23:0/3699246939 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08300cbed0 con 0x7f08300c9260 2023-06-22T08:00:18.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.936+0000 7f0817fff700 1 -- 172.21.15.23:0/3699246939 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0820003bc0 con 0x7f08300c9260 2023-06-22T08:00:18.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.937+0000 7f0817fff700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f081c06ace0 0x7f081c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:18.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.937+0000 7f0817fff700 1 -- 172.21.15.23:0/3699246939 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f082008aa20 con 0x7f08300c9260 2023-06-22T08:00:18.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.938+0000 7f082effd700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f081c06ace0 0x7f081c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:18.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.938+0000 7f082effd700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f081c06ace0 0x7f081c06d190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f08301f2b00 tx=0x7f0824007dd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:18.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:18.941+0000 7f0817fff700 1 -- 172.21.15.23:0/3699246939 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0820055950 con 0x7f08300c9260 2023-06-22T08:00:19.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.201+0000 7f0835dff700 1 -- 172.21.15.23:0/3699246939 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f083004ea20 con 0x7f08300c9260 2023-06-22T08:00:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.672+0000 7f0817fff700 1 -- 172.21.15.23:0/3699246939 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0820058f70 con 0x7f08300c9260 2023-06-22T08:00:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 -- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f081c06ace0 msgr2=0x7f081c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f081c06ace0 0x7f081c06d190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f08301f2b00 tx=0x7f0824007dd0 comp rx=0 tx=0).stop 2023-06-22T08:00:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 -- 172.21.15.23:0/3699246939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08300c9260 msgr2=0x7f08301eeed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08300c9260 0x7f08301eeed0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0820009c10 tx=0x7f0820004750 comp rx=0 tx=0).stop 2023-06-22T08:00:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 -- 172.21.15.23:0/3699246939 shutdown_connections 2023-06-22T08:00:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f081c06ace0 0x7f081c06d190 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0830152ba0 0x7f08300c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08300c9260 0x7f08301eeed0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 --2- 172.21.15.23:0/3699246939 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08300c9ee0 0x7f08300c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 -- 172.21.15.23:0/3699246939 >> 172.21.15.23:0/3699246939 conn(0x7f0830065b90 msgr2=0x7f083014f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 -- 172.21.15.23:0/3699246939 shutdown_connections 2023-06-22T08:00:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.675+0000 7f0835dff700 1 -- 172.21.15.23:0/3699246939 wait complete. 2023-06-22T08:00:19.698 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300} 2023-06-22T08:00:19.699 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-06-22T08:00:19.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.896+0000 7f86021ff700 1 Processor -- start 2023-06-22T08:00:19.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.896+0000 7f86021ff700 1 -- start start 2023-06-22T08:00:19.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.898+0000 7f86021ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85fc1593f0 0x7f85fc0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:19.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.898+0000 7f86021ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85fc0c9aa0 con 0x7f85fc152ba0 2023-06-22T08:00:19.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.898+0000 7f86021ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85fc06aed0 con 0x7f85fc0c4650 2023-06-22T08:00:19.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.898+0000 7f86021ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85fc0c4a60 con 0x7f85fc1593f0 2023-06-22T08:00:19.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.898+0000 7f85fb7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f85fc152ba0 0x7f85fc0c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40034/0 (socket says 172.21.15.23:40034) 2023-06-22T08:00:19.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.898+0000 7f85fb7fe700 1 -- 172.21.15.23:0/3445997546 learned_addr learned my addr 172.21.15.23:0/3445997546 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:19.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.898+0000 7f85faffd700 1 --2- 172.21.15.23:0/3445997546 >> v2:172.21.15.67:3300/0 conn(0x7f85fc1593f0 0x7f85fc0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.898+0000 7f85fa7fc700 1 -- 172.21.15.23:0/3445997546 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171131568 0 0) 0x7f85fc0c9aa0 con 0x7f85fc152ba0 2023-06-22T08:00:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.898+0000 7f85fa7fc700 1 -- 172.21.15.23:0/3445997546 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e40035b0 con 0x7f85fc152ba0 2023-06-22T08:00:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.899+0000 7f85fa7fc700 1 -- 172.21.15.23:0/3445997546 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961944754 0 0) 0x7f85fc06aed0 con 0x7f85fc0c4650 2023-06-22T08:00:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.899+0000 7f85fa7fc700 1 -- 172.21.15.23:0/3445997546 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85fc0c9aa0 con 0x7f85fc0c4650 2023-06-22T08:00:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.899+0000 7f85faffd700 1 -- 172.21.15.23:0/3445997546 >> v1:172.21.15.67:6789/0 conn(0x7f85fc0c4650 legacy=0x7f85fc0c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.899+0000 7f85faffd700 1 -- 172.21.15.23:0/3445997546 >> v1:172.21.15.23:6789/0 conn(0x7f85fc152ba0 legacy=0x7f85fc0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:19.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.899+0000 7f85faffd700 1 -- 172.21.15.23:0/3445997546 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85fc06b1b0 con 0x7f85fc1593f0 2023-06-22T08:00:19.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.899+0000 7f85faffd700 1 --2- 172.21.15.23:0/3445997546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85fc1593f0 0x7f85fc0c4110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009840 tx=0x7f85f0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38ecd6e1a66f119 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:19.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.899+0000 7f85fa7fc700 1 -- 172.21.15.23:0/3445997546 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0011830 con 0x7f85fc1593f0 2023-06-22T08:00:19.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.899+0000 7f85fa7fc700 1 -- 172.21.15.23:0/3445997546 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f0011990 con 0x7f85fc1593f0 2023-06-22T08:00:19.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.899+0000 7f85fa7fc700 1 -- 172.21.15.23:0/3445997546 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0011d20 con 0x7f85fc1593f0 2023-06-22T08:00:19.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.900+0000 7f86021ff700 1 -- 172.21.15.23:0/3445997546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85fc1593f0 msgr2=0x7f85fc0c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.900+0000 7f86021ff700 1 --2- 172.21.15.23:0/3445997546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85fc1593f0 0x7f85fc0c4110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009840 tx=0x7f85f0009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.900+0000 7f86021ff700 1 -- 172.21.15.23:0/3445997546 shutdown_connections 2023-06-22T08:00:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.900+0000 7f86021ff700 1 --2- 172.21.15.23:0/3445997546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85fc1593f0 0x7f85fc0c4110 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.900+0000 7f86021ff700 1 -- 172.21.15.23:0/3445997546 >> 172.21.15.23:0/3445997546 conn(0x7f85fc065b90 msgr2=0x7f85fc065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.900+0000 7f86021ff700 1 -- 172.21.15.23:0/3445997546 shutdown_connections 2023-06-22T08:00:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.900+0000 7f86021ff700 1 -- 172.21.15.23:0/3445997546 wait complete. 2023-06-22T08:00:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.902+0000 7f86021ff700 1 Processor -- start 2023-06-22T08:00:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.902+0000 7f86021ff700 1 -- start start 2023-06-22T08:00:19.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f86021ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85fc0c4650 0x7f85fc1eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:19.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f86021ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85fc152ba0 0x7f85fc0b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:19.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f86021ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85fc1593f0 0x7f85fc1f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:19.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f86021ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc1ef080 con 0x7f85fc152ba0 2023-06-22T08:00:19.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f86021ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc1ee6b0 con 0x7f85fc0c4650 2023-06-22T08:00:19.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f86021ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc1ee0c0 con 0x7f85fc1593f0 2023-06-22T08:00:19.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85faffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85fc152ba0 0x7f85fc0b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:19.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85faffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85fc152ba0 0x7f85fc0b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38678/0 (socket says 172.21.15.23:38678) 2023-06-22T08:00:19.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85faffd700 1 -- 172.21.15.23:0/2112654163 learned_addr learned my addr 172.21.15.23:0/2112654163 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:19.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85fb7fe700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85fc0c4650 0x7f85fc1eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:19.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85fbfff700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85fc1593f0 0x7f85fc1f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85faffd700 1 -- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85fc1593f0 msgr2=0x7f85fc1f1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85faffd700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85fc1593f0 0x7f85fc1f1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85faffd700 1 -- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85fc0c4650 msgr2=0x7f85fc1eeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85faffd700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85fc0c4650 0x7f85fc1eeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85faffd700 1 -- 172.21.15.23:0/2112654163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f00094a0 con 0x7f85fc152ba0 2023-06-22T08:00:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.903+0000 7f85fbfff700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85fc1593f0 0x7f85fc1f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.904+0000 7f85faffd700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85fc152ba0 0x7f85fc0b24c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f85f00090b0 tx=0x7f85f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.904+0000 7f85e3fff700 1 -- 172.21.15.23:0/2112654163 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f001a760 con 0x7f85fc152ba0 2023-06-22T08:00:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.904+0000 7f85e3fff700 1 -- 172.21.15.23:0/2112654163 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f001a8c0 con 0x7f85fc152ba0 2023-06-22T08:00:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.904+0000 7f85e3fff700 1 -- 172.21.15.23:0/2112654163 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f001ac30 con 0x7f85fc152ba0 2023-06-22T08:00:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.904+0000 7f86021ff700 1 -- 172.21.15.23:0/2112654163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85fc155fd0 con 0x7f85fc152ba0 2023-06-22T08:00:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.904+0000 7f86021ff700 1 -- 172.21.15.23:0/2112654163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85fc156a80 con 0x7f85fc152ba0 2023-06-22T08:00:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.905+0000 7f86021ff700 1 -- 172.21.15.23:0/2112654163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85fc0cbed0 con 0x7f85fc152ba0 2023-06-22T08:00:19.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.906+0000 7f85e3fff700 1 -- 172.21.15.23:0/2112654163 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f85f0022070 con 0x7f85fc152ba0 2023-06-22T08:00:19.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.907+0000 7f85e3fff700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85e806ace0 0x7f85e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:19.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.908+0000 7f85e3fff700 1 -- 172.21.15.23:0/2112654163 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f85f008a3f0 con 0x7f85fc152ba0 2023-06-22T08:00:19.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.908+0000 7f85fb7fe700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85e806ace0 0x7f85e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:19.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.909+0000 7f85fb7fe700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85e806ace0 0x7f85e806d190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f85ec007fd0 tx=0x7f85ec007e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:19.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:19.913+0000 7f85e3fff700 1 -- 172.21.15.23:0/2112654163 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f85f0055250 con 0x7f85fc152ba0 2023-06-22T08:00:20.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.167+0000 7f86021ff700 1 -- 172.21.15.23:0/2112654163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f85fc04ea20 con 0x7f85fc152ba0 2023-06-22T08:00:20.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.681+0000 7f85e3fff700 1 -- 172.21.15.23:0/2112654163 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v9) v1 ==== 105+0+0 (secure 0 0 0) 0x7f85f00150b0 con 0x7f85fc152ba0 2023-06-22T08:00:20.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 -- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85e806ace0 msgr2=0x7f85e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:20.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85e806ace0 0x7f85e806d190 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f85ec007fd0 tx=0x7f85ec007e00 comp rx=0 tx=0).stop 2023-06-22T08:00:20.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 -- 172.21.15.23:0/2112654163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85fc152ba0 msgr2=0x7f85fc0b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:20.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85fc152ba0 0x7f85fc0b24c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f85f00090b0 tx=0x7f85f0004750 comp rx=0 tx=0).stop 2023-06-22T08:00:20.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 -- 172.21.15.23:0/2112654163 shutdown_connections 2023-06-22T08:00:20.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85fc1593f0 0x7f85fc1f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:20.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85e806ace0 0x7f85e806d190 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:20.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85fc0c4650 0x7f85fc1eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:20.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 --2- 172.21.15.23:0/2112654163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85fc152ba0 0x7f85fc0b24c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:20.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 -- 172.21.15.23:0/2112654163 >> 172.21.15.23:0/2112654163 conn(0x7f85fc065b90 msgr2=0x7f85fc151540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:20.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 -- 172.21.15.23:0/2112654163 shutdown_connections 2023-06-22T08:00:20.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.684+0000 7f86021ff700 1 -- 172.21.15.23:0/2112654163 wait complete. 2023-06-22T08:00:20.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-22T08:00:20.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.910+0000 7f112e9ff700 1 Processor -- start 2023-06-22T08:00:20.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.911+0000 7f112e9ff700 1 -- start start 2023-06-22T08:00:20.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.912+0000 7f112e9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1128152ba0 0x7f11281593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:20.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.912+0000 7f112e9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1128159930 con 0x7f11280c9ee0 2023-06-22T08:00:20.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.912+0000 7f112e9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11280c9aa0 con 0x7f11280c9260 2023-06-22T08:00:20.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.912+0000 7f112e9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f112806aed0 con 0x7f1128152ba0 2023-06-22T08:00:20.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.912+0000 7f112cf9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f11280c9ee0 0x7f11280ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40036/0 (socket says 172.21.15.23:40036) 2023-06-22T08:00:20.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.912+0000 7f112cf9c700 1 -- 172.21.15.23:0/733260296 learned_addr learned my addr 172.21.15.23:0/733260296 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:20.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.912+0000 7f1127fff700 1 --2- 172.21.15.23:0/733260296 >> v2:172.21.15.67:3300/0 conn(0x7f1128152ba0 0x7f11281593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:20.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.913+0000 7f1126ffd700 1 -- 172.21.15.23:0/733260296 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966813054 0 0) 0x7f1120001490 con 0x7f11280c9ee0 2023-06-22T08:00:20.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.913+0000 7f1126ffd700 1 -- 172.21.15.23:0/733260296 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11100035b0 con 0x7f11280c9ee0 2023-06-22T08:00:20.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.913+0000 7f1126ffd700 1 -- 172.21.15.23:0/733260296 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834481999 0 0) 0x7f11280c9aa0 con 0x7f11280c9260 2023-06-22T08:00:20.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.913+0000 7f1126ffd700 1 -- 172.21.15.23:0/733260296 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1120001490 con 0x7f11280c9260 2023-06-22T08:00:20.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.913+0000 7f1127fff700 1 -- 172.21.15.23:0/733260296 >> v1:172.21.15.67:6789/0 conn(0x7f11280c9260 legacy=0x7f11280c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:20.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.913+0000 7f1127fff700 1 -- 172.21.15.23:0/733260296 >> v1:172.21.15.23:6789/0 conn(0x7f11280c9ee0 legacy=0x7f11280ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:20.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.913+0000 7f1127fff700 1 -- 172.21.15.23:0/733260296 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112806b1b0 con 0x7f1128152ba0 2023-06-22T08:00:20.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.913+0000 7f1127fff700 1 --2- 172.21.15.23:0/733260296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128152ba0 0x7f11281593f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1118009840 tx=0x7f1118009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f2663d68630f74a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:20.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.913+0000 7f1126ffd700 1 -- 172.21.15.23:0/733260296 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118011830 con 0x7f1128152ba0 2023-06-22T08:00:20.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.914+0000 7f1126ffd700 1 -- 172.21.15.23:0/733260296 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1118011990 con 0x7f1128152ba0 2023-06-22T08:00:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.914+0000 7f1126ffd700 1 -- 172.21.15.23:0/733260296 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118011d20 con 0x7f1128152ba0 2023-06-22T08:00:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.914+0000 7f112e9ff700 1 -- 172.21.15.23:0/733260296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128152ba0 msgr2=0x7f11281593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.914+0000 7f112e9ff700 1 --2- 172.21.15.23:0/733260296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128152ba0 0x7f11281593f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1118009840 tx=0x7f1118009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.914+0000 7f112e9ff700 1 -- 172.21.15.23:0/733260296 shutdown_connections 2023-06-22T08:00:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.914+0000 7f112e9ff700 1 --2- 172.21.15.23:0/733260296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128152ba0 0x7f11281593f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.914+0000 7f112e9ff700 1 -- 172.21.15.23:0/733260296 >> 172.21.15.23:0/733260296 conn(0x7f1128065b90 msgr2=0x7f1128065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.914+0000 7f112e9ff700 1 -- 172.21.15.23:0/733260296 shutdown_connections 2023-06-22T08:00:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.914+0000 7f112e9ff700 1 -- 172.21.15.23:0/733260296 wait complete. 2023-06-22T08:00:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.916+0000 7f112e9ff700 1 Processor -- start 2023-06-22T08:00:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.916+0000 7f112e9ff700 1 -- start start 2023-06-22T08:00:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f112e9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c9260 0x7f1128155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f112e9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9ee0 0x7f11280b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f112e9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128152ba0 0x7f11281eeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f112e9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11280b15c0 con 0x7f11280c9260 2023-06-22T08:00:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f112e9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112814d860 con 0x7f1128152ba0 2023-06-22T08:00:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f112e9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f112814db40 con 0x7f11280c9ee0 2023-06-22T08:00:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f1127fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c9260 0x7f1128155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f1127fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c9260 0x7f1128155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38688/0 (socket says 172.21.15.23:38688) 2023-06-22T08:00:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f1127fff700 1 -- 172.21.15.23:0/1456581256 learned_addr learned my addr 172.21.15.23:0/1456581256 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f112cf9c700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128152ba0 0x7f11281eeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.917+0000 7f11277fe700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9ee0 0x7f11280b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f1127fff700 1 -- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9ee0 msgr2=0x7f11280b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f1127fff700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9ee0 0x7f11280b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f1127fff700 1 -- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128152ba0 msgr2=0x7f11281eeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f1127fff700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128152ba0 0x7f11281eeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f1127fff700 1 -- 172.21.15.23:0/1456581256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11180094a0 con 0x7f11280c9260 2023-06-22T08:00:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f1127fff700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c9260 0x7f1128155f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1118014040 tx=0x7f1118004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f1124ff9700 1 -- 172.21.15.23:0/1456581256 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111801a6e0 con 0x7f11280c9260 2023-06-22T08:00:20.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f1124ff9700 1 -- 172.21.15.23:0/1456581256 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111801a840 con 0x7f11280c9260 2023-06-22T08:00:20.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f112e9ff700 1 -- 172.21.15.23:0/1456581256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112814de20 con 0x7f11280c9260 2023-06-22T08:00:20.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f1124ff9700 1 -- 172.21.15.23:0/1456581256 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111801abb0 con 0x7f11280c9260 2023-06-22T08:00:20.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.918+0000 7f112e9ff700 1 -- 172.21.15.23:0/1456581256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11281ee680 con 0x7f11280c9260 2023-06-22T08:00:20.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.919+0000 7f112e9ff700 1 -- 172.21.15.23:0/1456581256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11280cbed0 con 0x7f11280c9260 2023-06-22T08:00:20.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.920+0000 7f1124ff9700 1 -- 172.21.15.23:0/1456581256 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1118022070 con 0x7f11280c9260 2023-06-22T08:00:20.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.921+0000 7f1124ff9700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 0x7f111406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:20.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.921+0000 7f1124ff9700 1 -- 172.21.15.23:0/1456581256 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f111808a790 con 0x7f11280c9260 2023-06-22T08:00:20.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.922+0000 7f11277fe700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 0x7f111406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:20.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.922+0000 7f11277fe700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 0x7f111406d190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1128156ac0 tx=0x7f111c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:20.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:20.926+0000 7f1124ff9700 1 -- 172.21.15.23:0/1456581256 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f11180554b0 con 0x7f11280c9260 2023-06-22T08:00:21.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.180+0000 7f112e9ff700 1 -- 172.21.15.23:0/1456581256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f11280c8f30 con 0x7f11280c9260 2023-06-22T08:00:21.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.690+0000 7f1124ff9700 1 -- 172.21.15.23:0/1456581256 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v11) v1 ==== 122+0+0 (secure 0 0 0) 0x7f11180150c0 con 0x7f11280c9260 2023-06-22T08:00:21.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 -- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 msgr2=0x7f111406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:21.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 0x7f111406d190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1128156ac0 tx=0x7f111c009040 comp rx=0 tx=0).stop 2023-06-22T08:00:21.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 -- 172.21.15.23:0/1456581256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c9260 msgr2=0x7f1128155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:21.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c9260 0x7f1128155f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1118014040 tx=0x7f1118004750 comp rx=0 tx=0).stop 2023-06-22T08:00:21.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 -- 172.21.15.23:0/1456581256 shutdown_connections 2023-06-22T08:00:21.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 0x7f111406d190 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:21.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128152ba0 0x7f11281eeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:21.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c9260 0x7f1128155f90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:21.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 --2- 172.21.15.23:0/1456581256 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9ee0 0x7f11280b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:21.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 -- 172.21.15.23:0/1456581256 >> 172.21.15.23:0/1456581256 conn(0x7f1128065b90 msgr2=0x7f1128150a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:21.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 -- 172.21.15.23:0/1456581256 shutdown_connections 2023-06-22T08:00:21.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.692+0000 7f112e9ff700 1 -- 172.21.15.23:0/1456581256 wait complete. 2023-06-22T08:00:21.714 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-06-22T08:00:21.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.914+0000 7f6439bff700 1 Processor -- start 2023-06-22T08:00:21.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.914+0000 7f6439bff700 1 -- start start 2023-06-22T08:00:21.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.915+0000 7f6439bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6434152ba0 0x7f64341593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:21.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.915+0000 7f6439bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f64340c9260 0x7f64340c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:21.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.915+0000 7f6439bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6434159930 con 0x7f64340c4650 2023-06-22T08:00:21.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.915+0000 7f6439bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64340c9aa0 con 0x7f6434152ba0 2023-06-22T08:00:21.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.915+0000 7f6439bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643406aed0 con 0x7f64340c9260 2023-06-22T08:00:21.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f64337fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6434152ba0 0x7f64341593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:21.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f64337fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6434152ba0 0x7f64341593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38704/0 (socket says 172.21.15.23:38704) 2023-06-22T08:00:21.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f64337fe700 1 -- 172.21.15.23:0/150437789 learned_addr learned my addr 172.21.15.23:0/150437789 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:21.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f6433fff700 1 --1- 172.21.15.23:0/150437789 >> v1:172.21.15.67:6789/0 conn(0x7f64340c4650 0x7f64340c9ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39210/0 (socket says 172.21.15.23:39210) 2023-06-22T08:00:21.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f6432ffd700 1 --2- 172.21.15.23:0/150437789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f64340c9260 0x7f64340c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:21.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f64337fe700 1 -- 172.21.15.23:0/150437789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f64340c9260 msgr2=0x7f64340c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:21.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f64337fe700 1 --2- 172.21.15.23:0/150437789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f64340c9260 0x7f64340c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:21.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f64337fe700 1 -- 172.21.15.23:0/150437789 >> v1:172.21.15.67:6789/0 conn(0x7f64340c4650 legacy=0x7f64340c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:21.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f64337fe700 1 -- 172.21.15.23:0/150437789 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643406b1b0 con 0x7f6434152ba0 2023-06-22T08:00:21.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f64327fc700 1 -- 172.21.15.23:0/150437789 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265350879 0 0) 0x7f642c001490 con 0x7f64340c4650 2023-06-22T08:00:21.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.916+0000 7f64337fe700 1 --2- 172.21.15.23:0/150437789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6434152ba0 0x7f64341593f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f64240098f0 tx=0x7f6424009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5581c3681ead275 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:21.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f64327fc700 1 -- 172.21.15.23:0/150437789 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424011830 con 0x7f6434152ba0 2023-06-22T08:00:21.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f64327fc700 1 -- 172.21.15.23:0/150437789 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6424011990 con 0x7f6434152ba0 2023-06-22T08:00:21.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f64327fc700 1 -- 172.21.15.23:0/150437789 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424011d20 con 0x7f6434152ba0 2023-06-22T08:00:21.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f6439bff700 1 -- 172.21.15.23:0/150437789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6434152ba0 msgr2=0x7f64341593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:21.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f6439bff700 1 --2- 172.21.15.23:0/150437789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6434152ba0 0x7f64341593f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f64240098f0 tx=0x7f6424009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:21.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f6439bff700 1 -- 172.21.15.23:0/150437789 shutdown_connections 2023-06-22T08:00:21.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f6439bff700 1 --2- 172.21.15.23:0/150437789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6434152ba0 0x7f64341593f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:21.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f6439bff700 1 --2- 172.21.15.23:0/150437789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f64340c9260 0x7f64340c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:21.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f6439bff700 1 -- 172.21.15.23:0/150437789 >> 172.21.15.23:0/150437789 conn(0x7f6434065b90 msgr2=0x7f6434065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:21.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f6439bff700 1 -- 172.21.15.23:0/150437789 shutdown_connections 2023-06-22T08:00:21.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.917+0000 7f6439bff700 1 -- 172.21.15.23:0/150437789 wait complete. 2023-06-22T08:00:21.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.919+0000 7f6439bff700 1 Processor -- start 2023-06-22T08:00:21.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.919+0000 7f6439bff700 1 -- start start 2023-06-22T08:00:21.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f6439bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340c4650 0x7f64340b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:21.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f6439bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64340c9260 0x7f64340ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:21.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f6439bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6434152ba0 0x7f64340cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:21.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f6439bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64341f1160 con 0x7f64340c9260 2023-06-22T08:00:21.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f6439bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64341f1ce0 con 0x7f64340c4650 2023-06-22T08:00:21.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f6439bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64340b4690 con 0x7f6434152ba0 2023-06-22T08:00:21.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f6432ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64340c9260 0x7f64340ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:21.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f64337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340c4650 0x7f64340b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:21.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f64337fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340c4650 0x7f64340b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:48162/0 (socket says 172.21.15.23:48162) 2023-06-22T08:00:21.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f64337fe700 1 -- 172.21.15.23:0/1109052441 learned_addr learned my addr 172.21.15.23:0/1109052441 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:21.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.920+0000 7f6433fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6434152ba0 0x7f64340cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:21.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.921+0000 7f64337fe700 1 -- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6434152ba0 msgr2=0x7f64340cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:21.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.921+0000 7f64337fe700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6434152ba0 0x7f64340cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:21.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.921+0000 7f64337fe700 1 -- 172.21.15.23:0/1109052441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64340c9260 msgr2=0x7f64340ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:21.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.921+0000 7f64337fe700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64340c9260 0x7f64340ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:21.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.921+0000 7f64337fe700 1 -- 172.21.15.23:0/1109052441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6424009550 con 0x7f64340c4650 2023-06-22T08:00:21.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.921+0000 7f64337fe700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340c4650 0x7f64340b3480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6424009c40 tx=0x7f6424004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:21.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.921+0000 7f641bfff700 1 -- 172.21.15.23:0/1109052441 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424004510 con 0x7f64340c4650 2023-06-22T08:00:21.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.921+0000 7f641bfff700 1 -- 172.21.15.23:0/1109052441 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642401e070 con 0x7f64340c4650 2023-06-22T08:00:21.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.921+0000 7f641bfff700 1 -- 172.21.15.23:0/1109052441 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64240038c0 con 0x7f64340c4650 2023-06-22T08:00:21.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.921+0000 7f6439bff700 1 -- 172.21.15.23:0/1109052441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64341f13b0 con 0x7f64340c4650 2023-06-22T08:00:21.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.922+0000 7f6439bff700 1 -- 172.21.15.23:0/1109052441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64341f1970 con 0x7f64340c4650 2023-06-22T08:00:21.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.923+0000 7f6439bff700 1 -- 172.21.15.23:0/1109052441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64340cbed0 con 0x7f64340c4650 2023-06-22T08:00:21.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.924+0000 7f641bfff700 1 -- 172.21.15.23:0/1109052441 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6424003a20 con 0x7f64340c4650 2023-06-22T08:00:21.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.924+0000 7f641bfff700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f642006ace0 0x7f642006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:21.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.925+0000 7f641bfff700 1 -- 172.21.15.23:0/1109052441 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f642408a660 con 0x7f64340c4650 2023-06-22T08:00:21.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.925+0000 7f6432ffd700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f642006ace0 0x7f642006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:21.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.926+0000 7f6432ffd700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f642006ace0 0x7f642006d190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6434158110 tx=0x7f6428009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:21.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:21.930+0000 7f641bfff700 1 -- 172.21.15.23:0/1109052441 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6424055390 con 0x7f64340c4650 2023-06-22T08:00:22.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.185+0000 7f6439bff700 1 -- 172.21.15.23:0/1109052441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f64341f2070 con 0x7f64340c4650 2023-06-22T08:00:22.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.698+0000 7f641bfff700 1 -- 172.21.15.23:0/1109052441 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v12) v1 ==== 115+0+0 (secure 0 0 0) 0x7f64240589b0 con 0x7f64340c4650 2023-06-22T08:00:22.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 -- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f642006ace0 msgr2=0x7f642006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:22.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f642006ace0 0x7f642006d190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6434158110 tx=0x7f6428009210 comp rx=0 tx=0).stop 2023-06-22T08:00:22.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 -- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340c4650 msgr2=0x7f64340b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:22.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340c4650 0x7f64340b3480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6424009c40 tx=0x7f6424004750 comp rx=0 tx=0).stop 2023-06-22T08:00:22.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 -- 172.21.15.23:0/1109052441 shutdown_connections 2023-06-22T08:00:22.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6434152ba0 0x7f64340cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:22.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f642006ace0 0x7f642006d190 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:22.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64340c4650 0x7f64340b3480 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:22.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 --2- 172.21.15.23:0/1109052441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64340c9260 0x7f64340ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:22.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 -- 172.21.15.23:0/1109052441 >> 172.21.15.23:0/1109052441 conn(0x7f6434065b90 msgr2=0x7f6434151a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:22.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 -- 172.21.15.23:0/1109052441 shutdown_connections 2023-06-22T08:00:22.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.701+0000 7f6439bff700 1 -- 172.21.15.23:0/1109052441 wait complete. 2023-06-22T08:00:22.723 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-06-22T08:00:22.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:00:22.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.921+0000 7f01193ff700 1 Processor -- start 2023-06-22T08:00:22.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.921+0000 7f01193ff700 1 -- start start 2023-06-22T08:00:22.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.922+0000 7f01193ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 0x7f01140cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:22.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.922+0000 7f01193ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011415c370 con 0x7f01140c40b0 2023-06-22T08:00:22.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.922+0000 7f01193ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011415bda0 con 0x7f0114152ce0 2023-06-22T08:00:22.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.922+0000 7f01193ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01141530b0 con 0x7f01140c9fd0 2023-06-22T08:00:22.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.922+0000 7f01137fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f01140c40b0 0x7f01140c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40048/0 (socket says 172.21.15.23:40048) 2023-06-22T08:00:22.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.922+0000 7f01137fe700 1 -- 172.21.15.23:0/1240505772 learned_addr learned my addr 172.21.15.23:0/1240505772 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:22.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.922+0000 7f01127fc700 1 --2- 172.21.15.23:0/1240505772 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 0x7f01140cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:22.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.923+0000 7f0111ffb700 1 -- 172.21.15.23:0/1240505772 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878684098 0 0) 0x7f011415bda0 con 0x7f0114152ce0 2023-06-22T08:00:22.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.923+0000 7f0111ffb700 1 -- 172.21.15.23:0/1240505772 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00fc0035b0 con 0x7f0114152ce0 2023-06-22T08:00:22.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.923+0000 7f01127fc700 1 -- 172.21.15.23:0/1240505772 >> v1:172.21.15.67:6789/0 conn(0x7f0114152ce0 legacy=0x7f011415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:22.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.923+0000 7f0111ffb700 1 -- 172.21.15.23:0/1240505772 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903250932 0 0) 0x7f011415c370 con 0x7f01140c40b0 2023-06-22T08:00:22.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.923+0000 7f01127fc700 1 -- 172.21.15.23:0/1240505772 >> v1:172.21.15.23:6789/0 conn(0x7f01140c40b0 legacy=0x7f01140c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:22.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.923+0000 7f01127fc700 1 -- 172.21.15.23:0/1240505772 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01140c9460 con 0x7f01140c9fd0 2023-06-22T08:00:22.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.923+0000 7f01127fc700 1 --2- 172.21.15.23:0/1240505772 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 0x7f01140cf1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0108009870 tx=0x7f0108009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e90bc20a61b537b8 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:22.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.923+0000 7f0111ffb700 1 -- 172.21.15.23:0/1240505772 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108011830 con 0x7f01140c9fd0 2023-06-22T08:00:22.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.923+0000 7f0111ffb700 1 -- 172.21.15.23:0/1240505772 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108011990 con 0x7f01140c9fd0 2023-06-22T08:00:22.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.923+0000 7f0111ffb700 1 -- 172.21.15.23:0/1240505772 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108011d20 con 0x7f01140c9fd0 2023-06-22T08:00:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.924+0000 7f01193ff700 1 -- 172.21.15.23:0/1240505772 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 msgr2=0x7f01140cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.924+0000 7f01193ff700 1 --2- 172.21.15.23:0/1240505772 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 0x7f01140cf1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0108009870 tx=0x7f0108009b80 comp rx=0 tx=0).stop 2023-06-22T08:00:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.924+0000 7f01193ff700 1 -- 172.21.15.23:0/1240505772 shutdown_connections 2023-06-22T08:00:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.924+0000 7f01193ff700 1 --2- 172.21.15.23:0/1240505772 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 0x7f01140cf1b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.924+0000 7f01193ff700 1 -- 172.21.15.23:0/1240505772 >> 172.21.15.23:0/1240505772 conn(0x7f011406ae30 msgr2=0x7f011406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.924+0000 7f01193ff700 1 -- 172.21.15.23:0/1240505772 shutdown_connections 2023-06-22T08:00:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.924+0000 7f01193ff700 1 -- 172.21.15.23:0/1240505772 wait complete. 2023-06-22T08:00:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.926+0000 7f01193ff700 1 Processor -- start 2023-06-22T08:00:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.926+0000 7f01193ff700 1 -- start start 2023-06-22T08:00:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f01193ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01140c40b0 0x7f01140c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f01193ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 0x7f01141519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f01193ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0114152ce0 0x7f01141e1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:22.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f01193ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c8290 con 0x7f0114152ce0 2023-06-22T08:00:22.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f01193ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01141e4160 con 0x7f01140c40b0 2023-06-22T08:00:22.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f01193ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c5f90 con 0x7f01140c9fd0 2023-06-22T08:00:22.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f01137fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0114152ce0 0x7f01141e1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:22.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f01137fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0114152ce0 0x7f01141e1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38726/0 (socket says 172.21.15.23:38726) 2023-06-22T08:00:22.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f01137fe700 1 -- 172.21.15.23:0/3680763870 learned_addr learned my addr 172.21.15.23:0/3680763870 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:22.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f0112ffd700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01140c40b0 0x7f01140c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:22.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.927+0000 7f01127fc700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 0x7f01141519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:22.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f01137fe700 1 -- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 msgr2=0x7f01141519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:22.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f01137fe700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 0x7f01141519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:22.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f01137fe700 1 -- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01140c40b0 msgr2=0x7f01140c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:22.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f01137fe700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01140c40b0 0x7f01140c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:22.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f01137fe700 1 -- 172.21.15.23:0/3680763870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080094a0 con 0x7f0114152ce0 2023-06-22T08:00:22.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f01137fe700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0114152ce0 0x7f01141e1510 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f010c00cba0 tx=0x7f010c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:22.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f00fb7fe700 1 -- 172.21.15.23:0/3680763870 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c00a980 con 0x7f0114152ce0 2023-06-22T08:00:22.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f00fb7fe700 1 -- 172.21.15.23:0/3680763870 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c00aae0 con 0x7f0114152ce0 2023-06-22T08:00:22.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f00fb7fe700 1 -- 172.21.15.23:0/3680763870 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c00ae90 con 0x7f0114152ce0 2023-06-22T08:00:22.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f01193ff700 1 -- 172.21.15.23:0/3680763870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01140c7610 con 0x7f0114152ce0 2023-06-22T08:00:22.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.928+0000 7f01193ff700 1 -- 172.21.15.23:0/3680763870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01140c7bd0 con 0x7f0114152ce0 2023-06-22T08:00:22.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.930+0000 7f01193ff700 1 -- 172.21.15.23:0/3680763870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01140c83d0 con 0x7f0114152ce0 2023-06-22T08:00:22.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.930+0000 7f00fb7fe700 1 -- 172.21.15.23:0/3680763870 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f010c0070d0 con 0x7f0114152ce0 2023-06-22T08:00:22.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.931+0000 7f00fb7fe700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f010006ad10 0x7f010006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:22.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.931+0000 7f00fb7fe700 1 -- 172.21.15.23:0/3680763870 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f010c089090 con 0x7f0114152ce0 2023-06-22T08:00:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.932+0000 7f0112ffd700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f010006ad10 0x7f010006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.932+0000 7f0112ffd700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f010006ad10 0x7f010006d1c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0104001220 tx=0x7f0104009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:22.936+0000 7f00fb7fe700 1 -- 172.21.15.23:0/3680763870 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f01140c83d0 con 0x7f0114152ce0 2023-06-22T08:00:23.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.188+0000 7f01193ff700 1 -- 172.21.15.23:0/3680763870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01140c83d0 con 0x7f0114152ce0 2023-06-22T08:00:23.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.189+0000 7f00fb7fe700 1 -- 172.21.15.23:0/3680763870 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6196 (secure 0 0 0) 0x7f01140c83d0 con 0x7f0114152ce0 2023-06-22T08:00:23.192 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:23.192 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4268,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:00:22.690276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4285,"mds_1":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:creating","state_seq":1,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:00:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.191+0000 7f01193ff700 1 -- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f010006ad10 msgr2=0x7f010006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.191+0000 7f01193ff700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f010006ad10 0x7f010006d1c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0104001220 tx=0x7f0104009040 comp rx=0 tx=0).stop 2023-06-22T08:00:23.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.191+0000 7f01193ff700 1 -- 172.21.15.23:0/3680763870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0114152ce0 msgr2=0x7f01141e1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.191+0000 7f01193ff700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0114152ce0 0x7f01141e1510 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f010c00cba0 tx=0x7f010c00ceb0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.191+0000 7f01193ff700 1 -- 172.21.15.23:0/3680763870 shutdown_connections 2023-06-22T08:00:23.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.191+0000 7f01193ff700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0114152ce0 0x7f01141e1510 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.191+0000 7f01193ff700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f010006ad10 0x7f010006d1c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.191+0000 7f01193ff700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01140c40b0 0x7f01140c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.191+0000 7f01193ff700 1 --2- 172.21.15.23:0/3680763870 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01140c9fd0 0x7f01141519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.191+0000 7f01193ff700 1 -- 172.21.15.23:0/3680763870 >> 172.21.15.23:0/3680763870 conn(0x7f011406ae30 msgr2=0x7f01140cb760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:23.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.192+0000 7f01193ff700 1 -- 172.21.15.23:0/3680763870 shutdown_connections 2023-06-22T08:00:23.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.192+0000 7f01193ff700 1 -- 172.21.15.23:0/3680763870 wait complete. 2023-06-22T08:00:23.196 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 12 2023-06-22T08:00:23.215 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:23.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.410+0000 7f112e7ff700 1 Processor -- start 2023-06-22T08:00:23.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.411+0000 7f112e7ff700 1 -- start start 2023-06-22T08:00:23.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.412+0000 7f112e7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9fd0 0x7f11280cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.412+0000 7f112e7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f11280c40b0 0x7f11280c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.412+0000 7f112e7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112815c370 con 0x7f1128152ce0 2023-06-22T08:00:23.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.412+0000 7f112e7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11281530b0 con 0x7f11280c40b0 2023-06-22T08:00:23.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.412+0000 7f112e7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11280cf6f0 con 0x7f11280c9fd0 2023-06-22T08:00:23.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.412+0000 7f11277fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9fd0 0x7f11280cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.412+0000 7f112cd9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f11280c40b0 0x7f11280c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.412+0000 7f1127fff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1128152ce0 0x7f112815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39226/0 (socket says 172.21.15.23:39226) 2023-06-22T08:00:23.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.412+0000 7f1127fff700 1 -- 172.21.15.23:0/2997254228 learned_addr learned my addr 172.21.15.23:0/2997254228 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:23.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.413+0000 7f11277fe700 1 -- 172.21.15.23:0/2997254228 >> v2:172.21.15.67:3300/0 conn(0x7f11280c40b0 msgr2=0x7f11280c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:23.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.413+0000 7f11277fe700 1 --2- 172.21.15.23:0/2997254228 >> v2:172.21.15.67:3300/0 conn(0x7f11280c40b0 0x7f11280c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.413+0000 7f1126ffd700 1 -- 172.21.15.23:0/2997254228 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380126899 0 0) 0x7f112815c370 con 0x7f1128152ce0 2023-06-22T08:00:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.413+0000 7f11277fe700 1 -- 172.21.15.23:0/2997254228 >> v1:172.21.15.67:6789/0 conn(0x7f1128152ce0 legacy=0x7f112815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.413+0000 7f11277fe700 1 -- 172.21.15.23:0/2997254228 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112815bda0 con 0x7f11280c9fd0 2023-06-22T08:00:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.413+0000 7f11277fe700 1 --2- 172.21.15.23:0/2997254228 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9fd0 0x7f11280cf1b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f111c009840 tx=0x7f111c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b515745052be69e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.413+0000 7f1126ffd700 1 -- 172.21.15.23:0/2997254228 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111c011830 con 0x7f11280c9fd0 2023-06-22T08:00:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.413+0000 7f1126ffd700 1 -- 172.21.15.23:0/2997254228 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111c011990 con 0x7f11280c9fd0 2023-06-22T08:00:23.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.413+0000 7f1126ffd700 1 -- 172.21.15.23:0/2997254228 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111c011d20 con 0x7f11280c9fd0 2023-06-22T08:00:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.414+0000 7f112e7ff700 1 -- 172.21.15.23:0/2997254228 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9fd0 msgr2=0x7f11280cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.414+0000 7f112e7ff700 1 --2- 172.21.15.23:0/2997254228 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9fd0 0x7f11280cf1b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f111c009840 tx=0x7f111c009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.414+0000 7f112e7ff700 1 -- 172.21.15.23:0/2997254228 shutdown_connections 2023-06-22T08:00:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.414+0000 7f112e7ff700 1 --2- 172.21.15.23:0/2997254228 >> v2:172.21.15.67:3300/0 conn(0x7f11280c40b0 0x7f11280c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.414+0000 7f112e7ff700 1 --2- 172.21.15.23:0/2997254228 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11280c9fd0 0x7f11280cf1b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.414+0000 7f112e7ff700 1 -- 172.21.15.23:0/2997254228 >> 172.21.15.23:0/2997254228 conn(0x7f112806ae30 msgr2=0x7f112806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.414+0000 7f112e7ff700 1 -- 172.21.15.23:0/2997254228 shutdown_connections 2023-06-22T08:00:23.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.414+0000 7f112e7ff700 1 -- 172.21.15.23:0/2997254228 wait complete. 2023-06-22T08:00:23.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.416+0000 7f112e7ff700 1 Processor -- start 2023-06-22T08:00:23.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.416+0000 7f112e7ff700 1 -- start start 2023-06-22T08:00:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f112e7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c40b0 0x7f11280cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f112e7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11280c9fd0 0x7f112815b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f112e7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1128152ce0 0x7f11280cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f112e7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11280cc800 con 0x7f11280c40b0 2023-06-22T08:00:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f112e7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11281ee7c0 con 0x7f11280c9fd0 2023-06-22T08:00:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f112e7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11280cd0f0 con 0x7f1128152ce0 2023-06-22T08:00:23.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f1127fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c40b0 0x7f11280cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f1127fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c40b0 0x7f11280cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38730/0 (socket says 172.21.15.23:38730) 2023-06-22T08:00:23.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f1127fff700 1 -- 172.21.15.23:0/819371566 learned_addr learned my addr 172.21.15.23:0/819371566 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:23.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f11277fe700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11280c9fd0 0x7f112815b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f112cd9c700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1128152ce0 0x7f11280cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f1127fff700 1 -- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1128152ce0 msgr2=0x7f11280cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f1127fff700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1128152ce0 0x7f11280cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.417+0000 7f1127fff700 1 -- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11280c9fd0 msgr2=0x7f112815b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.418+0000 7f1127fff700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11280c9fd0 0x7f112815b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.418+0000 7f1127fff700 1 -- 172.21.15.23:0/819371566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c0094a0 con 0x7f11280c40b0 2023-06-22T08:00:23.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.418+0000 7f1127fff700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c40b0 0x7f11280cca40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f11180011e0 tx=0x7f111800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:23.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.418+0000 7f1124ff9700 1 -- 172.21.15.23:0/819371566 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118004c80 con 0x7f11280c40b0 2023-06-22T08:00:23.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.418+0000 7f1124ff9700 1 -- 172.21.15.23:0/819371566 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1118004e00 con 0x7f11280c40b0 2023-06-22T08:00:23.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.418+0000 7f1124ff9700 1 -- 172.21.15.23:0/819371566 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118007310 con 0x7f11280c40b0 2023-06-22T08:00:23.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.418+0000 7f112e7ff700 1 -- 172.21.15.23:0/819371566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11281eea80 con 0x7f11280c40b0 2023-06-22T08:00:23.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.418+0000 7f112e7ff700 1 -- 172.21.15.23:0/819371566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11281ef040 con 0x7f11280c40b0 2023-06-22T08:00:23.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.419+0000 7f112e7ff700 1 -- 172.21.15.23:0/819371566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112804ea20 con 0x7f11280c40b0 2023-06-22T08:00:23.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.420+0000 7f1124ff9700 1 -- 172.21.15.23:0/819371566 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1118007470 con 0x7f11280c40b0 2023-06-22T08:00:23.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.421+0000 7f1124ff9700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 0x7f111406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.421+0000 7f1124ff9700 1 -- 172.21.15.23:0/819371566 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1118089440 con 0x7f11280c40b0 2023-06-22T08:00:23.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.421+0000 7f11277fe700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 0x7f111406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.422+0000 7f11277fe700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 0x7f111406d190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f111c0090b0 tx=0x7f111c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:23.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.426+0000 7f1124ff9700 1 -- 172.21.15.23:0/819371566 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1118054160 con 0x7f11280c40b0 2023-06-22T08:00:23.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.634+0000 7f112e7ff700 1 -- 172.21.15.23:0/819371566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f112814ef10 con 0x7f11280c40b0 2023-06-22T08:00:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.635+0000 7f1124ff9700 1 -- 172.21.15.23:0/819371566 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f1118057780 con 0x7f11280c40b0 2023-06-22T08:00:23.639 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:23.639 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 -- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 msgr2=0x7f111406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 0x7f111406d190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f111c0090b0 tx=0x7f111c008030 comp rx=0 tx=0).stop 2023-06-22T08:00:23.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 -- 172.21.15.23:0/819371566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c40b0 msgr2=0x7f11280cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c40b0 0x7f11280cca40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f11180011e0 tx=0x7f111800ca30 comp rx=0 tx=0).stop 2023-06-22T08:00:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 -- 172.21.15.23:0/819371566 shutdown_connections 2023-06-22T08:00:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1128152ce0 0x7f11280cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11280c40b0 0x7f11280cca40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f111406ace0 0x7f111406d190 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 --2- 172.21.15.23:0/819371566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11280c9fd0 0x7f112815b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 -- 172.21.15.23:0/819371566 >> 172.21.15.23:0/819371566 conn(0x7f112806ae30 msgr2=0x7f1128151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 -- 172.21.15.23:0/819371566 shutdown_connections 2023-06-22T08:00:23.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.638+0000 7f112e7ff700 1 -- 172.21.15.23:0/819371566 wait complete. 2023-06-22T08:00:23.659 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-06-22T08:00:23.660 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-22T08:00:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.856+0000 7fb0bbfff700 1 Processor -- start 2023-06-22T08:00:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.856+0000 7fb0bbfff700 1 -- start start 2023-06-22T08:00:23.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.857+0000 7fb0bbfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 0x7fb0b40cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.857+0000 7fb0bbfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0b40c40b0 0x7fb0b40c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.857+0000 7fb0bbfff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b415c370 con 0x7fb0b40c9fd0 2023-06-22T08:00:23.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.857+0000 7fb0bbfff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b41530b0 con 0x7fb0b40c40b0 2023-06-22T08:00:23.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.857+0000 7fb0bbfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40cf6f0 con 0x7fb0b4152ce0 2023-06-22T08:00:23.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.857+0000 7fb0b9d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 0x7fb0b40cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.857+0000 7fb0ba59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0b40c40b0 0x7fb0b40c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.857+0000 7fb0b9d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 0x7fb0b40cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:48584/0 (socket says 172.21.15.23:48584) 2023-06-22T08:00:23.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.857+0000 7fb0b9d9b700 1 -- 172.21.15.23:0/1104357016 learned_addr learned my addr 172.21.15.23:0/1104357016 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:23.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.857+0000 7fb0b959a700 1 --1- 172.21.15.23:0/1104357016 >> v1:172.21.15.67:6789/0 conn(0x7fb0b40c9fd0 0x7fb0b415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39232/0 (socket says 172.21.15.23:39232) 2023-06-22T08:00:23.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.858+0000 7fb0b9d9b700 1 -- 172.21.15.23:0/1104357016 >> v2:172.21.15.67:3300/0 conn(0x7fb0b40c40b0 msgr2=0x7fb0b40c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:23.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.858+0000 7fb0b9d9b700 1 --2- 172.21.15.23:0/1104357016 >> v2:172.21.15.67:3300/0 conn(0x7fb0b40c40b0 0x7fb0b40c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.858+0000 7fb0b9d9b700 1 -- 172.21.15.23:0/1104357016 >> v1:172.21.15.67:6789/0 conn(0x7fb0b40c9fd0 legacy=0x7fb0b415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.858+0000 7fb0b9d9b700 1 -- 172.21.15.23:0/1104357016 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b415bda0 con 0x7fb0b4152ce0 2023-06-22T08:00:23.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.858+0000 7fb0b9d9b700 1 --2- 172.21.15.23:0/1104357016 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 0x7fb0b40cf1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00098f0 tx=0x7fb0b0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f237d50f962e4f92 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:23.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.858+0000 7fb0b8d99700 1 -- 172.21.15.23:0/1104357016 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0011830 con 0x7fb0b4152ce0 2023-06-22T08:00:23.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.858+0000 7fb0b8d99700 1 -- 172.21.15.23:0/1104357016 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b0011990 con 0x7fb0b4152ce0 2023-06-22T08:00:23.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.858+0000 7fb0b8d99700 1 -- 172.21.15.23:0/1104357016 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0011d20 con 0x7fb0b4152ce0 2023-06-22T08:00:23.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.859+0000 7fb0bbfff700 1 -- 172.21.15.23:0/1104357016 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 msgr2=0x7fb0b40cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.859+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/1104357016 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 0x7fb0b40cf1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00098f0 tx=0x7fb0b0009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:23.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.859+0000 7fb0bbfff700 1 -- 172.21.15.23:0/1104357016 shutdown_connections 2023-06-22T08:00:23.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.859+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/1104357016 >> v2:172.21.15.67:3300/0 conn(0x7fb0b40c40b0 0x7fb0b40c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.859+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/1104357016 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 0x7fb0b40cf1b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.859+0000 7fb0bbfff700 1 -- 172.21.15.23:0/1104357016 >> 172.21.15.23:0/1104357016 conn(0x7fb0b406ae30 msgr2=0x7fb0b406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:23.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.859+0000 7fb0bbfff700 1 -- 172.21.15.23:0/1104357016 shutdown_connections 2023-06-22T08:00:23.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.859+0000 7fb0bbfff700 1 -- 172.21.15.23:0/1104357016 wait complete. 2023-06-22T08:00:23.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.861+0000 7fb0bbfff700 1 Processor -- start 2023-06-22T08:00:23.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.861+0000 7fb0bbfff700 1 -- start start 2023-06-22T08:00:23.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0bbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b40c40b0 0x7fb0b41e1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0bbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b40c9fd0 0x7fb0b40ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0bbfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 0x7fb0b414b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0bbfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b41f2750 con 0x7fb0b40c9fd0 2023-06-22T08:00:23.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0bbfff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b41e1b50 con 0x7fb0b40c40b0 2023-06-22T08:00:23.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0bbfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b41f1b40 con 0x7fb0b4152ce0 2023-06-22T08:00:23.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0b9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b40c40b0 0x7fb0b41e1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0b959a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b40c9fd0 0x7fb0b40ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0b9d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b40c40b0 0x7fb0b41e1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:48200/0 (socket says 172.21.15.23:48200) 2023-06-22T08:00:23.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0b9d9b700 1 -- 172.21.15.23:0/2394441808 learned_addr learned my addr 172.21.15.23:0/2394441808 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:23.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.862+0000 7fb0ba59c700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 0x7fb0b414b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0b9d9b700 1 -- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 msgr2=0x7fb0b414b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0b9d9b700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 0x7fb0b414b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0b9d9b700 1 -- 172.21.15.23:0/2394441808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b40c9fd0 msgr2=0x7fb0b40ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:23.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0b9d9b700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b40c9fd0 0x7fb0b40ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:23.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0b9d9b700 1 -- 172.21.15.23:0/2394441808 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0009550 con 0x7fb0b40c40b0 2023-06-22T08:00:23.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0b9d9b700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b40c40b0 0x7fb0b41e1510 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009c40 tx=0x7fb0b0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:23.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0aa7fc700 1 -- 172.21.15.23:0/2394441808 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0004510 con 0x7fb0b40c40b0 2023-06-22T08:00:23.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0aa7fc700 1 -- 172.21.15.23:0/2394441808 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b00036a0 con 0x7fb0b40c40b0 2023-06-22T08:00:23.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0bbfff700 1 -- 172.21.15.23:0/2394441808 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b41f1e20 con 0x7fb0b40c40b0 2023-06-22T08:00:23.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0aa7fc700 1 -- 172.21.15.23:0/2394441808 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0003a50 con 0x7fb0b40c40b0 2023-06-22T08:00:23.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.863+0000 7fb0bbfff700 1 -- 172.21.15.23:0/2394441808 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b41f23e0 con 0x7fb0b40c40b0 2023-06-22T08:00:23.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.864+0000 7fb0bbfff700 1 -- 172.21.15.23:0/2394441808 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b404ea20 con 0x7fb0b40c40b0 2023-06-22T08:00:23.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.865+0000 7fb0aa7fc700 1 -- 172.21.15.23:0/2394441808 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb0b0003bb0 con 0x7fb0b40c40b0 2023-06-22T08:00:23.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.866+0000 7fb0aa7fc700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09806ace0 0x7fb09806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:23.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.866+0000 7fb0aa7fc700 1 -- 172.21.15.23:0/2394441808 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb0b008a970 con 0x7fb0b40c40b0 2023-06-22T08:00:23.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.867+0000 7fb0b959a700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09806ace0 0x7fb09806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:23.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.867+0000 7fb0b959a700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09806ace0 0x7fb09806d190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40c9100 tx=0x7fb0a4007ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:23.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:23.871+0000 7fb0aa7fc700 1 -- 172.21.15.23:0/2394441808 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb0b00556a0 con 0x7fb0b40c40b0 2023-06-22T08:00:24.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.074+0000 7fb0bbfff700 1 -- 172.21.15.23:0/2394441808 --> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0b40c9740 con 0x7fb09806ace0 2023-06-22T08:00:24.089 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.087+0000 7fb0aa7fc700 1 -- 172.21.15.23:0/2394441808 <== mgr.4100 v2:172.21.15.67:6832/3180103139 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367422 (secure 0 0 0) 0x7fb0a4001330 con 0x7fb09806ace0 2023-06-22T08:00:24.094 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 -- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09806ace0 msgr2=0x7fb09806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09806ace0 0x7fb09806d190 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40c9100 tx=0x7fb0a4007ed0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 -- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b40c40b0 msgr2=0x7fb0b41e1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b40c40b0 0x7fb0b41e1510 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009c40 tx=0x7fb0b0004750 comp rx=0 tx=0).stop 2023-06-22T08:00:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 -- 172.21.15.23:0/2394441808 shutdown_connections 2023-06-22T08:00:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0b4152ce0 0x7fb0b414b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09806ace0 0x7fb09806d190 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0b40c40b0 0x7fb0b41e1510 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/2394441808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b40c9fd0 0x7fb0b40ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 -- 172.21.15.23:0/2394441808 >> 172.21.15.23:0/2394441808 conn(0x7fb0b406ae30 msgr2=0x7fb0b414cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 -- 172.21.15.23:0/2394441808 shutdown_connections 2023-06-22T08:00:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.091+0000 7fb0bbfff700 1 -- 172.21.15.23:0/2394441808 wait complete. 2023-06-22T08:00:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-06-22T08:00:24.113 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2023-06-22T08:00:22.560510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594960,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219176,"kb_used_data":3560,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755544,"statfs":{"total":773094113280,"available":772869677056,"internally_reserved":0,"allocated":3645440,"data_stored":2178384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003436"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709823+0000","last_change":"2023-06-22T07:59:55.165740+0000","last_active":"2023-06-22T07:59:59.709823+0000","last_peered":"2023-06-22T07:59:59.709823+0000","last_clean":"2023-06-22T07:59:59.709823+0000","last_became_active":"2023-06-22T07:59:55.165398+0000","last_became_peered":"2023-06-22T07:59:55.165398+0000","last_unstale":"2023-06-22T07:59:59.709823+0000","last_undegraded":"2023-06-22T07:59:59.709823+0000","last_fullsized":"2023-06-22T07:59:59.709823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:37.596656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709026+0000","last_change":"2023-06-22T07:59:55.174603+0000","last_active":"2023-06-22T07:59:59.709026+0000","last_peered":"2023-06-22T07:59:59.709026+0000","last_clean":"2023-06-22T07:59:59.709026+0000","last_became_active":"2023-06-22T07:59:55.174346+0000","last_became_peered":"2023-06-22T07:59:55.174346+0000","last_unstale":"2023-06-22T07:59:59.709026+0000","last_undegraded":"2023-06-22T07:59:59.709026+0000","last_fullsized":"2023-06-22T07:59:59.709026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157321+0000","last_change":"2023-06-22T07:59:55.167525+0000","last_active":"2023-06-22T08:00:00.157321+0000","last_peered":"2023-06-22T08:00:00.157321+0000","last_clean":"2023-06-22T08:00:00.157321+0000","last_became_active":"2023-06-22T07:59:55.167181+0000","last_became_peered":"2023-06-22T07:59:55.167181+0000","last_unstale":"2023-06-22T08:00:00.157321+0000","last_undegraded":"2023-06-22T08:00:00.157321+0000","last_fullsized":"2023-06-22T08:00:00.157321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157927+0000","last_change":"2023-06-22T07:59:55.160549+0000","last_active":"2023-06-22T08:00:00.157927+0000","last_peered":"2023-06-22T08:00:00.157927+0000","last_clean":"2023-06-22T08:00:00.157927+0000","last_became_active":"2023-06-22T07:59:55.160239+0000","last_became_peered":"2023-06-22T07:59:55.160239+0000","last_unstale":"2023-06-22T08:00:00.157927+0000","last_undegraded":"2023-06-22T08:00:00.157927+0000","last_fullsized":"2023-06-22T08:00:00.157927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754390+0000","last_change":"2023-06-22T07:59:55.173006+0000","last_active":"2023-06-22T08:00:00.754390+0000","last_peered":"2023-06-22T08:00:00.754390+0000","last_clean":"2023-06-22T08:00:00.754390+0000","last_became_active":"2023-06-22T07:59:55.172578+0000","last_became_peered":"2023-06-22T07:59:55.172578+0000","last_unstale":"2023-06-22T08:00:00.754390+0000","last_undegraded":"2023-06-22T08:00:00.754390+0000","last_fullsized":"2023-06-22T08:00:00.754390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:12:44.799856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871378+0000","last_change":"2023-06-22T07:59:55.173858+0000","last_active":"2023-06-22T07:59:59.871378+0000","last_peered":"2023-06-22T07:59:59.871378+0000","last_clean":"2023-06-22T07:59:59.871378+0000","last_became_active":"2023-06-22T07:59:55.173599+0000","last_became_peered":"2023-06-22T07:59:55.173599+0000","last_unstale":"2023-06-22T07:59:59.871378+0000","last_undegraded":"2023-06-22T07:59:59.871378+0000","last_fullsized":"2023-06-22T07:59:59.871378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:37.597195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871174+0000","last_change":"2023-06-22T07:59:55.168354+0000","last_active":"2023-06-22T07:59:59.871174+0000","last_peered":"2023-06-22T07:59:59.871174+0000","last_clean":"2023-06-22T07:59:59.871174+0000","last_became_active":"2023-06-22T07:59:55.168050+0000","last_became_peered":"2023-06-22T07:59:55.168050+0000","last_unstale":"2023-06-22T07:59:59.871174+0000","last_undegraded":"2023-06-22T07:59:59.871174+0000","last_fullsized":"2023-06-22T07:59:59.871174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:08:45.847378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155602+0000","last_change":"2023-06-22T07:59:55.159738+0000","last_active":"2023-06-22T08:00:00.155602+0000","last_peered":"2023-06-22T08:00:00.155602+0000","last_clean":"2023-06-22T08:00:00.155602+0000","last_became_active":"2023-06-22T07:59:55.159501+0000","last_became_peered":"2023-06-22T07:59:55.159501+0000","last_unstale":"2023-06-22T08:00:00.155602+0000","last_undegraded":"2023-06-22T08:00:00.155602+0000","last_fullsized":"2023-06-22T08:00:00.155602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:55.160768+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:55.160508+0000","last_became_peered":"2023-06-22T07:59:55.160508+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:55:48.713145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155821+0000","last_change":"2023-06-22T07:59:55.166546+0000","last_active":"2023-06-22T08:00:00.155821+0000","last_peered":"2023-06-22T08:00:00.155821+0000","last_clean":"2023-06-22T08:00:00.155821+0000","last_became_active":"2023-06-22T07:59:55.166311+0000","last_became_peered":"2023-06-22T07:59:55.166311+0000","last_unstale":"2023-06-22T08:00:00.155821+0000","last_undegraded":"2023-06-22T08:00:00.155821+0000","last_fullsized":"2023-06-22T08:00:00.155821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:35.822548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155943+0000","last_change":"2023-06-22T07:59:55.162295+0000","last_active":"2023-06-22T08:00:00.155943+0000","last_peered":"2023-06-22T08:00:00.155943+0000","last_clean":"2023-06-22T08:00:00.155943+0000","last_became_active":"2023-06-22T07:59:55.162010+0000","last_became_peered":"2023-06-22T07:59:55.162010+0000","last_unstale":"2023-06-22T08:00:00.155943+0000","last_undegraded":"2023-06-22T08:00:00.155943+0000","last_fullsized":"2023-06-22T08:00:00.155943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709563+0000","last_change":"2023-06-22T07:59:55.158990+0000","last_active":"2023-06-22T07:59:59.709563+0000","last_peered":"2023-06-22T07:59:59.709563+0000","last_clean":"2023-06-22T07:59:59.709563+0000","last_became_active":"2023-06-22T07:59:55.158614+0000","last_became_peered":"2023-06-22T07:59:55.158614+0000","last_unstale":"2023-06-22T07:59:59.709563+0000","last_undegraded":"2023-06-22T07:59:59.709563+0000","last_fullsized":"2023-06-22T07:59:59.709563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755166+0000","last_change":"2023-06-22T07:59:55.161453+0000","last_active":"2023-06-22T08:00:00.755166+0000","last_peered":"2023-06-22T08:00:00.755166+0000","last_clean":"2023-06-22T08:00:00.755166+0000","last_became_active":"2023-06-22T07:59:55.161168+0000","last_became_peered":"2023-06-22T07:59:55.161168+0000","last_unstale":"2023-06-22T08:00:00.755166+0000","last_undegraded":"2023-06-22T08:00:00.755166+0000","last_fullsized":"2023-06-22T08:00:00.755166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155022+0000","last_change":"2023-06-22T07:59:55.164270+0000","last_active":"2023-06-22T08:00:00.155022+0000","last_peered":"2023-06-22T08:00:00.155022+0000","last_clean":"2023-06-22T08:00:00.155022+0000","last_became_active":"2023-06-22T07:59:55.163993+0000","last_became_peered":"2023-06-22T07:59:55.163993+0000","last_unstale":"2023-06-22T08:00:00.155022+0000","last_undegraded":"2023-06-22T08:00:00.155022+0000","last_fullsized":"2023-06-22T08:00:00.155022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:52.051830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157430+0000","last_change":"2023-06-22T07:59:55.169039+0000","last_active":"2023-06-22T08:00:00.157430+0000","last_peered":"2023-06-22T08:00:00.157430+0000","last_clean":"2023-06-22T08:00:00.157430+0000","last_became_active":"2023-06-22T07:59:55.168698+0000","last_became_peered":"2023-06-22T07:59:55.168698+0000","last_unstale":"2023-06-22T08:00:00.157430+0000","last_undegraded":"2023-06-22T08:00:00.157430+0000","last_fullsized":"2023-06-22T08:00:00.157430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871035+0000","last_change":"2023-06-22T07:59:55.158014+0000","last_active":"2023-06-22T07:59:59.871035+0000","last_peered":"2023-06-22T07:59:59.871035+0000","last_clean":"2023-06-22T07:59:59.871035+0000","last_became_active":"2023-06-22T07:59:55.157562+0000","last_became_peered":"2023-06-22T07:59:55.157562+0000","last_unstale":"2023-06-22T07:59:59.871035+0000","last_undegraded":"2023-06-22T07:59:59.871035+0000","last_fullsized":"2023-06-22T07:59:59.871035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:08:33.856001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754963+0000","last_change":"2023-06-22T07:59:55.165957+0000","last_active":"2023-06-22T08:00:00.754963+0000","last_peered":"2023-06-22T08:00:00.754963+0000","last_clean":"2023-06-22T08:00:00.754963+0000","last_became_active":"2023-06-22T07:59:55.165643+0000","last_became_peered":"2023-06-22T07:59:55.165643+0000","last_unstale":"2023-06-22T08:00:00.754963+0000","last_undegraded":"2023-06-22T08:00:00.754963+0000","last_fullsized":"2023-06-22T08:00:00.754963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:10.134743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157391+0000","last_change":"2023-06-22T07:59:55.159858+0000","last_active":"2023-06-22T08:00:00.157391+0000","last_peered":"2023-06-22T08:00:00.157391+0000","last_clean":"2023-06-22T08:00:00.157391+0000","last_became_active":"2023-06-22T07:59:55.159444+0000","last_became_peered":"2023-06-22T07:59:55.159444+0000","last_unstale":"2023-06-22T08:00:00.157391+0000","last_undegraded":"2023-06-22T08:00:00.157391+0000","last_fullsized":"2023-06-22T08:00:00.157391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709424+0000","last_change":"2023-06-22T07:59:55.157162+0000","last_active":"2023-06-22T07:59:59.709424+0000","last_peered":"2023-06-22T07:59:59.709424+0000","last_clean":"2023-06-22T07:59:59.709424+0000","last_became_active":"2023-06-22T07:59:55.156882+0000","last_became_peered":"2023-06-22T07:59:55.156882+0000","last_unstale":"2023-06-22T07:59:59.709424+0000","last_undegraded":"2023-06-22T07:59:59.709424+0000","last_fullsized":"2023-06-22T07:59:59.709424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155892+0000","last_change":"2023-06-22T07:59:55.163834+0000","last_active":"2023-06-22T08:00:00.155892+0000","last_peered":"2023-06-22T08:00:00.155892+0000","last_clean":"2023-06-22T08:00:00.155892+0000","last_became_active":"2023-06-22T07:59:55.163580+0000","last_became_peered":"2023-06-22T07:59:55.163580+0000","last_unstale":"2023-06-22T08:00:00.155892+0000","last_undegraded":"2023-06-22T08:00:00.155892+0000","last_fullsized":"2023-06-22T08:00:00.155892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155285+0000","last_change":"2023-06-22T07:59:55.164569+0000","last_active":"2023-06-22T08:00:00.155285+0000","last_peered":"2023-06-22T08:00:00.155285+0000","last_clean":"2023-06-22T08:00:00.155285+0000","last_became_active":"2023-06-22T07:59:55.164286+0000","last_became_peered":"2023-06-22T07:59:55.164286+0000","last_unstale":"2023-06-22T08:00:00.155285+0000","last_undegraded":"2023-06-22T08:00:00.155285+0000","last_fullsized":"2023-06-22T08:00:00.155285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:17.386317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754137+0000","last_change":"2023-06-22T07:59:55.159883+0000","last_active":"2023-06-22T08:00:00.754137+0000","last_peered":"2023-06-22T08:00:00.754137+0000","last_clean":"2023-06-22T08:00:00.754137+0000","last_became_active":"2023-06-22T07:59:55.159588+0000","last_became_peered":"2023-06-22T07:59:55.159588+0000","last_unstale":"2023-06-22T08:00:00.754137+0000","last_undegraded":"2023-06-22T08:00:00.754137+0000","last_fullsized":"2023-06-22T08:00:00.754137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754941+0000","last_change":"2023-06-22T07:59:55.160638+0000","last_active":"2023-06-22T08:00:00.754941+0000","last_peered":"2023-06-22T08:00:00.754941+0000","last_clean":"2023-06-22T08:00:00.754941+0000","last_became_active":"2023-06-22T07:59:55.160315+0000","last_became_peered":"2023-06-22T07:59:55.160315+0000","last_unstale":"2023-06-22T08:00:00.754941+0000","last_undegraded":"2023-06-22T08:00:00.754941+0000","last_fullsized":"2023-06-22T08:00:00.754941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:51:09.921626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155504+0000","last_change":"2023-06-22T07:59:55.165950+0000","last_active":"2023-06-22T08:00:00.155504+0000","last_peered":"2023-06-22T08:00:00.155504+0000","last_clean":"2023-06-22T08:00:00.155504+0000","last_became_active":"2023-06-22T07:59:55.165505+0000","last_became_peered":"2023-06-22T07:59:55.165505+0000","last_unstale":"2023-06-22T08:00:00.155504+0000","last_undegraded":"2023-06-22T08:00:00.155504+0000","last_fullsized":"2023-06-22T08:00:00.155504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:13.512588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755640+0000","last_change":"2023-06-22T07:59:55.157572+0000","last_active":"2023-06-22T08:00:00.755640+0000","last_peered":"2023-06-22T08:00:00.755640+0000","last_clean":"2023-06-22T08:00:00.755640+0000","last_became_active":"2023-06-22T07:59:55.157055+0000","last_became_peered":"2023-06-22T07:59:55.157055+0000","last_unstale":"2023-06-22T08:00:00.755640+0000","last_undegraded":"2023-06-22T08:00:00.755640+0000","last_fullsized":"2023-06-22T08:00:00.755640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:34:35.685117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754095+0000","last_change":"2023-06-22T07:59:55.164696+0000","last_active":"2023-06-22T08:00:00.754095+0000","last_peered":"2023-06-22T08:00:00.754095+0000","last_clean":"2023-06-22T08:00:00.754095+0000","last_became_active":"2023-06-22T07:59:55.164179+0000","last_became_peered":"2023-06-22T07:59:55.164179+0000","last_unstale":"2023-06-22T08:00:00.754095+0000","last_undegraded":"2023-06-22T08:00:00.754095+0000","last_fullsized":"2023-06-22T08:00:00.754095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871504+0000","last_change":"2023-06-22T07:59:55.168838+0000","last_active":"2023-06-22T07:59:59.871504+0000","last_peered":"2023-06-22T07:59:59.871504+0000","last_clean":"2023-06-22T07:59:59.871504+0000","last_became_active":"2023-06-22T07:59:55.168551+0000","last_became_peered":"2023-06-22T07:59:55.168551+0000","last_unstale":"2023-06-22T07:59:59.871504+0000","last_undegraded":"2023-06-22T07:59:59.871504+0000","last_fullsized":"2023-06-22T07:59:59.871504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:02:36.615449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708648+0000","last_change":"2023-06-22T07:59:59.195356+0000","last_active":"2023-06-22T07:59:59.708648+0000","last_peered":"2023-06-22T07:59:59.708648+0000","last_clean":"2023-06-22T07:59:59.708648+0000","last_became_active":"2023-06-22T07:59:59.194814+0000","last_became_peered":"2023-06-22T07:59:59.194814+0000","last_unstale":"2023-06-22T07:59:59.708648+0000","last_undegraded":"2023-06-22T07:59:59.708648+0000","last_fullsized":"2023-06-22T07:59:59.708648+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:12.223267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6,0],"acting":[2,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"27'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155337+0000","last_change":"2023-06-22T07:59:54.651326+0000","last_active":"2023-06-22T08:00:00.155337+0000","last_peered":"2023-06-22T08:00:00.155337+0000","last_clean":"2023-06-22T08:00:00.155337+0000","last_became_active":"2023-06-22T07:59:54.651010+0000","last_became_peered":"2023-06-22T07:59:54.651010+0000","last_unstale":"2023-06-22T08:00:00.155337+0000","last_undegraded":"2023-06-22T08:00:00.155337+0000","last_fullsized":"2023-06-22T08:00:00.155337+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:17.769117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754538+0000","last_change":"2023-06-22T07:59:55.160345+0000","last_active":"2023-06-22T08:00:00.754538+0000","last_peered":"2023-06-22T08:00:00.754538+0000","last_clean":"2023-06-22T08:00:00.754538+0000","last_became_active":"2023-06-22T07:59:55.160071+0000","last_became_peered":"2023-06-22T07:59:55.160071+0000","last_unstale":"2023-06-22T08:00:00.754538+0000","last_undegraded":"2023-06-22T08:00:00.754538+0000","last_fullsized":"2023-06-22T08:00:00.754538+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708662+0000","last_change":"2023-06-22T07:59:59.196477+0000","last_active":"2023-06-22T07:59:59.708662+0000","last_peered":"2023-06-22T07:59:59.708662+0000","last_clean":"2023-06-22T07:59:59.708662+0000","last_became_active":"2023-06-22T07:59:59.196007+0000","last_became_peered":"2023-06-22T07:59:59.196007+0000","last_unstale":"2023-06-22T07:59:59.708662+0000","last_undegraded":"2023-06-22T07:59:59.708662+0000","last_fullsized":"2023-06-22T07:59:59.708662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,5],"acting":[2,7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753791+0000","last_change":"2023-06-22T07:59:54.653192+0000","last_active":"2023-06-22T08:00:00.753791+0000","last_peered":"2023-06-22T08:00:00.753791+0000","last_clean":"2023-06-22T08:00:00.753791+0000","last_became_active":"2023-06-22T07:59:54.652926+0000","last_became_peered":"2023-06-22T07:59:54.652926+0000","last_unstale":"2023-06-22T08:00:00.753791+0000","last_undegraded":"2023-06-22T08:00:00.753791+0000","last_fullsized":"2023-06-22T08:00:00.753791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:12:59.656555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754609+0000","last_change":"2023-06-22T07:59:55.163171+0000","last_active":"2023-06-22T08:00:00.754609+0000","last_peered":"2023-06-22T08:00:00.754609+0000","last_clean":"2023-06-22T08:00:00.754609+0000","last_became_active":"2023-06-22T07:59:55.162869+0000","last_became_peered":"2023-06-22T07:59:55.162869+0000","last_unstale":"2023-06-22T08:00:00.754609+0000","last_undegraded":"2023-06-22T08:00:00.754609+0000","last_fullsized":"2023-06-22T08:00:00.754609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:37:59.063186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754586+0000","last_change":"2023-06-22T07:59:59.200516+0000","last_active":"2023-06-22T08:00:00.754586+0000","last_peered":"2023-06-22T08:00:00.754586+0000","last_clean":"2023-06-22T08:00:00.754586+0000","last_became_active":"2023-06-22T07:59:59.200096+0000","last_became_peered":"2023-06-22T07:59:59.200096+0000","last_unstale":"2023-06-22T08:00:00.754586+0000","last_undegraded":"2023-06-22T08:00:00.754586+0000","last_fullsized":"2023-06-22T08:00:00.754586+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:06.099105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2,7],"acting":[6,3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708683+0000","last_change":"2023-06-22T07:59:54.645950+0000","last_active":"2023-06-22T07:59:59.708683+0000","last_peered":"2023-06-22T07:59:59.708683+0000","last_clean":"2023-06-22T07:59:59.708683+0000","last_became_active":"2023-06-22T07:59:54.645520+0000","last_became_peered":"2023-06-22T07:59:54.645520+0000","last_unstale":"2023-06-22T07:59:59.708683+0000","last_undegraded":"2023-06-22T07:59:59.708683+0000","last_fullsized":"2023-06-22T07:59:59.708683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:34.816556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871992+0000","last_change":"2023-06-22T07:59:55.162767+0000","last_active":"2023-06-22T07:59:59.871992+0000","last_peered":"2023-06-22T07:59:59.871992+0000","last_clean":"2023-06-22T07:59:59.871992+0000","last_became_active":"2023-06-22T07:59:55.162451+0000","last_became_peered":"2023-06-22T07:59:55.162451+0000","last_unstale":"2023-06-22T07:59:59.871992+0000","last_undegraded":"2023-06-22T07:59:59.871992+0000","last_fullsized":"2023-06-22T07:59:59.871992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:38.977413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871071+0000","last_change":"2023-06-22T07:59:59.208215+0000","last_active":"2023-06-22T07:59:59.871071+0000","last_peered":"2023-06-22T07:59:59.871071+0000","last_clean":"2023-06-22T07:59:59.871071+0000","last_became_active":"2023-06-22T07:59:59.207608+0000","last_became_peered":"2023-06-22T07:59:59.207608+0000","last_unstale":"2023-06-22T07:59:59.871071+0000","last_undegraded":"2023-06-22T07:59:59.871071+0000","last_fullsized":"2023-06-22T07:59:59.871071+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:39:48.527076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,5],"acting":[3,2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709379+0000","last_change":"2023-06-22T07:59:54.652515+0000","last_active":"2023-06-22T07:59:59.709379+0000","last_peered":"2023-06-22T07:59:59.709379+0000","last_clean":"2023-06-22T07:59:59.709379+0000","last_became_active":"2023-06-22T07:59:54.652249+0000","last_became_peered":"2023-06-22T07:59:54.652249+0000","last_unstale":"2023-06-22T07:59:59.709379+0000","last_undegraded":"2023-06-22T07:59:59.709379+0000","last_fullsized":"2023-06-22T07:59:59.709379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:12.506596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755250+0000","last_change":"2023-06-22T07:59:55.158359+0000","last_active":"2023-06-22T08:00:00.755250+0000","last_peered":"2023-06-22T08:00:00.755250+0000","last_clean":"2023-06-22T08:00:00.755250+0000","last_became_active":"2023-06-22T07:59:55.157918+0000","last_became_peered":"2023-06-22T07:59:55.157918+0000","last_unstale":"2023-06-22T08:00:00.755250+0000","last_undegraded":"2023-06-22T08:00:00.755250+0000","last_fullsized":"2023-06-22T08:00:00.755250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708305+0000","last_change":"2023-06-22T07:59:59.201033+0000","last_active":"2023-06-22T07:59:59.708305+0000","last_peered":"2023-06-22T07:59:59.708305+0000","last_clean":"2023-06-22T07:59:59.708305+0000","last_became_active":"2023-06-22T07:59:59.200500+0000","last_became_peered":"2023-06-22T07:59:59.200500+0000","last_unstale":"2023-06-22T07:59:59.708305+0000","last_undegraded":"2023-06-22T07:59:59.708305+0000","last_fullsized":"2023-06-22T07:59:59.708305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:18.174604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,4],"acting":[2,1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708780+0000","last_change":"2023-06-22T07:59:54.654416+0000","last_active":"2023-06-22T07:59:59.708780+0000","last_peered":"2023-06-22T07:59:59.708780+0000","last_clean":"2023-06-22T07:59:59.708780+0000","last_became_active":"2023-06-22T07:59:54.654139+0000","last_became_peered":"2023-06-22T07:59:54.654139+0000","last_unstale":"2023-06-22T07:59:59.708780+0000","last_undegraded":"2023-06-22T07:59:59.708780+0000","last_fullsized":"2023-06-22T07:59:59.708780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755506+0000","last_change":"2023-06-22T07:59:55.159790+0000","last_active":"2023-06-22T08:00:00.755506+0000","last_peered":"2023-06-22T08:00:00.755506+0000","last_clean":"2023-06-22T08:00:00.755506+0000","last_became_active":"2023-06-22T07:59:55.159489+0000","last_became_peered":"2023-06-22T07:59:55.159489+0000","last_unstale":"2023-06-22T08:00:00.755506+0000","last_undegraded":"2023-06-22T08:00:00.755506+0000","last_fullsized":"2023-06-22T08:00:00.755506+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:15:18.373418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753814+0000","last_change":"2023-06-22T07:59:59.197822+0000","last_active":"2023-06-22T08:00:00.753814+0000","last_peered":"2023-06-22T08:00:00.753814+0000","last_clean":"2023-06-22T08:00:00.753814+0000","last_became_active":"2023-06-22T07:59:59.197431+0000","last_became_peered":"2023-06-22T07:59:59.197431+0000","last_unstale":"2023-06-22T08:00:00.753814+0000","last_undegraded":"2023-06-22T08:00:00.753814+0000","last_fullsized":"2023-06-22T08:00:00.753814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:41.139500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7,1],"acting":[4,6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"27'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157177+0000","last_change":"2023-06-22T07:59:54.644158+0000","last_active":"2023-06-22T08:00:00.157177+0000","last_peered":"2023-06-22T08:00:00.157177+0000","last_clean":"2023-06-22T08:00:00.157177+0000","last_became_active":"2023-06-22T07:59:54.643808+0000","last_became_peered":"2023-06-22T07:59:54.643808+0000","last_unstale":"2023-06-22T08:00:00.157177+0000","last_undegraded":"2023-06-22T08:00:00.157177+0000","last_fullsized":"2023-06-22T08:00:00.157177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:45.438691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754507+0000","last_change":"2023-06-22T07:59:55.157538+0000","last_active":"2023-06-22T08:00:00.754507+0000","last_peered":"2023-06-22T08:00:00.754507+0000","last_clean":"2023-06-22T08:00:00.754507+0000","last_became_active":"2023-06-22T07:59:55.156967+0000","last_became_peered":"2023-06-22T07:59:55.156967+0000","last_unstale":"2023-06-22T08:00:00.754507+0000","last_undegraded":"2023-06-22T08:00:00.754507+0000","last_fullsized":"2023-06-22T08:00:00.754507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:38:44.161245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753821+0000","last_change":"2023-06-22T07:59:59.201671+0000","last_active":"2023-06-22T08:00:00.753821+0000","last_peered":"2023-06-22T08:00:00.753821+0000","last_clean":"2023-06-22T08:00:00.753821+0000","last_became_active":"2023-06-22T07:59:59.201201+0000","last_became_peered":"2023-06-22T07:59:59.201201+0000","last_unstale":"2023-06-22T08:00:00.753821+0000","last_undegraded":"2023-06-22T08:00:00.753821+0000","last_fullsized":"2023-06-22T08:00:00.753821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4,3],"acting":[6,1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"28'10","reported_seq":31,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:17.682776+0000","last_change":"2023-06-22T07:59:54.640470+0000","last_active":"2023-06-22T08:00:17.682776+0000","last_peered":"2023-06-22T08:00:17.682776+0000","last_clean":"2023-06-22T08:00:17.682776+0000","last_became_active":"2023-06-22T07:59:54.640124+0000","last_became_peered":"2023-06-22T07:59:54.640124+0000","last_unstale":"2023-06-22T08:00:17.682776+0000","last_undegraded":"2023-06-22T08:00:17.682776+0000","last_fullsized":"2023-06-22T08:00:17.682776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:06:41.502702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708599+0000","last_change":"2023-06-22T07:59:55.165550+0000","last_active":"2023-06-22T07:59:59.708599+0000","last_peered":"2023-06-22T07:59:59.708599+0000","last_clean":"2023-06-22T07:59:59.708599+0000","last_became_active":"2023-06-22T07:59:55.164998+0000","last_became_peered":"2023-06-22T07:59:55.164998+0000","last_unstale":"2023-06-22T07:59:59.708599+0000","last_undegraded":"2023-06-22T07:59:59.708599+0000","last_fullsized":"2023-06-22T07:59:59.708599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872133+0000","last_change":"2023-06-22T07:59:59.207103+0000","last_active":"2023-06-22T07:59:59.872133+0000","last_peered":"2023-06-22T07:59:59.872133+0000","last_clean":"2023-06-22T07:59:59.872133+0000","last_became_active":"2023-06-22T07:59:59.206531+0000","last_became_peered":"2023-06-22T07:59:59.206531+0000","last_unstale":"2023-06-22T07:59:59.872133+0000","last_undegraded":"2023-06-22T07:59:59.872133+0000","last_fullsized":"2023-06-22T07:59:59.872133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:42:48.571228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0,6],"acting":[3,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157887+0000","last_change":"2023-06-22T07:59:54.644301+0000","last_active":"2023-06-22T08:00:00.157887+0000","last_peered":"2023-06-22T08:00:00.157887+0000","last_clean":"2023-06-22T08:00:00.157887+0000","last_became_active":"2023-06-22T07:59:54.643939+0000","last_became_peered":"2023-06-22T07:59:54.643939+0000","last_unstale":"2023-06-22T08:00:00.157887+0000","last_undegraded":"2023-06-22T08:00:00.157887+0000","last_fullsized":"2023-06-22T08:00:00.157887+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:07:42.118767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753877+0000","last_change":"2023-06-22T07:59:55.163311+0000","last_active":"2023-06-22T08:00:00.753877+0000","last_peered":"2023-06-22T08:00:00.753877+0000","last_clean":"2023-06-22T08:00:00.753877+0000","last_became_active":"2023-06-22T07:59:55.162991+0000","last_became_peered":"2023-06-22T07:59:55.162991+0000","last_unstale":"2023-06-22T08:00:00.753877+0000","last_undegraded":"2023-06-22T08:00:00.753877+0000","last_fullsized":"2023-06-22T08:00:00.753877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754227+0000","last_change":"2023-06-22T07:59:59.201324+0000","last_active":"2023-06-22T08:00:00.754227+0000","last_peered":"2023-06-22T08:00:00.754227+0000","last_clean":"2023-06-22T08:00:00.754227+0000","last_became_active":"2023-06-22T07:59:59.200877+0000","last_became_peered":"2023-06-22T07:59:59.200877+0000","last_unstale":"2023-06-22T08:00:00.754227+0000","last_undegraded":"2023-06-22T08:00:00.754227+0000","last_fullsized":"2023-06-22T08:00:00.754227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:14:17.094426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5,7],"acting":[4,2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754441+0000","last_change":"2023-06-22T07:59:54.651527+0000","last_active":"2023-06-22T08:00:00.754441+0000","last_peered":"2023-06-22T08:00:00.754441+0000","last_clean":"2023-06-22T08:00:00.754441+0000","last_became_active":"2023-06-22T07:59:54.651196+0000","last_became_peered":"2023-06-22T07:59:54.651196+0000","last_unstale":"2023-06-22T08:00:00.754441+0000","last_undegraded":"2023-06-22T08:00:00.754441+0000","last_fullsized":"2023-06-22T08:00:00.754441+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872142+0000","last_change":"2023-06-22T07:59:55.162610+0000","last_active":"2023-06-22T07:59:59.872142+0000","last_peered":"2023-06-22T07:59:59.872142+0000","last_clean":"2023-06-22T07:59:59.872142+0000","last_became_active":"2023-06-22T07:59:55.162334+0000","last_became_peered":"2023-06-22T07:59:55.162334+0000","last_unstale":"2023-06-22T07:59:59.872142+0000","last_undegraded":"2023-06-22T07:59:59.872142+0000","last_fullsized":"2023-06-22T07:59:59.872142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:42.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:59.199402+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:59.198979+0000","last_became_peered":"2023-06-22T07:59:59.198979+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:17:07.122732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1,6],"acting":[4,5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.870997+0000","last_change":"2023-06-22T07:59:54.649236+0000","last_active":"2023-06-22T07:59:59.870997+0000","last_peered":"2023-06-22T07:59:59.870997+0000","last_clean":"2023-06-22T07:59:59.870997+0000","last_became_active":"2023-06-22T07:59:54.648852+0000","last_became_peered":"2023-06-22T07:59:54.648852+0000","last_unstale":"2023-06-22T07:59:59.870997+0000","last_undegraded":"2023-06-22T07:59:59.870997+0000","last_fullsized":"2023-06-22T07:59:59.870997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:16:59.865638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709134+0000","last_change":"2023-06-22T07:59:55.165859+0000","last_active":"2023-06-22T07:59:59.709134+0000","last_peered":"2023-06-22T07:59:59.709134+0000","last_clean":"2023-06-22T07:59:59.709134+0000","last_became_active":"2023-06-22T07:59:55.165542+0000","last_became_peered":"2023-06-22T07:59:55.165542+0000","last_unstale":"2023-06-22T07:59:59.709134+0000","last_undegraded":"2023-06-22T07:59:59.709134+0000","last_fullsized":"2023-06-22T07:59:59.709134+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:32.147969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871894+0000","last_change":"2023-06-22T07:59:59.207794+0000","last_active":"2023-06-22T07:59:59.871894+0000","last_peered":"2023-06-22T07:59:59.871894+0000","last_clean":"2023-06-22T07:59:59.871894+0000","last_became_active":"2023-06-22T07:59:59.207364+0000","last_became_peered":"2023-06-22T07:59:59.207364+0000","last_unstale":"2023-06-22T07:59:59.871894+0000","last_undegraded":"2023-06-22T07:59:59.871894+0000","last_fullsized":"2023-06-22T07:59:59.871894+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:30.718491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4,0],"acting":[3,6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"27'8","reported_seq":29,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158928+0000","last_change":"2023-06-22T07:59:54.651423+0000","last_active":"2023-06-22T08:00:00.158928+0000","last_peered":"2023-06-22T08:00:00.158928+0000","last_clean":"2023-06-22T08:00:00.158928+0000","last_became_active":"2023-06-22T07:59:54.651167+0000","last_became_peered":"2023-06-22T07:59:54.651167+0000","last_unstale":"2023-06-22T08:00:00.158928+0000","last_undegraded":"2023-06-22T08:00:00.158928+0000","last_fullsized":"2023-06-22T08:00:00.158928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:31.141977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709649+0000","last_change":"2023-06-22T07:59:55.164478+0000","last_active":"2023-06-22T07:59:59.709649+0000","last_peered":"2023-06-22T07:59:59.709649+0000","last_clean":"2023-06-22T07:59:59.709649+0000","last_became_active":"2023-06-22T07:59:55.163922+0000","last_became_peered":"2023-06-22T07:59:55.163922+0000","last_unstale":"2023-06-22T07:59:59.709649+0000","last_undegraded":"2023-06-22T07:59:59.709649+0000","last_fullsized":"2023-06-22T07:59:59.709649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:43.772399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155085+0000","last_change":"2023-06-22T07:59:59.198496+0000","last_active":"2023-06-22T08:00:00.155085+0000","last_peered":"2023-06-22T08:00:00.155085+0000","last_clean":"2023-06-22T08:00:00.155085+0000","last_became_active":"2023-06-22T07:59:59.198067+0000","last_became_peered":"2023-06-22T07:59:59.198067+0000","last_unstale":"2023-06-22T08:00:00.155085+0000","last_undegraded":"2023-06-22T08:00:00.155085+0000","last_fullsized":"2023-06-22T08:00:00.155085+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:24.865757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,7],"acting":[1,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"27'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157653+0000","last_change":"2023-06-22T07:59:54.651366+0000","last_active":"2023-06-22T08:00:00.157653+0000","last_peered":"2023-06-22T08:00:00.157653+0000","last_clean":"2023-06-22T08:00:00.157653+0000","last_became_active":"2023-06-22T07:59:54.651059+0000","last_became_peered":"2023-06-22T07:59:54.651059+0000","last_unstale":"2023-06-22T08:00:00.157653+0000","last_undegraded":"2023-06-22T08:00:00.157653+0000","last_fullsized":"2023-06-22T08:00:00.157653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:37.183143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871131+0000","last_change":"2023-06-22T07:59:55.162820+0000","last_active":"2023-06-22T07:59:59.871131+0000","last_peered":"2023-06-22T07:59:59.871131+0000","last_clean":"2023-06-22T07:59:59.871131+0000","last_became_active":"2023-06-22T07:59:55.162505+0000","last_became_peered":"2023-06-22T07:59:55.162505+0000","last_unstale":"2023-06-22T07:59:59.871131+0000","last_undegraded":"2023-06-22T07:59:59.871131+0000","last_fullsized":"2023-06-22T07:59:59.871131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:46:38.312778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753744+0000","last_change":"2023-06-22T07:59:59.201059+0000","last_active":"2023-06-22T08:00:00.753744+0000","last_peered":"2023-06-22T08:00:00.753744+0000","last_clean":"2023-06-22T08:00:00.753744+0000","last_became_active":"2023-06-22T07:59:59.200460+0000","last_became_peered":"2023-06-22T07:59:59.200460+0000","last_unstale":"2023-06-22T08:00:00.753744+0000","last_undegraded":"2023-06-22T08:00:00.753744+0000","last_fullsized":"2023-06-22T08:00:00.753744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,0],"acting":[6,4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158090+0000","last_change":"2023-06-22T07:59:54.646017+0000","last_active":"2023-06-22T08:00:00.158090+0000","last_peered":"2023-06-22T08:00:00.158090+0000","last_clean":"2023-06-22T08:00:00.158090+0000","last_became_active":"2023-06-22T07:59:54.645554+0000","last_became_peered":"2023-06-22T07:59:54.645554+0000","last_unstale":"2023-06-22T08:00:00.158090+0000","last_undegraded":"2023-06-22T08:00:00.158090+0000","last_fullsized":"2023-06-22T08:00:00.158090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:01.550269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154806+0000","last_change":"2023-06-22T07:59:55.166283+0000","last_active":"2023-06-22T08:00:00.154806+0000","last_peered":"2023-06-22T08:00:00.154806+0000","last_clean":"2023-06-22T08:00:00.154806+0000","last_became_active":"2023-06-22T07:59:55.165977+0000","last_became_peered":"2023-06-22T07:59:55.165977+0000","last_unstale":"2023-06-22T08:00:00.154806+0000","last_undegraded":"2023-06-22T08:00:00.154806+0000","last_fullsized":"2023-06-22T08:00:00.154806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:21:29.484640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709141+0000","last_change":"2023-06-22T07:59:59.205411+0000","last_active":"2023-06-22T07:59:59.709141+0000","last_peered":"2023-06-22T07:59:59.709141+0000","last_clean":"2023-06-22T07:59:59.709141+0000","last_became_active":"2023-06-22T07:59:59.204980+0000","last_became_peered":"2023-06-22T07:59:59.204980+0000","last_unstale":"2023-06-22T07:59:59.709141+0000","last_undegraded":"2023-06-22T07:59:59.709141+0000","last_fullsized":"2023-06-22T07:59:59.709141+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:22.317952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871622+0000","last_change":"2023-06-22T07:59:54.654491+0000","last_active":"2023-06-22T07:59:59.871622+0000","last_peered":"2023-06-22T07:59:59.871622+0000","last_clean":"2023-06-22T07:59:59.871622+0000","last_became_active":"2023-06-22T07:59:54.654197+0000","last_became_peered":"2023-06-22T07:59:54.654197+0000","last_unstale":"2023-06-22T07:59:59.871622+0000","last_undegraded":"2023-06-22T07:59:59.871622+0000","last_fullsized":"2023-06-22T07:59:59.871622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:42:59.725455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708444+0000","last_change":"2023-06-22T07:59:55.172053+0000","last_active":"2023-06-22T07:59:59.708444+0000","last_peered":"2023-06-22T07:59:59.708444+0000","last_clean":"2023-06-22T07:59:59.708444+0000","last_became_active":"2023-06-22T07:59:55.165501+0000","last_became_peered":"2023-06-22T07:59:55.165501+0000","last_unstale":"2023-06-22T07:59:59.708444+0000","last_undegraded":"2023-06-22T07:59:59.708444+0000","last_fullsized":"2023-06-22T07:59:59.708444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753990+0000","last_change":"2023-06-22T07:59:59.199179+0000","last_active":"2023-06-22T08:00:00.753990+0000","last_peered":"2023-06-22T08:00:00.753990+0000","last_clean":"2023-06-22T08:00:00.753990+0000","last_became_active":"2023-06-22T07:59:59.198740+0000","last_became_peered":"2023-06-22T07:59:59.198740+0000","last_unstale":"2023-06-22T08:00:00.753990+0000","last_undegraded":"2023-06-22T08:00:00.753990+0000","last_fullsized":"2023-06-22T08:00:00.753990+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158086+0000","last_change":"2023-06-22T07:59:54.644520+0000","last_active":"2023-06-22T08:00:00.158086+0000","last_peered":"2023-06-22T08:00:00.158086+0000","last_clean":"2023-06-22T08:00:00.158086+0000","last_became_active":"2023-06-22T07:59:54.644204+0000","last_became_peered":"2023-06-22T07:59:54.644204+0000","last_unstale":"2023-06-22T08:00:00.158086+0000","last_undegraded":"2023-06-22T08:00:00.158086+0000","last_fullsized":"2023-06-22T08:00:00.158086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:07.674431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709657+0000","last_change":"2023-06-22T07:59:55.164111+0000","last_active":"2023-06-22T07:59:59.709657+0000","last_peered":"2023-06-22T07:59:59.709657+0000","last_clean":"2023-06-22T07:59:59.709657+0000","last_became_active":"2023-06-22T07:59:55.163802+0000","last_became_peered":"2023-06-22T07:59:55.163802+0000","last_unstale":"2023-06-22T07:59:59.709657+0000","last_undegraded":"2023-06-22T07:59:59.709657+0000","last_fullsized":"2023-06-22T07:59:59.709657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755252+0000","last_change":"2023-06-22T07:59:59.197721+0000","last_active":"2023-06-22T08:00:00.755252+0000","last_peered":"2023-06-22T08:00:00.755252+0000","last_clean":"2023-06-22T08:00:00.755252+0000","last_became_active":"2023-06-22T07:59:59.197259+0000","last_became_peered":"2023-06-22T07:59:59.197259+0000","last_unstale":"2023-06-22T08:00:00.755252+0000","last_undegraded":"2023-06-22T08:00:00.755252+0000","last_fullsized":"2023-06-22T08:00:00.755252+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5,1],"acting":[7,0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"27'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754436+0000","last_change":"2023-06-22T07:59:54.643124+0000","last_active":"2023-06-22T08:00:00.754436+0000","last_peered":"2023-06-22T08:00:00.754436+0000","last_clean":"2023-06-22T08:00:00.754436+0000","last_became_active":"2023-06-22T07:59:54.642833+0000","last_became_peered":"2023-06-22T07:59:54.642833+0000","last_unstale":"2023-06-22T08:00:00.754436+0000","last_undegraded":"2023-06-22T08:00:00.754436+0000","last_fullsized":"2023-06-22T08:00:00.754436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158064+0000","last_change":"2023-06-22T07:59:55.160551+0000","last_active":"2023-06-22T08:00:00.158064+0000","last_peered":"2023-06-22T08:00:00.158064+0000","last_clean":"2023-06-22T08:00:00.158064+0000","last_became_active":"2023-06-22T07:59:55.160221+0000","last_became_peered":"2023-06-22T07:59:55.160221+0000","last_unstale":"2023-06-22T08:00:00.158064+0000","last_undegraded":"2023-06-22T08:00:00.158064+0000","last_fullsized":"2023-06-22T08:00:00.158064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708974+0000","last_change":"2023-06-22T07:59:59.204643+0000","last_active":"2023-06-22T07:59:59.708974+0000","last_peered":"2023-06-22T07:59:59.708974+0000","last_clean":"2023-06-22T07:59:59.708974+0000","last_became_active":"2023-06-22T07:59:59.204182+0000","last_became_peered":"2023-06-22T07:59:59.204182+0000","last_unstale":"2023-06-22T07:59:59.708974+0000","last_undegraded":"2023-06-22T07:59:59.708974+0000","last_fullsized":"2023-06-22T07:59:59.708974+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4,2],"acting":[5,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871327+0000","last_change":"2023-06-22T07:59:54.654036+0000","last_active":"2023-06-22T07:59:59.871327+0000","last_peered":"2023-06-22T07:59:59.871327+0000","last_clean":"2023-06-22T07:59:59.871327+0000","last_became_active":"2023-06-22T07:59:54.653736+0000","last_became_peered":"2023-06-22T07:59:54.653736+0000","last_unstale":"2023-06-22T07:59:59.871327+0000","last_undegraded":"2023-06-22T07:59:59.871327+0000","last_fullsized":"2023-06-22T07:59:59.871327+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:44:09.331748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871999+0000","last_change":"2023-06-22T07:59:55.163181+0000","last_active":"2023-06-22T07:59:59.871999+0000","last_peered":"2023-06-22T07:59:59.871999+0000","last_clean":"2023-06-22T07:59:59.871999+0000","last_became_active":"2023-06-22T07:59:55.162860+0000","last_became_peered":"2023-06-22T07:59:55.162860+0000","last_unstale":"2023-06-22T07:59:59.871999+0000","last_undegraded":"2023-06-22T07:59:59.871999+0000","last_fullsized":"2023-06-22T07:59:59.871999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:59:25.037938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155510+0000","last_change":"2023-06-22T07:59:59.195710+0000","last_active":"2023-06-22T08:00:00.155510+0000","last_peered":"2023-06-22T08:00:00.155510+0000","last_clean":"2023-06-22T08:00:00.155510+0000","last_became_active":"2023-06-22T07:59:59.195246+0000","last_became_peered":"2023-06-22T07:59:59.195246+0000","last_unstale":"2023-06-22T08:00:00.155510+0000","last_undegraded":"2023-06-22T08:00:00.155510+0000","last_fullsized":"2023-06-22T08:00:00.155510+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:36.457493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,5],"acting":[1,0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871146+0000","last_change":"2023-06-22T07:59:54.649692+0000","last_active":"2023-06-22T07:59:59.871146+0000","last_peered":"2023-06-22T07:59:59.871146+0000","last_clean":"2023-06-22T07:59:59.871146+0000","last_became_active":"2023-06-22T07:59:54.649416+0000","last_became_peered":"2023-06-22T07:59:54.649416+0000","last_unstale":"2023-06-22T07:59:59.871146+0000","last_undegraded":"2023-06-22T07:59:59.871146+0000","last_fullsized":"2023-06-22T07:59:59.871146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:54:02.923433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708909+0000","last_change":"2023-06-22T07:59:55.175392+0000","last_active":"2023-06-22T07:59:59.708909+0000","last_peered":"2023-06-22T07:59:59.708909+0000","last_clean":"2023-06-22T07:59:59.708909+0000","last_became_active":"2023-06-22T07:59:55.175112+0000","last_became_peered":"2023-06-22T07:59:55.175112+0000","last_unstale":"2023-06-22T07:59:59.708909+0000","last_undegraded":"2023-06-22T07:59:59.708909+0000","last_fullsized":"2023-06-22T07:59:59.708909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871873+0000","last_change":"2023-06-22T07:59:59.206671+0000","last_active":"2023-06-22T07:59:59.871873+0000","last_peered":"2023-06-22T07:59:59.871873+0000","last_clean":"2023-06-22T07:59:59.871873+0000","last_became_active":"2023-06-22T07:59:59.206179+0000","last_became_peered":"2023-06-22T07:59:59.206179+0000","last_unstale":"2023-06-22T07:59:59.871873+0000","last_undegraded":"2023-06-22T07:59:59.871873+0000","last_fullsized":"2023-06-22T07:59:59.871873+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:00.427518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4,5],"acting":[3,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753414+0000","last_change":"2023-06-22T07:59:54.640508+0000","last_active":"2023-06-22T08:00:00.753414+0000","last_peered":"2023-06-22T08:00:00.753414+0000","last_clean":"2023-06-22T08:00:00.753414+0000","last_became_active":"2023-06-22T07:59:54.640108+0000","last_became_peered":"2023-06-22T07:59:54.640108+0000","last_unstale":"2023-06-22T08:00:00.753414+0000","last_undegraded":"2023-06-22T08:00:00.753414+0000","last_fullsized":"2023-06-22T08:00:00.753414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:19.193231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754822+0000","last_change":"2023-06-22T07:59:55.164398+0000","last_active":"2023-06-22T08:00:00.754822+0000","last_peered":"2023-06-22T08:00:00.754822+0000","last_clean":"2023-06-22T08:00:00.754822+0000","last_became_active":"2023-06-22T07:59:55.164086+0000","last_became_peered":"2023-06-22T07:59:55.164086+0000","last_unstale":"2023-06-22T08:00:00.754822+0000","last_undegraded":"2023-06-22T08:00:00.754822+0000","last_fullsized":"2023-06-22T08:00:00.754822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:38.189135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753997+0000","last_change":"2023-06-22T07:59:59.207752+0000","last_active":"2023-06-22T08:00:00.753997+0000","last_peered":"2023-06-22T08:00:00.753997+0000","last_clean":"2023-06-22T08:00:00.753997+0000","last_became_active":"2023-06-22T07:59:59.207360+0000","last_became_peered":"2023-06-22T07:59:59.207360+0000","last_unstale":"2023-06-22T08:00:00.753997+0000","last_undegraded":"2023-06-22T08:00:00.753997+0000","last_fullsized":"2023-06-22T08:00:00.753997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:08:25.633009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3,1],"acting":[4,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709114+0000","last_change":"2023-06-22T07:59:49.652772+0000","last_active":"2023-06-22T07:59:59.709114+0000","last_peered":"2023-06-22T07:59:59.709114+0000","last_clean":"2023-06-22T07:59:59.709114+0000","last_became_active":"2023-06-22T07:59:49.652487+0000","last_became_peered":"2023-06-22T07:59:49.652487+0000","last_unstale":"2023-06-22T07:59:59.709114+0000","last_undegraded":"2023-06-22T07:59:59.709114+0000","last_fullsized":"2023-06-22T07:59:59.709114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:11.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708571+0000","last_change":"2023-06-22T07:59:54.644538+0000","last_active":"2023-06-22T07:59:59.708571+0000","last_peered":"2023-06-22T07:59:59.708571+0000","last_clean":"2023-06-22T07:59:59.708571+0000","last_became_active":"2023-06-22T07:59:54.644253+0000","last_became_peered":"2023-06-22T07:59:54.644253+0000","last_unstale":"2023-06-22T07:59:59.708571+0000","last_undegraded":"2023-06-22T07:59:59.708571+0000","last_fullsized":"2023-06-22T07:59:59.708571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:09.128751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754228+0000","last_change":"2023-06-22T07:59:55.163802+0000","last_active":"2023-06-22T08:00:00.754228+0000","last_peered":"2023-06-22T08:00:00.754228+0000","last_clean":"2023-06-22T08:00:00.754228+0000","last_became_active":"2023-06-22T07:59:55.163481+0000","last_became_peered":"2023-06-22T07:59:55.163481+0000","last_unstale":"2023-06-22T08:00:00.754228+0000","last_undegraded":"2023-06-22T08:00:00.754228+0000","last_fullsized":"2023-06-22T08:00:00.754228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:16.916350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755246+0000","last_change":"2023-06-22T07:59:59.207344+0000","last_active":"2023-06-22T08:00:00.755246+0000","last_peered":"2023-06-22T08:00:00.755246+0000","last_clean":"2023-06-22T08:00:00.755246+0000","last_became_active":"2023-06-22T07:59:59.206975+0000","last_became_peered":"2023-06-22T07:59:59.206975+0000","last_unstale":"2023-06-22T08:00:00.755246+0000","last_undegraded":"2023-06-22T08:00:00.755246+0000","last_fullsized":"2023-06-22T08:00:00.755246+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3,1],"acting":[7,4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871513+0000","last_change":"2023-06-22T07:59:49.653053+0000","last_active":"2023-06-22T07:59:59.871513+0000","last_peered":"2023-06-22T07:59:59.871513+0000","last_clean":"2023-06-22T07:59:59.871513+0000","last_became_active":"2023-06-22T07:59:49.652861+0000","last_became_peered":"2023-06-22T07:59:49.652861+0000","last_unstale":"2023-06-22T07:59:59.871513+0000","last_undegraded":"2023-06-22T07:59:59.871513+0000","last_fullsized":"2023-06-22T07:59:59.871513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:28:40.388367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158202+0000","last_change":"2023-06-22T07:59:54.645403+0000","last_active":"2023-06-22T08:00:00.158202+0000","last_peered":"2023-06-22T08:00:00.158202+0000","last_clean":"2023-06-22T08:00:00.158202+0000","last_became_active":"2023-06-22T07:59:54.644973+0000","last_became_peered":"2023-06-22T07:59:54.644973+0000","last_unstale":"2023-06-22T08:00:00.158202+0000","last_undegraded":"2023-06-22T08:00:00.158202+0000","last_fullsized":"2023-06-22T08:00:00.158202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:15.910358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754101+0000","last_change":"2023-06-22T07:59:55.161484+0000","last_active":"2023-06-22T08:00:00.754101+0000","last_peered":"2023-06-22T08:00:00.754101+0000","last_clean":"2023-06-22T08:00:00.754101+0000","last_became_active":"2023-06-22T07:59:55.161194+0000","last_became_peered":"2023-06-22T07:59:55.161194+0000","last_unstale":"2023-06-22T08:00:00.754101+0000","last_undegraded":"2023-06-22T08:00:00.754101+0000","last_fullsized":"2023-06-22T08:00:00.754101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754988+0000","last_change":"2023-06-22T07:59:59.198241+0000","last_active":"2023-06-22T08:00:00.754988+0000","last_peered":"2023-06-22T08:00:00.754988+0000","last_clean":"2023-06-22T08:00:00.754988+0000","last_became_active":"2023-06-22T07:59:59.197689+0000","last_became_peered":"2023-06-22T07:59:59.197689+0000","last_unstale":"2023-06-22T08:00:00.754988+0000","last_undegraded":"2023-06-22T08:00:00.754988+0000","last_fullsized":"2023-06-22T08:00:00.754988+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:39.513874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4,1],"acting":[7,0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753886+0000","last_change":"2023-06-22T07:59:49.450044+0000","last_active":"2023-06-22T08:00:00.753886+0000","last_peered":"2023-06-22T08:00:00.753886+0000","last_clean":"2023-06-22T08:00:00.753886+0000","last_became_active":"2023-06-22T07:59:49.449731+0000","last_became_peered":"2023-06-22T07:59:49.449731+0000","last_unstale":"2023-06-22T08:00:00.753886+0000","last_undegraded":"2023-06-22T08:00:00.753886+0000","last_fullsized":"2023-06-22T08:00:00.753886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:40.262337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708613+0000","last_change":"2023-06-22T07:59:54.640462+0000","last_active":"2023-06-22T07:59:59.708613+0000","last_peered":"2023-06-22T07:59:59.708613+0000","last_clean":"2023-06-22T07:59:59.708613+0000","last_became_active":"2023-06-22T07:59:54.640123+0000","last_became_peered":"2023-06-22T07:59:54.640123+0000","last_unstale":"2023-06-22T07:59:59.708613+0000","last_undegraded":"2023-06-22T07:59:59.708613+0000","last_fullsized":"2023-06-22T07:59:59.708613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:20.316921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755350+0000","last_change":"2023-06-22T07:59:55.164651+0000","last_active":"2023-06-22T08:00:00.755350+0000","last_peered":"2023-06-22T08:00:00.755350+0000","last_clean":"2023-06-22T08:00:00.755350+0000","last_became_active":"2023-06-22T07:59:55.164235+0000","last_became_peered":"2023-06-22T07:59:55.164235+0000","last_unstale":"2023-06-22T08:00:00.755350+0000","last_undegraded":"2023-06-22T08:00:00.755350+0000","last_fullsized":"2023-06-22T08:00:00.755350+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:32.924136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157741+0000","last_change":"2023-06-22T07:59:59.204779+0000","last_active":"2023-06-22T08:00:00.157741+0000","last_peered":"2023-06-22T08:00:00.157741+0000","last_clean":"2023-06-22T08:00:00.157741+0000","last_became_active":"2023-06-22T07:59:59.204288+0000","last_became_peered":"2023-06-22T07:59:59.204288+0000","last_unstale":"2023-06-22T08:00:00.157741+0000","last_undegraded":"2023-06-22T08:00:00.157741+0000","last_fullsized":"2023-06-22T08:00:00.157741+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871497+0000","last_change":"2023-06-22T07:59:49.657008+0000","last_active":"2023-06-22T07:59:59.871497+0000","last_peered":"2023-06-22T07:59:59.871497+0000","last_clean":"2023-06-22T07:59:59.871497+0000","last_became_active":"2023-06-22T07:59:49.656752+0000","last_became_peered":"2023-06-22T07:59:49.656752+0000","last_unstale":"2023-06-22T07:59:59.871497+0000","last_undegraded":"2023-06-22T07:59:59.871497+0000","last_fullsized":"2023-06-22T07:59:59.871497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:50:37.639134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":185,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155134+0000","last_change":"2023-06-22T07:59:42.501363+0000","last_active":"2023-06-22T08:00:00.155134+0000","last_peered":"2023-06-22T08:00:00.155134+0000","last_clean":"2023-06-22T08:00:00.155134+0000","last_became_active":"2023-06-22T07:59:42.500320+0000","last_became_peered":"2023-06-22T07:59:42.500320+0000","last_unstale":"2023-06-22T08:00:00.155134+0000","last_undegraded":"2023-06-22T08:00:00.155134+0000","last_fullsized":"2023-06-22T08:00:00.155134+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_clean_scrub_stamp":"2023-06-22T07:59:41.381475+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:08:53.784030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753785+0000","last_change":"2023-06-22T07:59:54.645031+0000","last_active":"2023-06-22T08:00:00.753785+0000","last_peered":"2023-06-22T08:00:00.753785+0000","last_clean":"2023-06-22T08:00:00.753785+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:00.753785+0000","last_undegraded":"2023-06-22T08:00:00.753785+0000","last_fullsized":"2023-06-22T08:00:00.753785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:42.766407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872074+0000","last_change":"2023-06-22T07:59:55.162799+0000","last_active":"2023-06-22T07:59:59.872074+0000","last_peered":"2023-06-22T07:59:59.872074+0000","last_clean":"2023-06-22T07:59:59.872074+0000","last_became_active":"2023-06-22T07:59:55.162501+0000","last_became_peered":"2023-06-22T07:59:55.162501+0000","last_unstale":"2023-06-22T07:59:59.872074+0000","last_undegraded":"2023-06-22T07:59:59.872074+0000","last_fullsized":"2023-06-22T07:59:59.872074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:57:14.305897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871805+0000","last_change":"2023-06-22T07:59:59.205097+0000","last_active":"2023-06-22T07:59:59.871805+0000","last_peered":"2023-06-22T07:59:59.871805+0000","last_clean":"2023-06-22T07:59:59.871805+0000","last_became_active":"2023-06-22T07:59:59.201465+0000","last_became_peered":"2023-06-22T07:59:59.201465+0000","last_unstale":"2023-06-22T07:59:59.871805+0000","last_undegraded":"2023-06-22T07:59:59.871805+0000","last_fullsized":"2023-06-22T07:59:59.871805+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:17:45.352998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,0],"acting":[3,7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871056+0000","last_change":"2023-06-22T07:59:49.442558+0000","last_active":"2023-06-22T07:59:59.871056+0000","last_peered":"2023-06-22T07:59:59.871056+0000","last_clean":"2023-06-22T07:59:59.871056+0000","last_became_active":"2023-06-22T07:59:49.442154+0000","last_became_peered":"2023-06-22T07:59:49.442154+0000","last_unstale":"2023-06-22T07:59:59.871056+0000","last_undegraded":"2023-06-22T07:59:59.871056+0000","last_fullsized":"2023-06-22T07:59:59.871056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:46:16.233204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753311+0000","last_change":"2023-06-22T07:59:54.641746+0000","last_active":"2023-06-22T08:00:00.753311+0000","last_peered":"2023-06-22T08:00:00.753311+0000","last_clean":"2023-06-22T08:00:00.753311+0000","last_became_active":"2023-06-22T07:59:54.641431+0000","last_became_peered":"2023-06-22T07:59:54.641431+0000","last_unstale":"2023-06-22T08:00:00.753311+0000","last_undegraded":"2023-06-22T08:00:00.753311+0000","last_fullsized":"2023-06-22T08:00:00.753311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709468+0000","last_change":"2023-06-22T07:59:55.163214+0000","last_active":"2023-06-22T07:59:59.709468+0000","last_peered":"2023-06-22T07:59:59.709468+0000","last_clean":"2023-06-22T07:59:59.709468+0000","last_became_active":"2023-06-22T07:59:55.162907+0000","last_became_peered":"2023-06-22T07:59:55.162907+0000","last_unstale":"2023-06-22T07:59:59.709468+0000","last_undegraded":"2023-06-22T07:59:59.709468+0000","last_fullsized":"2023-06-22T07:59:59.709468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:46.988703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157450+0000","last_change":"2023-06-22T07:59:59.198945+0000","last_active":"2023-06-22T08:00:00.157450+0000","last_peered":"2023-06-22T08:00:00.157450+0000","last_clean":"2023-06-22T08:00:00.157450+0000","last_became_active":"2023-06-22T07:59:59.198561+0000","last_became_peered":"2023-06-22T07:59:59.198561+0000","last_unstale":"2023-06-22T08:00:00.157450+0000","last_undegraded":"2023-06-22T08:00:00.157450+0000","last_fullsized":"2023-06-22T08:00:00.157450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:06.135651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2,5],"acting":[0,7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871842+0000","last_change":"2023-06-22T07:59:49.442246+0000","last_active":"2023-06-22T07:59:59.871842+0000","last_peered":"2023-06-22T07:59:59.871842+0000","last_clean":"2023-06-22T07:59:59.871842+0000","last_became_active":"2023-06-22T07:59:49.442001+0000","last_became_peered":"2023-06-22T07:59:49.442001+0000","last_unstale":"2023-06-22T07:59:59.871842+0000","last_undegraded":"2023-06-22T07:59:59.871842+0000","last_fullsized":"2023-06-22T07:59:59.871842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:27:24.457978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754254+0000","last_change":"2023-06-22T07:59:54.643272+0000","last_active":"2023-06-22T08:00:00.754254+0000","last_peered":"2023-06-22T08:00:00.754254+0000","last_clean":"2023-06-22T08:00:00.754254+0000","last_became_active":"2023-06-22T07:59:54.642925+0000","last_became_peered":"2023-06-22T07:59:54.642925+0000","last_unstale":"2023-06-22T08:00:00.754254+0000","last_undegraded":"2023-06-22T08:00:00.754254+0000","last_fullsized":"2023-06-22T08:00:00.754254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:13.625768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754231+0000","last_change":"2023-06-22T07:59:55.167307+0000","last_active":"2023-06-22T08:00:00.754231+0000","last_peered":"2023-06-22T08:00:00.754231+0000","last_clean":"2023-06-22T08:00:00.754231+0000","last_became_active":"2023-06-22T07:59:55.167026+0000","last_became_peered":"2023-06-22T07:59:55.167026+0000","last_unstale":"2023-06-22T08:00:00.754231+0000","last_undegraded":"2023-06-22T08:00:00.754231+0000","last_fullsized":"2023-06-22T08:00:00.754231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871379+0000","last_change":"2023-06-22T07:59:59.208263+0000","last_active":"2023-06-22T07:59:59.871379+0000","last_peered":"2023-06-22T07:59:59.871379+0000","last_clean":"2023-06-22T07:59:59.871379+0000","last_became_active":"2023-06-22T07:59:59.207691+0000","last_became_peered":"2023-06-22T07:59:59.207691+0000","last_unstale":"2023-06-22T07:59:59.871379+0000","last_undegraded":"2023-06-22T07:59:59.871379+0000","last_fullsized":"2023-06-22T07:59:59.871379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:37:57.170857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,2],"acting":[3,7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755549+0000","last_change":"2023-06-22T07:59:54.644570+0000","last_active":"2023-06-22T08:00:00.755549+0000","last_peered":"2023-06-22T08:00:00.755549+0000","last_clean":"2023-06-22T08:00:00.755549+0000","last_became_active":"2023-06-22T07:59:54.644218+0000","last_became_peered":"2023-06-22T07:59:54.644218+0000","last_unstale":"2023-06-22T08:00:00.755549+0000","last_undegraded":"2023-06-22T08:00:00.755549+0000","last_fullsized":"2023-06-22T08:00:00.755549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:05:00.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754571+0000","last_change":"2023-06-22T07:59:49.655601+0000","last_active":"2023-06-22T08:00:00.754571+0000","last_peered":"2023-06-22T08:00:00.754571+0000","last_clean":"2023-06-22T08:00:00.754571+0000","last_became_active":"2023-06-22T07:59:49.655328+0000","last_became_peered":"2023-06-22T07:59:49.655328+0000","last_unstale":"2023-06-22T08:00:00.754571+0000","last_undegraded":"2023-06-22T08:00:00.754571+0000","last_fullsized":"2023-06-22T08:00:00.754571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:29:09.967964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755018+0000","last_change":"2023-06-22T07:59:55.163241+0000","last_active":"2023-06-22T08:00:00.755018+0000","last_peered":"2023-06-22T08:00:00.755018+0000","last_clean":"2023-06-22T08:00:00.755018+0000","last_became_active":"2023-06-22T07:59:55.162949+0000","last_became_peered":"2023-06-22T07:59:55.162949+0000","last_unstale":"2023-06-22T08:00:00.755018+0000","last_undegraded":"2023-06-22T08:00:00.755018+0000","last_fullsized":"2023-06-22T08:00:00.755018+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:51.267726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755485+0000","last_change":"2023-06-22T07:59:59.201187+0000","last_active":"2023-06-22T08:00:00.755485+0000","last_peered":"2023-06-22T08:00:00.755485+0000","last_clean":"2023-06-22T08:00:00.755485+0000","last_became_active":"2023-06-22T07:59:59.200688+0000","last_became_peered":"2023-06-22T07:59:59.200688+0000","last_unstale":"2023-06-22T08:00:00.755485+0000","last_undegraded":"2023-06-22T08:00:00.755485+0000","last_fullsized":"2023-06-22T08:00:00.755485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:52:15.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2,6],"acting":[4,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871926+0000","last_change":"2023-06-22T07:59:49.655839+0000","last_active":"2023-06-22T07:59:59.871926+0000","last_peered":"2023-06-22T07:59:59.871926+0000","last_clean":"2023-06-22T07:59:59.871926+0000","last_became_active":"2023-06-22T07:59:49.655494+0000","last_became_peered":"2023-06-22T07:59:49.655494+0000","last_unstale":"2023-06-22T07:59:59.871926+0000","last_undegraded":"2023-06-22T07:59:59.871926+0000","last_fullsized":"2023-06-22T07:59:59.871926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:01:23.950347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754455+0000","last_change":"2023-06-22T07:59:54.641464+0000","last_active":"2023-06-22T08:00:00.754455+0000","last_peered":"2023-06-22T08:00:00.754455+0000","last_clean":"2023-06-22T08:00:00.754455+0000","last_became_active":"2023-06-22T07:59:54.641161+0000","last_became_peered":"2023-06-22T07:59:54.641161+0000","last_unstale":"2023-06-22T08:00:00.754455+0000","last_undegraded":"2023-06-22T08:00:00.754455+0000","last_fullsized":"2023-06-22T08:00:00.754455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:46.346592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708316+0000","last_change":"2023-06-22T07:59:55.164629+0000","last_active":"2023-06-22T07:59:59.708316+0000","last_peered":"2023-06-22T07:59:59.708316+0000","last_clean":"2023-06-22T07:59:59.708316+0000","last_became_active":"2023-06-22T07:59:55.164329+0000","last_became_peered":"2023-06-22T07:59:55.164329+0000","last_unstale":"2023-06-22T07:59:59.708316+0000","last_undegraded":"2023-06-22T07:59:59.708316+0000","last_fullsized":"2023-06-22T07:59:59.708316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754950+0000","last_change":"2023-06-22T07:59:59.201778+0000","last_active":"2023-06-22T08:00:00.754950+0000","last_peered":"2023-06-22T08:00:00.754950+0000","last_clean":"2023-06-22T08:00:00.754950+0000","last_became_active":"2023-06-22T07:59:59.201326+0000","last_became_peered":"2023-06-22T07:59:59.201326+0000","last_unstale":"2023-06-22T08:00:00.754950+0000","last_undegraded":"2023-06-22T08:00:00.754950+0000","last_fullsized":"2023-06-22T08:00:00.754950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:17:31.817024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"27'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754232+0000","last_change":"2023-06-22T07:59:54.644816+0000","last_active":"2023-06-22T08:00:00.754232+0000","last_peered":"2023-06-22T08:00:00.754232+0000","last_clean":"2023-06-22T08:00:00.754232+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:00.754232+0000","last_undegraded":"2023-06-22T08:00:00.754232+0000","last_fullsized":"2023-06-22T08:00:00.754232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:33.819054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753735+0000","last_change":"2023-06-22T07:59:55.167979+0000","last_active":"2023-06-22T08:00:00.753735+0000","last_peered":"2023-06-22T08:00:00.753735+0000","last_clean":"2023-06-22T08:00:00.753735+0000","last_became_active":"2023-06-22T07:59:55.163775+0000","last_became_peered":"2023-06-22T07:59:55.163775+0000","last_unstale":"2023-06-22T08:00:00.753735+0000","last_undegraded":"2023-06-22T08:00:00.753735+0000","last_fullsized":"2023-06-22T08:00:00.753735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155135+0000","last_change":"2023-06-22T07:59:59.194581+0000","last_active":"2023-06-22T08:00:00.155135+0000","last_peered":"2023-06-22T08:00:00.155135+0000","last_clean":"2023-06-22T08:00:00.155135+0000","last_became_active":"2023-06-22T07:59:59.194135+0000","last_became_peered":"2023-06-22T07:59:59.194135+0000","last_unstale":"2023-06-22T08:00:00.155135+0000","last_undegraded":"2023-06-22T08:00:00.155135+0000","last_fullsized":"2023-06-22T08:00:00.155135+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"27'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.156863+0000","last_change":"2023-06-22T07:59:54.651138+0000","last_active":"2023-06-22T08:00:00.156863+0000","last_peered":"2023-06-22T08:00:00.156863+0000","last_clean":"2023-06-22T08:00:00.156863+0000","last_became_active":"2023-06-22T07:59:54.650803+0000","last_became_peered":"2023-06-22T07:59:54.650803+0000","last_unstale":"2023-06-22T08:00:00.156863+0000","last_undegraded":"2023-06-22T08:00:00.156863+0000","last_fullsized":"2023-06-22T08:00:00.156863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:45.982711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708617+0000","last_change":"2023-06-22T07:59:55.165814+0000","last_active":"2023-06-22T07:59:59.708617+0000","last_peered":"2023-06-22T07:59:59.708617+0000","last_clean":"2023-06-22T07:59:59.708617+0000","last_became_active":"2023-06-22T07:59:55.165270+0000","last_became_peered":"2023-06-22T07:59:55.165270+0000","last_unstale":"2023-06-22T07:59:59.708617+0000","last_undegraded":"2023-06-22T07:59:59.708617+0000","last_fullsized":"2023-06-22T07:59:59.708617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708465+0000","last_change":"2023-06-22T07:59:59.212042+0000","last_active":"2023-06-22T07:59:59.708465+0000","last_peered":"2023-06-22T07:59:59.708465+0000","last_clean":"2023-06-22T07:59:59.708465+0000","last_became_active":"2023-06-22T07:59:59.211666+0000","last_became_peered":"2023-06-22T07:59:59.211666+0000","last_unstale":"2023-06-22T07:59:59.708465+0000","last_undegraded":"2023-06-22T07:59:59.708465+0000","last_fullsized":"2023-06-22T07:59:59.708465+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,4],"acting":[2,7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158226+0000","last_change":"2023-06-22T07:59:54.650251+0000","last_active":"2023-06-22T08:00:00.158226+0000","last_peered":"2023-06-22T08:00:00.158226+0000","last_clean":"2023-06-22T08:00:00.158226+0000","last_became_active":"2023-06-22T07:59:54.649993+0000","last_became_peered":"2023-06-22T07:59:54.649993+0000","last_unstale":"2023-06-22T08:00:00.158226+0000","last_undegraded":"2023-06-22T08:00:00.158226+0000","last_fullsized":"2023-06-22T08:00:00.158226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:20:20.733896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708375+0000","last_change":"2023-06-22T07:59:55.172595+0000","last_active":"2023-06-22T07:59:59.708375+0000","last_peered":"2023-06-22T07:59:59.708375+0000","last_clean":"2023-06-22T07:59:59.708375+0000","last_became_active":"2023-06-22T07:59:55.165873+0000","last_became_peered":"2023-06-22T07:59:55.165873+0000","last_unstale":"2023-06-22T07:59:59.708375+0000","last_undegraded":"2023-06-22T07:59:59.708375+0000","last_fullsized":"2023-06-22T07:59:59.708375+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:02.592807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154969+0000","last_change":"2023-06-22T07:59:59.192708+0000","last_active":"2023-06-22T08:00:00.154969+0000","last_peered":"2023-06-22T08:00:00.154969+0000","last_clean":"2023-06-22T08:00:00.154969+0000","last_became_active":"2023-06-22T07:59:59.191693+0000","last_became_peered":"2023-06-22T07:59:59.191693+0000","last_unstale":"2023-06-22T08:00:00.154969+0000","last_undegraded":"2023-06-22T08:00:00.154969+0000","last_fullsized":"2023-06-22T08:00:00.154969+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T12:59:33.454962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5,7],"acting":[1,0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709255+0000","last_change":"2023-06-22T07:59:54.651179+0000","last_active":"2023-06-22T07:59:59.709255+0000","last_peered":"2023-06-22T07:59:59.709255+0000","last_clean":"2023-06-22T07:59:59.709255+0000","last_became_active":"2023-06-22T07:59:54.650852+0000","last_became_peered":"2023-06-22T07:59:54.650852+0000","last_unstale":"2023-06-22T07:59:59.709255+0000","last_undegraded":"2023-06-22T07:59:59.709255+0000","last_fullsized":"2023-06-22T07:59:59.709255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:01.351861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754043+0000","last_change":"2023-06-22T07:59:55.163050+0000","last_active":"2023-06-22T08:00:00.754043+0000","last_peered":"2023-06-22T08:00:00.754043+0000","last_clean":"2023-06-22T08:00:00.754043+0000","last_became_active":"2023-06-22T07:59:55.162754+0000","last_became_peered":"2023-06-22T07:59:55.162754+0000","last_unstale":"2023-06-22T08:00:00.754043+0000","last_undegraded":"2023-06-22T08:00:00.754043+0000","last_fullsized":"2023-06-22T08:00:00.754043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754797+0000","last_change":"2023-06-22T07:59:59.209558+0000","last_active":"2023-06-22T08:00:00.754797+0000","last_peered":"2023-06-22T08:00:00.754797+0000","last_clean":"2023-06-22T08:00:00.754797+0000","last_became_active":"2023-06-22T07:59:59.209129+0000","last_became_peered":"2023-06-22T07:59:59.209129+0000","last_unstale":"2023-06-22T08:00:00.754797+0000","last_undegraded":"2023-06-22T08:00:00.754797+0000","last_fullsized":"2023-06-22T08:00:00.754797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:21.581063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1,0],"acting":[4,3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154714+0000","last_change":"2023-06-22T07:59:54.652059+0000","last_active":"2023-06-22T08:00:00.154714+0000","last_peered":"2023-06-22T08:00:00.154714+0000","last_clean":"2023-06-22T08:00:00.154714+0000","last_became_active":"2023-06-22T07:59:54.651720+0000","last_became_peered":"2023-06-22T07:59:54.651720+0000","last_unstale":"2023-06-22T08:00:00.154714+0000","last_undegraded":"2023-06-22T08:00:00.154714+0000","last_fullsized":"2023-06-22T08:00:00.154714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:31.908657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708158+0000","last_change":"2023-06-22T07:59:55.172794+0000","last_active":"2023-06-22T07:59:59.708158+0000","last_peered":"2023-06-22T07:59:59.708158+0000","last_clean":"2023-06-22T07:59:59.708158+0000","last_became_active":"2023-06-22T07:59:55.172527+0000","last_became_peered":"2023-06-22T07:59:55.172527+0000","last_unstale":"2023-06-22T07:59:59.708158+0000","last_undegraded":"2023-06-22T07:59:59.708158+0000","last_fullsized":"2023-06-22T07:59:59.708158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155457+0000","last_change":"2023-06-22T07:59:55.165758+0000","last_active":"2023-06-22T08:00:00.155457+0000","last_peered":"2023-06-22T08:00:00.155457+0000","last_clean":"2023-06-22T08:00:00.155457+0000","last_became_active":"2023-06-22T07:59:55.165117+0000","last_became_peered":"2023-06-22T07:59:55.165117+0000","last_unstale":"2023-06-22T08:00:00.155457+0000","last_undegraded":"2023-06-22T08:00:00.155457+0000","last_fullsized":"2023-06-22T08:00:00.155457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:02.357853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753486+0000","last_change":"2023-06-22T07:59:55.162439+0000","last_active":"2023-06-22T08:00:00.753486+0000","last_peered":"2023-06-22T08:00:00.753486+0000","last_clean":"2023-06-22T08:00:00.753486+0000","last_became_active":"2023-06-22T07:59:55.162072+0000","last_became_peered":"2023-06-22T07:59:55.162072+0000","last_unstale":"2023-06-22T08:00:00.753486+0000","last_undegraded":"2023-06-22T08:00:00.753486+0000","last_fullsized":"2023-06-22T08:00:00.753486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157229+0000","last_change":"2023-06-22T07:59:55.159299+0000","last_active":"2023-06-22T08:00:00.157229+0000","last_peered":"2023-06-22T08:00:00.157229+0000","last_clean":"2023-06-22T08:00:00.157229+0000","last_became_active":"2023-06-22T07:59:55.158893+0000","last_became_peered":"2023-06-22T07:59:55.158893+0000","last_unstale":"2023-06-22T08:00:00.157229+0000","last_undegraded":"2023-06-22T08:00:00.157229+0000","last_fullsized":"2023-06-22T08:00:00.157229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709716+0000","last_change":"2023-06-22T07:59:55.162520+0000","last_active":"2023-06-22T07:59:59.709716+0000","last_peered":"2023-06-22T07:59:59.709716+0000","last_clean":"2023-06-22T07:59:59.709716+0000","last_became_active":"2023-06-22T07:59:55.162211+0000","last_became_peered":"2023-06-22T07:59:55.162211+0000","last_unstale":"2023-06-22T07:59:59.709716+0000","last_undegraded":"2023-06-22T07:59:59.709716+0000","last_fullsized":"2023-06-22T07:59:59.709716+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:13:00.662547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708947+0000","last_change":"2023-06-22T07:59:55.165364+0000","last_active":"2023-06-22T07:59:59.708947+0000","last_peered":"2023-06-22T07:59:59.708947+0000","last_clean":"2023-06-22T07:59:59.708947+0000","last_became_active":"2023-06-22T07:59:55.165021+0000","last_became_peered":"2023-06-22T07:59:55.165021+0000","last_unstale":"2023-06-22T07:59:59.708947+0000","last_undegraded":"2023-06-22T07:59:59.708947+0000","last_fullsized":"2023-06-22T07:59:59.708947+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:09:07.008945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4592,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476747,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":127352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":319488,"data_stored":127456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":299008,"data_stored":123558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":905216,"data_stored":714600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":897024,"data_stored":714508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":303104,"data_stored":123648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]}]},{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":299008,"data_stored":123558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542155,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":123704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-22T08:00:24.121 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-22T08:00:24.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.348+0000 7fd4eabff700 1 Processor -- start 2023-06-22T08:00:24.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.348+0000 7fd4eabff700 1 -- start start 2023-06-22T08:00:24.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.350+0000 7fd4eabff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd4e4152a10 0x7fd4e40c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.350+0000 7fd4eabff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e40c9200 0x7fd4e40cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.350+0000 7fd4eabff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e4159730 con 0x7fd4e40cb7a0 2023-06-22T08:00:24.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.350+0000 7fd4eabff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e40c5470 con 0x7fd4e4152a10 2023-06-22T08:00:24.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.350+0000 7fd4eabff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40cbbd0 con 0x7fd4e40c9200 2023-06-22T08:00:24.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.350+0000 7fd4e899b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd4e4152a10 0x7fd4e40c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.350+0000 7fd4e919c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fd4e40cb7a0 0x7fd4e415c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40052/0 (socket says 172.21.15.23:40052) 2023-06-22T08:00:24.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.350+0000 7fd4e919c700 1 -- 172.21.15.23:0/652337086 learned_addr learned my addr 172.21.15.23:0/652337086 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:24.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.350+0000 7fd4e3fff700 1 --2- 172.21.15.23:0/652337086 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e40c9200 0x7fd4e40cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.351+0000 7fd4e899b700 1 -- 172.21.15.23:0/652337086 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e40c9200 msgr2=0x7fd4e40cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:24.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.351+0000 7fd4e899b700 1 --2- 172.21.15.23:0/652337086 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e40c9200 0x7fd4e40cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.351+0000 7fd4e899b700 1 -- 172.21.15.23:0/652337086 >> v1:172.21.15.23:6789/0 conn(0x7fd4e40cb7a0 legacy=0x7fd4e415c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.351+0000 7fd4e899b700 1 -- 172.21.15.23:0/652337086 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e4158910 con 0x7fd4e4152a10 2023-06-22T08:00:24.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.351+0000 7fd4e37fe700 1 -- 172.21.15.23:0/652337086 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323223563 0 0) 0x7fd4e4159730 con 0x7fd4e40cb7a0 2023-06-22T08:00:24.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.351+0000 7fd4e899b700 1 --2- 172.21.15.23:0/652337086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e4152a10 0x7fd4e40c4a30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009840 tx=0x7fd4d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216cfb606e236f55 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:24.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.351+0000 7fd4e37fe700 1 -- 172.21.15.23:0/652337086 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8011830 con 0x7fd4e4152a10 2023-06-22T08:00:24.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.351+0000 7fd4e37fe700 1 -- 172.21.15.23:0/652337086 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d8011990 con 0x7fd4e4152a10 2023-06-22T08:00:24.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.351+0000 7fd4e37fe700 1 -- 172.21.15.23:0/652337086 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8011d20 con 0x7fd4e4152a10 2023-06-22T08:00:24.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.352+0000 7fd4eabff700 1 -- 172.21.15.23:0/652337086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e4152a10 msgr2=0x7fd4e40c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.352+0000 7fd4eabff700 1 --2- 172.21.15.23:0/652337086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e4152a10 0x7fd4e40c4a30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009840 tx=0x7fd4d8009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.352+0000 7fd4eabff700 1 -- 172.21.15.23:0/652337086 shutdown_connections 2023-06-22T08:00:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.352+0000 7fd4eabff700 1 --2- 172.21.15.23:0/652337086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e4152a10 0x7fd4e40c4a30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.352+0000 7fd4eabff700 1 --2- 172.21.15.23:0/652337086 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e40c9200 0x7fd4e40cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.352+0000 7fd4eabff700 1 -- 172.21.15.23:0/652337086 >> 172.21.15.23:0/652337086 conn(0x7fd4e4065b90 msgr2=0x7fd4e4065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.352+0000 7fd4eabff700 1 -- 172.21.15.23:0/652337086 shutdown_connections 2023-06-22T08:00:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.352+0000 7fd4eabff700 1 -- 172.21.15.23:0/652337086 wait complete. 2023-06-22T08:00:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.354+0000 7fd4eabff700 1 Processor -- start 2023-06-22T08:00:24.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.354+0000 7fd4eabff700 1 -- start start 2023-06-22T08:00:24.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4eabff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e40c9200 0x7fd4e4154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4eabff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e40cb7a0 0x7fd4e40b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4eabff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e4152a10 0x7fd4e41550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4eabff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40c8660 con 0x7fd4e40c9200 2023-06-22T08:00:24.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4eabff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40b4710 con 0x7fd4e40cb7a0 2023-06-22T08:00:24.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4eabff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40c7a90 con 0x7fd4e4152a10 2023-06-22T08:00:24.362 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e899b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e40c9200 0x7fd4e4154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e899b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e40c9200 0x7fd4e4154950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38762/0 (socket says 172.21.15.23:38762) 2023-06-22T08:00:24.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e899b700 1 -- 172.21.15.23:0/1526825076 learned_addr learned my addr 172.21.15.23:0/1526825076 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:24.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e3fff700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e40cb7a0 0x7fd4e40b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e919c700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e4152a10 0x7fd4e41550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e899b700 1 -- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e4152a10 msgr2=0x7fd4e41550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e899b700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e4152a10 0x7fd4e41550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e899b700 1 -- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e40cb7a0 msgr2=0x7fd4e40b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e899b700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e40cb7a0 0x7fd4e40b32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e899b700 1 -- 172.21.15.23:0/1526825076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d80094a0 con 0x7fd4e40c9200 2023-06-22T08:00:24.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.355+0000 7fd4e919c700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e4152a10 0x7fd4e41550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:24.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.356+0000 7fd4e899b700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e40c9200 0x7fd4e4154950 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009fd0 tx=0x7fd4d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:24.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.356+0000 7fd4e17fa700 1 -- 172.21.15.23:0/1526825076 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80036a0 con 0x7fd4e40c9200 2023-06-22T08:00:24.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.356+0000 7fd4e17fa700 1 -- 172.21.15.23:0/1526825076 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d801e070 con 0x7fd4e40c9200 2023-06-22T08:00:24.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.356+0000 7fd4e17fa700 1 -- 172.21.15.23:0/1526825076 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8003a40 con 0x7fd4e40c9200 2023-06-22T08:00:24.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.356+0000 7fd4eabff700 1 -- 172.21.15.23:0/1526825076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e40c7d30 con 0x7fd4e40c9200 2023-06-22T08:00:24.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.356+0000 7fd4eabff700 1 -- 172.21.15.23:0/1526825076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e40c82f0 con 0x7fd4e40c9200 2023-06-22T08:00:24.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.357+0000 7fd4eabff700 1 -- 172.21.15.23:0/1526825076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e404ea20 con 0x7fd4e40c9200 2023-06-22T08:00:24.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.358+0000 7fd4e17fa700 1 -- 172.21.15.23:0/1526825076 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd4d8003ba0 con 0x7fd4e40c9200 2023-06-22T08:00:24.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.360+0000 7fd4e17fa700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c806ace0 0x7fd4c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.360+0000 7fd4e17fa700 1 -- 172.21.15.23:0/1526825076 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd4d808a800 con 0x7fd4e40c9200 2023-06-22T08:00:24.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.360+0000 7fd4e3fff700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c806ace0 0x7fd4c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.361+0000 7fd4e3fff700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c806ace0 0x7fd4c806d190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd4e415b690 tx=0x7fd4d0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:24.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.365+0000 7fd4e17fa700 1 -- 172.21.15.23:0/1526825076 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd4d80555b0 con 0x7fd4e40c9200 2023-06-22T08:00:24.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.568+0000 7fd4eabff700 1 -- 172.21.15.23:0/1526825076 --> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd4e4158e90 con 0x7fd4c806ace0 2023-06-22T08:00:24.583 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.580+0000 7fd4e17fa700 1 -- 172.21.15.23:0/1526825076 <== mgr.4100 v2:172.21.15.67:6832/3180103139 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367395 (secure 0 0 0) 0x7fd4d0011860 con 0x7fd4c806ace0 2023-06-22T08:00:24.587 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:24.588 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 -- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c806ace0 msgr2=0x7fd4c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c806ace0 0x7fd4c806d190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd4e415b690 tx=0x7fd4d0009210 comp rx=0 tx=0).stop 2023-06-22T08:00:24.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 -- 172.21.15.23:0/1526825076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e40c9200 msgr2=0x7fd4e4154950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e40c9200 0x7fd4e4154950 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009fd0 tx=0x7fd4d8004750 comp rx=0 tx=0).stop 2023-06-22T08:00:24.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 -- 172.21.15.23:0/1526825076 shutdown_connections 2023-06-22T08:00:24.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e4152a10 0x7fd4e41550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e40c9200 0x7fd4e4154950 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c806ace0 0x7fd4c806d190 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 --2- 172.21.15.23:0/1526825076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e40cb7a0 0x7fd4e40b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 -- 172.21.15.23:0/1526825076 >> 172.21.15.23:0/1526825076 conn(0x7fd4e4065b90 msgr2=0x7fd4e4065f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:24.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 -- 172.21.15.23:0/1526825076 shutdown_connections 2023-06-22T08:00:24.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.585+0000 7fd4eabff700 1 -- 172.21.15.23:0/1526825076 wait complete. 2023-06-22T08:00:24.590 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-06-22T08:00:24.606 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2023-06-22T08:00:24.561289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594960,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219176,"kb_used_data":3560,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755544,"statfs":{"total":773094113280,"available":772869677056,"internally_reserved":0,"allocated":3645440,"data_stored":2178384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003753"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709823+0000","last_change":"2023-06-22T07:59:55.165740+0000","last_active":"2023-06-22T07:59:59.709823+0000","last_peered":"2023-06-22T07:59:59.709823+0000","last_clean":"2023-06-22T07:59:59.709823+0000","last_became_active":"2023-06-22T07:59:55.165398+0000","last_became_peered":"2023-06-22T07:59:55.165398+0000","last_unstale":"2023-06-22T07:59:59.709823+0000","last_undegraded":"2023-06-22T07:59:59.709823+0000","last_fullsized":"2023-06-22T07:59:59.709823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:37.596656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709026+0000","last_change":"2023-06-22T07:59:55.174603+0000","last_active":"2023-06-22T07:59:59.709026+0000","last_peered":"2023-06-22T07:59:59.709026+0000","last_clean":"2023-06-22T07:59:59.709026+0000","last_became_active":"2023-06-22T07:59:55.174346+0000","last_became_peered":"2023-06-22T07:59:55.174346+0000","last_unstale":"2023-06-22T07:59:59.709026+0000","last_undegraded":"2023-06-22T07:59:59.709026+0000","last_fullsized":"2023-06-22T07:59:59.709026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157321+0000","last_change":"2023-06-22T07:59:55.167525+0000","last_active":"2023-06-22T08:00:00.157321+0000","last_peered":"2023-06-22T08:00:00.157321+0000","last_clean":"2023-06-22T08:00:00.157321+0000","last_became_active":"2023-06-22T07:59:55.167181+0000","last_became_peered":"2023-06-22T07:59:55.167181+0000","last_unstale":"2023-06-22T08:00:00.157321+0000","last_undegraded":"2023-06-22T08:00:00.157321+0000","last_fullsized":"2023-06-22T08:00:00.157321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157927+0000","last_change":"2023-06-22T07:59:55.160549+0000","last_active":"2023-06-22T08:00:00.157927+0000","last_peered":"2023-06-22T08:00:00.157927+0000","last_clean":"2023-06-22T08:00:00.157927+0000","last_became_active":"2023-06-22T07:59:55.160239+0000","last_became_peered":"2023-06-22T07:59:55.160239+0000","last_unstale":"2023-06-22T08:00:00.157927+0000","last_undegraded":"2023-06-22T08:00:00.157927+0000","last_fullsized":"2023-06-22T08:00:00.157927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754390+0000","last_change":"2023-06-22T07:59:55.173006+0000","last_active":"2023-06-22T08:00:00.754390+0000","last_peered":"2023-06-22T08:00:00.754390+0000","last_clean":"2023-06-22T08:00:00.754390+0000","last_became_active":"2023-06-22T07:59:55.172578+0000","last_became_peered":"2023-06-22T07:59:55.172578+0000","last_unstale":"2023-06-22T08:00:00.754390+0000","last_undegraded":"2023-06-22T08:00:00.754390+0000","last_fullsized":"2023-06-22T08:00:00.754390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:12:44.799856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871378+0000","last_change":"2023-06-22T07:59:55.173858+0000","last_active":"2023-06-22T07:59:59.871378+0000","last_peered":"2023-06-22T07:59:59.871378+0000","last_clean":"2023-06-22T07:59:59.871378+0000","last_became_active":"2023-06-22T07:59:55.173599+0000","last_became_peered":"2023-06-22T07:59:55.173599+0000","last_unstale":"2023-06-22T07:59:59.871378+0000","last_undegraded":"2023-06-22T07:59:59.871378+0000","last_fullsized":"2023-06-22T07:59:59.871378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:37.597195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871174+0000","last_change":"2023-06-22T07:59:55.168354+0000","last_active":"2023-06-22T07:59:59.871174+0000","last_peered":"2023-06-22T07:59:59.871174+0000","last_clean":"2023-06-22T07:59:59.871174+0000","last_became_active":"2023-06-22T07:59:55.168050+0000","last_became_peered":"2023-06-22T07:59:55.168050+0000","last_unstale":"2023-06-22T07:59:59.871174+0000","last_undegraded":"2023-06-22T07:59:59.871174+0000","last_fullsized":"2023-06-22T07:59:59.871174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:08:45.847378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155602+0000","last_change":"2023-06-22T07:59:55.159738+0000","last_active":"2023-06-22T08:00:00.155602+0000","last_peered":"2023-06-22T08:00:00.155602+0000","last_clean":"2023-06-22T08:00:00.155602+0000","last_became_active":"2023-06-22T07:59:55.159501+0000","last_became_peered":"2023-06-22T07:59:55.159501+0000","last_unstale":"2023-06-22T08:00:00.155602+0000","last_undegraded":"2023-06-22T08:00:00.155602+0000","last_fullsized":"2023-06-22T08:00:00.155602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:55.160768+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:55.160508+0000","last_became_peered":"2023-06-22T07:59:55.160508+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:55:48.713145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155821+0000","last_change":"2023-06-22T07:59:55.166546+0000","last_active":"2023-06-22T08:00:00.155821+0000","last_peered":"2023-06-22T08:00:00.155821+0000","last_clean":"2023-06-22T08:00:00.155821+0000","last_became_active":"2023-06-22T07:59:55.166311+0000","last_became_peered":"2023-06-22T07:59:55.166311+0000","last_unstale":"2023-06-22T08:00:00.155821+0000","last_undegraded":"2023-06-22T08:00:00.155821+0000","last_fullsized":"2023-06-22T08:00:00.155821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:35.822548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155943+0000","last_change":"2023-06-22T07:59:55.162295+0000","last_active":"2023-06-22T08:00:00.155943+0000","last_peered":"2023-06-22T08:00:00.155943+0000","last_clean":"2023-06-22T08:00:00.155943+0000","last_became_active":"2023-06-22T07:59:55.162010+0000","last_became_peered":"2023-06-22T07:59:55.162010+0000","last_unstale":"2023-06-22T08:00:00.155943+0000","last_undegraded":"2023-06-22T08:00:00.155943+0000","last_fullsized":"2023-06-22T08:00:00.155943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709563+0000","last_change":"2023-06-22T07:59:55.158990+0000","last_active":"2023-06-22T07:59:59.709563+0000","last_peered":"2023-06-22T07:59:59.709563+0000","last_clean":"2023-06-22T07:59:59.709563+0000","last_became_active":"2023-06-22T07:59:55.158614+0000","last_became_peered":"2023-06-22T07:59:55.158614+0000","last_unstale":"2023-06-22T07:59:59.709563+0000","last_undegraded":"2023-06-22T07:59:59.709563+0000","last_fullsized":"2023-06-22T07:59:59.709563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755166+0000","last_change":"2023-06-22T07:59:55.161453+0000","last_active":"2023-06-22T08:00:00.755166+0000","last_peered":"2023-06-22T08:00:00.755166+0000","last_clean":"2023-06-22T08:00:00.755166+0000","last_became_active":"2023-06-22T07:59:55.161168+0000","last_became_peered":"2023-06-22T07:59:55.161168+0000","last_unstale":"2023-06-22T08:00:00.755166+0000","last_undegraded":"2023-06-22T08:00:00.755166+0000","last_fullsized":"2023-06-22T08:00:00.755166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155022+0000","last_change":"2023-06-22T07:59:55.164270+0000","last_active":"2023-06-22T08:00:00.155022+0000","last_peered":"2023-06-22T08:00:00.155022+0000","last_clean":"2023-06-22T08:00:00.155022+0000","last_became_active":"2023-06-22T07:59:55.163993+0000","last_became_peered":"2023-06-22T07:59:55.163993+0000","last_unstale":"2023-06-22T08:00:00.155022+0000","last_undegraded":"2023-06-22T08:00:00.155022+0000","last_fullsized":"2023-06-22T08:00:00.155022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:52.051830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157430+0000","last_change":"2023-06-22T07:59:55.169039+0000","last_active":"2023-06-22T08:00:00.157430+0000","last_peered":"2023-06-22T08:00:00.157430+0000","last_clean":"2023-06-22T08:00:00.157430+0000","last_became_active":"2023-06-22T07:59:55.168698+0000","last_became_peered":"2023-06-22T07:59:55.168698+0000","last_unstale":"2023-06-22T08:00:00.157430+0000","last_undegraded":"2023-06-22T08:00:00.157430+0000","last_fullsized":"2023-06-22T08:00:00.157430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871035+0000","last_change":"2023-06-22T07:59:55.158014+0000","last_active":"2023-06-22T07:59:59.871035+0000","last_peered":"2023-06-22T07:59:59.871035+0000","last_clean":"2023-06-22T07:59:59.871035+0000","last_became_active":"2023-06-22T07:59:55.157562+0000","last_became_peered":"2023-06-22T07:59:55.157562+0000","last_unstale":"2023-06-22T07:59:59.871035+0000","last_undegraded":"2023-06-22T07:59:59.871035+0000","last_fullsized":"2023-06-22T07:59:59.871035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:08:33.856001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754963+0000","last_change":"2023-06-22T07:59:55.165957+0000","last_active":"2023-06-22T08:00:00.754963+0000","last_peered":"2023-06-22T08:00:00.754963+0000","last_clean":"2023-06-22T08:00:00.754963+0000","last_became_active":"2023-06-22T07:59:55.165643+0000","last_became_peered":"2023-06-22T07:59:55.165643+0000","last_unstale":"2023-06-22T08:00:00.754963+0000","last_undegraded":"2023-06-22T08:00:00.754963+0000","last_fullsized":"2023-06-22T08:00:00.754963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:10.134743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157391+0000","last_change":"2023-06-22T07:59:55.159858+0000","last_active":"2023-06-22T08:00:00.157391+0000","last_peered":"2023-06-22T08:00:00.157391+0000","last_clean":"2023-06-22T08:00:00.157391+0000","last_became_active":"2023-06-22T07:59:55.159444+0000","last_became_peered":"2023-06-22T07:59:55.159444+0000","last_unstale":"2023-06-22T08:00:00.157391+0000","last_undegraded":"2023-06-22T08:00:00.157391+0000","last_fullsized":"2023-06-22T08:00:00.157391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709424+0000","last_change":"2023-06-22T07:59:55.157162+0000","last_active":"2023-06-22T07:59:59.709424+0000","last_peered":"2023-06-22T07:59:59.709424+0000","last_clean":"2023-06-22T07:59:59.709424+0000","last_became_active":"2023-06-22T07:59:55.156882+0000","last_became_peered":"2023-06-22T07:59:55.156882+0000","last_unstale":"2023-06-22T07:59:59.709424+0000","last_undegraded":"2023-06-22T07:59:59.709424+0000","last_fullsized":"2023-06-22T07:59:59.709424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155892+0000","last_change":"2023-06-22T07:59:55.163834+0000","last_active":"2023-06-22T08:00:00.155892+0000","last_peered":"2023-06-22T08:00:00.155892+0000","last_clean":"2023-06-22T08:00:00.155892+0000","last_became_active":"2023-06-22T07:59:55.163580+0000","last_became_peered":"2023-06-22T07:59:55.163580+0000","last_unstale":"2023-06-22T08:00:00.155892+0000","last_undegraded":"2023-06-22T08:00:00.155892+0000","last_fullsized":"2023-06-22T08:00:00.155892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155285+0000","last_change":"2023-06-22T07:59:55.164569+0000","last_active":"2023-06-22T08:00:00.155285+0000","last_peered":"2023-06-22T08:00:00.155285+0000","last_clean":"2023-06-22T08:00:00.155285+0000","last_became_active":"2023-06-22T07:59:55.164286+0000","last_became_peered":"2023-06-22T07:59:55.164286+0000","last_unstale":"2023-06-22T08:00:00.155285+0000","last_undegraded":"2023-06-22T08:00:00.155285+0000","last_fullsized":"2023-06-22T08:00:00.155285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:17.386317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754137+0000","last_change":"2023-06-22T07:59:55.159883+0000","last_active":"2023-06-22T08:00:00.754137+0000","last_peered":"2023-06-22T08:00:00.754137+0000","last_clean":"2023-06-22T08:00:00.754137+0000","last_became_active":"2023-06-22T07:59:55.159588+0000","last_became_peered":"2023-06-22T07:59:55.159588+0000","last_unstale":"2023-06-22T08:00:00.754137+0000","last_undegraded":"2023-06-22T08:00:00.754137+0000","last_fullsized":"2023-06-22T08:00:00.754137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754941+0000","last_change":"2023-06-22T07:59:55.160638+0000","last_active":"2023-06-22T08:00:00.754941+0000","last_peered":"2023-06-22T08:00:00.754941+0000","last_clean":"2023-06-22T08:00:00.754941+0000","last_became_active":"2023-06-22T07:59:55.160315+0000","last_became_peered":"2023-06-22T07:59:55.160315+0000","last_unstale":"2023-06-22T08:00:00.754941+0000","last_undegraded":"2023-06-22T08:00:00.754941+0000","last_fullsized":"2023-06-22T08:00:00.754941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:51:09.921626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155504+0000","last_change":"2023-06-22T07:59:55.165950+0000","last_active":"2023-06-22T08:00:00.155504+0000","last_peered":"2023-06-22T08:00:00.155504+0000","last_clean":"2023-06-22T08:00:00.155504+0000","last_became_active":"2023-06-22T07:59:55.165505+0000","last_became_peered":"2023-06-22T07:59:55.165505+0000","last_unstale":"2023-06-22T08:00:00.155504+0000","last_undegraded":"2023-06-22T08:00:00.155504+0000","last_fullsized":"2023-06-22T08:00:00.155504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:13.512588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755640+0000","last_change":"2023-06-22T07:59:55.157572+0000","last_active":"2023-06-22T08:00:00.755640+0000","last_peered":"2023-06-22T08:00:00.755640+0000","last_clean":"2023-06-22T08:00:00.755640+0000","last_became_active":"2023-06-22T07:59:55.157055+0000","last_became_peered":"2023-06-22T07:59:55.157055+0000","last_unstale":"2023-06-22T08:00:00.755640+0000","last_undegraded":"2023-06-22T08:00:00.755640+0000","last_fullsized":"2023-06-22T08:00:00.755640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:34:35.685117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754095+0000","last_change":"2023-06-22T07:59:55.164696+0000","last_active":"2023-06-22T08:00:00.754095+0000","last_peered":"2023-06-22T08:00:00.754095+0000","last_clean":"2023-06-22T08:00:00.754095+0000","last_became_active":"2023-06-22T07:59:55.164179+0000","last_became_peered":"2023-06-22T07:59:55.164179+0000","last_unstale":"2023-06-22T08:00:00.754095+0000","last_undegraded":"2023-06-22T08:00:00.754095+0000","last_fullsized":"2023-06-22T08:00:00.754095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871504+0000","last_change":"2023-06-22T07:59:55.168838+0000","last_active":"2023-06-22T07:59:59.871504+0000","last_peered":"2023-06-22T07:59:59.871504+0000","last_clean":"2023-06-22T07:59:59.871504+0000","last_became_active":"2023-06-22T07:59:55.168551+0000","last_became_peered":"2023-06-22T07:59:55.168551+0000","last_unstale":"2023-06-22T07:59:59.871504+0000","last_undegraded":"2023-06-22T07:59:59.871504+0000","last_fullsized":"2023-06-22T07:59:59.871504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:02:36.615449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708648+0000","last_change":"2023-06-22T07:59:59.195356+0000","last_active":"2023-06-22T07:59:59.708648+0000","last_peered":"2023-06-22T07:59:59.708648+0000","last_clean":"2023-06-22T07:59:59.708648+0000","last_became_active":"2023-06-22T07:59:59.194814+0000","last_became_peered":"2023-06-22T07:59:59.194814+0000","last_unstale":"2023-06-22T07:59:59.708648+0000","last_undegraded":"2023-06-22T07:59:59.708648+0000","last_fullsized":"2023-06-22T07:59:59.708648+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:12.223267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6,0],"acting":[2,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"27'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155337+0000","last_change":"2023-06-22T07:59:54.651326+0000","last_active":"2023-06-22T08:00:00.155337+0000","last_peered":"2023-06-22T08:00:00.155337+0000","last_clean":"2023-06-22T08:00:00.155337+0000","last_became_active":"2023-06-22T07:59:54.651010+0000","last_became_peered":"2023-06-22T07:59:54.651010+0000","last_unstale":"2023-06-22T08:00:00.155337+0000","last_undegraded":"2023-06-22T08:00:00.155337+0000","last_fullsized":"2023-06-22T08:00:00.155337+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:17.769117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754538+0000","last_change":"2023-06-22T07:59:55.160345+0000","last_active":"2023-06-22T08:00:00.754538+0000","last_peered":"2023-06-22T08:00:00.754538+0000","last_clean":"2023-06-22T08:00:00.754538+0000","last_became_active":"2023-06-22T07:59:55.160071+0000","last_became_peered":"2023-06-22T07:59:55.160071+0000","last_unstale":"2023-06-22T08:00:00.754538+0000","last_undegraded":"2023-06-22T08:00:00.754538+0000","last_fullsized":"2023-06-22T08:00:00.754538+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708662+0000","last_change":"2023-06-22T07:59:59.196477+0000","last_active":"2023-06-22T07:59:59.708662+0000","last_peered":"2023-06-22T07:59:59.708662+0000","last_clean":"2023-06-22T07:59:59.708662+0000","last_became_active":"2023-06-22T07:59:59.196007+0000","last_became_peered":"2023-06-22T07:59:59.196007+0000","last_unstale":"2023-06-22T07:59:59.708662+0000","last_undegraded":"2023-06-22T07:59:59.708662+0000","last_fullsized":"2023-06-22T07:59:59.708662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,5],"acting":[2,7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753791+0000","last_change":"2023-06-22T07:59:54.653192+0000","last_active":"2023-06-22T08:00:00.753791+0000","last_peered":"2023-06-22T08:00:00.753791+0000","last_clean":"2023-06-22T08:00:00.753791+0000","last_became_active":"2023-06-22T07:59:54.652926+0000","last_became_peered":"2023-06-22T07:59:54.652926+0000","last_unstale":"2023-06-22T08:00:00.753791+0000","last_undegraded":"2023-06-22T08:00:00.753791+0000","last_fullsized":"2023-06-22T08:00:00.753791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:12:59.656555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754609+0000","last_change":"2023-06-22T07:59:55.163171+0000","last_active":"2023-06-22T08:00:00.754609+0000","last_peered":"2023-06-22T08:00:00.754609+0000","last_clean":"2023-06-22T08:00:00.754609+0000","last_became_active":"2023-06-22T07:59:55.162869+0000","last_became_peered":"2023-06-22T07:59:55.162869+0000","last_unstale":"2023-06-22T08:00:00.754609+0000","last_undegraded":"2023-06-22T08:00:00.754609+0000","last_fullsized":"2023-06-22T08:00:00.754609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:37:59.063186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754586+0000","last_change":"2023-06-22T07:59:59.200516+0000","last_active":"2023-06-22T08:00:00.754586+0000","last_peered":"2023-06-22T08:00:00.754586+0000","last_clean":"2023-06-22T08:00:00.754586+0000","last_became_active":"2023-06-22T07:59:59.200096+0000","last_became_peered":"2023-06-22T07:59:59.200096+0000","last_unstale":"2023-06-22T08:00:00.754586+0000","last_undegraded":"2023-06-22T08:00:00.754586+0000","last_fullsized":"2023-06-22T08:00:00.754586+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:06.099105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2,7],"acting":[6,3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708683+0000","last_change":"2023-06-22T07:59:54.645950+0000","last_active":"2023-06-22T07:59:59.708683+0000","last_peered":"2023-06-22T07:59:59.708683+0000","last_clean":"2023-06-22T07:59:59.708683+0000","last_became_active":"2023-06-22T07:59:54.645520+0000","last_became_peered":"2023-06-22T07:59:54.645520+0000","last_unstale":"2023-06-22T07:59:59.708683+0000","last_undegraded":"2023-06-22T07:59:59.708683+0000","last_fullsized":"2023-06-22T07:59:59.708683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:34.816556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871992+0000","last_change":"2023-06-22T07:59:55.162767+0000","last_active":"2023-06-22T07:59:59.871992+0000","last_peered":"2023-06-22T07:59:59.871992+0000","last_clean":"2023-06-22T07:59:59.871992+0000","last_became_active":"2023-06-22T07:59:55.162451+0000","last_became_peered":"2023-06-22T07:59:55.162451+0000","last_unstale":"2023-06-22T07:59:59.871992+0000","last_undegraded":"2023-06-22T07:59:59.871992+0000","last_fullsized":"2023-06-22T07:59:59.871992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:38.977413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871071+0000","last_change":"2023-06-22T07:59:59.208215+0000","last_active":"2023-06-22T07:59:59.871071+0000","last_peered":"2023-06-22T07:59:59.871071+0000","last_clean":"2023-06-22T07:59:59.871071+0000","last_became_active":"2023-06-22T07:59:59.207608+0000","last_became_peered":"2023-06-22T07:59:59.207608+0000","last_unstale":"2023-06-22T07:59:59.871071+0000","last_undegraded":"2023-06-22T07:59:59.871071+0000","last_fullsized":"2023-06-22T07:59:59.871071+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:39:48.527076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,5],"acting":[3,2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709379+0000","last_change":"2023-06-22T07:59:54.652515+0000","last_active":"2023-06-22T07:59:59.709379+0000","last_peered":"2023-06-22T07:59:59.709379+0000","last_clean":"2023-06-22T07:59:59.709379+0000","last_became_active":"2023-06-22T07:59:54.652249+0000","last_became_peered":"2023-06-22T07:59:54.652249+0000","last_unstale":"2023-06-22T07:59:59.709379+0000","last_undegraded":"2023-06-22T07:59:59.709379+0000","last_fullsized":"2023-06-22T07:59:59.709379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:12.506596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755250+0000","last_change":"2023-06-22T07:59:55.158359+0000","last_active":"2023-06-22T08:00:00.755250+0000","last_peered":"2023-06-22T08:00:00.755250+0000","last_clean":"2023-06-22T08:00:00.755250+0000","last_became_active":"2023-06-22T07:59:55.157918+0000","last_became_peered":"2023-06-22T07:59:55.157918+0000","last_unstale":"2023-06-22T08:00:00.755250+0000","last_undegraded":"2023-06-22T08:00:00.755250+0000","last_fullsized":"2023-06-22T08:00:00.755250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708305+0000","last_change":"2023-06-22T07:59:59.201033+0000","last_active":"2023-06-22T07:59:59.708305+0000","last_peered":"2023-06-22T07:59:59.708305+0000","last_clean":"2023-06-22T07:59:59.708305+0000","last_became_active":"2023-06-22T07:59:59.200500+0000","last_became_peered":"2023-06-22T07:59:59.200500+0000","last_unstale":"2023-06-22T07:59:59.708305+0000","last_undegraded":"2023-06-22T07:59:59.708305+0000","last_fullsized":"2023-06-22T07:59:59.708305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:18.174604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,4],"acting":[2,1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708780+0000","last_change":"2023-06-22T07:59:54.654416+0000","last_active":"2023-06-22T07:59:59.708780+0000","last_peered":"2023-06-22T07:59:59.708780+0000","last_clean":"2023-06-22T07:59:59.708780+0000","last_became_active":"2023-06-22T07:59:54.654139+0000","last_became_peered":"2023-06-22T07:59:54.654139+0000","last_unstale":"2023-06-22T07:59:59.708780+0000","last_undegraded":"2023-06-22T07:59:59.708780+0000","last_fullsized":"2023-06-22T07:59:59.708780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755506+0000","last_change":"2023-06-22T07:59:55.159790+0000","last_active":"2023-06-22T08:00:00.755506+0000","last_peered":"2023-06-22T08:00:00.755506+0000","last_clean":"2023-06-22T08:00:00.755506+0000","last_became_active":"2023-06-22T07:59:55.159489+0000","last_became_peered":"2023-06-22T07:59:55.159489+0000","last_unstale":"2023-06-22T08:00:00.755506+0000","last_undegraded":"2023-06-22T08:00:00.755506+0000","last_fullsized":"2023-06-22T08:00:00.755506+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:15:18.373418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753814+0000","last_change":"2023-06-22T07:59:59.197822+0000","last_active":"2023-06-22T08:00:00.753814+0000","last_peered":"2023-06-22T08:00:00.753814+0000","last_clean":"2023-06-22T08:00:00.753814+0000","last_became_active":"2023-06-22T07:59:59.197431+0000","last_became_peered":"2023-06-22T07:59:59.197431+0000","last_unstale":"2023-06-22T08:00:00.753814+0000","last_undegraded":"2023-06-22T08:00:00.753814+0000","last_fullsized":"2023-06-22T08:00:00.753814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:41.139500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7,1],"acting":[4,6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"27'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157177+0000","last_change":"2023-06-22T07:59:54.644158+0000","last_active":"2023-06-22T08:00:00.157177+0000","last_peered":"2023-06-22T08:00:00.157177+0000","last_clean":"2023-06-22T08:00:00.157177+0000","last_became_active":"2023-06-22T07:59:54.643808+0000","last_became_peered":"2023-06-22T07:59:54.643808+0000","last_unstale":"2023-06-22T08:00:00.157177+0000","last_undegraded":"2023-06-22T08:00:00.157177+0000","last_fullsized":"2023-06-22T08:00:00.157177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:45.438691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754507+0000","last_change":"2023-06-22T07:59:55.157538+0000","last_active":"2023-06-22T08:00:00.754507+0000","last_peered":"2023-06-22T08:00:00.754507+0000","last_clean":"2023-06-22T08:00:00.754507+0000","last_became_active":"2023-06-22T07:59:55.156967+0000","last_became_peered":"2023-06-22T07:59:55.156967+0000","last_unstale":"2023-06-22T08:00:00.754507+0000","last_undegraded":"2023-06-22T08:00:00.754507+0000","last_fullsized":"2023-06-22T08:00:00.754507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:38:44.161245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753821+0000","last_change":"2023-06-22T07:59:59.201671+0000","last_active":"2023-06-22T08:00:00.753821+0000","last_peered":"2023-06-22T08:00:00.753821+0000","last_clean":"2023-06-22T08:00:00.753821+0000","last_became_active":"2023-06-22T07:59:59.201201+0000","last_became_peered":"2023-06-22T07:59:59.201201+0000","last_unstale":"2023-06-22T08:00:00.753821+0000","last_undegraded":"2023-06-22T08:00:00.753821+0000","last_fullsized":"2023-06-22T08:00:00.753821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4,3],"acting":[6,1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"28'10","reported_seq":31,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:17.682776+0000","last_change":"2023-06-22T07:59:54.640470+0000","last_active":"2023-06-22T08:00:17.682776+0000","last_peered":"2023-06-22T08:00:17.682776+0000","last_clean":"2023-06-22T08:00:17.682776+0000","last_became_active":"2023-06-22T07:59:54.640124+0000","last_became_peered":"2023-06-22T07:59:54.640124+0000","last_unstale":"2023-06-22T08:00:17.682776+0000","last_undegraded":"2023-06-22T08:00:17.682776+0000","last_fullsized":"2023-06-22T08:00:17.682776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:06:41.502702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708599+0000","last_change":"2023-06-22T07:59:55.165550+0000","last_active":"2023-06-22T07:59:59.708599+0000","last_peered":"2023-06-22T07:59:59.708599+0000","last_clean":"2023-06-22T07:59:59.708599+0000","last_became_active":"2023-06-22T07:59:55.164998+0000","last_became_peered":"2023-06-22T07:59:55.164998+0000","last_unstale":"2023-06-22T07:59:59.708599+0000","last_undegraded":"2023-06-22T07:59:59.708599+0000","last_fullsized":"2023-06-22T07:59:59.708599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872133+0000","last_change":"2023-06-22T07:59:59.207103+0000","last_active":"2023-06-22T07:59:59.872133+0000","last_peered":"2023-06-22T07:59:59.872133+0000","last_clean":"2023-06-22T07:59:59.872133+0000","last_became_active":"2023-06-22T07:59:59.206531+0000","last_became_peered":"2023-06-22T07:59:59.206531+0000","last_unstale":"2023-06-22T07:59:59.872133+0000","last_undegraded":"2023-06-22T07:59:59.872133+0000","last_fullsized":"2023-06-22T07:59:59.872133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:42:48.571228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0,6],"acting":[3,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157887+0000","last_change":"2023-06-22T07:59:54.644301+0000","last_active":"2023-06-22T08:00:00.157887+0000","last_peered":"2023-06-22T08:00:00.157887+0000","last_clean":"2023-06-22T08:00:00.157887+0000","last_became_active":"2023-06-22T07:59:54.643939+0000","last_became_peered":"2023-06-22T07:59:54.643939+0000","last_unstale":"2023-06-22T08:00:00.157887+0000","last_undegraded":"2023-06-22T08:00:00.157887+0000","last_fullsized":"2023-06-22T08:00:00.157887+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:07:42.118767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753877+0000","last_change":"2023-06-22T07:59:55.163311+0000","last_active":"2023-06-22T08:00:00.753877+0000","last_peered":"2023-06-22T08:00:00.753877+0000","last_clean":"2023-06-22T08:00:00.753877+0000","last_became_active":"2023-06-22T07:59:55.162991+0000","last_became_peered":"2023-06-22T07:59:55.162991+0000","last_unstale":"2023-06-22T08:00:00.753877+0000","last_undegraded":"2023-06-22T08:00:00.753877+0000","last_fullsized":"2023-06-22T08:00:00.753877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754227+0000","last_change":"2023-06-22T07:59:59.201324+0000","last_active":"2023-06-22T08:00:00.754227+0000","last_peered":"2023-06-22T08:00:00.754227+0000","last_clean":"2023-06-22T08:00:00.754227+0000","last_became_active":"2023-06-22T07:59:59.200877+0000","last_became_peered":"2023-06-22T07:59:59.200877+0000","last_unstale":"2023-06-22T08:00:00.754227+0000","last_undegraded":"2023-06-22T08:00:00.754227+0000","last_fullsized":"2023-06-22T08:00:00.754227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:14:17.094426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5,7],"acting":[4,2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754441+0000","last_change":"2023-06-22T07:59:54.651527+0000","last_active":"2023-06-22T08:00:00.754441+0000","last_peered":"2023-06-22T08:00:00.754441+0000","last_clean":"2023-06-22T08:00:00.754441+0000","last_became_active":"2023-06-22T07:59:54.651196+0000","last_became_peered":"2023-06-22T07:59:54.651196+0000","last_unstale":"2023-06-22T08:00:00.754441+0000","last_undegraded":"2023-06-22T08:00:00.754441+0000","last_fullsized":"2023-06-22T08:00:00.754441+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872142+0000","last_change":"2023-06-22T07:59:55.162610+0000","last_active":"2023-06-22T07:59:59.872142+0000","last_peered":"2023-06-22T07:59:59.872142+0000","last_clean":"2023-06-22T07:59:59.872142+0000","last_became_active":"2023-06-22T07:59:55.162334+0000","last_became_peered":"2023-06-22T07:59:55.162334+0000","last_unstale":"2023-06-22T07:59:59.872142+0000","last_undegraded":"2023-06-22T07:59:59.872142+0000","last_fullsized":"2023-06-22T07:59:59.872142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:42.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:59.199402+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:59.198979+0000","last_became_peered":"2023-06-22T07:59:59.198979+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:17:07.122732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1,6],"acting":[4,5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.870997+0000","last_change":"2023-06-22T07:59:54.649236+0000","last_active":"2023-06-22T07:59:59.870997+0000","last_peered":"2023-06-22T07:59:59.870997+0000","last_clean":"2023-06-22T07:59:59.870997+0000","last_became_active":"2023-06-22T07:59:54.648852+0000","last_became_peered":"2023-06-22T07:59:54.648852+0000","last_unstale":"2023-06-22T07:59:59.870997+0000","last_undegraded":"2023-06-22T07:59:59.870997+0000","last_fullsized":"2023-06-22T07:59:59.870997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:16:59.865638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709134+0000","last_change":"2023-06-22T07:59:55.165859+0000","last_active":"2023-06-22T07:59:59.709134+0000","last_peered":"2023-06-22T07:59:59.709134+0000","last_clean":"2023-06-22T07:59:59.709134+0000","last_became_active":"2023-06-22T07:59:55.165542+0000","last_became_peered":"2023-06-22T07:59:55.165542+0000","last_unstale":"2023-06-22T07:59:59.709134+0000","last_undegraded":"2023-06-22T07:59:59.709134+0000","last_fullsized":"2023-06-22T07:59:59.709134+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:32.147969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871894+0000","last_change":"2023-06-22T07:59:59.207794+0000","last_active":"2023-06-22T07:59:59.871894+0000","last_peered":"2023-06-22T07:59:59.871894+0000","last_clean":"2023-06-22T07:59:59.871894+0000","last_became_active":"2023-06-22T07:59:59.207364+0000","last_became_peered":"2023-06-22T07:59:59.207364+0000","last_unstale":"2023-06-22T07:59:59.871894+0000","last_undegraded":"2023-06-22T07:59:59.871894+0000","last_fullsized":"2023-06-22T07:59:59.871894+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:30.718491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4,0],"acting":[3,6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"27'8","reported_seq":29,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158928+0000","last_change":"2023-06-22T07:59:54.651423+0000","last_active":"2023-06-22T08:00:00.158928+0000","last_peered":"2023-06-22T08:00:00.158928+0000","last_clean":"2023-06-22T08:00:00.158928+0000","last_became_active":"2023-06-22T07:59:54.651167+0000","last_became_peered":"2023-06-22T07:59:54.651167+0000","last_unstale":"2023-06-22T08:00:00.158928+0000","last_undegraded":"2023-06-22T08:00:00.158928+0000","last_fullsized":"2023-06-22T08:00:00.158928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:31.141977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709649+0000","last_change":"2023-06-22T07:59:55.164478+0000","last_active":"2023-06-22T07:59:59.709649+0000","last_peered":"2023-06-22T07:59:59.709649+0000","last_clean":"2023-06-22T07:59:59.709649+0000","last_became_active":"2023-06-22T07:59:55.163922+0000","last_became_peered":"2023-06-22T07:59:55.163922+0000","last_unstale":"2023-06-22T07:59:59.709649+0000","last_undegraded":"2023-06-22T07:59:59.709649+0000","last_fullsized":"2023-06-22T07:59:59.709649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:43.772399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155085+0000","last_change":"2023-06-22T07:59:59.198496+0000","last_active":"2023-06-22T08:00:00.155085+0000","last_peered":"2023-06-22T08:00:00.155085+0000","last_clean":"2023-06-22T08:00:00.155085+0000","last_became_active":"2023-06-22T07:59:59.198067+0000","last_became_peered":"2023-06-22T07:59:59.198067+0000","last_unstale":"2023-06-22T08:00:00.155085+0000","last_undegraded":"2023-06-22T08:00:00.155085+0000","last_fullsized":"2023-06-22T08:00:00.155085+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:24.865757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,7],"acting":[1,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"27'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157653+0000","last_change":"2023-06-22T07:59:54.651366+0000","last_active":"2023-06-22T08:00:00.157653+0000","last_peered":"2023-06-22T08:00:00.157653+0000","last_clean":"2023-06-22T08:00:00.157653+0000","last_became_active":"2023-06-22T07:59:54.651059+0000","last_became_peered":"2023-06-22T07:59:54.651059+0000","last_unstale":"2023-06-22T08:00:00.157653+0000","last_undegraded":"2023-06-22T08:00:00.157653+0000","last_fullsized":"2023-06-22T08:00:00.157653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:37.183143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871131+0000","last_change":"2023-06-22T07:59:55.162820+0000","last_active":"2023-06-22T07:59:59.871131+0000","last_peered":"2023-06-22T07:59:59.871131+0000","last_clean":"2023-06-22T07:59:59.871131+0000","last_became_active":"2023-06-22T07:59:55.162505+0000","last_became_peered":"2023-06-22T07:59:55.162505+0000","last_unstale":"2023-06-22T07:59:59.871131+0000","last_undegraded":"2023-06-22T07:59:59.871131+0000","last_fullsized":"2023-06-22T07:59:59.871131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:46:38.312778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753744+0000","last_change":"2023-06-22T07:59:59.201059+0000","last_active":"2023-06-22T08:00:00.753744+0000","last_peered":"2023-06-22T08:00:00.753744+0000","last_clean":"2023-06-22T08:00:00.753744+0000","last_became_active":"2023-06-22T07:59:59.200460+0000","last_became_peered":"2023-06-22T07:59:59.200460+0000","last_unstale":"2023-06-22T08:00:00.753744+0000","last_undegraded":"2023-06-22T08:00:00.753744+0000","last_fullsized":"2023-06-22T08:00:00.753744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,0],"acting":[6,4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158090+0000","last_change":"2023-06-22T07:59:54.646017+0000","last_active":"2023-06-22T08:00:00.158090+0000","last_peered":"2023-06-22T08:00:00.158090+0000","last_clean":"2023-06-22T08:00:00.158090+0000","last_became_active":"2023-06-22T07:59:54.645554+0000","last_became_peered":"2023-06-22T07:59:54.645554+0000","last_unstale":"2023-06-22T08:00:00.158090+0000","last_undegraded":"2023-06-22T08:00:00.158090+0000","last_fullsized":"2023-06-22T08:00:00.158090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:01.550269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154806+0000","last_change":"2023-06-22T07:59:55.166283+0000","last_active":"2023-06-22T08:00:00.154806+0000","last_peered":"2023-06-22T08:00:00.154806+0000","last_clean":"2023-06-22T08:00:00.154806+0000","last_became_active":"2023-06-22T07:59:55.165977+0000","last_became_peered":"2023-06-22T07:59:55.165977+0000","last_unstale":"2023-06-22T08:00:00.154806+0000","last_undegraded":"2023-06-22T08:00:00.154806+0000","last_fullsized":"2023-06-22T08:00:00.154806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:21:29.484640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709141+0000","last_change":"2023-06-22T07:59:59.205411+0000","last_active":"2023-06-22T07:59:59.709141+0000","last_peered":"2023-06-22T07:59:59.709141+0000","last_clean":"2023-06-22T07:59:59.709141+0000","last_became_active":"2023-06-22T07:59:59.204980+0000","last_became_peered":"2023-06-22T07:59:59.204980+0000","last_unstale":"2023-06-22T07:59:59.709141+0000","last_undegraded":"2023-06-22T07:59:59.709141+0000","last_fullsized":"2023-06-22T07:59:59.709141+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:22.317952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871622+0000","last_change":"2023-06-22T07:59:54.654491+0000","last_active":"2023-06-22T07:59:59.871622+0000","last_peered":"2023-06-22T07:59:59.871622+0000","last_clean":"2023-06-22T07:59:59.871622+0000","last_became_active":"2023-06-22T07:59:54.654197+0000","last_became_peered":"2023-06-22T07:59:54.654197+0000","last_unstale":"2023-06-22T07:59:59.871622+0000","last_undegraded":"2023-06-22T07:59:59.871622+0000","last_fullsized":"2023-06-22T07:59:59.871622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:42:59.725455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708444+0000","last_change":"2023-06-22T07:59:55.172053+0000","last_active":"2023-06-22T07:59:59.708444+0000","last_peered":"2023-06-22T07:59:59.708444+0000","last_clean":"2023-06-22T07:59:59.708444+0000","last_became_active":"2023-06-22T07:59:55.165501+0000","last_became_peered":"2023-06-22T07:59:55.165501+0000","last_unstale":"2023-06-22T07:59:59.708444+0000","last_undegraded":"2023-06-22T07:59:59.708444+0000","last_fullsized":"2023-06-22T07:59:59.708444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753990+0000","last_change":"2023-06-22T07:59:59.199179+0000","last_active":"2023-06-22T08:00:00.753990+0000","last_peered":"2023-06-22T08:00:00.753990+0000","last_clean":"2023-06-22T08:00:00.753990+0000","last_became_active":"2023-06-22T07:59:59.198740+0000","last_became_peered":"2023-06-22T07:59:59.198740+0000","last_unstale":"2023-06-22T08:00:00.753990+0000","last_undegraded":"2023-06-22T08:00:00.753990+0000","last_fullsized":"2023-06-22T08:00:00.753990+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158086+0000","last_change":"2023-06-22T07:59:54.644520+0000","last_active":"2023-06-22T08:00:00.158086+0000","last_peered":"2023-06-22T08:00:00.158086+0000","last_clean":"2023-06-22T08:00:00.158086+0000","last_became_active":"2023-06-22T07:59:54.644204+0000","last_became_peered":"2023-06-22T07:59:54.644204+0000","last_unstale":"2023-06-22T08:00:00.158086+0000","last_undegraded":"2023-06-22T08:00:00.158086+0000","last_fullsized":"2023-06-22T08:00:00.158086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:07.674431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709657+0000","last_change":"2023-06-22T07:59:55.164111+0000","last_active":"2023-06-22T07:59:59.709657+0000","last_peered":"2023-06-22T07:59:59.709657+0000","last_clean":"2023-06-22T07:59:59.709657+0000","last_became_active":"2023-06-22T07:59:55.163802+0000","last_became_peered":"2023-06-22T07:59:55.163802+0000","last_unstale":"2023-06-22T07:59:59.709657+0000","last_undegraded":"2023-06-22T07:59:59.709657+0000","last_fullsized":"2023-06-22T07:59:59.709657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755252+0000","last_change":"2023-06-22T07:59:59.197721+0000","last_active":"2023-06-22T08:00:00.755252+0000","last_peered":"2023-06-22T08:00:00.755252+0000","last_clean":"2023-06-22T08:00:00.755252+0000","last_became_active":"2023-06-22T07:59:59.197259+0000","last_became_peered":"2023-06-22T07:59:59.197259+0000","last_unstale":"2023-06-22T08:00:00.755252+0000","last_undegraded":"2023-06-22T08:00:00.755252+0000","last_fullsized":"2023-06-22T08:00:00.755252+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5,1],"acting":[7,0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"27'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754436+0000","last_change":"2023-06-22T07:59:54.643124+0000","last_active":"2023-06-22T08:00:00.754436+0000","last_peered":"2023-06-22T08:00:00.754436+0000","last_clean":"2023-06-22T08:00:00.754436+0000","last_became_active":"2023-06-22T07:59:54.642833+0000","last_became_peered":"2023-06-22T07:59:54.642833+0000","last_unstale":"2023-06-22T08:00:00.754436+0000","last_undegraded":"2023-06-22T08:00:00.754436+0000","last_fullsized":"2023-06-22T08:00:00.754436+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158064+0000","last_change":"2023-06-22T07:59:55.160551+0000","last_active":"2023-06-22T08:00:00.158064+0000","last_peered":"2023-06-22T08:00:00.158064+0000","last_clean":"2023-06-22T08:00:00.158064+0000","last_became_active":"2023-06-22T07:59:55.160221+0000","last_became_peered":"2023-06-22T07:59:55.160221+0000","last_unstale":"2023-06-22T08:00:00.158064+0000","last_undegraded":"2023-06-22T08:00:00.158064+0000","last_fullsized":"2023-06-22T08:00:00.158064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708974+0000","last_change":"2023-06-22T07:59:59.204643+0000","last_active":"2023-06-22T07:59:59.708974+0000","last_peered":"2023-06-22T07:59:59.708974+0000","last_clean":"2023-06-22T07:59:59.708974+0000","last_became_active":"2023-06-22T07:59:59.204182+0000","last_became_peered":"2023-06-22T07:59:59.204182+0000","last_unstale":"2023-06-22T07:59:59.708974+0000","last_undegraded":"2023-06-22T07:59:59.708974+0000","last_fullsized":"2023-06-22T07:59:59.708974+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4,2],"acting":[5,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871327+0000","last_change":"2023-06-22T07:59:54.654036+0000","last_active":"2023-06-22T07:59:59.871327+0000","last_peered":"2023-06-22T07:59:59.871327+0000","last_clean":"2023-06-22T07:59:59.871327+0000","last_became_active":"2023-06-22T07:59:54.653736+0000","last_became_peered":"2023-06-22T07:59:54.653736+0000","last_unstale":"2023-06-22T07:59:59.871327+0000","last_undegraded":"2023-06-22T07:59:59.871327+0000","last_fullsized":"2023-06-22T07:59:59.871327+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:44:09.331748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871999+0000","last_change":"2023-06-22T07:59:55.163181+0000","last_active":"2023-06-22T07:59:59.871999+0000","last_peered":"2023-06-22T07:59:59.871999+0000","last_clean":"2023-06-22T07:59:59.871999+0000","last_became_active":"2023-06-22T07:59:55.162860+0000","last_became_peered":"2023-06-22T07:59:55.162860+0000","last_unstale":"2023-06-22T07:59:59.871999+0000","last_undegraded":"2023-06-22T07:59:59.871999+0000","last_fullsized":"2023-06-22T07:59:59.871999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:59:25.037938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155510+0000","last_change":"2023-06-22T07:59:59.195710+0000","last_active":"2023-06-22T08:00:00.155510+0000","last_peered":"2023-06-22T08:00:00.155510+0000","last_clean":"2023-06-22T08:00:00.155510+0000","last_became_active":"2023-06-22T07:59:59.195246+0000","last_became_peered":"2023-06-22T07:59:59.195246+0000","last_unstale":"2023-06-22T08:00:00.155510+0000","last_undegraded":"2023-06-22T08:00:00.155510+0000","last_fullsized":"2023-06-22T08:00:00.155510+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:36.457493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,5],"acting":[1,0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871146+0000","last_change":"2023-06-22T07:59:54.649692+0000","last_active":"2023-06-22T07:59:59.871146+0000","last_peered":"2023-06-22T07:59:59.871146+0000","last_clean":"2023-06-22T07:59:59.871146+0000","last_became_active":"2023-06-22T07:59:54.649416+0000","last_became_peered":"2023-06-22T07:59:54.649416+0000","last_unstale":"2023-06-22T07:59:59.871146+0000","last_undegraded":"2023-06-22T07:59:59.871146+0000","last_fullsized":"2023-06-22T07:59:59.871146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:54:02.923433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708909+0000","last_change":"2023-06-22T07:59:55.175392+0000","last_active":"2023-06-22T07:59:59.708909+0000","last_peered":"2023-06-22T07:59:59.708909+0000","last_clean":"2023-06-22T07:59:59.708909+0000","last_became_active":"2023-06-22T07:59:55.175112+0000","last_became_peered":"2023-06-22T07:59:55.175112+0000","last_unstale":"2023-06-22T07:59:59.708909+0000","last_undegraded":"2023-06-22T07:59:59.708909+0000","last_fullsized":"2023-06-22T07:59:59.708909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871873+0000","last_change":"2023-06-22T07:59:59.206671+0000","last_active":"2023-06-22T07:59:59.871873+0000","last_peered":"2023-06-22T07:59:59.871873+0000","last_clean":"2023-06-22T07:59:59.871873+0000","last_became_active":"2023-06-22T07:59:59.206179+0000","last_became_peered":"2023-06-22T07:59:59.206179+0000","last_unstale":"2023-06-22T07:59:59.871873+0000","last_undegraded":"2023-06-22T07:59:59.871873+0000","last_fullsized":"2023-06-22T07:59:59.871873+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:00.427518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4,5],"acting":[3,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753414+0000","last_change":"2023-06-22T07:59:54.640508+0000","last_active":"2023-06-22T08:00:00.753414+0000","last_peered":"2023-06-22T08:00:00.753414+0000","last_clean":"2023-06-22T08:00:00.753414+0000","last_became_active":"2023-06-22T07:59:54.640108+0000","last_became_peered":"2023-06-22T07:59:54.640108+0000","last_unstale":"2023-06-22T08:00:00.753414+0000","last_undegraded":"2023-06-22T08:00:00.753414+0000","last_fullsized":"2023-06-22T08:00:00.753414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:19.193231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754822+0000","last_change":"2023-06-22T07:59:55.164398+0000","last_active":"2023-06-22T08:00:00.754822+0000","last_peered":"2023-06-22T08:00:00.754822+0000","last_clean":"2023-06-22T08:00:00.754822+0000","last_became_active":"2023-06-22T07:59:55.164086+0000","last_became_peered":"2023-06-22T07:59:55.164086+0000","last_unstale":"2023-06-22T08:00:00.754822+0000","last_undegraded":"2023-06-22T08:00:00.754822+0000","last_fullsized":"2023-06-22T08:00:00.754822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:38.189135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753997+0000","last_change":"2023-06-22T07:59:59.207752+0000","last_active":"2023-06-22T08:00:00.753997+0000","last_peered":"2023-06-22T08:00:00.753997+0000","last_clean":"2023-06-22T08:00:00.753997+0000","last_became_active":"2023-06-22T07:59:59.207360+0000","last_became_peered":"2023-06-22T07:59:59.207360+0000","last_unstale":"2023-06-22T08:00:00.753997+0000","last_undegraded":"2023-06-22T08:00:00.753997+0000","last_fullsized":"2023-06-22T08:00:00.753997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:08:25.633009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3,1],"acting":[4,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709114+0000","last_change":"2023-06-22T07:59:49.652772+0000","last_active":"2023-06-22T07:59:59.709114+0000","last_peered":"2023-06-22T07:59:59.709114+0000","last_clean":"2023-06-22T07:59:59.709114+0000","last_became_active":"2023-06-22T07:59:49.652487+0000","last_became_peered":"2023-06-22T07:59:49.652487+0000","last_unstale":"2023-06-22T07:59:59.709114+0000","last_undegraded":"2023-06-22T07:59:59.709114+0000","last_fullsized":"2023-06-22T07:59:59.709114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:11.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708571+0000","last_change":"2023-06-22T07:59:54.644538+0000","last_active":"2023-06-22T07:59:59.708571+0000","last_peered":"2023-06-22T07:59:59.708571+0000","last_clean":"2023-06-22T07:59:59.708571+0000","last_became_active":"2023-06-22T07:59:54.644253+0000","last_became_peered":"2023-06-22T07:59:54.644253+0000","last_unstale":"2023-06-22T07:59:59.708571+0000","last_undegraded":"2023-06-22T07:59:59.708571+0000","last_fullsized":"2023-06-22T07:59:59.708571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:09.128751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754228+0000","last_change":"2023-06-22T07:59:55.163802+0000","last_active":"2023-06-22T08:00:00.754228+0000","last_peered":"2023-06-22T08:00:00.754228+0000","last_clean":"2023-06-22T08:00:00.754228+0000","last_became_active":"2023-06-22T07:59:55.163481+0000","last_became_peered":"2023-06-22T07:59:55.163481+0000","last_unstale":"2023-06-22T08:00:00.754228+0000","last_undegraded":"2023-06-22T08:00:00.754228+0000","last_fullsized":"2023-06-22T08:00:00.754228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:16.916350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755246+0000","last_change":"2023-06-22T07:59:59.207344+0000","last_active":"2023-06-22T08:00:00.755246+0000","last_peered":"2023-06-22T08:00:00.755246+0000","last_clean":"2023-06-22T08:00:00.755246+0000","last_became_active":"2023-06-22T07:59:59.206975+0000","last_became_peered":"2023-06-22T07:59:59.206975+0000","last_unstale":"2023-06-22T08:00:00.755246+0000","last_undegraded":"2023-06-22T08:00:00.755246+0000","last_fullsized":"2023-06-22T08:00:00.755246+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3,1],"acting":[7,4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871513+0000","last_change":"2023-06-22T07:59:49.653053+0000","last_active":"2023-06-22T07:59:59.871513+0000","last_peered":"2023-06-22T07:59:59.871513+0000","last_clean":"2023-06-22T07:59:59.871513+0000","last_became_active":"2023-06-22T07:59:49.652861+0000","last_became_peered":"2023-06-22T07:59:49.652861+0000","last_unstale":"2023-06-22T07:59:59.871513+0000","last_undegraded":"2023-06-22T07:59:59.871513+0000","last_fullsized":"2023-06-22T07:59:59.871513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:28:40.388367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158202+0000","last_change":"2023-06-22T07:59:54.645403+0000","last_active":"2023-06-22T08:00:00.158202+0000","last_peered":"2023-06-22T08:00:00.158202+0000","last_clean":"2023-06-22T08:00:00.158202+0000","last_became_active":"2023-06-22T07:59:54.644973+0000","last_became_peered":"2023-06-22T07:59:54.644973+0000","last_unstale":"2023-06-22T08:00:00.158202+0000","last_undegraded":"2023-06-22T08:00:00.158202+0000","last_fullsized":"2023-06-22T08:00:00.158202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:15.910358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754101+0000","last_change":"2023-06-22T07:59:55.161484+0000","last_active":"2023-06-22T08:00:00.754101+0000","last_peered":"2023-06-22T08:00:00.754101+0000","last_clean":"2023-06-22T08:00:00.754101+0000","last_became_active":"2023-06-22T07:59:55.161194+0000","last_became_peered":"2023-06-22T07:59:55.161194+0000","last_unstale":"2023-06-22T08:00:00.754101+0000","last_undegraded":"2023-06-22T08:00:00.754101+0000","last_fullsized":"2023-06-22T08:00:00.754101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754988+0000","last_change":"2023-06-22T07:59:59.198241+0000","last_active":"2023-06-22T08:00:00.754988+0000","last_peered":"2023-06-22T08:00:00.754988+0000","last_clean":"2023-06-22T08:00:00.754988+0000","last_became_active":"2023-06-22T07:59:59.197689+0000","last_became_peered":"2023-06-22T07:59:59.197689+0000","last_unstale":"2023-06-22T08:00:00.754988+0000","last_undegraded":"2023-06-22T08:00:00.754988+0000","last_fullsized":"2023-06-22T08:00:00.754988+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:39.513874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4,1],"acting":[7,0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753886+0000","last_change":"2023-06-22T07:59:49.450044+0000","last_active":"2023-06-22T08:00:00.753886+0000","last_peered":"2023-06-22T08:00:00.753886+0000","last_clean":"2023-06-22T08:00:00.753886+0000","last_became_active":"2023-06-22T07:59:49.449731+0000","last_became_peered":"2023-06-22T07:59:49.449731+0000","last_unstale":"2023-06-22T08:00:00.753886+0000","last_undegraded":"2023-06-22T08:00:00.753886+0000","last_fullsized":"2023-06-22T08:00:00.753886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:40.262337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708613+0000","last_change":"2023-06-22T07:59:54.640462+0000","last_active":"2023-06-22T07:59:59.708613+0000","last_peered":"2023-06-22T07:59:59.708613+0000","last_clean":"2023-06-22T07:59:59.708613+0000","last_became_active":"2023-06-22T07:59:54.640123+0000","last_became_peered":"2023-06-22T07:59:54.640123+0000","last_unstale":"2023-06-22T07:59:59.708613+0000","last_undegraded":"2023-06-22T07:59:59.708613+0000","last_fullsized":"2023-06-22T07:59:59.708613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:20.316921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755350+0000","last_change":"2023-06-22T07:59:55.164651+0000","last_active":"2023-06-22T08:00:00.755350+0000","last_peered":"2023-06-22T08:00:00.755350+0000","last_clean":"2023-06-22T08:00:00.755350+0000","last_became_active":"2023-06-22T07:59:55.164235+0000","last_became_peered":"2023-06-22T07:59:55.164235+0000","last_unstale":"2023-06-22T08:00:00.755350+0000","last_undegraded":"2023-06-22T08:00:00.755350+0000","last_fullsized":"2023-06-22T08:00:00.755350+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:32.924136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157741+0000","last_change":"2023-06-22T07:59:59.204779+0000","last_active":"2023-06-22T08:00:00.157741+0000","last_peered":"2023-06-22T08:00:00.157741+0000","last_clean":"2023-06-22T08:00:00.157741+0000","last_became_active":"2023-06-22T07:59:59.204288+0000","last_became_peered":"2023-06-22T07:59:59.204288+0000","last_unstale":"2023-06-22T08:00:00.157741+0000","last_undegraded":"2023-06-22T08:00:00.157741+0000","last_fullsized":"2023-06-22T08:00:00.157741+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871497+0000","last_change":"2023-06-22T07:59:49.657008+0000","last_active":"2023-06-22T07:59:59.871497+0000","last_peered":"2023-06-22T07:59:59.871497+0000","last_clean":"2023-06-22T07:59:59.871497+0000","last_became_active":"2023-06-22T07:59:49.656752+0000","last_became_peered":"2023-06-22T07:59:49.656752+0000","last_unstale":"2023-06-22T07:59:59.871497+0000","last_undegraded":"2023-06-22T07:59:59.871497+0000","last_fullsized":"2023-06-22T07:59:59.871497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:50:37.639134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":185,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155134+0000","last_change":"2023-06-22T07:59:42.501363+0000","last_active":"2023-06-22T08:00:00.155134+0000","last_peered":"2023-06-22T08:00:00.155134+0000","last_clean":"2023-06-22T08:00:00.155134+0000","last_became_active":"2023-06-22T07:59:42.500320+0000","last_became_peered":"2023-06-22T07:59:42.500320+0000","last_unstale":"2023-06-22T08:00:00.155134+0000","last_undegraded":"2023-06-22T08:00:00.155134+0000","last_fullsized":"2023-06-22T08:00:00.155134+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_clean_scrub_stamp":"2023-06-22T07:59:41.381475+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:08:53.784030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753785+0000","last_change":"2023-06-22T07:59:54.645031+0000","last_active":"2023-06-22T08:00:00.753785+0000","last_peered":"2023-06-22T08:00:00.753785+0000","last_clean":"2023-06-22T08:00:00.753785+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:00.753785+0000","last_undegraded":"2023-06-22T08:00:00.753785+0000","last_fullsized":"2023-06-22T08:00:00.753785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:42.766407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872074+0000","last_change":"2023-06-22T07:59:55.162799+0000","last_active":"2023-06-22T07:59:59.872074+0000","last_peered":"2023-06-22T07:59:59.872074+0000","last_clean":"2023-06-22T07:59:59.872074+0000","last_became_active":"2023-06-22T07:59:55.162501+0000","last_became_peered":"2023-06-22T07:59:55.162501+0000","last_unstale":"2023-06-22T07:59:59.872074+0000","last_undegraded":"2023-06-22T07:59:59.872074+0000","last_fullsized":"2023-06-22T07:59:59.872074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:57:14.305897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871805+0000","last_change":"2023-06-22T07:59:59.205097+0000","last_active":"2023-06-22T07:59:59.871805+0000","last_peered":"2023-06-22T07:59:59.871805+0000","last_clean":"2023-06-22T07:59:59.871805+0000","last_became_active":"2023-06-22T07:59:59.201465+0000","last_became_peered":"2023-06-22T07:59:59.201465+0000","last_unstale":"2023-06-22T07:59:59.871805+0000","last_undegraded":"2023-06-22T07:59:59.871805+0000","last_fullsized":"2023-06-22T07:59:59.871805+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:17:45.352998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,0],"acting":[3,7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871056+0000","last_change":"2023-06-22T07:59:49.442558+0000","last_active":"2023-06-22T07:59:59.871056+0000","last_peered":"2023-06-22T07:59:59.871056+0000","last_clean":"2023-06-22T07:59:59.871056+0000","last_became_active":"2023-06-22T07:59:49.442154+0000","last_became_peered":"2023-06-22T07:59:49.442154+0000","last_unstale":"2023-06-22T07:59:59.871056+0000","last_undegraded":"2023-06-22T07:59:59.871056+0000","last_fullsized":"2023-06-22T07:59:59.871056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:46:16.233204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753311+0000","last_change":"2023-06-22T07:59:54.641746+0000","last_active":"2023-06-22T08:00:00.753311+0000","last_peered":"2023-06-22T08:00:00.753311+0000","last_clean":"2023-06-22T08:00:00.753311+0000","last_became_active":"2023-06-22T07:59:54.641431+0000","last_became_peered":"2023-06-22T07:59:54.641431+0000","last_unstale":"2023-06-22T08:00:00.753311+0000","last_undegraded":"2023-06-22T08:00:00.753311+0000","last_fullsized":"2023-06-22T08:00:00.753311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709468+0000","last_change":"2023-06-22T07:59:55.163214+0000","last_active":"2023-06-22T07:59:59.709468+0000","last_peered":"2023-06-22T07:59:59.709468+0000","last_clean":"2023-06-22T07:59:59.709468+0000","last_became_active":"2023-06-22T07:59:55.162907+0000","last_became_peered":"2023-06-22T07:59:55.162907+0000","last_unstale":"2023-06-22T07:59:59.709468+0000","last_undegraded":"2023-06-22T07:59:59.709468+0000","last_fullsized":"2023-06-22T07:59:59.709468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:46.988703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157450+0000","last_change":"2023-06-22T07:59:59.198945+0000","last_active":"2023-06-22T08:00:00.157450+0000","last_peered":"2023-06-22T08:00:00.157450+0000","last_clean":"2023-06-22T08:00:00.157450+0000","last_became_active":"2023-06-22T07:59:59.198561+0000","last_became_peered":"2023-06-22T07:59:59.198561+0000","last_unstale":"2023-06-22T08:00:00.157450+0000","last_undegraded":"2023-06-22T08:00:00.157450+0000","last_fullsized":"2023-06-22T08:00:00.157450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:06.135651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2,5],"acting":[0,7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871842+0000","last_change":"2023-06-22T07:59:49.442246+0000","last_active":"2023-06-22T07:59:59.871842+0000","last_peered":"2023-06-22T07:59:59.871842+0000","last_clean":"2023-06-22T07:59:59.871842+0000","last_became_active":"2023-06-22T07:59:49.442001+0000","last_became_peered":"2023-06-22T07:59:49.442001+0000","last_unstale":"2023-06-22T07:59:59.871842+0000","last_undegraded":"2023-06-22T07:59:59.871842+0000","last_fullsized":"2023-06-22T07:59:59.871842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:27:24.457978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754254+0000","last_change":"2023-06-22T07:59:54.643272+0000","last_active":"2023-06-22T08:00:00.754254+0000","last_peered":"2023-06-22T08:00:00.754254+0000","last_clean":"2023-06-22T08:00:00.754254+0000","last_became_active":"2023-06-22T07:59:54.642925+0000","last_became_peered":"2023-06-22T07:59:54.642925+0000","last_unstale":"2023-06-22T08:00:00.754254+0000","last_undegraded":"2023-06-22T08:00:00.754254+0000","last_fullsized":"2023-06-22T08:00:00.754254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:13.625768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754231+0000","last_change":"2023-06-22T07:59:55.167307+0000","last_active":"2023-06-22T08:00:00.754231+0000","last_peered":"2023-06-22T08:00:00.754231+0000","last_clean":"2023-06-22T08:00:00.754231+0000","last_became_active":"2023-06-22T07:59:55.167026+0000","last_became_peered":"2023-06-22T07:59:55.167026+0000","last_unstale":"2023-06-22T08:00:00.754231+0000","last_undegraded":"2023-06-22T08:00:00.754231+0000","last_fullsized":"2023-06-22T08:00:00.754231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871379+0000","last_change":"2023-06-22T07:59:59.208263+0000","last_active":"2023-06-22T07:59:59.871379+0000","last_peered":"2023-06-22T07:59:59.871379+0000","last_clean":"2023-06-22T07:59:59.871379+0000","last_became_active":"2023-06-22T07:59:59.207691+0000","last_became_peered":"2023-06-22T07:59:59.207691+0000","last_unstale":"2023-06-22T07:59:59.871379+0000","last_undegraded":"2023-06-22T07:59:59.871379+0000","last_fullsized":"2023-06-22T07:59:59.871379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:37:57.170857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,2],"acting":[3,7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"27'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755549+0000","last_change":"2023-06-22T07:59:54.644570+0000","last_active":"2023-06-22T08:00:00.755549+0000","last_peered":"2023-06-22T08:00:00.755549+0000","last_clean":"2023-06-22T08:00:00.755549+0000","last_became_active":"2023-06-22T07:59:54.644218+0000","last_became_peered":"2023-06-22T07:59:54.644218+0000","last_unstale":"2023-06-22T08:00:00.755549+0000","last_undegraded":"2023-06-22T08:00:00.755549+0000","last_fullsized":"2023-06-22T08:00:00.755549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:05:00.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754571+0000","last_change":"2023-06-22T07:59:49.655601+0000","last_active":"2023-06-22T08:00:00.754571+0000","last_peered":"2023-06-22T08:00:00.754571+0000","last_clean":"2023-06-22T08:00:00.754571+0000","last_became_active":"2023-06-22T07:59:49.655328+0000","last_became_peered":"2023-06-22T07:59:49.655328+0000","last_unstale":"2023-06-22T08:00:00.754571+0000","last_undegraded":"2023-06-22T08:00:00.754571+0000","last_fullsized":"2023-06-22T08:00:00.754571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:29:09.967964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755018+0000","last_change":"2023-06-22T07:59:55.163241+0000","last_active":"2023-06-22T08:00:00.755018+0000","last_peered":"2023-06-22T08:00:00.755018+0000","last_clean":"2023-06-22T08:00:00.755018+0000","last_became_active":"2023-06-22T07:59:55.162949+0000","last_became_peered":"2023-06-22T07:59:55.162949+0000","last_unstale":"2023-06-22T08:00:00.755018+0000","last_undegraded":"2023-06-22T08:00:00.755018+0000","last_fullsized":"2023-06-22T08:00:00.755018+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:51.267726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755485+0000","last_change":"2023-06-22T07:59:59.201187+0000","last_active":"2023-06-22T08:00:00.755485+0000","last_peered":"2023-06-22T08:00:00.755485+0000","last_clean":"2023-06-22T08:00:00.755485+0000","last_became_active":"2023-06-22T07:59:59.200688+0000","last_became_peered":"2023-06-22T07:59:59.200688+0000","last_unstale":"2023-06-22T08:00:00.755485+0000","last_undegraded":"2023-06-22T08:00:00.755485+0000","last_fullsized":"2023-06-22T08:00:00.755485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:52:15.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2,6],"acting":[4,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871926+0000","last_change":"2023-06-22T07:59:49.655839+0000","last_active":"2023-06-22T07:59:59.871926+0000","last_peered":"2023-06-22T07:59:59.871926+0000","last_clean":"2023-06-22T07:59:59.871926+0000","last_became_active":"2023-06-22T07:59:49.655494+0000","last_became_peered":"2023-06-22T07:59:49.655494+0000","last_unstale":"2023-06-22T07:59:59.871926+0000","last_undegraded":"2023-06-22T07:59:59.871926+0000","last_fullsized":"2023-06-22T07:59:59.871926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:01:23.950347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754455+0000","last_change":"2023-06-22T07:59:54.641464+0000","last_active":"2023-06-22T08:00:00.754455+0000","last_peered":"2023-06-22T08:00:00.754455+0000","last_clean":"2023-06-22T08:00:00.754455+0000","last_became_active":"2023-06-22T07:59:54.641161+0000","last_became_peered":"2023-06-22T07:59:54.641161+0000","last_unstale":"2023-06-22T08:00:00.754455+0000","last_undegraded":"2023-06-22T08:00:00.754455+0000","last_fullsized":"2023-06-22T08:00:00.754455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:46.346592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708316+0000","last_change":"2023-06-22T07:59:55.164629+0000","last_active":"2023-06-22T07:59:59.708316+0000","last_peered":"2023-06-22T07:59:59.708316+0000","last_clean":"2023-06-22T07:59:59.708316+0000","last_became_active":"2023-06-22T07:59:55.164329+0000","last_became_peered":"2023-06-22T07:59:55.164329+0000","last_unstale":"2023-06-22T07:59:59.708316+0000","last_undegraded":"2023-06-22T07:59:59.708316+0000","last_fullsized":"2023-06-22T07:59:59.708316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754950+0000","last_change":"2023-06-22T07:59:59.201778+0000","last_active":"2023-06-22T08:00:00.754950+0000","last_peered":"2023-06-22T08:00:00.754950+0000","last_clean":"2023-06-22T08:00:00.754950+0000","last_became_active":"2023-06-22T07:59:59.201326+0000","last_became_peered":"2023-06-22T07:59:59.201326+0000","last_unstale":"2023-06-22T08:00:00.754950+0000","last_undegraded":"2023-06-22T08:00:00.754950+0000","last_fullsized":"2023-06-22T08:00:00.754950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:17:31.817024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"27'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754232+0000","last_change":"2023-06-22T07:59:54.644816+0000","last_active":"2023-06-22T08:00:00.754232+0000","last_peered":"2023-06-22T08:00:00.754232+0000","last_clean":"2023-06-22T08:00:00.754232+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:00.754232+0000","last_undegraded":"2023-06-22T08:00:00.754232+0000","last_fullsized":"2023-06-22T08:00:00.754232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:33.819054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753735+0000","last_change":"2023-06-22T07:59:55.167979+0000","last_active":"2023-06-22T08:00:00.753735+0000","last_peered":"2023-06-22T08:00:00.753735+0000","last_clean":"2023-06-22T08:00:00.753735+0000","last_became_active":"2023-06-22T07:59:55.163775+0000","last_became_peered":"2023-06-22T07:59:55.163775+0000","last_unstale":"2023-06-22T08:00:00.753735+0000","last_undegraded":"2023-06-22T08:00:00.753735+0000","last_fullsized":"2023-06-22T08:00:00.753735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155135+0000","last_change":"2023-06-22T07:59:59.194581+0000","last_active":"2023-06-22T08:00:00.155135+0000","last_peered":"2023-06-22T08:00:00.155135+0000","last_clean":"2023-06-22T08:00:00.155135+0000","last_became_active":"2023-06-22T07:59:59.194135+0000","last_became_peered":"2023-06-22T07:59:59.194135+0000","last_unstale":"2023-06-22T08:00:00.155135+0000","last_undegraded":"2023-06-22T08:00:00.155135+0000","last_fullsized":"2023-06-22T08:00:00.155135+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"27'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.156863+0000","last_change":"2023-06-22T07:59:54.651138+0000","last_active":"2023-06-22T08:00:00.156863+0000","last_peered":"2023-06-22T08:00:00.156863+0000","last_clean":"2023-06-22T08:00:00.156863+0000","last_became_active":"2023-06-22T07:59:54.650803+0000","last_became_peered":"2023-06-22T07:59:54.650803+0000","last_unstale":"2023-06-22T08:00:00.156863+0000","last_undegraded":"2023-06-22T08:00:00.156863+0000","last_fullsized":"2023-06-22T08:00:00.156863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:45.982711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708617+0000","last_change":"2023-06-22T07:59:55.165814+0000","last_active":"2023-06-22T07:59:59.708617+0000","last_peered":"2023-06-22T07:59:59.708617+0000","last_clean":"2023-06-22T07:59:59.708617+0000","last_became_active":"2023-06-22T07:59:55.165270+0000","last_became_peered":"2023-06-22T07:59:55.165270+0000","last_unstale":"2023-06-22T07:59:59.708617+0000","last_undegraded":"2023-06-22T07:59:59.708617+0000","last_fullsized":"2023-06-22T07:59:59.708617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708465+0000","last_change":"2023-06-22T07:59:59.212042+0000","last_active":"2023-06-22T07:59:59.708465+0000","last_peered":"2023-06-22T07:59:59.708465+0000","last_clean":"2023-06-22T07:59:59.708465+0000","last_became_active":"2023-06-22T07:59:59.211666+0000","last_became_peered":"2023-06-22T07:59:59.211666+0000","last_unstale":"2023-06-22T07:59:59.708465+0000","last_undegraded":"2023-06-22T07:59:59.708465+0000","last_fullsized":"2023-06-22T07:59:59.708465+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,4],"acting":[2,7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158226+0000","last_change":"2023-06-22T07:59:54.650251+0000","last_active":"2023-06-22T08:00:00.158226+0000","last_peered":"2023-06-22T08:00:00.158226+0000","last_clean":"2023-06-22T08:00:00.158226+0000","last_became_active":"2023-06-22T07:59:54.649993+0000","last_became_peered":"2023-06-22T07:59:54.649993+0000","last_unstale":"2023-06-22T08:00:00.158226+0000","last_undegraded":"2023-06-22T08:00:00.158226+0000","last_fullsized":"2023-06-22T08:00:00.158226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:20:20.733896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708375+0000","last_change":"2023-06-22T07:59:55.172595+0000","last_active":"2023-06-22T07:59:59.708375+0000","last_peered":"2023-06-22T07:59:59.708375+0000","last_clean":"2023-06-22T07:59:59.708375+0000","last_became_active":"2023-06-22T07:59:55.165873+0000","last_became_peered":"2023-06-22T07:59:55.165873+0000","last_unstale":"2023-06-22T07:59:59.708375+0000","last_undegraded":"2023-06-22T07:59:59.708375+0000","last_fullsized":"2023-06-22T07:59:59.708375+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:02.592807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154969+0000","last_change":"2023-06-22T07:59:59.192708+0000","last_active":"2023-06-22T08:00:00.154969+0000","last_peered":"2023-06-22T08:00:00.154969+0000","last_clean":"2023-06-22T08:00:00.154969+0000","last_became_active":"2023-06-22T07:59:59.191693+0000","last_became_peered":"2023-06-22T07:59:59.191693+0000","last_unstale":"2023-06-22T08:00:00.154969+0000","last_undegraded":"2023-06-22T08:00:00.154969+0000","last_fullsized":"2023-06-22T08:00:00.154969+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T12:59:33.454962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5,7],"acting":[1,0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709255+0000","last_change":"2023-06-22T07:59:54.651179+0000","last_active":"2023-06-22T07:59:59.709255+0000","last_peered":"2023-06-22T07:59:59.709255+0000","last_clean":"2023-06-22T07:59:59.709255+0000","last_became_active":"2023-06-22T07:59:54.650852+0000","last_became_peered":"2023-06-22T07:59:54.650852+0000","last_unstale":"2023-06-22T07:59:59.709255+0000","last_undegraded":"2023-06-22T07:59:59.709255+0000","last_fullsized":"2023-06-22T07:59:59.709255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:01.351861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754043+0000","last_change":"2023-06-22T07:59:55.163050+0000","last_active":"2023-06-22T08:00:00.754043+0000","last_peered":"2023-06-22T08:00:00.754043+0000","last_clean":"2023-06-22T08:00:00.754043+0000","last_became_active":"2023-06-22T07:59:55.162754+0000","last_became_peered":"2023-06-22T07:59:55.162754+0000","last_unstale":"2023-06-22T08:00:00.754043+0000","last_undegraded":"2023-06-22T08:00:00.754043+0000","last_fullsized":"2023-06-22T08:00:00.754043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754797+0000","last_change":"2023-06-22T07:59:59.209558+0000","last_active":"2023-06-22T08:00:00.754797+0000","last_peered":"2023-06-22T08:00:00.754797+0000","last_clean":"2023-06-22T08:00:00.754797+0000","last_became_active":"2023-06-22T07:59:59.209129+0000","last_became_peered":"2023-06-22T07:59:59.209129+0000","last_unstale":"2023-06-22T08:00:00.754797+0000","last_undegraded":"2023-06-22T08:00:00.754797+0000","last_fullsized":"2023-06-22T08:00:00.754797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:21.581063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1,0],"acting":[4,3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154714+0000","last_change":"2023-06-22T07:59:54.652059+0000","last_active":"2023-06-22T08:00:00.154714+0000","last_peered":"2023-06-22T08:00:00.154714+0000","last_clean":"2023-06-22T08:00:00.154714+0000","last_became_active":"2023-06-22T07:59:54.651720+0000","last_became_peered":"2023-06-22T07:59:54.651720+0000","last_unstale":"2023-06-22T08:00:00.154714+0000","last_undegraded":"2023-06-22T08:00:00.154714+0000","last_fullsized":"2023-06-22T08:00:00.154714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:31.908657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708158+0000","last_change":"2023-06-22T07:59:55.172794+0000","last_active":"2023-06-22T07:59:59.708158+0000","last_peered":"2023-06-22T07:59:59.708158+0000","last_clean":"2023-06-22T07:59:59.708158+0000","last_became_active":"2023-06-22T07:59:55.172527+0000","last_became_peered":"2023-06-22T07:59:55.172527+0000","last_unstale":"2023-06-22T07:59:59.708158+0000","last_undegraded":"2023-06-22T07:59:59.708158+0000","last_fullsized":"2023-06-22T07:59:59.708158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155457+0000","last_change":"2023-06-22T07:59:55.165758+0000","last_active":"2023-06-22T08:00:00.155457+0000","last_peered":"2023-06-22T08:00:00.155457+0000","last_clean":"2023-06-22T08:00:00.155457+0000","last_became_active":"2023-06-22T07:59:55.165117+0000","last_became_peered":"2023-06-22T07:59:55.165117+0000","last_unstale":"2023-06-22T08:00:00.155457+0000","last_undegraded":"2023-06-22T08:00:00.155457+0000","last_fullsized":"2023-06-22T08:00:00.155457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:02.357853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753486+0000","last_change":"2023-06-22T07:59:55.162439+0000","last_active":"2023-06-22T08:00:00.753486+0000","last_peered":"2023-06-22T08:00:00.753486+0000","last_clean":"2023-06-22T08:00:00.753486+0000","last_became_active":"2023-06-22T07:59:55.162072+0000","last_became_peered":"2023-06-22T07:59:55.162072+0000","last_unstale":"2023-06-22T08:00:00.753486+0000","last_undegraded":"2023-06-22T08:00:00.753486+0000","last_fullsized":"2023-06-22T08:00:00.753486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157229+0000","last_change":"2023-06-22T07:59:55.159299+0000","last_active":"2023-06-22T08:00:00.157229+0000","last_peered":"2023-06-22T08:00:00.157229+0000","last_clean":"2023-06-22T08:00:00.157229+0000","last_became_active":"2023-06-22T07:59:55.158893+0000","last_became_peered":"2023-06-22T07:59:55.158893+0000","last_unstale":"2023-06-22T08:00:00.157229+0000","last_undegraded":"2023-06-22T08:00:00.157229+0000","last_fullsized":"2023-06-22T08:00:00.157229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709716+0000","last_change":"2023-06-22T07:59:55.162520+0000","last_active":"2023-06-22T07:59:59.709716+0000","last_peered":"2023-06-22T07:59:59.709716+0000","last_clean":"2023-06-22T07:59:59.709716+0000","last_became_active":"2023-06-22T07:59:55.162211+0000","last_became_peered":"2023-06-22T07:59:55.162211+0000","last_unstale":"2023-06-22T07:59:59.709716+0000","last_undegraded":"2023-06-22T07:59:59.709716+0000","last_fullsized":"2023-06-22T07:59:59.709716+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:13:00.662547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708947+0000","last_change":"2023-06-22T07:59:55.165364+0000","last_active":"2023-06-22T07:59:59.708947+0000","last_peered":"2023-06-22T07:59:59.708947+0000","last_clean":"2023-06-22T07:59:59.708947+0000","last_became_active":"2023-06-22T07:59:55.165021+0000","last_became_peered":"2023-06-22T07:59:55.165021+0000","last_unstale":"2023-06-22T07:59:59.708947+0000","last_undegraded":"2023-06-22T07:59:59.708947+0000","last_fullsized":"2023-06-22T07:59:59.708947+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:09:07.008945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4592,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476748,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":127352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]}]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":319488,"data_stored":127456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":299008,"data_stored":123558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":905216,"data_stored":714600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":897024,"data_stored":714508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":303104,"data_stored":123648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]}]},{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":299008,"data_stored":123558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542155,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":123704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-22T08:00:24.615 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-06-22T08:00:24.616 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-06-22T08:00:24.616 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-06-22T08:00:24.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-06-22T08:00:24.617 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-06-22T08:00:24.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.841+0000 7f48f37ff700 1 Processor -- start 2023-06-22T08:00:24.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.841+0000 7f48f37ff700 1 -- start start 2023-06-22T08:00:24.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.842+0000 7f48f37ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48ec15a320 0x7f48ec0c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48f37ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec0c6a60 con 0x7f48ec0c3a40 2023-06-22T08:00:24.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48f37ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec14d690 con 0x7f48ec06d050 2023-06-22T08:00:24.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48f37ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ec157570 con 0x7f48ec15a320 2023-06-22T08:00:24.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48f159b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f48ec0c3a40 0x7f48ec159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40054/0 (socket says 172.21.15.23:40054) 2023-06-22T08:00:24.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48f159b700 1 -- 172.21.15.23:0/3010611457 learned_addr learned my addr 172.21.15.23:0/3010611457 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:24.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48f0d9a700 1 --2- 172.21.15.23:0/3010611457 >> v2:172.21.15.67:3300/0 conn(0x7f48ec15a320 0x7f48ec0c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48e3fff700 1 -- 172.21.15.23:0/3010611457 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862086460 0 0) 0x7f48ec0c6a60 con 0x7f48ec0c3a40 2023-06-22T08:00:24.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48e3fff700 1 -- 172.21.15.23:0/3010611457 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d80035b0 con 0x7f48ec0c3a40 2023-06-22T08:00:24.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48e3fff700 1 -- 172.21.15.23:0/3010611457 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203575693 0 0) 0x7f48ec14d690 con 0x7f48ec06d050 2023-06-22T08:00:24.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48e3fff700 1 -- 172.21.15.23:0/3010611457 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ec0c6a60 con 0x7f48ec06d050 2023-06-22T08:00:24.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48f0d9a700 1 -- 172.21.15.23:0/3010611457 >> v1:172.21.15.67:6789/0 conn(0x7f48ec06d050 legacy=0x7f48ec160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48f0d9a700 1 -- 172.21.15.23:0/3010611457 >> v1:172.21.15.23:6789/0 conn(0x7f48ec0c3a40 legacy=0x7f48ec159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.843+0000 7f48f0d9a700 1 -- 172.21.15.23:0/3010611457 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec0c6430 con 0x7f48ec15a320 2023-06-22T08:00:24.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.844+0000 7f48f0d9a700 1 --2- 172.21.15.23:0/3010611457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48ec15a320 0x7f48ec0c3e10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f48dc009840 tx=0x7f48dc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17777d276d95bda4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:24.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.844+0000 7f48e3fff700 1 -- 172.21.15.23:0/3010611457 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc011830 con 0x7f48ec15a320 2023-06-22T08:00:24.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.844+0000 7f48e3fff700 1 -- 172.21.15.23:0/3010611457 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48dc011990 con 0x7f48ec15a320 2023-06-22T08:00:24.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.844+0000 7f48e3fff700 1 -- 172.21.15.23:0/3010611457 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc011d20 con 0x7f48ec15a320 2023-06-22T08:00:24.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.845+0000 7f48f37ff700 1 -- 172.21.15.23:0/3010611457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48ec15a320 msgr2=0x7f48ec0c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.845+0000 7f48f37ff700 1 --2- 172.21.15.23:0/3010611457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48ec15a320 0x7f48ec0c3e10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f48dc009840 tx=0x7f48dc009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:24.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.845+0000 7f48f37ff700 1 -- 172.21.15.23:0/3010611457 shutdown_connections 2023-06-22T08:00:24.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.845+0000 7f48f37ff700 1 --2- 172.21.15.23:0/3010611457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48ec15a320 0x7f48ec0c3e10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.845+0000 7f48f37ff700 1 -- 172.21.15.23:0/3010611457 >> 172.21.15.23:0/3010611457 conn(0x7f48ec0ca3f0 msgr2=0x7f48ec0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:24.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.845+0000 7f48f37ff700 1 -- 172.21.15.23:0/3010611457 shutdown_connections 2023-06-22T08:00:24.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.845+0000 7f48f37ff700 1 -- 172.21.15.23:0/3010611457 wait complete. 2023-06-22T08:00:24.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.847+0000 7f48f37ff700 1 Processor -- start 2023-06-22T08:00:24.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.847+0000 7f48f37ff700 1 -- start start 2023-06-22T08:00:24.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f37ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48ec06d050 0x7f48ec1555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f37ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48ec0c3a40 0x7f48ec1585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f37ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f48ec15a320 0x7f48ec1436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f37ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec146510 con 0x7f48ec0c3a40 2023-06-22T08:00:24.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f37ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec157830 con 0x7f48ec06d050 2023-06-22T08:00:24.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f37ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec1ef0d0 con 0x7f48ec15a320 2023-06-22T08:00:24.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f0d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48ec0c3a40 0x7f48ec1585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f0d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48ec0c3a40 0x7f48ec1585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38764/0 (socket says 172.21.15.23:38764) 2023-06-22T08:00:24.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f0d9a700 1 -- 172.21.15.23:0/4039112374 learned_addr learned my addr 172.21.15.23:0/4039112374 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:24.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f159b700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48ec06d050 0x7f48ec1555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f1d9c700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f48ec15a320 0x7f48ec1436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f0d9a700 1 -- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f48ec15a320 msgr2=0x7f48ec1436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f0d9a700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f48ec15a320 0x7f48ec1436e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f0d9a700 1 -- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48ec06d050 msgr2=0x7f48ec1555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:24.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f0d9a700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48ec06d050 0x7f48ec1555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:24.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.848+0000 7f48f0d9a700 1 -- 172.21.15.23:0/4039112374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48dc0094a0 con 0x7f48ec0c3a40 2023-06-22T08:00:24.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.849+0000 7f48f0d9a700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48ec0c3a40 0x7f48ec1585d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f48dc009b90 tx=0x7f48dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:24.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.849+0000 7f48e1ffb700 1 -- 172.21.15.23:0/4039112374 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc01a970 con 0x7f48ec0c3a40 2023-06-22T08:00:24.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.849+0000 7f48e1ffb700 1 -- 172.21.15.23:0/4039112374 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48dc01aad0 con 0x7f48ec0c3a40 2023-06-22T08:00:24.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.849+0000 7f48e1ffb700 1 -- 172.21.15.23:0/4039112374 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc01ae40 con 0x7f48ec0c3a40 2023-06-22T08:00:24.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.849+0000 7f48f37ff700 1 -- 172.21.15.23:0/4039112374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ec1ef3b0 con 0x7f48ec0c3a40 2023-06-22T08:00:24.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.849+0000 7f48f37ff700 1 -- 172.21.15.23:0/4039112374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ec156590 con 0x7f48ec0c3a40 2023-06-22T08:00:24.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.850+0000 7f48f37ff700 1 -- 172.21.15.23:0/4039112374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ec0c7930 con 0x7f48ec0c3a40 2023-06-22T08:00:24.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.851+0000 7f48e1ffb700 1 -- 172.21.15.23:0/4039112374 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f48dc02d450 con 0x7f48ec0c3a40 2023-06-22T08:00:24.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.852+0000 7f48e1ffb700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48d006ace0 0x7f48d006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:24.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.852+0000 7f48e1ffb700 1 -- 172.21.15.23:0/4039112374 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f48dc08a9b0 con 0x7f48ec0c3a40 2023-06-22T08:00:24.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.852+0000 7f48f159b700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48d006ace0 0x7f48d006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:24.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.853+0000 7f48f159b700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48d006ace0 0x7f48d006d190 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007fd0 tx=0x7f48e8007e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:24.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:24.857+0000 7f48e1ffb700 1 -- 172.21.15.23:0/4039112374 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f48dc055920 con 0x7f48ec0c3a40 2023-06-22T08:00:25.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.118+0000 7f48f37ff700 1 -- 172.21.15.23:0/4039112374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f48ec06cc00 con 0x7f48ec0c3a40 2023-06-22T08:00:25.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.121+0000 7f48e1ffb700 1 -- 172.21.15.23:0/4039112374 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v6) v1 ==== 74+0+170842 (secure 0 0 0) 0x7f48dc058f40 con 0x7f48ec0c3a40 2023-06-22T08:00:25.126 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 -- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48d006ace0 msgr2=0x7f48d006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48d006ace0 0x7f48d006d190 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007fd0 tx=0x7f48e8007e00 comp rx=0 tx=0).stop 2023-06-22T08:00:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 -- 172.21.15.23:0/4039112374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48ec0c3a40 msgr2=0x7f48ec1585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48ec0c3a40 0x7f48ec1585d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f48dc009b90 tx=0x7f48dc004750 comp rx=0 tx=0).stop 2023-06-22T08:00:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 -- 172.21.15.23:0/4039112374 shutdown_connections 2023-06-22T08:00:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f48ec15a320 0x7f48ec1436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48d006ace0 0x7f48d006d190 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48ec06d050 0x7f48ec1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 --2- 172.21.15.23:0/4039112374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48ec0c3a40 0x7f48ec1585d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 -- 172.21.15.23:0/4039112374 >> 172.21.15.23:0/4039112374 conn(0x7f48ec0ca3f0 msgr2=0x7f48ec0c7220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:25.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 -- 172.21.15.23:0/4039112374 shutdown_connections 2023-06-22T08:00:25.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.124+0000 7f48f37ff700 1 -- 172.21.15.23:0/4039112374 wait complete. 2023-06-22T08:00:25.145 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":6,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3180103139},{"type":"v1","addr":"172.21.15.67:6833","nonce":3180103139}]},"active_addr":"172.21.15.67:6833/3180103139","active_change":"2023-06-22T07:59:34.542361+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4102,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1285077356}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2828739182}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":4226742434}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":4042990870}]}]} 2023-06-22T08:00:25.149 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-06-22T08:00:25.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-06-22T08:00:25.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:25.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.345+0000 7f56483ff700 1 Processor -- start 2023-06-22T08:00:25.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.345+0000 7f56483ff700 1 -- start start 2023-06-22T08:00:25.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.346+0000 7f56483ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56400c9fd0 0x7f56400cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.346+0000 7f56483ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f56400c40b0 0x7f56400c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.346+0000 7f56483ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f564015c370 con 0x7f5640152ce0 2023-06-22T08:00:25.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.346+0000 7f56483ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56401530b0 con 0x7f56400c40b0 2023-06-22T08:00:25.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.347+0000 7f56483ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56400cf6f0 con 0x7f56400c9fd0 2023-06-22T08:00:25.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.347+0000 7f564699c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f56400c40b0 0x7f56400c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.347+0000 7f564619b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5640152ce0 0x7f564015c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40062/0 (socket says 172.21.15.23:40062) 2023-06-22T08:00:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.347+0000 7f564619b700 1 -- 172.21.15.23:0/2870824728 learned_addr learned my addr 172.21.15.23:0/2870824728 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.347+0000 7f564599a700 1 --2- 172.21.15.23:0/2870824728 >> v2:172.21.15.67:3300/0 conn(0x7f56400c9fd0 0x7f56400cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.347+0000 7f564699c700 1 -- 172.21.15.23:0/2870824728 >> v2:172.21.15.67:3300/0 conn(0x7f56400c9fd0 msgr2=0x7f56400cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:25.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.347+0000 7f564699c700 1 --2- 172.21.15.23:0/2870824728 >> v2:172.21.15.67:3300/0 conn(0x7f56400c9fd0 0x7f56400cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.347+0000 7f564699c700 1 -- 172.21.15.23:0/2870824728 >> v1:172.21.15.23:6789/0 conn(0x7f5640152ce0 legacy=0x7f564015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.347+0000 7f564699c700 1 -- 172.21.15.23:0/2870824728 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564015bda0 con 0x7f56400c40b0 2023-06-22T08:00:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.347+0000 7f5645199700 1 -- 172.21.15.23:0/2870824728 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961844656 0 0) 0x7f564015c370 con 0x7f5640152ce0 2023-06-22T08:00:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f564699c700 1 --2- 172.21.15.23:0/2870824728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 0x7f56400c4530 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f56400cc380 tx=0x7f563800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aef663d0a868afa server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f5645199700 1 -- 172.21.15.23:0/2870824728 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563800e070 con 0x7f56400c40b0 2023-06-22T08:00:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f5645199700 1 -- 172.21.15.23:0/2870824728 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563800bbd0 con 0x7f56400c40b0 2023-06-22T08:00:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f5645199700 1 -- 172.21.15.23:0/2870824728 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638012430 con 0x7f56400c40b0 2023-06-22T08:00:25.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f56483ff700 1 -- 172.21.15.23:0/2870824728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 msgr2=0x7f56400c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f56483ff700 1 --2- 172.21.15.23:0/2870824728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 0x7f56400c4530 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f56400cc380 tx=0x7f563800b330 comp rx=0 tx=0).stop 2023-06-22T08:00:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f56483ff700 1 -- 172.21.15.23:0/2870824728 shutdown_connections 2023-06-22T08:00:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f56483ff700 1 --2- 172.21.15.23:0/2870824728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 0x7f56400c4530 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f56483ff700 1 --2- 172.21.15.23:0/2870824728 >> v2:172.21.15.67:3300/0 conn(0x7f56400c9fd0 0x7f56400cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f56483ff700 1 -- 172.21.15.23:0/2870824728 >> 172.21.15.23:0/2870824728 conn(0x7f564006ae30 msgr2=0x7f564006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.348+0000 7f56483ff700 1 -- 172.21.15.23:0/2870824728 shutdown_connections 2023-06-22T08:00:25.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.349+0000 7f56483ff700 1 -- 172.21.15.23:0/2870824728 wait complete. 2023-06-22T08:00:25.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.350+0000 7f56483ff700 1 Processor -- start 2023-06-22T08:00:25.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.350+0000 7f56483ff700 1 -- start start 2023-06-22T08:00:25.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.351+0000 7f56483ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 0x7f56401ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.351+0000 7f56483ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f56400c9fd0 0x7f5640155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.351+0000 7f56483ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5640152ce0 0x7f56400b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.351+0000 7f56483ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56401eddb0 con 0x7f56400c40b0 2023-06-22T08:00:25.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.351+0000 7f56483ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56401f1120 con 0x7f5640152ce0 2023-06-22T08:00:25.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.351+0000 7f56483ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56400ccef0 con 0x7f56400c9fd0 2023-06-22T08:00:25.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.351+0000 7f564619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 0x7f56401ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.351+0000 7f564619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 0x7f56401ee7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38782/0 (socket says 172.21.15.23:38782) 2023-06-22T08:00:25.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.351+0000 7f564619b700 1 -- 172.21.15.23:0/2741630901 learned_addr learned my addr 172.21.15.23:0/2741630901 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:25.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f564699c700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5640152ce0 0x7f56400b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f564599a700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f56400c9fd0 0x7f5640155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f564619b700 1 -- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f56400c9fd0 msgr2=0x7f5640155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f564619b700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f56400c9fd0 0x7f5640155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f564619b700 1 -- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5640152ce0 msgr2=0x7f56400b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f564619b700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5640152ce0 0x7f56400b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f564619b700 1 -- 172.21.15.23:0/2741630901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5638009a70 con 0x7f56400c40b0 2023-06-22T08:00:25.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f564619b700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 0x7f56401ee7c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f563c00c550 tx=0x7f563c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:25.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f5636ffd700 1 -- 172.21.15.23:0/2741630901 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c004c80 con 0x7f56400c40b0 2023-06-22T08:00:25.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f5636ffd700 1 -- 172.21.15.23:0/2741630901 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c004e00 con 0x7f56400c40b0 2023-06-22T08:00:25.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f5636ffd700 1 -- 172.21.15.23:0/2741630901 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c007310 con 0x7f56400c40b0 2023-06-22T08:00:25.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.352+0000 7f56483ff700 1 -- 172.21.15.23:0/2741630901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56400cc9c0 con 0x7f56400c40b0 2023-06-22T08:00:25.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.353+0000 7f56483ff700 1 -- 172.21.15.23:0/2741630901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5640156030 con 0x7f56400c40b0 2023-06-22T08:00:25.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.354+0000 7f56483ff700 1 -- 172.21.15.23:0/2741630901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564004ea20 con 0x7f56400c40b0 2023-06-22T08:00:25.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.355+0000 7f5636ffd700 1 -- 172.21.15.23:0/2741630901 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f563c007470 con 0x7f56400c40b0 2023-06-22T08:00:25.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.356+0000 7f5636ffd700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f562406ace0 0x7f562406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.356+0000 7f5636ffd700 1 -- 172.21.15.23:0/2741630901 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f563c089420 con 0x7f56400c40b0 2023-06-22T08:00:25.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.356+0000 7f564599a700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f562406ace0 0x7f562406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.357+0000 7f564599a700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f562406ace0 0x7f562406d190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f56400c5020 tx=0x7f5630008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:25.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.361+0000 7f5636ffd700 1 -- 172.21.15.23:0/2741630901 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f563c054140 con 0x7f56400c40b0 2023-06-22T08:00:25.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.570+0000 7f56483ff700 1 -- 172.21.15.23:0/2741630901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56400b13d0 con 0x7f56400c40b0 2023-06-22T08:00:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.571+0000 7f5636ffd700 1 -- 172.21.15.23:0/2741630901 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f563c057760 con 0x7f56400c40b0 2023-06-22T08:00:25.575 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:25.575 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 -- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f562406ace0 msgr2=0x7f562406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f562406ace0 0x7f562406d190 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f56400c5020 tx=0x7f5630008040 comp rx=0 tx=0).stop 2023-06-22T08:00:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 -- 172.21.15.23:0/2741630901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 msgr2=0x7f56401ee7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 0x7f56401ee7c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f563c00c550 tx=0x7f563c00c860 comp rx=0 tx=0).stop 2023-06-22T08:00:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 -- 172.21.15.23:0/2741630901 shutdown_connections 2023-06-22T08:00:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f562406ace0 0x7f562406d190 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5640152ce0 0x7f56400b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56400c40b0 0x7f56401ee7c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 --2- 172.21.15.23:0/2741630901 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f56400c9fd0 0x7f5640155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 -- 172.21.15.23:0/2741630901 >> 172.21.15.23:0/2741630901 conn(0x7f564006ae30 msgr2=0x7f5640151ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 -- 172.21.15.23:0/2741630901 shutdown_connections 2023-06-22T08:00:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.574+0000 7f56483ff700 1 -- 172.21.15.23:0/2741630901 wait complete. 2023-06-22T08:00:25.595 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-06-22T08:00:25.596 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:25.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.796+0000 7f9a5c9ff700 1 Processor -- start 2023-06-22T08:00:25.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.796+0000 7f9a5c9ff700 1 -- start start 2023-06-22T08:00:25.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a5c9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9a58152ce0 0x7f9a580cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a5c9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a580c40b0 0x7f9a580c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a5c9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5815c370 con 0x7f9a580c9fd0 2023-06-22T08:00:25.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a5c9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a581530b0 con 0x7f9a58152ce0 2023-06-22T08:00:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a5c9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a580cf6f0 con 0x7f9a580c40b0 2023-06-22T08:00:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a5659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9a58152ce0 0x7f9a580cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a56d9d700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a580c40b0 0x7f9a580c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a5659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9a58152ce0 0x7f9a580cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38796/0 (socket says 172.21.15.23:38796) 2023-06-22T08:00:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a5659c700 1 -- 172.21.15.23:0/3214858852 learned_addr learned my addr 172.21.15.23:0/3214858852 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a4ffff700 1 --1- 172.21.15.23:0/3214858852 >> v1:172.21.15.67:6789/0 conn(0x7f9a580c9fd0 0x7f9a5815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39256/0 (socket says 172.21.15.23:39256) 2023-06-22T08:00:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a5659c700 1 -- 172.21.15.23:0/3214858852 >> v2:172.21.15.67:3300/0 conn(0x7f9a580c40b0 msgr2=0x7f9a580c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:25.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.798+0000 7f9a5659c700 1 --2- 172.21.15.23:0/3214858852 >> v2:172.21.15.67:3300/0 conn(0x7f9a580c40b0 0x7f9a580c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.799+0000 7f9a5659c700 1 -- 172.21.15.23:0/3214858852 >> v1:172.21.15.67:6789/0 conn(0x7f9a580c9fd0 legacy=0x7f9a5815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.799+0000 7f9a5659c700 1 -- 172.21.15.23:0/3214858852 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5815bda0 con 0x7f9a58152ce0 2023-06-22T08:00:25.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.799+0000 7f9a5659c700 1 --2- 172.21.15.23:0/3214858852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a58152ce0 0x7f9a580cf1b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9a580b4380 tx=0x7f9a48009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be7b327c1a8b6d8 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:25.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.799+0000 7f9a55d9b700 1 -- 172.21.15.23:0/3214858852 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a48011830 con 0x7f9a58152ce0 2023-06-22T08:00:25.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.799+0000 7f9a55d9b700 1 -- 172.21.15.23:0/3214858852 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a48011990 con 0x7f9a58152ce0 2023-06-22T08:00:25.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.799+0000 7f9a55d9b700 1 -- 172.21.15.23:0/3214858852 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a48011d20 con 0x7f9a58152ce0 2023-06-22T08:00:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.799+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/3214858852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a58152ce0 msgr2=0x7f9a580cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.800+0000 7f9a5c9ff700 1 --2- 172.21.15.23:0/3214858852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a58152ce0 0x7f9a580cf1b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9a580b4380 tx=0x7f9a48009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.800+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/3214858852 shutdown_connections 2023-06-22T08:00:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.800+0000 7f9a5c9ff700 1 --2- 172.21.15.23:0/3214858852 >> v2:172.21.15.67:3300/0 conn(0x7f9a580c40b0 0x7f9a580c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.800+0000 7f9a5c9ff700 1 --2- 172.21.15.23:0/3214858852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a58152ce0 0x7f9a580cf1b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.800+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/3214858852 >> 172.21.15.23:0/3214858852 conn(0x7f9a5806ae30 msgr2=0x7f9a5806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.800+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/3214858852 shutdown_connections 2023-06-22T08:00:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.800+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/3214858852 wait complete. 2023-06-22T08:00:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.801+0000 7f9a5c9ff700 1 Processor -- start 2023-06-22T08:00:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.802+0000 7f9a5c9ff700 1 -- start start 2023-06-22T08:00:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.802+0000 7f9a5c9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9a580c40b0 0x7f9a581e1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.802+0000 7f9a5c9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a580c9fd0 0x7f9a581e2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5c9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58152ce0 0x7f9a581e2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5c9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a58156040 con 0x7f9a580c9fd0 2023-06-22T08:00:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5c9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a581ee830 con 0x7f9a58152ce0 2023-06-22T08:00:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5c9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a581eed60 con 0x7f9a580c40b0 2023-06-22T08:00:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a4ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a580c9fd0 0x7f9a581e2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5659c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9a580c40b0 0x7f9a581e1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a56d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58152ce0 0x7f9a581e2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a4ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a580c9fd0 0x7f9a581e2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38800/0 (socket says 172.21.15.23:38800) 2023-06-22T08:00:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a4ffff700 1 -- 172.21.15.23:0/1118252998 learned_addr learned my addr 172.21.15.23:0/1118252998 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5659c700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9a580c40b0 0x7f9a581e1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:48624/0 (socket says 172.21.15.23:48624) 2023-06-22T08:00:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5659c700 1 -- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58152ce0 msgr2=0x7f9a581e2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5659c700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58152ce0 0x7f9a581e2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5659c700 1 -- 172.21.15.23:0/1118252998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a580c9fd0 msgr2=0x7f9a581e2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5659c700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a580c9fd0 0x7f9a581e2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a5659c700 1 -- 172.21.15.23:0/1118252998 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a48009550 con 0x7f9a580c40b0 2023-06-22T08:00:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a4ffff700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a580c9fd0 0x7f9a581e2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.803+0000 7f9a56d9d700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58152ce0 0x7f9a581e2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.804+0000 7f9a5659c700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9a580c40b0 0x7f9a581e1610 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a48016000 tx=0x7f9a48004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.804+0000 7f9a4effd700 1 -- 172.21.15.23:0/1118252998 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a48004510 con 0x7f9a580c40b0 2023-06-22T08:00:25.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.804+0000 7f9a4effd700 1 -- 172.21.15.23:0/1118252998 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4801e070 con 0x7f9a580c40b0 2023-06-22T08:00:25.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.804+0000 7f9a4effd700 1 -- 172.21.15.23:0/1118252998 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a480038c0 con 0x7f9a580c40b0 2023-06-22T08:00:25.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.804+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/1118252998 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a580cec20 con 0x7f9a580c40b0 2023-06-22T08:00:25.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.804+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/1118252998 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a58155cd0 con 0x7f9a580c40b0 2023-06-22T08:00:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.805+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/1118252998 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5804ea20 con 0x7f9a580c40b0 2023-06-22T08:00:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.806+0000 7f9a4effd700 1 -- 172.21.15.23:0/1118252998 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9a48003a20 con 0x7f9a580c40b0 2023-06-22T08:00:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.807+0000 7f9a4effd700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9a3806ace0 0x7f9a3806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.807+0000 7f9a4effd700 1 -- 172.21.15.23:0/1118252998 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9a4808a9c0 con 0x7f9a580c40b0 2023-06-22T08:00:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.808+0000 7f9a4ffff700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9a3806ace0 0x7f9a3806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:25.813 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.808+0000 7f9a4ffff700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9a3806ace0 0x7f9a3806d190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9a580c8e50 tx=0x7f9a40007f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:25.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:25.811+0000 7f9a4effd700 1 -- 172.21.15.23:0/1118252998 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9a48055770 con 0x7f9a580c40b0 2023-06-22T08:00:26.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.021+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/1118252998 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a580c9db0 con 0x7f9a580c40b0 2023-06-22T08:00:26.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.022+0000 7f9a4effd700 1 -- 172.21.15.23:0/1118252998 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f9a48058d90 con 0x7f9a580c40b0 2023-06-22T08:00:26.026 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:26.026 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9a3806ace0 msgr2=0x7f9a3806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9a3806ace0 0x7f9a3806d190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9a580c8e50 tx=0x7f9a40007f90 comp rx=0 tx=0).stop 2023-06-22T08:00:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9a580c40b0 msgr2=0x7f9a581e1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9a580c40b0 0x7f9a581e1610 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a48016000 tx=0x7f9a48004750 comp rx=0 tx=0).stop 2023-06-22T08:00:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/1118252998 shutdown_connections 2023-06-22T08:00:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9a3806ace0 0x7f9a3806d190 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58152ce0 0x7f9a581e2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9a580c40b0 0x7f9a581e1610 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 --2- 172.21.15.23:0/1118252998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a580c9fd0 0x7f9a581e2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/1118252998 >> 172.21.15.23:0/1118252998 conn(0x7f9a5806ae30 msgr2=0x7f9a5814fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/1118252998 shutdown_connections 2023-06-22T08:00:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.025+0000 7f9a5c9ff700 1 -- 172.21.15.23:0/1118252998 wait complete. 2023-06-22T08:00:26.047 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-06-22T08:00:26.047 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-06-22T08:00:26.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-06-22T08:00:26.048 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-06-22T08:00:26.049 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-06-22T08:00:26.049 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-06-22T08:00:26.050 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-06-22T08:00:26.050 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-06-22T08:00:26.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.406+0000 7f32a4fff700 1 Processor -- start 2023-06-22T08:00:26.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.406+0000 7f32a4fff700 1 -- start start 2023-06-22T08:00:26.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.408+0000 7f32a4fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32a014eaf0 0x7f32a00c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.408+0000 7f32a4fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f32a0069eb0 0x7f32a006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.409+0000 7f32a4fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a00c3dd0 con 0x7f32a006e310 2023-06-22T08:00:26.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.409+0000 7f32a4fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a006a7f0 con 0x7f32a0069eb0 2023-06-22T08:00:26.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.409+0000 7f32a4fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a0158d10 con 0x7f32a014eaf0 2023-06-22T08:00:26.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.409+0000 7fdee59ff700 1 Processor -- start 2023-06-22T08:00:26.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.410+0000 7fdee59ff700 1 -- start start 2023-06-22T08:00:26.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.411+0000 7f329dd9b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f32a006e310 0x7f32a01548b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58618/0 (socket says 172.21.15.23:58618) 2023-06-22T08:00:26.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.411+0000 7f329dd9b700 1 -- 172.21.15.23:0/3340228979 learned_addr learned my addr 172.21.15.23:0/3340228979 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:26.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.411+0000 7f329ed9d700 1 --2- 172.21.15.23:0/3340228979 >> v2:172.21.15.23:3300/0 conn(0x7f32a0069eb0 0x7f32a006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7f329e59c700 1 --2- 172.21.15.23:0/3340228979 >> v2:172.21.15.67:3300/0 conn(0x7f32a014eaf0 0x7f32a00c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7fdee59ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdee006e310 0x7fdee00c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7fdee59ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdee01548b0 0x7fdee0154d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7f329d59a700 1 -- 172.21.15.23:0/3340228979 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050314816 0 0) 0x7f32a00c3dd0 con 0x7f32a006e310 2023-06-22T08:00:26.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7f329d59a700 1 -- 172.21.15.23:0/3340228979 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32880057c0 con 0x7f32a006e310 2023-06-22T08:00:26.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7fdee59ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee00c3dd0 con 0x7fdee00caf00 2023-06-22T08:00:26.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7fdee59ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee0158d10 con 0x7fdee01548b0 2023-06-22T08:00:26.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7fdee59ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00c4800 con 0x7fdee006e310 2023-06-22T08:00:26.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7f329ed9d700 1 -- 172.21.15.23:0/3340228979 >> v2:172.21.15.67:3300/0 conn(0x7f32a014eaf0 msgr2=0x7f32a00c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7f329ed9d700 1 --2- 172.21.15.23:0/3340228979 >> v2:172.21.15.67:3300/0 conn(0x7f32a014eaf0 0x7f32a00c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7f329ed9d700 1 -- 172.21.15.23:0/3340228979 >> v1:172.21.15.67:6789/0 conn(0x7f32a006e310 legacy=0x7f32a01548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7f329ed9d700 1 -- 172.21.15.23:0/3340228979 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a00c4800 con 0x7f32a0069eb0 2023-06-22T08:00:26.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.413+0000 7f329ed9d700 1 --2- 172.21.15.23:0/3340228979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a0069eb0 0x7f32a006a2b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f32a0066820 tx=0x7f3298009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db03dcb7491324b0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7fdedeffd700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdee00caf00 0x7fdee014eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58624/0 (socket says 172.21.15.23:58624) 2023-06-22T08:00:26.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.412+0000 7fdedeffd700 1 -- 172.21.15.23:0/1059029533 learned_addr learned my addr 172.21.15.23:0/1059029533 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:26.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.413+0000 7fdedf7fe700 1 --2- 172.21.15.23:0/1059029533 >> v2:172.21.15.67:3300/0 conn(0x7fdee01548b0 0x7fdee0154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.413+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12592658 0 0) 0x7fdee00c3dd0 con 0x7fdee00caf00 2023-06-22T08:00:26.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.413+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec80035b0 con 0x7fdee00caf00 2023-06-22T08:00:26.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.413+0000 7fdede7fc700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdee006e310 0x7fdee00c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.413+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2621820668 0 0) 0x7fdec80035b0 con 0x7fdee00caf00 2023-06-22T08:00:26.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdee006e310 msgr2=0x7fdee00c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdee006e310 0x7fdee00c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 >> v2:172.21.15.67:3300/0 conn(0x7fdee01548b0 msgr2=0x7fdee0154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 --2- 172.21.15.23:0/1059029533 >> v2:172.21.15.67:3300/0 conn(0x7fdee01548b0 0x7fdee0154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee00c4520 con 0x7fdee00caf00 2023-06-22T08:00:26.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7fded80019f0 con 0x7fdee00caf00 2023-06-22T08:00:26.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 >> v1:172.21.15.67:6789/0 conn(0x7fdee00caf00 legacy=0x7fdee014eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdec8003a00 0x7fdec8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdec8004850 0x7fdec8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdec8009a00 0x7fdec8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee0158d10 con 0x7fdec8003a00 2023-06-22T08:00:26.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00c4800 con 0x7fdec8004850 2023-06-22T08:00:26.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80147a0 con 0x7fdec8009a00 2023-06-22T08:00:26.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7f329d59a700 1 -- 172.21.15.23:0/3340228979 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329800b8b0 con 0x7f32a0069eb0 2023-06-22T08:00:26.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7f329d59a700 1 -- 172.21.15.23:0/3340228979 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329800ba10 con 0x7f32a0069eb0 2023-06-22T08:00:26.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7f329d59a700 1 -- 172.21.15.23:0/3340228979 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329800bd80 con 0x7f32a0069eb0 2023-06-22T08:00:26.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7f32a4fff700 1 -- 172.21.15.23:0/3340228979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a0069eb0 msgr2=0x7f32a006a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7f32a4fff700 1 --2- 172.21.15.23:0/3340228979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a0069eb0 0x7f32a006a2b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f32a0066820 tx=0x7f3298009f90 comp rx=0 tx=0).stop 2023-06-22T08:00:26.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7f32a4fff700 1 -- 172.21.15.23:0/3340228979 shutdown_connections 2023-06-22T08:00:26.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7f32a4fff700 1 --2- 172.21.15.23:0/3340228979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a0069eb0 0x7f32a006a2b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7f32a4fff700 1 --2- 172.21.15.23:0/3340228979 >> v2:172.21.15.67:3300/0 conn(0x7f32a014eaf0 0x7f32a00c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7f32a4fff700 1 -- 172.21.15.23:0/3340228979 >> 172.21.15.23:0/3340228979 conn(0x7f32a00ca3f0 msgr2=0x7f32a00ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7f32a4fff700 1 -- 172.21.15.23:0/3340228979 shutdown_connections 2023-06-22T08:00:26.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7f32a4fff700 1 -- 172.21.15.23:0/3340228979 wait complete. 2023-06-22T08:00:26.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.419+0000 7f32a4fff700 1 Processor -- start 2023-06-22T08:00:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.419+0000 7f32a4fff700 1 -- start start 2023-06-22T08:00:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.421+0000 7f32a4fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a0069eb0 0x7f32a01f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.421+0000 7f32a4fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f32a006e310 0x7f32a0149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.421+0000 7f32a4fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a014eaf0 0x7f32a00c0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.421+0000 7f32a4fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a00c1580 con 0x7f32a0069eb0 2023-06-22T08:00:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.421+0000 7f32a4fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a01f2e90 con 0x7f32a014eaf0 2023-06-22T08:00:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.421+0000 7f32a4fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a00c04f0 con 0x7f32a006e310 2023-06-22T08:00:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.422+0000 7f329e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a0069eb0 0x7f32a01f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.422+0000 7f329e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a0069eb0 0x7f32a01f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60724/0 (socket says 172.21.15.23:60724) 2023-06-22T08:00:26.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.422+0000 7f329e59c700 1 -- 172.21.15.23:0/1322132478 learned_addr learned my addr 172.21.15.23:0/1322132478 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.422+0000 7f329dd9b700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f32a006e310 0x7f32a0149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.422+0000 7f329ed9d700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a014eaf0 0x7f32a00c0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.423+0000 7f329dd9b700 1 -- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a014eaf0 msgr2=0x7f32a00c0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.423+0000 7f329dd9b700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32a014eaf0 0x7f32a00c0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.423+0000 7f329dd9b700 1 -- 172.21.15.23:0/1322132478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a0069eb0 msgr2=0x7f32a01f2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.423+0000 7f329dd9b700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a0069eb0 0x7f32a01f2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.423+0000 7f329dd9b700 1 -- 172.21.15.23:0/1322132478 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329800d040 con 0x7f32a006e310 2023-06-22T08:00:26.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.423+0000 7f329dd9b700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f32a006e310 0x7f32a0149010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f329000d710 tx=0x7f329000da20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.424+0000 7f328effd700 1 -- 172.21.15.23:0/1322132478 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329000f8d0 con 0x7f32a006e310 2023-06-22T08:00:26.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.424+0000 7f328effd700 1 -- 172.21.15.23:0/1322132478 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329000fa50 con 0x7f32a006e310 2023-06-22T08:00:26.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.424+0000 7f328effd700 1 -- 172.21.15.23:0/1322132478 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329000fe00 con 0x7f32a006e310 2023-06-22T08:00:26.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.424+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a01540d0 con 0x7f32a006e310 2023-06-22T08:00:26.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.425+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a00bfea0 con 0x7f32a006e310 2023-06-22T08:00:26.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdedeffd700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdec8003a00 0x7fdec8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.414+0000 7fdedf7fe700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdec8009a00 0x7fdec8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.415+0000 7fdede7fc700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdec8004850 0x7fdec8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.415+0000 7fdedeffd700 1 -- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdec8009a00 msgr2=0x7fdec8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.415+0000 7fdedeffd700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdec8009a00 0x7fdec8005200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.415+0000 7fdedeffd700 1 -- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdec8004850 msgr2=0x7fdec8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.415+0000 7fdedeffd700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdec8004850 0x7fdec8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.415+0000 7fdedeffd700 1 -- 172.21.15.23:0/1059029533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec80035b0 con 0x7fdec8003a00 2023-06-22T08:00:26.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.415+0000 7fdede7fc700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdec8004850 0x7fdec8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T08:00:26.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7fdedeffd700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdec8003a00 0x7fdec8003e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fded800a790 tx=0x7fdec8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33017e2981fe6c1d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded800a250 con 0x7fdec8003a00 2023-06-22T08:00:26.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded8012600 con 0x7fdec8003a00 2023-06-22T08:00:26.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.416+0000 7fdeddffb700 1 -- 172.21.15.23:0/1059029533 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded80129f0 con 0x7fdec8003a00 2023-06-22T08:00:26.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.417+0000 7fdee59ff700 1 -- 172.21.15.23:0/1059029533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdec8003a00 msgr2=0x7fdec8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.417+0000 7fdee59ff700 1 --2- 172.21.15.23:0/1059029533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdec8003a00 0x7fdec8003e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fded800a790 tx=0x7fdec8005880 comp rx=0 tx=0).stop 2023-06-22T08:00:26.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.417+0000 7fdedf7fe700 1 -- 172.21.15.23:0/1059029533 reap_dead start 2023-06-22T08:00:26.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.417+0000 7fdee59ff700 1 -- 172.21.15.23:0/1059029533 shutdown_connections 2023-06-22T08:00:26.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.417+0000 7fdee59ff700 1 -- 172.21.15.23:0/1059029533 >> 172.21.15.23:0/1059029533 conn(0x7fdee00ca3f0 msgr2=0x7fdee00ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.419+0000 7fdee59ff700 1 -- 172.21.15.23:0/1059029533 shutdown_connections 2023-06-22T08:00:26.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.419+0000 7fdee59ff700 1 -- 172.21.15.23:0/1059029533 wait complete. 2023-06-22T08:00:26.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.422+0000 7fdee59ff700 1 Processor -- start 2023-06-22T08:00:26.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.423+0000 7fdee59ff700 1 -- start start 2023-06-22T08:00:26.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.424+0000 7fdee59ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee0157030 0x7fdee0152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.425+0000 7fdee59ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdee01544e0 0x7fdee01fc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.425+0000 7fdedeffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee0157030 0x7fdee0152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.425+0000 7fdedeffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee0157030 0x7fdee0152720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:47844/0 (socket says 172.21.15.23:47844) 2023-06-22T08:00:26.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.425+0000 7fdede7fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdee01544e0 0x7fdee01fc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.425+0000 7fdede7fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdee01544e0 0x7fdee01fc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:36866/0 (socket says 172.21.15.23:36866) 2023-06-22T08:00:26.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.426+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3284000fc0 con 0x7f32a006e310 2023-06-22T08:00:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.426+0000 7fdee59ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdee0157400 0x7fdee01fc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.426+0000 7fdee59ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee00c19f0 con 0x7fdee0157400 2023-06-22T08:00:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.426+0000 7fdee59ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee0201100 con 0x7fdee0157030 2023-06-22T08:00:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.426+0000 7fdee59ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee0200440 con 0x7fdee01544e0 2023-06-22T08:00:26.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.426+0000 7fdedeffd700 1 -- 172.21.15.23:0/2365544436 learned_addr learned my addr 172.21.15.23:0/2365544436 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.428+0000 7fdedf7fe700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdee0157400 0x7fdee01fc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.428+0000 7fdedf7fe700 1 -- 172.21.15.23:0/2365544436 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdee01544e0 msgr2=0x7fdee01fc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.428+0000 7fdedf7fe700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdee01544e0 0x7fdee01fc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.428+0000 7fdedf7fe700 1 -- 172.21.15.23:0/2365544436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee0157030 msgr2=0x7fdee0152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.428+0000 7fdedf7fe700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee0157030 0x7fdee0152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.428+0000 7fdedf7fe700 1 -- 172.21.15.23:0/2365544436 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee02006e0 con 0x7fdee0157400 2023-06-22T08:00:26.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.429+0000 7f96ee9ff700 1 Processor -- start 2023-06-22T08:00:26.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.429+0000 7f96ee9ff700 1 -- start start 2023-06-22T08:00:26.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.431+0000 7f96ee9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96e8069eb0 0x7f96e806d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.431+0000 7f96ee9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e80c4d70 con 0x7f96e815d9b0 2023-06-22T08:00:26.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.431+0000 7f96ee9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e80c5610 con 0x7f96e806a280 2023-06-22T08:00:26.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.431+0000 7f96ee9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e80703c0 con 0x7f96e8069eb0 2023-06-22T08:00:26.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.432+0000 7fdedf7fe700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdee0157400 0x7fdee01fc7e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdee006b440 tx=0x7fded0010a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.432+0000 7f96e7fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96e8069eb0 0x7f96e806d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.432+0000 7f96e7fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96e8069eb0 0x7f96e806d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:47850/0 (socket says 172.21.15.23:47850) 2023-06-22T08:00:26.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.432+0000 7f96e7fff700 1 -- 172.21.15.23:0/4235449006 learned_addr learned my addr 172.21.15.23:0/4235449006 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.427+0000 7fb0d55ff700 1 Processor -- start 2023-06-22T08:00:26.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.427+0000 7fb0d55ff700 1 -- start start 2023-06-22T08:00:26.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.429+0000 7fb0d55ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0d0069eb0 0x7fb0d006d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.430+0000 7fb0d55ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d00c4d70 con 0x7fb0d006a280 2023-06-22T08:00:26.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.430+0000 7fb0d55ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d00c5610 con 0x7fb0d015d9b0 2023-06-22T08:00:26.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.430+0000 7fb0d55ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d00703c0 con 0x7fb0d0069eb0 2023-06-22T08:00:26.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.430+0000 7fb0cf7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb0d006a280 0x7fb0d01675a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41312/0 (socket says 172.21.15.23:41312) 2023-06-22T08:00:26.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.430+0000 7fb0cf7fe700 1 -- 172.21.15.23:0/1753750911 learned_addr learned my addr 172.21.15.23:0/1753750911 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:26.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7fdec7fff700 1 -- 172.21.15.23:0/2365544436 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0014800 con 0x7fdee0157400 2023-06-22T08:00:26.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7fdec7fff700 1 -- 172.21.15.23:0/2365544436 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded0014960 con 0x7fdee0157400 2023-06-22T08:00:26.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7fdec7fff700 1 -- 172.21.15.23:0/2365544436 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0014d10 con 0x7fdee0157400 2023-06-22T08:00:26.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7f96e7fff700 1 -- 172.21.15.23:0/4235449006 >> v1:172.21.15.67:6789/0 conn(0x7f96e806a280 legacy=0x7f96e81675a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7fdee59ff700 1 -- 172.21.15.23:0/2365544436 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee0200e20 con 0x7fdee0157400 2023-06-22T08:00:26.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.431+0000 7f328effd700 1 -- 172.21.15.23:0/1322132478 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f329000a050 con 0x7f32a006e310 2023-06-22T08:00:26.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7f328effd700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f328006adb0 0x7f328006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7f328effd700 1 -- 172.21.15.23:0/1322132478 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f329008a080 con 0x7f32a006e310 2023-06-22T08:00:26.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7f328effd700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.23:6800/458708400,v1:172.21.15.23:6801/458708400] conn(0x7f3280074a80 0x7f3280076eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7f328effd700 1 -- 172.21.15.23:0/1322132478 --> [v2:172.21.15.23:6800/458708400,v1:172.21.15.23:6801/458708400] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3280077580 con 0x7f3280074a80 2023-06-22T08:00:26.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7f328effd700 1 -- 172.21.15.23:0/1322132478 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f329008a430 con 0x7f32a006e310 2023-06-22T08:00:26.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7f329e59c700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f328006adb0 0x7f328006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7f329ed9d700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.23:6800/458708400,v1:172.21.15.23:6801/458708400] conn(0x7f3280074a80 0x7f3280076eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.433+0000 7fdee59ff700 1 -- 172.21.15.23:0/2365544436 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee0202170 con 0x7fdee0157400 2023-06-22T08:00:26.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7fb0ceffd700 1 --2- 172.21.15.23:0/1753750911 >> v2:172.21.15.67:3300/0 conn(0x7fb0d0069eb0 0x7fb0d006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990677679 0 0) 0x7fb0d00c4d70 con 0x7fb0d006a280 2023-06-22T08:00:26.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0bc0035b0 con 0x7fb0d006a280 2023-06-22T08:00:26.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 650654232 0 0) 0x7fb0bc0035b0 con 0x7fb0d006a280 2023-06-22T08:00:26.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 >> v2:172.21.15.67:3300/0 conn(0x7fb0d0069eb0 msgr2=0x7fb0d006d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7fb0ce7fc700 1 --2- 172.21.15.23:0/1753750911 >> v2:172.21.15.67:3300/0 conn(0x7fb0d0069eb0 0x7fb0d006d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 >> v1:172.21.15.67:6789/0 conn(0x7fb0d015d9b0 legacy=0x7fb0d006fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d0071340 con 0x7fb0d006a280 2023-06-22T08:00:26.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7fb0c80019f0 con 0x7fb0d006a280 2023-06-22T08:00:26.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.434+0000 7f96e7fff700 1 -- 172.21.15.23:0/4235449006 >> v1:172.21.15.23:6789/0 conn(0x7f96e815d9b0 legacy=0x7f96e806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7fdec5ffb700 1 -- 172.21.15.23:0/2365544436 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdeb8000fc0 con 0x7fdee0157400 2023-06-22T08:00:26.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 >> v1:172.21.15.23:6789/0 conn(0x7fb0d006a280 legacy=0x7fb0d01675a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7fb0ce7fc700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc003a00 0x7fb0bc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7fb0ce7fc700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0bc004850 0x7fb0bc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7fb0ce7fc700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc005200 0x7fb0bc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d00703c0 con 0x7fb0bc003a00 2023-06-22T08:00:26.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc00c060 con 0x7fb0bc005200 2023-06-22T08:00:26.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc00c220 con 0x7fb0bc004850 2023-06-22T08:00:26.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7f96e7fff700 1 -- 172.21.15.23:0/4235449006 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e8071340 con 0x7f96e8069eb0 2023-06-22T08:00:26.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7f329e59c700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f328006adb0 0x7f328006d260 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3294007af0 tx=0x7f3294008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7f329ed9d700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.23:6800/458708400,v1:172.21.15.23:6801/458708400] conn(0x7f3280074a80 0x7f3280076eb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7f1a25bff700 1 Processor -- start 2023-06-22T08:00:26.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7f1a25bff700 1 -- start start 2023-06-22T08:00:26.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.435+0000 7fb0cf7fe700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc005200 0x7fb0bc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7fb0ceffd700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc003a00 0x7fb0bc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7fb0c7fff700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0bc004850 0x7fb0bc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7fb0cf7fe700 1 -- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0bc004850 msgr2=0x7fb0bc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7fb0cf7fe700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0bc004850 0x7fb0bc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7fb0cf7fe700 1 -- 172.21.15.23:0/1753750911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc003a00 msgr2=0x7fb0bc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7fb0cf7fe700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc003a00 0x7fb0bc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7fb0cf7fe700 1 -- 172.21.15.23:0/1753750911 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d0071340 con 0x7fb0bc005200 2023-06-22T08:00:26.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7fb0cf7fe700 1 -- 172.21.15.23:0/1753750911 reap_dead start 2023-06-22T08:00:26.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.436+0000 7fb0c7fff700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0bc004850 0x7fb0bc004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:26.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.437+0000 7fb0cf7fe700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc005200 0x7fb0bc00ba10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800a860 tx=0x7fb0bc005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da73305a170dba3a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.437+0000 7f96e7fff700 1 --2- 172.21.15.23:0/4235449006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e8069eb0 0x7f96e806d050 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f96e00098f0 tx=0x7f96e0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b04313bdc680b0ea server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7fe61299d700 1 Processor -- start 2023-06-22T08:00:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7f96e6ffd700 1 -- 172.21.15.23:0/4235449006 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0004230 con 0x7f96e8069eb0 2023-06-22T08:00:26.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7f96e6ffd700 1 -- 172.21.15.23:0/4235449006 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e0004390 con 0x7f96e8069eb0 2023-06-22T08:00:26.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7f96e6ffd700 1 -- 172.21.15.23:0/4235449006 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e00036a0 con 0x7f96e8069eb0 2023-06-22T08:00:26.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7f328effd700 1 -- 172.21.15.23:0/1322132478 <== osd.0 v2:172.21.15.23:6800/458708400 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3280077580 con 0x7f3280074a80 2023-06-22T08:00:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7f1a25bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1a200c5f20 0x7f1a20069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7f1a25bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a200c3dd0 con 0x7f1a2006e310 2023-06-22T08:00:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7f1a25bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a200c4800 con 0x7f1a200caf00 2023-06-22T08:00:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7f1a25bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a20158d10 con 0x7f1a200c5f20 2023-06-22T08:00:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.439+0000 7f1a1effd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1a2006e310 0x7f1a201548b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41334/0 (socket says 172.21.15.23:41334) 2023-06-22T08:00:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.439+0000 7f1a1effd700 1 -- 172.21.15.23:0/3341860282 learned_addr learned my addr 172.21.15.23:0/3341860282 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.439+0000 7f96ee9ff700 1 -- 172.21.15.23:0/4235449006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e8069eb0 msgr2=0x7f96e806d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.439+0000 7f96ee9ff700 1 --2- 172.21.15.23:0/4235449006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e8069eb0 0x7f96e806d050 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f96e00098f0 tx=0x7f96e0009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.439+0000 7f96ee9ff700 1 -- 172.21.15.23:0/4235449006 shutdown_connections 2023-06-22T08:00:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.439+0000 7f96ee9ff700 1 --2- 172.21.15.23:0/4235449006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e8069eb0 0x7f96e806d050 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.439+0000 7f96ee9ff700 1 -- 172.21.15.23:0/4235449006 >> 172.21.15.23:0/4235449006 conn(0x7f96e80ca3f0 msgr2=0x7f96e80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.439+0000 7f1a1ffff700 1 --2- 172.21.15.23:0/3341860282 >> v2:172.21.15.23:3300/0 conn(0x7f1a200c5f20 0x7f1a20069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7fdec7fff700 1 -- 172.21.15.23:0/2365544436 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fded0020460 con 0x7fdee0157400 2023-06-22T08:00:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.439+0000 7fdec7fff700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdecc06adb0 0x7fdecc06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fdec7fff700 1 -- 172.21.15.23:0/2365544436 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fded008a0b0 con 0x7fdee0157400 2023-06-22T08:00:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fdec7fff700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.23:6816/3464394342,v1:172.21.15.23:6817/3464394342] conn(0x7fdecc074a80 0x7fdecc076eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fdec7fff700 1 -- 172.21.15.23:0/2365544436 --> [v2:172.21.15.23:6816/3464394342,v1:172.21.15.23:6817/3464394342] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdecc077580 con 0x7fdecc074a80 2023-06-22T08:00:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fdec7fff700 1 -- 172.21.15.23:0/2365544436 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fded008a460 con 0x7fdee0157400 2023-06-22T08:00:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7fdedeffd700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdecc06adb0 0x7fdecc06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7f96ee9ff700 1 -- 172.21.15.23:0/4235449006 shutdown_connections 2023-06-22T08:00:26.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7f1a1ffff700 1 -- 172.21.15.23:0/3341860282 >> v1:172.21.15.67:6789/0 conn(0x7f1a200caf00 legacy=0x7f1a2014eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7f1a1ffff700 1 -- 172.21.15.23:0/3341860282 >> v1:172.21.15.23:6789/0 conn(0x7f1a2006e310 legacy=0x7f1a201548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7f1a1ffff700 1 -- 172.21.15.23:0/3341860282 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a200c4520 con 0x7f1a200c5f20 2023-06-22T08:00:26.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c800a2d0 con 0x7fb0bc005200 2023-06-22T08:00:26.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c80072f0 con 0x7fb0bc005200 2023-06-22T08:00:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fb0ce7fc700 1 -- 172.21.15.23:0/1753750911 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80076a0 con 0x7fb0bc005200 2023-06-22T08:00:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7fb0d55ff700 1 -- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc005200 msgr2=0x7fb0bc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7fb0d55ff700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc005200 0x7fb0bc00ba10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800a860 tx=0x7fb0bc005880 comp rx=0 tx=0).stop 2023-06-22T08:00:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7f6704bff700 1 Processor -- start 2023-06-22T08:00:26.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.438+0000 7fe61299d700 1 -- start start 2023-06-22T08:00:26.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fe61299d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe60c1570a0 0x7fe60c070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fe61299d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 0x7fe60c15d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fe61299d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c06a5c0 con 0x7fe60c160b80 2023-06-22T08:00:26.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fe61299d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60c157470 con 0x7fe60c1570a0 2023-06-22T08:00:26.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7fe61299d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c0702a0 con 0x7fe60c15cf00 2023-06-22T08:00:26.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7fdede7fc700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.23:6816/3464394342,v1:172.21.15.23:6817/3464394342] conn(0x7fdecc074a80 0x7fdecc076eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7f1a1ffff700 1 --2- 172.21.15.23:0/3341860282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a200c5f20 0x7f1a20069eb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009840 tx=0x7f1a10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e12facb67d4086b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7fe61119a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe60c160b80 0x7fe60c0c6cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41348/0 (socket says 172.21.15.23:41348) 2023-06-22T08:00:26.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7fe61119a700 1 -- 172.21.15.23:0/728920667 learned_addr learned my addr 172.21.15.23:0/728920667 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7fb0d55ff700 1 -- 172.21.15.23:0/1753750911 shutdown_connections 2023-06-22T08:00:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7fb0d55ff700 1 --2- 172.21.15.23:0/1753750911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc005200 0x7fb0bc00ba10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7fb0d55ff700 1 -- 172.21.15.23:0/1753750911 >> 172.21.15.23:0/1753750911 conn(0x7fb0d00ca3f0 msgr2=0x7fb0d00ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.442+0000 7f1a1e7fc700 1 -- 172.21.15.23:0/3341860282 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a10004230 con 0x7f1a200c5f20 2023-06-22T08:00:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.442+0000 7f1a1e7fc700 1 -- 172.21.15.23:0/3341860282 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a10004390 con 0x7f1a200c5f20 2023-06-22T08:00:26.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.442+0000 7f1a1e7fc700 1 -- 172.21.15.23:0/3341860282 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a100036a0 con 0x7f1a200c5f20 2023-06-22T08:00:26.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.441+0000 7f96ee9ff700 1 -- 172.21.15.23:0/4235449006 wait complete. 2023-06-22T08:00:26.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.442+0000 7fb0d55ff700 1 -- 172.21.15.23:0/1753750911 shutdown_connections 2023-06-22T08:00:26.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.442+0000 7fdedeffd700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdecc06adb0 0x7fdecc06d260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fded8004c50 tx=0x7fded8009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.442+0000 7fdede7fc700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.23:6816/3464394342,v1:172.21.15.23:6817/3464394342] conn(0x7fdecc074a80 0x7fdecc076eb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.440+0000 7f6704bff700 1 -- start start 2023-06-22T08:00:26.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.443+0000 7f6704bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6700069eb0 0x7f670006d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.443+0000 7f6704bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f670006a280 0x7f67000c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.443+0000 7f6704bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67000c4d70 con 0x7f670015d9b0 2023-06-22T08:00:26.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.443+0000 7f6704bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67000703c0 con 0x7f670006a280 2023-06-22T08:00:26.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.443+0000 7f6704bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67000c5610 con 0x7f6700069eb0 2023-06-22T08:00:26.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.443+0000 7fe61199b700 1 --2- 172.21.15.23:0/728920667 >> v2:172.21.15.23:3300/0 conn(0x7fe60c1570a0 0x7fe60c070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.443+0000 7fe61219c700 1 --2- 172.21.15.23:0/728920667 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 0x7fe60c15d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7f96ee9ff700 1 Processor -- start 2023-06-22T08:00:26.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7fe61219c700 1 -- 172.21.15.23:0/728920667 >> v2:172.21.15.23:3300/0 conn(0x7fe60c1570a0 msgr2=0x7fe60c070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7fe61219c700 1 --2- 172.21.15.23:0/728920667 >> v2:172.21.15.23:3300/0 conn(0x7fe60c1570a0 0x7fe60c070fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7fe61219c700 1 -- 172.21.15.23:0/728920667 >> v1:172.21.15.23:6789/0 conn(0x7fe60c160b80 legacy=0x7fe60c0c6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7fe61219c700 1 -- 172.21.15.23:0/728920667 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c15d8c0 con 0x7fe60c15cf00 2023-06-22T08:00:26.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7fdec7fff700 1 -- 172.21.15.23:0/2365544436 <== osd.1 v2:172.21.15.23:6816/3464394342 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fdecc077580 con 0x7fdecc074a80 2023-06-22T08:00:26.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7f66fe59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6700069eb0 0x7f670006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7f66fe59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6700069eb0 0x7f670006d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:47888/0 (socket says 172.21.15.23:47888) 2023-06-22T08:00:26.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7f66fe59c700 1 -- 172.21.15.23:0/613898359 learned_addr learned my addr 172.21.15.23:0/613898359 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7f66fed9d700 1 --2- 172.21.15.23:0/613898359 >> v2:172.21.15.23:3300/0 conn(0x7f670006a280 0x7f67000c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7fe61219c700 1 --2- 172.21.15.23:0/728920667 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 0x7fe60c15d380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe604009840 tx=0x7fe604009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f93f1f4c5947036a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.442+0000 7fb0d55ff700 1 -- 172.21.15.23:0/1753750911 wait complete. 2023-06-22T08:00:26.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.445+0000 7fb0d55ff700 1 Processor -- start 2023-06-22T08:00:26.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.445+0000 7f66fed9d700 1 -- 172.21.15.23:0/613898359 >> v2:172.21.15.67:3300/0 conn(0x7f6700069eb0 msgr2=0x7f670006d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.445+0000 7f66fed9d700 1 --2- 172.21.15.23:0/613898359 >> v2:172.21.15.67:3300/0 conn(0x7f6700069eb0 0x7f670006d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.445+0000 7f66fed9d700 1 -- 172.21.15.23:0/613898359 >> v1:172.21.15.23:6789/0 conn(0x7f670015d9b0 legacy=0x7f670006fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.445+0000 7f66fed9d700 1 -- 172.21.15.23:0/613898359 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67000c5fe0 con 0x7f670006a280 2023-06-22T08:00:26.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7f1a25bff700 1 -- 172.21.15.23:0/3341860282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a200c5f20 msgr2=0x7f1a20069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7f1a25bff700 1 --2- 172.21.15.23:0/3341860282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a200c5f20 0x7f1a20069eb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009840 tx=0x7f1a10009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.444+0000 7f96ee9ff700 1 -- start start 2023-06-22T08:00:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7f96ee9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96e8069eb0 0x7f96e815bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7f96ee9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96e806a280 0x7f96e81f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7f96ee9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e815d9b0 0x7f96e81f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7f96ee9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e8160990 con 0x7f96e806a280 2023-06-22T08:00:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7f96ee9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e8160be0 con 0x7f96e815d9b0 2023-06-22T08:00:26.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7f96ee9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e8160ea0 con 0x7f96e8069eb0 2023-06-22T08:00:26.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7f66fed9d700 1 --2- 172.21.15.23:0/613898359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f670006a280 0x7f67000c5aa0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f67000c6dd0 tx=0x7f66f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a0e876437a9b33 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7f96e7fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96e8069eb0 0x7f96e815bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.447+0000 7f96e7fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96e8069eb0 0x7f96e815bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:36904/0 (socket says 172.21.15.23:36904) 2023-06-22T08:00:26.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.447+0000 7f96e7fff700 1 -- 172.21.15.23:0/1856714876 learned_addr learned my addr 172.21.15.23:0/1856714876 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.447+0000 7f96e77fe700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96e806a280 0x7f96e81f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.447+0000 7fa009fff700 1 Processor -- start 2023-06-22T08:00:26.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.447+0000 7fa009fff700 1 -- start start 2023-06-22T08:00:26.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7f96e77fe700 1 -- 172.21.15.23:0/1856714876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96e8069eb0 msgr2=0x7f96e815bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7f96e77fe700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96e8069eb0 0x7f96e815bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7f96e77fe700 1 -- 172.21.15.23:0/1856714876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e815d9b0 msgr2=0x7f96e81f33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7f96e77fe700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96e815d9b0 0x7f96e81f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7f96e77fe700 1 -- 172.21.15.23:0/1856714876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e0009550 con 0x7f96e806a280 2023-06-22T08:00:26.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.446+0000 7fb0d55ff700 1 -- start start 2023-06-22T08:00:26.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7fb0d55ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc005200 0x7fb0d0135490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7fb0d55ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0d0157670 0x7fb0d01544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7fb0d55ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0d00c3220 0x7fb0d00c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7fb0d55ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d00b78f0 con 0x7fb0bc005200 2023-06-22T08:00:26.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7fb0d55ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d01359d0 con 0x7fb0d00c3220 2023-06-22T08:00:26.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7fb0d55ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d0156180 con 0x7fb0d0157670 2023-06-22T08:00:26.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7fb0ceffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc005200 0x7fb0d0135490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7fb0ceffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc005200 0x7fb0d0135490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60792/0 (socket says 172.21.15.23:60792) 2023-06-22T08:00:26.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.448+0000 7fb0ceffd700 1 -- 172.21.15.23:0/3960667444 learned_addr learned my addr 172.21.15.23:0/3960667444 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fb0c7fff700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0d0157670 0x7fb0d01544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fb0cf7fe700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0d00c3220 0x7fb0d00c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7f1a25bff700 1 -- 172.21.15.23:0/3341860282 shutdown_connections 2023-06-22T08:00:26.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7f1a25bff700 1 --2- 172.21.15.23:0/3341860282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a200c5f20 0x7f1a20069eb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7f1a25bff700 1 -- 172.21.15.23:0/3341860282 >> 172.21.15.23:0/3341860282 conn(0x7f1a200ca3f0 msgr2=0x7f1a200ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fa009fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa0040c3a40 0x7fa004156f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fb0ceffd700 1 -- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0d0157670 msgr2=0x7fb0d01544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fb0ceffd700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0d0157670 0x7fb0d01544a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fb0ceffd700 1 -- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0d00c3220 msgr2=0x7fb0d00c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fb0ceffd700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0d00c3220 0x7fb0d00c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fb0ceffd700 1 -- 172.21.15.23:0/3960667444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d013e970 con 0x7fb0bc005200 2023-06-22T08:00:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fa009fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa00406d050 0x7fa0040c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fa009fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00414d710 con 0x7fa004159c10 2023-06-22T08:00:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fa009fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa004157570 con 0x7fa0040c3a40 2023-06-22T08:00:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fa009fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0040c55b0 con 0x7fa00406d050 2023-06-22T08:00:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7f96e77fe700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96e806a280 0x7f96e81f2e60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f96d000bd20 tx=0x7f96d000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7f1a25bff700 1 -- 172.21.15.23:0/3341860282 shutdown_connections 2023-06-22T08:00:26.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7fe610999700 1 -- 172.21.15.23:0/728920667 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604004230 con 0x7fe60c15cf00 2023-06-22T08:00:26.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7fe610999700 1 -- 172.21.15.23:0/728920667 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe604004390 con 0x7fe60c15cf00 2023-06-22T08:00:26.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7fe610999700 1 -- 172.21.15.23:0/728920667 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6040036a0 con 0x7fe60c15cf00 2023-06-22T08:00:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7f9ffaffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa00406d050 0x7fa0040c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7fa003fff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa004159c10 0x7fa00415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58646/0 (socket says 172.21.15.23:58646) 2023-06-22T08:00:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7fa003fff700 1 -- 172.21.15.23:0/1200512409 learned_addr learned my addr 172.21.15.23:0/1200512409 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699220906 0 0) 0x7f9ffc001110 con 0x7fa004159c10 2023-06-22T08:00:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ff00035b0 con 0x7fa004159c10 2023-06-22T08:00:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7fa0037fe700 1 --2- 172.21.15.23:0/1200512409 >> v2:172.21.15.23:3300/0 conn(0x7fa0040c3a40 0x7fa004156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7f66fd59a700 1 -- 172.21.15.23:0/613898359 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f800e070 con 0x7f670006a280 2023-06-22T08:00:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7f66fd59a700 1 -- 172.21.15.23:0/613898359 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f800bae0 con 0x7f670006a280 2023-06-22T08:00:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7f66fd59a700 1 -- 172.21.15.23:0/613898359 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f800be90 con 0x7f670006a280 2023-06-22T08:00:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7f6704bff700 1 -- 172.21.15.23:0/613898359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f670006a280 msgr2=0x7f67000c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7f6704bff700 1 --2- 172.21.15.23:0/613898359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f670006a280 0x7f67000c5aa0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f67000c6dd0 tx=0x7f66f800b330 comp rx=0 tx=0).stop 2023-06-22T08:00:26.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 452852076 0 0) 0x7f9ff00035b0 con 0x7fa004159c10 2023-06-22T08:00:26.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa00406d050 msgr2=0x7fa0040c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa00406d050 0x7fa0040c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 >> v2:172.21.15.23:3300/0 conn(0x7fa0040c3a40 msgr2=0x7fa004156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 --2- 172.21.15.23:0/1200512409 >> v2:172.21.15.23:3300/0 conn(0x7fa0040c3a40 0x7fa004156f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0040c6a60 con 0x7fa004159c10 2023-06-22T08:00:26.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7f9ffc001800 con 0x7fa004159c10 2023-06-22T08:00:26.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 >> v1:172.21.15.67:6789/0 conn(0x7fa004159c10 legacy=0x7fa00415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ff0003a00 0x7f9ff0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ff0004430 0x7f9ff00048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9ff0005010 0x7f9ff000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa004157570 con 0x7f9ff0003a00 2023-06-22T08:00:26.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff000bf50 con 0x7f9ff0004430 2023-06-22T08:00:26.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.451+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff000c090 con 0x7f9ff0005010 2023-06-22T08:00:26.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.449+0000 7f1a25bff700 1 -- 172.21.15.23:0/3341860282 wait complete. 2023-06-22T08:00:26.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.452+0000 7f6704bff700 1 -- 172.21.15.23:0/613898359 shutdown_connections 2023-06-22T08:00:26.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.452+0000 7f6704bff700 1 --2- 172.21.15.23:0/613898359 >> v2:172.21.15.67:3300/0 conn(0x7f6700069eb0 0x7f670006d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.452+0000 7f6704bff700 1 --2- 172.21.15.23:0/613898359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f670006a280 0x7f67000c5aa0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.452+0000 7f6704bff700 1 -- 172.21.15.23:0/613898359 >> 172.21.15.23:0/613898359 conn(0x7f67000ca3f0 msgr2=0x7f67000ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.452+0000 7fa003fff700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9ff0005010 0x7f9ff000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.450+0000 7fb0ceffd700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc005200 0x7fb0d0135490 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0011870 tx=0x7fb0c0011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.452+0000 7fa0037fe700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ff0003a00 0x7f9ff0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.452+0000 7f9ffaffd700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ff0004430 0x7f9ff00048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7fa0037fe700 1 -- 172.21.15.23:0/1200512409 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9ff0005010 msgr2=0x7f9ff000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7fa0037fe700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9ff0005010 0x7f9ff000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7fa0037fe700 1 -- 172.21.15.23:0/1200512409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ff0004430 msgr2=0x7f9ff00048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7fa0037fe700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ff0004430 0x7f9ff00048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7fa0037fe700 1 -- 172.21.15.23:0/1200512409 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0040c6a60 con 0x7f9ff0003a00 2023-06-22T08:00:26.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7fe61299d700 1 -- 172.21.15.23:0/728920667 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 msgr2=0x7fe60c15d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7fe61299d700 1 --2- 172.21.15.23:0/728920667 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 0x7fe60c15d380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe604009840 tx=0x7fe604009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:26.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7f96e4ff9700 1 -- 172.21.15.23:0/1856714876 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d000cd10 con 0x7f96e806a280 2023-06-22T08:00:26.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7f96e4ff9700 1 -- 172.21.15.23:0/1856714876 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d0014440 con 0x7f96e806a280 2023-06-22T08:00:26.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7f96e4ff9700 1 -- 172.21.15.23:0/1856714876 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d00147f0 con 0x7f96e806a280 2023-06-22T08:00:26.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e815ea00 con 0x7f96e806a280 2023-06-22T08:00:26.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7f6704bff700 1 -- 172.21.15.23:0/613898359 shutdown_connections 2023-06-22T08:00:26.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7f1a25bff700 1 Processor -- start 2023-06-22T08:00:26.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7fa0037fe700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ff0003a00 0x7f9ff0003e10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009b70 tx=0x7f9ff0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1ee52e3a3777a7 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.453+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e815efc0 con 0x7f96e806a280 2023-06-22T08:00:26.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.454+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff4004020 con 0x7f9ff0003a00 2023-06-22T08:00:26.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.454+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff4004180 con 0x7f9ff0003a00 2023-06-22T08:00:26.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.454+0000 7fa002ffd700 1 -- 172.21.15.23:0/1200512409 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff40044f0 con 0x7f9ff0003a00 2023-06-22T08:00:26.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.454+0000 7fb0c77fe700 1 -- 172.21.15.23:0/3960667444 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0011e70 con 0x7fb0bc005200 2023-06-22T08:00:26.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.454+0000 7fb0c77fe700 1 -- 172.21.15.23:0/3960667444 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c0004240 con 0x7fb0bc005200 2023-06-22T08:00:26.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.454+0000 7fb0c77fe700 1 -- 172.21.15.23:0/3960667444 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c000f730 con 0x7fb0bc005200 2023-06-22T08:00:26.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.454+0000 7f1a25bff700 1 -- start start 2023-06-22T08:00:26.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.454+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f96e80c4fc0 con 0x7f96e806a280 2023-06-22T08:00:26.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fa009fff700 1 -- 172.21.15.23:0/1200512409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ff0003a00 msgr2=0x7f9ff0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fa009fff700 1 --2- 172.21.15.23:0/1200512409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ff0003a00 0x7f9ff0003e10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009b70 tx=0x7f9ff0005880 comp rx=0 tx=0).stop 2023-06-22T08:00:26.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fa003fff700 1 -- 172.21.15.23:0/1200512409 reap_dead start 2023-06-22T08:00:26.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fa009fff700 1 -- 172.21.15.23:0/1200512409 shutdown_connections 2023-06-22T08:00:26.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fa009fff700 1 -- 172.21.15.23:0/1200512409 >> 172.21.15.23:0/1200512409 conn(0x7fa0040ca3f0 msgr2=0x7fa0040ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fe61299d700 1 -- 172.21.15.23:0/728920667 shutdown_connections 2023-06-22T08:00:26.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fe61299d700 1 --2- 172.21.15.23:0/728920667 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 0x7fe60c15d380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fe61299d700 1 --2- 172.21.15.23:0/728920667 >> v2:172.21.15.23:3300/0 conn(0x7fe60c1570a0 0x7fe60c070fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fe61299d700 1 -- 172.21.15.23:0/728920667 >> 172.21.15.23:0/728920667 conn(0x7fe60c06e3b0 msgr2=0x7fe60c069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fa009fff700 1 -- 172.21.15.23:0/1200512409 shutdown_connections 2023-06-22T08:00:26.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7f1a25bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2006e310 0x7f1a200703a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.456+0000 7f1a25bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a200c5f20 0x7f1a20136780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.456+0000 7f1a25bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1a200caf00 0x7f1a20136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.456+0000 7f1a25bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a20136330 con 0x7f1a200c5f20 2023-06-22T08:00:26.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.456+0000 7f1a25bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a20155b10 con 0x7f1a2006e310 2023-06-22T08:00:26.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.456+0000 7f1a25bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a20153d60 con 0x7f1a200caf00 2023-06-22T08:00:26.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.456+0000 7f1a1ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1a200caf00 0x7f1a20136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.456+0000 7f1a1ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1a200caf00 0x7f1a20136cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:36942/0 (socket says 172.21.15.23:36942) 2023-06-22T08:00:26.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.456+0000 7f1a1ffff700 1 -- 172.21.15.23:0/1765569455 learned_addr learned my addr 172.21.15.23:0/1765569455 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d013ec50 con 0x7fb0bc005200 2023-06-22T08:00:26.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d013f210 con 0x7fb0bc005200 2023-06-22T08:00:26.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.456+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb0d00c4fc0 con 0x7fb0bc005200 2023-06-22T08:00:26.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.455+0000 7fa009fff700 1 -- 172.21.15.23:0/1200512409 wait complete. 2023-06-22T08:00:26.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.454+0000 7f6704bff700 1 -- 172.21.15.23:0/613898359 wait complete. 2023-06-22T08:00:26.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.457+0000 7fe61299d700 1 -- 172.21.15.23:0/728920667 shutdown_connections 2023-06-22T08:00:26.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.457+0000 7fe61299d700 1 -- 172.21.15.23:0/728920667 wait complete. 2023-06-22T08:00:26.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.458+0000 7f1a1f7fe700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2006e310 0x7f1a200703a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.458+0000 7f1a1effd700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a200c5f20 0x7f1a20136780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.458+0000 7f6704bff700 1 Processor -- start 2023-06-22T08:00:26.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.458+0000 7f96e4ff9700 1 -- 172.21.15.23:0/1856714876 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f96d0014950 con 0x7f96e806a280 2023-06-22T08:00:26.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f96e4ff9700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f96c806adb0 0x7f96c806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f96e4ff9700 1 -- 172.21.15.23:0/1856714876 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f96d008a2a0 con 0x7f96e806a280 2023-06-22T08:00:26.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f96e4ff9700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.23:6808/1093903508,v1:172.21.15.23:6809/1093903508] conn(0x7f96c8074a80 0x7f96c8076eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f96e4ff9700 1 -- 172.21.15.23:0/1856714876 --> [v2:172.21.15.23:6808/1093903508,v1:172.21.15.23:6809/1093903508] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96c8077580 con 0x7f96c8074a80 2023-06-22T08:00:26.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f96e4ff9700 1 -- 172.21.15.23:0/1856714876 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96d008a650 con 0x7f96e806a280 2023-06-22T08:00:26.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.461+0000 7f96e7fff700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f96c806adb0 0x7f96c806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.461+0000 7f96ecf9c700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.23:6808/1093903508,v1:172.21.15.23:6809/1093903508] conn(0x7f96c8074a80 0x7f96c8076eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.462+0000 7f96e7fff700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f96c806adb0 0x7f96c806d260 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f96e0004720 tx=0x7f96e0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.462+0000 7f96ecf9c700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.23:6808/1093903508,v1:172.21.15.23:6809/1093903508] conn(0x7f96c8074a80 0x7f96c8076eb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.459+0000 7f1a1effd700 1 -- 172.21.15.23:0/1765569455 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1a200caf00 msgr2=0x7f1a20136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.459+0000 7f1a1effd700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1a200caf00 0x7f1a20136cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.459+0000 7f1a1effd700 1 -- 172.21.15.23:0/1765569455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2006e310 msgr2=0x7f1a200703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.459+0000 7f1a1effd700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a2006e310 0x7f1a200703a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.459+0000 7f1a1effd700 1 -- 172.21.15.23:0/1765569455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a100094a0 con 0x7f1a200c5f20 2023-06-22T08:00:26.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.459+0000 7f1a1effd700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a200c5f20 0x7f1a20136780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800cc50 tx=0x7f1a1800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f1a07fff700 1 -- 172.21.15.23:0/1765569455 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1800ab40 con 0x7f1a200c5f20 2023-06-22T08:00:26.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f1a07fff700 1 -- 172.21.15.23:0/1765569455 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1800aca0 con 0x7f1a200c5f20 2023-06-22T08:00:26.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f1a07fff700 1 -- 172.21.15.23:0/1765569455 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18007310 con 0x7f1a200c5f20 2023-06-22T08:00:26.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a20154040 con 0x7f1a200c5f20 2023-06-22T08:00:26.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a20154600 con 0x7f1a200c5f20 2023-06-22T08:00:26.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.461+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1a20158230 con 0x7f1a200c5f20 2023-06-22T08:00:26.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7f1a07fff700 1 -- 172.21.15.23:0/1765569455 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1a18007470 con 0x7f1a200c5f20 2023-06-22T08:00:26.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7f6704bff700 1 -- start start 2023-06-22T08:00:26.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.461+0000 7f6704bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6700069eb0 0x7f67000c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.461+0000 7f6704bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f670006a280 0x7f6700155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.461+0000 7f6704bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f670015d9b0 0x7f670013c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.461+0000 7f6704bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670013c2f0 con 0x7f670006a280 2023-06-22T08:00:26.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.461+0000 7f6704bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67001fe2d0 con 0x7f6700069eb0 2023-06-22T08:00:26.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.461+0000 7f6704bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67001fe500 con 0x7f670015d9b0 2023-06-22T08:00:26.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.462+0000 7f66fed9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f670015d9b0 0x7f670013c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.462+0000 7f66fed9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f670015d9b0 0x7f670013c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:36954/0 (socket says 172.21.15.23:36954) 2023-06-22T08:00:26.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.462+0000 7f66fed9d700 1 -- 172.21.15.23:0/3321618784 learned_addr learned my addr 172.21.15.23:0/3321618784 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7f66fed9d700 1 -- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6700069eb0 msgr2=0x7f67000c9be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T08:00:26.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7f66fed9d700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6700069eb0 0x7f67000c9be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7f66fed9d700 1 -- 172.21.15.23:0/3321618784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f670006a280 msgr2=0x7f6700155d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T08:00:26.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7f66fed9d700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f670006a280 0x7f6700155d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7f66fed9d700 1 -- 172.21.15.23:0/3321618784 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f8009a70 con 0x7f670015d9b0 2023-06-22T08:00:26.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.464+0000 7f66fed9d700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f670015d9b0 0x7f670013c710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f67000668d0 tx=0x7f66f8004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7fe61299d700 1 Processor -- start 2023-06-22T08:00:26.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.458+0000 7fa009fff700 1 Processor -- start 2023-06-22T08:00:26.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.459+0000 7fa009fff700 1 -- start start 2023-06-22T08:00:26.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7fa009fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0041eb260 0x7fa0041eb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7fa009fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa00414b210 0x7fa00414bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7fa009fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa004155dd0 0x7fa0041409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7fa009fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0040ca170 con 0x7fa004155dd0 2023-06-22T08:00:26.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7fa009fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0041ebd20 con 0x7fa0041eb260 2023-06-22T08:00:26.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.460+0000 7fa009fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0041ef090 con 0x7fa00414b210 2023-06-22T08:00:26.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.462+0000 7fa003fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa004155dd0 0x7fa0041409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.462+0000 7fa003fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa004155dd0 0x7fa0041409d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60824/0 (socket says 172.21.15.23:60824) 2023-06-22T08:00:26.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.462+0000 7fa003fff700 1 -- 172.21.15.23:0/194067813 learned_addr learned my addr 172.21.15.23:0/194067813 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.462+0000 7f9ffaffd700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa00414b210 0x7fa00414bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fa003fff700 1 -- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa00414b210 msgr2=0x7fa00414bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fa003fff700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa00414b210 0x7fa00414bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fa003fff700 1 -- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0041eb260 msgr2=0x7fa0041eb630 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T08:00:26.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fa003fff700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0041eb260 0x7fa0041eb630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fa003fff700 1 -- 172.21.15.23:0/194067813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0040c9d90 con 0x7fa004155dd0 2023-06-22T08:00:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fa003fff700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa004155dd0 0x7fa0041409d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa004072640 tx=0x7f9ffc007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.464+0000 7fa000ff9700 1 -- 172.21.15.23:0/194067813 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc00f5b0 con 0x7fa004155dd0 2023-06-22T08:00:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.464+0000 7fa000ff9700 1 -- 172.21.15.23:0/194067813 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ffc00f710 con 0x7fa004155dd0 2023-06-22T08:00:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.464+0000 7fa000ff9700 1 -- 172.21.15.23:0/194067813 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc00faa0 con 0x7fa004155dd0 2023-06-22T08:00:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.464+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00414c170 con 0x7fa004155dd0 2023-06-22T08:00:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.464+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa004156770 con 0x7fa004155dd0 2023-06-22T08:00:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7f66eeffd700 1 -- 172.21.15.23:0/3321618784 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f80046a0 con 0x7f670015d9b0 2023-06-22T08:00:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7f66eeffd700 1 -- 172.21.15.23:0/3321618784 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f8004800 con 0x7f670015d9b0 2023-06-22T08:00:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7f66eeffd700 1 -- 172.21.15.23:0/3321618784 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f8004970 con 0x7f670015d9b0 2023-06-22T08:00:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.466+0000 7f96e4ff9700 1 -- 172.21.15.23:0/1856714876 <== osd.2 v2:172.21.15.23:6808/1093903508 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f96c8077580 con 0x7f96c8074a80 2023-06-22T08:00:26.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fb0c77fe700 1 -- 172.21.15.23:0/3960667444 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb0c000f890 con 0x7fb0bc005200 2023-06-22T08:00:26.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.466+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67001fe7e0 con 0x7f670015d9b0 2023-06-22T08:00:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.466+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67001feda0 con 0x7f670015d9b0 2023-06-22T08:00:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7fb0c77fe700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0b006adb0 0x7fb0b006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7fb0c77fe700 1 -- 172.21.15.23:0/3960667444 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb0c008a260 con 0x7fb0bc005200 2023-06-22T08:00:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.466+0000 7fb0c77fe700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:6809/1534166692,v1:172.21.15.67:6811/1534166692] conn(0x7fb0b0074a80 0x7fb0b0076eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.466+0000 7fb0c77fe700 1 -- 172.21.15.23:0/3960667444 --> [v2:172.21.15.67:6809/1534166692,v1:172.21.15.67:6811/1534166692] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0b0077580 con 0x7fb0b0074a80 2023-06-22T08:00:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.466+0000 7fb0c77fe700 1 -- 172.21.15.23:0/3960667444 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0c008a610 con 0x7fb0bc005200 2023-06-22T08:00:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.462+0000 7fe61299d700 1 -- start start 2023-06-22T08:00:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fe61299d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 0x7fe60c15f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fe61299d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe60c160b80 0x7fe60c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fe61299d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe60c13e710 0x7fe60c141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fe61299d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c13fd10 con 0x7fe60c13e710 2023-06-22T08:00:26.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fe61299d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c13d470 con 0x7fe60c160b80 2023-06-22T08:00:26.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.463+0000 7fe61299d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c13fa30 con 0x7fe60c15cf00 2023-06-22T08:00:26.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.464+0000 7fe61199b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 0x7fe60c15f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.464+0000 7fe61199b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 0x7fe60c15f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:36976/0 (socket says 172.21.15.23:36976) 2023-06-22T08:00:26.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.464+0000 7fe61199b700 1 -- 172.21.15.23:0/1953182758 learned_addr learned my addr 172.21.15.23:0/1953182758 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.464+0000 7fe61219c700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe60c13e710 0x7fe60c141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7fe61219c700 1 -- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 msgr2=0x7fe60c15f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7fe61219c700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe60c15cf00 0x7fe60c15f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7fe61219c700 1 -- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe60c160b80 msgr2=0x7fe60c15f550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T08:00:26.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7fe61219c700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe60c160b80 0x7fe60c15f550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.465+0000 7fe61219c700 1 -- 172.21.15.23:0/1953182758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6040094a0 con 0x7fe60c13e710 2023-06-22T08:00:26.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.466+0000 7fe61219c700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe60c13e710 0x7fe60c141210 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe604015000 tx=0x7fe604004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.466+0000 7f1a07fff700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1a0c06ade0 0x7f1a0c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.466+0000 7f1a07fff700 1 -- 172.21.15.23:0/1765569455 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1a18089f00 con 0x7f1a200c5f20 2023-06-22T08:00:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.467+0000 7f1a07fff700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.23:6824/236939976,v1:172.21.15.23:6825/236939976] conn(0x7f1a0c074ab0 0x7f1a0c076ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.467+0000 7f1a07fff700 1 -- 172.21.15.23:0/1765569455 --> [v2:172.21.15.23:6824/236939976,v1:172.21.15.23:6825/236939976] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a0c0775b0 con 0x7f1a0c074ab0 2023-06-22T08:00:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.467+0000 7f1a07fff700 1 -- 172.21.15.23:0/1765569455 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a1808a2b0 con 0x7f1a200c5f20 2023-06-22T08:00:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.467+0000 7f1a1ffff700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.23:6824/236939976,v1:172.21.15.23:6825/236939976] conn(0x7f1a0c074ab0 0x7f1a0c076ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.467+0000 7fe5fe7fc700 1 -- 172.21.15.23:0/1953182758 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604003b50 con 0x7fe60c13e710 2023-06-22T08:00:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.467+0000 7fe5fe7fc700 1 -- 172.21.15.23:0/1953182758 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe604042070 con 0x7fe60c13e710 2023-06-22T08:00:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.467+0000 7fe5fe7fc700 1 -- 172.21.15.23:0/1953182758 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604011630 con 0x7fe60c13e710 2023-06-22T08:00:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.468+0000 7f1a1ffff700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.23:6824/236939976,v1:172.21.15.23:6825/236939976] conn(0x7f1a0c074ab0 0x7f1a0c076ee0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.469+0000 7fa000ff9700 1 -- 172.21.15.23:0/194067813 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9ffc00fc00 con 0x7fa004155dd0 2023-06-22T08:00:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.470+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60c13ceb0 con 0x7fe60c13e710 2023-06-22T08:00:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.470+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60c140cf0 con 0x7fe60c13e710 2023-06-22T08:00:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.470+0000 7fb0c7fff700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0b006adb0 0x7fb0b006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.470+0000 7f1a07fff700 1 -- 172.21.15.23:0/1765569455 <== osd.3 v2:172.21.15.23:6824/236939976 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1a0c0775b0 con 0x7f1a0c074ab0 2023-06-22T08:00:26.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.470+0000 7fa000ff9700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fe406ade0 0x7f9fe406d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.470+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f66e4000fc0 con 0x7f670015d9b0 2023-06-22T08:00:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.470+0000 7fb0cf7fe700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:6809/1534166692,v1:172.21.15.67:6811/1534166692] conn(0x7fb0b0074a80 0x7fb0b0076eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.470+0000 7fa000ff9700 1 -- 172.21.15.23:0/194067813 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9ffc089350 con 0x7fa004155dd0 2023-06-22T08:00:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.471+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe60c06aaf0 con 0x7fe60c13e710 2023-06-22T08:00:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.470+0000 7f1a1f7fe700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1a0c06ade0 0x7f1a0c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.472+0000 7f1a1f7fe700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1a0c06ade0 0x7f1a0c06d290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1a14000f60 tx=0x7f1a14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.472+0000 7fb0cf7fe700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:6809/1534166692,v1:172.21.15.67:6811/1534166692] conn(0x7fb0b0074a80 0x7fb0b0076eb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.472+0000 7fb0c7fff700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0b006adb0 0x7fb0b006d260 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00c1450 tx=0x7fb0b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.473+0000 7fb0c77fe700 1 -- 172.21.15.23:0/3960667444 <== osd.4 v2:172.21.15.67:6809/1534166692 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb0b0077580 con 0x7fb0b0074a80 2023-06-22T08:00:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.472+0000 7f66eeffd700 1 -- 172.21.15.23:0/3321618784 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f66f8012630 con 0x7f670015d9b0 2023-06-22T08:00:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.471+0000 7fa009fff700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:6800/471678163,v1:172.21.15.67:6801/471678163] conn(0x7f9fe80015e0 0x7f9fe8003a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.471+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 --> [v2:172.21.15.67:6800/471678163,v1:172.21.15.67:6801/471678163] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fe8006bc0 con 0x7f9fe80015e0 2023-06-22T08:00:26.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.472+0000 7f9ffaffd700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:6800/471678163,v1:172.21.15.67:6801/471678163] conn(0x7f9fe80015e0 0x7f9fe8003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.472+0000 7fa0037fe700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fe406ade0 0x7f9fe406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.473+0000 7f9ffaffd700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:6800/471678163,v1:172.21.15.67:6801/471678163] conn(0x7f9fe80015e0 0x7f9fe8003a90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.473+0000 7f66eeffd700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66e006adb0 0x7f66e006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.474+0000 7f66eeffd700 1 -- 172.21.15.23:0/3321618784 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f66f808b610 con 0x7f670015d9b0 2023-06-22T08:00:26.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.474+0000 7f66eeffd700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:6808/1884490442,v1:172.21.15.67:6810/1884490442] conn(0x7f66e0074a80 0x7f66e0076eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.474+0000 7f66eeffd700 1 -- 172.21.15.23:0/3321618784 --> [v2:172.21.15.67:6808/1884490442,v1:172.21.15.67:6810/1884490442] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66e0077580 con 0x7f66e0074a80 2023-06-22T08:00:26.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.474+0000 7f66eeffd700 1 -- 172.21.15.23:0/3321618784 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66f808dc30 con 0x7f670015d9b0 2023-06-22T08:00:26.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.474+0000 7f66fe59c700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66e006adb0 0x7f66e006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.475+0000 7fa000ff9700 1 -- 172.21.15.23:0/194067813 <== osd.7 v2:172.21.15.67:6800/471678163 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9fe8006bc0 con 0x7f9fe80015e0 2023-06-22T08:00:26.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.474+0000 7f66fdd9b700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:6808/1884490442,v1:172.21.15.67:6810/1884490442] conn(0x7f66e0074a80 0x7f66e0076eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.475+0000 7f66fe59c700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66e006adb0 0x7f66e006d260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f66f0009a60 tx=0x7f66f00092c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.476+0000 7f66fdd9b700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:6808/1884490442,v1:172.21.15.67:6810/1884490442] conn(0x7f66e0074a80 0x7f66e0076eb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.477+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 --> [v2:172.21.15.23:6800/458708400,v1:172.21.15.23:6801/458708400] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3284002d40 con 0x7f3280074a80 2023-06-22T08:00:26.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.477+0000 7f66eeffd700 1 -- 172.21.15.23:0/3321618784 <== osd.5 v2:172.21.15.67:6808/1884490442 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f66e0077580 con 0x7f66e0074a80 2023-06-22T08:00:26.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.476+0000 7fe5fe7fc700 1 -- 172.21.15.23:0/1953182758 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe604044030 con 0x7fe60c13e710 2023-06-22T08:00:26.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.477+0000 7f328effd700 1 -- 172.21.15.23:0/1322132478 <== osd.0 v2:172.21.15.23:6800/458708400 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3284002d40 con 0x7f3280074a80 2023-06-22T08:00:26.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.478+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 >> [v2:172.21.15.23:6800/458708400,v1:172.21.15.23:6801/458708400] conn(0x7f3280074a80 msgr2=0x7f3280076eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.478+0000 7f32a4fff700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.23:6800/458708400,v1:172.21.15.23:6801/458708400] conn(0x7f3280074a80 0x7f3280076eb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.478+0000 7fe5fe7fc700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe5f806af10 0x7fe5f806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.478+0000 7fe5fe7fc700 1 -- 172.21.15.23:0/1953182758 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe6040ad7b0 con 0x7fe60c13e710 2023-06-22T08:00:26.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.478+0000 7fe61199b700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe5f806af10 0x7fe5f806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.478+0000 7fe5fe7fc700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:6824/3723802251,v1:172.21.15.67:6825/3723802251] conn(0x7fe5f8074be0 0x7fe5f8077010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.478+0000 7fe5fe7fc700 1 -- 172.21.15.23:0/1953182758 --> [v2:172.21.15.67:6824/3723802251,v1:172.21.15.67:6825/3723802251] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5f80776e0 con 0x7fe5f8074be0 2023-06-22T08:00:26.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.478+0000 7fe5fe7fc700 1 -- 172.21.15.23:0/1953182758 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6040adb60 con 0x7fe60c13e710 2023-06-22T08:00:26.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.479+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f328006adb0 msgr2=0x7f328006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.479+0000 7f32a4fff700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f328006adb0 0x7f328006d260 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3294007af0 tx=0x7f3294008040 comp rx=0 tx=0).stop 2023-06-22T08:00:26.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.479+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f32a006e310 msgr2=0x7f32a0149010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.479+0000 7f32a4fff700 1 --2- 172.21.15.23:0/1322132478 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f32a006e310 0x7f32a0149010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f329000d710 tx=0x7f329000da20 comp rx=0 tx=0).stop 2023-06-22T08:00:26.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.479+0000 7f329ed9d700 1 -- 172.21.15.23:0/1322132478 reap_dead start 2023-06-22T08:00:26.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.479+0000 7fe61119a700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:6824/3723802251,v1:172.21.15.67:6825/3723802251] conn(0x7fe5f8074be0 0x7fe5f8077010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.480+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 shutdown_connections 2023-06-22T08:00:26.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.480+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 >> 172.21.15.23:0/1322132478 conn(0x7f32a00ca3f0 msgr2=0x7f32a0070470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.479+0000 7fe61199b700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe5f806af10 0x7fe5f806d3c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe600009c20 tx=0x7fe600009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.479+0000 7fa0037fe700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fe406ade0 0x7f9fe406d290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009b70 tx=0x7f9ff4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.480+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 shutdown_connections 2023-06-22T08:00:26.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.480+0000 7f32a4fff700 1 -- 172.21.15.23:0/1322132478 wait complete. 2023-06-22T08:00:26.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.480+0000 7fe61119a700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:6824/3723802251,v1:172.21.15.67:6825/3723802251] conn(0x7fe5f8074be0 0x7fe5f8077010 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.481+0000 7fe5fe7fc700 1 -- 172.21.15.23:0/1953182758 <== osd.6 v2:172.21.15.67:6824/3723802251 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe5f80776e0 con 0x7fe5f8074be0 2023-06-22T08:00:26.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.483+0000 7fdec5ffb700 1 -- 172.21.15.23:0/2365544436 --> [v2:172.21.15.23:6816/3464394342,v1:172.21.15.23:6817/3464394342] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdeb8002d40 con 0x7fdecc074a80 2023-06-22T08:00:26.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.485+0000 7fdec7fff700 1 -- 172.21.15.23:0/2365544436 <== osd.1 v2:172.21.15.23:6816/3464394342 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdeb8002d40 con 0x7fdecc074a80 2023-06-22T08:00:26.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.485+0000 7fdec5ffb700 1 -- 172.21.15.23:0/2365544436 >> [v2:172.21.15.23:6816/3464394342,v1:172.21.15.23:6817/3464394342] conn(0x7fdecc074a80 msgr2=0x7fdecc076eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.485+0000 7fdec5ffb700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.23:6816/3464394342,v1:172.21.15.23:6817/3464394342] conn(0x7fdecc074a80 0x7fdecc076eb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.486+0000 7fdec5ffb700 1 -- 172.21.15.23:0/2365544436 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdecc06adb0 msgr2=0x7fdecc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.486+0000 7fdec5ffb700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdecc06adb0 0x7fdecc06d260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fded8004c50 tx=0x7fded8009030 comp rx=0 tx=0).stop 2023-06-22T08:00:26.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.486+0000 7fdec5ffb700 1 -- 172.21.15.23:0/2365544436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdee0157400 msgr2=0x7fdee01fc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.486+0000 7fdec5ffb700 1 --2- 172.21.15.23:0/2365544436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdee0157400 0x7fdee01fc7e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdee006b440 tx=0x7fded0010a70 comp rx=0 tx=0).stop 2023-06-22T08:00:26.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.486+0000 7fdedf7fe700 1 -- 172.21.15.23:0/2365544436 reap_dead start 2023-06-22T08:00:26.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.487+0000 7fdec5ffb700 1 -- 172.21.15.23:0/2365544436 shutdown_connections 2023-06-22T08:00:26.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.487+0000 7fdec5ffb700 1 -- 172.21.15.23:0/2365544436 >> 172.21.15.23:0/2365544436 conn(0x7fdee00ca3f0 msgr2=0x7fdee00c3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.488+0000 7fdec5ffb700 1 -- 172.21.15.23:0/2365544436 shutdown_connections 2023-06-22T08:00:26.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.488+0000 7fdec5ffb700 1 -- 172.21.15.23:0/2365544436 wait complete. 2023-06-22T08:00:26.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.506+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 --> [v2:172.21.15.23:6808/1093903508,v1:172.21.15.23:6809/1093903508] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f96e8071620 con 0x7f96c8074a80 2023-06-22T08:00:26.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.506+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 --> [v2:172.21.15.23:6824/236939976,v1:172.21.15.23:6825/236939976] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1a20157f50 con 0x7f1a0c074ab0 2023-06-22T08:00:26.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.507+0000 7f1a07fff700 1 -- 172.21.15.23:0/1765569455 <== osd.3 v2:172.21.15.23:6824/236939976 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1a10006c60 con 0x7f1a0c074ab0 2023-06-22T08:00:26.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.508+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 >> [v2:172.21.15.23:6824/236939976,v1:172.21.15.23:6825/236939976] conn(0x7f1a0c074ab0 msgr2=0x7f1a0c076ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.508+0000 7f1a25bff700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.23:6824/236939976,v1:172.21.15.23:6825/236939976] conn(0x7f1a0c074ab0 0x7f1a0c076ee0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.509+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 --> [v2:172.21.15.67:6808/1884490442,v1:172.21.15.67:6810/1884490442] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f66e4002d40 con 0x7f66e0074a80 2023-06-22T08:00:26.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.509+0000 7f96e4ff9700 1 -- 172.21.15.23:0/1856714876 <== osd.2 v2:172.21.15.23:6808/1093903508 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f96d80050b0 con 0x7f96c8074a80 2023-06-22T08:00:26.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.509+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1a0c06ade0 msgr2=0x7f1a0c06d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.509+0000 7f1a25bff700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1a0c06ade0 0x7f1a0c06d290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1a14000f60 tx=0x7f1a14009040 comp rx=0 tx=0).stop 2023-06-22T08:00:26.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.509+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a200c5f20 msgr2=0x7f1a20136780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.509+0000 7f1a25bff700 1 --2- 172.21.15.23:0/1765569455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a200c5f20 0x7f1a20136780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800cc50 tx=0x7f1a1800cf60 comp rx=0 tx=0).stop 2023-06-22T08:00:26.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.510+0000 7f1a1ffff700 1 -- 172.21.15.23:0/1765569455 reap_dead start 2023-06-22T08:00:26.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.510+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 shutdown_connections 2023-06-22T08:00:26.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.510+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 >> 172.21.15.23:0/1765569455 conn(0x7f1a200ca3f0 msgr2=0x7f1a2014eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.510+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 shutdown_connections 2023-06-22T08:00:26.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.510+0000 7f66eeffd700 1 -- 172.21.15.23:0/3321618784 <== osd.5 v2:172.21.15.67:6808/1884490442 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f66e4002d40 con 0x7f66e0074a80 2023-06-22T08:00:26.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.510+0000 7f1a25bff700 1 -- 172.21.15.23:0/1765569455 wait complete. 2023-06-22T08:00:26.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 >> [v2:172.21.15.23:6808/1093903508,v1:172.21.15.23:6809/1093903508] conn(0x7f96c8074a80 msgr2=0x7f96c8076eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f96ee9ff700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.23:6808/1093903508,v1:172.21.15.23:6809/1093903508] conn(0x7f96c8074a80 0x7f96c8076eb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:6808/1884490442,v1:172.21.15.67:6810/1884490442] conn(0x7f66e0074a80 msgr2=0x7f66e0076eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f6704bff700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:6808/1884490442,v1:172.21.15.67:6810/1884490442] conn(0x7f66e0074a80 0x7f66e0076eb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f96c806adb0 msgr2=0x7f96c806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f96ee9ff700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f96c806adb0 0x7f96c806d260 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f96e0004720 tx=0x7f96e0008030 comp rx=0 tx=0).stop 2023-06-22T08:00:26.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96e806a280 msgr2=0x7f96e81f2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f96ee9ff700 1 --2- 172.21.15.23:0/1856714876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96e806a280 0x7f96e81f2e60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f96d000bd20 tx=0x7f96d000be00 comp rx=0 tx=0).stop 2023-06-22T08:00:26.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66e006adb0 msgr2=0x7f66e006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f6704bff700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66e006adb0 0x7f66e006d260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f66f0009a60 tx=0x7f66f00092c0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 --> [v2:172.21.15.67:6809/1534166692,v1:172.21.15.67:6811/1534166692] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb0d0071620 con 0x7fb0b0074a80 2023-06-22T08:00:26.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f670015d9b0 msgr2=0x7f670013c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.511+0000 7f6704bff700 1 --2- 172.21.15.23:0/3321618784 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f670015d9b0 0x7f670013c710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f67000668d0 tx=0x7f66f8004ad0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.553 INFO:teuthology.orchestra.run.smithi023.stdout:60129542156 2023-06-22T08:00:26.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.512+0000 7f96ecf9c700 1 -- 172.21.15.23:0/1856714876 reap_dead start 2023-06-22T08:00:26.555 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-06-22T08:00:26.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.512+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 shutdown_connections 2023-06-22T08:00:26.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.512+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 >> 172.21.15.23:0/1856714876 conn(0x7f96e80ca3f0 msgr2=0x7f96e8155a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.512+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 shutdown_connections 2023-06-22T08:00:26.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.512+0000 7f96ee9ff700 1 -- 172.21.15.23:0/1856714876 wait complete. 2023-06-22T08:00:26.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.512+0000 7f66fed9d700 1 -- 172.21.15.23:0/3321618784 reap_dead start 2023-06-22T08:00:26.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.512+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 shutdown_connections 2023-06-22T08:00:26.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.512+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 >> 172.21.15.23:0/3321618784 conn(0x7f67000ca3f0 msgr2=0x7f6700153080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.512+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 shutdown_connections 2023-06-22T08:00:26.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.512+0000 7f6704bff700 1 -- 172.21.15.23:0/3321618784 wait complete. 2023-06-22T08:00:26.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.513+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 --> [v2:172.21.15.67:6800/471678163,v1:172.21.15.67:6801/471678163] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9fe8005c40 con 0x7f9fe80015e0 2023-06-22T08:00:26.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.513+0000 7fb0c77fe700 1 -- 172.21.15.23:0/3960667444 <== osd.4 v2:172.21.15.67:6809/1534166692 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb0c80070c0 con 0x7fb0b0074a80 2023-06-22T08:00:26.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.513+0000 7fa000ff9700 1 -- 172.21.15.23:0/194067813 <== osd.7 v2:172.21.15.67:6800/471678163 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9fe8005c40 con 0x7f9fe80015e0 2023-06-22T08:00:26.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.514+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:6809/1534166692,v1:172.21.15.67:6811/1534166692] conn(0x7fb0b0074a80 msgr2=0x7fb0b0076eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.514+0000 7fb0d55ff700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:6809/1534166692,v1:172.21.15.67:6811/1534166692] conn(0x7fb0b0074a80 0x7fb0b0076eb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.514+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0b006adb0 msgr2=0x7fb0b006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.514+0000 7fb0d55ff700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0b006adb0 0x7fb0b006d260 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00c1450 tx=0x7fb0b8009040 comp rx=0 tx=0).stop 2023-06-22T08:00:26.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.514+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc005200 msgr2=0x7fb0d0135490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.514+0000 7fb0d55ff700 1 --2- 172.21.15.23:0/3960667444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc005200 0x7fb0d0135490 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0011870 tx=0x7fb0c0011c30 comp rx=0 tx=0).stop 2023-06-22T08:00:26.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.514+0000 7fb0cf7fe700 1 -- 172.21.15.23:0/3960667444 reap_dead start 2023-06-22T08:00:26.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.514+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 shutdown_connections 2023-06-22T08:00:26.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.514+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 >> 172.21.15.23:0/3960667444 conn(0x7fb0d00ca3f0 msgr2=0x7fb0d006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.515+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 shutdown_connections 2023-06-22T08:00:26.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.515+0000 7fb0d55ff700 1 -- 172.21.15.23:0/3960667444 wait complete. 2023-06-22T08:00:26.564 INFO:teuthology.orchestra.run.smithi023.stdout:55834574860 2023-06-22T08:00:26.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-06-22T08:00:26.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.516+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:6800/471678163,v1:172.21.15.67:6801/471678163] conn(0x7f9fe80015e0 msgr2=0x7f9fe8003a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.516+0000 7fa009fff700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:6800/471678163,v1:172.21.15.67:6801/471678163] conn(0x7f9fe80015e0 0x7f9fe8003a90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.516+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fe406ade0 msgr2=0x7f9fe406d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.516+0000 7fa009fff700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fe406ade0 0x7f9fe406d290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009b70 tx=0x7f9ff4008030 comp rx=0 tx=0).stop 2023-06-22T08:00:26.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.516+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa004155dd0 msgr2=0x7fa0041409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.516+0000 7fa009fff700 1 --2- 172.21.15.23:0/194067813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa004155dd0 0x7fa0041409d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa004072640 tx=0x7f9ffc007620 comp rx=0 tx=0).stop 2023-06-22T08:00:26.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.516+0000 7fa003fff700 1 -- 172.21.15.23:0/194067813 reap_dead start 2023-06-22T08:00:26.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.517+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 --> [v2:172.21.15.67:6824/3723802251,v1:172.21.15.67:6825/3723802251] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe60c06a810 con 0x7fe5f8074be0 2023-06-22T08:00:26.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.517+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 shutdown_connections 2023-06-22T08:00:26.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.517+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 >> 172.21.15.23:0/194067813 conn(0x7fa0040ca3f0 msgr2=0x7fa00406fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.517+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 shutdown_connections 2023-06-22T08:00:26.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.517+0000 7fa009fff700 1 -- 172.21.15.23:0/194067813 wait complete. 2023-06-22T08:00:26.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.517+0000 7fe5fe7fc700 1 -- 172.21.15.23:0/1953182758 <== osd.6 v2:172.21.15.67:6824/3723802251 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe608001d10 con 0x7fe5f8074be0 2023-06-22T08:00:26.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.518+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:6824/3723802251,v1:172.21.15.67:6825/3723802251] conn(0x7fe5f8074be0 msgr2=0x7fe5f8077010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.518+0000 7fe61299d700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:6824/3723802251,v1:172.21.15.67:6825/3723802251] conn(0x7fe5f8074be0 0x7fe5f8077010 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.518+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe5f806af10 msgr2=0x7fe5f806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.518+0000 7fe61299d700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe5f806af10 0x7fe5f806d3c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe600009c20 tx=0x7fe600009210 comp rx=0 tx=0).stop 2023-06-22T08:00:26.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.518+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe60c13e710 msgr2=0x7fe60c141210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.518+0000 7fe61299d700 1 --2- 172.21.15.23:0/1953182758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe60c13e710 0x7fe60c141210 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe604015000 tx=0x7fe604004750 comp rx=0 tx=0).stop 2023-06-22T08:00:26.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.518+0000 7fe61219c700 1 -- 172.21.15.23:0/1953182758 reap_dead start 2023-06-22T08:00:26.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.518+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 shutdown_connections 2023-06-22T08:00:26.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.518+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 >> 172.21.15.23:0/1953182758 conn(0x7fe60c1570a0 msgr2=0x7fe60c152880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.519+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 shutdown_connections 2023-06-22T08:00:26.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.519+0000 7fe61299d700 1 -- 172.21.15.23:0/1953182758 wait complete. 2023-06-22T08:00:26.571 INFO:teuthology.orchestra.run.smithi023.stdout:60129542156 2023-06-22T08:00:26.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-06-22T08:00:26.572 INFO:teuthology.orchestra.run.smithi023.stdout:55834574861 2023-06-22T08:00:26.572 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-06-22T08:00:26.573 INFO:teuthology.orchestra.run.smithi023.stdout:68719476749 2023-06-22T08:00:26.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-06-22T08:00:26.574 INFO:teuthology.orchestra.run.smithi023.stdout:55834574860 2023-06-22T08:00:26.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-06-22T08:00:26.575 INFO:teuthology.orchestra.run.smithi023.stdout:55834574860 2023-06-22T08:00:26.575 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-06-22T08:00:26.576 INFO:teuthology.orchestra.run.smithi023.stdout:55834574861 2023-06-22T08:00:26.576 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-06-22T08:00:26.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.954+0000 7eff5f1ff700 1 Processor -- start 2023-06-22T08:00:26.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.954+0000 7eff5f1ff700 1 -- start start 2023-06-22T08:00:26.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.957+0000 7eff5f1ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7eff5806d050 0x7eff5806a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.957+0000 7eff5f1ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff580c4d70 con 0x7eff5815d9b0 2023-06-22T08:00:26.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.957+0000 7eff5f1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff580c5610 con 0x7eff58069eb0 2023-06-22T08:00:26.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.957+0000 7eff5f1ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff580703c0 con 0x7eff5806d050 2023-06-22T08:00:26.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.958+0000 7eff5cf9b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7eff58069eb0 0x7eff5806fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58660/0 (socket says 172.21.15.23:58660) 2023-06-22T08:00:26.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.958+0000 7eff5cf9b700 1 -- 172.21.15.23:0/2951057728 learned_addr learned my addr 172.21.15.23:0/2951057728 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:26.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.959+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712129104 0 0) 0x7eff580c5610 con 0x7eff58069eb0 2023-06-22T08:00:26.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.959+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff400035b0 con 0x7eff58069eb0 2023-06-22T08:00:26.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.959+0000 7f8904bff700 1 Processor -- start 2023-06-22T08:00:26.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.959+0000 7f8904bff700 1 -- start start 2023-06-22T08:00:26.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.959+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4218950946 0 0) 0x7eff400035b0 con 0x7eff58069eb0 2023-06-22T08:00:26.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.959+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 >> v2:172.21.15.23:3300/0 conn(0x7eff5806d050 msgr2=0x7eff5806a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.959+0000 7eff577fe700 1 --2- 172.21.15.23:0/2951057728 >> v2:172.21.15.23:3300/0 conn(0x7eff5806d050 0x7eff5806a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.959+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 >> v1:172.21.15.23:6789/0 conn(0x7eff5815d9b0 legacy=0x7eff58165590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.959+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff58071340 con 0x7eff58069eb0 2023-06-22T08:00:26.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.959+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7eff48001670 con 0x7eff58069eb0 2023-06-22T08:00:26.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.960+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 >> v1:172.21.15.67:6789/0 conn(0x7eff58069eb0 legacy=0x7eff5806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.960+0000 7eff577fe700 1 --2- 172.21.15.23:0/2951057728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff40003a00 0x7eff40003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.960+0000 7eff577fe700 1 --2- 172.21.15.23:0/2951057728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff40004850 0x7eff40004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.960+0000 7eff577fe700 1 --2- 172.21.15.23:0/2951057728 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 0x7eff4000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.960+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff580703c0 con 0x7eff40004850 2023-06-22T08:00:26.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.960+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4000c1d0 con 0x7eff40003a00 2023-06-22T08:00:26.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.960+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4000c390 con 0x7eff40005320 2023-06-22T08:00:26.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.960+0000 7eff5d79c700 1 --2- 172.21.15.23:0/2951057728 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 0x7eff4000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.961+0000 7eff5cf9b700 1 --2- 172.21.15.23:0/2951057728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff40003a00 0x7eff40003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.961+0000 7eff5d79c700 1 -- 172.21.15.23:0/2951057728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff40003a00 msgr2=0x7eff40003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.961+0000 7eff5d79c700 1 --2- 172.21.15.23:0/2951057728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff40003a00 0x7eff40003df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.961+0000 7eff5d79c700 1 -- 172.21.15.23:0/2951057728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff40004850 msgr2=0x7eff40004cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:00:26.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.961+0000 7eff5d79c700 1 --2- 172.21.15.23:0/2951057728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff40004850 0x7eff40004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.961+0000 7eff5d79c700 1 -- 172.21.15.23:0/2951057728 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff58071340 con 0x7eff40005320 2023-06-22T08:00:26.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.961+0000 7eff5d79c700 1 -- 172.21.15.23:0/2951057728 reap_dead start 2023-06-22T08:00:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.962+0000 7eff5d79c700 1 --2- 172.21.15.23:0/2951057728 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 0x7eff4000ba10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7eff50009e40 tx=0x7eff40005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83f47dfc4126b1bd server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.962+0000 7f8904bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890015d9b0 0x7f890006d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.962+0000 7f8904bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89000c4d70 con 0x7f8900069eb0 2023-06-22T08:00:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.962+0000 7f8904bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89000c5610 con 0x7f890006a280 2023-06-22T08:00:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.962+0000 7f8904bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89000703c0 con 0x7f890015d9b0 2023-06-22T08:00:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.962+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50003c00 con 0x7eff40005320 2023-06-22T08:00:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.962+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff50003d80 con 0x7eff40005320 2023-06-22T08:00:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.962+0000 7eff577fe700 1 -- 172.21.15.23:0/2951057728 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5000bae0 con 0x7eff40005320 2023-06-22T08:00:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.963+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2951057728 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 msgr2=0x7eff4000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.963+0000 7eff5f1ff700 1 --2- 172.21.15.23:0/2951057728 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 0x7eff4000ba10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7eff50009e40 tx=0x7eff40005880 comp rx=0 tx=0).stop 2023-06-22T08:00:26.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.964+0000 7f88fe59c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f8900069eb0 0x7f890006fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41382/0 (socket says 172.21.15.23:41382) 2023-06-22T08:00:26.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.964+0000 7f88fe59c700 1 -- 172.21.15.23:0/13633542 learned_addr learned my addr 172.21.15.23:0/13633542 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:26.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.964+0000 7f88fdd9b700 1 --2- 172.21.15.23:0/13633542 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890015d9b0 0x7f890006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.965+0000 7f88fdd9b700 1 -- 172.21.15.23:0/13633542 >> v1:172.21.15.67:6789/0 conn(0x7f890006a280 legacy=0x7f89001675a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.965+0000 7f88fdd9b700 1 -- 172.21.15.23:0/13633542 >> v1:172.21.15.23:6789/0 conn(0x7f8900069eb0 legacy=0x7f890006fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.965+0000 7f88fdd9b700 1 -- 172.21.15.23:0/13633542 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8900071340 con 0x7f890015d9b0 2023-06-22T08:00:26.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.965+0000 7f88fdd9b700 1 --2- 172.21.15.23:0/13633542 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890015d9b0 0x7f890006d050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009870 tx=0x7f88f0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53e0a4963329f3a4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.965+0000 7f88fd59a700 1 -- 172.21.15.23:0/13633542 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0004230 con 0x7f890015d9b0 2023-06-22T08:00:26.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.966+0000 7f88fd59a700 1 -- 172.21.15.23:0/13633542 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0004390 con 0x7f890015d9b0 2023-06-22T08:00:26.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.966+0000 7f88fd59a700 1 -- 172.21.15.23:0/13633542 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f00036a0 con 0x7f890015d9b0 2023-06-22T08:00:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.966+0000 7f8904bff700 1 -- 172.21.15.23:0/13633542 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890015d9b0 msgr2=0x7f890006d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.966+0000 7f8904bff700 1 --2- 172.21.15.23:0/13633542 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890015d9b0 0x7f890006d050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009870 tx=0x7f88f0009b80 comp rx=0 tx=0).stop 2023-06-22T08:00:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.967+0000 7f8904bff700 1 -- 172.21.15.23:0/13633542 shutdown_connections 2023-06-22T08:00:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.967+0000 7f8904bff700 1 --2- 172.21.15.23:0/13633542 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890015d9b0 0x7f890006d050 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.967+0000 7f8904bff700 1 -- 172.21.15.23:0/13633542 >> 172.21.15.23:0/13633542 conn(0x7f89000ca3f0 msgr2=0x7f89000ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.967+0000 7f8904bff700 1 -- 172.21.15.23:0/13633542 shutdown_connections 2023-06-22T08:00:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.967+0000 7f8904bff700 1 -- 172.21.15.23:0/13633542 wait complete. 2023-06-22T08:00:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.969+0000 7f8904bff700 1 Processor -- start 2023-06-22T08:00:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.969+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2951057728 shutdown_connections 2023-06-22T08:00:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.969+0000 7eff5f1ff700 1 --2- 172.21.15.23:0/2951057728 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 0x7eff4000ba10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.969+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2951057728 >> 172.21.15.23:0/2951057728 conn(0x7eff580ca3f0 msgr2=0x7eff580ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.969+0000 7f8904bff700 1 -- start start 2023-06-22T08:00:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.969+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2951057728 shutdown_connections 2023-06-22T08:00:26.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.971+0000 7f8904bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8900069eb0 0x7f890015a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.971+0000 7f8904bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890006a280 0x7f89000c0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.971+0000 7f8904bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890015d9b0 0x7f8900159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.971+0000 7f8904bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000c3a30 con 0x7f8900069eb0 2023-06-22T08:00:26.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.971+0000 7f8904bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8900151a40 con 0x7f890015d9b0 2023-06-22T08:00:26.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.971+0000 7f8904bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89000b7640 con 0x7f890006a280 2023-06-22T08:00:26.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.970+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2951057728 wait complete. 2023-06-22T08:00:26.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.971+0000 7f88fdd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890006a280 0x7f89000c0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.971+0000 7f88fed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890015d9b0 0x7f8900159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.971+0000 7f88fed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890015d9b0 0x7f8900159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:47950/0 (socket says 172.21.15.23:47950) 2023-06-22T08:00:26.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.972+0000 7f88fed9d700 1 -- 172.21.15.23:0/729105698 learned_addr learned my addr 172.21.15.23:0/729105698 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.972+0000 7f88fed9d700 1 -- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890006a280 msgr2=0x7f89000c0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.972+0000 7f88fed9d700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890006a280 0x7f89000c0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.972+0000 7f88fed9d700 1 -- 172.21.15.23:0/729105698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8900069eb0 msgr2=0x7f890015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.972+0000 7f88fed9d700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8900069eb0 0x7f890015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.972+0000 7f88fed9d700 1 -- 172.21.15.23:0/729105698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f00094a0 con 0x7f890015d9b0 2023-06-22T08:00:26.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.972+0000 7f88fed9d700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890015d9b0 0x7f8900159130 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f88f400c5b0 tx=0x7f88f400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.973+0000 7eff5f1ff700 1 Processor -- start 2023-06-22T08:00:26.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.973+0000 7f88eeffd700 1 -- 172.21.15.23:0/729105698 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f40054e0 con 0x7f890015d9b0 2023-06-22T08:00:26.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.973+0000 7f88eeffd700 1 -- 172.21.15.23:0/729105698 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f4003680 con 0x7f890015d9b0 2023-06-22T08:00:26.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.973+0000 7f88eeffd700 1 -- 172.21.15.23:0/729105698 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4003a30 con 0x7f890015d9b0 2023-06-22T08:00:26.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.973+0000 7f2bfa3ff700 1 Processor -- start 2023-06-22T08:00:26.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.973+0000 7eff5f1ff700 1 -- start start 2023-06-22T08:00:26.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.974+0000 7eff5f1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 0x7eff58151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.974+0000 7eff5f1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff581fba90 0x7eff5815c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.974+0000 7eff5f1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff58160510 0x7eff58160980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.974+0000 7eff5f1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff58158410 con 0x7eff58160510 2023-06-22T08:00:26.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.974+0000 7eff5f1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff58160ec0 con 0x7eff581fba90 2023-06-22T08:00:26.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.974+0000 7eff5f1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5815efb0 con 0x7eff40005320 2023-06-22T08:00:26.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.973+0000 7f8904bff700 1 -- 172.21.15.23:0/729105698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89000b7360 con 0x7f890015d9b0 2023-06-22T08:00:26.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.973+0000 7f8904bff700 1 -- 172.21.15.23:0/729105698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89000c3370 con 0x7f890015d9b0 2023-06-22T08:00:26.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.974+0000 7f2bfa3ff700 1 -- start start 2023-06-22T08:00:26.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.976+0000 7eff57fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff581fba90 0x7eff5815c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.976+0000 7eff57fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff581fba90 0x7eff5815c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:47956/0 (socket says 172.21.15.23:47956) 2023-06-22T08:00:26.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.976+0000 7eff57fff700 1 -- 172.21.15.23:0/2249495348 learned_addr learned my addr 172.21.15.23:0/2249495348 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.976+0000 7eff5d79c700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff58160510 0x7eff58160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.976+0000 7eff5cf9b700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 0x7eff58151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.976+0000 7f8904bff700 1 -- 172.21.15.23:0/729105698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e40052f0 con 0x7f890015d9b0 2023-06-22T08:00:26.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.977+0000 7eff5d79c700 1 -- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 msgr2=0x7eff58151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.977+0000 7eff5d79c700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 0x7eff58151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.977+0000 7eff5d79c700 1 -- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff581fba90 msgr2=0x7eff5815c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.977+0000 7eff5d79c700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff581fba90 0x7eff5815c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.977+0000 7eff5d79c700 1 -- 172.21.15.23:0/2249495348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff58157d70 con 0x7eff58160510 2023-06-22T08:00:26.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.977+0000 7eff5d79c700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff58160510 0x7eff58160980 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7eff5815dd80 tx=0x7eff50004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.978+0000 7eff557fa700 1 -- 172.21.15.23:0/2249495348 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50015830 con 0x7eff58160510 2023-06-22T08:00:26.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.978+0000 7eff557fa700 1 -- 172.21.15.23:0/2249495348 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff50015990 con 0x7eff58160510 2023-06-22T08:00:26.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.978+0000 7eff557fa700 1 -- 172.21.15.23:0/2249495348 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50015d40 con 0x7eff58160510 2023-06-22T08:00:26.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.979+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2249495348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff58157a90 con 0x7eff58160510 2023-06-22T08:00:26.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.979+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2249495348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5815ecd0 con 0x7eff58160510 2023-06-22T08:00:26.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.981+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2249495348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5806cc60 con 0x7eff58160510 2023-06-22T08:00:26.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.981+0000 7f88eeffd700 1 -- 172.21.15.23:0/729105698 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f88f4003b90 con 0x7f890015d9b0 2023-06-22T08:00:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.982+0000 7f88eeffd700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006adb0 0x7f88e006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.982+0000 7f88eeffd700 1 -- 172.21.15.23:0/729105698 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f88f4088f60 con 0x7f890015d9b0 2023-06-22T08:00:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.984+0000 7f88fe59c700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006adb0 0x7f88e006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.982+0000 7f2bfa3ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2bf4069eb0 0x7f2bf406d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.982+0000 7f2bfa3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2bf415d9b0 0x7f2bf406a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.982+0000 7f2bfa3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bf40c5aa0 0x7f2bf406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.982+0000 7f2bfa3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf406a7c0 con 0x7f2bf415d9b0 2023-06-22T08:00:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.982+0000 7f2bfa3ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf40c4d70 con 0x7f2bf4069eb0 2023-06-22T08:00:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.982+0000 7f2bfa3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf40c5610 con 0x7f2bf40c5aa0 2023-06-22T08:00:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.983+0000 7f2bf37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2bf415d9b0 0x7f2bf406a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.983+0000 7f2bf37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2bf415d9b0 0x7f2bf406a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60892/0 (socket says 172.21.15.23:60892) 2023-06-22T08:00:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.983+0000 7f2bf37fe700 1 -- 172.21.15.23:0/1671704135 learned_addr learned my addr 172.21.15.23:0/1671704135 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.984+0000 7f2bf899c700 1 --2- 172.21.15.23:0/1671704135 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bf40c5aa0 0x7f2bf406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.984+0000 7f2bf37fe700 1 -- 172.21.15.23:0/1671704135 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bf40c5aa0 msgr2=0x7f2bf406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.984+0000 7f2bf37fe700 1 --2- 172.21.15.23:0/1671704135 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bf40c5aa0 0x7f2bf406fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.984+0000 7f2bf37fe700 1 -- 172.21.15.23:0/1671704135 >> v2:172.21.15.67:3300/0 conn(0x7f2bf4069eb0 msgr2=0x7f2bf406d050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-22T08:00:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.984+0000 7f2bf37fe700 1 --2- 172.21.15.23:0/1671704135 >> v2:172.21.15.67:3300/0 conn(0x7f2bf4069eb0 0x7f2bf406d050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.984+0000 7f2bf37fe700 1 -- 172.21.15.23:0/1671704135 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf4071340 con 0x7f2bf415d9b0 2023-06-22T08:00:26.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.984+0000 7f2bf899c700 1 --2- 172.21.15.23:0/1671704135 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bf40c5aa0 0x7f2bf406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T08:00:26.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.985+0000 7f2bf37fe700 1 --2- 172.21.15.23:0/1671704135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 0x7f2bf406a280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40c6dd0 tx=0x7f2be4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d53657f670c578f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.983+0000 7eff557fa700 1 -- 172.21.15.23:0/2249495348 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7eff50003ef0 con 0x7eff58160510 2023-06-22T08:00:26.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.985+0000 7eff557fa700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7eff4406ade0 0x7eff4406d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.985+0000 7eff557fa700 1 -- 172.21.15.23:0/2249495348 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7eff50011070 con 0x7eff58160510 2023-06-22T08:00:26.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.985+0000 7f88fe59c700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006adb0 0x7f88e006d260 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f88f8000f80 tx=0x7f88f8001710 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.985+0000 7f88eeffd700 1 -- 172.21.15.23:0/729105698 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f88f40b5310 con 0x7f890015d9b0 2023-06-22T08:00:26.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.986+0000 7eff5cf9b700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7eff4406ade0 0x7eff4406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.987+0000 7eff5cf9b700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7eff4406ade0 0x7eff4406d290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7eff48001900 tx=0x7eff48009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.988+0000 7fde125ff700 1 Processor -- start 2023-06-22T08:00:26.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.988+0000 7fde125ff700 1 -- start start 2023-06-22T08:00:26.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.990+0000 7fde125ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde0c070610 0x7fde0c071150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:26.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.991+0000 7fde125ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c1589b0 con 0x7fde0c0c5ae0 2023-06-22T08:00:26.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.991+0000 7fde125ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c157de0 con 0x7fde0c070d80 2023-06-22T08:00:26.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.991+0000 7fde125ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c071690 con 0x7fde0c070610 2023-06-22T08:00:26.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.991+0000 7fde0b7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde0c070610 0x7fde0c071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:26.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.991+0000 7fde0b7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde0c070610 0x7fde0c071150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:47980/0 (socket says 172.21.15.23:47980) 2023-06-22T08:00:26.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.991+0000 7fde0b7fe700 1 -- 172.21.15.23:0/3583798846 learned_addr learned my addr 172.21.15.23:0/3583798846 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:26.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.993+0000 7fde0b7fe700 1 -- 172.21.15.23:0/3583798846 >> v1:172.21.15.67:6789/0 conn(0x7fde0c070d80 legacy=0x7fde0c0c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.993+0000 7fde0b7fe700 1 -- 172.21.15.23:0/3583798846 >> v1:172.21.15.23:6789/0 conn(0x7fde0c0c5ae0 legacy=0x7fde0c15f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.993+0000 7fde0b7fe700 1 -- 172.21.15.23:0/3583798846 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c158080 con 0x7fde0c070610 2023-06-22T08:00:26.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.993+0000 7fde0affd700 1 -- 172.21.15.23:0/3583798846 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571786749 0 0) 0x7fde0c157de0 con 0x7fde0c070d80 2023-06-22T08:00:26.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.993+0000 7fde0b7fe700 1 --2- 172.21.15.23:0/3583798846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0c070610 0x7fde0c071150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0088f0 tx=0x7fddfc00d330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56a911c010f5c0fd server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:26.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.993+0000 7f2bf2ffd700 1 -- 172.21.15.23:0/1671704135 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be400b870 con 0x7f2bf415d9b0 2023-06-22T08:00:26.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.993+0000 7f2bf2ffd700 1 -- 172.21.15.23:0/1671704135 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be400b9d0 con 0x7f2bf415d9b0 2023-06-22T08:00:26.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.993+0000 7f2bf2ffd700 1 -- 172.21.15.23:0/1671704135 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be400bd60 con 0x7f2bf415d9b0 2023-06-22T08:00:26.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.994+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/1671704135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 msgr2=0x7f2bf406a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.994+0000 7f2bfa3ff700 1 --2- 172.21.15.23:0/1671704135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 0x7f2bf406a280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40c6dd0 tx=0x7f2be4009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:26.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.995+0000 7eff557fa700 1 -- 172.21.15.23:0/2249495348 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7eff50055370 con 0x7eff58160510 2023-06-22T08:00:26.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.994+0000 7fde0affd700 1 -- 172.21.15.23:0/3583798846 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc010070 con 0x7fde0c070610 2023-06-22T08:00:26.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.994+0000 7fde0affd700 1 -- 172.21.15.23:0/3583798846 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc00da80 con 0x7fde0c070610 2023-06-22T08:00:26.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.994+0000 7fde0affd700 1 -- 172.21.15.23:0/3583798846 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc00ddf0 con 0x7fde0c070610 2023-06-22T08:00:26.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.994+0000 7fde125ff700 1 -- 172.21.15.23:0/3583798846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0c070610 msgr2=0x7fde0c071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:26.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.994+0000 7fde125ff700 1 --2- 172.21.15.23:0/3583798846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0c070610 0x7fde0c071150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0088f0 tx=0x7fddfc00d330 comp rx=0 tx=0).stop 2023-06-22T08:00:26.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.995+0000 7fde125ff700 1 -- 172.21.15.23:0/3583798846 shutdown_connections 2023-06-22T08:00:26.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.995+0000 7fde125ff700 1 --2- 172.21.15.23:0/3583798846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0c070610 0x7fde0c071150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.995+0000 7fde125ff700 1 -- 172.21.15.23:0/3583798846 >> 172.21.15.23:0/3583798846 conn(0x7fde0c0ca3f0 msgr2=0x7fde0c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.995+0000 7fde125ff700 1 -- 172.21.15.23:0/3583798846 shutdown_connections 2023-06-22T08:00:27.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.996+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/1671704135 shutdown_connections 2023-06-22T08:00:27.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.996+0000 7f2bfa3ff700 1 --2- 172.21.15.23:0/1671704135 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bf40c5aa0 0x7f2bf406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.996+0000 7f2bfa3ff700 1 --2- 172.21.15.23:0/1671704135 >> v2:172.21.15.67:3300/0 conn(0x7f2bf4069eb0 0x7f2bf406d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.996+0000 7f2bfa3ff700 1 --2- 172.21.15.23:0/1671704135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 0x7f2bf406a280 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.996+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/1671704135 >> 172.21.15.23:0/1671704135 conn(0x7f2bf40ca3f0 msgr2=0x7f2bf40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.996+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/1671704135 shutdown_connections 2023-06-22T08:00:27.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.996+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/1671704135 wait complete. 2023-06-22T08:00:27.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.999+0000 7f2bfa3ff700 1 Processor -- start 2023-06-22T08:00:27.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.999+0000 7f2bfa3ff700 1 -- start start 2023-06-22T08:00:27.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.996+0000 7fde125ff700 1 -- 172.21.15.23:0/3583798846 wait complete. 2023-06-22T08:00:27.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.999+0000 7fde125ff700 1 Processor -- start 2023-06-22T08:00:27.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.999+0000 7fde125ff700 1 -- start start 2023-06-22T08:00:27.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.000+0000 7fde125ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde0c070610 0x7fde0c144b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.000+0000 7fde125ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde0c070d80 0x7fde0c14cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.000+0000 7fde125ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0c0c5ae0 0x7fde0c14d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.000+0000 7fde125ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c1ef440 con 0x7fde0c070d80 2023-06-22T08:00:27.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.000+0000 7fde125ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c1451f0 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.000+0000 7fde125ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c1454d0 con 0x7fde0c070610 2023-06-22T08:00:27.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.999+0000 7f9362bff700 1 Processor -- start 2023-06-22T08:00:27.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:26.999+0000 7f9362bff700 1 -- start start 2023-06-22T08:00:27.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7fde10b9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0c0c5ae0 0x7fde0c14d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7fba32fff700 1 Processor -- start 2023-06-22T08:00:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7fde0b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde0c070d80 0x7fde0c14cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7fde0b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde0c070d80 0x7fde0c14cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60908/0 (socket says 172.21.15.23:60908) 2023-06-22T08:00:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7fde0b7fe700 1 -- 172.21.15.23:0/815911575 learned_addr learned my addr 172.21.15.23:0/815911575 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7f2bfa3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bf4069eb0 0x7f2bf41f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7f2bfa3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bf40c5aa0 0x7f2bf41608c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7f2bfa3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 0x7f2bf41e9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7f2bfa3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf41f3160 con 0x7f2bf415d9b0 2023-06-22T08:00:27.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7f2bfa3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf40c4710 con 0x7f2bf4069eb0 2023-06-22T08:00:27.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7f2bfa3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf41f6610 con 0x7f2bf40c5aa0 2023-06-22T08:00:27.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.002+0000 7f2bf899c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 0x7f2bf41e9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.002+0000 7f2bf899c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 0x7f2bf41e9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60912/0 (socket says 172.21.15.23:60912) 2023-06-22T08:00:27.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.002+0000 7f2bf899c700 1 -- 172.21.15.23:0/3690612003 learned_addr learned my addr 172.21.15.23:0/3690612003 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:27.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.001+0000 7fba32fff700 1 -- start start 2023-06-22T08:00:27.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.003+0000 7f9362bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f935c152ce0 0x7f935c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.003+0000 7f9362bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f935c0c9fd0 0x7f935c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.003+0000 7f9362bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935c15c370 con 0x7f935c158880 2023-06-22T08:00:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.003+0000 7f9362bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c1530b0 con 0x7f935c0c9fd0 2023-06-22T08:00:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.003+0000 7f9362bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f935c0cf6f0 con 0x7f935c152ce0 2023-06-22T08:00:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.003+0000 7fde10b9c700 1 -- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde0c070610 msgr2=0x7fde0c144b10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T08:00:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.003+0000 7fde10b9c700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde0c070610 0x7fde0c144b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.003+0000 7fde10b9c700 1 -- 172.21.15.23:0/815911575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde0c070d80 msgr2=0x7fde0c14cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.003+0000 7fde10b9c700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde0c070d80 0x7fde0c14cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.003+0000 7fde10b9c700 1 -- 172.21.15.23:0/815911575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc0085d0 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f936099b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f935c152ce0 0x7f935c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f936099b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f935c152ce0 0x7f935c0cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:48012/0 (socket says 172.21.15.23:48012) 2023-06-22T08:00:27.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f936099b700 1 -- 172.21.15.23:0/4230238116 learned_addr learned my addr 172.21.15.23:0/4230238116 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:27.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f77389ff700 1 Processor -- start 2023-06-22T08:00:27.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f77389ff700 1 -- start start 2023-06-22T08:00:27.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f2bf899c700 1 -- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bf40c5aa0 msgr2=0x7f2bf41608c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T08:00:27.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f2bf899c700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bf40c5aa0 0x7f2bf41608c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f2bf899c700 1 -- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bf4069eb0 msgr2=0x7f2bf41f60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f2bf899c700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bf4069eb0 0x7f2bf41f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f2bf899c700 1 -- 172.21.15.23:0/3690612003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be4009550 con 0x7f2bf415d9b0 2023-06-22T08:00:27.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f936099b700 1 -- 172.21.15.23:0/4230238116 >> v2:172.21.15.23:3300/0 conn(0x7f935c0c9fd0 msgr2=0x7f935c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:27.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f936099b700 1 --2- 172.21.15.23:0/4230238116 >> v2:172.21.15.23:3300/0 conn(0x7f935c0c9fd0 0x7f935c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f936099b700 1 -- 172.21.15.23:0/4230238116 >> v1:172.21.15.67:6789/0 conn(0x7f935c158880 legacy=0x7f935c158d00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-22T08:00:27.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7f936099b700 1 -- 172.21.15.23:0/4230238116 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c15bda0 con 0x7f935c152ce0 2023-06-22T08:00:27.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7fba32fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071c20 0x7fba2c0c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7fba32fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c15bf50 con 0x7fba2c0c4a30 2023-06-22T08:00:27.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7fba32fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c0c9550 con 0x7fba2c071850 2023-06-22T08:00:27.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.004+0000 7fba32fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c0c4eb0 con 0x7fba2c071c20 2023-06-22T08:00:27.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7f936099b700 1 --2- 172.21.15.23:0/4230238116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f935c152ce0 0x7f935c0cf1b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f935c08ba10 tx=0x7f934c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1610b2d4ec5708db server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7fba2bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071c20 0x7fba2c0c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7fba2bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071c20 0x7fba2c0c9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:37032/0 (socket says 172.21.15.23:37032) 2023-06-22T08:00:27.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7fba2bfff700 1 -- 172.21.15.23:0/2549461100 learned_addr learned my addr 172.21.15.23:0/2549461100 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:27.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7f1470fff700 1 Processor -- start 2023-06-22T08:00:27.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7fba2bfff700 1 -- 172.21.15.23:0/2549461100 >> v1:172.21.15.67:6789/0 conn(0x7fba2c071850 legacy=0x7fba2c152750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-22T08:00:27.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7fba2bfff700 1 -- 172.21.15.23:0/2549461100 >> v1:172.21.15.23:6789/0 conn(0x7fba2c0c4a30 legacy=0x7fba2c15c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:27.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7fba2bfff700 1 -- 172.21.15.23:0/2549461100 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c158c00 con 0x7fba2c071c20 2023-06-22T08:00:27.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7fde10b9c700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0c0c5ae0 0x7fde0c14d110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fde0c0c5f40 tx=0x7fde0400ce70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7fde08ff9700 1 -- 172.21.15.23:0/815911575 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04003c00 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7fde08ff9700 1 -- 172.21.15.23:0/815911575 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04003d60 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7fde08ff9700 1 -- 172.21.15.23:0/815911575 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0400aa80 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7f935b7fe700 1 -- 172.21.15.23:0/4230238116 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c00e070 con 0x7f935c152ce0 2023-06-22T08:00:27.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7f935b7fe700 1 -- 172.21.15.23:0/4230238116 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c003a20 con 0x7f935c152ce0 2023-06-22T08:00:27.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7f935b7fe700 1 -- 172.21.15.23:0/4230238116 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c003dd0 con 0x7f935c152ce0 2023-06-22T08:00:27.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7f9362bff700 1 -- 172.21.15.23:0/4230238116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f935c152ce0 msgr2=0x7f935c0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7f9362bff700 1 --2- 172.21.15.23:0/4230238116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f935c152ce0 0x7f935c0cf1b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f935c08ba10 tx=0x7f934c00b330 comp rx=0 tx=0).stop 2023-06-22T08:00:27.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7f2bf899c700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 0x7f2bf41e9930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2be800c820 tx=0x7f2be800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7fde125ff700 1 -- 172.21.15.23:0/815911575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c1457b0 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7fde125ff700 1 -- 172.21.15.23:0/815911575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c1ec9d0 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7fba2bfff700 1 --2- 172.21.15.23:0/2549461100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071c20 0x7fba2c0c9880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fba1c009920 tx=0x7fba1c009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65fe9e0e467c8ae9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f2bf0ff9700 1 -- 172.21.15.23:0/3690612003 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be8011800 con 0x7f2bf415d9b0 2023-06-22T08:00:27.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f2bf0ff9700 1 -- 172.21.15.23:0/3690612003 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be8011960 con 0x7f2bf415d9b0 2023-06-22T08:00:27.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f2bf0ff9700 1 -- 172.21.15.23:0/3690612003 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be8011cf0 con 0x7f2bf415d9b0 2023-06-22T08:00:27.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/3690612003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf41f2340 con 0x7f2bf415d9b0 2023-06-22T08:00:27.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/3690612003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf41f28c0 con 0x7f2bf415d9b0 2023-06-22T08:00:27.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7fba2b7fe700 1 -- 172.21.15.23:0/2549461100 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c004230 con 0x7fba2c071c20 2023-06-22T08:00:27.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7fba2b7fe700 1 -- 172.21.15.23:0/2549461100 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1c004390 con 0x7fba2c071c20 2023-06-22T08:00:27.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7fba2b7fe700 1 -- 172.21.15.23:0/2549461100 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c009e70 con 0x7fba2c071c20 2023-06-22T08:00:27.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.008+0000 7fba32fff700 1 -- 172.21.15.23:0/2549461100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071c20 msgr2=0x7fba2c0c9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.008+0000 7fba32fff700 1 --2- 172.21.15.23:0/2549461100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071c20 0x7fba2c0c9880 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fba1c009920 tx=0x7fba1c009c30 comp rx=0 tx=0).stop 2023-06-22T08:00:27.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.008+0000 7fde125ff700 1 -- 172.21.15.23:0/815911575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c04ea20 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.009+0000 7fba32fff700 1 -- 172.21.15.23:0/2549461100 shutdown_connections 2023-06-22T08:00:27.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.009+0000 7fba32fff700 1 --2- 172.21.15.23:0/2549461100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071c20 0x7fba2c0c9880 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fba1c009920 tx=0x7fba1c009c30 comp rx=0 tx=0).stop 2023-06-22T08:00:27.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.010+0000 7fba32fff700 1 -- 172.21.15.23:0/2549461100 >> 172.21.15.23:0/2549461100 conn(0x7fba2c06b8b0 msgr2=0x7fba2c06bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.010+0000 7f9362bff700 1 -- 172.21.15.23:0/4230238116 shutdown_connections 2023-06-22T08:00:27.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.010+0000 7f9362bff700 1 --2- 172.21.15.23:0/4230238116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f935c152ce0 0x7f935c0cf1b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.010+0000 7f9362bff700 1 --2- 172.21.15.23:0/4230238116 >> v2:172.21.15.23:3300/0 conn(0x7f935c0c9fd0 0x7f935c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.010+0000 7f9362bff700 1 -- 172.21.15.23:0/4230238116 >> 172.21.15.23:0/4230238116 conn(0x7f935c06ae30 msgr2=0x7f935c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.010+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/3690612003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd40052f0 con 0x7f2bf415d9b0 2023-06-22T08:00:27.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.010+0000 7f9362bff700 1 -- 172.21.15.23:0/4230238116 shutdown_connections 2023-06-22T08:00:27.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.010+0000 7fba32fff700 1 -- 172.21.15.23:0/2549461100 shutdown_connections 2023-06-22T08:00:27.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7f77389ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7734071850 0x7f77340c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7f77389ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7734071c20 0x7f77340c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7f77389ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773415bf50 con 0x7f7734152750 2023-06-22T08:00:27.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7f77389ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77340c9550 con 0x7f7734071850 2023-06-22T08:00:27.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.006+0000 7f77389ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7734158c00 con 0x7f7734071c20 2023-06-22T08:00:27.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f7732d9d700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7734152750 0x7f773415c3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58704/0 (socket says 172.21.15.23:58704) 2023-06-22T08:00:27.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f7732d9d700 1 -- 172.21.15.23:0/1008621101 learned_addr learned my addr 172.21.15.23:0/1008621101 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:27.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f773259c700 1 --2- 172.21.15.23:0/1008621101 >> v2:172.21.15.23:3300/0 conn(0x7f7734071850 0x7f77340c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f7731d9b700 1 --2- 172.21.15.23:0/1008621101 >> v2:172.21.15.67:3300/0 conn(0x7f7734071c20 0x7f77340c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f773259c700 1 -- 172.21.15.23:0/1008621101 >> v2:172.21.15.67:3300/0 conn(0x7f7734071c20 msgr2=0x7f77340c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:27.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f773259c700 1 --2- 172.21.15.23:0/1008621101 >> v2:172.21.15.67:3300/0 conn(0x7f7734071c20 0x7f77340c4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f773259c700 1 -- 172.21.15.23:0/1008621101 >> v1:172.21.15.67:6789/0 conn(0x7f7734152750 legacy=0x7f773415c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f773259c700 1 -- 172.21.15.23:0/1008621101 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7734158ee0 con 0x7f7734071850 2023-06-22T08:00:27.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.008+0000 7f773259c700 1 --2- 172.21.15.23:0/1008621101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7734071850 0x7f77340c9880 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f77340cf8d0 tx=0x7f7724009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2452997b446af71c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.005+0000 7f1470fff700 1 -- start start 2023-06-22T08:00:27.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f1470fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f146c0c5f20 0x7f146c160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f1470fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146c0c3dd0 con 0x7f146c14eaf0 2023-06-22T08:00:27.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f1470fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146c0c4800 con 0x7f146c06e310 2023-06-22T08:00:27.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.007+0000 7f1470fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f146c158d10 con 0x7f146c0c5f20 2023-06-22T08:00:27.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.008+0000 7f146ad9d700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f146c0c5f20 0x7f146c160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.008+0000 7f146ad9d700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f146c0c5f20 0x7f146c160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:48026/0 (socket says 172.21.15.23:48026) 2023-06-22T08:00:27.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.008+0000 7f146ad9d700 1 -- 172.21.15.23:0/1385319756 learned_addr learned my addr 172.21.15.23:0/1385319756 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:27.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.008+0000 7f1463fff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f146c06e310 0x7f146c069eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58714/0 (socket says 172.21.15.23:58714) 2023-06-22T08:00:27.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.009+0000 7f146ad9d700 1 -- 172.21.15.23:0/1385319756 >> v1:172.21.15.67:6789/0 conn(0x7f146c06e310 legacy=0x7f146c069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.009+0000 7f146ad9d700 1 -- 172.21.15.23:0/1385319756 >> v1:172.21.15.23:6789/0 conn(0x7f146c14eaf0 legacy=0x7f146c1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:27.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.009+0000 7f146ad9d700 1 -- 172.21.15.23:0/1385319756 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c0c4520 con 0x7f146c0c5f20 2023-06-22T08:00:27.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.009+0000 7f146ad9d700 1 --2- 172.21.15.23:0/1385319756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c0c5f20 0x7f146c160160 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1464009e50 tx=0x7f146400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a2f381b9b5ee57 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.010+0000 7f9362bff700 1 -- 172.21.15.23:0/4230238116 wait complete. 2023-06-22T08:00:27.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.013+0000 7f9362bff700 1 Processor -- start 2023-06-22T08:00:27.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.013+0000 7f2bf0ff9700 1 -- 172.21.15.23:0/3690612003 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2be8024480 con 0x7f2bf415d9b0 2023-06-22T08:00:27.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.013+0000 7f1469d9b700 1 -- 172.21.15.23:0/1385319756 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400e070 con 0x7f146c0c5f20 2023-06-22T08:00:27.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.014+0000 7f1469d9b700 1 -- 172.21.15.23:0/1385319756 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1464003a20 con 0x7f146c0c5f20 2023-06-22T08:00:27.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.014+0000 7f1469d9b700 1 -- 172.21.15.23:0/1385319756 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464003db0 con 0x7f146c0c5f20 2023-06-22T08:00:27.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.014+0000 7f773159a700 1 -- 172.21.15.23:0/1008621101 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724011830 con 0x7f7734071850 2023-06-22T08:00:27.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.014+0000 7f773159a700 1 -- 172.21.15.23:0/1008621101 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7724011990 con 0x7f7734071850 2023-06-22T08:00:27.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.014+0000 7f773159a700 1 -- 172.21.15.23:0/1008621101 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724011d20 con 0x7f7734071850 2023-06-22T08:00:27.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.012+0000 7fde08ff9700 1 -- 172.21.15.23:0/815911575 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fde0400abe0 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.014+0000 7fde08ff9700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fddf006ad80 0x7fddf006d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.014+0000 7fde08ff9700 1 -- 172.21.15.23:0/815911575 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fde04089480 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7fde0bfff700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fddf006ad80 0x7fddf006d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f77389ff700 1 -- 172.21.15.23:0/1008621101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7734071850 msgr2=0x7f77340c9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f77389ff700 1 --2- 172.21.15.23:0/1008621101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7734071850 0x7f77340c9880 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f77340cf8d0 tx=0x7f7724009b20 comp rx=0 tx=0).stop 2023-06-22T08:00:27.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.014+0000 7f9362bff700 1 -- start start 2023-06-22T08:00:27.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f9362bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f935c0c9fd0 0x7f935c0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f9362bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f935c152ce0 0x7f935c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f9362bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f935c158880 0x7f935c0ce750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f9362bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0cced0 con 0x7f935c0c9fd0 2023-06-22T08:00:27.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f9362bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c0cd080 con 0x7f935c152ce0 2023-06-22T08:00:27.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f9362bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c157790 con 0x7f935c158880 2023-06-22T08:00:27.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f2bf0ff9700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2be006adb0 0x7f2be006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f2bf0ff9700 1 -- 172.21.15.23:0/3690612003 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f2be8089d80 con 0x7f2bf415d9b0 2023-06-22T08:00:27.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.015+0000 7f2bf3fff700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2be006adb0 0x7f2be006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.010+0000 7fba32fff700 1 -- 172.21.15.23:0/2549461100 wait complete. 2023-06-22T08:00:27.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.013+0000 7fba32fff700 1 Processor -- start 2023-06-22T08:00:27.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.016+0000 7f936119c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f935c158880 0x7f935c0ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.016+0000 7f936119c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f935c158880 0x7f935c0ce750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:37046/0 (socket says 172.21.15.23:37046) 2023-06-22T08:00:27.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.016+0000 7f936119c700 1 -- 172.21.15.23:0/575118701 learned_addr learned my addr 172.21.15.23:0/575118701 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:27.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.016+0000 7f936099b700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f935c0c9fd0 0x7f935c0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.016+0000 7f935bfff700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f935c152ce0 0x7f935c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.016+0000 7f1470fff700 1 -- 172.21.15.23:0/1385319756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c0c5f20 msgr2=0x7f146c160160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.016+0000 7f1470fff700 1 --2- 172.21.15.23:0/1385319756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c0c5f20 0x7f146c160160 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1464009e50 tx=0x7f146400b330 comp rx=0 tx=0).stop 2023-06-22T08:00:27.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f936119c700 1 -- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f935c152ce0 msgr2=0x7f935c15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f936119c700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f935c152ce0 0x7f935c15b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f936119c700 1 -- 172.21.15.23:0/575118701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f935c0c9fd0 msgr2=0x7f935c0cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f936119c700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f935c0c9fd0 0x7f935c0cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.016+0000 7fba32fff700 1 -- start start 2023-06-22T08:00:27.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f936119c700 1 -- 172.21.15.23:0/575118701 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c009b30 con 0x7f935c158880 2023-06-22T08:00:27.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f1470fff700 1 -- 172.21.15.23:0/1385319756 shutdown_connections 2023-06-22T08:00:27.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f1470fff700 1 --2- 172.21.15.23:0/1385319756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c0c5f20 0x7f146c160160 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f1470fff700 1 -- 172.21.15.23:0/1385319756 >> 172.21.15.23:0/1385319756 conn(0x7f146c0ca3f0 msgr2=0x7f146c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7fde0bfff700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fddf006ad80 0x7fddf006d230 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde00000f60 tx=0x7fde000078d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f936119c700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f935c158880 0x7f935c0ce750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f935400b3e0 tx=0x7f935400b6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7fde08ff9700 1 -- 172.21.15.23:0/815911575 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fde04054190 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f1470fff700 1 -- 172.21.15.23:0/1385319756 shutdown_connections 2023-06-22T08:00:27.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.017+0000 7f2bf3fff700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2be006adb0 0x7f2be006d260 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00abb0 tx=0x7f2bec009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.018+0000 7fba32fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071850 0x7fba2c156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.018+0000 7fba32fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba2c0c4a30 0x7fba2c154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.018+0000 7fba32fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c154770 0x7fba2c152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.018+0000 7fba32fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c158130 con 0x7fba2c0c4a30 2023-06-22T08:00:27.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.018+0000 7fba32fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c1586f0 con 0x7fba2c154770 2023-06-22T08:00:27.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.018+0000 7fba32fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c14dd40 con 0x7fba2c071850 2023-06-22T08:00:27.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.019+0000 7fba3159c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c154770 0x7fba2c152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.019+0000 7fba3159c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c154770 0x7fba2c152620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:48046/0 (socket says 172.21.15.23:48046) 2023-06-22T08:00:27.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.019+0000 7fba3159c700 1 -- 172.21.15.23:0/3435902057 learned_addr learned my addr 172.21.15.23:0/3435902057 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:27.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.019+0000 7fba30d9b700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071850 0x7fba2c156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.019+0000 7fba2bfff700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba2c0c4a30 0x7fba2c154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.019+0000 7f2bf0ff9700 1 -- 172.21.15.23:0/3690612003 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2be8054a90 con 0x7f2bf415d9b0 2023-06-22T08:00:27.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.020+0000 7fba2bfff700 1 -- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071850 msgr2=0x7fba2c156fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.020+0000 7fba2bfff700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071850 0x7fba2c156fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.020+0000 7fba2bfff700 1 -- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c154770 msgr2=0x7fba2c152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.020+0000 7fba2bfff700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c154770 0x7fba2c152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.020+0000 7fba2bfff700 1 -- 172.21.15.23:0/3435902057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c009550 con 0x7fba2c0c4a30 2023-06-22T08:00:27.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.020+0000 7f93597fa700 1 -- 172.21.15.23:0/575118701 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354011620 con 0x7f935c158880 2023-06-22T08:00:27.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.020+0000 7f93597fa700 1 -- 172.21.15.23:0/575118701 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93540117a0 con 0x7f935c158880 2023-06-22T08:00:27.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.020+0000 7f93597fa700 1 -- 172.21.15.23:0/575118701 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354011b50 con 0x7f935c158880 2023-06-22T08:00:27.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.018+0000 7f1470fff700 1 -- 172.21.15.23:0/1385319756 wait complete. 2023-06-22T08:00:27.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.020+0000 7f1470fff700 1 Processor -- start 2023-06-22T08:00:27.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.021+0000 7f77389ff700 1 -- 172.21.15.23:0/1008621101 shutdown_connections 2023-06-22T08:00:27.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.021+0000 7f77389ff700 1 --2- 172.21.15.23:0/1008621101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7734071850 0x7f77340c9880 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.021+0000 7f77389ff700 1 --2- 172.21.15.23:0/1008621101 >> v2:172.21.15.67:3300/0 conn(0x7f7734071c20 0x7f77340c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.021+0000 7f77389ff700 1 -- 172.21.15.23:0/1008621101 >> 172.21.15.23:0/1008621101 conn(0x7f773406b8b0 msgr2=0x7f773406bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.021+0000 7f9362bff700 1 -- 172.21.15.23:0/575118701 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c157a70 con 0x7f935c158880 2023-06-22T08:00:27.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.021+0000 7f9362bff700 1 -- 172.21.15.23:0/575118701 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c158030 con 0x7f935c158880 2023-06-22T08:00:27.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.020+0000 7fba2bfff700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba2c0c4a30 0x7fba2c154230 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fba1c015000 tx=0x7fba1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.021+0000 7fba297fa700 1 -- 172.21.15.23:0/3435902057 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c00bd70 con 0x7fba2c0c4a30 2023-06-22T08:00:27.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.021+0000 7fba297fa700 1 -- 172.21.15.23:0/3435902057 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1c0036a0 con 0x7fba2c0c4a30 2023-06-22T08:00:27.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.021+0000 7fba297fa700 1 -- 172.21.15.23:0/3435902057 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c003a50 con 0x7fba2c0c4a30 2023-06-22T08:00:27.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.022+0000 7fba32fff700 1 -- 172.21.15.23:0/3435902057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2c154f40 con 0x7fba2c0c4a30 2023-06-22T08:00:27.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.022+0000 7fba32fff700 1 -- 172.21.15.23:0/3435902057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2c158380 con 0x7fba2c0c4a30 2023-06-22T08:00:27.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.021+0000 7f1470fff700 1 -- start start 2023-06-22T08:00:27.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.023+0000 7f1470fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c06e310 0x7f146c1f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.023+0000 7f1470fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f146c0c5f20 0x7f146c0c0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.023+0000 7f1470fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f146c14eaf0 0x7f146c156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.023+0000 7f1470fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c1f52d0 con 0x7f146c0c5f20 2023-06-22T08:00:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.023+0000 7f1470fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c1f6130 con 0x7f146c06e310 2023-06-22T08:00:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.023+0000 7f1470fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146c1f5520 con 0x7f146c14eaf0 2023-06-22T08:00:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.024+0000 7f146ad9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f146c14eaf0 0x7f146c156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.024+0000 7f146ad9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f146c14eaf0 0x7f146c156e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:37066/0 (socket says 172.21.15.23:37066) 2023-06-22T08:00:27.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.024+0000 7f146ad9d700 1 -- 172.21.15.23:0/89964291 learned_addr learned my addr 172.21.15.23:0/89964291 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:27.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.025+0000 7f93597fa700 1 -- 172.21.15.23:0/575118701 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9354011cb0 con 0x7f935c158880 2023-06-22T08:00:27.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.025+0000 7fba32fff700 1 -- 172.21.15.23:0/3435902057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2c04ea20 con 0x7fba2c0c4a30 2023-06-22T08:00:27.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.025+0000 7f146a59c700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c06e310 0x7f146c1f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.025+0000 7f146ad9d700 1 -- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c06e310 msgr2=0x7f146c1f2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.025+0000 7f146ad9d700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c06e310 0x7f146c1f2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.025+0000 7f146ad9d700 1 -- 172.21.15.23:0/89964291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f146c0c5f20 msgr2=0x7f146c0c0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.025+0000 7f146ad9d700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f146c0c5f20 0x7f146c0c0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.025+0000 7f146ad9d700 1 -- 172.21.15.23:0/89964291 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1464009b30 con 0x7f146c14eaf0 2023-06-22T08:00:27.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.025+0000 7f146ad9d700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f146c14eaf0 0x7f146c156e20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f146400b730 tx=0x7f1464004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.026+0000 7f77389ff700 1 -- 172.21.15.23:0/1008621101 shutdown_connections 2023-06-22T08:00:27.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.026+0000 7f1462ffd700 1 -- 172.21.15.23:0/89964291 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464004440 con 0x7f146c14eaf0 2023-06-22T08:00:27.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.026+0000 7f1462ffd700 1 -- 172.21.15.23:0/89964291 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14640045c0 con 0x7f146c14eaf0 2023-06-22T08:00:27.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.026+0000 7f1462ffd700 1 -- 172.21.15.23:0/89964291 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464004970 con 0x7f146c14eaf0 2023-06-22T08:00:27.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.026+0000 7f1470fff700 1 -- 172.21.15.23:0/89964291 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146c1f5800 con 0x7f146c14eaf0 2023-06-22T08:00:27.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.026+0000 7f1470fff700 1 -- 172.21.15.23:0/89964291 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146c1f5dc0 con 0x7f146c14eaf0 2023-06-22T08:00:27.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.027+0000 7fba297fa700 1 -- 172.21.15.23:0/3435902057 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fba1c022070 con 0x7fba2c0c4a30 2023-06-22T08:00:27.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.027+0000 7f93597fa700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f934806ae90 0x7f934806d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.027+0000 7f93597fa700 1 -- 172.21.15.23:0/575118701 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f93540895e0 con 0x7f935c158880 2023-06-22T08:00:27.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.028+0000 7fba297fa700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba1806adb0 0x7fba1806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.029+0000 7fba297fa700 1 -- 172.21.15.23:0/3435902057 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fba1c08b050 con 0x7fba2c0c4a30 2023-06-22T08:00:27.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.026+0000 7f77389ff700 1 -- 172.21.15.23:0/1008621101 wait complete. 2023-06-22T08:00:27.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.029+0000 7f77389ff700 1 Processor -- start 2023-06-22T08:00:27.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.030+0000 7fba30d9b700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba1806adb0 0x7fba1806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.030+0000 7f9362bff700 1 -- 172.21.15.23:0/575118701 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c15ccf0 con 0x7f935c158880 2023-06-22T08:00:27.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.031+0000 7fba30d9b700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba1806adb0 0x7fba1806d260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fba20006fd0 tx=0x7fba20009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.031+0000 7f936099b700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f934806ae90 0x7f934806d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.031+0000 7f1462ffd700 1 -- 172.21.15.23:0/89964291 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1464019030 con 0x7f146c14eaf0 2023-06-22T08:00:27.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.032+0000 7f936099b700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f934806ae90 0x7f934806d340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f934c000f80 tx=0x7f934c009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.032+0000 7f1470fff700 1 -- 172.21.15.23:0/89964291 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14500052f0 con 0x7f146c14eaf0 2023-06-22T08:00:27.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.033+0000 7f1462ffd700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f145806af20 0x7f145806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.033+0000 7f1462ffd700 1 -- 172.21.15.23:0/89964291 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f146408ad10 con 0x7f146c14eaf0 2023-06-22T08:00:27.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.034+0000 7f146a59c700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f145806af20 0x7f145806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.034+0000 7f146a59c700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f145806af20 0x7f145806d3d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f145c004f30 tx=0x7f145c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.033+0000 7f77389ff700 1 -- start start 2023-06-22T08:00:27.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.035+0000 7f77389ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7734071850 0x7f773414b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.035+0000 7f77389ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7734071c20 0x7f77341501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.035+0000 7f77389ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7734152750 0x7f7734149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.035+0000 7f77389ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773414bd50 con 0x7f7734071c20 2023-06-22T08:00:27.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.035+0000 7f77389ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77340b43c0 con 0x7f7734071850 2023-06-22T08:00:27.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.035+0000 7f77389ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773415b740 con 0x7f7734152750 2023-06-22T08:00:27.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.035+0000 7f7732d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7734152750 0x7f7734149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.035+0000 7f7732d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7734152750 0x7f7734149eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:37068/0 (socket says 172.21.15.23:37068) 2023-06-22T08:00:27.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.035+0000 7f7732d9d700 1 -- 172.21.15.23:0/3823563226 learned_addr learned my addr 172.21.15.23:0/3823563226 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:27.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.036+0000 7f93597fa700 1 -- 172.21.15.23:0/575118701 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9354054380 con 0x7f935c158880 2023-06-22T08:00:27.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.036+0000 7f7732d9d700 1 -- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7734071850 msgr2=0x7f773414b810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T08:00:27.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.036+0000 7f7732d9d700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7734071850 0x7f773414b810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.036+0000 7f7732d9d700 1 -- 172.21.15.23:0/3823563226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7734071c20 msgr2=0x7f77341501d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T08:00:27.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.036+0000 7f7732d9d700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7734071c20 0x7f77341501d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.036+0000 7f7732d9d700 1 -- 172.21.15.23:0/3823563226 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77240094a0 con 0x7f7734152750 2023-06-22T08:00:27.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.036+0000 7f7732d9d700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7734152750 0x7f7734149eb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f772c00caa0 tx=0x7f772c00ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.039+0000 7fba297fa700 1 -- 172.21.15.23:0/3435902057 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fba1c055f40 con 0x7fba2c0c4a30 2023-06-22T08:00:27.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.041+0000 7f1462ffd700 1 -- 172.21.15.23:0/89964291 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1464055c60 con 0x7f146c14eaf0 2023-06-22T08:00:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.041+0000 7f7722ffd700 1 -- 172.21.15.23:0/3823563226 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c00aa80 con 0x7f7734152750 2023-06-22T08:00:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.041+0000 7f7722ffd700 1 -- 172.21.15.23:0/3823563226 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c00abe0 con 0x7f7734152750 2023-06-22T08:00:27.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.041+0000 7f7722ffd700 1 -- 172.21.15.23:0/3823563226 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c012400 con 0x7f7734152750 2023-06-22T08:00:27.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.041+0000 7f77389ff700 1 -- 172.21.15.23:0/3823563226 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773414b380 con 0x7f7734152750 2023-06-22T08:00:27.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.041+0000 7f77389ff700 1 -- 172.21.15.23:0/3823563226 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773414e990 con 0x7f7734152750 2023-06-22T08:00:27.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.045+0000 7f7722ffd700 1 -- 172.21.15.23:0/3823563226 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f772c01b3b0 con 0x7f7734152750 2023-06-22T08:00:27.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.046+0000 7f7722ffd700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f771406adb0 0x7f771406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:27.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.046+0000 7f7722ffd700 1 -- 172.21.15.23:0/3823563226 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f772c089270 con 0x7f7734152750 2023-06-22T08:00:27.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.048+0000 7f77389ff700 1 -- 172.21.15.23:0/3823563226 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77180052f0 con 0x7f7734152750 2023-06-22T08:00:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.050+0000 7f773259c700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f771406adb0 0x7f771406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.051+0000 7f773259c700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f771406adb0 0x7f771406d260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7724014040 tx=0x7f7724008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:27.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.055+0000 7f7722ffd700 1 -- 172.21.15.23:0/3823563226 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f772c054150 con 0x7f7734152750 2023-06-22T08:00:27.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.442+0000 7f8904bff700 1 -- 172.21.15.23:0/729105698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f88e4005160 con 0x7f890015d9b0 2023-06-22T08:00:27.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.443+0000 7f88eeffd700 1 -- 172.21.15.23:0/729105698 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f88f4017090 con 0x7f890015d9b0 2023-06-22T08:00:27.450 INFO:teuthology.orchestra.run.smithi023.stdout:55834574858 2023-06-22T08:00:27.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.450+0000 7f8904bff700 1 -- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006adb0 msgr2=0x7f88e006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.450+0000 7f8904bff700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006adb0 0x7f88e006d260 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f88f8000f80 tx=0x7f88f8001710 comp rx=0 tx=0).stop 2023-06-22T08:00:27.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.451+0000 7f8904bff700 1 -- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890015d9b0 msgr2=0x7f8900159130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.451+0000 7f8904bff700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890015d9b0 0x7f8900159130 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f88f400c5b0 tx=0x7f88f400c8c0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.451+0000 7f8904bff700 1 -- 172.21.15.23:0/729105698 shutdown_connections 2023-06-22T08:00:27.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.451+0000 7f8904bff700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006adb0 0x7f88e006d260 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.451+0000 7f8904bff700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f890015d9b0 0x7f8900159130 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.452+0000 7f8904bff700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8900069eb0 0x7f890015a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.452+0000 7f8904bff700 1 --2- 172.21.15.23:0/729105698 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f890006a280 0x7f89000c0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.452+0000 7f8904bff700 1 -- 172.21.15.23:0/729105698 >> 172.21.15.23:0/729105698 conn(0x7f89000ca3f0 msgr2=0x7f8900154580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.452+0000 7f8904bff700 1 -- 172.21.15.23:0/729105698 shutdown_connections 2023-06-22T08:00:27.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.452+0000 7f8904bff700 1 -- 172.21.15.23:0/729105698 wait complete. 2023-06-22T08:00:27.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.467+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2249495348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7eff58151b90 con 0x7eff58160510 2023-06-22T08:00:27.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.468+0000 7eff557fa700 1 -- 172.21.15.23:0/2249495348 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff50058990 con 0x7eff58160510 2023-06-22T08:00:27.474 INFO:teuthology.orchestra.run.smithi023.stdout:60129542156 2023-06-22T08:00:27.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.472+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7eff4406ade0 msgr2=0x7eff4406d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.472+0000 7eff5f1ff700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7eff4406ade0 0x7eff4406d290 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7eff48001900 tx=0x7eff48009040 comp rx=0 tx=0).stop 2023-06-22T08:00:27.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.472+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2249495348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff58160510 msgr2=0x7eff58160980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.472+0000 7eff5f1ff700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff58160510 0x7eff58160980 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7eff5815dd80 tx=0x7eff50004be0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.473+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2249495348 shutdown_connections 2023-06-22T08:00:27.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.473+0000 7eff5f1ff700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7eff58160510 0x7eff58160980 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.473+0000 7eff5f1ff700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff40005320 0x7eff58151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.473+0000 7eff5f1ff700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7eff4406ade0 0x7eff4406d290 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.473+0000 7eff5f1ff700 1 --2- 172.21.15.23:0/2249495348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff581fba90 0x7eff5815c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.473+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2249495348 >> 172.21.15.23:0/2249495348 conn(0x7eff580ca3f0 msgr2=0x7eff58155560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.479+0000 7fde125ff700 1 -- 172.21.15.23:0/815911575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fde0c0c6330 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.480+0000 7fde08ff9700 1 -- 172.21.15.23:0/815911575 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde04017090 con 0x7fde0c0c5ae0 2023-06-22T08:00:27.483 INFO:teuthology.orchestra.run.smithi023.stdout:55834574861 2023-06-22T08:00:27.486 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574858 for osd.1 2023-06-22T08:00:27.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.483+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2249495348 shutdown_connections 2023-06-22T08:00:27.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.484+0000 7eff5f1ff700 1 -- 172.21.15.23:0/2249495348 wait complete. 2023-06-22T08:00:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.484+0000 7fde125ff700 1 -- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fddf006ad80 msgr2=0x7fddf006d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.484+0000 7fde125ff700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fddf006ad80 0x7fddf006d230 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde00000f60 tx=0x7fde000078d0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.484+0000 7fde125ff700 1 -- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0c0c5ae0 msgr2=0x7fde0c14d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.484+0000 7fde125ff700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0c0c5ae0 0x7fde0c14d110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fde0c0c5f40 tx=0x7fde0400ce70 comp rx=0 tx=0).stop 2023-06-22T08:00:27.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.490+0000 7fde125ff700 1 -- 172.21.15.23:0/815911575 shutdown_connections 2023-06-22T08:00:27.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.490+0000 7fde125ff700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fddf006ad80 0x7fddf006d230 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.490+0000 7fde125ff700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0c0c5ae0 0x7fde0c14d110 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.490+0000 7fde125ff700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde0c070610 0x7fde0c144b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.490+0000 7fde125ff700 1 --2- 172.21.15.23:0/815911575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde0c070d80 0x7fde0c14cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.490+0000 7fde125ff700 1 -- 172.21.15.23:0/815911575 >> 172.21.15.23:0/815911575 conn(0x7fde0c0ca3f0 msgr2=0x7fde0c0c4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.490+0000 7fde125ff700 1 -- 172.21.15.23:0/815911575 shutdown_connections 2023-06-22T08:00:27.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.490+0000 7fde125ff700 1 -- 172.21.15.23:0/815911575 wait complete. 2023-06-22T08:00:27.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.490+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/3690612003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2bd4005870 con 0x7f2bf415d9b0 2023-06-22T08:00:27.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.491+0000 7f2bf0ff9700 1 -- 172.21.15.23:0/3690612003 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2be801f090 con 0x7f2bf415d9b0 2023-06-22T08:00:27.496 INFO:teuthology.orchestra.run.smithi023.stdout:60129542156 2023-06-22T08:00:27.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.495+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2be006adb0 msgr2=0x7f2be006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.495+0000 7f2bfa3ff700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2be006adb0 0x7f2be006d260 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00abb0 tx=0x7f2bec009250 comp rx=0 tx=0).stop 2023-06-22T08:00:27.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.495+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/3690612003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 msgr2=0x7f2bf41e9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.495+0000 7f2bfa3ff700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 0x7f2bf41e9930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2be800c820 tx=0x7f2be800cbe0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.495+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/3690612003 shutdown_connections 2023-06-22T08:00:27.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.495+0000 7f2bfa3ff700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bf415d9b0 0x7f2bf41e9930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.495+0000 7f2bfa3ff700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2be006adb0 0x7f2be006d260 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00abb0 tx=0x7f2bec009250 comp rx=0 tx=0).stop 2023-06-22T08:00:27.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.495+0000 7f2bfa3ff700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bf4069eb0 0x7f2bf41f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.495+0000 7f2bfa3ff700 1 --2- 172.21.15.23:0/3690612003 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2bf40c5aa0 0x7f2bf41608c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.495+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/3690612003 >> 172.21.15.23:0/3690612003 conn(0x7f2bf40ca3f0 msgr2=0x7f2bf41532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.496+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/3690612003 shutdown_connections 2023-06-22T08:00:27.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.496+0000 7f2bfa3ff700 1 -- 172.21.15.23:0/3690612003 wait complete. 2023-06-22T08:00:27.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.500+0000 7f9362bff700 1 -- 172.21.15.23:0/575118701 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f935c04ea20 con 0x7f935c158880 2023-06-22T08:00:27.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.502+0000 7f93597fa700 1 -- 172.21.15.23:0/575118701 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9354015030 con 0x7f935c158880 2023-06-22T08:00:27.508 INFO:teuthology.orchestra.run.smithi023.stdout:68719476749 2023-06-22T08:00:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.506+0000 7f9362bff700 1 -- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f934806ae90 msgr2=0x7f934806d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.506+0000 7f9362bff700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f934806ae90 0x7f934806d340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f934c000f80 tx=0x7f934c009000 comp rx=0 tx=0).stop 2023-06-22T08:00:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.506+0000 7f9362bff700 1 -- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f935c158880 msgr2=0x7f935c0ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.506+0000 7f9362bff700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f935c158880 0x7f935c0ce750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f935400b3e0 tx=0x7f935400b6f0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.506+0000 7f9362bff700 1 -- 172.21.15.23:0/575118701 shutdown_connections 2023-06-22T08:00:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.506+0000 7f9362bff700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f935c158880 0x7f935c0ce750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.506+0000 7f9362bff700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f935c0c9fd0 0x7f935c0cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.506+0000 7f9362bff700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f934806ae90 0x7f934806d340 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f934c000f80 tx=0x7f934c009000 comp rx=0 tx=0).stop 2023-06-22T08:00:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.506+0000 7f9362bff700 1 --2- 172.21.15.23:0/575118701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f935c152ce0 0x7f935c15b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.506+0000 7f9362bff700 1 -- 172.21.15.23:0/575118701 >> 172.21.15.23:0/575118701 conn(0x7f935c06ae30 msgr2=0x7f935c0c3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.508+0000 7f9362bff700 1 -- 172.21.15.23:0/575118701 shutdown_connections 2023-06-22T08:00:27.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.508+0000 7f77389ff700 1 -- 172.21.15.23:0/3823563226 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f77180055a0 con 0x7f7734152750 2023-06-22T08:00:27.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.509+0000 7f7722ffd700 1 -- 172.21.15.23:0/3823563226 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f772c017090 con 0x7f7734152750 2023-06-22T08:00:27.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.509+0000 7f9362bff700 1 -- 172.21.15.23:0/575118701 wait complete. 2023-06-22T08:00:27.514 INFO:teuthology.orchestra.run.smithi023.stdout:55834574861 2023-06-22T08:00:27.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.510+0000 7f1470fff700 1 -- 172.21.15.23:0/89964291 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f14500055a0 con 0x7f146c14eaf0 2023-06-22T08:00:27.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.510+0000 7f1462ffd700 1 -- 172.21.15.23:0/89964291 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1464059280 con 0x7f146c14eaf0 2023-06-22T08:00:27.516 INFO:teuthology.orchestra.run.smithi023.stdout:55834574860 2023-06-22T08:00:27.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.513+0000 7f77389ff700 1 -- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f771406adb0 msgr2=0x7f771406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.513+0000 7f77389ff700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f771406adb0 0x7f771406d260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7724014040 tx=0x7f7724008030 comp rx=0 tx=0).stop 2023-06-22T08:00:27.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.513+0000 7f77389ff700 1 -- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7734152750 msgr2=0x7f7734149eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.513+0000 7f77389ff700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7734152750 0x7f7734149eb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f772c00caa0 tx=0x7f772c00ce60 comp rx=0 tx=0).stop 2023-06-22T08:00:27.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.515+0000 7f77389ff700 1 -- 172.21.15.23:0/3823563226 shutdown_connections 2023-06-22T08:00:27.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.515+0000 7f77389ff700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7734152750 0x7f7734149eb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.515+0000 7f77389ff700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f771406adb0 0x7f771406d260 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.515+0000 7f77389ff700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7734071850 0x7f773414b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.515+0000 7f77389ff700 1 --2- 172.21.15.23:0/3823563226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7734071c20 0x7f77341501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.515+0000 7f77389ff700 1 -- 172.21.15.23:0/3823563226 >> 172.21.15.23:0/3823563226 conn(0x7f773406b8b0 msgr2=0x7f773414edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.515+0000 7f77389ff700 1 -- 172.21.15.23:0/3823563226 shutdown_connections 2023-06-22T08:00:27.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.515+0000 7f77389ff700 1 -- 172.21.15.23:0/3823563226 wait complete. 2023-06-22T08:00:27.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.516+0000 7fba32fff700 1 -- 172.21.15.23:0/3435902057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fba2c04ea20 con 0x7fba2c0c4a30 2023-06-22T08:00:27.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.516+0000 7fba297fa700 1 -- 172.21.15.23:0/3435902057 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fba1c059560 con 0x7fba2c0c4a30 2023-06-22T08:00:27.523 INFO:teuthology.orchestra.run.smithi023.stdout:55834574860 2023-06-22T08:00:27.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.517+0000 7f1470fff700 1 -- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f145806af20 msgr2=0x7f145806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.517+0000 7f1470fff700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f145806af20 0x7f145806d3d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f145c004f30 tx=0x7f145c009040 comp rx=0 tx=0).stop 2023-06-22T08:00:27.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.517+0000 7f1470fff700 1 -- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f146c14eaf0 msgr2=0x7f146c156e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.517+0000 7f1470fff700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f146c14eaf0 0x7f146c156e20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f146400b730 tx=0x7f1464004ad0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.518+0000 7f1470fff700 1 -- 172.21.15.23:0/89964291 shutdown_connections 2023-06-22T08:00:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.518+0000 7f1470fff700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f146c14eaf0 0x7f146c156e20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.518+0000 7f1470fff700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f145806af20 0x7f145806d3d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.518+0000 7f1470fff700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f146c06e310 0x7f146c1f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.518+0000 7f1470fff700 1 --2- 172.21.15.23:0/89964291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f146c0c5f20 0x7f146c0c0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.518+0000 7f1470fff700 1 -- 172.21.15.23:0/89964291 >> 172.21.15.23:0/89964291 conn(0x7f146c0ca3f0 msgr2=0x7f146c06db60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.520+0000 7fba32fff700 1 -- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba1806adb0 msgr2=0x7fba1806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.520+0000 7fba32fff700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba1806adb0 0x7fba1806d260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fba20006fd0 tx=0x7fba20009190 comp rx=0 tx=0).stop 2023-06-22T08:00:27.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.520+0000 7fba32fff700 1 -- 172.21.15.23:0/3435902057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba2c0c4a30 msgr2=0x7fba2c154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:27.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.520+0000 7fba32fff700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba2c0c4a30 0x7fba2c154230 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fba1c015000 tx=0x7fba1c004750 comp rx=0 tx=0).stop 2023-06-22T08:00:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.521+0000 7f1470fff700 1 -- 172.21.15.23:0/89964291 shutdown_connections 2023-06-22T08:00:27.529 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574861 for osd.5 2023-06-22T08:00:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.521+0000 7f1470fff700 1 -- 172.21.15.23:0/89964291 wait complete. 2023-06-22T08:00:27.530 DEBUG:teuthology.parallel:result is None 2023-06-22T08:00:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.521+0000 7fba32fff700 1 -- 172.21.15.23:0/3435902057 shutdown_connections 2023-06-22T08:00:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.521+0000 7fba32fff700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba1806adb0 0x7fba1806d260 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.521+0000 7fba32fff700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c154770 0x7fba2c152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.521+0000 7fba32fff700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fba2c071850 0x7fba2c156fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.521+0000 7fba32fff700 1 --2- 172.21.15.23:0/3435902057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba2c0c4a30 0x7fba2c154230 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.521+0000 7fba32fff700 1 -- 172.21.15.23:0/3435902057 >> 172.21.15.23:0/3435902057 conn(0x7fba2c06b8b0 msgr2=0x7fba2c1558f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.521+0000 7fba32fff700 1 -- 172.21.15.23:0/3435902057 shutdown_connections 2023-06-22T08:00:27.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:27.522+0000 7fba32fff700 1 -- 172.21.15.23:0/3435902057 wait complete. 2023-06-22T08:00:27.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542156 got 60129542156 for osd.0 2023-06-22T08:00:27.535 DEBUG:teuthology.parallel:result is None 2023-06-22T08:00:27.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542156 got 60129542156 for osd.3 2023-06-22T08:00:27.536 DEBUG:teuthology.parallel:result is None 2023-06-22T08:00:27.544 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574861 for osd.6 2023-06-22T08:00:27.545 DEBUG:teuthology.parallel:result is None 2023-06-22T08:00:27.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476749 got 68719476749 for osd.2 2023-06-22T08:00:27.546 DEBUG:teuthology.parallel:result is None 2023-06-22T08:00:27.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.7 2023-06-22T08:00:27.549 DEBUG:teuthology.parallel:result is None 2023-06-22T08:00:27.553 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.4 2023-06-22T08:00:27.553 DEBUG:teuthology.parallel:result is None 2023-06-22T08:00:28.487 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-06-22T08:00:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.696+0000 7f937f5ff700 1 Processor -- start 2023-06-22T08:00:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.696+0000 7f937f5ff700 1 -- start start 2023-06-22T08:00:28.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.697+0000 7f937f5ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f93780c9fd0 0x7f93780cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.697+0000 7f937f5ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937815c370 con 0x7f93780c40b0 2023-06-22T08:00:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.697+0000 7f937f5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937815bda0 con 0x7f9378152ce0 2023-06-22T08:00:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.697+0000 7f937f5ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93781530b0 con 0x7f93780c9fd0 2023-06-22T08:00:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.698+0000 7f937cb9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f93780c9fd0 0x7f93780cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.698+0000 7f937db9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f93780c40b0 0x7f93780c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41420/0 (socket says 172.21.15.23:41420) 2023-06-22T08:00:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.698+0000 7f937db9c700 1 -- 172.21.15.23:0/807641009 learned_addr learned my addr 172.21.15.23:0/807641009 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:28.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.698+0000 7f937cb9a700 1 -- 172.21.15.23:0/807641009 >> v1:172.21.15.67:6789/0 conn(0x7f9378152ce0 legacy=0x7f937815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.698+0000 7f937cb9a700 1 -- 172.21.15.23:0/807641009 >> v1:172.21.15.23:6789/0 conn(0x7f93780c40b0 legacy=0x7f93780c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.698+0000 7f936ffff700 1 -- 172.21.15.23:0/807641009 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504398288 0 0) 0x7f937815c370 con 0x7f93780c40b0 2023-06-22T08:00:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.698+0000 7f937cb9a700 1 -- 172.21.15.23:0/807641009 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93780c9460 con 0x7f93780c9fd0 2023-06-22T08:00:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.698+0000 7f937cb9a700 1 --2- 172.21.15.23:0/807641009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 0x7f93780cf1b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9368009840 tx=0x7f9368009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912d6125afd6a5cc server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.699+0000 7f936ffff700 1 -- 172.21.15.23:0/807641009 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368011830 con 0x7f93780c9fd0 2023-06-22T08:00:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.699+0000 7f936ffff700 1 -- 172.21.15.23:0/807641009 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9368011990 con 0x7f93780c9fd0 2023-06-22T08:00:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.699+0000 7f936ffff700 1 -- 172.21.15.23:0/807641009 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368011d20 con 0x7f93780c9fd0 2023-06-22T08:00:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.699+0000 7f937f5ff700 1 -- 172.21.15.23:0/807641009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 msgr2=0x7f93780cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.699+0000 7f937f5ff700 1 --2- 172.21.15.23:0/807641009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 0x7f93780cf1b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9368009840 tx=0x7f9368009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.699+0000 7f937f5ff700 1 -- 172.21.15.23:0/807641009 shutdown_connections 2023-06-22T08:00:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.699+0000 7f937f5ff700 1 --2- 172.21.15.23:0/807641009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 0x7f93780cf1b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.699+0000 7f937f5ff700 1 -- 172.21.15.23:0/807641009 >> 172.21.15.23:0/807641009 conn(0x7f937806ae30 msgr2=0x7f937806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.699+0000 7f937f5ff700 1 -- 172.21.15.23:0/807641009 shutdown_connections 2023-06-22T08:00:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.700+0000 7f937f5ff700 1 -- 172.21.15.23:0/807641009 wait complete. 2023-06-22T08:00:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.702+0000 7f937f5ff700 1 Processor -- start 2023-06-22T08:00:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.702+0000 7f937f5ff700 1 -- start start 2023-06-22T08:00:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937f5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f93780c40b0 0x7f93781e38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937f5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 0x7f93781512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937f5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9378152ce0 0x7f93780cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937f5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93781ef3d0 con 0x7f93780c9fd0 2023-06-22T08:00:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937f5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93780cdff0 con 0x7f9378152ce0 2023-06-22T08:00:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937f5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93781ee750 con 0x7f93780c40b0 2023-06-22T08:00:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937cb9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 0x7f93781512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937cb9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 0x7f93781512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60972/0 (socket says 172.21.15.23:60972) 2023-06-22T08:00:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937cb9a700 1 -- 172.21.15.23:0/2721535041 learned_addr learned my addr 172.21.15.23:0/2721535041 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937db9c700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9378152ce0 0x7f93780cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.703+0000 7f937d39b700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f93780c40b0 0x7f93781e38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f937cb9a700 1 -- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f93780c40b0 msgr2=0x7f93781e38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f937cb9a700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f93780c40b0 0x7f93781e38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f937cb9a700 1 -- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9378152ce0 msgr2=0x7f93780cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f937cb9a700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9378152ce0 0x7f93780cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f937cb9a700 1 -- 172.21.15.23:0/2721535041 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93680094a0 con 0x7f93780c9fd0 2023-06-22T08:00:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f937d39b700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f93780c40b0 0x7f93781e38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T08:00:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f937cb9a700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 0x7f93781512f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f937808bc30 tx=0x7f9368004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f936dffb700 1 -- 172.21.15.23:0/2721535041 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93680036a0 con 0x7f93780c9fd0 2023-06-22T08:00:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f936dffb700 1 -- 172.21.15.23:0/2721535041 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936801f070 con 0x7f93780c9fd0 2023-06-22T08:00:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f937f5ff700 1 -- 172.21.15.23:0/2721535041 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93781eea30 con 0x7f93780c9fd0 2023-06-22T08:00:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f936dffb700 1 -- 172.21.15.23:0/2721535041 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368003a60 con 0x7f93780c9fd0 2023-06-22T08:00:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.704+0000 7f937f5ff700 1 -- 172.21.15.23:0/2721535041 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93781eeff0 con 0x7f93780c9fd0 2023-06-22T08:00:28.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.705+0000 7f937f5ff700 1 -- 172.21.15.23:0/2721535041 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93781ef180 con 0x7f93780c9fd0 2023-06-22T08:00:28.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.706+0000 7f936dffb700 1 -- 172.21.15.23:0/2721535041 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9368003bc0 con 0x7f93780c9fd0 2023-06-22T08:00:28.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.707+0000 7f936dffb700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f935c06ace0 0x7f935c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:28.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.707+0000 7f936dffb700 1 -- 172.21.15.23:0/2721535041 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9368021070 con 0x7f93780c9fd0 2023-06-22T08:00:28.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.707+0000 7f937d39b700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f935c06ace0 0x7f935c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:28.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.708+0000 7f937d39b700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f935c06ace0 0x7f935c06d190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9374000f60 tx=0x7f9374009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:28.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.711+0000 7f936dffb700 1 -- 172.21.15.23:0/2721535041 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f93680560b0 con 0x7f93780c9fd0 2023-06-22T08:00:28.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.918+0000 7f937f5ff700 1 -- 172.21.15.23:0/2721535041 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9378151c40 con 0x7f93780c9fd0 2023-06-22T08:00:28.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.918+0000 7f936dffb700 1 -- 172.21.15.23:0/2721535041 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f93680596d0 con 0x7f93780c9fd0 2023-06-22T08:00:28.922 INFO:teuthology.orchestra.run.smithi023.stdout:55834574860 2023-06-22T08:00:28.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 -- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f935c06ace0 msgr2=0x7f935c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:28.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f935c06ace0 0x7f935c06d190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9374000f60 tx=0x7f9374009040 comp rx=0 tx=0).stop 2023-06-22T08:00:28.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 -- 172.21.15.23:0/2721535041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 msgr2=0x7f93781512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:28.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 0x7f93781512f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f937808bc30 tx=0x7f9368004750 comp rx=0 tx=0).stop 2023-06-22T08:00:28.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 -- 172.21.15.23:0/2721535041 shutdown_connections 2023-06-22T08:00:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f935c06ace0 0x7f935c06d190 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9378152ce0 0x7f93780cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f93780c40b0 0x7f93781e38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 --2- 172.21.15.23:0/2721535041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93780c9fd0 0x7f93781512f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 -- 172.21.15.23:0/2721535041 >> 172.21.15.23:0/2721535041 conn(0x7f937806ae30 msgr2=0x7f937815b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 -- 172.21.15.23:0/2721535041 shutdown_connections 2023-06-22T08:00:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:28.921+0000 7f937f5ff700 1 -- 172.21.15.23:0/2721535041 wait complete. 2023-06-22T08:00:28.942 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.1 2023-06-22T08:00:28.942 DEBUG:teuthology.parallel:result is None 2023-06-22T08:00:28.942 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-06-22T08:00:28.943 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-22T08:00:29.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.137+0000 7f40ab7ff700 1 Processor -- start 2023-06-22T08:00:29.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.138+0000 7f40ab7ff700 1 -- start start 2023-06-22T08:00:29.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40ab7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f40a4152ce0 0x7f40a40cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40ab7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40a40c40b0 0x7f40a40c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40ab7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a415c370 con 0x7f40a40c9fd0 2023-06-22T08:00:29.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40ab7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a41530b0 con 0x7f40a4152ce0 2023-06-22T08:00:29.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40ab7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a40cf6f0 con 0x7f40a40c40b0 2023-06-22T08:00:29.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40a959b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f40a4152ce0 0x7f40a40cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40a959b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f40a4152ce0 0x7f40a40cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60974/0 (socket says 172.21.15.23:60974) 2023-06-22T08:00:29.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40a959b700 1 -- 172.21.15.23:0/3589742251 learned_addr learned my addr 172.21.15.23:0/3589742251 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:29.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40a9d9c700 1 --2- 172.21.15.23:0/3589742251 >> v2:172.21.15.67:3300/0 conn(0x7f40a40c40b0 0x7f40a40c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40a959b700 1 -- 172.21.15.23:0/3589742251 >> v2:172.21.15.67:3300/0 conn(0x7f40a40c40b0 msgr2=0x7f40a40c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:29.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40a959b700 1 --2- 172.21.15.23:0/3589742251 >> v2:172.21.15.67:3300/0 conn(0x7f40a40c40b0 0x7f40a40c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.139+0000 7f40a959b700 1 -- 172.21.15.23:0/3589742251 >> v1:172.21.15.67:6789/0 conn(0x7f40a40c9fd0 legacy=0x7f40a415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:29.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.140+0000 7f40a959b700 1 -- 172.21.15.23:0/3589742251 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a415bda0 con 0x7f40a4152ce0 2023-06-22T08:00:29.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.140+0000 7f40a959b700 1 --2- 172.21.15.23:0/3589742251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a4152ce0 0x7f40a40cf1b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40a40b4380 tx=0x7f40a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8f64edd81b61e8 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:29.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.140+0000 7f409bfff700 1 -- 172.21.15.23:0/3589742251 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a000b870 con 0x7f40a4152ce0 2023-06-22T08:00:29.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.140+0000 7f409bfff700 1 -- 172.21.15.23:0/3589742251 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a000b9d0 con 0x7f40a4152ce0 2023-06-22T08:00:29.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.140+0000 7f409bfff700 1 -- 172.21.15.23:0/3589742251 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a000bd60 con 0x7f40a4152ce0 2023-06-22T08:00:29.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.141+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3589742251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a4152ce0 msgr2=0x7f40a40cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.141+0000 7f40ab7ff700 1 --2- 172.21.15.23:0/3589742251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a4152ce0 0x7f40a40cf1b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40a40b4380 tx=0x7f40a0009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.141+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3589742251 shutdown_connections 2023-06-22T08:00:29.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.141+0000 7f40ab7ff700 1 --2- 172.21.15.23:0/3589742251 >> v2:172.21.15.67:3300/0 conn(0x7f40a40c40b0 0x7f40a40c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.141+0000 7f40ab7ff700 1 --2- 172.21.15.23:0/3589742251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a4152ce0 0x7f40a40cf1b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.141+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3589742251 >> 172.21.15.23:0/3589742251 conn(0x7f40a406ae30 msgr2=0x7f40a406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:29.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.141+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3589742251 shutdown_connections 2023-06-22T08:00:29.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.141+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3589742251 wait complete. 2023-06-22T08:00:29.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.142+0000 7f40ab7ff700 1 Processor -- start 2023-06-22T08:00:29.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.143+0000 7f40ab7ff700 1 -- start start 2023-06-22T08:00:29.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.143+0000 7f40ab7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a40c40b0 0x7f40a41e1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.143+0000 7f40ab7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40a40c9fd0 0x7f40a41e2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.143+0000 7f40ab7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40a4152ce0 0x7f40a41e2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.143+0000 7f40ab7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a4156040 con 0x7f40a40c40b0 2023-06-22T08:00:29.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.143+0000 7f40ab7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a41ee830 con 0x7f40a4152ce0 2023-06-22T08:00:29.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.143+0000 7f40ab7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a41eed60 con 0x7f40a40c9fd0 2023-06-22T08:00:29.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a959b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a40c40b0 0x7f40a41e1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a959b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a40c40b0 0x7f40a41e1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60980/0 (socket says 172.21.15.23:60980) 2023-06-22T08:00:29.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a959b700 1 -- 172.21.15.23:0/3977827551 learned_addr learned my addr 172.21.15.23:0/3977827551 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:29.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a8d9a700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40a40c9fd0 0x7f40a41e2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a9d9c700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40a4152ce0 0x7f40a41e2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a959b700 1 -- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40a40c9fd0 msgr2=0x7f40a41e2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a959b700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40a40c9fd0 0x7f40a41e2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a959b700 1 -- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40a4152ce0 msgr2=0x7f40a41e2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a959b700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40a4152ce0 0x7f40a41e2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a959b700 1 -- 172.21.15.23:0/3977827551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a0009550 con 0x7f40a40c40b0 2023-06-22T08:00:29.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.144+0000 7f40a959b700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a40c40b0 0x7f40a41e1610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f40a408bc30 tx=0x7f40a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:29.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.145+0000 7f4099ffb700 1 -- 172.21.15.23:0/3977827551 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0004310 con 0x7f40a40c40b0 2023-06-22T08:00:29.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.145+0000 7f4099ffb700 1 -- 172.21.15.23:0/3977827551 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a001e070 con 0x7f40a40c40b0 2023-06-22T08:00:29.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.145+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3977827551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a40cec20 con 0x7f40a40c40b0 2023-06-22T08:00:29.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.145+0000 7f4099ffb700 1 -- 172.21.15.23:0/3977827551 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a00036a0 con 0x7f40a40c40b0 2023-06-22T08:00:29.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.145+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3977827551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a4155cd0 con 0x7f40a40c40b0 2023-06-22T08:00:29.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.146+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3977827551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a404ea20 con 0x7f40a40c40b0 2023-06-22T08:00:29.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.147+0000 7f4099ffb700 1 -- 172.21.15.23:0/3977827551 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f40a0020030 con 0x7f40a40c40b0 2023-06-22T08:00:29.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.148+0000 7f4099ffb700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f408806ace0 0x7f408806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.148+0000 7f4099ffb700 1 -- 172.21.15.23:0/3977827551 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f40a008a6b0 con 0x7f40a40c40b0 2023-06-22T08:00:29.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.149+0000 7f40a8d9a700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f408806ace0 0x7f408806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.149+0000 7f40a8d9a700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f408806ace0 0x7f408806d190 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f40a40c8e50 tx=0x7f4094009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:29.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.153+0000 7f4099ffb700 1 -- 172.21.15.23:0/3977827551 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f40a0055480 con 0x7f40a40c40b0 2023-06-22T08:00:29.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.354+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3977827551 --> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40a40c9740 con 0x7f408806ace0 2023-06-22T08:00:29.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.366+0000 7f4099ffb700 1 -- 172.21.15.23:0/3977827551 <== mgr.4100 v2:172.21.15.67:6832/3180103139 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367440 (secure 0 0 0) 0x7f4094001560 con 0x7f408806ace0 2023-06-22T08:00:29.373 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:29.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f408806ace0 msgr2=0x7f408806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f408806ace0 0x7f408806d190 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f40a40c8e50 tx=0x7f4094009040 comp rx=0 tx=0).stop 2023-06-22T08:00:29.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3977827551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a40c40b0 msgr2=0x7f40a41e1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a40c40b0 0x7f40a41e1610 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f40a408bc30 tx=0x7f40a0004750 comp rx=0 tx=0).stop 2023-06-22T08:00:29.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3977827551 shutdown_connections 2023-06-22T08:00:29.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f408806ace0 0x7f408806d190 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40a4152ce0 0x7f40a41e2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40a40c40b0 0x7f40a41e1610 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 --2- 172.21.15.23:0/3977827551 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f40a40c9fd0 0x7f40a41e2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3977827551 >> 172.21.15.23:0/3977827551 conn(0x7f40a406ae30 msgr2=0x7f40a414fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:29.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3977827551 shutdown_connections 2023-06-22T08:00:29.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.371+0000 7f40ab7ff700 1 -- 172.21.15.23:0/3977827551 wait complete. 2023-06-22T08:00:29.377 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-06-22T08:00:29.392 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2023-06-22T08:00:28.562416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219224,"kb_used_data":3608,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755496,"statfs":{"total":773094113280,"available":772869627904,"internally_reserved":0,"allocated":3694592,"data_stored":2181100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1499,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003741"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709823+0000","last_change":"2023-06-22T07:59:55.165740+0000","last_active":"2023-06-22T07:59:59.709823+0000","last_peered":"2023-06-22T07:59:59.709823+0000","last_clean":"2023-06-22T07:59:59.709823+0000","last_became_active":"2023-06-22T07:59:55.165398+0000","last_became_peered":"2023-06-22T07:59:55.165398+0000","last_unstale":"2023-06-22T07:59:59.709823+0000","last_undegraded":"2023-06-22T07:59:59.709823+0000","last_fullsized":"2023-06-22T07:59:59.709823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:37.596656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709026+0000","last_change":"2023-06-22T07:59:55.174603+0000","last_active":"2023-06-22T07:59:59.709026+0000","last_peered":"2023-06-22T07:59:59.709026+0000","last_clean":"2023-06-22T07:59:59.709026+0000","last_became_active":"2023-06-22T07:59:55.174346+0000","last_became_peered":"2023-06-22T07:59:55.174346+0000","last_unstale":"2023-06-22T07:59:59.709026+0000","last_undegraded":"2023-06-22T07:59:59.709026+0000","last_fullsized":"2023-06-22T07:59:59.709026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157321+0000","last_change":"2023-06-22T07:59:55.167525+0000","last_active":"2023-06-22T08:00:00.157321+0000","last_peered":"2023-06-22T08:00:00.157321+0000","last_clean":"2023-06-22T08:00:00.157321+0000","last_became_active":"2023-06-22T07:59:55.167181+0000","last_became_peered":"2023-06-22T07:59:55.167181+0000","last_unstale":"2023-06-22T08:00:00.157321+0000","last_undegraded":"2023-06-22T08:00:00.157321+0000","last_fullsized":"2023-06-22T08:00:00.157321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157927+0000","last_change":"2023-06-22T07:59:55.160549+0000","last_active":"2023-06-22T08:00:00.157927+0000","last_peered":"2023-06-22T08:00:00.157927+0000","last_clean":"2023-06-22T08:00:00.157927+0000","last_became_active":"2023-06-22T07:59:55.160239+0000","last_became_peered":"2023-06-22T07:59:55.160239+0000","last_unstale":"2023-06-22T08:00:00.157927+0000","last_undegraded":"2023-06-22T08:00:00.157927+0000","last_fullsized":"2023-06-22T08:00:00.157927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754390+0000","last_change":"2023-06-22T07:59:55.173006+0000","last_active":"2023-06-22T08:00:00.754390+0000","last_peered":"2023-06-22T08:00:00.754390+0000","last_clean":"2023-06-22T08:00:00.754390+0000","last_became_active":"2023-06-22T07:59:55.172578+0000","last_became_peered":"2023-06-22T07:59:55.172578+0000","last_unstale":"2023-06-22T08:00:00.754390+0000","last_undegraded":"2023-06-22T08:00:00.754390+0000","last_fullsized":"2023-06-22T08:00:00.754390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:12:44.799856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871378+0000","last_change":"2023-06-22T07:59:55.173858+0000","last_active":"2023-06-22T07:59:59.871378+0000","last_peered":"2023-06-22T07:59:59.871378+0000","last_clean":"2023-06-22T07:59:59.871378+0000","last_became_active":"2023-06-22T07:59:55.173599+0000","last_became_peered":"2023-06-22T07:59:55.173599+0000","last_unstale":"2023-06-22T07:59:59.871378+0000","last_undegraded":"2023-06-22T07:59:59.871378+0000","last_fullsized":"2023-06-22T07:59:59.871378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:37.597195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871174+0000","last_change":"2023-06-22T07:59:55.168354+0000","last_active":"2023-06-22T07:59:59.871174+0000","last_peered":"2023-06-22T07:59:59.871174+0000","last_clean":"2023-06-22T07:59:59.871174+0000","last_became_active":"2023-06-22T07:59:55.168050+0000","last_became_peered":"2023-06-22T07:59:55.168050+0000","last_unstale":"2023-06-22T07:59:59.871174+0000","last_undegraded":"2023-06-22T07:59:59.871174+0000","last_fullsized":"2023-06-22T07:59:59.871174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:08:45.847378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155602+0000","last_change":"2023-06-22T07:59:55.159738+0000","last_active":"2023-06-22T08:00:00.155602+0000","last_peered":"2023-06-22T08:00:00.155602+0000","last_clean":"2023-06-22T08:00:00.155602+0000","last_became_active":"2023-06-22T07:59:55.159501+0000","last_became_peered":"2023-06-22T07:59:55.159501+0000","last_unstale":"2023-06-22T08:00:00.155602+0000","last_undegraded":"2023-06-22T08:00:00.155602+0000","last_fullsized":"2023-06-22T08:00:00.155602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:55.160768+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:55.160508+0000","last_became_peered":"2023-06-22T07:59:55.160508+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:55:48.713145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155821+0000","last_change":"2023-06-22T07:59:55.166546+0000","last_active":"2023-06-22T08:00:00.155821+0000","last_peered":"2023-06-22T08:00:00.155821+0000","last_clean":"2023-06-22T08:00:00.155821+0000","last_became_active":"2023-06-22T07:59:55.166311+0000","last_became_peered":"2023-06-22T07:59:55.166311+0000","last_unstale":"2023-06-22T08:00:00.155821+0000","last_undegraded":"2023-06-22T08:00:00.155821+0000","last_fullsized":"2023-06-22T08:00:00.155821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:35.822548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155943+0000","last_change":"2023-06-22T07:59:55.162295+0000","last_active":"2023-06-22T08:00:00.155943+0000","last_peered":"2023-06-22T08:00:00.155943+0000","last_clean":"2023-06-22T08:00:00.155943+0000","last_became_active":"2023-06-22T07:59:55.162010+0000","last_became_peered":"2023-06-22T07:59:55.162010+0000","last_unstale":"2023-06-22T08:00:00.155943+0000","last_undegraded":"2023-06-22T08:00:00.155943+0000","last_fullsized":"2023-06-22T08:00:00.155943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709563+0000","last_change":"2023-06-22T07:59:55.158990+0000","last_active":"2023-06-22T07:59:59.709563+0000","last_peered":"2023-06-22T07:59:59.709563+0000","last_clean":"2023-06-22T07:59:59.709563+0000","last_became_active":"2023-06-22T07:59:55.158614+0000","last_became_peered":"2023-06-22T07:59:55.158614+0000","last_unstale":"2023-06-22T07:59:59.709563+0000","last_undegraded":"2023-06-22T07:59:59.709563+0000","last_fullsized":"2023-06-22T07:59:59.709563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755166+0000","last_change":"2023-06-22T07:59:55.161453+0000","last_active":"2023-06-22T08:00:00.755166+0000","last_peered":"2023-06-22T08:00:00.755166+0000","last_clean":"2023-06-22T08:00:00.755166+0000","last_became_active":"2023-06-22T07:59:55.161168+0000","last_became_peered":"2023-06-22T07:59:55.161168+0000","last_unstale":"2023-06-22T08:00:00.755166+0000","last_undegraded":"2023-06-22T08:00:00.755166+0000","last_fullsized":"2023-06-22T08:00:00.755166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155022+0000","last_change":"2023-06-22T07:59:55.164270+0000","last_active":"2023-06-22T08:00:00.155022+0000","last_peered":"2023-06-22T08:00:00.155022+0000","last_clean":"2023-06-22T08:00:00.155022+0000","last_became_active":"2023-06-22T07:59:55.163993+0000","last_became_peered":"2023-06-22T07:59:55.163993+0000","last_unstale":"2023-06-22T08:00:00.155022+0000","last_undegraded":"2023-06-22T08:00:00.155022+0000","last_fullsized":"2023-06-22T08:00:00.155022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:52.051830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157430+0000","last_change":"2023-06-22T07:59:55.169039+0000","last_active":"2023-06-22T08:00:00.157430+0000","last_peered":"2023-06-22T08:00:00.157430+0000","last_clean":"2023-06-22T08:00:00.157430+0000","last_became_active":"2023-06-22T07:59:55.168698+0000","last_became_peered":"2023-06-22T07:59:55.168698+0000","last_unstale":"2023-06-22T08:00:00.157430+0000","last_undegraded":"2023-06-22T08:00:00.157430+0000","last_fullsized":"2023-06-22T08:00:00.157430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871035+0000","last_change":"2023-06-22T07:59:55.158014+0000","last_active":"2023-06-22T07:59:59.871035+0000","last_peered":"2023-06-22T07:59:59.871035+0000","last_clean":"2023-06-22T07:59:59.871035+0000","last_became_active":"2023-06-22T07:59:55.157562+0000","last_became_peered":"2023-06-22T07:59:55.157562+0000","last_unstale":"2023-06-22T07:59:59.871035+0000","last_undegraded":"2023-06-22T07:59:59.871035+0000","last_fullsized":"2023-06-22T07:59:59.871035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:08:33.856001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754963+0000","last_change":"2023-06-22T07:59:55.165957+0000","last_active":"2023-06-22T08:00:00.754963+0000","last_peered":"2023-06-22T08:00:00.754963+0000","last_clean":"2023-06-22T08:00:00.754963+0000","last_became_active":"2023-06-22T07:59:55.165643+0000","last_became_peered":"2023-06-22T07:59:55.165643+0000","last_unstale":"2023-06-22T08:00:00.754963+0000","last_undegraded":"2023-06-22T08:00:00.754963+0000","last_fullsized":"2023-06-22T08:00:00.754963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:10.134743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157391+0000","last_change":"2023-06-22T07:59:55.159858+0000","last_active":"2023-06-22T08:00:00.157391+0000","last_peered":"2023-06-22T08:00:00.157391+0000","last_clean":"2023-06-22T08:00:00.157391+0000","last_became_active":"2023-06-22T07:59:55.159444+0000","last_became_peered":"2023-06-22T07:59:55.159444+0000","last_unstale":"2023-06-22T08:00:00.157391+0000","last_undegraded":"2023-06-22T08:00:00.157391+0000","last_fullsized":"2023-06-22T08:00:00.157391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709424+0000","last_change":"2023-06-22T07:59:55.157162+0000","last_active":"2023-06-22T07:59:59.709424+0000","last_peered":"2023-06-22T07:59:59.709424+0000","last_clean":"2023-06-22T07:59:59.709424+0000","last_became_active":"2023-06-22T07:59:55.156882+0000","last_became_peered":"2023-06-22T07:59:55.156882+0000","last_unstale":"2023-06-22T07:59:59.709424+0000","last_undegraded":"2023-06-22T07:59:59.709424+0000","last_fullsized":"2023-06-22T07:59:59.709424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155892+0000","last_change":"2023-06-22T07:59:55.163834+0000","last_active":"2023-06-22T08:00:00.155892+0000","last_peered":"2023-06-22T08:00:00.155892+0000","last_clean":"2023-06-22T08:00:00.155892+0000","last_became_active":"2023-06-22T07:59:55.163580+0000","last_became_peered":"2023-06-22T07:59:55.163580+0000","last_unstale":"2023-06-22T08:00:00.155892+0000","last_undegraded":"2023-06-22T08:00:00.155892+0000","last_fullsized":"2023-06-22T08:00:00.155892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155285+0000","last_change":"2023-06-22T07:59:55.164569+0000","last_active":"2023-06-22T08:00:00.155285+0000","last_peered":"2023-06-22T08:00:00.155285+0000","last_clean":"2023-06-22T08:00:00.155285+0000","last_became_active":"2023-06-22T07:59:55.164286+0000","last_became_peered":"2023-06-22T07:59:55.164286+0000","last_unstale":"2023-06-22T08:00:00.155285+0000","last_undegraded":"2023-06-22T08:00:00.155285+0000","last_fullsized":"2023-06-22T08:00:00.155285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:17.386317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754137+0000","last_change":"2023-06-22T07:59:55.159883+0000","last_active":"2023-06-22T08:00:00.754137+0000","last_peered":"2023-06-22T08:00:00.754137+0000","last_clean":"2023-06-22T08:00:00.754137+0000","last_became_active":"2023-06-22T07:59:55.159588+0000","last_became_peered":"2023-06-22T07:59:55.159588+0000","last_unstale":"2023-06-22T08:00:00.754137+0000","last_undegraded":"2023-06-22T08:00:00.754137+0000","last_fullsized":"2023-06-22T08:00:00.754137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754941+0000","last_change":"2023-06-22T07:59:55.160638+0000","last_active":"2023-06-22T08:00:00.754941+0000","last_peered":"2023-06-22T08:00:00.754941+0000","last_clean":"2023-06-22T08:00:00.754941+0000","last_became_active":"2023-06-22T07:59:55.160315+0000","last_became_peered":"2023-06-22T07:59:55.160315+0000","last_unstale":"2023-06-22T08:00:00.754941+0000","last_undegraded":"2023-06-22T08:00:00.754941+0000","last_fullsized":"2023-06-22T08:00:00.754941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:51:09.921626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155504+0000","last_change":"2023-06-22T07:59:55.165950+0000","last_active":"2023-06-22T08:00:00.155504+0000","last_peered":"2023-06-22T08:00:00.155504+0000","last_clean":"2023-06-22T08:00:00.155504+0000","last_became_active":"2023-06-22T07:59:55.165505+0000","last_became_peered":"2023-06-22T07:59:55.165505+0000","last_unstale":"2023-06-22T08:00:00.155504+0000","last_undegraded":"2023-06-22T08:00:00.155504+0000","last_fullsized":"2023-06-22T08:00:00.155504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:13.512588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755640+0000","last_change":"2023-06-22T07:59:55.157572+0000","last_active":"2023-06-22T08:00:00.755640+0000","last_peered":"2023-06-22T08:00:00.755640+0000","last_clean":"2023-06-22T08:00:00.755640+0000","last_became_active":"2023-06-22T07:59:55.157055+0000","last_became_peered":"2023-06-22T07:59:55.157055+0000","last_unstale":"2023-06-22T08:00:00.755640+0000","last_undegraded":"2023-06-22T08:00:00.755640+0000","last_fullsized":"2023-06-22T08:00:00.755640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:34:35.685117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754095+0000","last_change":"2023-06-22T07:59:55.164696+0000","last_active":"2023-06-22T08:00:00.754095+0000","last_peered":"2023-06-22T08:00:00.754095+0000","last_clean":"2023-06-22T08:00:00.754095+0000","last_became_active":"2023-06-22T07:59:55.164179+0000","last_became_peered":"2023-06-22T07:59:55.164179+0000","last_unstale":"2023-06-22T08:00:00.754095+0000","last_undegraded":"2023-06-22T08:00:00.754095+0000","last_fullsized":"2023-06-22T08:00:00.754095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871504+0000","last_change":"2023-06-22T07:59:55.168838+0000","last_active":"2023-06-22T07:59:59.871504+0000","last_peered":"2023-06-22T07:59:59.871504+0000","last_clean":"2023-06-22T07:59:59.871504+0000","last_became_active":"2023-06-22T07:59:55.168551+0000","last_became_peered":"2023-06-22T07:59:55.168551+0000","last_unstale":"2023-06-22T07:59:59.871504+0000","last_undegraded":"2023-06-22T07:59:59.871504+0000","last_fullsized":"2023-06-22T07:59:59.871504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:02:36.615449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708648+0000","last_change":"2023-06-22T07:59:59.195356+0000","last_active":"2023-06-22T07:59:59.708648+0000","last_peered":"2023-06-22T07:59:59.708648+0000","last_clean":"2023-06-22T07:59:59.708648+0000","last_became_active":"2023-06-22T07:59:59.194814+0000","last_became_peered":"2023-06-22T07:59:59.194814+0000","last_unstale":"2023-06-22T07:59:59.708648+0000","last_undegraded":"2023-06-22T07:59:59.708648+0000","last_fullsized":"2023-06-22T07:59:59.708648+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:12.223267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6,0],"acting":[2,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"27'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155337+0000","last_change":"2023-06-22T07:59:54.651326+0000","last_active":"2023-06-22T08:00:00.155337+0000","last_peered":"2023-06-22T08:00:00.155337+0000","last_clean":"2023-06-22T08:00:00.155337+0000","last_became_active":"2023-06-22T07:59:54.651010+0000","last_became_peered":"2023-06-22T07:59:54.651010+0000","last_unstale":"2023-06-22T08:00:00.155337+0000","last_undegraded":"2023-06-22T08:00:00.155337+0000","last_fullsized":"2023-06-22T08:00:00.155337+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:17.769117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754538+0000","last_change":"2023-06-22T07:59:55.160345+0000","last_active":"2023-06-22T08:00:00.754538+0000","last_peered":"2023-06-22T08:00:00.754538+0000","last_clean":"2023-06-22T08:00:00.754538+0000","last_became_active":"2023-06-22T07:59:55.160071+0000","last_became_peered":"2023-06-22T07:59:55.160071+0000","last_unstale":"2023-06-22T08:00:00.754538+0000","last_undegraded":"2023-06-22T08:00:00.754538+0000","last_fullsized":"2023-06-22T08:00:00.754538+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708662+0000","last_change":"2023-06-22T07:59:59.196477+0000","last_active":"2023-06-22T07:59:59.708662+0000","last_peered":"2023-06-22T07:59:59.708662+0000","last_clean":"2023-06-22T07:59:59.708662+0000","last_became_active":"2023-06-22T07:59:59.196007+0000","last_became_peered":"2023-06-22T07:59:59.196007+0000","last_unstale":"2023-06-22T07:59:59.708662+0000","last_undegraded":"2023-06-22T07:59:59.708662+0000","last_fullsized":"2023-06-22T07:59:59.708662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,5],"acting":[2,7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753791+0000","last_change":"2023-06-22T07:59:54.653192+0000","last_active":"2023-06-22T08:00:00.753791+0000","last_peered":"2023-06-22T08:00:00.753791+0000","last_clean":"2023-06-22T08:00:00.753791+0000","last_became_active":"2023-06-22T07:59:54.652926+0000","last_became_peered":"2023-06-22T07:59:54.652926+0000","last_unstale":"2023-06-22T08:00:00.753791+0000","last_undegraded":"2023-06-22T08:00:00.753791+0000","last_fullsized":"2023-06-22T08:00:00.753791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:12:59.656555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754609+0000","last_change":"2023-06-22T07:59:55.163171+0000","last_active":"2023-06-22T08:00:00.754609+0000","last_peered":"2023-06-22T08:00:00.754609+0000","last_clean":"2023-06-22T08:00:00.754609+0000","last_became_active":"2023-06-22T07:59:55.162869+0000","last_became_peered":"2023-06-22T07:59:55.162869+0000","last_unstale":"2023-06-22T08:00:00.754609+0000","last_undegraded":"2023-06-22T08:00:00.754609+0000","last_fullsized":"2023-06-22T08:00:00.754609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:37:59.063186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754586+0000","last_change":"2023-06-22T07:59:59.200516+0000","last_active":"2023-06-22T08:00:00.754586+0000","last_peered":"2023-06-22T08:00:00.754586+0000","last_clean":"2023-06-22T08:00:00.754586+0000","last_became_active":"2023-06-22T07:59:59.200096+0000","last_became_peered":"2023-06-22T07:59:59.200096+0000","last_unstale":"2023-06-22T08:00:00.754586+0000","last_undegraded":"2023-06-22T08:00:00.754586+0000","last_fullsized":"2023-06-22T08:00:00.754586+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:06.099105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2,7],"acting":[6,3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709623+0000","last_change":"2023-06-22T07:59:54.645950+0000","last_active":"2023-06-22T07:59:59.709623+0000","last_peered":"2023-06-22T07:59:59.709623+0000","last_clean":"2023-06-22T07:59:59.709623+0000","last_became_active":"2023-06-22T07:59:54.645520+0000","last_became_peered":"2023-06-22T07:59:54.645520+0000","last_unstale":"2023-06-22T07:59:59.709623+0000","last_undegraded":"2023-06-22T07:59:59.709623+0000","last_fullsized":"2023-06-22T07:59:59.709623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:34.816556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871992+0000","last_change":"2023-06-22T07:59:55.162767+0000","last_active":"2023-06-22T07:59:59.871992+0000","last_peered":"2023-06-22T07:59:59.871992+0000","last_clean":"2023-06-22T07:59:59.871992+0000","last_became_active":"2023-06-22T07:59:55.162451+0000","last_became_peered":"2023-06-22T07:59:55.162451+0000","last_unstale":"2023-06-22T07:59:59.871992+0000","last_undegraded":"2023-06-22T07:59:59.871992+0000","last_fullsized":"2023-06-22T07:59:59.871992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:38.977413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871071+0000","last_change":"2023-06-22T07:59:59.208215+0000","last_active":"2023-06-22T07:59:59.871071+0000","last_peered":"2023-06-22T07:59:59.871071+0000","last_clean":"2023-06-22T07:59:59.871071+0000","last_became_active":"2023-06-22T07:59:59.207608+0000","last_became_peered":"2023-06-22T07:59:59.207608+0000","last_unstale":"2023-06-22T07:59:59.871071+0000","last_undegraded":"2023-06-22T07:59:59.871071+0000","last_fullsized":"2023-06-22T07:59:59.871071+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:39:48.527076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,5],"acting":[3,2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.696994+0000","last_change":"2023-06-22T07:59:54.652515+0000","last_active":"2023-06-22T08:00:25.696994+0000","last_peered":"2023-06-22T08:00:25.696994+0000","last_clean":"2023-06-22T08:00:25.696994+0000","last_became_active":"2023-06-22T07:59:54.652249+0000","last_became_peered":"2023-06-22T07:59:54.652249+0000","last_unstale":"2023-06-22T08:00:25.696994+0000","last_undegraded":"2023-06-22T08:00:25.696994+0000","last_fullsized":"2023-06-22T08:00:25.696994+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:12.506596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755250+0000","last_change":"2023-06-22T07:59:55.158359+0000","last_active":"2023-06-22T08:00:00.755250+0000","last_peered":"2023-06-22T08:00:00.755250+0000","last_clean":"2023-06-22T08:00:00.755250+0000","last_became_active":"2023-06-22T07:59:55.157918+0000","last_became_peered":"2023-06-22T07:59:55.157918+0000","last_unstale":"2023-06-22T08:00:00.755250+0000","last_undegraded":"2023-06-22T08:00:00.755250+0000","last_fullsized":"2023-06-22T08:00:00.755250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708305+0000","last_change":"2023-06-22T07:59:59.201033+0000","last_active":"2023-06-22T07:59:59.708305+0000","last_peered":"2023-06-22T07:59:59.708305+0000","last_clean":"2023-06-22T07:59:59.708305+0000","last_became_active":"2023-06-22T07:59:59.200500+0000","last_became_peered":"2023-06-22T07:59:59.200500+0000","last_unstale":"2023-06-22T07:59:59.708305+0000","last_undegraded":"2023-06-22T07:59:59.708305+0000","last_fullsized":"2023-06-22T07:59:59.708305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:18.174604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,4],"acting":[2,1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.694471+0000","last_change":"2023-06-22T07:59:54.654416+0000","last_active":"2023-06-22T08:00:25.694471+0000","last_peered":"2023-06-22T08:00:25.694471+0000","last_clean":"2023-06-22T08:00:25.694471+0000","last_became_active":"2023-06-22T07:59:54.654139+0000","last_became_peered":"2023-06-22T07:59:54.654139+0000","last_unstale":"2023-06-22T08:00:25.694471+0000","last_undegraded":"2023-06-22T08:00:25.694471+0000","last_fullsized":"2023-06-22T08:00:25.694471+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755506+0000","last_change":"2023-06-22T07:59:55.159790+0000","last_active":"2023-06-22T08:00:00.755506+0000","last_peered":"2023-06-22T08:00:00.755506+0000","last_clean":"2023-06-22T08:00:00.755506+0000","last_became_active":"2023-06-22T07:59:55.159489+0000","last_became_peered":"2023-06-22T07:59:55.159489+0000","last_unstale":"2023-06-22T08:00:00.755506+0000","last_undegraded":"2023-06-22T08:00:00.755506+0000","last_fullsized":"2023-06-22T08:00:00.755506+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:15:18.373418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753814+0000","last_change":"2023-06-22T07:59:59.197822+0000","last_active":"2023-06-22T08:00:00.753814+0000","last_peered":"2023-06-22T08:00:00.753814+0000","last_clean":"2023-06-22T08:00:00.753814+0000","last_became_active":"2023-06-22T07:59:59.197431+0000","last_became_peered":"2023-06-22T07:59:59.197431+0000","last_unstale":"2023-06-22T08:00:00.753814+0000","last_undegraded":"2023-06-22T08:00:00.753814+0000","last_fullsized":"2023-06-22T08:00:00.753814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:41.139500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7,1],"acting":[4,6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157224+0000","last_change":"2023-06-22T07:59:54.644158+0000","last_active":"2023-06-22T08:00:00.157224+0000","last_peered":"2023-06-22T08:00:00.157224+0000","last_clean":"2023-06-22T08:00:00.157224+0000","last_became_active":"2023-06-22T07:59:54.643808+0000","last_became_peered":"2023-06-22T07:59:54.643808+0000","last_unstale":"2023-06-22T08:00:00.157224+0000","last_undegraded":"2023-06-22T08:00:00.157224+0000","last_fullsized":"2023-06-22T08:00:00.157224+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:45.438691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754507+0000","last_change":"2023-06-22T07:59:55.157538+0000","last_active":"2023-06-22T08:00:00.754507+0000","last_peered":"2023-06-22T08:00:00.754507+0000","last_clean":"2023-06-22T08:00:00.754507+0000","last_became_active":"2023-06-22T07:59:55.156967+0000","last_became_peered":"2023-06-22T07:59:55.156967+0000","last_unstale":"2023-06-22T08:00:00.754507+0000","last_undegraded":"2023-06-22T08:00:00.754507+0000","last_fullsized":"2023-06-22T08:00:00.754507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:38:44.161245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753821+0000","last_change":"2023-06-22T07:59:59.201671+0000","last_active":"2023-06-22T08:00:00.753821+0000","last_peered":"2023-06-22T08:00:00.753821+0000","last_clean":"2023-06-22T08:00:00.753821+0000","last_became_active":"2023-06-22T07:59:59.201201+0000","last_became_peered":"2023-06-22T07:59:59.201201+0000","last_unstale":"2023-06-22T08:00:00.753821+0000","last_undegraded":"2023-06-22T08:00:00.753821+0000","last_fullsized":"2023-06-22T08:00:00.753821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4,3],"acting":[6,1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"28'12","reported_seq":33,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.696162+0000","last_change":"2023-06-22T07:59:54.640470+0000","last_active":"2023-06-22T08:00:25.696162+0000","last_peered":"2023-06-22T08:00:25.696162+0000","last_clean":"2023-06-22T08:00:25.696162+0000","last_became_active":"2023-06-22T07:59:54.640124+0000","last_became_peered":"2023-06-22T07:59:54.640124+0000","last_unstale":"2023-06-22T08:00:25.696162+0000","last_undegraded":"2023-06-22T08:00:25.696162+0000","last_fullsized":"2023-06-22T08:00:25.696162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:06:41.502702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708599+0000","last_change":"2023-06-22T07:59:55.165550+0000","last_active":"2023-06-22T07:59:59.708599+0000","last_peered":"2023-06-22T07:59:59.708599+0000","last_clean":"2023-06-22T07:59:59.708599+0000","last_became_active":"2023-06-22T07:59:55.164998+0000","last_became_peered":"2023-06-22T07:59:55.164998+0000","last_unstale":"2023-06-22T07:59:59.708599+0000","last_undegraded":"2023-06-22T07:59:59.708599+0000","last_fullsized":"2023-06-22T07:59:59.708599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872133+0000","last_change":"2023-06-22T07:59:59.207103+0000","last_active":"2023-06-22T07:59:59.872133+0000","last_peered":"2023-06-22T07:59:59.872133+0000","last_clean":"2023-06-22T07:59:59.872133+0000","last_became_active":"2023-06-22T07:59:59.206531+0000","last_became_peered":"2023-06-22T07:59:59.206531+0000","last_unstale":"2023-06-22T07:59:59.872133+0000","last_undegraded":"2023-06-22T07:59:59.872133+0000","last_fullsized":"2023-06-22T07:59:59.872133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:42:48.571228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0,6],"acting":[3,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.694690+0000","last_change":"2023-06-22T07:59:54.644301+0000","last_active":"2023-06-22T08:00:25.694690+0000","last_peered":"2023-06-22T08:00:25.694690+0000","last_clean":"2023-06-22T08:00:25.694690+0000","last_became_active":"2023-06-22T07:59:54.643939+0000","last_became_peered":"2023-06-22T07:59:54.643939+0000","last_unstale":"2023-06-22T08:00:25.694690+0000","last_undegraded":"2023-06-22T08:00:25.694690+0000","last_fullsized":"2023-06-22T08:00:25.694690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:07:42.118767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753877+0000","last_change":"2023-06-22T07:59:55.163311+0000","last_active":"2023-06-22T08:00:00.753877+0000","last_peered":"2023-06-22T08:00:00.753877+0000","last_clean":"2023-06-22T08:00:00.753877+0000","last_became_active":"2023-06-22T07:59:55.162991+0000","last_became_peered":"2023-06-22T07:59:55.162991+0000","last_unstale":"2023-06-22T08:00:00.753877+0000","last_undegraded":"2023-06-22T08:00:00.753877+0000","last_fullsized":"2023-06-22T08:00:00.753877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754227+0000","last_change":"2023-06-22T07:59:59.201324+0000","last_active":"2023-06-22T08:00:00.754227+0000","last_peered":"2023-06-22T08:00:00.754227+0000","last_clean":"2023-06-22T08:00:00.754227+0000","last_became_active":"2023-06-22T07:59:59.200877+0000","last_became_peered":"2023-06-22T07:59:59.200877+0000","last_unstale":"2023-06-22T08:00:00.754227+0000","last_undegraded":"2023-06-22T08:00:00.754227+0000","last_fullsized":"2023-06-22T08:00:00.754227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:14:17.094426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5,7],"acting":[4,2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.694163+0000","last_change":"2023-06-22T07:59:54.651527+0000","last_active":"2023-06-22T08:00:25.694163+0000","last_peered":"2023-06-22T08:00:25.694163+0000","last_clean":"2023-06-22T08:00:25.694163+0000","last_became_active":"2023-06-22T07:59:54.651196+0000","last_became_peered":"2023-06-22T07:59:54.651196+0000","last_unstale":"2023-06-22T08:00:25.694163+0000","last_undegraded":"2023-06-22T08:00:25.694163+0000","last_fullsized":"2023-06-22T08:00:25.694163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872142+0000","last_change":"2023-06-22T07:59:55.162610+0000","last_active":"2023-06-22T07:59:59.872142+0000","last_peered":"2023-06-22T07:59:59.872142+0000","last_clean":"2023-06-22T07:59:59.872142+0000","last_became_active":"2023-06-22T07:59:55.162334+0000","last_became_peered":"2023-06-22T07:59:55.162334+0000","last_unstale":"2023-06-22T07:59:59.872142+0000","last_undegraded":"2023-06-22T07:59:59.872142+0000","last_fullsized":"2023-06-22T07:59:59.872142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:42.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:59.199402+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:59.198979+0000","last_became_peered":"2023-06-22T07:59:59.198979+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:17:07.122732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1,6],"acting":[4,5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.870997+0000","last_change":"2023-06-22T07:59:54.649236+0000","last_active":"2023-06-22T07:59:59.870997+0000","last_peered":"2023-06-22T07:59:59.870997+0000","last_clean":"2023-06-22T07:59:59.870997+0000","last_became_active":"2023-06-22T07:59:54.648852+0000","last_became_peered":"2023-06-22T07:59:54.648852+0000","last_unstale":"2023-06-22T07:59:59.870997+0000","last_undegraded":"2023-06-22T07:59:59.870997+0000","last_fullsized":"2023-06-22T07:59:59.870997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:16:59.865638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709134+0000","last_change":"2023-06-22T07:59:55.165859+0000","last_active":"2023-06-22T07:59:59.709134+0000","last_peered":"2023-06-22T07:59:59.709134+0000","last_clean":"2023-06-22T07:59:59.709134+0000","last_became_active":"2023-06-22T07:59:55.165542+0000","last_became_peered":"2023-06-22T07:59:55.165542+0000","last_unstale":"2023-06-22T07:59:59.709134+0000","last_undegraded":"2023-06-22T07:59:59.709134+0000","last_fullsized":"2023-06-22T07:59:59.709134+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:32.147969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871894+0000","last_change":"2023-06-22T07:59:59.207794+0000","last_active":"2023-06-22T07:59:59.871894+0000","last_peered":"2023-06-22T07:59:59.871894+0000","last_clean":"2023-06-22T07:59:59.871894+0000","last_became_active":"2023-06-22T07:59:59.207364+0000","last_became_peered":"2023-06-22T07:59:59.207364+0000","last_unstale":"2023-06-22T07:59:59.871894+0000","last_undegraded":"2023-06-22T07:59:59.871894+0000","last_fullsized":"2023-06-22T07:59:59.871894+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:30.718491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4,0],"acting":[3,6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"28'10","reported_seq":31,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.694836+0000","last_change":"2023-06-22T07:59:54.651423+0000","last_active":"2023-06-22T08:00:25.694836+0000","last_peered":"2023-06-22T08:00:25.694836+0000","last_clean":"2023-06-22T08:00:25.694836+0000","last_became_active":"2023-06-22T07:59:54.651167+0000","last_became_peered":"2023-06-22T07:59:54.651167+0000","last_unstale":"2023-06-22T08:00:25.694836+0000","last_undegraded":"2023-06-22T08:00:25.694836+0000","last_fullsized":"2023-06-22T08:00:25.694836+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:31.141977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709649+0000","last_change":"2023-06-22T07:59:55.164478+0000","last_active":"2023-06-22T07:59:59.709649+0000","last_peered":"2023-06-22T07:59:59.709649+0000","last_clean":"2023-06-22T07:59:59.709649+0000","last_became_active":"2023-06-22T07:59:55.163922+0000","last_became_peered":"2023-06-22T07:59:55.163922+0000","last_unstale":"2023-06-22T07:59:59.709649+0000","last_undegraded":"2023-06-22T07:59:59.709649+0000","last_fullsized":"2023-06-22T07:59:59.709649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:43.772399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155085+0000","last_change":"2023-06-22T07:59:59.198496+0000","last_active":"2023-06-22T08:00:00.155085+0000","last_peered":"2023-06-22T08:00:00.155085+0000","last_clean":"2023-06-22T08:00:00.155085+0000","last_became_active":"2023-06-22T07:59:59.198067+0000","last_became_peered":"2023-06-22T07:59:59.198067+0000","last_unstale":"2023-06-22T08:00:00.155085+0000","last_undegraded":"2023-06-22T08:00:00.155085+0000","last_fullsized":"2023-06-22T08:00:00.155085+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:24.865757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,7],"acting":[1,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"27'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157653+0000","last_change":"2023-06-22T07:59:54.651366+0000","last_active":"2023-06-22T08:00:00.157653+0000","last_peered":"2023-06-22T08:00:00.157653+0000","last_clean":"2023-06-22T08:00:00.157653+0000","last_became_active":"2023-06-22T07:59:54.651059+0000","last_became_peered":"2023-06-22T07:59:54.651059+0000","last_unstale":"2023-06-22T08:00:00.157653+0000","last_undegraded":"2023-06-22T08:00:00.157653+0000","last_fullsized":"2023-06-22T08:00:00.157653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:37.183143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871131+0000","last_change":"2023-06-22T07:59:55.162820+0000","last_active":"2023-06-22T07:59:59.871131+0000","last_peered":"2023-06-22T07:59:59.871131+0000","last_clean":"2023-06-22T07:59:59.871131+0000","last_became_active":"2023-06-22T07:59:55.162505+0000","last_became_peered":"2023-06-22T07:59:55.162505+0000","last_unstale":"2023-06-22T07:59:59.871131+0000","last_undegraded":"2023-06-22T07:59:59.871131+0000","last_fullsized":"2023-06-22T07:59:59.871131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:46:38.312778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753744+0000","last_change":"2023-06-22T07:59:59.201059+0000","last_active":"2023-06-22T08:00:00.753744+0000","last_peered":"2023-06-22T08:00:00.753744+0000","last_clean":"2023-06-22T08:00:00.753744+0000","last_became_active":"2023-06-22T07:59:59.200460+0000","last_became_peered":"2023-06-22T07:59:59.200460+0000","last_unstale":"2023-06-22T08:00:00.753744+0000","last_undegraded":"2023-06-22T08:00:00.753744+0000","last_fullsized":"2023-06-22T08:00:00.753744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,0],"acting":[6,4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158090+0000","last_change":"2023-06-22T07:59:54.646017+0000","last_active":"2023-06-22T08:00:00.158090+0000","last_peered":"2023-06-22T08:00:00.158090+0000","last_clean":"2023-06-22T08:00:00.158090+0000","last_became_active":"2023-06-22T07:59:54.645554+0000","last_became_peered":"2023-06-22T07:59:54.645554+0000","last_unstale":"2023-06-22T08:00:00.158090+0000","last_undegraded":"2023-06-22T08:00:00.158090+0000","last_fullsized":"2023-06-22T08:00:00.158090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:01.550269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154806+0000","last_change":"2023-06-22T07:59:55.166283+0000","last_active":"2023-06-22T08:00:00.154806+0000","last_peered":"2023-06-22T08:00:00.154806+0000","last_clean":"2023-06-22T08:00:00.154806+0000","last_became_active":"2023-06-22T07:59:55.165977+0000","last_became_peered":"2023-06-22T07:59:55.165977+0000","last_unstale":"2023-06-22T08:00:00.154806+0000","last_undegraded":"2023-06-22T08:00:00.154806+0000","last_fullsized":"2023-06-22T08:00:00.154806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:21:29.484640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709141+0000","last_change":"2023-06-22T07:59:59.205411+0000","last_active":"2023-06-22T07:59:59.709141+0000","last_peered":"2023-06-22T07:59:59.709141+0000","last_clean":"2023-06-22T07:59:59.709141+0000","last_became_active":"2023-06-22T07:59:59.204980+0000","last_became_peered":"2023-06-22T07:59:59.204980+0000","last_unstale":"2023-06-22T07:59:59.709141+0000","last_undegraded":"2023-06-22T07:59:59.709141+0000","last_fullsized":"2023-06-22T07:59:59.709141+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:22.317952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871622+0000","last_change":"2023-06-22T07:59:54.654491+0000","last_active":"2023-06-22T07:59:59.871622+0000","last_peered":"2023-06-22T07:59:59.871622+0000","last_clean":"2023-06-22T07:59:59.871622+0000","last_became_active":"2023-06-22T07:59:54.654197+0000","last_became_peered":"2023-06-22T07:59:54.654197+0000","last_unstale":"2023-06-22T07:59:59.871622+0000","last_undegraded":"2023-06-22T07:59:59.871622+0000","last_fullsized":"2023-06-22T07:59:59.871622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:42:59.725455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708444+0000","last_change":"2023-06-22T07:59:55.172053+0000","last_active":"2023-06-22T07:59:59.708444+0000","last_peered":"2023-06-22T07:59:59.708444+0000","last_clean":"2023-06-22T07:59:59.708444+0000","last_became_active":"2023-06-22T07:59:55.165501+0000","last_became_peered":"2023-06-22T07:59:55.165501+0000","last_unstale":"2023-06-22T07:59:59.708444+0000","last_undegraded":"2023-06-22T07:59:59.708444+0000","last_fullsized":"2023-06-22T07:59:59.708444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753990+0000","last_change":"2023-06-22T07:59:59.199179+0000","last_active":"2023-06-22T08:00:00.753990+0000","last_peered":"2023-06-22T08:00:00.753990+0000","last_clean":"2023-06-22T08:00:00.753990+0000","last_became_active":"2023-06-22T07:59:59.198740+0000","last_became_peered":"2023-06-22T07:59:59.198740+0000","last_unstale":"2023-06-22T08:00:00.753990+0000","last_undegraded":"2023-06-22T08:00:00.753990+0000","last_fullsized":"2023-06-22T08:00:00.753990+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158086+0000","last_change":"2023-06-22T07:59:54.644520+0000","last_active":"2023-06-22T08:00:00.158086+0000","last_peered":"2023-06-22T08:00:00.158086+0000","last_clean":"2023-06-22T08:00:00.158086+0000","last_became_active":"2023-06-22T07:59:54.644204+0000","last_became_peered":"2023-06-22T07:59:54.644204+0000","last_unstale":"2023-06-22T08:00:00.158086+0000","last_undegraded":"2023-06-22T08:00:00.158086+0000","last_fullsized":"2023-06-22T08:00:00.158086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:07.674431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709657+0000","last_change":"2023-06-22T07:59:55.164111+0000","last_active":"2023-06-22T07:59:59.709657+0000","last_peered":"2023-06-22T07:59:59.709657+0000","last_clean":"2023-06-22T07:59:59.709657+0000","last_became_active":"2023-06-22T07:59:55.163802+0000","last_became_peered":"2023-06-22T07:59:55.163802+0000","last_unstale":"2023-06-22T07:59:59.709657+0000","last_undegraded":"2023-06-22T07:59:59.709657+0000","last_fullsized":"2023-06-22T07:59:59.709657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755252+0000","last_change":"2023-06-22T07:59:59.197721+0000","last_active":"2023-06-22T08:00:00.755252+0000","last_peered":"2023-06-22T08:00:00.755252+0000","last_clean":"2023-06-22T08:00:00.755252+0000","last_became_active":"2023-06-22T07:59:59.197259+0000","last_became_peered":"2023-06-22T07:59:59.197259+0000","last_unstale":"2023-06-22T08:00:00.755252+0000","last_undegraded":"2023-06-22T08:00:00.755252+0000","last_fullsized":"2023-06-22T08:00:00.755252+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5,1],"acting":[7,0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.697877+0000","last_change":"2023-06-22T07:59:54.643124+0000","last_active":"2023-06-22T08:00:25.697877+0000","last_peered":"2023-06-22T08:00:25.697877+0000","last_clean":"2023-06-22T08:00:25.697877+0000","last_became_active":"2023-06-22T07:59:54.642833+0000","last_became_peered":"2023-06-22T07:59:54.642833+0000","last_unstale":"2023-06-22T08:00:25.697877+0000","last_undegraded":"2023-06-22T08:00:25.697877+0000","last_fullsized":"2023-06-22T08:00:25.697877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158064+0000","last_change":"2023-06-22T07:59:55.160551+0000","last_active":"2023-06-22T08:00:00.158064+0000","last_peered":"2023-06-22T08:00:00.158064+0000","last_clean":"2023-06-22T08:00:00.158064+0000","last_became_active":"2023-06-22T07:59:55.160221+0000","last_became_peered":"2023-06-22T07:59:55.160221+0000","last_unstale":"2023-06-22T08:00:00.158064+0000","last_undegraded":"2023-06-22T08:00:00.158064+0000","last_fullsized":"2023-06-22T08:00:00.158064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708974+0000","last_change":"2023-06-22T07:59:59.204643+0000","last_active":"2023-06-22T07:59:59.708974+0000","last_peered":"2023-06-22T07:59:59.708974+0000","last_clean":"2023-06-22T07:59:59.708974+0000","last_became_active":"2023-06-22T07:59:59.204182+0000","last_became_peered":"2023-06-22T07:59:59.204182+0000","last_unstale":"2023-06-22T07:59:59.708974+0000","last_undegraded":"2023-06-22T07:59:59.708974+0000","last_fullsized":"2023-06-22T07:59:59.708974+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4,2],"acting":[5,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871327+0000","last_change":"2023-06-22T07:59:54.654036+0000","last_active":"2023-06-22T07:59:59.871327+0000","last_peered":"2023-06-22T07:59:59.871327+0000","last_clean":"2023-06-22T07:59:59.871327+0000","last_became_active":"2023-06-22T07:59:54.653736+0000","last_became_peered":"2023-06-22T07:59:54.653736+0000","last_unstale":"2023-06-22T07:59:59.871327+0000","last_undegraded":"2023-06-22T07:59:59.871327+0000","last_fullsized":"2023-06-22T07:59:59.871327+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:44:09.331748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871999+0000","last_change":"2023-06-22T07:59:55.163181+0000","last_active":"2023-06-22T07:59:59.871999+0000","last_peered":"2023-06-22T07:59:59.871999+0000","last_clean":"2023-06-22T07:59:59.871999+0000","last_became_active":"2023-06-22T07:59:55.162860+0000","last_became_peered":"2023-06-22T07:59:55.162860+0000","last_unstale":"2023-06-22T07:59:59.871999+0000","last_undegraded":"2023-06-22T07:59:59.871999+0000","last_fullsized":"2023-06-22T07:59:59.871999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:59:25.037938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155510+0000","last_change":"2023-06-22T07:59:59.195710+0000","last_active":"2023-06-22T08:00:00.155510+0000","last_peered":"2023-06-22T08:00:00.155510+0000","last_clean":"2023-06-22T08:00:00.155510+0000","last_became_active":"2023-06-22T07:59:59.195246+0000","last_became_peered":"2023-06-22T07:59:59.195246+0000","last_unstale":"2023-06-22T08:00:00.155510+0000","last_undegraded":"2023-06-22T08:00:00.155510+0000","last_fullsized":"2023-06-22T08:00:00.155510+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:36.457493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,5],"acting":[1,0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.697779+0000","last_change":"2023-06-22T07:59:54.649692+0000","last_active":"2023-06-22T08:00:25.697779+0000","last_peered":"2023-06-22T08:00:25.697779+0000","last_clean":"2023-06-22T08:00:25.697779+0000","last_became_active":"2023-06-22T07:59:54.649416+0000","last_became_peered":"2023-06-22T07:59:54.649416+0000","last_unstale":"2023-06-22T08:00:25.697779+0000","last_undegraded":"2023-06-22T08:00:25.697779+0000","last_fullsized":"2023-06-22T08:00:25.697779+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:54:02.923433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708909+0000","last_change":"2023-06-22T07:59:55.175392+0000","last_active":"2023-06-22T07:59:59.708909+0000","last_peered":"2023-06-22T07:59:59.708909+0000","last_clean":"2023-06-22T07:59:59.708909+0000","last_became_active":"2023-06-22T07:59:55.175112+0000","last_became_peered":"2023-06-22T07:59:55.175112+0000","last_unstale":"2023-06-22T07:59:59.708909+0000","last_undegraded":"2023-06-22T07:59:59.708909+0000","last_fullsized":"2023-06-22T07:59:59.708909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871873+0000","last_change":"2023-06-22T07:59:59.206671+0000","last_active":"2023-06-22T07:59:59.871873+0000","last_peered":"2023-06-22T07:59:59.871873+0000","last_clean":"2023-06-22T07:59:59.871873+0000","last_became_active":"2023-06-22T07:59:59.206179+0000","last_became_peered":"2023-06-22T07:59:59.206179+0000","last_unstale":"2023-06-22T07:59:59.871873+0000","last_undegraded":"2023-06-22T07:59:59.871873+0000","last_fullsized":"2023-06-22T07:59:59.871873+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:00.427518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4,5],"acting":[3,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.698074+0000","last_change":"2023-06-22T07:59:54.640508+0000","last_active":"2023-06-22T08:00:25.698074+0000","last_peered":"2023-06-22T08:00:25.698074+0000","last_clean":"2023-06-22T08:00:25.698074+0000","last_became_active":"2023-06-22T07:59:54.640108+0000","last_became_peered":"2023-06-22T07:59:54.640108+0000","last_unstale":"2023-06-22T08:00:25.698074+0000","last_undegraded":"2023-06-22T08:00:25.698074+0000","last_fullsized":"2023-06-22T08:00:25.698074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:19.193231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754822+0000","last_change":"2023-06-22T07:59:55.164398+0000","last_active":"2023-06-22T08:00:00.754822+0000","last_peered":"2023-06-22T08:00:00.754822+0000","last_clean":"2023-06-22T08:00:00.754822+0000","last_became_active":"2023-06-22T07:59:55.164086+0000","last_became_peered":"2023-06-22T07:59:55.164086+0000","last_unstale":"2023-06-22T08:00:00.754822+0000","last_undegraded":"2023-06-22T08:00:00.754822+0000","last_fullsized":"2023-06-22T08:00:00.754822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:38.189135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753997+0000","last_change":"2023-06-22T07:59:59.207752+0000","last_active":"2023-06-22T08:00:00.753997+0000","last_peered":"2023-06-22T08:00:00.753997+0000","last_clean":"2023-06-22T08:00:00.753997+0000","last_became_active":"2023-06-22T07:59:59.207360+0000","last_became_peered":"2023-06-22T07:59:59.207360+0000","last_unstale":"2023-06-22T08:00:00.753997+0000","last_undegraded":"2023-06-22T08:00:00.753997+0000","last_fullsized":"2023-06-22T08:00:00.753997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:08:25.633009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3,1],"acting":[4,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709114+0000","last_change":"2023-06-22T07:59:49.652772+0000","last_active":"2023-06-22T07:59:59.709114+0000","last_peered":"2023-06-22T07:59:59.709114+0000","last_clean":"2023-06-22T07:59:59.709114+0000","last_became_active":"2023-06-22T07:59:49.652487+0000","last_became_peered":"2023-06-22T07:59:49.652487+0000","last_unstale":"2023-06-22T07:59:59.709114+0000","last_undegraded":"2023-06-22T07:59:59.709114+0000","last_fullsized":"2023-06-22T07:59:59.709114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:11.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708614+0000","last_change":"2023-06-22T07:59:54.644538+0000","last_active":"2023-06-22T07:59:59.708614+0000","last_peered":"2023-06-22T07:59:59.708614+0000","last_clean":"2023-06-22T07:59:59.708614+0000","last_became_active":"2023-06-22T07:59:54.644253+0000","last_became_peered":"2023-06-22T07:59:54.644253+0000","last_unstale":"2023-06-22T07:59:59.708614+0000","last_undegraded":"2023-06-22T07:59:59.708614+0000","last_fullsized":"2023-06-22T07:59:59.708614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:09.128751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754228+0000","last_change":"2023-06-22T07:59:55.163802+0000","last_active":"2023-06-22T08:00:00.754228+0000","last_peered":"2023-06-22T08:00:00.754228+0000","last_clean":"2023-06-22T08:00:00.754228+0000","last_became_active":"2023-06-22T07:59:55.163481+0000","last_became_peered":"2023-06-22T07:59:55.163481+0000","last_unstale":"2023-06-22T08:00:00.754228+0000","last_undegraded":"2023-06-22T08:00:00.754228+0000","last_fullsized":"2023-06-22T08:00:00.754228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:16.916350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755246+0000","last_change":"2023-06-22T07:59:59.207344+0000","last_active":"2023-06-22T08:00:00.755246+0000","last_peered":"2023-06-22T08:00:00.755246+0000","last_clean":"2023-06-22T08:00:00.755246+0000","last_became_active":"2023-06-22T07:59:59.206975+0000","last_became_peered":"2023-06-22T07:59:59.206975+0000","last_unstale":"2023-06-22T08:00:00.755246+0000","last_undegraded":"2023-06-22T08:00:00.755246+0000","last_fullsized":"2023-06-22T08:00:00.755246+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3,1],"acting":[7,4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871513+0000","last_change":"2023-06-22T07:59:49.653053+0000","last_active":"2023-06-22T07:59:59.871513+0000","last_peered":"2023-06-22T07:59:59.871513+0000","last_clean":"2023-06-22T07:59:59.871513+0000","last_became_active":"2023-06-22T07:59:49.652861+0000","last_became_peered":"2023-06-22T07:59:49.652861+0000","last_unstale":"2023-06-22T07:59:59.871513+0000","last_undegraded":"2023-06-22T07:59:59.871513+0000","last_fullsized":"2023-06-22T07:59:59.871513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:28:40.388367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.701026+0000","last_change":"2023-06-22T07:59:54.645403+0000","last_active":"2023-06-22T08:00:25.701026+0000","last_peered":"2023-06-22T08:00:25.701026+0000","last_clean":"2023-06-22T08:00:25.701026+0000","last_became_active":"2023-06-22T07:59:54.644973+0000","last_became_peered":"2023-06-22T07:59:54.644973+0000","last_unstale":"2023-06-22T08:00:25.701026+0000","last_undegraded":"2023-06-22T08:00:25.701026+0000","last_fullsized":"2023-06-22T08:00:25.701026+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:15.910358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754101+0000","last_change":"2023-06-22T07:59:55.161484+0000","last_active":"2023-06-22T08:00:00.754101+0000","last_peered":"2023-06-22T08:00:00.754101+0000","last_clean":"2023-06-22T08:00:00.754101+0000","last_became_active":"2023-06-22T07:59:55.161194+0000","last_became_peered":"2023-06-22T07:59:55.161194+0000","last_unstale":"2023-06-22T08:00:00.754101+0000","last_undegraded":"2023-06-22T08:00:00.754101+0000","last_fullsized":"2023-06-22T08:00:00.754101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754988+0000","last_change":"2023-06-22T07:59:59.198241+0000","last_active":"2023-06-22T08:00:00.754988+0000","last_peered":"2023-06-22T08:00:00.754988+0000","last_clean":"2023-06-22T08:00:00.754988+0000","last_became_active":"2023-06-22T07:59:59.197689+0000","last_became_peered":"2023-06-22T07:59:59.197689+0000","last_unstale":"2023-06-22T08:00:00.754988+0000","last_undegraded":"2023-06-22T08:00:00.754988+0000","last_fullsized":"2023-06-22T08:00:00.754988+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:39.513874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4,1],"acting":[7,0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753886+0000","last_change":"2023-06-22T07:59:49.450044+0000","last_active":"2023-06-22T08:00:00.753886+0000","last_peered":"2023-06-22T08:00:00.753886+0000","last_clean":"2023-06-22T08:00:00.753886+0000","last_became_active":"2023-06-22T07:59:49.449731+0000","last_became_peered":"2023-06-22T07:59:49.449731+0000","last_unstale":"2023-06-22T08:00:00.753886+0000","last_undegraded":"2023-06-22T08:00:00.753886+0000","last_fullsized":"2023-06-22T08:00:00.753886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:40.262337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708682+0000","last_change":"2023-06-22T07:59:54.640462+0000","last_active":"2023-06-22T07:59:59.708682+0000","last_peered":"2023-06-22T07:59:59.708682+0000","last_clean":"2023-06-22T07:59:59.708682+0000","last_became_active":"2023-06-22T07:59:54.640123+0000","last_became_peered":"2023-06-22T07:59:54.640123+0000","last_unstale":"2023-06-22T07:59:59.708682+0000","last_undegraded":"2023-06-22T07:59:59.708682+0000","last_fullsized":"2023-06-22T07:59:59.708682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:20.316921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755350+0000","last_change":"2023-06-22T07:59:55.164651+0000","last_active":"2023-06-22T08:00:00.755350+0000","last_peered":"2023-06-22T08:00:00.755350+0000","last_clean":"2023-06-22T08:00:00.755350+0000","last_became_active":"2023-06-22T07:59:55.164235+0000","last_became_peered":"2023-06-22T07:59:55.164235+0000","last_unstale":"2023-06-22T08:00:00.755350+0000","last_undegraded":"2023-06-22T08:00:00.755350+0000","last_fullsized":"2023-06-22T08:00:00.755350+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:32.924136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157741+0000","last_change":"2023-06-22T07:59:59.204779+0000","last_active":"2023-06-22T08:00:00.157741+0000","last_peered":"2023-06-22T08:00:00.157741+0000","last_clean":"2023-06-22T08:00:00.157741+0000","last_became_active":"2023-06-22T07:59:59.204288+0000","last_became_peered":"2023-06-22T07:59:59.204288+0000","last_unstale":"2023-06-22T08:00:00.157741+0000","last_undegraded":"2023-06-22T08:00:00.157741+0000","last_fullsized":"2023-06-22T08:00:00.157741+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871497+0000","last_change":"2023-06-22T07:59:49.657008+0000","last_active":"2023-06-22T07:59:59.871497+0000","last_peered":"2023-06-22T07:59:59.871497+0000","last_clean":"2023-06-22T07:59:59.871497+0000","last_became_active":"2023-06-22T07:59:49.656752+0000","last_became_peered":"2023-06-22T07:59:49.656752+0000","last_unstale":"2023-06-22T07:59:59.871497+0000","last_undegraded":"2023-06-22T07:59:59.871497+0000","last_fullsized":"2023-06-22T07:59:59.871497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:50:37.639134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":185,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155134+0000","last_change":"2023-06-22T07:59:42.501363+0000","last_active":"2023-06-22T08:00:00.155134+0000","last_peered":"2023-06-22T08:00:00.155134+0000","last_clean":"2023-06-22T08:00:00.155134+0000","last_became_active":"2023-06-22T07:59:42.500320+0000","last_became_peered":"2023-06-22T07:59:42.500320+0000","last_unstale":"2023-06-22T08:00:00.155134+0000","last_undegraded":"2023-06-22T08:00:00.155134+0000","last_fullsized":"2023-06-22T08:00:00.155134+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_clean_scrub_stamp":"2023-06-22T07:59:41.381475+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:08:53.784030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753785+0000","last_change":"2023-06-22T07:59:54.645031+0000","last_active":"2023-06-22T08:00:00.753785+0000","last_peered":"2023-06-22T08:00:00.753785+0000","last_clean":"2023-06-22T08:00:00.753785+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:00.753785+0000","last_undegraded":"2023-06-22T08:00:00.753785+0000","last_fullsized":"2023-06-22T08:00:00.753785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:42.766407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872074+0000","last_change":"2023-06-22T07:59:55.162799+0000","last_active":"2023-06-22T07:59:59.872074+0000","last_peered":"2023-06-22T07:59:59.872074+0000","last_clean":"2023-06-22T07:59:59.872074+0000","last_became_active":"2023-06-22T07:59:55.162501+0000","last_became_peered":"2023-06-22T07:59:55.162501+0000","last_unstale":"2023-06-22T07:59:59.872074+0000","last_undegraded":"2023-06-22T07:59:59.872074+0000","last_fullsized":"2023-06-22T07:59:59.872074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:57:14.305897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871805+0000","last_change":"2023-06-22T07:59:59.205097+0000","last_active":"2023-06-22T07:59:59.871805+0000","last_peered":"2023-06-22T07:59:59.871805+0000","last_clean":"2023-06-22T07:59:59.871805+0000","last_became_active":"2023-06-22T07:59:59.201465+0000","last_became_peered":"2023-06-22T07:59:59.201465+0000","last_unstale":"2023-06-22T07:59:59.871805+0000","last_undegraded":"2023-06-22T07:59:59.871805+0000","last_fullsized":"2023-06-22T07:59:59.871805+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:17:45.352998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,0],"acting":[3,7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871056+0000","last_change":"2023-06-22T07:59:49.442558+0000","last_active":"2023-06-22T07:59:59.871056+0000","last_peered":"2023-06-22T07:59:59.871056+0000","last_clean":"2023-06-22T07:59:59.871056+0000","last_became_active":"2023-06-22T07:59:49.442154+0000","last_became_peered":"2023-06-22T07:59:49.442154+0000","last_unstale":"2023-06-22T07:59:59.871056+0000","last_undegraded":"2023-06-22T07:59:59.871056+0000","last_fullsized":"2023-06-22T07:59:59.871056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:46:16.233204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753373+0000","last_change":"2023-06-22T07:59:54.641746+0000","last_active":"2023-06-22T08:00:00.753373+0000","last_peered":"2023-06-22T08:00:00.753373+0000","last_clean":"2023-06-22T08:00:00.753373+0000","last_became_active":"2023-06-22T07:59:54.641431+0000","last_became_peered":"2023-06-22T07:59:54.641431+0000","last_unstale":"2023-06-22T08:00:00.753373+0000","last_undegraded":"2023-06-22T08:00:00.753373+0000","last_fullsized":"2023-06-22T08:00:00.753373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709468+0000","last_change":"2023-06-22T07:59:55.163214+0000","last_active":"2023-06-22T07:59:59.709468+0000","last_peered":"2023-06-22T07:59:59.709468+0000","last_clean":"2023-06-22T07:59:59.709468+0000","last_became_active":"2023-06-22T07:59:55.162907+0000","last_became_peered":"2023-06-22T07:59:55.162907+0000","last_unstale":"2023-06-22T07:59:59.709468+0000","last_undegraded":"2023-06-22T07:59:59.709468+0000","last_fullsized":"2023-06-22T07:59:59.709468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:46.988703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157450+0000","last_change":"2023-06-22T07:59:59.198945+0000","last_active":"2023-06-22T08:00:00.157450+0000","last_peered":"2023-06-22T08:00:00.157450+0000","last_clean":"2023-06-22T08:00:00.157450+0000","last_became_active":"2023-06-22T07:59:59.198561+0000","last_became_peered":"2023-06-22T07:59:59.198561+0000","last_unstale":"2023-06-22T08:00:00.157450+0000","last_undegraded":"2023-06-22T08:00:00.157450+0000","last_fullsized":"2023-06-22T08:00:00.157450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:06.135651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2,5],"acting":[0,7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871842+0000","last_change":"2023-06-22T07:59:49.442246+0000","last_active":"2023-06-22T07:59:59.871842+0000","last_peered":"2023-06-22T07:59:59.871842+0000","last_clean":"2023-06-22T07:59:59.871842+0000","last_became_active":"2023-06-22T07:59:49.442001+0000","last_became_peered":"2023-06-22T07:59:49.442001+0000","last_unstale":"2023-06-22T07:59:59.871842+0000","last_undegraded":"2023-06-22T07:59:59.871842+0000","last_fullsized":"2023-06-22T07:59:59.871842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:27:24.457978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754254+0000","last_change":"2023-06-22T07:59:54.643272+0000","last_active":"2023-06-22T08:00:00.754254+0000","last_peered":"2023-06-22T08:00:00.754254+0000","last_clean":"2023-06-22T08:00:00.754254+0000","last_became_active":"2023-06-22T07:59:54.642925+0000","last_became_peered":"2023-06-22T07:59:54.642925+0000","last_unstale":"2023-06-22T08:00:00.754254+0000","last_undegraded":"2023-06-22T08:00:00.754254+0000","last_fullsized":"2023-06-22T08:00:00.754254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:13.625768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754231+0000","last_change":"2023-06-22T07:59:55.167307+0000","last_active":"2023-06-22T08:00:00.754231+0000","last_peered":"2023-06-22T08:00:00.754231+0000","last_clean":"2023-06-22T08:00:00.754231+0000","last_became_active":"2023-06-22T07:59:55.167026+0000","last_became_peered":"2023-06-22T07:59:55.167026+0000","last_unstale":"2023-06-22T08:00:00.754231+0000","last_undegraded":"2023-06-22T08:00:00.754231+0000","last_fullsized":"2023-06-22T08:00:00.754231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871379+0000","last_change":"2023-06-22T07:59:59.208263+0000","last_active":"2023-06-22T07:59:59.871379+0000","last_peered":"2023-06-22T07:59:59.871379+0000","last_clean":"2023-06-22T07:59:59.871379+0000","last_became_active":"2023-06-22T07:59:59.207691+0000","last_became_peered":"2023-06-22T07:59:59.207691+0000","last_unstale":"2023-06-22T07:59:59.871379+0000","last_undegraded":"2023-06-22T07:59:59.871379+0000","last_fullsized":"2023-06-22T07:59:59.871379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:37:57.170857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,2],"acting":[3,7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.697986+0000","last_change":"2023-06-22T07:59:54.644570+0000","last_active":"2023-06-22T08:00:25.697986+0000","last_peered":"2023-06-22T08:00:25.697986+0000","last_clean":"2023-06-22T08:00:25.697986+0000","last_became_active":"2023-06-22T07:59:54.644218+0000","last_became_peered":"2023-06-22T07:59:54.644218+0000","last_unstale":"2023-06-22T08:00:25.697986+0000","last_undegraded":"2023-06-22T08:00:25.697986+0000","last_fullsized":"2023-06-22T08:00:25.697986+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:05:00.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754571+0000","last_change":"2023-06-22T07:59:49.655601+0000","last_active":"2023-06-22T08:00:00.754571+0000","last_peered":"2023-06-22T08:00:00.754571+0000","last_clean":"2023-06-22T08:00:00.754571+0000","last_became_active":"2023-06-22T07:59:49.655328+0000","last_became_peered":"2023-06-22T07:59:49.655328+0000","last_unstale":"2023-06-22T08:00:00.754571+0000","last_undegraded":"2023-06-22T08:00:00.754571+0000","last_fullsized":"2023-06-22T08:00:00.754571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:29:09.967964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755018+0000","last_change":"2023-06-22T07:59:55.163241+0000","last_active":"2023-06-22T08:00:00.755018+0000","last_peered":"2023-06-22T08:00:00.755018+0000","last_clean":"2023-06-22T08:00:00.755018+0000","last_became_active":"2023-06-22T07:59:55.162949+0000","last_became_peered":"2023-06-22T07:59:55.162949+0000","last_unstale":"2023-06-22T08:00:00.755018+0000","last_undegraded":"2023-06-22T08:00:00.755018+0000","last_fullsized":"2023-06-22T08:00:00.755018+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:51.267726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755485+0000","last_change":"2023-06-22T07:59:59.201187+0000","last_active":"2023-06-22T08:00:00.755485+0000","last_peered":"2023-06-22T08:00:00.755485+0000","last_clean":"2023-06-22T08:00:00.755485+0000","last_became_active":"2023-06-22T07:59:59.200688+0000","last_became_peered":"2023-06-22T07:59:59.200688+0000","last_unstale":"2023-06-22T08:00:00.755485+0000","last_undegraded":"2023-06-22T08:00:00.755485+0000","last_fullsized":"2023-06-22T08:00:00.755485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:52:15.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2,6],"acting":[4,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871926+0000","last_change":"2023-06-22T07:59:49.655839+0000","last_active":"2023-06-22T07:59:59.871926+0000","last_peered":"2023-06-22T07:59:59.871926+0000","last_clean":"2023-06-22T07:59:59.871926+0000","last_became_active":"2023-06-22T07:59:49.655494+0000","last_became_peered":"2023-06-22T07:59:49.655494+0000","last_unstale":"2023-06-22T07:59:59.871926+0000","last_undegraded":"2023-06-22T07:59:59.871926+0000","last_fullsized":"2023-06-22T07:59:59.871926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:01:23.950347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754455+0000","last_change":"2023-06-22T07:59:54.641464+0000","last_active":"2023-06-22T08:00:00.754455+0000","last_peered":"2023-06-22T08:00:00.754455+0000","last_clean":"2023-06-22T08:00:00.754455+0000","last_became_active":"2023-06-22T07:59:54.641161+0000","last_became_peered":"2023-06-22T07:59:54.641161+0000","last_unstale":"2023-06-22T08:00:00.754455+0000","last_undegraded":"2023-06-22T08:00:00.754455+0000","last_fullsized":"2023-06-22T08:00:00.754455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:46.346592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708316+0000","last_change":"2023-06-22T07:59:55.164629+0000","last_active":"2023-06-22T07:59:59.708316+0000","last_peered":"2023-06-22T07:59:59.708316+0000","last_clean":"2023-06-22T07:59:59.708316+0000","last_became_active":"2023-06-22T07:59:55.164329+0000","last_became_peered":"2023-06-22T07:59:55.164329+0000","last_unstale":"2023-06-22T07:59:59.708316+0000","last_undegraded":"2023-06-22T07:59:59.708316+0000","last_fullsized":"2023-06-22T07:59:59.708316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754950+0000","last_change":"2023-06-22T07:59:59.201778+0000","last_active":"2023-06-22T08:00:00.754950+0000","last_peered":"2023-06-22T08:00:00.754950+0000","last_clean":"2023-06-22T08:00:00.754950+0000","last_became_active":"2023-06-22T07:59:59.201326+0000","last_became_peered":"2023-06-22T07:59:59.201326+0000","last_unstale":"2023-06-22T08:00:00.754950+0000","last_undegraded":"2023-06-22T08:00:00.754950+0000","last_fullsized":"2023-06-22T08:00:00.754950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:17:31.817024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.703139+0000","last_change":"2023-06-22T07:59:54.644816+0000","last_active":"2023-06-22T08:00:25.703139+0000","last_peered":"2023-06-22T08:00:25.703139+0000","last_clean":"2023-06-22T08:00:25.703139+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:25.703139+0000","last_undegraded":"2023-06-22T08:00:25.703139+0000","last_fullsized":"2023-06-22T08:00:25.703139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:33.819054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753735+0000","last_change":"2023-06-22T07:59:55.167979+0000","last_active":"2023-06-22T08:00:00.753735+0000","last_peered":"2023-06-22T08:00:00.753735+0000","last_clean":"2023-06-22T08:00:00.753735+0000","last_became_active":"2023-06-22T07:59:55.163775+0000","last_became_peered":"2023-06-22T07:59:55.163775+0000","last_unstale":"2023-06-22T08:00:00.753735+0000","last_undegraded":"2023-06-22T08:00:00.753735+0000","last_fullsized":"2023-06-22T08:00:00.753735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155135+0000","last_change":"2023-06-22T07:59:59.194581+0000","last_active":"2023-06-22T08:00:00.155135+0000","last_peered":"2023-06-22T08:00:00.155135+0000","last_clean":"2023-06-22T08:00:00.155135+0000","last_became_active":"2023-06-22T07:59:59.194135+0000","last_became_peered":"2023-06-22T07:59:59.194135+0000","last_unstale":"2023-06-22T08:00:00.155135+0000","last_undegraded":"2023-06-22T08:00:00.155135+0000","last_fullsized":"2023-06-22T08:00:00.155135+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"27'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.156863+0000","last_change":"2023-06-22T07:59:54.651138+0000","last_active":"2023-06-22T08:00:00.156863+0000","last_peered":"2023-06-22T08:00:00.156863+0000","last_clean":"2023-06-22T08:00:00.156863+0000","last_became_active":"2023-06-22T07:59:54.650803+0000","last_became_peered":"2023-06-22T07:59:54.650803+0000","last_unstale":"2023-06-22T08:00:00.156863+0000","last_undegraded":"2023-06-22T08:00:00.156863+0000","last_fullsized":"2023-06-22T08:00:00.156863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:45.982711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708617+0000","last_change":"2023-06-22T07:59:55.165814+0000","last_active":"2023-06-22T07:59:59.708617+0000","last_peered":"2023-06-22T07:59:59.708617+0000","last_clean":"2023-06-22T07:59:59.708617+0000","last_became_active":"2023-06-22T07:59:55.165270+0000","last_became_peered":"2023-06-22T07:59:55.165270+0000","last_unstale":"2023-06-22T07:59:59.708617+0000","last_undegraded":"2023-06-22T07:59:59.708617+0000","last_fullsized":"2023-06-22T07:59:59.708617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708465+0000","last_change":"2023-06-22T07:59:59.212042+0000","last_active":"2023-06-22T07:59:59.708465+0000","last_peered":"2023-06-22T07:59:59.708465+0000","last_clean":"2023-06-22T07:59:59.708465+0000","last_became_active":"2023-06-22T07:59:59.211666+0000","last_became_peered":"2023-06-22T07:59:59.211666+0000","last_unstale":"2023-06-22T07:59:59.708465+0000","last_undegraded":"2023-06-22T07:59:59.708465+0000","last_fullsized":"2023-06-22T07:59:59.708465+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,4],"acting":[2,7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158226+0000","last_change":"2023-06-22T07:59:54.650251+0000","last_active":"2023-06-22T08:00:00.158226+0000","last_peered":"2023-06-22T08:00:00.158226+0000","last_clean":"2023-06-22T08:00:00.158226+0000","last_became_active":"2023-06-22T07:59:54.649993+0000","last_became_peered":"2023-06-22T07:59:54.649993+0000","last_unstale":"2023-06-22T08:00:00.158226+0000","last_undegraded":"2023-06-22T08:00:00.158226+0000","last_fullsized":"2023-06-22T08:00:00.158226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:20:20.733896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708375+0000","last_change":"2023-06-22T07:59:55.172595+0000","last_active":"2023-06-22T07:59:59.708375+0000","last_peered":"2023-06-22T07:59:59.708375+0000","last_clean":"2023-06-22T07:59:59.708375+0000","last_became_active":"2023-06-22T07:59:55.165873+0000","last_became_peered":"2023-06-22T07:59:55.165873+0000","last_unstale":"2023-06-22T07:59:59.708375+0000","last_undegraded":"2023-06-22T07:59:59.708375+0000","last_fullsized":"2023-06-22T07:59:59.708375+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:02.592807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154969+0000","last_change":"2023-06-22T07:59:59.192708+0000","last_active":"2023-06-22T08:00:00.154969+0000","last_peered":"2023-06-22T08:00:00.154969+0000","last_clean":"2023-06-22T08:00:00.154969+0000","last_became_active":"2023-06-22T07:59:59.191693+0000","last_became_peered":"2023-06-22T07:59:59.191693+0000","last_unstale":"2023-06-22T08:00:00.154969+0000","last_undegraded":"2023-06-22T08:00:00.154969+0000","last_fullsized":"2023-06-22T08:00:00.154969+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T12:59:33.454962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5,7],"acting":[1,0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709255+0000","last_change":"2023-06-22T07:59:54.651179+0000","last_active":"2023-06-22T07:59:59.709255+0000","last_peered":"2023-06-22T07:59:59.709255+0000","last_clean":"2023-06-22T07:59:59.709255+0000","last_became_active":"2023-06-22T07:59:54.650852+0000","last_became_peered":"2023-06-22T07:59:54.650852+0000","last_unstale":"2023-06-22T07:59:59.709255+0000","last_undegraded":"2023-06-22T07:59:59.709255+0000","last_fullsized":"2023-06-22T07:59:59.709255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:01.351861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754043+0000","last_change":"2023-06-22T07:59:55.163050+0000","last_active":"2023-06-22T08:00:00.754043+0000","last_peered":"2023-06-22T08:00:00.754043+0000","last_clean":"2023-06-22T08:00:00.754043+0000","last_became_active":"2023-06-22T07:59:55.162754+0000","last_became_peered":"2023-06-22T07:59:55.162754+0000","last_unstale":"2023-06-22T08:00:00.754043+0000","last_undegraded":"2023-06-22T08:00:00.754043+0000","last_fullsized":"2023-06-22T08:00:00.754043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754797+0000","last_change":"2023-06-22T07:59:59.209558+0000","last_active":"2023-06-22T08:00:00.754797+0000","last_peered":"2023-06-22T08:00:00.754797+0000","last_clean":"2023-06-22T08:00:00.754797+0000","last_became_active":"2023-06-22T07:59:59.209129+0000","last_became_peered":"2023-06-22T07:59:59.209129+0000","last_unstale":"2023-06-22T08:00:00.754797+0000","last_undegraded":"2023-06-22T08:00:00.754797+0000","last_fullsized":"2023-06-22T08:00:00.754797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:21.581063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1,0],"acting":[4,3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154714+0000","last_change":"2023-06-22T07:59:54.652059+0000","last_active":"2023-06-22T08:00:00.154714+0000","last_peered":"2023-06-22T08:00:00.154714+0000","last_clean":"2023-06-22T08:00:00.154714+0000","last_became_active":"2023-06-22T07:59:54.651720+0000","last_became_peered":"2023-06-22T07:59:54.651720+0000","last_unstale":"2023-06-22T08:00:00.154714+0000","last_undegraded":"2023-06-22T08:00:00.154714+0000","last_fullsized":"2023-06-22T08:00:00.154714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:31.908657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708158+0000","last_change":"2023-06-22T07:59:55.172794+0000","last_active":"2023-06-22T07:59:59.708158+0000","last_peered":"2023-06-22T07:59:59.708158+0000","last_clean":"2023-06-22T07:59:59.708158+0000","last_became_active":"2023-06-22T07:59:55.172527+0000","last_became_peered":"2023-06-22T07:59:55.172527+0000","last_unstale":"2023-06-22T07:59:59.708158+0000","last_undegraded":"2023-06-22T07:59:59.708158+0000","last_fullsized":"2023-06-22T07:59:59.708158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155457+0000","last_change":"2023-06-22T07:59:55.165758+0000","last_active":"2023-06-22T08:00:00.155457+0000","last_peered":"2023-06-22T08:00:00.155457+0000","last_clean":"2023-06-22T08:00:00.155457+0000","last_became_active":"2023-06-22T07:59:55.165117+0000","last_became_peered":"2023-06-22T07:59:55.165117+0000","last_unstale":"2023-06-22T08:00:00.155457+0000","last_undegraded":"2023-06-22T08:00:00.155457+0000","last_fullsized":"2023-06-22T08:00:00.155457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:02.357853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753486+0000","last_change":"2023-06-22T07:59:55.162439+0000","last_active":"2023-06-22T08:00:00.753486+0000","last_peered":"2023-06-22T08:00:00.753486+0000","last_clean":"2023-06-22T08:00:00.753486+0000","last_became_active":"2023-06-22T07:59:55.162072+0000","last_became_peered":"2023-06-22T07:59:55.162072+0000","last_unstale":"2023-06-22T08:00:00.753486+0000","last_undegraded":"2023-06-22T08:00:00.753486+0000","last_fullsized":"2023-06-22T08:00:00.753486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157229+0000","last_change":"2023-06-22T07:59:55.159299+0000","last_active":"2023-06-22T08:00:00.157229+0000","last_peered":"2023-06-22T08:00:00.157229+0000","last_clean":"2023-06-22T08:00:00.157229+0000","last_became_active":"2023-06-22T07:59:55.158893+0000","last_became_peered":"2023-06-22T07:59:55.158893+0000","last_unstale":"2023-06-22T08:00:00.157229+0000","last_undegraded":"2023-06-22T08:00:00.157229+0000","last_fullsized":"2023-06-22T08:00:00.157229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709716+0000","last_change":"2023-06-22T07:59:55.162520+0000","last_active":"2023-06-22T07:59:59.709716+0000","last_peered":"2023-06-22T07:59:59.709716+0000","last_clean":"2023-06-22T07:59:59.709716+0000","last_became_active":"2023-06-22T07:59:55.162211+0000","last_became_peered":"2023-06-22T07:59:55.162211+0000","last_unstale":"2023-06-22T07:59:59.709716+0000","last_undegraded":"2023-06-22T07:59:59.709716+0000","last_fullsized":"2023-06-22T07:59:59.709716+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:13:00.662547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708947+0000","last_change":"2023-06-22T07:59:55.165364+0000","last_active":"2023-06-22T07:59:59.708947+0000","last_peered":"2023-06-22T07:59:59.708947+0000","last_clean":"2023-06-22T07:59:59.708947+0000","last_became_active":"2023-06-22T07:59:55.165021+0000","last_became_peered":"2023-06-22T07:59:55.165021+0000","last_unstale":"2023-06-22T07:59:59.708947+0000","last_undegraded":"2023-06-22T07:59:59.708947+0000","last_fullsized":"2023-06-22T07:59:59.708947+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:09:07.008945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5950,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":11900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476749,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":315392,"data_stored":127374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]}]},{"osd":3,"up_from":14,"seq":60129542157,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":327680,"data_stored":127636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":124222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4989999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.173999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5129999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":905216,"data_stored":714600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":901120,"data_stored":714598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":124252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":299008,"data_stored":123558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":323584,"data_stored":124860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-22T08:00:29.401 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-22T08:00:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.627+0000 7fef95dff700 1 Processor -- start 2023-06-22T08:00:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.628+0000 7fef95dff700 1 -- start start 2023-06-22T08:00:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.629+0000 7fef95dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fef900cf1b0 0x7fef900c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.629+0000 7fef95dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9015c370 con 0x7fef900c9fd0 2023-06-22T08:00:29.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.629+0000 7fef95dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9015bda0 con 0x7fef90152ce0 2023-06-22T08:00:29.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.629+0000 7fef95dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef901530b0 con 0x7fef900cf1b0 2023-06-22T08:00:29.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.629+0000 7fef8ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fef900cf1b0 0x7fef900c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.629+0000 7fef8effd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fef900c9fd0 0x7fef900c40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41428/0 (socket says 172.21.15.23:41428) 2023-06-22T08:00:29.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.629+0000 7fef8ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fef900cf1b0 0x7fef900c47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60984/0 (socket says 172.21.15.23:60984) 2023-06-22T08:00:29.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.629+0000 7fef8effd700 1 -- 172.21.15.23:0/3921233438 learned_addr learned my addr 172.21.15.23:0/3921233438 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.630+0000 7fef8ffff700 1 -- 172.21.15.23:0/3921233438 >> v1:172.21.15.67:6789/0 conn(0x7fef90152ce0 legacy=0x7fef9015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.630+0000 7fef8ffff700 1 -- 172.21.15.23:0/3921233438 >> v1:172.21.15.23:6789/0 conn(0x7fef900c9fd0 legacy=0x7fef900c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.630+0000 7fef8ffff700 1 -- 172.21.15.23:0/3921233438 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef900cf630 con 0x7fef900cf1b0 2023-06-22T08:00:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.630+0000 7fef8ffff700 1 --2- 172.21.15.23:0/3921233438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef900cf1b0 0x7fef900c47c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fef88009d90 tx=0x7fef8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e72fcba24c5f39 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.630+0000 7fef8e7fc700 1 -- 172.21.15.23:0/3921233438 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8800e070 con 0x7fef900cf1b0 2023-06-22T08:00:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.630+0000 7fef8e7fc700 1 -- 172.21.15.23:0/3921233438 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8800bad0 con 0x7fef900cf1b0 2023-06-22T08:00:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.630+0000 7fef8e7fc700 1 -- 172.21.15.23:0/3921233438 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8800be80 con 0x7fef900cf1b0 2023-06-22T08:00:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.631+0000 7fef95dff700 1 -- 172.21.15.23:0/3921233438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef900cf1b0 msgr2=0x7fef900c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.631+0000 7fef95dff700 1 --2- 172.21.15.23:0/3921233438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef900cf1b0 0x7fef900c47c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fef88009d90 tx=0x7fef8800b330 comp rx=0 tx=0).stop 2023-06-22T08:00:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.631+0000 7fef95dff700 1 -- 172.21.15.23:0/3921233438 shutdown_connections 2023-06-22T08:00:29.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.631+0000 7fef95dff700 1 --2- 172.21.15.23:0/3921233438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef900cf1b0 0x7fef900c47c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.631+0000 7fef95dff700 1 -- 172.21.15.23:0/3921233438 >> 172.21.15.23:0/3921233438 conn(0x7fef9006ae30 msgr2=0x7fef9006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:29.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.631+0000 7fef95dff700 1 -- 172.21.15.23:0/3921233438 shutdown_connections 2023-06-22T08:00:29.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.631+0000 7fef95dff700 1 -- 172.21.15.23:0/3921233438 wait complete. 2023-06-22T08:00:29.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.633+0000 7fef95dff700 1 Processor -- start 2023-06-22T08:00:29.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.633+0000 7fef95dff700 1 -- start start 2023-06-22T08:00:29.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef95dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef900c9fd0 0x7fef901ef010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef95dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef900cf1b0 0x7fef9015b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef95dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef90152ce0 0x7fef900b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef95dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef901ee600 con 0x7fef90152ce0 2023-06-22T08:00:29.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef95dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef90151530 con 0x7fef900cf1b0 2023-06-22T08:00:29.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef95dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef900b32b0 con 0x7fef900c9fd0 2023-06-22T08:00:29.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef8ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef90152ce0 0x7fef900b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef8f7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef900c9fd0 0x7fef901ef010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef8effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef900cf1b0 0x7fef9015b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef8ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef90152ce0 0x7fef900b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60988/0 (socket says 172.21.15.23:60988) 2023-06-22T08:00:29.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef8ffff700 1 -- 172.21.15.23:0/3667201599 learned_addr learned my addr 172.21.15.23:0/3667201599 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:29.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef8ffff700 1 -- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef900c9fd0 msgr2=0x7fef901ef010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef8ffff700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef900c9fd0 0x7fef901ef010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef8ffff700 1 -- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef900cf1b0 msgr2=0x7fef9015b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef8ffff700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef900cf1b0 0x7fef9015b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.634+0000 7fef8ffff700 1 -- 172.21.15.23:0/3667201599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef88009a70 con 0x7fef90152ce0 2023-06-22T08:00:29.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.635+0000 7fef8ffff700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef90152ce0 0x7fef900b3e90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fef8800b730 tx=0x7fef88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:29.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.635+0000 7fef77fff700 1 -- 172.21.15.23:0/3667201599 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88012430 con 0x7fef90152ce0 2023-06-22T08:00:29.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.635+0000 7fef77fff700 1 -- 172.21.15.23:0/3667201599 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef88012590 con 0x7fef90152ce0 2023-06-22T08:00:29.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.635+0000 7fef77fff700 1 -- 172.21.15.23:0/3667201599 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88012940 con 0x7fef90152ce0 2023-06-22T08:00:29.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.635+0000 7fef95dff700 1 -- 172.21.15.23:0/3667201599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef900b3590 con 0x7fef90152ce0 2023-06-22T08:00:29.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.635+0000 7fef95dff700 1 -- 172.21.15.23:0/3667201599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef901ee810 con 0x7fef90152ce0 2023-06-22T08:00:29.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.636+0000 7fef95dff700 1 -- 172.21.15.23:0/3667201599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef900b3a50 con 0x7fef90152ce0 2023-06-22T08:00:29.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.637+0000 7fef77fff700 1 -- 172.21.15.23:0/3667201599 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fef88012aa0 con 0x7fef90152ce0 2023-06-22T08:00:29.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.638+0000 7fef77fff700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef7c06ace0 0x7fef7c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:29.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.638+0000 7fef77fff700 1 -- 172.21.15.23:0/3667201599 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fef8808bcc0 con 0x7fef90152ce0 2023-06-22T08:00:29.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.639+0000 7fef8f7fe700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef7c06ace0 0x7fef7c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:29.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.639+0000 7fef8f7fe700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef7c06ace0 0x7fef7c06d190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fef80007fd0 tx=0x7fef80009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:29.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.643+0000 7fef77fff700 1 -- 172.21.15.23:0/3667201599 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fef88056b10 con 0x7fef90152ce0 2023-06-22T08:00:29.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.849+0000 7fef95dff700 1 -- 172.21.15.23:0/3667201599 --> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef900c9740 con 0x7fef7c06ace0 2023-06-22T08:00:29.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.861+0000 7fef77fff700 1 -- 172.21.15.23:0/3667201599 <== mgr.4100 v2:172.21.15.67:6832/3180103139 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367440 (secure 0 0 0) 0x7fef80001cc0 con 0x7fef7c06ace0 2023-06-22T08:00:29.868 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:29.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 -- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef7c06ace0 msgr2=0x7fef7c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef7c06ace0 0x7fef7c06d190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fef80007fd0 tx=0x7fef80009040 comp rx=0 tx=0).stop 2023-06-22T08:00:29.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 -- 172.21.15.23:0/3667201599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef90152ce0 msgr2=0x7fef900b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:29.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef90152ce0 0x7fef900b3e90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fef8800b730 tx=0x7fef88004ad0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 -- 172.21.15.23:0/3667201599 shutdown_connections 2023-06-22T08:00:29.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef90152ce0 0x7fef900b3e90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef900c9fd0 0x7fef901ef010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef7c06ace0 0x7fef7c06d190 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 --2- 172.21.15.23:0/3667201599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef900cf1b0 0x7fef9015b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:29.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 -- 172.21.15.23:0/3667201599 >> 172.21.15.23:0/3667201599 conn(0x7fef9006ae30 msgr2=0x7fef900662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:29.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 -- 172.21.15.23:0/3667201599 shutdown_connections 2023-06-22T08:00:29.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:29.866+0000 7fef95dff700 1 -- 172.21.15.23:0/3667201599 wait complete. 2023-06-22T08:00:29.872 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-06-22T08:00:29.887 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2023-06-22T08:00:28.562416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219224,"kb_used_data":3608,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755496,"statfs":{"total":773094113280,"available":772869627904,"internally_reserved":0,"allocated":3694592,"data_stored":2181100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1499,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003741"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709823+0000","last_change":"2023-06-22T07:59:55.165740+0000","last_active":"2023-06-22T07:59:59.709823+0000","last_peered":"2023-06-22T07:59:59.709823+0000","last_clean":"2023-06-22T07:59:59.709823+0000","last_became_active":"2023-06-22T07:59:55.165398+0000","last_became_peered":"2023-06-22T07:59:55.165398+0000","last_unstale":"2023-06-22T07:59:59.709823+0000","last_undegraded":"2023-06-22T07:59:59.709823+0000","last_fullsized":"2023-06-22T07:59:59.709823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:37.596656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709026+0000","last_change":"2023-06-22T07:59:55.174603+0000","last_active":"2023-06-22T07:59:59.709026+0000","last_peered":"2023-06-22T07:59:59.709026+0000","last_clean":"2023-06-22T07:59:59.709026+0000","last_became_active":"2023-06-22T07:59:55.174346+0000","last_became_peered":"2023-06-22T07:59:55.174346+0000","last_unstale":"2023-06-22T07:59:59.709026+0000","last_undegraded":"2023-06-22T07:59:59.709026+0000","last_fullsized":"2023-06-22T07:59:59.709026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157321+0000","last_change":"2023-06-22T07:59:55.167525+0000","last_active":"2023-06-22T08:00:00.157321+0000","last_peered":"2023-06-22T08:00:00.157321+0000","last_clean":"2023-06-22T08:00:00.157321+0000","last_became_active":"2023-06-22T07:59:55.167181+0000","last_became_peered":"2023-06-22T07:59:55.167181+0000","last_unstale":"2023-06-22T08:00:00.157321+0000","last_undegraded":"2023-06-22T08:00:00.157321+0000","last_fullsized":"2023-06-22T08:00:00.157321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157927+0000","last_change":"2023-06-22T07:59:55.160549+0000","last_active":"2023-06-22T08:00:00.157927+0000","last_peered":"2023-06-22T08:00:00.157927+0000","last_clean":"2023-06-22T08:00:00.157927+0000","last_became_active":"2023-06-22T07:59:55.160239+0000","last_became_peered":"2023-06-22T07:59:55.160239+0000","last_unstale":"2023-06-22T08:00:00.157927+0000","last_undegraded":"2023-06-22T08:00:00.157927+0000","last_fullsized":"2023-06-22T08:00:00.157927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754390+0000","last_change":"2023-06-22T07:59:55.173006+0000","last_active":"2023-06-22T08:00:00.754390+0000","last_peered":"2023-06-22T08:00:00.754390+0000","last_clean":"2023-06-22T08:00:00.754390+0000","last_became_active":"2023-06-22T07:59:55.172578+0000","last_became_peered":"2023-06-22T07:59:55.172578+0000","last_unstale":"2023-06-22T08:00:00.754390+0000","last_undegraded":"2023-06-22T08:00:00.754390+0000","last_fullsized":"2023-06-22T08:00:00.754390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:12:44.799856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871378+0000","last_change":"2023-06-22T07:59:55.173858+0000","last_active":"2023-06-22T07:59:59.871378+0000","last_peered":"2023-06-22T07:59:59.871378+0000","last_clean":"2023-06-22T07:59:59.871378+0000","last_became_active":"2023-06-22T07:59:55.173599+0000","last_became_peered":"2023-06-22T07:59:55.173599+0000","last_unstale":"2023-06-22T07:59:59.871378+0000","last_undegraded":"2023-06-22T07:59:59.871378+0000","last_fullsized":"2023-06-22T07:59:59.871378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:37.597195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871174+0000","last_change":"2023-06-22T07:59:55.168354+0000","last_active":"2023-06-22T07:59:59.871174+0000","last_peered":"2023-06-22T07:59:59.871174+0000","last_clean":"2023-06-22T07:59:59.871174+0000","last_became_active":"2023-06-22T07:59:55.168050+0000","last_became_peered":"2023-06-22T07:59:55.168050+0000","last_unstale":"2023-06-22T07:59:59.871174+0000","last_undegraded":"2023-06-22T07:59:59.871174+0000","last_fullsized":"2023-06-22T07:59:59.871174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:08:45.847378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155602+0000","last_change":"2023-06-22T07:59:55.159738+0000","last_active":"2023-06-22T08:00:00.155602+0000","last_peered":"2023-06-22T08:00:00.155602+0000","last_clean":"2023-06-22T08:00:00.155602+0000","last_became_active":"2023-06-22T07:59:55.159501+0000","last_became_peered":"2023-06-22T07:59:55.159501+0000","last_unstale":"2023-06-22T08:00:00.155602+0000","last_undegraded":"2023-06-22T08:00:00.155602+0000","last_fullsized":"2023-06-22T08:00:00.155602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:55.160768+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:55.160508+0000","last_became_peered":"2023-06-22T07:59:55.160508+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:55:48.713145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155821+0000","last_change":"2023-06-22T07:59:55.166546+0000","last_active":"2023-06-22T08:00:00.155821+0000","last_peered":"2023-06-22T08:00:00.155821+0000","last_clean":"2023-06-22T08:00:00.155821+0000","last_became_active":"2023-06-22T07:59:55.166311+0000","last_became_peered":"2023-06-22T07:59:55.166311+0000","last_unstale":"2023-06-22T08:00:00.155821+0000","last_undegraded":"2023-06-22T08:00:00.155821+0000","last_fullsized":"2023-06-22T08:00:00.155821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:35.822548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155943+0000","last_change":"2023-06-22T07:59:55.162295+0000","last_active":"2023-06-22T08:00:00.155943+0000","last_peered":"2023-06-22T08:00:00.155943+0000","last_clean":"2023-06-22T08:00:00.155943+0000","last_became_active":"2023-06-22T07:59:55.162010+0000","last_became_peered":"2023-06-22T07:59:55.162010+0000","last_unstale":"2023-06-22T08:00:00.155943+0000","last_undegraded":"2023-06-22T08:00:00.155943+0000","last_fullsized":"2023-06-22T08:00:00.155943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709563+0000","last_change":"2023-06-22T07:59:55.158990+0000","last_active":"2023-06-22T07:59:59.709563+0000","last_peered":"2023-06-22T07:59:59.709563+0000","last_clean":"2023-06-22T07:59:59.709563+0000","last_became_active":"2023-06-22T07:59:55.158614+0000","last_became_peered":"2023-06-22T07:59:55.158614+0000","last_unstale":"2023-06-22T07:59:59.709563+0000","last_undegraded":"2023-06-22T07:59:59.709563+0000","last_fullsized":"2023-06-22T07:59:59.709563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755166+0000","last_change":"2023-06-22T07:59:55.161453+0000","last_active":"2023-06-22T08:00:00.755166+0000","last_peered":"2023-06-22T08:00:00.755166+0000","last_clean":"2023-06-22T08:00:00.755166+0000","last_became_active":"2023-06-22T07:59:55.161168+0000","last_became_peered":"2023-06-22T07:59:55.161168+0000","last_unstale":"2023-06-22T08:00:00.755166+0000","last_undegraded":"2023-06-22T08:00:00.755166+0000","last_fullsized":"2023-06-22T08:00:00.755166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155022+0000","last_change":"2023-06-22T07:59:55.164270+0000","last_active":"2023-06-22T08:00:00.155022+0000","last_peered":"2023-06-22T08:00:00.155022+0000","last_clean":"2023-06-22T08:00:00.155022+0000","last_became_active":"2023-06-22T07:59:55.163993+0000","last_became_peered":"2023-06-22T07:59:55.163993+0000","last_unstale":"2023-06-22T08:00:00.155022+0000","last_undegraded":"2023-06-22T08:00:00.155022+0000","last_fullsized":"2023-06-22T08:00:00.155022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:52.051830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157430+0000","last_change":"2023-06-22T07:59:55.169039+0000","last_active":"2023-06-22T08:00:00.157430+0000","last_peered":"2023-06-22T08:00:00.157430+0000","last_clean":"2023-06-22T08:00:00.157430+0000","last_became_active":"2023-06-22T07:59:55.168698+0000","last_became_peered":"2023-06-22T07:59:55.168698+0000","last_unstale":"2023-06-22T08:00:00.157430+0000","last_undegraded":"2023-06-22T08:00:00.157430+0000","last_fullsized":"2023-06-22T08:00:00.157430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871035+0000","last_change":"2023-06-22T07:59:55.158014+0000","last_active":"2023-06-22T07:59:59.871035+0000","last_peered":"2023-06-22T07:59:59.871035+0000","last_clean":"2023-06-22T07:59:59.871035+0000","last_became_active":"2023-06-22T07:59:55.157562+0000","last_became_peered":"2023-06-22T07:59:55.157562+0000","last_unstale":"2023-06-22T07:59:59.871035+0000","last_undegraded":"2023-06-22T07:59:59.871035+0000","last_fullsized":"2023-06-22T07:59:59.871035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:08:33.856001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754963+0000","last_change":"2023-06-22T07:59:55.165957+0000","last_active":"2023-06-22T08:00:00.754963+0000","last_peered":"2023-06-22T08:00:00.754963+0000","last_clean":"2023-06-22T08:00:00.754963+0000","last_became_active":"2023-06-22T07:59:55.165643+0000","last_became_peered":"2023-06-22T07:59:55.165643+0000","last_unstale":"2023-06-22T08:00:00.754963+0000","last_undegraded":"2023-06-22T08:00:00.754963+0000","last_fullsized":"2023-06-22T08:00:00.754963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:10.134743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157391+0000","last_change":"2023-06-22T07:59:55.159858+0000","last_active":"2023-06-22T08:00:00.157391+0000","last_peered":"2023-06-22T08:00:00.157391+0000","last_clean":"2023-06-22T08:00:00.157391+0000","last_became_active":"2023-06-22T07:59:55.159444+0000","last_became_peered":"2023-06-22T07:59:55.159444+0000","last_unstale":"2023-06-22T08:00:00.157391+0000","last_undegraded":"2023-06-22T08:00:00.157391+0000","last_fullsized":"2023-06-22T08:00:00.157391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709424+0000","last_change":"2023-06-22T07:59:55.157162+0000","last_active":"2023-06-22T07:59:59.709424+0000","last_peered":"2023-06-22T07:59:59.709424+0000","last_clean":"2023-06-22T07:59:59.709424+0000","last_became_active":"2023-06-22T07:59:55.156882+0000","last_became_peered":"2023-06-22T07:59:55.156882+0000","last_unstale":"2023-06-22T07:59:59.709424+0000","last_undegraded":"2023-06-22T07:59:59.709424+0000","last_fullsized":"2023-06-22T07:59:59.709424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155892+0000","last_change":"2023-06-22T07:59:55.163834+0000","last_active":"2023-06-22T08:00:00.155892+0000","last_peered":"2023-06-22T08:00:00.155892+0000","last_clean":"2023-06-22T08:00:00.155892+0000","last_became_active":"2023-06-22T07:59:55.163580+0000","last_became_peered":"2023-06-22T07:59:55.163580+0000","last_unstale":"2023-06-22T08:00:00.155892+0000","last_undegraded":"2023-06-22T08:00:00.155892+0000","last_fullsized":"2023-06-22T08:00:00.155892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155285+0000","last_change":"2023-06-22T07:59:55.164569+0000","last_active":"2023-06-22T08:00:00.155285+0000","last_peered":"2023-06-22T08:00:00.155285+0000","last_clean":"2023-06-22T08:00:00.155285+0000","last_became_active":"2023-06-22T07:59:55.164286+0000","last_became_peered":"2023-06-22T07:59:55.164286+0000","last_unstale":"2023-06-22T08:00:00.155285+0000","last_undegraded":"2023-06-22T08:00:00.155285+0000","last_fullsized":"2023-06-22T08:00:00.155285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:17.386317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754137+0000","last_change":"2023-06-22T07:59:55.159883+0000","last_active":"2023-06-22T08:00:00.754137+0000","last_peered":"2023-06-22T08:00:00.754137+0000","last_clean":"2023-06-22T08:00:00.754137+0000","last_became_active":"2023-06-22T07:59:55.159588+0000","last_became_peered":"2023-06-22T07:59:55.159588+0000","last_unstale":"2023-06-22T08:00:00.754137+0000","last_undegraded":"2023-06-22T08:00:00.754137+0000","last_fullsized":"2023-06-22T08:00:00.754137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754941+0000","last_change":"2023-06-22T07:59:55.160638+0000","last_active":"2023-06-22T08:00:00.754941+0000","last_peered":"2023-06-22T08:00:00.754941+0000","last_clean":"2023-06-22T08:00:00.754941+0000","last_became_active":"2023-06-22T07:59:55.160315+0000","last_became_peered":"2023-06-22T07:59:55.160315+0000","last_unstale":"2023-06-22T08:00:00.754941+0000","last_undegraded":"2023-06-22T08:00:00.754941+0000","last_fullsized":"2023-06-22T08:00:00.754941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:51:09.921626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155504+0000","last_change":"2023-06-22T07:59:55.165950+0000","last_active":"2023-06-22T08:00:00.155504+0000","last_peered":"2023-06-22T08:00:00.155504+0000","last_clean":"2023-06-22T08:00:00.155504+0000","last_became_active":"2023-06-22T07:59:55.165505+0000","last_became_peered":"2023-06-22T07:59:55.165505+0000","last_unstale":"2023-06-22T08:00:00.155504+0000","last_undegraded":"2023-06-22T08:00:00.155504+0000","last_fullsized":"2023-06-22T08:00:00.155504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:13.512588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755640+0000","last_change":"2023-06-22T07:59:55.157572+0000","last_active":"2023-06-22T08:00:00.755640+0000","last_peered":"2023-06-22T08:00:00.755640+0000","last_clean":"2023-06-22T08:00:00.755640+0000","last_became_active":"2023-06-22T07:59:55.157055+0000","last_became_peered":"2023-06-22T07:59:55.157055+0000","last_unstale":"2023-06-22T08:00:00.755640+0000","last_undegraded":"2023-06-22T08:00:00.755640+0000","last_fullsized":"2023-06-22T08:00:00.755640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:34:35.685117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754095+0000","last_change":"2023-06-22T07:59:55.164696+0000","last_active":"2023-06-22T08:00:00.754095+0000","last_peered":"2023-06-22T08:00:00.754095+0000","last_clean":"2023-06-22T08:00:00.754095+0000","last_became_active":"2023-06-22T07:59:55.164179+0000","last_became_peered":"2023-06-22T07:59:55.164179+0000","last_unstale":"2023-06-22T08:00:00.754095+0000","last_undegraded":"2023-06-22T08:00:00.754095+0000","last_fullsized":"2023-06-22T08:00:00.754095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871504+0000","last_change":"2023-06-22T07:59:55.168838+0000","last_active":"2023-06-22T07:59:59.871504+0000","last_peered":"2023-06-22T07:59:59.871504+0000","last_clean":"2023-06-22T07:59:59.871504+0000","last_became_active":"2023-06-22T07:59:55.168551+0000","last_became_peered":"2023-06-22T07:59:55.168551+0000","last_unstale":"2023-06-22T07:59:59.871504+0000","last_undegraded":"2023-06-22T07:59:59.871504+0000","last_fullsized":"2023-06-22T07:59:59.871504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:02:36.615449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708648+0000","last_change":"2023-06-22T07:59:59.195356+0000","last_active":"2023-06-22T07:59:59.708648+0000","last_peered":"2023-06-22T07:59:59.708648+0000","last_clean":"2023-06-22T07:59:59.708648+0000","last_became_active":"2023-06-22T07:59:59.194814+0000","last_became_peered":"2023-06-22T07:59:59.194814+0000","last_unstale":"2023-06-22T07:59:59.708648+0000","last_undegraded":"2023-06-22T07:59:59.708648+0000","last_fullsized":"2023-06-22T07:59:59.708648+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:12.223267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6,0],"acting":[2,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"27'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155337+0000","last_change":"2023-06-22T07:59:54.651326+0000","last_active":"2023-06-22T08:00:00.155337+0000","last_peered":"2023-06-22T08:00:00.155337+0000","last_clean":"2023-06-22T08:00:00.155337+0000","last_became_active":"2023-06-22T07:59:54.651010+0000","last_became_peered":"2023-06-22T07:59:54.651010+0000","last_unstale":"2023-06-22T08:00:00.155337+0000","last_undegraded":"2023-06-22T08:00:00.155337+0000","last_fullsized":"2023-06-22T08:00:00.155337+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:17.769117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754538+0000","last_change":"2023-06-22T07:59:55.160345+0000","last_active":"2023-06-22T08:00:00.754538+0000","last_peered":"2023-06-22T08:00:00.754538+0000","last_clean":"2023-06-22T08:00:00.754538+0000","last_became_active":"2023-06-22T07:59:55.160071+0000","last_became_peered":"2023-06-22T07:59:55.160071+0000","last_unstale":"2023-06-22T08:00:00.754538+0000","last_undegraded":"2023-06-22T08:00:00.754538+0000","last_fullsized":"2023-06-22T08:00:00.754538+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708662+0000","last_change":"2023-06-22T07:59:59.196477+0000","last_active":"2023-06-22T07:59:59.708662+0000","last_peered":"2023-06-22T07:59:59.708662+0000","last_clean":"2023-06-22T07:59:59.708662+0000","last_became_active":"2023-06-22T07:59:59.196007+0000","last_became_peered":"2023-06-22T07:59:59.196007+0000","last_unstale":"2023-06-22T07:59:59.708662+0000","last_undegraded":"2023-06-22T07:59:59.708662+0000","last_fullsized":"2023-06-22T07:59:59.708662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,5],"acting":[2,7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753791+0000","last_change":"2023-06-22T07:59:54.653192+0000","last_active":"2023-06-22T08:00:00.753791+0000","last_peered":"2023-06-22T08:00:00.753791+0000","last_clean":"2023-06-22T08:00:00.753791+0000","last_became_active":"2023-06-22T07:59:54.652926+0000","last_became_peered":"2023-06-22T07:59:54.652926+0000","last_unstale":"2023-06-22T08:00:00.753791+0000","last_undegraded":"2023-06-22T08:00:00.753791+0000","last_fullsized":"2023-06-22T08:00:00.753791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:12:59.656555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754609+0000","last_change":"2023-06-22T07:59:55.163171+0000","last_active":"2023-06-22T08:00:00.754609+0000","last_peered":"2023-06-22T08:00:00.754609+0000","last_clean":"2023-06-22T08:00:00.754609+0000","last_became_active":"2023-06-22T07:59:55.162869+0000","last_became_peered":"2023-06-22T07:59:55.162869+0000","last_unstale":"2023-06-22T08:00:00.754609+0000","last_undegraded":"2023-06-22T08:00:00.754609+0000","last_fullsized":"2023-06-22T08:00:00.754609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:37:59.063186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754586+0000","last_change":"2023-06-22T07:59:59.200516+0000","last_active":"2023-06-22T08:00:00.754586+0000","last_peered":"2023-06-22T08:00:00.754586+0000","last_clean":"2023-06-22T08:00:00.754586+0000","last_became_active":"2023-06-22T07:59:59.200096+0000","last_became_peered":"2023-06-22T07:59:59.200096+0000","last_unstale":"2023-06-22T08:00:00.754586+0000","last_undegraded":"2023-06-22T08:00:00.754586+0000","last_fullsized":"2023-06-22T08:00:00.754586+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:06.099105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2,7],"acting":[6,3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709623+0000","last_change":"2023-06-22T07:59:54.645950+0000","last_active":"2023-06-22T07:59:59.709623+0000","last_peered":"2023-06-22T07:59:59.709623+0000","last_clean":"2023-06-22T07:59:59.709623+0000","last_became_active":"2023-06-22T07:59:54.645520+0000","last_became_peered":"2023-06-22T07:59:54.645520+0000","last_unstale":"2023-06-22T07:59:59.709623+0000","last_undegraded":"2023-06-22T07:59:59.709623+0000","last_fullsized":"2023-06-22T07:59:59.709623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:34.816556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871992+0000","last_change":"2023-06-22T07:59:55.162767+0000","last_active":"2023-06-22T07:59:59.871992+0000","last_peered":"2023-06-22T07:59:59.871992+0000","last_clean":"2023-06-22T07:59:59.871992+0000","last_became_active":"2023-06-22T07:59:55.162451+0000","last_became_peered":"2023-06-22T07:59:55.162451+0000","last_unstale":"2023-06-22T07:59:59.871992+0000","last_undegraded":"2023-06-22T07:59:59.871992+0000","last_fullsized":"2023-06-22T07:59:59.871992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:38.977413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871071+0000","last_change":"2023-06-22T07:59:59.208215+0000","last_active":"2023-06-22T07:59:59.871071+0000","last_peered":"2023-06-22T07:59:59.871071+0000","last_clean":"2023-06-22T07:59:59.871071+0000","last_became_active":"2023-06-22T07:59:59.207608+0000","last_became_peered":"2023-06-22T07:59:59.207608+0000","last_unstale":"2023-06-22T07:59:59.871071+0000","last_undegraded":"2023-06-22T07:59:59.871071+0000","last_fullsized":"2023-06-22T07:59:59.871071+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:39:48.527076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,5],"acting":[3,2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.696994+0000","last_change":"2023-06-22T07:59:54.652515+0000","last_active":"2023-06-22T08:00:25.696994+0000","last_peered":"2023-06-22T08:00:25.696994+0000","last_clean":"2023-06-22T08:00:25.696994+0000","last_became_active":"2023-06-22T07:59:54.652249+0000","last_became_peered":"2023-06-22T07:59:54.652249+0000","last_unstale":"2023-06-22T08:00:25.696994+0000","last_undegraded":"2023-06-22T08:00:25.696994+0000","last_fullsized":"2023-06-22T08:00:25.696994+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:12.506596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755250+0000","last_change":"2023-06-22T07:59:55.158359+0000","last_active":"2023-06-22T08:00:00.755250+0000","last_peered":"2023-06-22T08:00:00.755250+0000","last_clean":"2023-06-22T08:00:00.755250+0000","last_became_active":"2023-06-22T07:59:55.157918+0000","last_became_peered":"2023-06-22T07:59:55.157918+0000","last_unstale":"2023-06-22T08:00:00.755250+0000","last_undegraded":"2023-06-22T08:00:00.755250+0000","last_fullsized":"2023-06-22T08:00:00.755250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708305+0000","last_change":"2023-06-22T07:59:59.201033+0000","last_active":"2023-06-22T07:59:59.708305+0000","last_peered":"2023-06-22T07:59:59.708305+0000","last_clean":"2023-06-22T07:59:59.708305+0000","last_became_active":"2023-06-22T07:59:59.200500+0000","last_became_peered":"2023-06-22T07:59:59.200500+0000","last_unstale":"2023-06-22T07:59:59.708305+0000","last_undegraded":"2023-06-22T07:59:59.708305+0000","last_fullsized":"2023-06-22T07:59:59.708305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:18.174604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,4],"acting":[2,1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.694471+0000","last_change":"2023-06-22T07:59:54.654416+0000","last_active":"2023-06-22T08:00:25.694471+0000","last_peered":"2023-06-22T08:00:25.694471+0000","last_clean":"2023-06-22T08:00:25.694471+0000","last_became_active":"2023-06-22T07:59:54.654139+0000","last_became_peered":"2023-06-22T07:59:54.654139+0000","last_unstale":"2023-06-22T08:00:25.694471+0000","last_undegraded":"2023-06-22T08:00:25.694471+0000","last_fullsized":"2023-06-22T08:00:25.694471+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755506+0000","last_change":"2023-06-22T07:59:55.159790+0000","last_active":"2023-06-22T08:00:00.755506+0000","last_peered":"2023-06-22T08:00:00.755506+0000","last_clean":"2023-06-22T08:00:00.755506+0000","last_became_active":"2023-06-22T07:59:55.159489+0000","last_became_peered":"2023-06-22T07:59:55.159489+0000","last_unstale":"2023-06-22T08:00:00.755506+0000","last_undegraded":"2023-06-22T08:00:00.755506+0000","last_fullsized":"2023-06-22T08:00:00.755506+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:15:18.373418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753814+0000","last_change":"2023-06-22T07:59:59.197822+0000","last_active":"2023-06-22T08:00:00.753814+0000","last_peered":"2023-06-22T08:00:00.753814+0000","last_clean":"2023-06-22T08:00:00.753814+0000","last_became_active":"2023-06-22T07:59:59.197431+0000","last_became_peered":"2023-06-22T07:59:59.197431+0000","last_unstale":"2023-06-22T08:00:00.753814+0000","last_undegraded":"2023-06-22T08:00:00.753814+0000","last_fullsized":"2023-06-22T08:00:00.753814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:41.139500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7,1],"acting":[4,6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157224+0000","last_change":"2023-06-22T07:59:54.644158+0000","last_active":"2023-06-22T08:00:00.157224+0000","last_peered":"2023-06-22T08:00:00.157224+0000","last_clean":"2023-06-22T08:00:00.157224+0000","last_became_active":"2023-06-22T07:59:54.643808+0000","last_became_peered":"2023-06-22T07:59:54.643808+0000","last_unstale":"2023-06-22T08:00:00.157224+0000","last_undegraded":"2023-06-22T08:00:00.157224+0000","last_fullsized":"2023-06-22T08:00:00.157224+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:45.438691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754507+0000","last_change":"2023-06-22T07:59:55.157538+0000","last_active":"2023-06-22T08:00:00.754507+0000","last_peered":"2023-06-22T08:00:00.754507+0000","last_clean":"2023-06-22T08:00:00.754507+0000","last_became_active":"2023-06-22T07:59:55.156967+0000","last_became_peered":"2023-06-22T07:59:55.156967+0000","last_unstale":"2023-06-22T08:00:00.754507+0000","last_undegraded":"2023-06-22T08:00:00.754507+0000","last_fullsized":"2023-06-22T08:00:00.754507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:38:44.161245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753821+0000","last_change":"2023-06-22T07:59:59.201671+0000","last_active":"2023-06-22T08:00:00.753821+0000","last_peered":"2023-06-22T08:00:00.753821+0000","last_clean":"2023-06-22T08:00:00.753821+0000","last_became_active":"2023-06-22T07:59:59.201201+0000","last_became_peered":"2023-06-22T07:59:59.201201+0000","last_unstale":"2023-06-22T08:00:00.753821+0000","last_undegraded":"2023-06-22T08:00:00.753821+0000","last_fullsized":"2023-06-22T08:00:00.753821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4,3],"acting":[6,1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"28'12","reported_seq":33,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.696162+0000","last_change":"2023-06-22T07:59:54.640470+0000","last_active":"2023-06-22T08:00:25.696162+0000","last_peered":"2023-06-22T08:00:25.696162+0000","last_clean":"2023-06-22T08:00:25.696162+0000","last_became_active":"2023-06-22T07:59:54.640124+0000","last_became_peered":"2023-06-22T07:59:54.640124+0000","last_unstale":"2023-06-22T08:00:25.696162+0000","last_undegraded":"2023-06-22T08:00:25.696162+0000","last_fullsized":"2023-06-22T08:00:25.696162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:06:41.502702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708599+0000","last_change":"2023-06-22T07:59:55.165550+0000","last_active":"2023-06-22T07:59:59.708599+0000","last_peered":"2023-06-22T07:59:59.708599+0000","last_clean":"2023-06-22T07:59:59.708599+0000","last_became_active":"2023-06-22T07:59:55.164998+0000","last_became_peered":"2023-06-22T07:59:55.164998+0000","last_unstale":"2023-06-22T07:59:59.708599+0000","last_undegraded":"2023-06-22T07:59:59.708599+0000","last_fullsized":"2023-06-22T07:59:59.708599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872133+0000","last_change":"2023-06-22T07:59:59.207103+0000","last_active":"2023-06-22T07:59:59.872133+0000","last_peered":"2023-06-22T07:59:59.872133+0000","last_clean":"2023-06-22T07:59:59.872133+0000","last_became_active":"2023-06-22T07:59:59.206531+0000","last_became_peered":"2023-06-22T07:59:59.206531+0000","last_unstale":"2023-06-22T07:59:59.872133+0000","last_undegraded":"2023-06-22T07:59:59.872133+0000","last_fullsized":"2023-06-22T07:59:59.872133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:42:48.571228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0,6],"acting":[3,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.694690+0000","last_change":"2023-06-22T07:59:54.644301+0000","last_active":"2023-06-22T08:00:25.694690+0000","last_peered":"2023-06-22T08:00:25.694690+0000","last_clean":"2023-06-22T08:00:25.694690+0000","last_became_active":"2023-06-22T07:59:54.643939+0000","last_became_peered":"2023-06-22T07:59:54.643939+0000","last_unstale":"2023-06-22T08:00:25.694690+0000","last_undegraded":"2023-06-22T08:00:25.694690+0000","last_fullsized":"2023-06-22T08:00:25.694690+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:07:42.118767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753877+0000","last_change":"2023-06-22T07:59:55.163311+0000","last_active":"2023-06-22T08:00:00.753877+0000","last_peered":"2023-06-22T08:00:00.753877+0000","last_clean":"2023-06-22T08:00:00.753877+0000","last_became_active":"2023-06-22T07:59:55.162991+0000","last_became_peered":"2023-06-22T07:59:55.162991+0000","last_unstale":"2023-06-22T08:00:00.753877+0000","last_undegraded":"2023-06-22T08:00:00.753877+0000","last_fullsized":"2023-06-22T08:00:00.753877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754227+0000","last_change":"2023-06-22T07:59:59.201324+0000","last_active":"2023-06-22T08:00:00.754227+0000","last_peered":"2023-06-22T08:00:00.754227+0000","last_clean":"2023-06-22T08:00:00.754227+0000","last_became_active":"2023-06-22T07:59:59.200877+0000","last_became_peered":"2023-06-22T07:59:59.200877+0000","last_unstale":"2023-06-22T08:00:00.754227+0000","last_undegraded":"2023-06-22T08:00:00.754227+0000","last_fullsized":"2023-06-22T08:00:00.754227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:14:17.094426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5,7],"acting":[4,2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.694163+0000","last_change":"2023-06-22T07:59:54.651527+0000","last_active":"2023-06-22T08:00:25.694163+0000","last_peered":"2023-06-22T08:00:25.694163+0000","last_clean":"2023-06-22T08:00:25.694163+0000","last_became_active":"2023-06-22T07:59:54.651196+0000","last_became_peered":"2023-06-22T07:59:54.651196+0000","last_unstale":"2023-06-22T08:00:25.694163+0000","last_undegraded":"2023-06-22T08:00:25.694163+0000","last_fullsized":"2023-06-22T08:00:25.694163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872142+0000","last_change":"2023-06-22T07:59:55.162610+0000","last_active":"2023-06-22T07:59:59.872142+0000","last_peered":"2023-06-22T07:59:59.872142+0000","last_clean":"2023-06-22T07:59:59.872142+0000","last_became_active":"2023-06-22T07:59:55.162334+0000","last_became_peered":"2023-06-22T07:59:55.162334+0000","last_unstale":"2023-06-22T07:59:59.872142+0000","last_undegraded":"2023-06-22T07:59:59.872142+0000","last_fullsized":"2023-06-22T07:59:59.872142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:42.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:59.199402+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:59.198979+0000","last_became_peered":"2023-06-22T07:59:59.198979+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:17:07.122732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1,6],"acting":[4,5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.870997+0000","last_change":"2023-06-22T07:59:54.649236+0000","last_active":"2023-06-22T07:59:59.870997+0000","last_peered":"2023-06-22T07:59:59.870997+0000","last_clean":"2023-06-22T07:59:59.870997+0000","last_became_active":"2023-06-22T07:59:54.648852+0000","last_became_peered":"2023-06-22T07:59:54.648852+0000","last_unstale":"2023-06-22T07:59:59.870997+0000","last_undegraded":"2023-06-22T07:59:59.870997+0000","last_fullsized":"2023-06-22T07:59:59.870997+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:16:59.865638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709134+0000","last_change":"2023-06-22T07:59:55.165859+0000","last_active":"2023-06-22T07:59:59.709134+0000","last_peered":"2023-06-22T07:59:59.709134+0000","last_clean":"2023-06-22T07:59:59.709134+0000","last_became_active":"2023-06-22T07:59:55.165542+0000","last_became_peered":"2023-06-22T07:59:55.165542+0000","last_unstale":"2023-06-22T07:59:59.709134+0000","last_undegraded":"2023-06-22T07:59:59.709134+0000","last_fullsized":"2023-06-22T07:59:59.709134+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:32.147969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871894+0000","last_change":"2023-06-22T07:59:59.207794+0000","last_active":"2023-06-22T07:59:59.871894+0000","last_peered":"2023-06-22T07:59:59.871894+0000","last_clean":"2023-06-22T07:59:59.871894+0000","last_became_active":"2023-06-22T07:59:59.207364+0000","last_became_peered":"2023-06-22T07:59:59.207364+0000","last_unstale":"2023-06-22T07:59:59.871894+0000","last_undegraded":"2023-06-22T07:59:59.871894+0000","last_fullsized":"2023-06-22T07:59:59.871894+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:30.718491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4,0],"acting":[3,6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"28'10","reported_seq":31,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.694836+0000","last_change":"2023-06-22T07:59:54.651423+0000","last_active":"2023-06-22T08:00:25.694836+0000","last_peered":"2023-06-22T08:00:25.694836+0000","last_clean":"2023-06-22T08:00:25.694836+0000","last_became_active":"2023-06-22T07:59:54.651167+0000","last_became_peered":"2023-06-22T07:59:54.651167+0000","last_unstale":"2023-06-22T08:00:25.694836+0000","last_undegraded":"2023-06-22T08:00:25.694836+0000","last_fullsized":"2023-06-22T08:00:25.694836+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:31.141977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709649+0000","last_change":"2023-06-22T07:59:55.164478+0000","last_active":"2023-06-22T07:59:59.709649+0000","last_peered":"2023-06-22T07:59:59.709649+0000","last_clean":"2023-06-22T07:59:59.709649+0000","last_became_active":"2023-06-22T07:59:55.163922+0000","last_became_peered":"2023-06-22T07:59:55.163922+0000","last_unstale":"2023-06-22T07:59:59.709649+0000","last_undegraded":"2023-06-22T07:59:59.709649+0000","last_fullsized":"2023-06-22T07:59:59.709649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:43.772399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155085+0000","last_change":"2023-06-22T07:59:59.198496+0000","last_active":"2023-06-22T08:00:00.155085+0000","last_peered":"2023-06-22T08:00:00.155085+0000","last_clean":"2023-06-22T08:00:00.155085+0000","last_became_active":"2023-06-22T07:59:59.198067+0000","last_became_peered":"2023-06-22T07:59:59.198067+0000","last_unstale":"2023-06-22T08:00:00.155085+0000","last_undegraded":"2023-06-22T08:00:00.155085+0000","last_fullsized":"2023-06-22T08:00:00.155085+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:24.865757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,7],"acting":[1,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"27'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157653+0000","last_change":"2023-06-22T07:59:54.651366+0000","last_active":"2023-06-22T08:00:00.157653+0000","last_peered":"2023-06-22T08:00:00.157653+0000","last_clean":"2023-06-22T08:00:00.157653+0000","last_became_active":"2023-06-22T07:59:54.651059+0000","last_became_peered":"2023-06-22T07:59:54.651059+0000","last_unstale":"2023-06-22T08:00:00.157653+0000","last_undegraded":"2023-06-22T08:00:00.157653+0000","last_fullsized":"2023-06-22T08:00:00.157653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:37.183143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871131+0000","last_change":"2023-06-22T07:59:55.162820+0000","last_active":"2023-06-22T07:59:59.871131+0000","last_peered":"2023-06-22T07:59:59.871131+0000","last_clean":"2023-06-22T07:59:59.871131+0000","last_became_active":"2023-06-22T07:59:55.162505+0000","last_became_peered":"2023-06-22T07:59:55.162505+0000","last_unstale":"2023-06-22T07:59:59.871131+0000","last_undegraded":"2023-06-22T07:59:59.871131+0000","last_fullsized":"2023-06-22T07:59:59.871131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:46:38.312778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753744+0000","last_change":"2023-06-22T07:59:59.201059+0000","last_active":"2023-06-22T08:00:00.753744+0000","last_peered":"2023-06-22T08:00:00.753744+0000","last_clean":"2023-06-22T08:00:00.753744+0000","last_became_active":"2023-06-22T07:59:59.200460+0000","last_became_peered":"2023-06-22T07:59:59.200460+0000","last_unstale":"2023-06-22T08:00:00.753744+0000","last_undegraded":"2023-06-22T08:00:00.753744+0000","last_fullsized":"2023-06-22T08:00:00.753744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,0],"acting":[6,4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158090+0000","last_change":"2023-06-22T07:59:54.646017+0000","last_active":"2023-06-22T08:00:00.158090+0000","last_peered":"2023-06-22T08:00:00.158090+0000","last_clean":"2023-06-22T08:00:00.158090+0000","last_became_active":"2023-06-22T07:59:54.645554+0000","last_became_peered":"2023-06-22T07:59:54.645554+0000","last_unstale":"2023-06-22T08:00:00.158090+0000","last_undegraded":"2023-06-22T08:00:00.158090+0000","last_fullsized":"2023-06-22T08:00:00.158090+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:01.550269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154806+0000","last_change":"2023-06-22T07:59:55.166283+0000","last_active":"2023-06-22T08:00:00.154806+0000","last_peered":"2023-06-22T08:00:00.154806+0000","last_clean":"2023-06-22T08:00:00.154806+0000","last_became_active":"2023-06-22T07:59:55.165977+0000","last_became_peered":"2023-06-22T07:59:55.165977+0000","last_unstale":"2023-06-22T08:00:00.154806+0000","last_undegraded":"2023-06-22T08:00:00.154806+0000","last_fullsized":"2023-06-22T08:00:00.154806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:21:29.484640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709141+0000","last_change":"2023-06-22T07:59:59.205411+0000","last_active":"2023-06-22T07:59:59.709141+0000","last_peered":"2023-06-22T07:59:59.709141+0000","last_clean":"2023-06-22T07:59:59.709141+0000","last_became_active":"2023-06-22T07:59:59.204980+0000","last_became_peered":"2023-06-22T07:59:59.204980+0000","last_unstale":"2023-06-22T07:59:59.709141+0000","last_undegraded":"2023-06-22T07:59:59.709141+0000","last_fullsized":"2023-06-22T07:59:59.709141+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:22.317952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871622+0000","last_change":"2023-06-22T07:59:54.654491+0000","last_active":"2023-06-22T07:59:59.871622+0000","last_peered":"2023-06-22T07:59:59.871622+0000","last_clean":"2023-06-22T07:59:59.871622+0000","last_became_active":"2023-06-22T07:59:54.654197+0000","last_became_peered":"2023-06-22T07:59:54.654197+0000","last_unstale":"2023-06-22T07:59:59.871622+0000","last_undegraded":"2023-06-22T07:59:59.871622+0000","last_fullsized":"2023-06-22T07:59:59.871622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:42:59.725455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708444+0000","last_change":"2023-06-22T07:59:55.172053+0000","last_active":"2023-06-22T07:59:59.708444+0000","last_peered":"2023-06-22T07:59:59.708444+0000","last_clean":"2023-06-22T07:59:59.708444+0000","last_became_active":"2023-06-22T07:59:55.165501+0000","last_became_peered":"2023-06-22T07:59:55.165501+0000","last_unstale":"2023-06-22T07:59:59.708444+0000","last_undegraded":"2023-06-22T07:59:59.708444+0000","last_fullsized":"2023-06-22T07:59:59.708444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753990+0000","last_change":"2023-06-22T07:59:59.199179+0000","last_active":"2023-06-22T08:00:00.753990+0000","last_peered":"2023-06-22T08:00:00.753990+0000","last_clean":"2023-06-22T08:00:00.753990+0000","last_became_active":"2023-06-22T07:59:59.198740+0000","last_became_peered":"2023-06-22T07:59:59.198740+0000","last_unstale":"2023-06-22T08:00:00.753990+0000","last_undegraded":"2023-06-22T08:00:00.753990+0000","last_fullsized":"2023-06-22T08:00:00.753990+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158086+0000","last_change":"2023-06-22T07:59:54.644520+0000","last_active":"2023-06-22T08:00:00.158086+0000","last_peered":"2023-06-22T08:00:00.158086+0000","last_clean":"2023-06-22T08:00:00.158086+0000","last_became_active":"2023-06-22T07:59:54.644204+0000","last_became_peered":"2023-06-22T07:59:54.644204+0000","last_unstale":"2023-06-22T08:00:00.158086+0000","last_undegraded":"2023-06-22T08:00:00.158086+0000","last_fullsized":"2023-06-22T08:00:00.158086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:07.674431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709657+0000","last_change":"2023-06-22T07:59:55.164111+0000","last_active":"2023-06-22T07:59:59.709657+0000","last_peered":"2023-06-22T07:59:59.709657+0000","last_clean":"2023-06-22T07:59:59.709657+0000","last_became_active":"2023-06-22T07:59:55.163802+0000","last_became_peered":"2023-06-22T07:59:55.163802+0000","last_unstale":"2023-06-22T07:59:59.709657+0000","last_undegraded":"2023-06-22T07:59:59.709657+0000","last_fullsized":"2023-06-22T07:59:59.709657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755252+0000","last_change":"2023-06-22T07:59:59.197721+0000","last_active":"2023-06-22T08:00:00.755252+0000","last_peered":"2023-06-22T08:00:00.755252+0000","last_clean":"2023-06-22T08:00:00.755252+0000","last_became_active":"2023-06-22T07:59:59.197259+0000","last_became_peered":"2023-06-22T07:59:59.197259+0000","last_unstale":"2023-06-22T08:00:00.755252+0000","last_undegraded":"2023-06-22T08:00:00.755252+0000","last_fullsized":"2023-06-22T08:00:00.755252+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5,1],"acting":[7,0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.697877+0000","last_change":"2023-06-22T07:59:54.643124+0000","last_active":"2023-06-22T08:00:25.697877+0000","last_peered":"2023-06-22T08:00:25.697877+0000","last_clean":"2023-06-22T08:00:25.697877+0000","last_became_active":"2023-06-22T07:59:54.642833+0000","last_became_peered":"2023-06-22T07:59:54.642833+0000","last_unstale":"2023-06-22T08:00:25.697877+0000","last_undegraded":"2023-06-22T08:00:25.697877+0000","last_fullsized":"2023-06-22T08:00:25.697877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158064+0000","last_change":"2023-06-22T07:59:55.160551+0000","last_active":"2023-06-22T08:00:00.158064+0000","last_peered":"2023-06-22T08:00:00.158064+0000","last_clean":"2023-06-22T08:00:00.158064+0000","last_became_active":"2023-06-22T07:59:55.160221+0000","last_became_peered":"2023-06-22T07:59:55.160221+0000","last_unstale":"2023-06-22T08:00:00.158064+0000","last_undegraded":"2023-06-22T08:00:00.158064+0000","last_fullsized":"2023-06-22T08:00:00.158064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708974+0000","last_change":"2023-06-22T07:59:59.204643+0000","last_active":"2023-06-22T07:59:59.708974+0000","last_peered":"2023-06-22T07:59:59.708974+0000","last_clean":"2023-06-22T07:59:59.708974+0000","last_became_active":"2023-06-22T07:59:59.204182+0000","last_became_peered":"2023-06-22T07:59:59.204182+0000","last_unstale":"2023-06-22T07:59:59.708974+0000","last_undegraded":"2023-06-22T07:59:59.708974+0000","last_fullsized":"2023-06-22T07:59:59.708974+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4,2],"acting":[5,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871327+0000","last_change":"2023-06-22T07:59:54.654036+0000","last_active":"2023-06-22T07:59:59.871327+0000","last_peered":"2023-06-22T07:59:59.871327+0000","last_clean":"2023-06-22T07:59:59.871327+0000","last_became_active":"2023-06-22T07:59:54.653736+0000","last_became_peered":"2023-06-22T07:59:54.653736+0000","last_unstale":"2023-06-22T07:59:59.871327+0000","last_undegraded":"2023-06-22T07:59:59.871327+0000","last_fullsized":"2023-06-22T07:59:59.871327+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:44:09.331748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871999+0000","last_change":"2023-06-22T07:59:55.163181+0000","last_active":"2023-06-22T07:59:59.871999+0000","last_peered":"2023-06-22T07:59:59.871999+0000","last_clean":"2023-06-22T07:59:59.871999+0000","last_became_active":"2023-06-22T07:59:55.162860+0000","last_became_peered":"2023-06-22T07:59:55.162860+0000","last_unstale":"2023-06-22T07:59:59.871999+0000","last_undegraded":"2023-06-22T07:59:59.871999+0000","last_fullsized":"2023-06-22T07:59:59.871999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:59:25.037938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155510+0000","last_change":"2023-06-22T07:59:59.195710+0000","last_active":"2023-06-22T08:00:00.155510+0000","last_peered":"2023-06-22T08:00:00.155510+0000","last_clean":"2023-06-22T08:00:00.155510+0000","last_became_active":"2023-06-22T07:59:59.195246+0000","last_became_peered":"2023-06-22T07:59:59.195246+0000","last_unstale":"2023-06-22T08:00:00.155510+0000","last_undegraded":"2023-06-22T08:00:00.155510+0000","last_fullsized":"2023-06-22T08:00:00.155510+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:36.457493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,5],"acting":[1,0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.697779+0000","last_change":"2023-06-22T07:59:54.649692+0000","last_active":"2023-06-22T08:00:25.697779+0000","last_peered":"2023-06-22T08:00:25.697779+0000","last_clean":"2023-06-22T08:00:25.697779+0000","last_became_active":"2023-06-22T07:59:54.649416+0000","last_became_peered":"2023-06-22T07:59:54.649416+0000","last_unstale":"2023-06-22T08:00:25.697779+0000","last_undegraded":"2023-06-22T08:00:25.697779+0000","last_fullsized":"2023-06-22T08:00:25.697779+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:54:02.923433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708909+0000","last_change":"2023-06-22T07:59:55.175392+0000","last_active":"2023-06-22T07:59:59.708909+0000","last_peered":"2023-06-22T07:59:59.708909+0000","last_clean":"2023-06-22T07:59:59.708909+0000","last_became_active":"2023-06-22T07:59:55.175112+0000","last_became_peered":"2023-06-22T07:59:55.175112+0000","last_unstale":"2023-06-22T07:59:59.708909+0000","last_undegraded":"2023-06-22T07:59:59.708909+0000","last_fullsized":"2023-06-22T07:59:59.708909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871873+0000","last_change":"2023-06-22T07:59:59.206671+0000","last_active":"2023-06-22T07:59:59.871873+0000","last_peered":"2023-06-22T07:59:59.871873+0000","last_clean":"2023-06-22T07:59:59.871873+0000","last_became_active":"2023-06-22T07:59:59.206179+0000","last_became_peered":"2023-06-22T07:59:59.206179+0000","last_unstale":"2023-06-22T07:59:59.871873+0000","last_undegraded":"2023-06-22T07:59:59.871873+0000","last_fullsized":"2023-06-22T07:59:59.871873+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:00.427518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4,5],"acting":[3,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.698074+0000","last_change":"2023-06-22T07:59:54.640508+0000","last_active":"2023-06-22T08:00:25.698074+0000","last_peered":"2023-06-22T08:00:25.698074+0000","last_clean":"2023-06-22T08:00:25.698074+0000","last_became_active":"2023-06-22T07:59:54.640108+0000","last_became_peered":"2023-06-22T07:59:54.640108+0000","last_unstale":"2023-06-22T08:00:25.698074+0000","last_undegraded":"2023-06-22T08:00:25.698074+0000","last_fullsized":"2023-06-22T08:00:25.698074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:19.193231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754822+0000","last_change":"2023-06-22T07:59:55.164398+0000","last_active":"2023-06-22T08:00:00.754822+0000","last_peered":"2023-06-22T08:00:00.754822+0000","last_clean":"2023-06-22T08:00:00.754822+0000","last_became_active":"2023-06-22T07:59:55.164086+0000","last_became_peered":"2023-06-22T07:59:55.164086+0000","last_unstale":"2023-06-22T08:00:00.754822+0000","last_undegraded":"2023-06-22T08:00:00.754822+0000","last_fullsized":"2023-06-22T08:00:00.754822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:38.189135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753997+0000","last_change":"2023-06-22T07:59:59.207752+0000","last_active":"2023-06-22T08:00:00.753997+0000","last_peered":"2023-06-22T08:00:00.753997+0000","last_clean":"2023-06-22T08:00:00.753997+0000","last_became_active":"2023-06-22T07:59:59.207360+0000","last_became_peered":"2023-06-22T07:59:59.207360+0000","last_unstale":"2023-06-22T08:00:00.753997+0000","last_undegraded":"2023-06-22T08:00:00.753997+0000","last_fullsized":"2023-06-22T08:00:00.753997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:08:25.633009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3,1],"acting":[4,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709114+0000","last_change":"2023-06-22T07:59:49.652772+0000","last_active":"2023-06-22T07:59:59.709114+0000","last_peered":"2023-06-22T07:59:59.709114+0000","last_clean":"2023-06-22T07:59:59.709114+0000","last_became_active":"2023-06-22T07:59:49.652487+0000","last_became_peered":"2023-06-22T07:59:49.652487+0000","last_unstale":"2023-06-22T07:59:59.709114+0000","last_undegraded":"2023-06-22T07:59:59.709114+0000","last_fullsized":"2023-06-22T07:59:59.709114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:11.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708614+0000","last_change":"2023-06-22T07:59:54.644538+0000","last_active":"2023-06-22T07:59:59.708614+0000","last_peered":"2023-06-22T07:59:59.708614+0000","last_clean":"2023-06-22T07:59:59.708614+0000","last_became_active":"2023-06-22T07:59:54.644253+0000","last_became_peered":"2023-06-22T07:59:54.644253+0000","last_unstale":"2023-06-22T07:59:59.708614+0000","last_undegraded":"2023-06-22T07:59:59.708614+0000","last_fullsized":"2023-06-22T07:59:59.708614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:09.128751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754228+0000","last_change":"2023-06-22T07:59:55.163802+0000","last_active":"2023-06-22T08:00:00.754228+0000","last_peered":"2023-06-22T08:00:00.754228+0000","last_clean":"2023-06-22T08:00:00.754228+0000","last_became_active":"2023-06-22T07:59:55.163481+0000","last_became_peered":"2023-06-22T07:59:55.163481+0000","last_unstale":"2023-06-22T08:00:00.754228+0000","last_undegraded":"2023-06-22T08:00:00.754228+0000","last_fullsized":"2023-06-22T08:00:00.754228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:16.916350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755246+0000","last_change":"2023-06-22T07:59:59.207344+0000","last_active":"2023-06-22T08:00:00.755246+0000","last_peered":"2023-06-22T08:00:00.755246+0000","last_clean":"2023-06-22T08:00:00.755246+0000","last_became_active":"2023-06-22T07:59:59.206975+0000","last_became_peered":"2023-06-22T07:59:59.206975+0000","last_unstale":"2023-06-22T08:00:00.755246+0000","last_undegraded":"2023-06-22T08:00:00.755246+0000","last_fullsized":"2023-06-22T08:00:00.755246+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3,1],"acting":[7,4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871513+0000","last_change":"2023-06-22T07:59:49.653053+0000","last_active":"2023-06-22T07:59:59.871513+0000","last_peered":"2023-06-22T07:59:59.871513+0000","last_clean":"2023-06-22T07:59:59.871513+0000","last_became_active":"2023-06-22T07:59:49.652861+0000","last_became_peered":"2023-06-22T07:59:49.652861+0000","last_unstale":"2023-06-22T07:59:59.871513+0000","last_undegraded":"2023-06-22T07:59:59.871513+0000","last_fullsized":"2023-06-22T07:59:59.871513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:28:40.388367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.701026+0000","last_change":"2023-06-22T07:59:54.645403+0000","last_active":"2023-06-22T08:00:25.701026+0000","last_peered":"2023-06-22T08:00:25.701026+0000","last_clean":"2023-06-22T08:00:25.701026+0000","last_became_active":"2023-06-22T07:59:54.644973+0000","last_became_peered":"2023-06-22T07:59:54.644973+0000","last_unstale":"2023-06-22T08:00:25.701026+0000","last_undegraded":"2023-06-22T08:00:25.701026+0000","last_fullsized":"2023-06-22T08:00:25.701026+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:15.910358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754101+0000","last_change":"2023-06-22T07:59:55.161484+0000","last_active":"2023-06-22T08:00:00.754101+0000","last_peered":"2023-06-22T08:00:00.754101+0000","last_clean":"2023-06-22T08:00:00.754101+0000","last_became_active":"2023-06-22T07:59:55.161194+0000","last_became_peered":"2023-06-22T07:59:55.161194+0000","last_unstale":"2023-06-22T08:00:00.754101+0000","last_undegraded":"2023-06-22T08:00:00.754101+0000","last_fullsized":"2023-06-22T08:00:00.754101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754988+0000","last_change":"2023-06-22T07:59:59.198241+0000","last_active":"2023-06-22T08:00:00.754988+0000","last_peered":"2023-06-22T08:00:00.754988+0000","last_clean":"2023-06-22T08:00:00.754988+0000","last_became_active":"2023-06-22T07:59:59.197689+0000","last_became_peered":"2023-06-22T07:59:59.197689+0000","last_unstale":"2023-06-22T08:00:00.754988+0000","last_undegraded":"2023-06-22T08:00:00.754988+0000","last_fullsized":"2023-06-22T08:00:00.754988+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:39.513874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4,1],"acting":[7,0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753886+0000","last_change":"2023-06-22T07:59:49.450044+0000","last_active":"2023-06-22T08:00:00.753886+0000","last_peered":"2023-06-22T08:00:00.753886+0000","last_clean":"2023-06-22T08:00:00.753886+0000","last_became_active":"2023-06-22T07:59:49.449731+0000","last_became_peered":"2023-06-22T07:59:49.449731+0000","last_unstale":"2023-06-22T08:00:00.753886+0000","last_undegraded":"2023-06-22T08:00:00.753886+0000","last_fullsized":"2023-06-22T08:00:00.753886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:40.262337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708682+0000","last_change":"2023-06-22T07:59:54.640462+0000","last_active":"2023-06-22T07:59:59.708682+0000","last_peered":"2023-06-22T07:59:59.708682+0000","last_clean":"2023-06-22T07:59:59.708682+0000","last_became_active":"2023-06-22T07:59:54.640123+0000","last_became_peered":"2023-06-22T07:59:54.640123+0000","last_unstale":"2023-06-22T07:59:59.708682+0000","last_undegraded":"2023-06-22T07:59:59.708682+0000","last_fullsized":"2023-06-22T07:59:59.708682+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:20.316921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755350+0000","last_change":"2023-06-22T07:59:55.164651+0000","last_active":"2023-06-22T08:00:00.755350+0000","last_peered":"2023-06-22T08:00:00.755350+0000","last_clean":"2023-06-22T08:00:00.755350+0000","last_became_active":"2023-06-22T07:59:55.164235+0000","last_became_peered":"2023-06-22T07:59:55.164235+0000","last_unstale":"2023-06-22T08:00:00.755350+0000","last_undegraded":"2023-06-22T08:00:00.755350+0000","last_fullsized":"2023-06-22T08:00:00.755350+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:32.924136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157741+0000","last_change":"2023-06-22T07:59:59.204779+0000","last_active":"2023-06-22T08:00:00.157741+0000","last_peered":"2023-06-22T08:00:00.157741+0000","last_clean":"2023-06-22T08:00:00.157741+0000","last_became_active":"2023-06-22T07:59:59.204288+0000","last_became_peered":"2023-06-22T07:59:59.204288+0000","last_unstale":"2023-06-22T08:00:00.157741+0000","last_undegraded":"2023-06-22T08:00:00.157741+0000","last_fullsized":"2023-06-22T08:00:00.157741+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871497+0000","last_change":"2023-06-22T07:59:49.657008+0000","last_active":"2023-06-22T07:59:59.871497+0000","last_peered":"2023-06-22T07:59:59.871497+0000","last_clean":"2023-06-22T07:59:59.871497+0000","last_became_active":"2023-06-22T07:59:49.656752+0000","last_became_peered":"2023-06-22T07:59:49.656752+0000","last_unstale":"2023-06-22T07:59:59.871497+0000","last_undegraded":"2023-06-22T07:59:59.871497+0000","last_fullsized":"2023-06-22T07:59:59.871497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:50:37.639134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":185,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155134+0000","last_change":"2023-06-22T07:59:42.501363+0000","last_active":"2023-06-22T08:00:00.155134+0000","last_peered":"2023-06-22T08:00:00.155134+0000","last_clean":"2023-06-22T08:00:00.155134+0000","last_became_active":"2023-06-22T07:59:42.500320+0000","last_became_peered":"2023-06-22T07:59:42.500320+0000","last_unstale":"2023-06-22T08:00:00.155134+0000","last_undegraded":"2023-06-22T08:00:00.155134+0000","last_fullsized":"2023-06-22T08:00:00.155134+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_clean_scrub_stamp":"2023-06-22T07:59:41.381475+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:08:53.784030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753785+0000","last_change":"2023-06-22T07:59:54.645031+0000","last_active":"2023-06-22T08:00:00.753785+0000","last_peered":"2023-06-22T08:00:00.753785+0000","last_clean":"2023-06-22T08:00:00.753785+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:00.753785+0000","last_undegraded":"2023-06-22T08:00:00.753785+0000","last_fullsized":"2023-06-22T08:00:00.753785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:42.766407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872074+0000","last_change":"2023-06-22T07:59:55.162799+0000","last_active":"2023-06-22T07:59:59.872074+0000","last_peered":"2023-06-22T07:59:59.872074+0000","last_clean":"2023-06-22T07:59:59.872074+0000","last_became_active":"2023-06-22T07:59:55.162501+0000","last_became_peered":"2023-06-22T07:59:55.162501+0000","last_unstale":"2023-06-22T07:59:59.872074+0000","last_undegraded":"2023-06-22T07:59:59.872074+0000","last_fullsized":"2023-06-22T07:59:59.872074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:57:14.305897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871805+0000","last_change":"2023-06-22T07:59:59.205097+0000","last_active":"2023-06-22T07:59:59.871805+0000","last_peered":"2023-06-22T07:59:59.871805+0000","last_clean":"2023-06-22T07:59:59.871805+0000","last_became_active":"2023-06-22T07:59:59.201465+0000","last_became_peered":"2023-06-22T07:59:59.201465+0000","last_unstale":"2023-06-22T07:59:59.871805+0000","last_undegraded":"2023-06-22T07:59:59.871805+0000","last_fullsized":"2023-06-22T07:59:59.871805+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:17:45.352998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,0],"acting":[3,7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871056+0000","last_change":"2023-06-22T07:59:49.442558+0000","last_active":"2023-06-22T07:59:59.871056+0000","last_peered":"2023-06-22T07:59:59.871056+0000","last_clean":"2023-06-22T07:59:59.871056+0000","last_became_active":"2023-06-22T07:59:49.442154+0000","last_became_peered":"2023-06-22T07:59:49.442154+0000","last_unstale":"2023-06-22T07:59:59.871056+0000","last_undegraded":"2023-06-22T07:59:59.871056+0000","last_fullsized":"2023-06-22T07:59:59.871056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:46:16.233204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753373+0000","last_change":"2023-06-22T07:59:54.641746+0000","last_active":"2023-06-22T08:00:00.753373+0000","last_peered":"2023-06-22T08:00:00.753373+0000","last_clean":"2023-06-22T08:00:00.753373+0000","last_became_active":"2023-06-22T07:59:54.641431+0000","last_became_peered":"2023-06-22T07:59:54.641431+0000","last_unstale":"2023-06-22T08:00:00.753373+0000","last_undegraded":"2023-06-22T08:00:00.753373+0000","last_fullsized":"2023-06-22T08:00:00.753373+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709468+0000","last_change":"2023-06-22T07:59:55.163214+0000","last_active":"2023-06-22T07:59:59.709468+0000","last_peered":"2023-06-22T07:59:59.709468+0000","last_clean":"2023-06-22T07:59:59.709468+0000","last_became_active":"2023-06-22T07:59:55.162907+0000","last_became_peered":"2023-06-22T07:59:55.162907+0000","last_unstale":"2023-06-22T07:59:59.709468+0000","last_undegraded":"2023-06-22T07:59:59.709468+0000","last_fullsized":"2023-06-22T07:59:59.709468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:46.988703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157450+0000","last_change":"2023-06-22T07:59:59.198945+0000","last_active":"2023-06-22T08:00:00.157450+0000","last_peered":"2023-06-22T08:00:00.157450+0000","last_clean":"2023-06-22T08:00:00.157450+0000","last_became_active":"2023-06-22T07:59:59.198561+0000","last_became_peered":"2023-06-22T07:59:59.198561+0000","last_unstale":"2023-06-22T08:00:00.157450+0000","last_undegraded":"2023-06-22T08:00:00.157450+0000","last_fullsized":"2023-06-22T08:00:00.157450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:06.135651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2,5],"acting":[0,7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871842+0000","last_change":"2023-06-22T07:59:49.442246+0000","last_active":"2023-06-22T07:59:59.871842+0000","last_peered":"2023-06-22T07:59:59.871842+0000","last_clean":"2023-06-22T07:59:59.871842+0000","last_became_active":"2023-06-22T07:59:49.442001+0000","last_became_peered":"2023-06-22T07:59:49.442001+0000","last_unstale":"2023-06-22T07:59:59.871842+0000","last_undegraded":"2023-06-22T07:59:59.871842+0000","last_fullsized":"2023-06-22T07:59:59.871842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:27:24.457978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754254+0000","last_change":"2023-06-22T07:59:54.643272+0000","last_active":"2023-06-22T08:00:00.754254+0000","last_peered":"2023-06-22T08:00:00.754254+0000","last_clean":"2023-06-22T08:00:00.754254+0000","last_became_active":"2023-06-22T07:59:54.642925+0000","last_became_peered":"2023-06-22T07:59:54.642925+0000","last_unstale":"2023-06-22T08:00:00.754254+0000","last_undegraded":"2023-06-22T08:00:00.754254+0000","last_fullsized":"2023-06-22T08:00:00.754254+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:13.625768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754231+0000","last_change":"2023-06-22T07:59:55.167307+0000","last_active":"2023-06-22T08:00:00.754231+0000","last_peered":"2023-06-22T08:00:00.754231+0000","last_clean":"2023-06-22T08:00:00.754231+0000","last_became_active":"2023-06-22T07:59:55.167026+0000","last_became_peered":"2023-06-22T07:59:55.167026+0000","last_unstale":"2023-06-22T08:00:00.754231+0000","last_undegraded":"2023-06-22T08:00:00.754231+0000","last_fullsized":"2023-06-22T08:00:00.754231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871379+0000","last_change":"2023-06-22T07:59:59.208263+0000","last_active":"2023-06-22T07:59:59.871379+0000","last_peered":"2023-06-22T07:59:59.871379+0000","last_clean":"2023-06-22T07:59:59.871379+0000","last_became_active":"2023-06-22T07:59:59.207691+0000","last_became_peered":"2023-06-22T07:59:59.207691+0000","last_unstale":"2023-06-22T07:59:59.871379+0000","last_undegraded":"2023-06-22T07:59:59.871379+0000","last_fullsized":"2023-06-22T07:59:59.871379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:37:57.170857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,2],"acting":[3,7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.697986+0000","last_change":"2023-06-22T07:59:54.644570+0000","last_active":"2023-06-22T08:00:25.697986+0000","last_peered":"2023-06-22T08:00:25.697986+0000","last_clean":"2023-06-22T08:00:25.697986+0000","last_became_active":"2023-06-22T07:59:54.644218+0000","last_became_peered":"2023-06-22T07:59:54.644218+0000","last_unstale":"2023-06-22T08:00:25.697986+0000","last_undegraded":"2023-06-22T08:00:25.697986+0000","last_fullsized":"2023-06-22T08:00:25.697986+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:05:00.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754571+0000","last_change":"2023-06-22T07:59:49.655601+0000","last_active":"2023-06-22T08:00:00.754571+0000","last_peered":"2023-06-22T08:00:00.754571+0000","last_clean":"2023-06-22T08:00:00.754571+0000","last_became_active":"2023-06-22T07:59:49.655328+0000","last_became_peered":"2023-06-22T07:59:49.655328+0000","last_unstale":"2023-06-22T08:00:00.754571+0000","last_undegraded":"2023-06-22T08:00:00.754571+0000","last_fullsized":"2023-06-22T08:00:00.754571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:29:09.967964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755018+0000","last_change":"2023-06-22T07:59:55.163241+0000","last_active":"2023-06-22T08:00:00.755018+0000","last_peered":"2023-06-22T08:00:00.755018+0000","last_clean":"2023-06-22T08:00:00.755018+0000","last_became_active":"2023-06-22T07:59:55.162949+0000","last_became_peered":"2023-06-22T07:59:55.162949+0000","last_unstale":"2023-06-22T08:00:00.755018+0000","last_undegraded":"2023-06-22T08:00:00.755018+0000","last_fullsized":"2023-06-22T08:00:00.755018+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:51.267726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755485+0000","last_change":"2023-06-22T07:59:59.201187+0000","last_active":"2023-06-22T08:00:00.755485+0000","last_peered":"2023-06-22T08:00:00.755485+0000","last_clean":"2023-06-22T08:00:00.755485+0000","last_became_active":"2023-06-22T07:59:59.200688+0000","last_became_peered":"2023-06-22T07:59:59.200688+0000","last_unstale":"2023-06-22T08:00:00.755485+0000","last_undegraded":"2023-06-22T08:00:00.755485+0000","last_fullsized":"2023-06-22T08:00:00.755485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:52:15.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2,6],"acting":[4,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871926+0000","last_change":"2023-06-22T07:59:49.655839+0000","last_active":"2023-06-22T07:59:59.871926+0000","last_peered":"2023-06-22T07:59:59.871926+0000","last_clean":"2023-06-22T07:59:59.871926+0000","last_became_active":"2023-06-22T07:59:49.655494+0000","last_became_peered":"2023-06-22T07:59:49.655494+0000","last_unstale":"2023-06-22T07:59:59.871926+0000","last_undegraded":"2023-06-22T07:59:59.871926+0000","last_fullsized":"2023-06-22T07:59:59.871926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:01:23.950347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754455+0000","last_change":"2023-06-22T07:59:54.641464+0000","last_active":"2023-06-22T08:00:00.754455+0000","last_peered":"2023-06-22T08:00:00.754455+0000","last_clean":"2023-06-22T08:00:00.754455+0000","last_became_active":"2023-06-22T07:59:54.641161+0000","last_became_peered":"2023-06-22T07:59:54.641161+0000","last_unstale":"2023-06-22T08:00:00.754455+0000","last_undegraded":"2023-06-22T08:00:00.754455+0000","last_fullsized":"2023-06-22T08:00:00.754455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:46.346592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708316+0000","last_change":"2023-06-22T07:59:55.164629+0000","last_active":"2023-06-22T07:59:59.708316+0000","last_peered":"2023-06-22T07:59:59.708316+0000","last_clean":"2023-06-22T07:59:59.708316+0000","last_became_active":"2023-06-22T07:59:55.164329+0000","last_became_peered":"2023-06-22T07:59:55.164329+0000","last_unstale":"2023-06-22T07:59:59.708316+0000","last_undegraded":"2023-06-22T07:59:59.708316+0000","last_fullsized":"2023-06-22T07:59:59.708316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754950+0000","last_change":"2023-06-22T07:59:59.201778+0000","last_active":"2023-06-22T08:00:00.754950+0000","last_peered":"2023-06-22T08:00:00.754950+0000","last_clean":"2023-06-22T08:00:00.754950+0000","last_became_active":"2023-06-22T07:59:59.201326+0000","last_became_peered":"2023-06-22T07:59:59.201326+0000","last_unstale":"2023-06-22T08:00:00.754950+0000","last_undegraded":"2023-06-22T08:00:00.754950+0000","last_fullsized":"2023-06-22T08:00:00.754950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:17:31.817024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.703139+0000","last_change":"2023-06-22T07:59:54.644816+0000","last_active":"2023-06-22T08:00:25.703139+0000","last_peered":"2023-06-22T08:00:25.703139+0000","last_clean":"2023-06-22T08:00:25.703139+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:25.703139+0000","last_undegraded":"2023-06-22T08:00:25.703139+0000","last_fullsized":"2023-06-22T08:00:25.703139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:33.819054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753735+0000","last_change":"2023-06-22T07:59:55.167979+0000","last_active":"2023-06-22T08:00:00.753735+0000","last_peered":"2023-06-22T08:00:00.753735+0000","last_clean":"2023-06-22T08:00:00.753735+0000","last_became_active":"2023-06-22T07:59:55.163775+0000","last_became_peered":"2023-06-22T07:59:55.163775+0000","last_unstale":"2023-06-22T08:00:00.753735+0000","last_undegraded":"2023-06-22T08:00:00.753735+0000","last_fullsized":"2023-06-22T08:00:00.753735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155135+0000","last_change":"2023-06-22T07:59:59.194581+0000","last_active":"2023-06-22T08:00:00.155135+0000","last_peered":"2023-06-22T08:00:00.155135+0000","last_clean":"2023-06-22T08:00:00.155135+0000","last_became_active":"2023-06-22T07:59:59.194135+0000","last_became_peered":"2023-06-22T07:59:59.194135+0000","last_unstale":"2023-06-22T08:00:00.155135+0000","last_undegraded":"2023-06-22T08:00:00.155135+0000","last_fullsized":"2023-06-22T08:00:00.155135+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"27'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.156863+0000","last_change":"2023-06-22T07:59:54.651138+0000","last_active":"2023-06-22T08:00:00.156863+0000","last_peered":"2023-06-22T08:00:00.156863+0000","last_clean":"2023-06-22T08:00:00.156863+0000","last_became_active":"2023-06-22T07:59:54.650803+0000","last_became_peered":"2023-06-22T07:59:54.650803+0000","last_unstale":"2023-06-22T08:00:00.156863+0000","last_undegraded":"2023-06-22T08:00:00.156863+0000","last_fullsized":"2023-06-22T08:00:00.156863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:45.982711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708617+0000","last_change":"2023-06-22T07:59:55.165814+0000","last_active":"2023-06-22T07:59:59.708617+0000","last_peered":"2023-06-22T07:59:59.708617+0000","last_clean":"2023-06-22T07:59:59.708617+0000","last_became_active":"2023-06-22T07:59:55.165270+0000","last_became_peered":"2023-06-22T07:59:55.165270+0000","last_unstale":"2023-06-22T07:59:59.708617+0000","last_undegraded":"2023-06-22T07:59:59.708617+0000","last_fullsized":"2023-06-22T07:59:59.708617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708465+0000","last_change":"2023-06-22T07:59:59.212042+0000","last_active":"2023-06-22T07:59:59.708465+0000","last_peered":"2023-06-22T07:59:59.708465+0000","last_clean":"2023-06-22T07:59:59.708465+0000","last_became_active":"2023-06-22T07:59:59.211666+0000","last_became_peered":"2023-06-22T07:59:59.211666+0000","last_unstale":"2023-06-22T07:59:59.708465+0000","last_undegraded":"2023-06-22T07:59:59.708465+0000","last_fullsized":"2023-06-22T07:59:59.708465+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,4],"acting":[2,7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158226+0000","last_change":"2023-06-22T07:59:54.650251+0000","last_active":"2023-06-22T08:00:00.158226+0000","last_peered":"2023-06-22T08:00:00.158226+0000","last_clean":"2023-06-22T08:00:00.158226+0000","last_became_active":"2023-06-22T07:59:54.649993+0000","last_became_peered":"2023-06-22T07:59:54.649993+0000","last_unstale":"2023-06-22T08:00:00.158226+0000","last_undegraded":"2023-06-22T08:00:00.158226+0000","last_fullsized":"2023-06-22T08:00:00.158226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:20:20.733896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708375+0000","last_change":"2023-06-22T07:59:55.172595+0000","last_active":"2023-06-22T07:59:59.708375+0000","last_peered":"2023-06-22T07:59:59.708375+0000","last_clean":"2023-06-22T07:59:59.708375+0000","last_became_active":"2023-06-22T07:59:55.165873+0000","last_became_peered":"2023-06-22T07:59:55.165873+0000","last_unstale":"2023-06-22T07:59:59.708375+0000","last_undegraded":"2023-06-22T07:59:59.708375+0000","last_fullsized":"2023-06-22T07:59:59.708375+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:02.592807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154969+0000","last_change":"2023-06-22T07:59:59.192708+0000","last_active":"2023-06-22T08:00:00.154969+0000","last_peered":"2023-06-22T08:00:00.154969+0000","last_clean":"2023-06-22T08:00:00.154969+0000","last_became_active":"2023-06-22T07:59:59.191693+0000","last_became_peered":"2023-06-22T07:59:59.191693+0000","last_unstale":"2023-06-22T08:00:00.154969+0000","last_undegraded":"2023-06-22T08:00:00.154969+0000","last_fullsized":"2023-06-22T08:00:00.154969+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T12:59:33.454962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5,7],"acting":[1,0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709255+0000","last_change":"2023-06-22T07:59:54.651179+0000","last_active":"2023-06-22T07:59:59.709255+0000","last_peered":"2023-06-22T07:59:59.709255+0000","last_clean":"2023-06-22T07:59:59.709255+0000","last_became_active":"2023-06-22T07:59:54.650852+0000","last_became_peered":"2023-06-22T07:59:54.650852+0000","last_unstale":"2023-06-22T07:59:59.709255+0000","last_undegraded":"2023-06-22T07:59:59.709255+0000","last_fullsized":"2023-06-22T07:59:59.709255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:01.351861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754043+0000","last_change":"2023-06-22T07:59:55.163050+0000","last_active":"2023-06-22T08:00:00.754043+0000","last_peered":"2023-06-22T08:00:00.754043+0000","last_clean":"2023-06-22T08:00:00.754043+0000","last_became_active":"2023-06-22T07:59:55.162754+0000","last_became_peered":"2023-06-22T07:59:55.162754+0000","last_unstale":"2023-06-22T08:00:00.754043+0000","last_undegraded":"2023-06-22T08:00:00.754043+0000","last_fullsized":"2023-06-22T08:00:00.754043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754797+0000","last_change":"2023-06-22T07:59:59.209558+0000","last_active":"2023-06-22T08:00:00.754797+0000","last_peered":"2023-06-22T08:00:00.754797+0000","last_clean":"2023-06-22T08:00:00.754797+0000","last_became_active":"2023-06-22T07:59:59.209129+0000","last_became_peered":"2023-06-22T07:59:59.209129+0000","last_unstale":"2023-06-22T08:00:00.754797+0000","last_undegraded":"2023-06-22T08:00:00.754797+0000","last_fullsized":"2023-06-22T08:00:00.754797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:21.581063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1,0],"acting":[4,3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154714+0000","last_change":"2023-06-22T07:59:54.652059+0000","last_active":"2023-06-22T08:00:00.154714+0000","last_peered":"2023-06-22T08:00:00.154714+0000","last_clean":"2023-06-22T08:00:00.154714+0000","last_became_active":"2023-06-22T07:59:54.651720+0000","last_became_peered":"2023-06-22T07:59:54.651720+0000","last_unstale":"2023-06-22T08:00:00.154714+0000","last_undegraded":"2023-06-22T08:00:00.154714+0000","last_fullsized":"2023-06-22T08:00:00.154714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:31.908657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708158+0000","last_change":"2023-06-22T07:59:55.172794+0000","last_active":"2023-06-22T07:59:59.708158+0000","last_peered":"2023-06-22T07:59:59.708158+0000","last_clean":"2023-06-22T07:59:59.708158+0000","last_became_active":"2023-06-22T07:59:55.172527+0000","last_became_peered":"2023-06-22T07:59:55.172527+0000","last_unstale":"2023-06-22T07:59:59.708158+0000","last_undegraded":"2023-06-22T07:59:59.708158+0000","last_fullsized":"2023-06-22T07:59:59.708158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155457+0000","last_change":"2023-06-22T07:59:55.165758+0000","last_active":"2023-06-22T08:00:00.155457+0000","last_peered":"2023-06-22T08:00:00.155457+0000","last_clean":"2023-06-22T08:00:00.155457+0000","last_became_active":"2023-06-22T07:59:55.165117+0000","last_became_peered":"2023-06-22T07:59:55.165117+0000","last_unstale":"2023-06-22T08:00:00.155457+0000","last_undegraded":"2023-06-22T08:00:00.155457+0000","last_fullsized":"2023-06-22T08:00:00.155457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:02.357853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753486+0000","last_change":"2023-06-22T07:59:55.162439+0000","last_active":"2023-06-22T08:00:00.753486+0000","last_peered":"2023-06-22T08:00:00.753486+0000","last_clean":"2023-06-22T08:00:00.753486+0000","last_became_active":"2023-06-22T07:59:55.162072+0000","last_became_peered":"2023-06-22T07:59:55.162072+0000","last_unstale":"2023-06-22T08:00:00.753486+0000","last_undegraded":"2023-06-22T08:00:00.753486+0000","last_fullsized":"2023-06-22T08:00:00.753486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157229+0000","last_change":"2023-06-22T07:59:55.159299+0000","last_active":"2023-06-22T08:00:00.157229+0000","last_peered":"2023-06-22T08:00:00.157229+0000","last_clean":"2023-06-22T08:00:00.157229+0000","last_became_active":"2023-06-22T07:59:55.158893+0000","last_became_peered":"2023-06-22T07:59:55.158893+0000","last_unstale":"2023-06-22T08:00:00.157229+0000","last_undegraded":"2023-06-22T08:00:00.157229+0000","last_fullsized":"2023-06-22T08:00:00.157229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709716+0000","last_change":"2023-06-22T07:59:55.162520+0000","last_active":"2023-06-22T07:59:59.709716+0000","last_peered":"2023-06-22T07:59:59.709716+0000","last_clean":"2023-06-22T07:59:59.709716+0000","last_became_active":"2023-06-22T07:59:55.162211+0000","last_became_peered":"2023-06-22T07:59:55.162211+0000","last_unstale":"2023-06-22T07:59:59.709716+0000","last_undegraded":"2023-06-22T07:59:59.709716+0000","last_fullsized":"2023-06-22T07:59:59.709716+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:13:00.662547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708947+0000","last_change":"2023-06-22T07:59:55.165364+0000","last_active":"2023-06-22T07:59:59.708947+0000","last_peered":"2023-06-22T07:59:59.708947+0000","last_clean":"2023-06-22T07:59:59.708947+0000","last_became_active":"2023-06-22T07:59:55.165021+0000","last_became_peered":"2023-06-22T07:59:55.165021+0000","last_unstale":"2023-06-22T07:59:59.708947+0000","last_undegraded":"2023-06-22T07:59:59.708947+0000","last_fullsized":"2023-06-22T07:59:59.708947+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:09:07.008945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5950,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":11900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476749,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":315392,"data_stored":127374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]}]},{"osd":3,"up_from":14,"seq":60129542157,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":327680,"data_stored":127636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":124222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.4989999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.173999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.486}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5129999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":905216,"data_stored":714600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":901120,"data_stored":714598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":124252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":299008,"data_stored":123558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":316,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":323584,"data_stored":124860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-22T08:00:29.895 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-06-22T08:00:29.895 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-06-22T08:00:29.895 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-06-22T08:00:29.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-06-22T08:00:30.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.121+0000 7f51e81ff700 1 Processor -- start 2023-06-22T08:00:30.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.121+0000 7f51e81ff700 1 -- start start 2023-06-22T08:00:30.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.122+0000 7f51e81ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f51e015be00 0x7f51e015c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:30.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.122+0000 7f51e81ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e0159700 con 0x7f51e0152dc0 2023-06-22T08:00:30.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.122+0000 7f51e81ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e0158af0 con 0x7f51e00ca120 2023-06-22T08:00:30.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.122+0000 7f51e81ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e015cde0 con 0x7f51e015be00 2023-06-22T08:00:30.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.123+0000 7f51e5f9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f51e0152dc0 0x7f51e00c9230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41432/0 (socket says 172.21.15.23:41432) 2023-06-22T08:00:30.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.123+0000 7f51e5f9b700 1 -- 172.21.15.23:0/3585354411 learned_addr learned my addr 172.21.15.23:0/3585354411 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:30.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.123+0000 7f51e679c700 1 --2- 172.21.15.23:0/3585354411 >> v2:172.21.15.67:3300/0 conn(0x7f51e015be00 0x7f51e015c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:30.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.123+0000 7f51e4f99700 1 -- 172.21.15.23:0/3585354411 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823838153 0 0) 0x7f51e0159700 con 0x7f51e0152dc0 2023-06-22T08:00:30.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.123+0000 7f51e4f99700 1 -- 172.21.15.23:0/3585354411 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51cc0035b0 con 0x7f51e0152dc0 2023-06-22T08:00:30.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.123+0000 7f51e679c700 1 -- 172.21.15.23:0/3585354411 >> v1:172.21.15.67:6789/0 conn(0x7f51e00ca120 legacy=0x7f51e00ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:30.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.123+0000 7f51e679c700 1 -- 172.21.15.23:0/3585354411 >> v1:172.21.15.23:6789/0 conn(0x7f51e0152dc0 legacy=0x7f51e00c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:30.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.123+0000 7f51e679c700 1 -- 172.21.15.23:0/3585354411 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e0158dd0 con 0x7f51e015be00 2023-06-22T08:00:30.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.123+0000 7f51e679c700 1 --2- 172.21.15.23:0/3585354411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e015be00 0x7f51e015c220 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009d90 tx=0x7f51d800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f29241daf61434b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:30.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.124+0000 7f51e4f99700 1 -- 172.21.15.23:0/3585354411 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d800e070 con 0x7f51e015be00 2023-06-22T08:00:30.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.124+0000 7f51e4f99700 1 -- 172.21.15.23:0/3585354411 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d800bb20 con 0x7f51e015be00 2023-06-22T08:00:30.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.124+0000 7f51e4f99700 1 -- 172.21.15.23:0/3585354411 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d8012430 con 0x7f51e015be00 2023-06-22T08:00:30.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.124+0000 7f51e81ff700 1 -- 172.21.15.23:0/3585354411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e015be00 msgr2=0x7f51e015c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:30.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.124+0000 7f51e81ff700 1 --2- 172.21.15.23:0/3585354411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e015be00 0x7f51e015c220 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009d90 tx=0x7f51d800b330 comp rx=0 tx=0).stop 2023-06-22T08:00:30.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.124+0000 7f51e81ff700 1 -- 172.21.15.23:0/3585354411 shutdown_connections 2023-06-22T08:00:30.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.124+0000 7f51e81ff700 1 --2- 172.21.15.23:0/3585354411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e015be00 0x7f51e015c220 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:30.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.124+0000 7f51e81ff700 1 -- 172.21.15.23:0/3585354411 >> 172.21.15.23:0/3585354411 conn(0x7f51e006ae30 msgr2=0x7f51e006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:30.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.124+0000 7f51e81ff700 1 -- 172.21.15.23:0/3585354411 shutdown_connections 2023-06-22T08:00:30.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.124+0000 7f51e81ff700 1 -- 172.21.15.23:0/3585354411 wait complete. 2023-06-22T08:00:30.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.126+0000 7f51e81ff700 1 Processor -- start 2023-06-22T08:00:30.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.126+0000 7f51e81ff700 1 -- start start 2023-06-22T08:00:30.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e81ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51e00ca120 0x7f51e01f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:30.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e81ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e0152dc0 0x7f51e00b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:30.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e81ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f51e015be00 0x7f51e01e5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:30.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e81ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e01e3930 con 0x7f51e00ca120 2023-06-22T08:00:30.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e81ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e00b3ee0 con 0x7f51e0152dc0 2023-06-22T08:00:30.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e81ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e01e4aa0 con 0x7f51e015be00 2023-06-22T08:00:30.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e5f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51e00ca120 0x7f51e01f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:30.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e579a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e0152dc0 0x7f51e00b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:30.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e5f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51e00ca120 0x7f51e01f1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60992/0 (socket says 172.21.15.23:60992) 2023-06-22T08:00:30.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e579a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e0152dc0 0x7f51e00b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:48138/0 (socket says 172.21.15.23:48138) 2023-06-22T08:00:30.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e5f9b700 1 -- 172.21.15.23:0/1326718251 learned_addr learned my addr 172.21.15.23:0/1326718251 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:30.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.127+0000 7f51e679c700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f51e015be00 0x7f51e01e5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:30.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.128+0000 7f51e5f9b700 1 -- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f51e015be00 msgr2=0x7f51e01e5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:30.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.128+0000 7f51e5f9b700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f51e015be00 0x7f51e01e5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:30.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.128+0000 7f51e5f9b700 1 -- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e0152dc0 msgr2=0x7f51e00b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:30.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.128+0000 7f51e5f9b700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e0152dc0 0x7f51e00b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:30.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.128+0000 7f51e5f9b700 1 -- 172.21.15.23:0/1326718251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d8009a70 con 0x7f51e00ca120 2023-06-22T08:00:30.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.128+0000 7f51e5f9b700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51e00ca120 0x7f51e01f1b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00c700 tx=0x7f51dc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:30.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.128+0000 7f51d67fc700 1 -- 172.21.15.23:0/1326718251 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc005d70 con 0x7f51e00ca120 2023-06-22T08:00:30.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.128+0000 7f51d67fc700 1 -- 172.21.15.23:0/1326718251 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc0070d0 con 0x7f51e00ca120 2023-06-22T08:00:30.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.128+0000 7f51d67fc700 1 -- 172.21.15.23:0/1326718251 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc007460 con 0x7f51e00ca120 2023-06-22T08:00:30.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.128+0000 7f51e81ff700 1 -- 172.21.15.23:0/1326718251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e01f2060 con 0x7f51e00ca120 2023-06-22T08:00:30.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.129+0000 7f51e81ff700 1 -- 172.21.15.23:0/1326718251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e00b3890 con 0x7f51e00ca120 2023-06-22T08:00:30.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.130+0000 7f51e81ff700 1 -- 172.21.15.23:0/1326718251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e00b41c0 con 0x7f51e00ca120 2023-06-22T08:00:30.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.130+0000 7f51d67fc700 1 -- 172.21.15.23:0/1326718251 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f51dc0075c0 con 0x7f51e00ca120 2023-06-22T08:00:30.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.131+0000 7f51d67fc700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f51c406ace0 0x7f51c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:30.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.132+0000 7f51d67fc700 1 -- 172.21.15.23:0/1326718251 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f51dc088bd0 con 0x7f51e00ca120 2023-06-22T08:00:30.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.132+0000 7f51e579a700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f51c406ace0 0x7f51c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:30.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.132+0000 7f51e579a700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f51c406ace0 0x7f51c406d190 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f51e00c93a0 tx=0x7f51d0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:30.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.136+0000 7f51d67fc700 1 -- 172.21.15.23:0/1326718251 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f51dc053980 con 0x7f51e00ca120 2023-06-22T08:00:30.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.387+0000 7f51e81ff700 1 -- 172.21.15.23:0/1326718251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f51e004ea20 con 0x7f51e00ca120 2023-06-22T08:00:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.388+0000 7f51d67fc700 1 -- 172.21.15.23:0/1326718251 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f51dc0170e0 con 0x7f51e00ca120 2023-06-22T08:00:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:30.391 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-06-22T08:00:30.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 -- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f51c406ace0 msgr2=0x7f51c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:30.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f51c406ace0 0x7f51c406d190 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f51e00c93a0 tx=0x7f51d0009040 comp rx=0 tx=0).stop 2023-06-22T08:00:30.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 -- 172.21.15.23:0/1326718251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51e00ca120 msgr2=0x7f51e01f1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:30.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51e00ca120 0x7f51e01f1b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00c700 tx=0x7f51dc00ca10 comp rx=0 tx=0).stop 2023-06-22T08:00:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 -- 172.21.15.23:0/1326718251 shutdown_connections 2023-06-22T08:00:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f51e015be00 0x7f51e01e5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51e00ca120 0x7f51e01f1b20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f51c406ace0 0x7f51c406d190 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 --2- 172.21.15.23:0/1326718251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51e0152dc0 0x7f51e00b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 -- 172.21.15.23:0/1326718251 >> 172.21.15.23:0/1326718251 conn(0x7f51e006ae30 msgr2=0x7f51e00c4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 -- 172.21.15.23:0/1326718251 shutdown_connections 2023-06-22T08:00:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:30.390+0000 7f51e81ff700 1 -- 172.21.15.23:0/1326718251 wait complete. 2023-06-22T08:00:30.597 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:00:30.593+0000 7fa5334b2700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-06-22T08:00:33.411 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-06-22T08:00:33.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.617+0000 7f28d11ff700 1 Processor -- start 2023-06-22T08:00:33.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.617+0000 7f28d11ff700 1 -- start start 2023-06-22T08:00:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28d11ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc152dc0 0x7f28cc0c9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28d11ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f28cc0c9770 0x7f28cc0ca120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28d11ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28cc159700 con 0x7f28cc0ca660 2023-06-22T08:00:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28d11ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28cc15cde0 con 0x7f28cc0c9770 2023-06-22T08:00:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28d11ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc1588e0 con 0x7f28cc152dc0 2023-06-22T08:00:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28ca59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f28cc0c9770 0x7f28cc0ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28ca59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f28cc0c9770 0x7f28cc0ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60994/0 (socket says 172.21.15.23:60994) 2023-06-22T08:00:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28ca59c700 1 -- 172.21.15.23:0/316084531 learned_addr learned my addr 172.21.15.23:0/316084531 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28cad9d700 1 --2- 172.21.15.23:0/316084531 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc152dc0 0x7f28cc0c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28ca59c700 1 -- 172.21.15.23:0/316084531 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc152dc0 msgr2=0x7f28cc0c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28ca59c700 1 --2- 172.21.15.23:0/316084531 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc152dc0 0x7f28cc0c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28ca59c700 1 -- 172.21.15.23:0/316084531 >> v1:172.21.15.67:6789/0 conn(0x7f28cc0ca660 legacy=0x7f28cc15be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.619+0000 7f28ca59c700 1 -- 172.21.15.23:0/316084531 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28cc158af0 con 0x7f28cc0c9770 2023-06-22T08:00:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.620+0000 7f28ca59c700 1 --2- 172.21.15.23:0/316084531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28cc0c9770 0x7f28cc0ca120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f28c00098f0 tx=0x7f28c0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49eab2b9774de88a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.620+0000 7f28c9d9b700 1 -- 172.21.15.23:0/316084531 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0011830 con 0x7f28cc0c9770 2023-06-22T08:00:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.620+0000 7f28c9d9b700 1 -- 172.21.15.23:0/316084531 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c0011990 con 0x7f28cc0c9770 2023-06-22T08:00:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.620+0000 7f28c9d9b700 1 -- 172.21.15.23:0/316084531 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0011d20 con 0x7f28cc0c9770 2023-06-22T08:00:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.620+0000 7f28d11ff700 1 -- 172.21.15.23:0/316084531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28cc0c9770 msgr2=0x7f28cc0ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.620+0000 7f28d11ff700 1 --2- 172.21.15.23:0/316084531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28cc0c9770 0x7f28cc0ca120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f28c00098f0 tx=0x7f28c0009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:33.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.620+0000 7f28d11ff700 1 -- 172.21.15.23:0/316084531 shutdown_connections 2023-06-22T08:00:33.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.620+0000 7f28d11ff700 1 --2- 172.21.15.23:0/316084531 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc152dc0 0x7f28cc0c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:33.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.620+0000 7f28d11ff700 1 --2- 172.21.15.23:0/316084531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28cc0c9770 0x7f28cc0ca120 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:33.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.621+0000 7f28d11ff700 1 -- 172.21.15.23:0/316084531 >> 172.21.15.23:0/316084531 conn(0x7f28cc06ae30 msgr2=0x7f28cc06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:33.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.621+0000 7f28d11ff700 1 -- 172.21.15.23:0/316084531 shutdown_connections 2023-06-22T08:00:33.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.621+0000 7f28d11ff700 1 -- 172.21.15.23:0/316084531 wait complete. 2023-06-22T08:00:33.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.622+0000 7f28d11ff700 1 Processor -- start 2023-06-22T08:00:33.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.622+0000 7f28d11ff700 1 -- start start 2023-06-22T08:00:33.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.623+0000 7f28d11ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc0c9770 0x7f28cc1e32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:33.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.623+0000 7f28d11ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc0ca660 0x7f28cc14e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:33.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.623+0000 7f28d11ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28cc152dc0 0x7f28cc14e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:33.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.623+0000 7f28d11ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc15c660 con 0x7f28cc152dc0 2023-06-22T08:00:33.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.623+0000 7f28d11ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc14df40 con 0x7f28cc0c9770 2023-06-22T08:00:33.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.623+0000 7f28d11ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28cc15c380 con 0x7f28cc0ca660 2023-06-22T08:00:33.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.623+0000 7f28cad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc0c9770 0x7f28cc1e32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:33.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.623+0000 7f28cb59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28cc152dc0 0x7f28cc14e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:33.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28ca59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc0ca660 0x7f28cc14e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:33.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28cad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc0c9770 0x7f28cc1e32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:48154/0 (socket says 172.21.15.23:48154) 2023-06-22T08:00:33.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28cad9d700 1 -- 172.21.15.23:0/710962581 learned_addr learned my addr 172.21.15.23:0/710962581 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:33.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28ca59c700 1 -- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc0c9770 msgr2=0x7f28cc1e32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28ca59c700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc0c9770 0x7f28cc1e32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28ca59c700 1 -- 172.21.15.23:0/710962581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28cc152dc0 msgr2=0x7f28cc14e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28ca59c700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28cc152dc0 0x7f28cc14e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28ca59c700 1 -- 172.21.15.23:0/710962581 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c0009550 con 0x7f28cc0ca660 2023-06-22T08:00:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28cb59e700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28cc152dc0 0x7f28cc14e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28ca59c700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc0ca660 0x7f28cc14e3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f28c0016000 tx=0x7f28c0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.624+0000 7f28b37fe700 1 -- 172.21.15.23:0/710962581 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0004400 con 0x7f28cc0ca660 2023-06-22T08:00:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.625+0000 7f28b37fe700 1 -- 172.21.15.23:0/710962581 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c001f070 con 0x7f28cc0ca660 2023-06-22T08:00:33.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.625+0000 7f28d11ff700 1 -- 172.21.15.23:0/710962581 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28cc159be0 con 0x7f28cc0ca660 2023-06-22T08:00:33.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.625+0000 7f28b37fe700 1 -- 172.21.15.23:0/710962581 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c00038e0 con 0x7f28cc0ca660 2023-06-22T08:00:33.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.625+0000 7f28d11ff700 1 -- 172.21.15.23:0/710962581 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28cc15a1a0 con 0x7f28cc0ca660 2023-06-22T08:00:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.626+0000 7f28d11ff700 1 -- 172.21.15.23:0/710962581 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28cc04ea20 con 0x7f28cc0ca660 2023-06-22T08:00:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.627+0000 7f28b37fe700 1 -- 172.21.15.23:0/710962581 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f28c0021030 con 0x7f28cc0ca660 2023-06-22T08:00:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.628+0000 7f28b37fe700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f28b806ace0 0x7f28b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.628+0000 7f28b37fe700 1 -- 172.21.15.23:0/710962581 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f28c008a070 con 0x7f28cc0ca660 2023-06-22T08:00:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.628+0000 7f28cad9d700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f28b806ace0 0x7f28b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.629+0000 7f28cad9d700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f28b806ace0 0x7f28b806d190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009c00 tx=0x7f28bc009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:33.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.632+0000 7f28b37fe700 1 -- 172.21.15.23:0/710962581 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f28c0054d10 con 0x7f28cc0ca660 2023-06-22T08:00:33.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.882+0000 7f28d11ff700 1 -- 172.21.15.23:0/710962581 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f28cc06ac10 con 0x7f28cc0ca660 2023-06-22T08:00:33.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.883+0000 7f28b37fe700 1 -- 172.21.15.23:0/710962581 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7f28c0058330 con 0x7f28cc0ca660 2023-06-22T08:00:33.886 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:33.886 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-06-22T08:00:33.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 -- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f28b806ace0 msgr2=0x7f28b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:33.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f28b806ace0 0x7f28b806d190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009c00 tx=0x7f28bc009210 comp rx=0 tx=0).stop 2023-06-22T08:00:33.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 -- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc0ca660 msgr2=0x7f28cc14e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:33.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc0ca660 0x7f28cc14e3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f28c0016000 tx=0x7f28c0004750 comp rx=0 tx=0).stop 2023-06-22T08:00:33.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 -- 172.21.15.23:0/710962581 shutdown_connections 2023-06-22T08:00:33.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28cc152dc0 0x7f28cc14e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:33.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f28b806ace0 0x7f28b806d190 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:33.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28cc0c9770 0x7f28cc1e32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:33.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 --2- 172.21.15.23:0/710962581 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28cc0ca660 0x7f28cc14e3f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:33.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 -- 172.21.15.23:0/710962581 >> 172.21.15.23:0/710962581 conn(0x7f28cc06ae30 msgr2=0x7f28cc1501d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:33.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.885+0000 7f28d11ff700 1 -- 172.21.15.23:0/710962581 shutdown_connections 2023-06-22T08:00:33.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:33.886+0000 7f28d11ff700 1 -- 172.21.15.23:0/710962581 wait complete. 2023-06-22T08:00:35.597 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:00:35.594+0000 7fa5334b2700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-06-22T08:00:35.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:00:35.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:00:36.906 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-06-22T08:00:37.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.112+0000 7fa4c73ff700 1 Processor -- start 2023-06-22T08:00:37.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.112+0000 7fa4c73ff700 1 -- start start 2023-06-22T08:00:37.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.113+0000 7fa4c73ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4c006ab30 0x7fa4c0069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.113+0000 7fa4c73ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa4c006af00 0x7fa4c0071410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.113+0000 7fa4c73ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c00c5ee0 con 0x7fa4c00c5260 2023-06-22T08:00:37.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.113+0000 7fa4c73ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c00c66f0 con 0x7fa4c006af00 2023-06-22T08:00:37.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.113+0000 7fa4c73ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c0070f80 con 0x7fa4c006ab30 2023-06-22T08:00:37.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.114+0000 7fa4c499a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa4c006af00 0x7fa4c0071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.114+0000 7fa4c519b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4c006ab30 0x7fa4c0069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.114+0000 7fa4c499a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa4c006af00 0x7fa4c0071410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33256/0 (socket says 172.21.15.23:33256) 2023-06-22T08:00:37.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.114+0000 7fa4c499a700 1 -- 172.21.15.23:0/4120919179 learned_addr learned my addr 172.21.15.23:0/4120919179 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:37.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.114+0000 7fa4c499a700 1 -- 172.21.15.23:0/4120919179 >> v2:172.21.15.67:3300/0 conn(0x7fa4c006ab30 msgr2=0x7fa4c0069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:37.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.114+0000 7fa4c499a700 1 --2- 172.21.15.23:0/4120919179 >> v2:172.21.15.67:3300/0 conn(0x7fa4c006ab30 0x7fa4c0069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.114+0000 7fa4c499a700 1 -- 172.21.15.23:0/4120919179 >> v1:172.21.15.23:6789/0 conn(0x7fa4c00c5260 legacy=0x7fa4c0152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.114+0000 7fa4c499a700 1 -- 172.21.15.23:0/4120919179 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c015cc10 con 0x7fa4c006af00 2023-06-22T08:00:37.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.114+0000 7fa4c499a700 1 --2- 172.21.15.23:0/4120919179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 0x7fa4c0071410 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00700e0 tx=0x7fa4b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f281595a4a27162 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:37.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.114+0000 7fa4b7fff700 1 -- 172.21.15.23:0/4120919179 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0011830 con 0x7fa4c006af00 2023-06-22T08:00:37.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.115+0000 7fa4b7fff700 1 -- 172.21.15.23:0/4120919179 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b0011990 con 0x7fa4c006af00 2023-06-22T08:00:37.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.115+0000 7fa4b7fff700 1 -- 172.21.15.23:0/4120919179 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0011d20 con 0x7fa4c006af00 2023-06-22T08:00:37.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.115+0000 7fa4c73ff700 1 -- 172.21.15.23:0/4120919179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 msgr2=0x7fa4c0071410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.115+0000 7fa4c73ff700 1 --2- 172.21.15.23:0/4120919179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 0x7fa4c0071410 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00700e0 tx=0x7fa4b0009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.115+0000 7fa4c73ff700 1 -- 172.21.15.23:0/4120919179 shutdown_connections 2023-06-22T08:00:37.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.115+0000 7fa4c73ff700 1 --2- 172.21.15.23:0/4120919179 >> v2:172.21.15.67:3300/0 conn(0x7fa4c006ab30 0x7fa4c0069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.115+0000 7fa4c73ff700 1 --2- 172.21.15.23:0/4120919179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 0x7fa4c0071410 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.115+0000 7fa4c73ff700 1 -- 172.21.15.23:0/4120919179 >> 172.21.15.23:0/4120919179 conn(0x7fa4c006e940 msgr2=0x7fa4c006ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:37.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.115+0000 7fa4c73ff700 1 -- 172.21.15.23:0/4120919179 shutdown_connections 2023-06-22T08:00:37.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.115+0000 7fa4c73ff700 1 -- 172.21.15.23:0/4120919179 wait complete. 2023-06-22T08:00:37.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.117+0000 7fa4c73ff700 1 Processor -- start 2023-06-22T08:00:37.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.117+0000 7fa4c73ff700 1 -- start start 2023-06-22T08:00:37.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c73ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4c006ab30 0x7fa4c0157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c73ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 0x7fa4c0156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c73ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa4c00c5260 0x7fa4c00c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c73ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00ca4b0 con 0x7fa4c006af00 2023-06-22T08:00:37.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c73ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c01f20f0 con 0x7fa4c006ab30 2023-06-22T08:00:37.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c73ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c01f25e0 con 0x7fa4c00c5260 2023-06-22T08:00:37.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c499a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 0x7fa4c0156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c499a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 0x7fa4c0156720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33258/0 (socket says 172.21.15.23:33258) 2023-06-22T08:00:37.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c499a700 1 -- 172.21.15.23:0/1977237463 learned_addr learned my addr 172.21.15.23:0/1977237463 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:37.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c519b700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4c006ab30 0x7fa4c0157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.118+0000 7fa4c599c700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa4c00c5260 0x7fa4c00c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.119+0000 7fa4c499a700 1 -- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa4c00c5260 msgr2=0x7fa4c00c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.119+0000 7fa4c499a700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa4c00c5260 0x7fa4c00c48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.119+0000 7fa4c499a700 1 -- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4c006ab30 msgr2=0x7fa4c0157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.119+0000 7fa4c499a700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4c006ab30 0x7fa4c0157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.119+0000 7fa4c499a700 1 -- 172.21.15.23:0/1977237463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0009550 con 0x7fa4c006af00 2023-06-22T08:00:37.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.119+0000 7fa4c499a700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 0x7fa4c0156720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009c10 tx=0x7fa4b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:37.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.119+0000 7fa4b5ffb700 1 -- 172.21.15.23:0/1977237463 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0004510 con 0x7fa4c006af00 2023-06-22T08:00:37.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.119+0000 7fa4b5ffb700 1 -- 172.21.15.23:0/1977237463 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b001e070 con 0x7fa4c006af00 2023-06-22T08:00:37.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.119+0000 7fa4c73ff700 1 -- 172.21.15.23:0/1977237463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c01ee310 con 0x7fa4c006af00 2023-06-22T08:00:37.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.119+0000 7fa4b5ffb700 1 -- 172.21.15.23:0/1977237463 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b00038e0 con 0x7fa4c006af00 2023-06-22T08:00:37.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.120+0000 7fa4c73ff700 1 -- 172.21.15.23:0/1977237463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c01f2300 con 0x7fa4c006af00 2023-06-22T08:00:37.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.121+0000 7fa4c73ff700 1 -- 172.21.15.23:0/1977237463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c01ee170 con 0x7fa4c006af00 2023-06-22T08:00:37.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.122+0000 7fa4b5ffb700 1 -- 172.21.15.23:0/1977237463 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa4b0003a40 con 0x7fa4c006af00 2023-06-22T08:00:37.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.123+0000 7fa4b5ffb700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4a406ace0 0x7fa4a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.123+0000 7fa4b5ffb700 1 -- 172.21.15.23:0/1977237463 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa4b008a4e0 con 0x7fa4c006af00 2023-06-22T08:00:37.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.123+0000 7fa4c519b700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4a406ace0 0x7fa4a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.123+0000 7fa4c519b700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4a406ace0 0x7fa4a406d190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009b40 tx=0x7fa4bc009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:37.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.127+0000 7fa4b5ffb700 1 -- 172.21.15.23:0/1977237463 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa4c01ee170 con 0x7fa4c006af00 2023-06-22T08:00:37.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.378+0000 7fa4c73ff700 1 -- 172.21.15.23:0/1977237463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa4c01ee170 con 0x7fa4c006af00 2023-06-22T08:00:37.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.379+0000 7fa4b5ffb700 1 -- 172.21.15.23:0/1977237463 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa4c01ee170 con 0x7fa4c006af00 2023-06-22T08:00:37.382 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:37.382 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-06-22T08:00:37.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 -- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4a406ace0 msgr2=0x7fa4a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4a406ace0 0x7fa4a406d190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009b40 tx=0x7fa4bc009210 comp rx=0 tx=0).stop 2023-06-22T08:00:37.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 -- 172.21.15.23:0/1977237463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 msgr2=0x7fa4c0156720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 0x7fa4c0156720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009c10 tx=0x7fa4b0004750 comp rx=0 tx=0).stop 2023-06-22T08:00:37.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 -- 172.21.15.23:0/1977237463 shutdown_connections 2023-06-22T08:00:37.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa4c00c5260 0x7fa4c00c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4a406ace0 0x7fa4a406d190 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4c006ab30 0x7fa4c0157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 --2- 172.21.15.23:0/1977237463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c006af00 0x7fa4c0156720 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 -- 172.21.15.23:0/1977237463 >> 172.21.15.23:0/1977237463 conn(0x7fa4c006e940 msgr2=0x7fa4c00c7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:37.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 -- 172.21.15.23:0/1977237463 shutdown_connections 2023-06-22T08:00:37.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.381+0000 7fa4c73ff700 1 -- 172.21.15.23:0/1977237463 wait complete. 2023-06-22T08:00:37.402 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-06-22T08:00:37.403 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:37.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.599+0000 7f104f7ff700 1 Processor -- start 2023-06-22T08:00:37.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.600+0000 7f104f7ff700 1 -- start start 2023-06-22T08:00:37.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.601+0000 7f104f7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10480c9fd0 0x7f10480cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.601+0000 7f104f7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f10480c40b0 0x7f10480c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.601+0000 7f104f7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f104815c370 con 0x7f1048152ce0 2023-06-22T08:00:37.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.601+0000 7f104f7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10481530b0 con 0x7f10480c40b0 2023-06-22T08:00:37.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.601+0000 7f104f7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10480cf6f0 con 0x7f10480c9fd0 2023-06-22T08:00:37.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.601+0000 7f104dd9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f10480c40b0 0x7f10480c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.601+0000 7f104dd9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f10480c40b0 0x7f10480c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33260/0 (socket says 172.21.15.23:33260) 2023-06-22T08:00:37.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.601+0000 7f104dd9c700 1 -- 172.21.15.23:0/3053745989 learned_addr learned my addr 172.21.15.23:0/3053745989 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:37.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.601+0000 7f104cd9a700 1 --2- 172.21.15.23:0/3053745989 >> v2:172.21.15.67:3300/0 conn(0x7f10480c9fd0 0x7f10480cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.602+0000 7f104dd9c700 1 -- 172.21.15.23:0/3053745989 >> v2:172.21.15.67:3300/0 conn(0x7f10480c9fd0 msgr2=0x7f10480cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:37.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.602+0000 7f104dd9c700 1 --2- 172.21.15.23:0/3053745989 >> v2:172.21.15.67:3300/0 conn(0x7f10480c9fd0 0x7f10480cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.602+0000 7f104dd9c700 1 -- 172.21.15.23:0/3053745989 >> v1:172.21.15.67:6789/0 conn(0x7f1048152ce0 legacy=0x7f104815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.602+0000 7f104dd9c700 1 -- 172.21.15.23:0/3053745989 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104815bda0 con 0x7f10480c40b0 2023-06-22T08:00:37.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.602+0000 7f104dd9c700 1 --2- 172.21.15.23:0/3053745989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480c40b0 0x7f10480c4530 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f10480cc380 tx=0x7f104000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf75f2cf535bc3c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:37.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.602+0000 7f103ffff700 1 -- 172.21.15.23:0/3053745989 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104000e070 con 0x7f10480c40b0 2023-06-22T08:00:37.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.602+0000 7f103ffff700 1 -- 172.21.15.23:0/3053745989 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104000b910 con 0x7f10480c40b0 2023-06-22T08:00:37.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.602+0000 7f103ffff700 1 -- 172.21.15.23:0/3053745989 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104000bcc0 con 0x7f10480c40b0 2023-06-22T08:00:37.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.603+0000 7f104f7ff700 1 -- 172.21.15.23:0/3053745989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480c40b0 msgr2=0x7f10480c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.603+0000 7f104f7ff700 1 --2- 172.21.15.23:0/3053745989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480c40b0 0x7f10480c4530 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f10480cc380 tx=0x7f104000b330 comp rx=0 tx=0).stop 2023-06-22T08:00:37.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.603+0000 7f104f7ff700 1 -- 172.21.15.23:0/3053745989 shutdown_connections 2023-06-22T08:00:37.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.603+0000 7f104f7ff700 1 --2- 172.21.15.23:0/3053745989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480c40b0 0x7f10480c4530 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.603+0000 7f104f7ff700 1 --2- 172.21.15.23:0/3053745989 >> v2:172.21.15.67:3300/0 conn(0x7f10480c9fd0 0x7f10480cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.603+0000 7f104f7ff700 1 -- 172.21.15.23:0/3053745989 >> 172.21.15.23:0/3053745989 conn(0x7f104806ae30 msgr2=0x7f104806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:37.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.603+0000 7f104f7ff700 1 -- 172.21.15.23:0/3053745989 shutdown_connections 2023-06-22T08:00:37.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.603+0000 7f104f7ff700 1 -- 172.21.15.23:0/3053745989 wait complete. 2023-06-22T08:00:37.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.604+0000 7f104f7ff700 1 Processor -- start 2023-06-22T08:00:37.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.605+0000 7f104f7ff700 1 -- start start 2023-06-22T08:00:37.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104f7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480c40b0 0x7f10481e3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104f7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10480c9fd0 0x7f10481e3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104f7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1048152ce0 0x7f10481ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104f7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10480cd8a0 con 0x7f10480c40b0 2023-06-22T08:00:37.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104f7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10480ce3e0 con 0x7f1048152ce0 2023-06-22T08:00:37.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104f7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10481eee00 con 0x7f10480c9fd0 2023-06-22T08:00:37.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104d59b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480c40b0 0x7f10481e3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104cd9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10480c9fd0 0x7f10481e3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104cd9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10480c9fd0 0x7f10481e3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:51136/0 (socket says 172.21.15.23:51136) 2023-06-22T08:00:37.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104cd9a700 1 -- 172.21.15.23:0/3550476440 learned_addr learned my addr 172.21.15.23:0/3550476440 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:37.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104dd9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1048152ce0 0x7f10481ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104d59b700 1 -- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10480c9fd0 msgr2=0x7f10481e3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104d59b700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10480c9fd0 0x7f10481e3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104d59b700 1 -- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1048152ce0 msgr2=0x7f10481ee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104d59b700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1048152ce0 0x7f10481ee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104d59b700 1 -- 172.21.15.23:0/3550476440 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1040009b30 con 0x7f10480c40b0 2023-06-22T08:00:37.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104cd9a700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10480c9fd0 0x7f10481e3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:37.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.606+0000 7f104dd9c700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1048152ce0 0x7f10481ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:37.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.607+0000 7f104d59b700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480c40b0 0x7f10481e3970 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1044000f60 tx=0x7f104400c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:37.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.607+0000 7f103dffb700 1 -- 172.21.15.23:0/3550476440 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10440052d0 con 0x7f10480c40b0 2023-06-22T08:00:37.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.607+0000 7f103dffb700 1 -- 172.21.15.23:0/3550476440 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1044005430 con 0x7f10480c40b0 2023-06-22T08:00:37.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.607+0000 7f104f7ff700 1 -- 172.21.15.23:0/3550476440 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10480cdb40 con 0x7f10480c40b0 2023-06-22T08:00:37.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.607+0000 7f103dffb700 1 -- 172.21.15.23:0/3550476440 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10440038c0 con 0x7f10480c40b0 2023-06-22T08:00:37.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.607+0000 7f104f7ff700 1 -- 172.21.15.23:0/3550476440 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10480ce100 con 0x7f10480c40b0 2023-06-22T08:00:37.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.608+0000 7f104f7ff700 1 -- 172.21.15.23:0/3550476440 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104804ea20 con 0x7f10480c40b0 2023-06-22T08:00:37.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.609+0000 7f103dffb700 1 -- 172.21.15.23:0/3550476440 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1044003c30 con 0x7f10480c40b0 2023-06-22T08:00:37.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.610+0000 7f103dffb700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f102c06ace0 0x7f102c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:37.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.610+0000 7f103dffb700 1 -- 172.21.15.23:0/3550476440 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1044088d40 con 0x7f10480c40b0 2023-06-22T08:00:37.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.610+0000 7f104cd9a700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f102c06ace0 0x7f102c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:37.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.611+0000 7f104cd9a700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f102c06ace0 0x7f102c06d190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f104814ccb0 tx=0x7f1038009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:37.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.614+0000 7f103dffb700 1 -- 172.21.15.23:0/3550476440 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f10440539d0 con 0x7f10480c40b0 2023-06-22T08:00:37.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.823+0000 7f104f7ff700 1 -- 172.21.15.23:0/3550476440 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10480c9db0 con 0x7f10480c40b0 2023-06-22T08:00:37.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.824+0000 7f103dffb700 1 -- 172.21.15.23:0/3550476440 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f1044017020 con 0x7f10480c40b0 2023-06-22T08:00:37.828 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:37.828 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:37.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 -- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f102c06ace0 msgr2=0x7f102c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f102c06ace0 0x7f102c06d190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f104814ccb0 tx=0x7f1038009210 comp rx=0 tx=0).stop 2023-06-22T08:00:37.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 -- 172.21.15.23:0/3550476440 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480c40b0 msgr2=0x7f10481e3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480c40b0 0x7f10481e3970 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1044000f60 tx=0x7f104400c6e0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 -- 172.21.15.23:0/3550476440 shutdown_connections 2023-06-22T08:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f102c06ace0 0x7f102c06d190 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1048152ce0 0x7f10481ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480c40b0 0x7f10481e3970 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 --2- 172.21.15.23:0/3550476440 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10480c9fd0 0x7f10481e3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 -- 172.21.15.23:0/3550476440 >> 172.21.15.23:0/3550476440 conn(0x7f104806ae30 msgr2=0x7f10480cb760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:37.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 -- 172.21.15.23:0/3550476440 shutdown_connections 2023-06-22T08:00:37.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:37.827+0000 7f104f7ff700 1 -- 172.21.15.23:0/3550476440 wait complete. 2023-06-22T08:00:37.848 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T07:59:40.617747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-22T07:59:48.190528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-22T07:59:53.576494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-22T07:59:54.105472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-22T07:59:57.198896+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-22T08:00:37.849 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T08:00:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.049+0000 7f4aaabff700 1 Processor -- start 2023-06-22T08:00:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.049+0000 7f4aaabff700 1 -- start start 2023-06-22T08:00:38.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.050+0000 7f4aaabff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4aa4152ba0 0x7f4aa41593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.050+0000 7f4aaabff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa4159930 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.050+0000 7f4aaabff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa40c9aa0 con 0x7f4aa40c9260 2023-06-22T08:00:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.050+0000 7f4aaabff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa406aed0 con 0x7f4aa4152ba0 2023-06-22T08:00:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.050+0000 7f4aa899b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4aa4152ba0 0x7f4aa41593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.051+0000 7f4aa919c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4aa40c9ee0 0x7f4aa40ca2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60932/0 (socket says 172.21.15.23:60932) 2023-06-22T08:00:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.051+0000 7f4aa3fff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4aa40c9260 0x7f4aa40c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:41738/0 (socket says 172.21.15.23:41738) 2023-06-22T08:00:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.051+0000 7f4aa919c700 1 -- 172.21.15.23:0/4262101005 learned_addr learned my addr 172.21.15.23:0/4262101005 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.051+0000 7f4aa899b700 1 -- 172.21.15.23:0/4262101005 >> v1:172.21.15.67:6789/0 conn(0x7f4aa40c9260 legacy=0x7f4aa40c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.051+0000 7f4aa899b700 1 -- 172.21.15.23:0/4262101005 >> v1:172.21.15.23:6789/0 conn(0x7f4aa40c9ee0 legacy=0x7f4aa40ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.051+0000 7f4aa899b700 1 -- 172.21.15.23:0/4262101005 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa406b1b0 con 0x7f4aa4152ba0 2023-06-22T08:00:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.051+0000 7f4aa37fe700 1 -- 172.21.15.23:0/4262101005 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769331612 0 0) 0x7f4aa40c9aa0 con 0x7f4aa40c9260 2023-06-22T08:00:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.051+0000 7f4aa899b700 1 --2- 172.21.15.23:0/4262101005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4152ba0 0x7f4aa41593f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009840 tx=0x7f4a98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59fcbe60587bcc4c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.051+0000 7f4aa37fe700 1 -- 172.21.15.23:0/4262101005 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a98011830 con 0x7f4aa4152ba0 2023-06-22T08:00:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.052+0000 7f4aa37fe700 1 -- 172.21.15.23:0/4262101005 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a98011990 con 0x7f4aa4152ba0 2023-06-22T08:00:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.052+0000 7f4aa37fe700 1 -- 172.21.15.23:0/4262101005 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a98011d20 con 0x7f4aa4152ba0 2023-06-22T08:00:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.052+0000 7f4aaabff700 1 -- 172.21.15.23:0/4262101005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4152ba0 msgr2=0x7f4aa41593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.052+0000 7f4aaabff700 1 --2- 172.21.15.23:0/4262101005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4152ba0 0x7f4aa41593f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009840 tx=0x7f4a98009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.052+0000 7f4aaabff700 1 -- 172.21.15.23:0/4262101005 shutdown_connections 2023-06-22T08:00:38.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.052+0000 7f4aaabff700 1 --2- 172.21.15.23:0/4262101005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4152ba0 0x7f4aa41593f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.052+0000 7f4aaabff700 1 -- 172.21.15.23:0/4262101005 >> 172.21.15.23:0/4262101005 conn(0x7f4aa4065b90 msgr2=0x7f4aa4065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:38.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.052+0000 7f4aaabff700 1 -- 172.21.15.23:0/4262101005 shutdown_connections 2023-06-22T08:00:38.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.052+0000 7f4aaabff700 1 -- 172.21.15.23:0/4262101005 wait complete. 2023-06-22T08:00:38.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.054+0000 7f4aaabff700 1 Processor -- start 2023-06-22T08:00:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.054+0000 7f4aaabff700 1 -- start start 2023-06-22T08:00:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aaabff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4aa40c9260 0x7f4aa414dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aaabff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa40c9ee0 0x7f4aa41510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aaabff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4aa4152ba0 0x7f4aa41515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aaabff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40ceb90 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aaabff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40c74a0 con 0x7f4aa4152ba0 2023-06-22T08:00:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aaabff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa41eee10 con 0x7f4aa40c9260 2023-06-22T08:00:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aa3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa40c9ee0 0x7f4aa41510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aa3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa40c9ee0 0x7f4aa41510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33288/0 (socket says 172.21.15.23:33288) 2023-06-22T08:00:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aa3fff700 1 -- 172.21.15.23:0/3911776054 learned_addr learned my addr 172.21.15.23:0/3911776054 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aa919c700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4aa4152ba0 0x7f4aa41515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.055+0000 7f4aa899b700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4aa40c9260 0x7f4aa414dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aa3fff700 1 -- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4aa40c9260 msgr2=0x7f4aa414dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aa3fff700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4aa40c9260 0x7f4aa414dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aa3fff700 1 -- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4aa4152ba0 msgr2=0x7f4aa41515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aa3fff700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4aa4152ba0 0x7f4aa41515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aa3fff700 1 -- 172.21.15.23:0/3911776054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a980094a0 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aa919c700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4aa4152ba0 0x7f4aa41515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aa3fff700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa40c9ee0 0x7f4aa41510a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000c520 tx=0x7f4a9000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aa17fa700 1 -- 172.21.15.23:0/3911776054 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90004c80 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aa17fa700 1 -- 172.21.15.23:0/3911776054 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a90004de0 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aaabff700 1 -- 172.21.15.23:0/3911776054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa40c6b70 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aa17fa700 1 -- 172.21.15.23:0/3911776054 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900072f0 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.056+0000 7f4aaabff700 1 -- 172.21.15.23:0/3911776054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa40c7130 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.057+0000 7f4aaabff700 1 -- 172.21.15.23:0/3911776054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa40cbed0 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.058+0000 7f4aa17fa700 1 -- 172.21.15.23:0/3911776054 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4a90007450 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.059+0000 7f4aa17fa700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4a8806ace0 0x7f4a8806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.059+0000 7f4aa17fa700 1 -- 172.21.15.23:0/3911776054 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4a90089530 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.060+0000 7f4aa899b700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4a8806ace0 0x7f4a8806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.060+0000 7f4aa899b700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4a8806ace0 0x7f4a8806d190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4a980090b0 tx=0x7f4a9801c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:38.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.064+0000 7f4aa17fa700 1 -- 172.21.15.23:0/3911776054 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4a90054250 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.271+0000 7f4aaabff700 1 -- 172.21.15.23:0/3911776054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4aa404ea20 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.272+0000 7f4aa17fa700 1 -- 172.21.15.23:0/3911776054 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a90007700 con 0x7f4aa40c9ee0 2023-06-22T08:00:38.275 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-06-22T08:00:38.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 -- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4a8806ace0 msgr2=0x7f4a8806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4a8806ace0 0x7f4a8806d190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4a980090b0 tx=0x7f4a9801c040 comp rx=0 tx=0).stop 2023-06-22T08:00:38.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 -- 172.21.15.23:0/3911776054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa40c9ee0 msgr2=0x7f4aa41510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa40c9ee0 0x7f4aa41510a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000c520 tx=0x7f4a9000c830 comp rx=0 tx=0).stop 2023-06-22T08:00:38.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 -- 172.21.15.23:0/3911776054 shutdown_connections 2023-06-22T08:00:38.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4a8806ace0 0x7f4a8806d190 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4aa4152ba0 0x7f4aa41515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4aa40c9260 0x7f4aa414dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 --2- 172.21.15.23:0/3911776054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa40c9ee0 0x7f4aa41510a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 -- 172.21.15.23:0/3911776054 >> 172.21.15.23:0/3911776054 conn(0x7f4aa4065b90 msgr2=0x7f4aa40cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:38.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 -- 172.21.15.23:0/3911776054 shutdown_connections 2023-06-22T08:00:38.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.274+0000 7f4aaabff700 1 -- 172.21.15.23:0/3911776054 wait complete. 2023-06-22T08:00:38.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T08:00:38.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.492+0000 7fa6d0dff700 1 Processor -- start 2023-06-22T08:00:38.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.493+0000 7fa6d0dff700 1 -- start start 2023-06-22T08:00:38.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.494+0000 7fa6d0dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa6cc0c9ee0 0x7fa6cc0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.494+0000 7fa6d0dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc0c9aa0 con 0x7fa6cc152ba0 2023-06-22T08:00:38.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.494+0000 7fa6d0dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc06aed0 con 0x7fa6cc1593f0 2023-06-22T08:00:38.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.494+0000 7fa6d0dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6cc06b1b0 con 0x7fa6cc0c9ee0 2023-06-22T08:00:38.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.494+0000 7fa6cad9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa6cc0c9ee0 0x7fa6cc0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.494+0000 7fa6ca59c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa6cc152ba0 0x7fa6cc0c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60948/0 (socket says 172.21.15.23:60948) 2023-06-22T08:00:38.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.494+0000 7fa6ca59c700 1 -- 172.21.15.23:0/3812590769 learned_addr learned my addr 172.21.15.23:0/3812590769 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:38.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.495+0000 7fa6cad9d700 1 -- 172.21.15.23:0/3812590769 >> v1:172.21.15.67:6789/0 conn(0x7fa6cc1593f0 legacy=0x7fa6cc0c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.495+0000 7fa6cad9d700 1 -- 172.21.15.23:0/3812590769 >> v1:172.21.15.23:6789/0 conn(0x7fa6cc152ba0 legacy=0x7fa6cc0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.495+0000 7fa6cad9d700 1 -- 172.21.15.23:0/3812590769 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc06b490 con 0x7fa6cc0c9ee0 2023-06-22T08:00:38.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.495+0000 7fa6c9d9b700 1 -- 172.21.15.23:0/3812590769 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484685847 0 0) 0x7fa6cc0c9aa0 con 0x7fa6cc152ba0 2023-06-22T08:00:38.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.495+0000 7fa6cad9d700 1 --2- 172.21.15.23:0/3812590769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc0c9ee0 0x7fa6cc0ca2f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009d90 tx=0x7fa6c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7715e988a37592 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:38.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.495+0000 7fa6c9d9b700 1 -- 172.21.15.23:0/3812590769 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c400e070 con 0x7fa6cc0c9ee0 2023-06-22T08:00:38.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.495+0000 7fa6c9d9b700 1 -- 172.21.15.23:0/3812590769 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c400bbd0 con 0x7fa6cc0c9ee0 2023-06-22T08:00:38.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.495+0000 7fa6c9d9b700 1 -- 172.21.15.23:0/3812590769 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4012430 con 0x7fa6cc0c9ee0 2023-06-22T08:00:38.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.496+0000 7fa6d0dff700 1 -- 172.21.15.23:0/3812590769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc0c9ee0 msgr2=0x7fa6cc0ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.496+0000 7fa6d0dff700 1 --2- 172.21.15.23:0/3812590769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc0c9ee0 0x7fa6cc0ca2f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009d90 tx=0x7fa6c400b330 comp rx=0 tx=0).stop 2023-06-22T08:00:38.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.496+0000 7fa6d0dff700 1 -- 172.21.15.23:0/3812590769 shutdown_connections 2023-06-22T08:00:38.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.496+0000 7fa6d0dff700 1 --2- 172.21.15.23:0/3812590769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc0c9ee0 0x7fa6cc0ca2f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.496+0000 7fa6d0dff700 1 -- 172.21.15.23:0/3812590769 >> 172.21.15.23:0/3812590769 conn(0x7fa6cc065b90 msgr2=0x7fa6cc065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:38.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.496+0000 7fa6d0dff700 1 -- 172.21.15.23:0/3812590769 shutdown_connections 2023-06-22T08:00:38.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.496+0000 7fa6d0dff700 1 -- 172.21.15.23:0/3812590769 wait complete. 2023-06-22T08:00:38.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.498+0000 7fa6d0dff700 1 Processor -- start 2023-06-22T08:00:38.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.498+0000 7fa6d0dff700 1 -- start start 2023-06-22T08:00:38.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6d0dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6cc0c9ee0 0x7fa6cc0c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6d0dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc152ba0 0x7fa6cc150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6d0dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6cc1593f0 0x7fa6cc14e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6d0dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc0ceff0 con 0x7fa6cc152ba0 2023-06-22T08:00:38.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6d0dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc1531b0 con 0x7fa6cc0c9ee0 2023-06-22T08:00:38.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6d0dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc0ccad0 con 0x7fa6cc1593f0 2023-06-22T08:00:38.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6c3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc152ba0 0x7fa6cc150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6c3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc152ba0 0x7fa6cc150400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33302/0 (socket says 172.21.15.23:33302) 2023-06-22T08:00:38.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6c3fff700 1 -- 172.21.15.23:0/2354492388 learned_addr learned my addr 172.21.15.23:0/2354492388 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:38.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6ca59c700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6cc0c9ee0 0x7fa6cc0c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.499+0000 7fa6cad9d700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6cc1593f0 0x7fa6cc14e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6c3fff700 1 -- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6cc1593f0 msgr2=0x7fa6cc14e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6c3fff700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6cc1593f0 0x7fa6cc14e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6c3fff700 1 -- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6cc0c9ee0 msgr2=0x7fa6cc0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6c3fff700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6cc0c9ee0 0x7fa6cc0c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6c3fff700 1 -- 172.21.15.23:0/2354492388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c4009a70 con 0x7fa6cc152ba0 2023-06-22T08:00:38.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6c3fff700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc152ba0 0x7fa6cc150400 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c700 tx=0x7fa6b400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:38.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6c2ffd700 1 -- 172.21.15.23:0/2354492388 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b40052d0 con 0x7fa6cc152ba0 2023-06-22T08:00:38.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6c2ffd700 1 -- 172.21.15.23:0/2354492388 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b4005430 con 0x7fa6cc152ba0 2023-06-22T08:00:38.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6d0dff700 1 -- 172.21.15.23:0/2354492388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6cc0ccdb0 con 0x7fa6cc152ba0 2023-06-22T08:00:38.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6c2ffd700 1 -- 172.21.15.23:0/2354492388 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4003880 con 0x7fa6cc152ba0 2023-06-22T08:00:38.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.500+0000 7fa6d0dff700 1 -- 172.21.15.23:0/2354492388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc0cd3f0 con 0x7fa6cc152ba0 2023-06-22T08:00:38.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.502+0000 7fa6d0dff700 1 -- 172.21.15.23:0/2354492388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6cc0cbeb0 con 0x7fa6cc152ba0 2023-06-22T08:00:38.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.502+0000 7fa6c2ffd700 1 -- 172.21.15.23:0/2354492388 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa6b40039e0 con 0x7fa6cc152ba0 2023-06-22T08:00:38.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.503+0000 7fa6c2ffd700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa6ac06ace0 0x7fa6ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.503+0000 7fa6c2ffd700 1 -- 172.21.15.23:0/2354492388 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa6b4088a30 con 0x7fa6cc152ba0 2023-06-22T08:00:38.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.503+0000 7fa6ca59c700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa6ac06ace0 0x7fa6ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.504+0000 7fa6ca59c700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa6ac06ace0 0x7fa6ac06d190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc007b10 tx=0x7fa6bc007a60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:38.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.508+0000 7fa6c2ffd700 1 -- 172.21.15.23:0/2354492388 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa6b4053800 con 0x7fa6cc152ba0 2023-06-22T08:00:38.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.716+0000 7fa6d0dff700 1 -- 172.21.15.23:0/2354492388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa6cc04ea20 con 0x7fa6cc152ba0 2023-06-22T08:00:38.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.716+0000 7fa6c2ffd700 1 -- 172.21.15.23:0/2354492388 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa6b4056e20 con 0x7fa6cc152ba0 2023-06-22T08:00:38.720 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2023-06-22T08:00:38.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 -- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa6ac06ace0 msgr2=0x7fa6ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa6ac06ace0 0x7fa6ac06d190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc007b10 tx=0x7fa6bc007a60 comp rx=0 tx=0).stop 2023-06-22T08:00:38.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 -- 172.21.15.23:0/2354492388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc152ba0 msgr2=0x7fa6cc150400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc152ba0 0x7fa6cc150400 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c700 tx=0x7fa6b400ca10 comp rx=0 tx=0).stop 2023-06-22T08:00:38.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 -- 172.21.15.23:0/2354492388 shutdown_connections 2023-06-22T08:00:38.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6cc1593f0 0x7fa6cc14e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa6ac06ace0 0x7fa6ac06d190 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6cc0c9ee0 0x7fa6cc0c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 --2- 172.21.15.23:0/2354492388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6cc152ba0 0x7fa6cc150400 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 -- 172.21.15.23:0/2354492388 >> 172.21.15.23:0/2354492388 conn(0x7fa6cc065b90 msgr2=0x7fa6cc1569d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:38.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 -- 172.21.15.23:0/2354492388 shutdown_connections 2023-06-22T08:00:38.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.719+0000 7fa6d0dff700 1 -- 172.21.15.23:0/2354492388 wait complete. 2023-06-22T08:00:38.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-22T08:00:38.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.939+0000 7ff4507ff700 1 Processor -- start 2023-06-22T08:00:38.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.939+0000 7ff4507ff700 1 -- start start 2023-06-22T08:00:38.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.940+0000 7ff4507ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4481593f0 0x7ff4480c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.940+0000 7ff4507ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4480c4650 0x7ff4480c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.940+0000 7ff4507ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4480c9aa0 con 0x7ff448152ba0 2023-06-22T08:00:38.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.940+0000 7ff4507ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff44806aed0 con 0x7ff4480c4650 2023-06-22T08:00:38.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.940+0000 7ff4507ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44806b1b0 con 0x7ff4481593f0 2023-06-22T08:00:38.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44e59b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff448152ba0 0x7ff4480c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:41762/0 (socket says 172.21.15.23:41762) 2023-06-22T08:00:38.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44ed9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4480c4650 0x7ff4480c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44e59b700 1 -- 172.21.15.23:0/1380763260 learned_addr learned my addr 172.21.15.23:0/1380763260 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:38.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44dd9a700 1 --2- 172.21.15.23:0/1380763260 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4481593f0 0x7ff4480c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44d599700 1 -- 172.21.15.23:0/1380763260 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097483393 0 0) 0x7ff4480c9aa0 con 0x7ff448152ba0 2023-06-22T08:00:38.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44d599700 1 -- 172.21.15.23:0/1380763260 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4340035b0 con 0x7ff448152ba0 2023-06-22T08:00:38.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44ed9c700 1 -- 172.21.15.23:0/1380763260 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4481593f0 msgr2=0x7ff4480c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44ed9c700 1 --2- 172.21.15.23:0/1380763260 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4481593f0 0x7ff4480c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44ed9c700 1 -- 172.21.15.23:0/1380763260 >> v1:172.21.15.67:6789/0 conn(0x7ff448152ba0 legacy=0x7ff4480c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44ed9c700 1 -- 172.21.15.23:0/1380763260 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44806b490 con 0x7ff4480c4650 2023-06-22T08:00:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.941+0000 7ff44dd9a700 1 --2- 172.21.15.23:0/1380763260 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4481593f0 0x7ff4480c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.942+0000 7ff44ed9c700 1 --2- 172.21.15.23:0/1380763260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c4650 0x7ff4480c9ee0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff440009d90 tx=0x7ff44000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf33474e2252eb92 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.942+0000 7ff44d599700 1 -- 172.21.15.23:0/1380763260 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44000e070 con 0x7ff4480c4650 2023-06-22T08:00:38.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.942+0000 7ff44d599700 1 -- 172.21.15.23:0/1380763260 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44000bb20 con 0x7ff4480c4650 2023-06-22T08:00:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.942+0000 7ff44d599700 1 -- 172.21.15.23:0/1380763260 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44000be90 con 0x7ff4480c4650 2023-06-22T08:00:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.942+0000 7ff4507ff700 1 -- 172.21.15.23:0/1380763260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c4650 msgr2=0x7ff4480c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.942+0000 7ff4507ff700 1 --2- 172.21.15.23:0/1380763260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c4650 0x7ff4480c9ee0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff440009d90 tx=0x7ff44000b330 comp rx=0 tx=0).stop 2023-06-22T08:00:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.942+0000 7ff4507ff700 1 -- 172.21.15.23:0/1380763260 shutdown_connections 2023-06-22T08:00:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.942+0000 7ff4507ff700 1 --2- 172.21.15.23:0/1380763260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c4650 0x7ff4480c9ee0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.942+0000 7ff4507ff700 1 --2- 172.21.15.23:0/1380763260 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4481593f0 0x7ff4480c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.942+0000 7ff4507ff700 1 -- 172.21.15.23:0/1380763260 >> 172.21.15.23:0/1380763260 conn(0x7ff448065b90 msgr2=0x7ff448065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.943+0000 7ff4507ff700 1 -- 172.21.15.23:0/1380763260 shutdown_connections 2023-06-22T08:00:38.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.943+0000 7ff4507ff700 1 -- 172.21.15.23:0/1380763260 wait complete. 2023-06-22T08:00:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.944+0000 7ff4507ff700 1 Processor -- start 2023-06-22T08:00:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.944+0000 7ff4507ff700 1 -- start start 2023-06-22T08:00:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.945+0000 7ff4507ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4480c4650 0x7ff448151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.945+0000 7ff4507ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff448152ba0 0x7ff4481ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.945+0000 7ff4507ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4481593f0 0x7ff4481ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.945+0000 7ff4507ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481f18f0 con 0x7ff4480c4650 2023-06-22T08:00:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.945+0000 7ff4507ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481e2b30 con 0x7ff4481593f0 2023-06-22T08:00:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.945+0000 7ff4507ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481f1b40 con 0x7ff448152ba0 2023-06-22T08:00:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44e59b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4480c4650 0x7ff448151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44dd9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff448152ba0 0x7ff4481ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44e59b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4480c4650 0x7ff448151040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33314/0 (socket says 172.21.15.23:33314) 2023-06-22T08:00:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44e59b700 1 -- 172.21.15.23:0/1763191624 learned_addr learned my addr 172.21.15.23:0/1763191624 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:38.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44ed9c700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4481593f0 0x7ff4481ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44ed9c700 1 -- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff448152ba0 msgr2=0x7ff4481ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44ed9c700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff448152ba0 0x7ff4481ee380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44ed9c700 1 -- 172.21.15.23:0/1763191624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4480c4650 msgr2=0x7ff448151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:38.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44ed9c700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4480c4650 0x7ff448151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:38.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44ed9c700 1 -- 172.21.15.23:0/1763191624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4380094a0 con 0x7ff4481593f0 2023-06-22T08:00:38.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44e59b700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4480c4650 0x7ff448151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.946+0000 7ff44dd9a700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff448152ba0 0x7ff4481ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.947+0000 7ff44ed9c700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4481593f0 0x7ff4481ee8c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff440016000 tx=0x7ff440004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.947+0000 7ff43effd700 1 -- 172.21.15.23:0/1763191624 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440004880 con 0x7ff4481593f0 2023-06-22T08:00:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.947+0000 7ff4507ff700 1 -- 172.21.15.23:0/1763191624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff440009a70 con 0x7ff4481593f0 2023-06-22T08:00:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.947+0000 7ff43effd700 1 -- 172.21.15.23:0/1763191624 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff440020070 con 0x7ff4481593f0 2023-06-22T08:00:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.947+0000 7ff43effd700 1 -- 172.21.15.23:0/1763191624 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4400276c0 con 0x7ff4481593f0 2023-06-22T08:00:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.947+0000 7ff4507ff700 1 -- 172.21.15.23:0/1763191624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4481f2100 con 0x7ff4481593f0 2023-06-22T08:00:38.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.948+0000 7ff4507ff700 1 -- 172.21.15.23:0/1763191624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4480cbed0 con 0x7ff4481593f0 2023-06-22T08:00:38.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.949+0000 7ff43effd700 1 -- 172.21.15.23:0/1763191624 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff440027820 con 0x7ff4481593f0 2023-06-22T08:00:38.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.950+0000 7ff43effd700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ace0 0x7ff42c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:38.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.950+0000 7ff43effd700 1 -- 172.21.15.23:0/1763191624 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff440022070 con 0x7ff4481593f0 2023-06-22T08:00:38.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.950+0000 7ff44e59b700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ace0 0x7ff42c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:38.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.951+0000 7ff44e59b700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ace0 0x7ff42c06d190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff4440011e0 tx=0x7ff444009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:38.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:38.954+0000 7ff43effd700 1 -- 172.21.15.23:0/1763191624 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff440056320 con 0x7ff4481593f0 2023-06-22T08:00:39.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.161+0000 7ff4507ff700 1 -- 172.21.15.23:0/1763191624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff44804ea20 con 0x7ff4481593f0 2023-06-22T08:00:39.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.161+0000 7ff43effd700 1 -- 172.21.15.23:0/1763191624 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff440059940 con 0x7ff4481593f0 2023-06-22T08:00:39.165 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:39.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 -- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ace0 msgr2=0x7ff42c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ace0 0x7ff42c06d190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff4440011e0 tx=0x7ff444009040 comp rx=0 tx=0).stop 2023-06-22T08:00:39.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 -- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4481593f0 msgr2=0x7ff4481ee8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4481593f0 0x7ff4481ee8c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff440016000 tx=0x7ff440004ad0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 -- 172.21.15.23:0/1763191624 shutdown_connections 2023-06-22T08:00:39.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ace0 0x7ff42c06d190 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4481593f0 0x7ff4481ee8c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4480c4650 0x7ff448151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 --2- 172.21.15.23:0/1763191624 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff448152ba0 0x7ff4481ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 -- 172.21.15.23:0/1763191624 >> 172.21.15.23:0/1763191624 conn(0x7ff448065b90 msgr2=0x7ff448065f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:39.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 -- 172.21.15.23:0/1763191624 shutdown_connections 2023-06-22T08:00:39.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.164+0000 7ff4507ff700 1 -- 172.21.15.23:0/1763191624 wait complete. 2023-06-22T08:00:39.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-22T08:00:39.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.387+0000 7fefe39ff700 1 Processor -- start 2023-06-22T08:00:39.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.387+0000 7fefe39ff700 1 -- start start 2023-06-22T08:00:39.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe39ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fefdc1593f0 0x7fefdc0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe39ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefdc0c4650 0x7fefdc0c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe39ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefdc0c9aa0 con 0x7fefdc152ba0 2023-06-22T08:00:39.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe39ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefdc06aed0 con 0x7fefdc1593f0 2023-06-22T08:00:39.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe39ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc06b1b0 con 0x7fefdc0c4650 2023-06-22T08:00:39.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe0f9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fefdc1593f0 0x7fefdc0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe0f9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fefdc1593f0 0x7fefdc0c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33328/0 (socket says 172.21.15.23:33328) 2023-06-22T08:00:39.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe0f9a700 1 -- 172.21.15.23:0/1887809447 learned_addr learned my addr 172.21.15.23:0/1887809447 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:39.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe1f9c700 1 --2- 172.21.15.23:0/1887809447 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefdc0c4650 0x7fefdc0c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe0f9a700 1 -- 172.21.15.23:0/1887809447 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefdc0c4650 msgr2=0x7fefdc0c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:39.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe0f9a700 1 --2- 172.21.15.23:0/1887809447 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefdc0c4650 0x7fefdc0c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.389+0000 7fefe0f9a700 1 -- 172.21.15.23:0/1887809447 >> v1:172.21.15.67:6789/0 conn(0x7fefdc152ba0 legacy=0x7fefdc0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.390+0000 7fefe0f9a700 1 -- 172.21.15.23:0/1887809447 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc06b490 con 0x7fefdc1593f0 2023-06-22T08:00:39.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.390+0000 7fefe0f9a700 1 --2- 172.21.15.23:0/1887809447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc1593f0 0x7fefdc0c4110 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fefcc0098f0 tx=0x7fefcc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2863a9482c6f738a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:39.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.390+0000 7fefd3fff700 1 -- 172.21.15.23:0/1887809447 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc011830 con 0x7fefdc1593f0 2023-06-22T08:00:39.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.390+0000 7fefd3fff700 1 -- 172.21.15.23:0/1887809447 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefcc011990 con 0x7fefdc1593f0 2023-06-22T08:00:39.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.390+0000 7fefd3fff700 1 -- 172.21.15.23:0/1887809447 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc011d20 con 0x7fefdc1593f0 2023-06-22T08:00:39.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.390+0000 7fefe39ff700 1 -- 172.21.15.23:0/1887809447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc1593f0 msgr2=0x7fefdc0c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.390+0000 7fefe39ff700 1 --2- 172.21.15.23:0/1887809447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc1593f0 0x7fefdc0c4110 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fefcc0098f0 tx=0x7fefcc009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.391+0000 7fefe39ff700 1 -- 172.21.15.23:0/1887809447 shutdown_connections 2023-06-22T08:00:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.391+0000 7fefe39ff700 1 --2- 172.21.15.23:0/1887809447 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefdc0c4650 0x7fefdc0c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.391+0000 7fefe39ff700 1 --2- 172.21.15.23:0/1887809447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc1593f0 0x7fefdc0c4110 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.391+0000 7fefe39ff700 1 -- 172.21.15.23:0/1887809447 >> 172.21.15.23:0/1887809447 conn(0x7fefdc065b90 msgr2=0x7fefdc065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.391+0000 7fefe39ff700 1 -- 172.21.15.23:0/1887809447 shutdown_connections 2023-06-22T08:00:39.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.391+0000 7fefe39ff700 1 -- 172.21.15.23:0/1887809447 wait complete. 2023-06-22T08:00:39.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.392+0000 7fefe39ff700 1 Processor -- start 2023-06-22T08:00:39.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.393+0000 7fefe39ff700 1 -- start start 2023-06-22T08:00:39.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.393+0000 7fefe39ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc0c4650 0x7fefdc1f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.393+0000 7fefe39ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefdc152ba0 0x7fefdc1e4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.393+0000 7fefe39ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefdc1593f0 0x7fefdc0b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe39ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc14c0c0 con 0x7fefdc0c4650 2023-06-22T08:00:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe39ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc1f2470 con 0x7fefdc1593f0 2023-06-22T08:00:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe39ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc0c62f0 con 0x7fefdc152ba0 2023-06-22T08:00:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe179b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc0c4650 0x7fefdc1f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe179b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc0c4650 0x7fefdc1f1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33338/0 (socket says 172.21.15.23:33338) 2023-06-22T08:00:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe179b700 1 -- 172.21.15.23:0/2381951652 learned_addr learned my addr 172.21.15.23:0/2381951652 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe1f9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefdc1593f0 0x7fefdc0b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe0f9a700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefdc152ba0 0x7fefdc1e4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe179b700 1 -- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefdc152ba0 msgr2=0x7fefdc1e4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe179b700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefdc152ba0 0x7fefdc1e4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe179b700 1 -- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefdc1593f0 msgr2=0x7fefdc0b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe179b700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefdc1593f0 0x7fefdc0b35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.394+0000 7fefe179b700 1 -- 172.21.15.23:0/2381951652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefcc009550 con 0x7fefdc0c4650 2023-06-22T08:00:39.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.395+0000 7fefe179b700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc0c4650 0x7fefdc1f1f30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fefd800c4e0 tx=0x7fefd800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:39.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.395+0000 7fefd1ffb700 1 -- 172.21.15.23:0/2381951652 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd80052d0 con 0x7fefdc0c4650 2023-06-22T08:00:39.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.395+0000 7fefd1ffb700 1 -- 172.21.15.23:0/2381951652 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd800ce90 con 0x7fefdc0c4650 2023-06-22T08:00:39.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.395+0000 7fefd1ffb700 1 -- 172.21.15.23:0/2381951652 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8003680 con 0x7fefdc0c4650 2023-06-22T08:00:39.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.395+0000 7fefe39ff700 1 -- 172.21.15.23:0/2381951652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefdc0c6590 con 0x7fefdc0c4650 2023-06-22T08:00:39.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.395+0000 7fefe39ff700 1 -- 172.21.15.23:0/2381951652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefdc1ee5e0 con 0x7fefdc0c4650 2023-06-22T08:00:39.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.396+0000 7fefe39ff700 1 -- 172.21.15.23:0/2381951652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefdc0cbed0 con 0x7fefdc0c4650 2023-06-22T08:00:39.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.397+0000 7fefd1ffb700 1 -- 172.21.15.23:0/2381951652 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fefd80037e0 con 0x7fefdc0c4650 2023-06-22T08:00:39.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.398+0000 7fefd1ffb700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fefc006ace0 0x7fefc006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.398+0000 7fefd1ffb700 1 -- 172.21.15.23:0/2381951652 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fefd8088b50 con 0x7fefdc0c4650 2023-06-22T08:00:39.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.398+0000 7fefe0f9a700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fefc006ace0 0x7fefc006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.399+0000 7fefe0f9a700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fefc006ace0 0x7fefc006d190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fefcc009c40 tx=0x7fefcc01d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:39.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.402+0000 7fefd1ffb700 1 -- 172.21.15.23:0/2381951652 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fefd8053870 con 0x7fefdc0c4650 2023-06-22T08:00:39.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.608+0000 7fefe39ff700 1 -- 172.21.15.23:0/2381951652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fefdc04ea20 con 0x7fefdc0c4650 2023-06-22T08:00:39.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.608+0000 7fefd1ffb700 1 -- 172.21.15.23:0/2381951652 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7fefd8056e90 con 0x7fefdc0c4650 2023-06-22T08:00:39.611 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-06-22T08:00:39.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 -- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fefc006ace0 msgr2=0x7fefc006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fefc006ace0 0x7fefc006d190 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fefcc009c40 tx=0x7fefcc01d040 comp rx=0 tx=0).stop 2023-06-22T08:00:39.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 -- 172.21.15.23:0/2381951652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc0c4650 msgr2=0x7fefdc1f1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc0c4650 0x7fefdc1f1f30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fefd800c4e0 tx=0x7fefd800c7f0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 -- 172.21.15.23:0/2381951652 shutdown_connections 2023-06-22T08:00:39.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fefc006ace0 0x7fefc006d190 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefdc1593f0 0x7fefdc0b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefdc0c4650 0x7fefdc1f1f30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 --2- 172.21.15.23:0/2381951652 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fefdc152ba0 0x7fefdc1e4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 -- 172.21.15.23:0/2381951652 >> 172.21.15.23:0/2381951652 conn(0x7fefdc065b90 msgr2=0x7fefdc0cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:39.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.610+0000 7fefe39ff700 1 -- 172.21.15.23:0/2381951652 shutdown_connections 2023-06-22T08:00:39.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.611+0000 7fefe39ff700 1 -- 172.21.15.23:0/2381951652 wait complete. 2023-06-22T08:00:39.632 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-22T08:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.831+0000 7f5cdf3ff700 1 Processor -- start 2023-06-22T08:00:39.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.831+0000 7f5cdf3ff700 1 -- start start 2023-06-22T08:00:39.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.832+0000 7f5cdf3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5cd8152ba0 0x7f5cd81593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.832+0000 7f5cdf3ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5cd80c9ee0 0x7f5cd80ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.832+0000 7f5cdf3ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd8159930 con 0x7f5cd80c9260 2023-06-22T08:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.832+0000 7f5cdf3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd80c9aa0 con 0x7f5cd8152ba0 2023-06-22T08:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.832+0000 7f5cdf3ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd806aed0 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.832+0000 7f5cdd19b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5cd8152ba0 0x7f5cd81593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.832+0000 7f5cdd19b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5cd8152ba0 0x7f5cd81593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33354/0 (socket says 172.21.15.23:33354) 2023-06-22T08:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.832+0000 7f5cdd19b700 1 -- 172.21.15.23:0/3447840239 learned_addr learned my addr 172.21.15.23:0/3447840239 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:39.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.833+0000 7f5cdd99c700 1 --2- 172.21.15.23:0/3447840239 >> v2:172.21.15.67:3300/0 conn(0x7f5cd80c9ee0 0x7f5cd80ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.833+0000 7f5cdd19b700 1 -- 172.21.15.23:0/3447840239 >> v2:172.21.15.67:3300/0 conn(0x7f5cd80c9ee0 msgr2=0x7f5cd80ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:39.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.833+0000 7f5cdd19b700 1 --2- 172.21.15.23:0/3447840239 >> v2:172.21.15.67:3300/0 conn(0x7f5cd80c9ee0 0x7f5cd80ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.833+0000 7f5cdd19b700 1 -- 172.21.15.23:0/3447840239 >> v1:172.21.15.23:6789/0 conn(0x7f5cd80c9260 legacy=0x7f5cd80c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.833+0000 7f5cdd19b700 1 -- 172.21.15.23:0/3447840239 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd806b1b0 con 0x7f5cd8152ba0 2023-06-22T08:00:39.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.833+0000 7f5ccffff700 1 -- 172.21.15.23:0/3447840239 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48572009 0 0) 0x7f5cc8001110 con 0x7f5cd80c9260 2023-06-22T08:00:39.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.833+0000 7f5cdd19b700 1 --2- 172.21.15.23:0/3447840239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd8152ba0 0x7f5cd81593f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5cd808b390 tx=0x7f5cd4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8806e946db9b50d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:39.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.833+0000 7f5ccffff700 1 -- 172.21.15.23:0/3447840239 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4011830 con 0x7f5cd8152ba0 2023-06-22T08:00:39.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.833+0000 7f5ccffff700 1 -- 172.21.15.23:0/3447840239 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd4011990 con 0x7f5cd8152ba0 2023-06-22T08:00:39.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.833+0000 7f5ccffff700 1 -- 172.21.15.23:0/3447840239 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4011d20 con 0x7f5cd8152ba0 2023-06-22T08:00:39.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.834+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3447840239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd8152ba0 msgr2=0x7f5cd81593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.834+0000 7f5cdf3ff700 1 --2- 172.21.15.23:0/3447840239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd8152ba0 0x7f5cd81593f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5cd808b390 tx=0x7f5cd4009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.834+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3447840239 shutdown_connections 2023-06-22T08:00:39.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.834+0000 7f5cdf3ff700 1 --2- 172.21.15.23:0/3447840239 >> v2:172.21.15.67:3300/0 conn(0x7f5cd80c9ee0 0x7f5cd80ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.834+0000 7f5cdf3ff700 1 --2- 172.21.15.23:0/3447840239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd8152ba0 0x7f5cd81593f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.834+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3447840239 >> 172.21.15.23:0/3447840239 conn(0x7f5cd8065b90 msgr2=0x7f5cd8065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:39.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.834+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3447840239 shutdown_connections 2023-06-22T08:00:39.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.834+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3447840239 wait complete. 2023-06-22T08:00:39.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.836+0000 7f5cdf3ff700 1 Processor -- start 2023-06-22T08:00:39.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.836+0000 7f5cdf3ff700 1 -- start start 2023-06-22T08:00:39.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdf3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cd80c9260 0x7f5cd80c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdf3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd80c9ee0 0x7f5cd814b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdf3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd8152ba0 0x7f5cd814d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdf3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd81ef000 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdf3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80c7cb0 con 0x7f5cd80c9260 2023-06-22T08:00:39.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdf3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd80c5ef0 con 0x7f5cd8152ba0 2023-06-22T08:00:39.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdc99a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd80c9ee0 0x7f5cd814b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdc99a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd80c9ee0 0x7f5cd814b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33364/0 (socket says 172.21.15.23:33364) 2023-06-22T08:00:39.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdc99a700 1 -- 172.21.15.23:0/3434538251 learned_addr learned my addr 172.21.15.23:0/3434538251 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:39.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdd19b700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cd80c9260 0x7f5cd80c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.837+0000 7f5cdd99c700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd8152ba0 0x7f5cd814d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5cdc99a700 1 -- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd8152ba0 msgr2=0x7f5cd814d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5cdc99a700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd8152ba0 0x7f5cd814d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5cdc99a700 1 -- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cd80c9260 msgr2=0x7f5cd80c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:39.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5cdc99a700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cd80c9260 0x7f5cd80c7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:39.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5cdc99a700 1 -- 172.21.15.23:0/3434538251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4009550 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5cdc99a700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd80c9ee0 0x7f5cd814b800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5cc800c750 tx=0x7f5cc800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:39.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5ccdffb700 1 -- 172.21.15.23:0/3434538251 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc80054e0 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5ccdffb700 1 -- 172.21.15.23:0/3434538251 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc8003680 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5ccdffb700 1 -- 172.21.15.23:0/3434538251 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8003a30 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3434538251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd80c61d0 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.838+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3434538251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd80c6f70 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.839+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3434538251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd80cbed0 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.840+0000 7f5ccdffb700 1 -- 172.21.15.23:0/3434538251 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5cc8003b90 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.841+0000 7f5ccdffb700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ace0 0x7f5cbc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:39.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.841+0000 7f5ccdffb700 1 -- 172.21.15.23:0/3434538251 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f5cc8088ad0 con 0x7f5cd80c9ee0 2023-06-22T08:00:39.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.842+0000 7f5cdd19b700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ace0 0x7f5cbc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:39.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.842+0000 7f5cdd19b700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ace0 0x7f5cbc06d190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4014040 tx=0x7f5cd4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:39.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:39.846+0000 7f5ccdffb700 1 -- 172.21.15.23:0/3434538251 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5cc8057280 con 0x7f5cd80c9ee0 2023-06-22T08:00:40.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.052+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3434538251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5cd8159930 con 0x7f5cd80c9ee0 2023-06-22T08:00:40.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.053+0000 7f5ccdffb700 1 -- 172.21.15.23:0/3434538251 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5cc8017020 con 0x7f5cd80c9ee0 2023-06-22T08:00:40.056 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:40.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ace0 msgr2=0x7f5cbc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ace0 0x7f5cbc06d190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4014040 tx=0x7f5cd4008030 comp rx=0 tx=0).stop 2023-06-22T08:00:40.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3434538251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd80c9ee0 msgr2=0x7f5cd814b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd80c9ee0 0x7f5cd814b800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5cc800c750 tx=0x7f5cc800ca60 comp rx=0 tx=0).stop 2023-06-22T08:00:40.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3434538251 shutdown_connections 2023-06-22T08:00:40.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd8152ba0 0x7f5cd814d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ace0 0x7f5cbc06d190 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cd80c9260 0x7f5cd80c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 --2- 172.21.15.23:0/3434538251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd80c9ee0 0x7f5cd814b800 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3434538251 >> 172.21.15.23:0/3434538251 conn(0x7f5cd8065b90 msgr2=0x7f5cd8151b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:40.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3434538251 shutdown_connections 2023-06-22T08:00:40.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.055+0000 7f5cdf3ff700 1 -- 172.21.15.23:0/3434538251 wait complete. 2023-06-22T08:00:40.076 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:00:40.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.273+0000 7f85c89ff700 1 Processor -- start 2023-06-22T08:00:40.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.273+0000 7f85c89ff700 1 -- start start 2023-06-22T08:00:40.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.274+0000 7f85c89ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85c40c9fd0 0x7f85c40cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.274+0000 7f85c89ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f85c40c40b0 0x7f85c40c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.274+0000 7f85c89ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c415c370 con 0x7f85c4152ce0 2023-06-22T08:00:40.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.274+0000 7f85c89ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c41530b0 con 0x7f85c40c40b0 2023-06-22T08:00:40.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.274+0000 7f85c89ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c40cf6f0 con 0x7f85c40c9fd0 2023-06-22T08:00:40.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c259c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f85c4152ce0 0x7f85c415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60968/0 (socket says 172.21.15.23:60968) 2023-06-22T08:00:40.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c259c700 1 -- 172.21.15.23:0/3192816258 learned_addr learned my addr 172.21.15.23:0/3192816258 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:40.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c2d9d700 1 --2- 172.21.15.23:0/3192816258 >> v2:172.21.15.23:3300/0 conn(0x7f85c40c40b0 0x7f85c40c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c1d9b700 1 --2- 172.21.15.23:0/3192816258 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85c40c9fd0 0x7f85c40cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c159a700 1 -- 172.21.15.23:0/3192816258 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162577533 0 0) 0x7f85c415c370 con 0x7f85c4152ce0 2023-06-22T08:00:40.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c159a700 1 -- 172.21.15.23:0/3192816258 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ac0035b0 con 0x7f85c4152ce0 2023-06-22T08:00:40.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c2d9d700 1 -- 172.21.15.23:0/3192816258 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85c40c9fd0 msgr2=0x7f85c40cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:40.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c2d9d700 1 --2- 172.21.15.23:0/3192816258 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85c40c9fd0 0x7f85c40cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c2d9d700 1 -- 172.21.15.23:0/3192816258 >> v1:172.21.15.23:6789/0 conn(0x7f85c4152ce0 legacy=0x7f85c415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c2d9d700 1 -- 172.21.15.23:0/3192816258 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c415bda0 con 0x7f85c40c40b0 2023-06-22T08:00:40.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.275+0000 7f85c2d9d700 1 --2- 172.21.15.23:0/3192816258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 0x7f85c40c4530 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f85bc009d90 tx=0x7f85bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b2db0523c93f70 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:40.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c159a700 1 -- 172.21.15.23:0/3192816258 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc00e070 con 0x7f85c40c40b0 2023-06-22T08:00:40.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c159a700 1 -- 172.21.15.23:0/3192816258 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85bc00bb20 con 0x7f85c40c40b0 2023-06-22T08:00:40.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c159a700 1 -- 172.21.15.23:0/3192816258 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc012430 con 0x7f85c40c40b0 2023-06-22T08:00:40.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c89ff700 1 -- 172.21.15.23:0/3192816258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 msgr2=0x7f85c40c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c89ff700 1 --2- 172.21.15.23:0/3192816258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 0x7f85c40c4530 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f85bc009d90 tx=0x7f85bc00b330 comp rx=0 tx=0).stop 2023-06-22T08:00:40.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c89ff700 1 -- 172.21.15.23:0/3192816258 shutdown_connections 2023-06-22T08:00:40.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c89ff700 1 --2- 172.21.15.23:0/3192816258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 0x7f85c40c4530 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c89ff700 1 --2- 172.21.15.23:0/3192816258 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85c40c9fd0 0x7f85c40cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c89ff700 1 -- 172.21.15.23:0/3192816258 >> 172.21.15.23:0/3192816258 conn(0x7f85c406ae30 msgr2=0x7f85c406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:40.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c89ff700 1 -- 172.21.15.23:0/3192816258 shutdown_connections 2023-06-22T08:00:40.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.276+0000 7f85c89ff700 1 -- 172.21.15.23:0/3192816258 wait complete. 2023-06-22T08:00:40.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.279+0000 7f85c89ff700 1 Processor -- start 2023-06-22T08:00:40.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.279+0000 7f85c89ff700 1 -- start start 2023-06-22T08:00:40.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.280+0000 7f85c89ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 0x7f85c40c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.280+0000 7f85c89ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85c40c9fd0 0x7f85c40ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.280+0000 7f85c89ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4152ce0 0x7f85c41e38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.280+0000 7f85c89ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c41f1840 con 0x7f85c40c40b0 2023-06-22T08:00:40.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.280+0000 7f85c89ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c41f1ae0 con 0x7f85c4152ce0 2023-06-22T08:00:40.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.280+0000 7f85c89ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c41f1dc0 con 0x7f85c40c9fd0 2023-06-22T08:00:40.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.280+0000 7f85c259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 0x7f85c40c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c1d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85c40c9fd0 0x7f85c40ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 0x7f85c40c7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33384/0 (socket says 172.21.15.23:33384) 2023-06-22T08:00:40.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c259c700 1 -- 172.21.15.23:0/51479041 learned_addr learned my addr 172.21.15.23:0/51479041 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:40.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c2d9d700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4152ce0 0x7f85c41e38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c259c700 1 -- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85c40c9fd0 msgr2=0x7f85c40ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c259c700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85c40c9fd0 0x7f85c40ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c259c700 1 -- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4152ce0 msgr2=0x7f85c41e38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c259c700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4152ce0 0x7f85c41e38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c259c700 1 -- 172.21.15.23:0/51479041 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc009a70 con 0x7f85c40c40b0 2023-06-22T08:00:40.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c259c700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 0x7f85c40c7be0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f85b400c700 tx=0x7f85b400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:40.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85b2ffd700 1 -- 172.21.15.23:0/51479041 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4005d70 con 0x7f85c40c40b0 2023-06-22T08:00:40.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85b2ffd700 1 -- 172.21.15.23:0/51479041 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b40070d0 con 0x7f85c40c40b0 2023-06-22T08:00:40.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85b2ffd700 1 -- 172.21.15.23:0/51479041 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4007460 con 0x7f85c40c40b0 2023-06-22T08:00:40.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.281+0000 7f85c89ff700 1 -- 172.21.15.23:0/51479041 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c41f20a0 con 0x7f85c40c40b0 2023-06-22T08:00:40.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.282+0000 7f85c89ff700 1 -- 172.21.15.23:0/51479041 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c41f2660 con 0x7f85c40c40b0 2023-06-22T08:00:40.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.282+0000 7f85c89ff700 1 -- 172.21.15.23:0/51479041 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c404ea20 con 0x7f85c40c40b0 2023-06-22T08:00:40.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.283+0000 7f85b2ffd700 1 -- 172.21.15.23:0/51479041 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f85b40075c0 con 0x7f85c40c40b0 2023-06-22T08:00:40.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.284+0000 7f85b2ffd700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85a406ad10 0x7f85a406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.284+0000 7f85b2ffd700 1 -- 172.21.15.23:0/51479041 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f85b4088c40 con 0x7f85c40c40b0 2023-06-22T08:00:40.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.285+0000 7f85c1d9b700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85a406ad10 0x7f85a406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.285+0000 7f85c1d9b700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85a406ad10 0x7f85a406d1c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f85c40cb3d0 tx=0x7f85b8009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:40.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.289+0000 7f85b2ffd700 1 -- 172.21.15.23:0/51479041 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f85b4053960 con 0x7f85c40c40b0 2023-06-22T08:00:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.539+0000 7f85c89ff700 1 -- 172.21.15.23:0/51479041 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85c40c8c40 con 0x7f85c40c40b0 2023-06-22T08:00:40.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.540+0000 7f85b2ffd700 1 -- 172.21.15.23:0/51479041 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6244 (secure 0 0 0) 0x7f85b4017090 con 0x7f85c40c40b0 2023-06-22T08:00:40.543 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:40.543 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:00:36.552649+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:creating","state_seq":1,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:00:40.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.542+0000 7f85c89ff700 1 -- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85a406ad10 msgr2=0x7f85a406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.542+0000 7f85c89ff700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85a406ad10 0x7f85a406d1c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f85c40cb3d0 tx=0x7f85b8009210 comp rx=0 tx=0).stop 2023-06-22T08:00:40.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.542+0000 7f85c89ff700 1 -- 172.21.15.23:0/51479041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 msgr2=0x7f85c40c7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.542+0000 7f85c89ff700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 0x7f85c40c7be0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f85b400c700 tx=0x7f85b400ca10 comp rx=0 tx=0).stop 2023-06-22T08:00:40.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.543+0000 7f85c89ff700 1 -- 172.21.15.23:0/51479041 shutdown_connections 2023-06-22T08:00:40.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.543+0000 7f85c89ff700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f85a406ad10 0x7f85a406d1c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.543+0000 7f85c89ff700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85c4152ce0 0x7f85c41e38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.543+0000 7f85c89ff700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85c40c40b0 0x7f85c40c7be0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.543+0000 7f85c89ff700 1 --2- 172.21.15.23:0/51479041 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85c40c9fd0 0x7f85c40ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.543+0000 7f85c89ff700 1 -- 172.21.15.23:0/51479041 >> 172.21.15.23:0/51479041 conn(0x7f85c406ae30 msgr2=0x7f85c414c430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:40.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.543+0000 7f85c89ff700 1 -- 172.21.15.23:0/51479041 shutdown_connections 2023-06-22T08:00:40.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.543+0000 7f85c89ff700 1 -- 172.21.15.23:0/51479041 wait complete. 2023-06-22T08:00:40.547 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 18 2023-06-22T08:00:40.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:40.597 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:00:40.594+0000 7fa5334b2700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-06-22T08:00:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.761+0000 7f0942bff700 1 Processor -- start 2023-06-22T08:00:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.761+0000 7f0942bff700 1 -- start start 2023-06-22T08:00:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.762+0000 7f0942bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f093c0c9fd0 0x7f093c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.762+0000 7f0942bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093c15c370 con 0x7f093c0c40b0 2023-06-22T08:00:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.762+0000 7f0942bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093c15bda0 con 0x7f093c152ce0 2023-06-22T08:00:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.762+0000 7f0942bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093c1530b0 con 0x7f093c0c9fd0 2023-06-22T08:00:40.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.762+0000 7f093bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f093c0c9fd0 0x7f093c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.762+0000 7f094119c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f093c0c40b0 0x7f093c0c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60982/0 (socket says 172.21.15.23:60982) 2023-06-22T08:00:40.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.762+0000 7f094119c700 1 -- 172.21.15.23:0/315102753 learned_addr learned my addr 172.21.15.23:0/315102753 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:40.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.763+0000 7f093b7fe700 1 -- 172.21.15.23:0/315102753 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658534833 0 0) 0x7f093c15c370 con 0x7f093c0c40b0 2023-06-22T08:00:40.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.763+0000 7f093bfff700 1 -- 172.21.15.23:0/315102753 >> v1:172.21.15.67:6789/0 conn(0x7f093c152ce0 legacy=0x7f093c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.763+0000 7f093bfff700 1 -- 172.21.15.23:0/315102753 >> v1:172.21.15.23:6789/0 conn(0x7f093c0c40b0 legacy=0x7f093c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.763+0000 7f093bfff700 1 -- 172.21.15.23:0/315102753 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c0c9460 con 0x7f093c0c9fd0 2023-06-22T08:00:40.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.763+0000 7f093bfff700 1 --2- 172.21.15.23:0/315102753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 0x7f093c0cf1b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0928009840 tx=0x7f0928009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3b777da3587e95 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:40.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.763+0000 7f093b7fe700 1 -- 172.21.15.23:0/315102753 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928011830 con 0x7f093c0c9fd0 2023-06-22T08:00:40.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.763+0000 7f093b7fe700 1 -- 172.21.15.23:0/315102753 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0928011990 con 0x7f093c0c9fd0 2023-06-22T08:00:40.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.763+0000 7f093b7fe700 1 -- 172.21.15.23:0/315102753 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928011d20 con 0x7f093c0c9fd0 2023-06-22T08:00:40.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.764+0000 7f0942bff700 1 -- 172.21.15.23:0/315102753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 msgr2=0x7f093c0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.764+0000 7f0942bff700 1 --2- 172.21.15.23:0/315102753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 0x7f093c0cf1b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0928009840 tx=0x7f0928009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:40.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.764+0000 7f0942bff700 1 -- 172.21.15.23:0/315102753 shutdown_connections 2023-06-22T08:00:40.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.764+0000 7f0942bff700 1 --2- 172.21.15.23:0/315102753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 0x7f093c0cf1b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.764+0000 7f0942bff700 1 -- 172.21.15.23:0/315102753 >> 172.21.15.23:0/315102753 conn(0x7f093c06ae30 msgr2=0x7f093c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:40.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.764+0000 7f0942bff700 1 -- 172.21.15.23:0/315102753 shutdown_connections 2023-06-22T08:00:40.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.764+0000 7f0942bff700 1 -- 172.21.15.23:0/315102753 wait complete. 2023-06-22T08:00:40.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.766+0000 7f0942bff700 1 Processor -- start 2023-06-22T08:00:40.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.767+0000 7f0942bff700 1 -- start start 2023-06-22T08:00:40.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f0942bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f093c0c40b0 0x7f093c1e38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f0942bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 0x7f093c1512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f0942bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f093c152ce0 0x7f093c0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f0942bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c1ef3d0 con 0x7f093c0c9fd0 2023-06-22T08:00:40.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f0942bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c0cdff0 con 0x7f093c0c40b0 2023-06-22T08:00:40.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f0942bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093c1ee750 con 0x7f093c152ce0 2023-06-22T08:00:40.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f093bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 0x7f093c1512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f093bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 0x7f093c1512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33400/0 (socket says 172.21.15.23:33400) 2023-06-22T08:00:40.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f093bfff700 1 -- 172.21.15.23:0/3541315974 learned_addr learned my addr 172.21.15.23:0/3541315974 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:40.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f094099b700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f093c0c40b0 0x7f093c1e38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.768+0000 7f094119c700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f093c152ce0 0x7f093c0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f093bfff700 1 -- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f093c152ce0 msgr2=0x7f093c0cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f093bfff700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f093c152ce0 0x7f093c0cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f093bfff700 1 -- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f093c0c40b0 msgr2=0x7f093c1e38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f093bfff700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f093c0c40b0 0x7f093c1e38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f093bfff700 1 -- 172.21.15.23:0/3541315974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09280094a0 con 0x7f093c0c9fd0 2023-06-22T08:00:40.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f093bfff700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 0x7f093c1512f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f093c08bc30 tx=0x7f0928004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:40.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f09397fa700 1 -- 172.21.15.23:0/3541315974 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09280045d0 con 0x7f093c0c9fd0 2023-06-22T08:00:40.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f09397fa700 1 -- 172.21.15.23:0/3541315974 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092801f070 con 0x7f093c0c9fd0 2023-06-22T08:00:40.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f09397fa700 1 -- 172.21.15.23:0/3541315974 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09280038e0 con 0x7f093c0c9fd0 2023-06-22T08:00:40.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f0942bff700 1 -- 172.21.15.23:0/3541315974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093c1eea30 con 0x7f093c0c9fd0 2023-06-22T08:00:40.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.769+0000 7f0942bff700 1 -- 172.21.15.23:0/3541315974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093c1eeff0 con 0x7f093c0c9fd0 2023-06-22T08:00:40.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.770+0000 7f0942bff700 1 -- 172.21.15.23:0/3541315974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093c1ef1b0 con 0x7f093c0c9fd0 2023-06-22T08:00:40.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.771+0000 7f09397fa700 1 -- 172.21.15.23:0/3541315974 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0928003a40 con 0x7f093c0c9fd0 2023-06-22T08:00:40.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.772+0000 7f09397fa700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f092006ace0 0x7f092006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:40.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.772+0000 7f09397fa700 1 -- 172.21.15.23:0/3541315974 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0928021070 con 0x7f093c0c9fd0 2023-06-22T08:00:40.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.773+0000 7f094099b700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f092006ace0 0x7f092006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:40.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.773+0000 7f094099b700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f092006ace0 0x7f092006d190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0930007fd0 tx=0x7f0930009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:40.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.777+0000 7f09397fa700 1 -- 172.21.15.23:0/3541315974 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0928055f90 con 0x7f093c0c9fd0 2023-06-22T08:00:40.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.989+0000 7f0942bff700 1 -- 172.21.15.23:0/3541315974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f093c151c40 con 0x7f093c0c9fd0 2023-06-22T08:00:40.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.991+0000 7f09397fa700 1 -- 172.21.15.23:0/3541315974 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f09280595b0 con 0x7f093c0c9fd0 2023-06-22T08:00:40.994 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:40.994 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:40.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 -- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f092006ace0 msgr2=0x7f092006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f092006ace0 0x7f092006d190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0930007fd0 tx=0x7f0930009040 comp rx=0 tx=0).stop 2023-06-22T08:00:40.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 -- 172.21.15.23:0/3541315974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 msgr2=0x7f093c1512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:40.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 0x7f093c1512f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f093c08bc30 tx=0x7f0928004750 comp rx=0 tx=0).stop 2023-06-22T08:00:40.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 -- 172.21.15.23:0/3541315974 shutdown_connections 2023-06-22T08:00:40.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f093c152ce0 0x7f093c0cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f092006ace0 0x7f092006d190 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f093c0c40b0 0x7f093c1e38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 --2- 172.21.15.23:0/3541315974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093c0c9fd0 0x7f093c1512f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:40.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 -- 172.21.15.23:0/3541315974 >> 172.21.15.23:0/3541315974 conn(0x7f093c06ae30 msgr2=0x7f093c15b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:40.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 -- 172.21.15.23:0/3541315974 shutdown_connections 2023-06-22T08:00:40.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:40.993+0000 7f0942bff700 1 -- 172.21.15.23:0/3541315974 wait complete. 2023-06-22T08:00:41.014 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T07:59:40.617747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-22T07:59:48.190528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-22T07:59:53.576494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-22T07:59:54.105472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-22T07:59:57.198896+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-22T08:00:41.015 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T08:00:41.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.213+0000 7fb1757ff700 1 Processor -- start 2023-06-22T08:00:41.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.213+0000 7fb1757ff700 1 -- start start 2023-06-22T08:00:41.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.214+0000 7fb1757ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb170152ba0 0x7fb1701593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb1757ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1700c9260 0x7fb1700c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb1757ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb170159930 con 0x7fb1700c4650 2023-06-22T08:00:41.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb1757ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1700c9aa0 con 0x7fb170152ba0 2023-06-22T08:00:41.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb1757ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17006aed0 con 0x7fb1700c9260 2023-06-22T08:00:41.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb16effd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb170152ba0 0x7fb1701593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb16effd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb170152ba0 0x7fb1701593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33414/0 (socket says 172.21.15.23:33414) 2023-06-22T08:00:41.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb16effd700 1 -- 172.21.15.23:0/50775694 learned_addr learned my addr 172.21.15.23:0/50775694 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:41.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb16e7fc700 1 --2- 172.21.15.23:0/50775694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1700c9260 0x7fb1700c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb16effd700 1 -- 172.21.15.23:0/50775694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1700c9260 msgr2=0x7fb1700c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:41.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb16effd700 1 --2- 172.21.15.23:0/50775694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1700c9260 0x7fb1700c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb16effd700 1 -- 172.21.15.23:0/50775694 >> v1:172.21.15.67:6789/0 conn(0x7fb1700c4650 legacy=0x7fb1700c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb16effd700 1 -- 172.21.15.23:0/50775694 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17006b1b0 con 0x7fb170152ba0 2023-06-22T08:00:41.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.215+0000 7fb16dffb700 1 -- 172.21.15.23:0/50775694 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363258218 0 0) 0x7fb168001490 con 0x7fb1700c4650 2023-06-22T08:00:41.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.216+0000 7fb16effd700 1 --2- 172.21.15.23:0/50775694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb170152ba0 0x7fb1701593f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb1600098f0 tx=0x7fb160009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986a5bb27bad40fb server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:41.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.216+0000 7fb16dffb700 1 -- 172.21.15.23:0/50775694 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160011830 con 0x7fb170152ba0 2023-06-22T08:00:41.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.216+0000 7fb16dffb700 1 -- 172.21.15.23:0/50775694 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb160011990 con 0x7fb170152ba0 2023-06-22T08:00:41.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.216+0000 7fb16dffb700 1 -- 172.21.15.23:0/50775694 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160011d20 con 0x7fb170152ba0 2023-06-22T08:00:41.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.216+0000 7fb1757ff700 1 -- 172.21.15.23:0/50775694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb170152ba0 msgr2=0x7fb1701593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.216+0000 7fb1757ff700 1 --2- 172.21.15.23:0/50775694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb170152ba0 0x7fb1701593f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb1600098f0 tx=0x7fb160009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:41.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.217+0000 7fb1757ff700 1 -- 172.21.15.23:0/50775694 shutdown_connections 2023-06-22T08:00:41.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.217+0000 7fb1757ff700 1 --2- 172.21.15.23:0/50775694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb170152ba0 0x7fb1701593f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.217+0000 7fb1757ff700 1 --2- 172.21.15.23:0/50775694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1700c9260 0x7fb1700c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.217+0000 7fb1757ff700 1 -- 172.21.15.23:0/50775694 >> 172.21.15.23:0/50775694 conn(0x7fb170065b90 msgr2=0x7fb170065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:41.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.217+0000 7fb1757ff700 1 -- 172.21.15.23:0/50775694 shutdown_connections 2023-06-22T08:00:41.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.217+0000 7fb1757ff700 1 -- 172.21.15.23:0/50775694 wait complete. 2023-06-22T08:00:41.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.218+0000 7fb1757ff700 1 Processor -- start 2023-06-22T08:00:41.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.219+0000 7fb1757ff700 1 -- start start 2023-06-22T08:00:41.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.219+0000 7fb1757ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1700c4650 0x7fb1700c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.219+0000 7fb1757ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1700c9260 0x7fb17014dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb1757ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb170152ba0 0x7fb1700c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb1757ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1700ce6c0 con 0x7fb1700c4650 2023-06-22T08:00:41.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb1757ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1701f2780 con 0x7fb1700c9260 2023-06-22T08:00:41.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb1757ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1701f1920 con 0x7fb170152ba0 2023-06-22T08:00:41.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1700c4650 0x7fb1700c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1700c4650 0x7fb1700c7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33420/0 (socket says 172.21.15.23:33420) 2023-06-22T08:00:41.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16effd700 1 -- 172.21.15.23:0/98938671 learned_addr learned my addr 172.21.15.23:0/98938671 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:41.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16e7fc700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1700c9260 0x7fb17014dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16f7fe700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb170152ba0 0x7fb1700c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16effd700 1 -- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb170152ba0 msgr2=0x7fb1700c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16effd700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb170152ba0 0x7fb1700c83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16effd700 1 -- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1700c9260 msgr2=0x7fb17014dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16effd700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1700c9260 0x7fb17014dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16effd700 1 -- 172.21.15.23:0/98938671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160009550 con 0x7fb1700c4650 2023-06-22T08:00:41.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.220+0000 7fb16effd700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1700c4650 0x7fb1700c7c20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb160009c40 tx=0x7fb160004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:41.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.221+0000 7fb1577fe700 1 -- 172.21.15.23:0/98938671 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160004550 con 0x7fb1700c4650 2023-06-22T08:00:41.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.221+0000 7fb1577fe700 1 -- 172.21.15.23:0/98938671 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16001f070 con 0x7fb1700c4650 2023-06-22T08:00:41.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.221+0000 7fb1577fe700 1 -- 172.21.15.23:0/98938671 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1600038e0 con 0x7fb1700c4650 2023-06-22T08:00:41.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.221+0000 7fb1757ff700 1 -- 172.21.15.23:0/98938671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1701f1c00 con 0x7fb1700c4650 2023-06-22T08:00:41.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.221+0000 7fb1757ff700 1 -- 172.21.15.23:0/98938671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1701f21c0 con 0x7fb1700c4650 2023-06-22T08:00:41.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.222+0000 7fb1757ff700 1 -- 172.21.15.23:0/98938671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1700cbed0 con 0x7fb1700c4650 2023-06-22T08:00:41.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.223+0000 7fb1577fe700 1 -- 172.21.15.23:0/98938671 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb160003a40 con 0x7fb1700c4650 2023-06-22T08:00:41.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.224+0000 7fb1577fe700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb15c06ace0 0x7fb15c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.224+0000 7fb1577fe700 1 -- 172.21.15.23:0/98938671 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb160021070 con 0x7fb1700c4650 2023-06-22T08:00:41.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.224+0000 7fb16e7fc700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb15c06ace0 0x7fb15c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.225+0000 7fb16e7fc700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb15c06ace0 0x7fb15c06d190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb170151e20 tx=0x7fb164009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:41.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.228+0000 7fb1577fe700 1 -- 172.21.15.23:0/98938671 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb160055d80 con 0x7fb1700c4650 2023-06-22T08:00:41.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.439+0000 7fb1757ff700 1 -- 172.21.15.23:0/98938671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb1701500a0 con 0x7fb1700c4650 2023-06-22T08:00:41.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.439+0000 7fb1577fe700 1 -- 172.21.15.23:0/98938671 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1600150a0 con 0x7fb1700c4650 2023-06-22T08:00:41.443 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-06-22T08:00:41.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 -- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb15c06ace0 msgr2=0x7fb15c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb15c06ace0 0x7fb15c06d190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb170151e20 tx=0x7fb164009210 comp rx=0 tx=0).stop 2023-06-22T08:00:41.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 -- 172.21.15.23:0/98938671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1700c4650 msgr2=0x7fb1700c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1700c4650 0x7fb1700c7c20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb160009c40 tx=0x7fb160004750 comp rx=0 tx=0).stop 2023-06-22T08:00:41.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 -- 172.21.15.23:0/98938671 shutdown_connections 2023-06-22T08:00:41.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb170152ba0 0x7fb1700c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1700c4650 0x7fb1700c7c20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb15c06ace0 0x7fb15c06d190 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 --2- 172.21.15.23:0/98938671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1700c9260 0x7fb17014dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 -- 172.21.15.23:0/98938671 >> 172.21.15.23:0/98938671 conn(0x7fb170065b90 msgr2=0x7fb170150bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:41.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 -- 172.21.15.23:0/98938671 shutdown_connections 2023-06-22T08:00:41.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.442+0000 7fb1757ff700 1 -- 172.21.15.23:0/98938671 wait complete. 2023-06-22T08:00:41.463 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T08:00:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.670+0000 7f329cdff700 1 Processor -- start 2023-06-22T08:00:41.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.670+0000 7f329cdff700 1 -- start start 2023-06-22T08:00:41.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.672+0000 7f329cdff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32981593f0 0x7f32980c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.672+0000 7f329cdff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32980c9aa0 con 0x7f32980c4650 2023-06-22T08:00:41.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.672+0000 7f329cdff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f329806aed0 con 0x7f3298152ba0 2023-06-22T08:00:41.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.672+0000 7f329cdff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32980c4a60 con 0x7f32981593f0 2023-06-22T08:00:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.672+0000 7f329659c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3298152ba0 0x7f32980c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:41794/0 (socket says 172.21.15.23:41794) 2023-06-22T08:00:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.672+0000 7f3295d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32981593f0 0x7f32980c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.672+0000 7f329659c700 1 -- 172.21.15.23:0/4182003170 learned_addr learned my addr 172.21.15.23:0/4182003170 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:41.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.673+0000 7f329559a700 1 -- 172.21.15.23:0/4182003170 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171196492 0 0) 0x7f329806aed0 con 0x7f3298152ba0 2023-06-22T08:00:41.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.673+0000 7f329559a700 1 -- 172.21.15.23:0/4182003170 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32800035b0 con 0x7f3298152ba0 2023-06-22T08:00:41.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.673+0000 7f329559a700 1 -- 172.21.15.23:0/4182003170 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533510231 0 0) 0x7f32980c9aa0 con 0x7f32980c4650 2023-06-22T08:00:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.673+0000 7f3295d9b700 1 -- 172.21.15.23:0/4182003170 >> v1:172.21.15.67:6789/0 conn(0x7f3298152ba0 legacy=0x7f32980c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.673+0000 7f3295d9b700 1 -- 172.21.15.23:0/4182003170 >> v1:172.21.15.23:6789/0 conn(0x7f32980c4650 legacy=0x7f32980c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.673+0000 7f3295d9b700 1 -- 172.21.15.23:0/4182003170 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329806b1b0 con 0x7f32981593f0 2023-06-22T08:00:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.673+0000 7f3295d9b700 1 --2- 172.21.15.23:0/4182003170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32981593f0 0x7f32980c4110 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f328c009840 tx=0x7f328c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b795bacb2fe0ead4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.673+0000 7f329559a700 1 -- 172.21.15.23:0/4182003170 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c011830 con 0x7f32981593f0 2023-06-22T08:00:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.673+0000 7f329559a700 1 -- 172.21.15.23:0/4182003170 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c011990 con 0x7f32981593f0 2023-06-22T08:00:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.673+0000 7f329559a700 1 -- 172.21.15.23:0/4182003170 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c011d20 con 0x7f32981593f0 2023-06-22T08:00:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.674+0000 7f329cdff700 1 -- 172.21.15.23:0/4182003170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32981593f0 msgr2=0x7f32980c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.674+0000 7f329cdff700 1 --2- 172.21.15.23:0/4182003170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32981593f0 0x7f32980c4110 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f328c009840 tx=0x7f328c009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.674+0000 7f329cdff700 1 -- 172.21.15.23:0/4182003170 shutdown_connections 2023-06-22T08:00:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.674+0000 7f329cdff700 1 --2- 172.21.15.23:0/4182003170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32981593f0 0x7f32980c4110 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.674+0000 7f329cdff700 1 -- 172.21.15.23:0/4182003170 >> 172.21.15.23:0/4182003170 conn(0x7f3298065b90 msgr2=0x7f3298065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.674+0000 7f329cdff700 1 -- 172.21.15.23:0/4182003170 shutdown_connections 2023-06-22T08:00:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.674+0000 7f329cdff700 1 -- 172.21.15.23:0/4182003170 wait complete. 2023-06-22T08:00:41.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.676+0000 7f329cdff700 1 Processor -- start 2023-06-22T08:00:41.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.677+0000 7f329cdff700 1 -- start start 2023-06-22T08:00:41.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f329cdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f32980c4650 0x7f3298154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f329cdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3298152ba0 0x7f32980ce510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f329cdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32981593f0 0x7f32980c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f329cdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3298156580 con 0x7f3298152ba0 2023-06-22T08:00:41.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f329cdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3298155170 con 0x7f32981593f0 2023-06-22T08:00:41.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f329cdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32980c8ba0 con 0x7f32980c4650 2023-06-22T08:00:41.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f329659c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f32980c4650 0x7f3298154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f3295d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3298152ba0 0x7f32980ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f3295d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3298152ba0 0x7f32980ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33436/0 (socket says 172.21.15.23:33436) 2023-06-22T08:00:41.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f3295d9b700 1 -- 172.21.15.23:0/1004370329 learned_addr learned my addr 172.21.15.23:0/1004370329 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:41.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.678+0000 7f3296d9d700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32981593f0 0x7f32980c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f3295d9b700 1 -- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f32980c4650 msgr2=0x7f3298154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f3295d9b700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f32980c4650 0x7f3298154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f3295d9b700 1 -- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32981593f0 msgr2=0x7f32980c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f3295d9b700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32981593f0 0x7f32980c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f3295d9b700 1 -- 172.21.15.23:0/1004370329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c0094a0 con 0x7f3298152ba0 2023-06-22T08:00:41.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f3295d9b700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3298152ba0 0x7f32980ce510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f328c0090b0 tx=0x7f328c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:41.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f3286ffd700 1 -- 172.21.15.23:0/1004370329 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c01a7e0 con 0x7f3298152ba0 2023-06-22T08:00:41.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f3286ffd700 1 -- 172.21.15.23:0/1004370329 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c01a940 con 0x7f3298152ba0 2023-06-22T08:00:41.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f3286ffd700 1 -- 172.21.15.23:0/1004370329 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c01acd0 con 0x7f3298152ba0 2023-06-22T08:00:41.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f329cdff700 1 -- 172.21.15.23:0/1004370329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32981562a0 con 0x7f3298152ba0 2023-06-22T08:00:41.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.679+0000 7f329cdff700 1 -- 172.21.15.23:0/1004370329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32981535a0 con 0x7f3298152ba0 2023-06-22T08:00:41.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.680+0000 7f329cdff700 1 -- 172.21.15.23:0/1004370329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32980cbed0 con 0x7f3298152ba0 2023-06-22T08:00:41.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.681+0000 7f3286ffd700 1 -- 172.21.15.23:0/1004370329 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f328c022070 con 0x7f3298152ba0 2023-06-22T08:00:41.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.682+0000 7f3286ffd700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f327806ad10 0x7f327806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:41.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.682+0000 7f3286ffd700 1 -- 172.21.15.23:0/1004370329 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f328c08a5b0 con 0x7f3298152ba0 2023-06-22T08:00:41.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.682+0000 7f329659c700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f327806ad10 0x7f327806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:41.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.683+0000 7f329659c700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f327806ad10 0x7f327806d1c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3288007fd0 tx=0x7f3288007d80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:41.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.687+0000 7f3286ffd700 1 -- 172.21.15.23:0/1004370329 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f328c055480 con 0x7f3298152ba0 2023-06-22T08:00:41.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.893+0000 7f329cdff700 1 -- 172.21.15.23:0/1004370329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f329804ea20 con 0x7f3298152ba0 2023-06-22T08:00:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.893+0000 7f3286ffd700 1 -- 172.21.15.23:0/1004370329 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7f328c0150a0 con 0x7f3298152ba0 2023-06-22T08:00:41.897 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2023-06-22T08:00:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 -- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f327806ad10 msgr2=0x7f327806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f327806ad10 0x7f327806d1c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3288007fd0 tx=0x7f3288007d80 comp rx=0 tx=0).stop 2023-06-22T08:00:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 -- 172.21.15.23:0/1004370329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3298152ba0 msgr2=0x7f32980ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3298152ba0 0x7f32980ce510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f328c0090b0 tx=0x7f328c004750 comp rx=0 tx=0).stop 2023-06-22T08:00:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 -- 172.21.15.23:0/1004370329 shutdown_connections 2023-06-22T08:00:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f327806ad10 0x7f327806d1c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32981593f0 0x7f32980c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f32980c4650 0x7f3298154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 --2- 172.21.15.23:0/1004370329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3298152ba0 0x7f32980ce510 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 -- 172.21.15.23:0/1004370329 >> 172.21.15.23:0/1004370329 conn(0x7f3298065b90 msgr2=0x7f3298157260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 -- 172.21.15.23:0/1004370329 shutdown_connections 2023-06-22T08:00:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:41.896+0000 7f329cdff700 1 -- 172.21.15.23:0/1004370329 wait complete. 2023-06-22T08:00:41.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-22T08:00:42.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.116+0000 7fb2003ff700 1 Processor -- start 2023-06-22T08:00:42.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.116+0000 7fb2003ff700 1 -- start start 2023-06-22T08:00:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb2003ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb1f81593f0 0x7fb1f80c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb2003ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f80c9aa0 con 0x7fb1f8152ba0 2023-06-22T08:00:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb2003ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f806aed0 con 0x7fb1f80c4650 2023-06-22T08:00:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb2003ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f80c4a60 con 0x7fb1f81593f0 2023-06-22T08:00:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb1fe19b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb1f8152ba0 0x7fb1f80c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:32768/0 (socket says 172.21.15.23:32768) 2023-06-22T08:00:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb1fe19b700 1 -- 172.21.15.23:0/2340818582 learned_addr learned my addr 172.21.15.23:0/2340818582 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:42.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb1fd99a700 1 --2- 172.21.15.23:0/2340818582 >> v2:172.21.15.67:3300/0 conn(0x7fb1f81593f0 0x7fb1f80c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:42.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb1fd199700 1 -- 172.21.15.23:0/2340818582 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385593497 0 0) 0x7fb1f80c9aa0 con 0x7fb1f8152ba0 2023-06-22T08:00:42.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb1fd199700 1 -- 172.21.15.23:0/2340818582 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e40035b0 con 0x7fb1f8152ba0 2023-06-22T08:00:42.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb1fd199700 1 -- 172.21.15.23:0/2340818582 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060519404 0 0) 0x7fb1f806aed0 con 0x7fb1f80c4650 2023-06-22T08:00:42.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.118+0000 7fb1fd199700 1 -- 172.21.15.23:0/2340818582 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f80c9aa0 con 0x7fb1f80c4650 2023-06-22T08:00:42.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.119+0000 7fb1fd99a700 1 -- 172.21.15.23:0/2340818582 >> v1:172.21.15.67:6789/0 conn(0x7fb1f80c4650 legacy=0x7fb1f80c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.119+0000 7fb1fd99a700 1 -- 172.21.15.23:0/2340818582 >> v1:172.21.15.23:6789/0 conn(0x7fb1f8152ba0 legacy=0x7fb1f80c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.119+0000 7fb1fd99a700 1 -- 172.21.15.23:0/2340818582 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f806b1b0 con 0x7fb1f81593f0 2023-06-22T08:00:42.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.119+0000 7fb1fd99a700 1 --2- 172.21.15.23:0/2340818582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f81593f0 0x7fb1f80c4110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009840 tx=0x7fb1e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4376c64347cae42a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:42.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.119+0000 7fb1fd199700 1 -- 172.21.15.23:0/2340818582 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8011830 con 0x7fb1f81593f0 2023-06-22T08:00:42.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.119+0000 7fb1fd199700 1 -- 172.21.15.23:0/2340818582 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e8011990 con 0x7fb1f81593f0 2023-06-22T08:00:42.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.119+0000 7fb1fd199700 1 -- 172.21.15.23:0/2340818582 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8011d20 con 0x7fb1f81593f0 2023-06-22T08:00:42.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.120+0000 7fb2003ff700 1 -- 172.21.15.23:0/2340818582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f81593f0 msgr2=0x7fb1f80c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.120+0000 7fb2003ff700 1 --2- 172.21.15.23:0/2340818582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f81593f0 0x7fb1f80c4110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009840 tx=0x7fb1e8009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:42.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.120+0000 7fb2003ff700 1 -- 172.21.15.23:0/2340818582 shutdown_connections 2023-06-22T08:00:42.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.120+0000 7fb2003ff700 1 --2- 172.21.15.23:0/2340818582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f81593f0 0x7fb1f80c4110 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.120+0000 7fb2003ff700 1 -- 172.21.15.23:0/2340818582 >> 172.21.15.23:0/2340818582 conn(0x7fb1f8065b90 msgr2=0x7fb1f8065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:42.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.120+0000 7fb2003ff700 1 -- 172.21.15.23:0/2340818582 shutdown_connections 2023-06-22T08:00:42.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.120+0000 7fb2003ff700 1 -- 172.21.15.23:0/2340818582 wait complete. 2023-06-22T08:00:42.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.122+0000 7fb2003ff700 1 Processor -- start 2023-06-22T08:00:42.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.122+0000 7fb2003ff700 1 -- start start 2023-06-22T08:00:42.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb2003ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f80c4650 0x7fb1f81eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:42.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb2003ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1f8152ba0 0x7fb1f80b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:42.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb2003ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1f81593f0 0x7fb1f81f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:42.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb2003ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f81ef080 con 0x7fb1f8152ba0 2023-06-22T08:00:42.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb2003ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f81ee6b0 con 0x7fb1f80c4650 2023-06-22T08:00:42.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb2003ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f81ee0c0 con 0x7fb1f81593f0 2023-06-22T08:00:42.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb1fd99a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1f8152ba0 0x7fb1f80b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:42.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb1fd99a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1f8152ba0 0x7fb1f80b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33450/0 (socket says 172.21.15.23:33450) 2023-06-22T08:00:42.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb1fe19b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f80c4650 0x7fb1f81eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:42.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb1fd99a700 1 -- 172.21.15.23:0/3771990570 learned_addr learned my addr 172.21.15.23:0/3771990570 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:42.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.123+0000 7fb1fe99c700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1f81593f0 0x7fb1f81f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:42.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb1fd99a700 1 -- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1f81593f0 msgr2=0x7fb1f81f1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb1fd99a700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1f81593f0 0x7fb1f81f1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb1fd99a700 1 -- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f80c4650 msgr2=0x7fb1f81eeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb1fd99a700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f80c4650 0x7fb1f81eeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb1fd99a700 1 -- 172.21.15.23:0/3771990570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e80094a0 con 0x7fb1f8152ba0 2023-06-22T08:00:42.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb1fe19b700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f80c4650 0x7fb1f81eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:42.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb1fd99a700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1f8152ba0 0x7fb1f80b24c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80090b0 tx=0x7fb1e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:42.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb1eeffd700 1 -- 172.21.15.23:0/3771990570 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e801a6e0 con 0x7fb1f8152ba0 2023-06-22T08:00:42.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb1eeffd700 1 -- 172.21.15.23:0/3771990570 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e801a840 con 0x7fb1f8152ba0 2023-06-22T08:00:42.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb2003ff700 1 -- 172.21.15.23:0/3771990570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f8155fd0 con 0x7fb1f8152ba0 2023-06-22T08:00:42.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb1eeffd700 1 -- 172.21.15.23:0/3771990570 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e801abb0 con 0x7fb1f8152ba0 2023-06-22T08:00:42.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.124+0000 7fb2003ff700 1 -- 172.21.15.23:0/3771990570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f8156a80 con 0x7fb1f8152ba0 2023-06-22T08:00:42.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.125+0000 7fb2003ff700 1 -- 172.21.15.23:0/3771990570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f80cbed0 con 0x7fb1f8152ba0 2023-06-22T08:00:42.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.126+0000 7fb1eeffd700 1 -- 172.21.15.23:0/3771990570 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb1e8022070 con 0x7fb1f8152ba0 2023-06-22T08:00:42.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.128+0000 7fb1eeffd700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb1dc06ace0 0x7fb1dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:42.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.128+0000 7fb1eeffd700 1 -- 172.21.15.23:0/3771990570 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb1e808a3b0 con 0x7fb1f8152ba0 2023-06-22T08:00:42.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.128+0000 7fb1fe19b700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb1dc06ace0 0x7fb1dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:42.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.129+0000 7fb1fe19b700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb1dc06ace0 0x7fb1dc06d190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4007fd0 tx=0x7fb1f4007e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:42.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.132+0000 7fb1eeffd700 1 -- 172.21.15.23:0/3771990570 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb1e8055210 con 0x7fb1f8152ba0 2023-06-22T08:00:42.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.338+0000 7fb2003ff700 1 -- 172.21.15.23:0/3771990570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb1f804ea20 con 0x7fb1f8152ba0 2023-06-22T08:00:42.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.339+0000 7fb1eeffd700 1 -- 172.21.15.23:0/3771990570 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb1e8015020 con 0x7fb1f8152ba0 2023-06-22T08:00:42.342 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:42.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 -- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb1dc06ace0 msgr2=0x7fb1dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb1dc06ace0 0x7fb1dc06d190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4007fd0 tx=0x7fb1f4007e00 comp rx=0 tx=0).stop 2023-06-22T08:00:42.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 -- 172.21.15.23:0/3771990570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1f8152ba0 msgr2=0x7fb1f80b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1f8152ba0 0x7fb1f80b24c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80090b0 tx=0x7fb1e8004750 comp rx=0 tx=0).stop 2023-06-22T08:00:42.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 -- 172.21.15.23:0/3771990570 shutdown_connections 2023-06-22T08:00:42.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1f81593f0 0x7fb1f81f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb1dc06ace0 0x7fb1dc06d190 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f80c4650 0x7fb1f81eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 --2- 172.21.15.23:0/3771990570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1f8152ba0 0x7fb1f80b24c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 -- 172.21.15.23:0/3771990570 >> 172.21.15.23:0/3771990570 conn(0x7fb1f8065b90 msgr2=0x7fb1f8151540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:42.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 -- 172.21.15.23:0/3771990570 shutdown_connections 2023-06-22T08:00:42.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.341+0000 7fb2003ff700 1 -- 172.21.15.23:0/3771990570 wait complete. 2023-06-22T08:00:42.362 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-22T08:00:42.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.561+0000 7efcf6fff700 1 Processor -- start 2023-06-22T08:00:42.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.561+0000 7efcf6fff700 1 -- start start 2023-06-22T08:00:42.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.563+0000 7efcf6fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efcf00c9ee0 0x7efcf00ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:42.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.563+0000 7efcf6fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf00c9aa0 con 0x7efcf0152ba0 2023-06-22T08:00:42.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.563+0000 7efcf6fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf006aed0 con 0x7efcf01593f0 2023-06-22T08:00:42.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.563+0000 7efcf6fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf006b1b0 con 0x7efcf00c9ee0 2023-06-22T08:00:42.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.563+0000 7efcf4d9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7efcf0152ba0 0x7efcf00c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:32780/0 (socket says 172.21.15.23:32780) 2023-06-22T08:00:42.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.563+0000 7efcf4d9b700 1 -- 172.21.15.23:0/2336845459 learned_addr learned my addr 172.21.15.23:0/2336845459 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:42.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.563+0000 7efcf559c700 1 --2- 172.21.15.23:0/2336845459 >> v2:172.21.15.23:3300/0 conn(0x7efcf00c9ee0 0x7efcf00ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.563+0000 7efcf559c700 1 -- 172.21.15.23:0/2336845459 >> v1:172.21.15.67:6789/0 conn(0x7efcf01593f0 legacy=0x7efcf00c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.563+0000 7efcf559c700 1 -- 172.21.15.23:0/2336845459 >> v1:172.21.15.23:6789/0 conn(0x7efcf0152ba0 legacy=0x7efcf00c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.563+0000 7efcf559c700 1 -- 172.21.15.23:0/2336845459 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf006b490 con 0x7efcf00c9ee0 2023-06-22T08:00:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.564+0000 7efcf559c700 1 --2- 172.21.15.23:0/2336845459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcf00c9ee0 0x7efcf00ca2f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efce8009d90 tx=0x7efce800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357ef048eba78a46 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.564+0000 7efcef7fe700 1 -- 172.21.15.23:0/2336845459 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce800e070 con 0x7efcf00c9ee0 2023-06-22T08:00:42.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.564+0000 7efcef7fe700 1 -- 172.21.15.23:0/2336845459 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce800bbd0 con 0x7efcf00c9ee0 2023-06-22T08:00:42.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.564+0000 7efcef7fe700 1 -- 172.21.15.23:0/2336845459 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce8012430 con 0x7efcf00c9ee0 2023-06-22T08:00:42.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.564+0000 7efcf6fff700 1 -- 172.21.15.23:0/2336845459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcf00c9ee0 msgr2=0x7efcf00ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.564+0000 7efcf6fff700 1 --2- 172.21.15.23:0/2336845459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcf00c9ee0 0x7efcf00ca2f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efce8009d90 tx=0x7efce800b330 comp rx=0 tx=0).stop 2023-06-22T08:00:42.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.564+0000 7efcf6fff700 1 -- 172.21.15.23:0/2336845459 shutdown_connections 2023-06-22T08:00:42.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.564+0000 7efcf6fff700 1 --2- 172.21.15.23:0/2336845459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcf00c9ee0 0x7efcf00ca2f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.569 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.564+0000 7efcf6fff700 1 -- 172.21.15.23:0/2336845459 >> 172.21.15.23:0/2336845459 conn(0x7efcf0065b90 msgr2=0x7efcf0065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:42.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.565+0000 7efcf6fff700 1 -- 172.21.15.23:0/2336845459 shutdown_connections 2023-06-22T08:00:42.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.565+0000 7efcf6fff700 1 -- 172.21.15.23:0/2336845459 wait complete. 2023-06-22T08:00:42.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.566+0000 7efcf6fff700 1 Processor -- start 2023-06-22T08:00:42.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.566+0000 7efcf6fff700 1 -- start start 2023-06-22T08:00:42.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.567+0000 7efcf6fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efcf00c9ee0 0x7efcf00cea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:42.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.567+0000 7efcf6fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcf0152ba0 0x7efcf00c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.567+0000 7efcf6fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf01593f0 0x7efcf00c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.567+0000 7efcf6fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf0155b30 con 0x7efcf0152ba0 2023-06-22T08:00:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.567+0000 7efcf6fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf01515e0 con 0x7efcf01593f0 2023-06-22T08:00:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.567+0000 7efcf6fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf014fbc0 con 0x7efcf00c9ee0 2023-06-22T08:00:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efcf4d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efcf00c9ee0 0x7efcf00cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efceffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcf0152ba0 0x7efcf00c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efcf4d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efcf00c9ee0 0x7efcf00cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:51264/0 (socket says 172.21.15.23:51264) 2023-06-22T08:00:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efcf4d9b700 1 -- 172.21.15.23:0/826927405 learned_addr learned my addr 172.21.15.23:0/826927405 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efcf559c700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf01593f0 0x7efcf00c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efcf4d9b700 1 -- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf01593f0 msgr2=0x7efcf00c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efcf4d9b700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf01593f0 0x7efcf00c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efcf4d9b700 1 -- 172.21.15.23:0/826927405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcf0152ba0 msgr2=0x7efcf00c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efcf4d9b700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcf0152ba0 0x7efcf00c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efcf4d9b700 1 -- 172.21.15.23:0/826927405 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce8009a70 con 0x7efcf00c9ee0 2023-06-22T08:00:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efceffff700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcf0152ba0 0x7efcf00c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.568+0000 7efcf4d9b700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efcf00c9ee0 0x7efcf00cea60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efce400c520 tx=0x7efce400c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.569+0000 7efced7fa700 1 -- 172.21.15.23:0/826927405 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4005d70 con 0x7efcf00c9ee0 2023-06-22T08:00:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.569+0000 7efced7fa700 1 -- 172.21.15.23:0/826927405 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce40070d0 con 0x7efcf00c9ee0 2023-06-22T08:00:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.569+0000 7efced7fa700 1 -- 172.21.15.23:0/826927405 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4007440 con 0x7efcf00c9ee0 2023-06-22T08:00:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.569+0000 7efcf6fff700 1 -- 172.21.15.23:0/826927405 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf0155200 con 0x7efcf00c9ee0 2023-06-22T08:00:42.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.569+0000 7efcf6fff700 1 -- 172.21.15.23:0/826927405 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf01557c0 con 0x7efcf00c9ee0 2023-06-22T08:00:42.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.570+0000 7efcf6fff700 1 -- 172.21.15.23:0/826927405 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf00cbed0 con 0x7efcf00c9ee0 2023-06-22T08:00:42.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.571+0000 7efced7fa700 1 -- 172.21.15.23:0/826927405 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efce40075a0 con 0x7efcf00c9ee0 2023-06-22T08:00:42.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.572+0000 7efced7fa700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcd406ace0 0x7efcd406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:42.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.572+0000 7efced7fa700 1 -- 172.21.15.23:0/826927405 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7efce4088d20 con 0x7efcf00c9ee0 2023-06-22T08:00:42.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.572+0000 7efceffff700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcd406ace0 0x7efcd406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.573+0000 7efceffff700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcd406ace0 0x7efcd406d190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efcf00c5860 tx=0x7efcdc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:42.579 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.575+0000 7efced7fa700 1 -- 172.21.15.23:0/826927405 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efce4053ae0 con 0x7efcf00c9ee0 2023-06-22T08:00:42.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.790+0000 7efcf6fff700 1 -- 172.21.15.23:0/826927405 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efcf0155ca0 con 0x7efcf00c9ee0 2023-06-22T08:00:42.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.791+0000 7efced7fa700 1 -- 172.21.15.23:0/826927405 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7efce4017020 con 0x7efcf00c9ee0 2023-06-22T08:00:42.794 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-06-22T08:00:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.793+0000 7efcf6fff700 1 -- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcd406ace0 msgr2=0x7efcd406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.793+0000 7efcf6fff700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcd406ace0 0x7efcd406d190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efcf00c5860 tx=0x7efcdc009040 comp rx=0 tx=0).stop 2023-06-22T08:00:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.794+0000 7efcf6fff700 1 -- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efcf00c9ee0 msgr2=0x7efcf00cea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.794+0000 7efcf6fff700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efcf00c9ee0 0x7efcf00cea60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efce400c520 tx=0x7efce400c830 comp rx=0 tx=0).stop 2023-06-22T08:00:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.794+0000 7efcf6fff700 1 -- 172.21.15.23:0/826927405 shutdown_connections 2023-06-22T08:00:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.794+0000 7efcf6fff700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efcd406ace0 0x7efcd406d190 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.794+0000 7efcf6fff700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcf01593f0 0x7efcf00c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.794+0000 7efcf6fff700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efcf00c9ee0 0x7efcf00cea60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.794+0000 7efcf6fff700 1 --2- 172.21.15.23:0/826927405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcf0152ba0 0x7efcf00c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.794+0000 7efcf6fff700 1 -- 172.21.15.23:0/826927405 >> 172.21.15.23:0/826927405 conn(0x7efcf0065b90 msgr2=0x7efcf00c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.794+0000 7efcf6fff700 1 -- 172.21.15.23:0/826927405 shutdown_connections 2023-06-22T08:00:42.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:42.794+0000 7efcf6fff700 1 -- 172.21.15.23:0/826927405 wait complete. 2023-06-22T08:00:42.814 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-22T08:00:43.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.014+0000 7fb3997ff700 1 Processor -- start 2023-06-22T08:00:43.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.014+0000 7fb3997ff700 1 -- start start 2023-06-22T08:00:43.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.015+0000 7fb3997ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb3941593f0 0x7fb3940c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.015+0000 7fb3997ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 0x7fb3940c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.015+0000 7fb3997ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3940c9aa0 con 0x7fb394152ba0 2023-06-22T08:00:43.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.015+0000 7fb3997ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39406aed0 con 0x7fb3941593f0 2023-06-22T08:00:43.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.016+0000 7fb3997ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39406b1b0 con 0x7fb3940c4650 2023-06-22T08:00:43.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.016+0000 7fb3927fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb3941593f0 0x7fb3940c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.016+0000 7fb3927fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb3941593f0 0x7fb3940c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33488/0 (socket says 172.21.15.23:33488) 2023-06-22T08:00:43.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.016+0000 7fb3927fc700 1 -- 172.21.15.23:0/3844844086 learned_addr learned my addr 172.21.15.23:0/3844844086 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:43.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.016+0000 7fb3937fe700 1 --2- 172.21.15.23:0/3844844086 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 0x7fb3940c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.016+0000 7fb3927fc700 1 -- 172.21.15.23:0/3844844086 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 msgr2=0x7fb3940c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:43.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.016+0000 7fb3927fc700 1 --2- 172.21.15.23:0/3844844086 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 0x7fb3940c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.016+0000 7fb3927fc700 1 -- 172.21.15.23:0/3844844086 >> v1:172.21.15.23:6789/0 conn(0x7fb394152ba0 legacy=0x7fb3940c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.016+0000 7fb3927fc700 1 -- 172.21.15.23:0/3844844086 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39406b490 con 0x7fb3941593f0 2023-06-22T08:00:43.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.016+0000 7fb391ffb700 1 -- 172.21.15.23:0/3844844086 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042321330 0 0) 0x7fb3940c9aa0 con 0x7fb394152ba0 2023-06-22T08:00:43.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb3927fc700 1 --2- 172.21.15.23:0/3844844086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3941593f0 0x7fb3940c4110 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb3880098f0 tx=0x7fb388009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405dc28295a6157 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:43.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb391ffb700 1 -- 172.21.15.23:0/3844844086 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388011830 con 0x7fb3941593f0 2023-06-22T08:00:43.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb391ffb700 1 -- 172.21.15.23:0/3844844086 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb388011990 con 0x7fb3941593f0 2023-06-22T08:00:43.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb391ffb700 1 -- 172.21.15.23:0/3844844086 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388011d20 con 0x7fb3941593f0 2023-06-22T08:00:43.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb3997ff700 1 -- 172.21.15.23:0/3844844086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3941593f0 msgr2=0x7fb3940c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb3997ff700 1 --2- 172.21.15.23:0/3844844086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3941593f0 0x7fb3940c4110 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb3880098f0 tx=0x7fb388009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:43.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb3997ff700 1 -- 172.21.15.23:0/3844844086 shutdown_connections 2023-06-22T08:00:43.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb3997ff700 1 --2- 172.21.15.23:0/3844844086 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 0x7fb3940c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb3997ff700 1 --2- 172.21.15.23:0/3844844086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3941593f0 0x7fb3940c4110 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb3997ff700 1 -- 172.21.15.23:0/3844844086 >> 172.21.15.23:0/3844844086 conn(0x7fb394065b90 msgr2=0x7fb394065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:43.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.017+0000 7fb3997ff700 1 -- 172.21.15.23:0/3844844086 shutdown_connections 2023-06-22T08:00:43.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.018+0000 7fb3997ff700 1 -- 172.21.15.23:0/3844844086 wait complete. 2023-06-22T08:00:43.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.019+0000 7fb3997ff700 1 Processor -- start 2023-06-22T08:00:43.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.019+0000 7fb3997ff700 1 -- start start 2023-06-22T08:00:43.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.020+0000 7fb3997ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 0x7fb3940ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.020+0000 7fb3997ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394152ba0 0x7fb3940cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.020+0000 7fb3997ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3941593f0 0x7fb3941f1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.020+0000 7fb3997ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940cca10 con 0x7fb394152ba0 2023-06-22T08:00:43.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.020+0000 7fb3997ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3941530f0 con 0x7fb3941593f0 2023-06-22T08:00:43.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.020+0000 7fb3997ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3941f1030 con 0x7fb3940c4650 2023-06-22T08:00:43.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb3927fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394152ba0 0x7fb3940cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb392ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 0x7fb3940ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb3927fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394152ba0 0x7fb3940cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33494/0 (socket says 172.21.15.23:33494) 2023-06-22T08:00:43.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb3927fc700 1 -- 172.21.15.23:0/220032865 learned_addr learned my addr 172.21.15.23:0/220032865 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:43.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb392ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 0x7fb3940ccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:51286/0 (socket says 172.21.15.23:51286) 2023-06-22T08:00:43.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb3937fe700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3941593f0 0x7fb3941f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb3927fc700 1 -- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 msgr2=0x7fb3940ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb3927fc700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 0x7fb3940ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb3927fc700 1 -- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3941593f0 msgr2=0x7fb3941f1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb3927fc700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3941593f0 0x7fb3941f1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb3927fc700 1 -- 172.21.15.23:0/220032865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb388009550 con 0x7fb394152ba0 2023-06-22T08:00:43.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.021+0000 7fb3927fc700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394152ba0 0x7fb3940cd3e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb39408b530 tx=0x7fb388004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:43.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.022+0000 7fb37b7fe700 1 -- 172.21.15.23:0/220032865 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388004510 con 0x7fb394152ba0 2023-06-22T08:00:43.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.022+0000 7fb37b7fe700 1 -- 172.21.15.23:0/220032865 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3880036a0 con 0x7fb394152ba0 2023-06-22T08:00:43.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.022+0000 7fb37b7fe700 1 -- 172.21.15.23:0/220032865 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388003a10 con 0x7fb394152ba0 2023-06-22T08:00:43.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.022+0000 7fb3997ff700 1 -- 172.21.15.23:0/220032865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3940c8490 con 0x7fb394152ba0 2023-06-22T08:00:43.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.022+0000 7fb3997ff700 1 -- 172.21.15.23:0/220032865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3940c8a50 con 0x7fb394152ba0 2023-06-22T08:00:43.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.023+0000 7fb3997ff700 1 -- 172.21.15.23:0/220032865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3940cbed0 con 0x7fb394152ba0 2023-06-22T08:00:43.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.024+0000 7fb37b7fe700 1 -- 172.21.15.23:0/220032865 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb388003b70 con 0x7fb394152ba0 2023-06-22T08:00:43.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.024+0000 7fb37b7fe700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb38006ad10 0x7fb38006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.025+0000 7fb37b7fe700 1 -- 172.21.15.23:0/220032865 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb38808ad20 con 0x7fb394152ba0 2023-06-22T08:00:43.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.025+0000 7fb392ffd700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb38006ad10 0x7fb38006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.025+0000 7fb392ffd700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb38006ad10 0x7fb38006d1c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb384007fd0 tx=0x7fb384009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:43.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.029+0000 7fb37b7fe700 1 -- 172.21.15.23:0/220032865 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb388055af0 con 0x7fb394152ba0 2023-06-22T08:00:43.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.238+0000 7fb3997ff700 1 -- 172.21.15.23:0/220032865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb3940ccbc0 con 0x7fb394152ba0 2023-06-22T08:00:43.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.238+0000 7fb37b7fe700 1 -- 172.21.15.23:0/220032865 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb3940ccbc0 con 0x7fb394152ba0 2023-06-22T08:00:43.242 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:43.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 -- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb38006ad10 msgr2=0x7fb38006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb38006ad10 0x7fb38006d1c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb384007fd0 tx=0x7fb384009040 comp rx=0 tx=0).stop 2023-06-22T08:00:43.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 -- 172.21.15.23:0/220032865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394152ba0 msgr2=0x7fb3940cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394152ba0 0x7fb3940cd3e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb39408b530 tx=0x7fb388004750 comp rx=0 tx=0).stop 2023-06-22T08:00:43.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 -- 172.21.15.23:0/220032865 shutdown_connections 2023-06-22T08:00:43.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb38006ad10 0x7fb38006d1c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3941593f0 0x7fb3941f1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3940c4650 0x7fb3940ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 --2- 172.21.15.23:0/220032865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394152ba0 0x7fb3940cd3e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 -- 172.21.15.23:0/220032865 >> 172.21.15.23:0/220032865 conn(0x7fb394065b90 msgr2=0x7fb3940c3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:43.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 -- 172.21.15.23:0/220032865 shutdown_connections 2023-06-22T08:00:43.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.241+0000 7fb3997ff700 1 -- 172.21.15.23:0/220032865 wait complete. 2023-06-22T08:00:43.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:00:43.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.458+0000 7f6829fff700 1 Processor -- start 2023-06-22T08:00:43.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.458+0000 7f6829fff700 1 -- start start 2023-06-22T08:00:43.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.459+0000 7f6829fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f68240c9fd0 0x7f68240cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.459+0000 7f6829fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f68240c40b0 0x7f68240c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6829fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f682415c370 con 0x7f6824152ce0 2023-06-22T08:00:43.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6829fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68241530b0 con 0x7f68240c40b0 2023-06-22T08:00:43.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6829fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68240cf6f0 con 0x7f68240c9fd0 2023-06-22T08:00:43.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6823fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f68240c40b0 0x7f68240c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6823fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f68240c40b0 0x7f68240c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33496/0 (socket says 172.21.15.23:33496) 2023-06-22T08:00:43.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6823fff700 1 -- 172.21.15.23:0/3878217797 learned_addr learned my addr 172.21.15.23:0/3878217797 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:43.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6822ffd700 1 --2- 172.21.15.23:0/3878217797 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f68240c9fd0 0x7f68240cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f68237fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f6824152ce0 0x7f682415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:32798/0 (socket says 172.21.15.23:32798) 2023-06-22T08:00:43.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6823fff700 1 -- 172.21.15.23:0/3878217797 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f68240c9fd0 msgr2=0x7f68240cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:43.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6823fff700 1 --2- 172.21.15.23:0/3878217797 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f68240c9fd0 0x7f68240cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6823fff700 1 -- 172.21.15.23:0/3878217797 >> v1:172.21.15.23:6789/0 conn(0x7f6824152ce0 legacy=0x7f682415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f6823fff700 1 -- 172.21.15.23:0/3878217797 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682415bda0 con 0x7f68240c40b0 2023-06-22T08:00:43.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.460+0000 7f68227fc700 1 -- 172.21.15.23:0/3878217797 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366423399 0 0) 0x7f682415c370 con 0x7f6824152ce0 2023-06-22T08:00:43.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.461+0000 7f6823fff700 1 --2- 172.21.15.23:0/3878217797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 0x7f68240c4530 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f681c009e50 tx=0x7f681c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1044164b8b50d7f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:43.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.461+0000 7f68227fc700 1 -- 172.21.15.23:0/3878217797 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c00e070 con 0x7f68240c40b0 2023-06-22T08:00:43.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.461+0000 7f68227fc700 1 -- 172.21.15.23:0/3878217797 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c00b910 con 0x7f68240c40b0 2023-06-22T08:00:43.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.461+0000 7f68227fc700 1 -- 172.21.15.23:0/3878217797 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c00bcc0 con 0x7f68240c40b0 2023-06-22T08:00:43.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.461+0000 7f6829fff700 1 -- 172.21.15.23:0/3878217797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 msgr2=0x7f68240c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.461+0000 7f6829fff700 1 --2- 172.21.15.23:0/3878217797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 0x7f68240c4530 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f681c009e50 tx=0x7f681c00b330 comp rx=0 tx=0).stop 2023-06-22T08:00:43.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.461+0000 7f6829fff700 1 -- 172.21.15.23:0/3878217797 shutdown_connections 2023-06-22T08:00:43.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.461+0000 7f6829fff700 1 --2- 172.21.15.23:0/3878217797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 0x7f68240c4530 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.461+0000 7f6829fff700 1 --2- 172.21.15.23:0/3878217797 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f68240c9fd0 0x7f68240cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.461+0000 7f6829fff700 1 -- 172.21.15.23:0/3878217797 >> 172.21.15.23:0/3878217797 conn(0x7f682406ae30 msgr2=0x7f682406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:43.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.462+0000 7f6829fff700 1 -- 172.21.15.23:0/3878217797 shutdown_connections 2023-06-22T08:00:43.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.462+0000 7f6829fff700 1 -- 172.21.15.23:0/3878217797 wait complete. 2023-06-22T08:00:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.463+0000 7f6829fff700 1 Processor -- start 2023-06-22T08:00:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.463+0000 7f6829fff700 1 -- start start 2023-06-22T08:00:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.464+0000 7f6829fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 0x7f682414f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.464+0000 7f6829fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68240c9fd0 0x7f68241e2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.464+0000 7f6829fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6824152ce0 0x7f68241e32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.464+0000 7f6829fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682414eb00 con 0x7f68240c40b0 2023-06-22T08:00:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.464+0000 7f6829fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68241e3810 con 0x7f68240c9fd0 2023-06-22T08:00:43.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.464+0000 7f6829fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68241e1510 con 0x7f6824152ce0 2023-06-22T08:00:43.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f68237fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 0x7f682414f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f68237fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 0x7f682414f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33504/0 (socket says 172.21.15.23:33504) 2023-06-22T08:00:43.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f6822ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68240c9fd0 0x7f68241e2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f6823fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6824152ce0 0x7f68241e32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f68237fe700 1 -- 172.21.15.23:0/1049354452 learned_addr learned my addr 172.21.15.23:0/1049354452 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:43.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f6822ffd700 1 -- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6824152ce0 msgr2=0x7f68241e32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f6822ffd700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6824152ce0 0x7f68241e32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f6822ffd700 1 -- 172.21.15.23:0/1049354452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 msgr2=0x7f682414f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f6822ffd700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 0x7f682414f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f6822ffd700 1 -- 172.21.15.23:0/1049354452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c009b30 con 0x7f68240c9fd0 2023-06-22T08:00:43.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f68237fe700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 0x7f682414f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:43.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.465+0000 7f6822ffd700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68240c9fd0 0x7f68241e2d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f681800e930 tx=0x7f681800ec40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:43.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.466+0000 7f680bfff700 1 -- 172.21.15.23:0/1049354452 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6818013680 con 0x7f68240c9fd0 2023-06-22T08:00:43.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.466+0000 7f680bfff700 1 -- 172.21.15.23:0/1049354452 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68180137e0 con 0x7f68240c9fd0 2023-06-22T08:00:43.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.466+0000 7f680bfff700 1 -- 172.21.15.23:0/1049354452 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6818013b50 con 0x7f68240c9fd0 2023-06-22T08:00:43.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.466+0000 7f6829fff700 1 -- 172.21.15.23:0/1049354452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f682414e160 con 0x7f68240c9fd0 2023-06-22T08:00:43.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.466+0000 7f6829fff700 1 -- 172.21.15.23:0/1049354452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68241ee880 con 0x7f68240c9fd0 2023-06-22T08:00:43.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.467+0000 7f6829fff700 1 -- 172.21.15.23:0/1049354452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68241eea10 con 0x7f68240c9fd0 2023-06-22T08:00:43.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.468+0000 7f680bfff700 1 -- 172.21.15.23:0/1049354452 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6818013cb0 con 0x7f68240c9fd0 2023-06-22T08:00:43.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.469+0000 7f680bfff700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f681006ace0 0x7f681006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.469+0000 7f680bfff700 1 -- 172.21.15.23:0/1049354452 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6818089a70 con 0x7f68240c9fd0 2023-06-22T08:00:43.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.469+0000 7f68237fe700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f681006ace0 0x7f681006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.470+0000 7f68237fe700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f681006ace0 0x7f681006d190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6814000f60 tx=0x7f6814007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.472+0000 7f680bfff700 1 -- 172.21.15.23:0/1049354452 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f68180548a0 con 0x7f68240c9fd0 2023-06-22T08:00:43.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.724+0000 7f6829fff700 1 -- 172.21.15.23:0/1049354452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f682404ea20 con 0x7f68240c9fd0 2023-06-22T08:00:43.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.725+0000 7f680bfff700 1 -- 172.21.15.23:0/1049354452 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6243 (secure 0 0 0) 0x7f6818057ec0 con 0x7f68240c9fd0 2023-06-22T08:00:43.729 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:43.729 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:00:41.586436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:00:43.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 -- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f681006ace0 msgr2=0x7f681006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f681006ace0 0x7f681006d190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6814000f60 tx=0x7f6814007ea0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 -- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68240c9fd0 msgr2=0x7f68241e2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68240c9fd0 0x7f68241e2d90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f681800e930 tx=0x7f681800ec40 comp rx=0 tx=0).stop 2023-06-22T08:00:43.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 -- 172.21.15.23:0/1049354452 shutdown_connections 2023-06-22T08:00:43.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6824152ce0 0x7f68241e32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68240c40b0 0x7f682414f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f681006ace0 0x7f681006d190 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 --2- 172.21.15.23:0/1049354452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68240c9fd0 0x7f68241e2d90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 -- 172.21.15.23:0/1049354452 >> 172.21.15.23:0/1049354452 conn(0x7f682406ae30 msgr2=0x7f68241454c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 -- 172.21.15.23:0/1049354452 shutdown_connections 2023-06-22T08:00:43.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.728+0000 7f6829fff700 1 -- 172.21.15.23:0/1049354452 wait complete. 2023-06-22T08:00:43.733 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 19 2023-06-22T08:00:43.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:43.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.944+0000 7fc0925ff700 1 Processor -- start 2023-06-22T08:00:43.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.944+0000 7fc0925ff700 1 -- start start 2023-06-22T08:00:43.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.945+0000 7fc0925ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc08c152ce0 0x7fc08c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.945+0000 7fc0925ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc08c0c9fd0 0x7fc08c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.945+0000 7fc0925ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc08c158880 0x7fc08c158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.945+0000 7fc0925ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08c1530b0 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.945+0000 7fc0925ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08c0cf6f0 con 0x7fc08c158880 2023-06-22T08:00:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.945+0000 7fc0925ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c15c370 con 0x7fc08c152ce0 2023-06-22T08:00:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc08c0c9fd0 0x7fc08c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc08c152ce0 0x7fc08c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc090b9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc08c158880 0x7fc08c158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc08c0c9fd0 0x7fc08c15c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33516/0 (socket says 172.21.15.23:33516) 2023-06-22T08:00:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08b7fe700 1 -- 172.21.15.23:0/2802306862 learned_addr learned my addr 172.21.15.23:0/2802306862 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:43.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08b7fe700 1 -- 172.21.15.23:0/2802306862 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc08c152ce0 msgr2=0x7fc08c0cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:43.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08b7fe700 1 --2- 172.21.15.23:0/2802306862 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc08c152ce0 0x7fc08c0cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08b7fe700 1 -- 172.21.15.23:0/2802306862 >> v2:172.21.15.67:3300/0 conn(0x7fc08c158880 msgr2=0x7fc08c158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:43.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08b7fe700 1 --2- 172.21.15.23:0/2802306862 >> v2:172.21.15.67:3300/0 conn(0x7fc08c158880 0x7fc08c158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08b7fe700 1 -- 172.21.15.23:0/2802306862 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c15bda0 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08b7fe700 1 --2- 172.21.15.23:0/2802306862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 0x7fc08c15c800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc08c08b9e0 tx=0x7fc080009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757611dddf4784e7 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:43.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08affd700 1 -- 172.21.15.23:0/2802306862 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080011830 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08affd700 1 -- 172.21.15.23:0/2802306862 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc080011990 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.946+0000 7fc08affd700 1 -- 172.21.15.23:0/2802306862 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080011d20 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.947+0000 7fc0925ff700 1 -- 172.21.15.23:0/2802306862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 msgr2=0x7fc08c15c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.947+0000 7fc0925ff700 1 --2- 172.21.15.23:0/2802306862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 0x7fc08c15c800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc08c08b9e0 tx=0x7fc080009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.947+0000 7fc0925ff700 1 -- 172.21.15.23:0/2802306862 shutdown_connections 2023-06-22T08:00:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.947+0000 7fc0925ff700 1 --2- 172.21.15.23:0/2802306862 >> v2:172.21.15.67:3300/0 conn(0x7fc08c158880 0x7fc08c158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.947+0000 7fc0925ff700 1 --2- 172.21.15.23:0/2802306862 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc08c152ce0 0x7fc08c0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.947+0000 7fc0925ff700 1 --2- 172.21.15.23:0/2802306862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 0x7fc08c15c800 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.947+0000 7fc0925ff700 1 -- 172.21.15.23:0/2802306862 >> 172.21.15.23:0/2802306862 conn(0x7fc08c06ae30 msgr2=0x7fc08c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.947+0000 7fc0925ff700 1 -- 172.21.15.23:0/2802306862 shutdown_connections 2023-06-22T08:00:43.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.947+0000 7fc0925ff700 1 -- 172.21.15.23:0/2802306862 wait complete. 2023-06-22T08:00:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.950+0000 7fc0925ff700 1 Processor -- start 2023-06-22T08:00:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.950+0000 7fc0925ff700 1 -- start start 2023-06-22T08:00:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc0925ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 0x7fc08c1ee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc0925ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc08c152ce0 0x7fc08c14e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc0925ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08c158880 0x7fc08c14a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc0925ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c157100 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc0925ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c1f1410 con 0x7fc08c158880 2023-06-22T08:00:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc0925ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c1f2270 con 0x7fc08c152ce0 2023-06-22T08:00:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc08bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 0x7fc08c1ee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc08bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 0x7fc08c1ee8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33532/0 (socket says 172.21.15.23:33532) 2023-06-22T08:00:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc08bfff700 1 -- 172.21.15.23:0/705322168 learned_addr learned my addr 172.21.15.23:0/705322168 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc090b9c700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08c158880 0x7fc08c14a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.951+0000 7fc08b7fe700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc08c152ce0 0x7fc08c14e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc08bfff700 1 -- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc08c152ce0 msgr2=0x7fc08c14e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc08bfff700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc08c152ce0 0x7fc08c14e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc08bfff700 1 -- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08c158880 msgr2=0x7fc08c14a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:43.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc08bfff700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08c158880 0x7fc08c14a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:43.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc08bfff700 1 -- 172.21.15.23:0/705322168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc080009550 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc08bfff700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 0x7fc08c1ee8b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00e9b0 tx=0x7fc07c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:43.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc088ff9700 1 -- 172.21.15.23:0/705322168 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c009710 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc088ff9700 1 -- 172.21.15.23:0/705322168 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07c00ee50 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc088ff9700 1 -- 172.21.15.23:0/705322168 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c00ca50 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc0925ff700 1 -- 172.21.15.23:0/705322168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c1eedf0 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.952+0000 7fc0925ff700 1 -- 172.21.15.23:0/705322168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c1f1940 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.953+0000 7fc0925ff700 1 -- 172.21.15.23:0/705322168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c04ea20 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.954+0000 7fc088ff9700 1 -- 172.21.15.23:0/705322168 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc07c00cbb0 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.955+0000 7fc088ff9700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc07806ace0 0x7fc07806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.956+0000 7fc088ff9700 1 -- 172.21.15.23:0/705322168 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc07c089b50 con 0x7fc08c0c9fd0 2023-06-22T08:00:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.956+0000 7fc08b7fe700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc07806ace0 0x7fc07806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.956+0000 7fc08b7fe700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc07806ace0 0x7fc07806d190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc080009c10 tx=0x7fc080008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:43.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:43.960+0000 7fc088ff9700 1 -- 172.21.15.23:0/705322168 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc07c058300 con 0x7fc08c0c9fd0 2023-06-22T08:00:44.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.171+0000 7fc0925ff700 1 -- 172.21.15.23:0/705322168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc08c0c9db0 con 0x7fc08c0c9fd0 2023-06-22T08:00:44.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.173+0000 7fc088ff9700 1 -- 172.21.15.23:0/705322168 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7fc07c00ce60 con 0x7fc08c0c9fd0 2023-06-22T08:00:44.176 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:44.176 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:44.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 -- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc07806ace0 msgr2=0x7fc07806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:44.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc07806ace0 0x7fc07806d190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc080009c10 tx=0x7fc080008030 comp rx=0 tx=0).stop 2023-06-22T08:00:44.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 -- 172.21.15.23:0/705322168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 msgr2=0x7fc08c1ee8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:44.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 0x7fc08c1ee8b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00e9b0 tx=0x7fc07c00ecc0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 -- 172.21.15.23:0/705322168 shutdown_connections 2023-06-22T08:00:44.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc07806ace0 0x7fc07806d190 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc08c158880 0x7fc08c14a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc08c0c9fd0 0x7fc08c1ee8b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 --2- 172.21.15.23:0/705322168 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc08c152ce0 0x7fc08c14e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 -- 172.21.15.23:0/705322168 >> 172.21.15.23:0/705322168 conn(0x7fc08c06ae30 msgr2=0x7fc08c1566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:44.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 -- 172.21.15.23:0/705322168 shutdown_connections 2023-06-22T08:00:44.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.175+0000 7fc0925ff700 1 -- 172.21.15.23:0/705322168 wait complete. 2023-06-22T08:00:44.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:00:44.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.393+0000 7f0e47dff700 1 Processor -- start 2023-06-22T08:00:44.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.394+0000 7f0e47dff700 1 -- start start 2023-06-22T08:00:44.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.395+0000 7f0e47dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0e40152bf0 0x7f0e4015c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:44.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.395+0000 7f0e47dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0e400c7050 0x7f0e40158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:44.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.395+0000 7f0e47dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e4015cc90 con 0x7f0e40159410 2023-06-22T08:00:44.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.395+0000 7f0e47dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e400c5d00 con 0x7f0e40152bf0 2023-06-22T08:00:44.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.395+0000 7f0e47dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4006b900 con 0x7f0e400c7050 2023-06-22T08:00:44.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.396+0000 7f0e45b9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0e40152bf0 0x7f0e4015c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:44.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.396+0000 7f0e4639c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f0e40159410 0x7f0e400c75a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:32806/0 (socket says 172.21.15.23:32806) 2023-06-22T08:00:44.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.396+0000 7f0e4639c700 1 -- 172.21.15.23:0/3089122182 learned_addr learned my addr 172.21.15.23:0/3089122182 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:44.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.396+0000 7f0e4539a700 1 --2- 172.21.15.23:0/3089122182 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0e400c7050 0x7f0e40158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:44.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.396+0000 7f0e45b9b700 1 -- 172.21.15.23:0/3089122182 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0e400c7050 msgr2=0x7f0e40158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:44.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.396+0000 7f0e45b9b700 1 --2- 172.21.15.23:0/3089122182 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0e400c7050 0x7f0e40158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.396+0000 7f0e45b9b700 1 -- 172.21.15.23:0/3089122182 >> v1:172.21.15.23:6789/0 conn(0x7f0e40159410 legacy=0x7f0e400c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:44.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.396+0000 7f0e45b9b700 1 -- 172.21.15.23:0/3089122182 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4006b620 con 0x7f0e40152bf0 2023-06-22T08:00:44.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.396+0000 7f0e44b99700 1 -- 172.21.15.23:0/3089122182 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703104848 0 0) 0x7f0e4015cc90 con 0x7f0e40159410 2023-06-22T08:00:44.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.396+0000 7f0e45b9b700 1 --2- 172.21.15.23:0/3089122182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e40152bf0 0x7f0e4015c750 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c009840 tx=0x7f0e3c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8bb7acf560fcea server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:44.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e44b99700 1 -- 172.21.15.23:0/3089122182 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c011830 con 0x7f0e40152bf0 2023-06-22T08:00:44.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e44b99700 1 -- 172.21.15.23:0/3089122182 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3c011990 con 0x7f0e40152bf0 2023-06-22T08:00:44.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e44b99700 1 -- 172.21.15.23:0/3089122182 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c011d20 con 0x7f0e40152bf0 2023-06-22T08:00:44.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e47dff700 1 -- 172.21.15.23:0/3089122182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e40152bf0 msgr2=0x7f0e4015c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:44.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e47dff700 1 --2- 172.21.15.23:0/3089122182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e40152bf0 0x7f0e4015c750 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c009840 tx=0x7f0e3c009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:44.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e47dff700 1 -- 172.21.15.23:0/3089122182 shutdown_connections 2023-06-22T08:00:44.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e47dff700 1 --2- 172.21.15.23:0/3089122182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e40152bf0 0x7f0e4015c750 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e47dff700 1 --2- 172.21.15.23:0/3089122182 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0e400c7050 0x7f0e40158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e47dff700 1 -- 172.21.15.23:0/3089122182 >> 172.21.15.23:0/3089122182 conn(0x7f0e4006e0f0 msgr2=0x7f0e4006e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:44.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e47dff700 1 -- 172.21.15.23:0/3089122182 shutdown_connections 2023-06-22T08:00:44.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.397+0000 7f0e47dff700 1 -- 172.21.15.23:0/3089122182 wait complete. 2023-06-22T08:00:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.399+0000 7f0e47dff700 1 Processor -- start 2023-06-22T08:00:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.399+0000 7f0e47dff700 1 -- start start 2023-06-22T08:00:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e47dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e400c7050 0x7f0e401f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e47dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0e40152bf0 0x7f0e401eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e47dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e40159410 0x7f0e40070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e47dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e401534a0 con 0x7f0e40159410 2023-06-22T08:00:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e47dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e401f25d0 con 0x7f0e400c7050 2023-06-22T08:00:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e47dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e40159a00 con 0x7f0e40152bf0 2023-06-22T08:00:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e45b9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e400c7050 0x7f0e401f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e4639c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e40159410 0x7f0e40070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e4539a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0e40152bf0 0x7f0e401eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e45b9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e400c7050 0x7f0e401f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:40078/0 (socket says 172.21.15.23:40078) 2023-06-22T08:00:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.400+0000 7f0e45b9b700 1 -- 172.21.15.23:0/2257992217 learned_addr learned my addr 172.21.15.23:0/2257992217 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.401+0000 7f0e45b9b700 1 -- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0e40152bf0 msgr2=0x7f0e401eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.401+0000 7f0e45b9b700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0e40152bf0 0x7f0e401eea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.401+0000 7f0e45b9b700 1 -- 172.21.15.23:0/2257992217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e40159410 msgr2=0x7f0e40070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.401+0000 7f0e45b9b700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e40159410 0x7f0e40070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.401+0000 7f0e45b9b700 1 -- 172.21.15.23:0/2257992217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c0094a0 con 0x7f0e400c7050 2023-06-22T08:00:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.401+0000 7f0e45b9b700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e400c7050 0x7f0e401f2090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0090b0 tx=0x7f0e3c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.401+0000 7f0e367fc700 1 -- 172.21.15.23:0/2257992217 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c0036a0 con 0x7f0e400c7050 2023-06-22T08:00:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.401+0000 7f0e367fc700 1 -- 172.21.15.23:0/2257992217 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3c01f070 con 0x7f0e400c7050 2023-06-22T08:00:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.401+0000 7f0e47dff700 1 -- 172.21.15.23:0/2257992217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e40159c70 con 0x7f0e400c7050 2023-06-22T08:00:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.401+0000 7f0e367fc700 1 -- 172.21.15.23:0/2257992217 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c003a40 con 0x7f0e400c7050 2023-06-22T08:00:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.402+0000 7f0e47dff700 1 -- 172.21.15.23:0/2257992217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e401ee500 con 0x7f0e400c7050 2023-06-22T08:00:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.403+0000 7f0e47dff700 1 -- 172.21.15.23:0/2257992217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e4004ea20 con 0x7f0e400c7050 2023-06-22T08:00:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.403+0000 7f0e367fc700 1 -- 172.21.15.23:0/2257992217 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0e3c003ba0 con 0x7f0e400c7050 2023-06-22T08:00:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.404+0000 7f0e367fc700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0e2406ace0 0x7f0e2406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.404+0000 7f0e367fc700 1 -- 172.21.15.23:0/2257992217 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0e3c021070 con 0x7f0e400c7050 2023-06-22T08:00:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.405+0000 7f0e4539a700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0e2406ace0 0x7f0e2406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.405+0000 7f0e4539a700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0e2406ace0 0x7f0e2406d190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0e4015cc70 tx=0x7f0e30009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.409+0000 7f0e367fc700 1 -- 172.21.15.23:0/2257992217 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0e3c055d50 con 0x7f0e400c7050 2023-06-22T08:00:44.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.663+0000 7f0e47dff700 1 -- 172.21.15.23:0/2257992217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e40066200 con 0x7f0e400c7050 2023-06-22T08:00:44.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.664+0000 7f0e367fc700 1 -- 172.21.15.23:0/2257992217 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6243 (secure 0 0 0) 0x7f0e3c02b090 con 0x7f0e400c7050 2023-06-22T08:00:44.667 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:44.667 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:00:41.586436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:00:44.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 -- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0e2406ace0 msgr2=0x7f0e2406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:44.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0e2406ace0 0x7f0e2406d190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0e4015cc70 tx=0x7f0e30009210 comp rx=0 tx=0).stop 2023-06-22T08:00:44.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 -- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e400c7050 msgr2=0x7f0e401f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:44.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e400c7050 0x7f0e401f2090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0090b0 tx=0x7f0e3c004750 comp rx=0 tx=0).stop 2023-06-22T08:00:44.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 -- 172.21.15.23:0/2257992217 shutdown_connections 2023-06-22T08:00:44.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e40159410 0x7f0e40070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0e2406ace0 0x7f0e2406d190 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e400c7050 0x7f0e401f2090 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 --2- 172.21.15.23:0/2257992217 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0e40152bf0 0x7f0e401eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:44.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 -- 172.21.15.23:0/2257992217 >> 172.21.15.23:0/2257992217 conn(0x7f0e4006e0f0 msgr2=0x7f0e4006e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:44.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 -- 172.21.15.23:0/2257992217 shutdown_connections 2023-06-22T08:00:44.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:44.666+0000 7f0e47dff700 1 -- 172.21.15.23:0/2257992217 wait complete. 2023-06-22T08:00:44.671 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 19 2023-06-22T08:00:45.291 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-22T07:59:55.139239+0000', 'modified': '2023-06-22T08:00:41.586436+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4285, 'mds_1': 4279, 'mds_2': 4272, 'mds_3': 4269, 'mds_4': 4268}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4268': {'gid': 4268, 'name': 'e', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.23:6837/1762110734', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6836', 'nonce': 1762110734}, {'type': 'v1', 'addr': '172.21.15.23:6837', 'nonce': 1762110734}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4269': {'gid': 4269, 'name': 'c', 'rank': 3, 'incarnation': 16, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.23:6834/3365097525', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6832', 'nonce': 3365097525}, {'type': 'v1', 'addr': '172.21.15.23:6834', 'nonce': 3365097525}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4272': {'gid': 4272, 'name': 'a', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.23:6835/3894174037', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6833', 'nonce': 3894174037}, {'type': 'v1', 'addr': '172.21.15.23:6835', 'nonce': 3894174037}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4279': {'gid': 4279, 'name': 'b', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.67:6836/1962314516', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 1962314516}, {'type': 'v1', 'addr': '172.21.15.67:6836', 'nonce': 1962314516}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_4285': {'gid': 4285, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6839/567663545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6838', 'nonce': 567663545}, {'type': 'v1', 'addr': '172.21.15.67:6839', 'nonce': 567663545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-06-22T08:00:45.292 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-06-22T08:00:45.292 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-06-22T08:00:45.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.494+0000 7f5c801ff700 1 Processor -- start 2023-06-22T08:00:45.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.494+0000 7f5c801ff700 1 -- start start 2023-06-22T08:00:45.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.495+0000 7f5c801ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5c780c97d0 0x7f5c78159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.495+0000 7f5c801ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5c780c45f0 0x7f5c780c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.495+0000 7f5c801ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c780c9c30 con 0x7f5c78152d70 2023-06-22T08:00:45.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.495+0000 7f5c801ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c780c95e0 con 0x7f5c780c45f0 2023-06-22T08:00:45.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.495+0000 7f5c801ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7815bea0 con 0x7f5c780c97d0 2023-06-22T08:00:45.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.495+0000 7f5c7df9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5c78152d70 0x7f5c780cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:32816/0 (socket says 172.21.15.23:32816) 2023-06-22T08:00:45.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.495+0000 7f5c7df9b700 1 -- 172.21.15.23:0/3701181594 learned_addr learned my addr 172.21.15.23:0/3701181594 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:45.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.495+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/3701181594 >> v2:172.21.15.23:3300/0 conn(0x7f5c780c45f0 0x7f5c780c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.496+0000 7f5c7d79a700 1 --2- 172.21.15.23:0/3701181594 >> v2:172.21.15.67:3300/0 conn(0x7f5c780c97d0 0x7f5c78159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.496+0000 7f5c7e79c700 1 -- 172.21.15.23:0/3701181594 >> v2:172.21.15.67:3300/0 conn(0x7f5c780c97d0 msgr2=0x7f5c78159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:45.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.496+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/3701181594 >> v2:172.21.15.67:3300/0 conn(0x7f5c780c97d0 0x7f5c78159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.496+0000 7f5c7e79c700 1 -- 172.21.15.23:0/3701181594 >> v1:172.21.15.23:6789/0 conn(0x7f5c78152d70 legacy=0x7f5c780cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:45.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.496+0000 7f5c7e79c700 1 -- 172.21.15.23:0/3701181594 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7815c160 con 0x7f5c780c45f0 2023-06-22T08:00:45.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.496+0000 7f5c7cf99700 1 -- 172.21.15.23:0/3701181594 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246778175 0 0) 0x7f5c780c9c30 con 0x7f5c78152d70 2023-06-22T08:00:45.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.496+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/3701181594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c780c45f0 0x7f5c780c4a10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5c7808ba40 tx=0x7f5c7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf5a71d3d898922 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:45.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.496+0000 7f5c7cf99700 1 -- 172.21.15.23:0/3701181594 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7000e070 con 0x7f5c780c45f0 2023-06-22T08:00:45.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.496+0000 7f5c7cf99700 1 -- 172.21.15.23:0/3701181594 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7000bb20 con 0x7f5c780c45f0 2023-06-22T08:00:45.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.497+0000 7f5c7cf99700 1 -- 172.21.15.23:0/3701181594 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70012430 con 0x7f5c780c45f0 2023-06-22T08:00:45.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.497+0000 7f5c801ff700 1 -- 172.21.15.23:0/3701181594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c780c45f0 msgr2=0x7f5c780c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:45.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.497+0000 7f5c801ff700 1 --2- 172.21.15.23:0/3701181594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c780c45f0 0x7f5c780c4a10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5c7808ba40 tx=0x7f5c7000b330 comp rx=0 tx=0).stop 2023-06-22T08:00:45.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.497+0000 7f5c801ff700 1 -- 172.21.15.23:0/3701181594 shutdown_connections 2023-06-22T08:00:45.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.497+0000 7f5c801ff700 1 --2- 172.21.15.23:0/3701181594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c780c45f0 0x7f5c780c4a10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.497+0000 7f5c801ff700 1 --2- 172.21.15.23:0/3701181594 >> v2:172.21.15.67:3300/0 conn(0x7f5c780c97d0 0x7f5c78159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.497+0000 7f5c801ff700 1 -- 172.21.15.23:0/3701181594 >> 172.21.15.23:0/3701181594 conn(0x7f5c7806ae30 msgr2=0x7f5c7806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:45.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.497+0000 7f5c801ff700 1 -- 172.21.15.23:0/3701181594 shutdown_connections 2023-06-22T08:00:45.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.497+0000 7f5c801ff700 1 -- 172.21.15.23:0/3701181594 wait complete. 2023-06-22T08:00:45.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.499+0000 7f5c801ff700 1 Processor -- start 2023-06-22T08:00:45.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.499+0000 7f5c801ff700 1 -- start start 2023-06-22T08:00:45.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c801ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5c780c45f0 0x7f5c7815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c801ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c780c97d0 0x7f5c78154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c801ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c78152d70 0x7f5c78155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c801ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781573a0 con 0x7f5c78152d70 2023-06-22T08:00:45.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c801ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781f22d0 con 0x7f5c780c97d0 2023-06-22T08:00:45.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c801ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c78156790 con 0x7f5c780c45f0 2023-06-22T08:00:45.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c7e79c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c78152d70 0x7f5c78155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c7e79c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c78152d70 0x7f5c78155020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33566/0 (socket says 172.21.15.23:33566) 2023-06-22T08:00:45.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1700493700 learned_addr learned my addr 172.21.15.23:0/1700493700 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:45.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5c780c45f0 0x7f5c7815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.500+0000 7f5c7d79a700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c780c97d0 0x7f5c78154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.501+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5c780c45f0 msgr2=0x7f5c7815aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:45.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.501+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5c780c45f0 0x7f5c7815aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.501+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c780c97d0 msgr2=0x7f5c78154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:45.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.501+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c780c97d0 0x7f5c78154ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.501+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1700493700 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c70009a70 con 0x7f5c78152d70 2023-06-22T08:00:45.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.501+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c78152d70 0x7f5c78155020 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5c7808bcd0 tx=0x7f5c70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:45.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.501+0000 7f5c6e7fc700 1 -- 172.21.15.23:0/1700493700 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70012430 con 0x7f5c78152d70 2023-06-22T08:00:45.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.501+0000 7f5c6e7fc700 1 -- 172.21.15.23:0/1700493700 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c70020070 con 0x7f5c78152d70 2023-06-22T08:00:45.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.501+0000 7f5c6e7fc700 1 -- 172.21.15.23:0/1700493700 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70004280 con 0x7f5c78152d70 2023-06-22T08:00:45.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.501+0000 7f5c801ff700 1 -- 172.21.15.23:0/1700493700 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c78156a70 con 0x7f5c78152d70 2023-06-22T08:00:45.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.502+0000 7f5c801ff700 1 -- 172.21.15.23:0/1700493700 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c78157030 con 0x7f5c78152d70 2023-06-22T08:00:45.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.504+0000 7f5c6e7fc700 1 -- 172.21.15.23:0/1700493700 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5c700043e0 con 0x7f5c78152d70 2023-06-22T08:00:45.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.505+0000 7f5c6e7fc700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5c5c06ad10 0x7f5c5c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.505+0000 7f5c6e7fc700 1 -- 172.21.15.23:0/1700493700 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f5c70022070 con 0x7f5c78152d70 2023-06-22T08:00:45.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.505+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5c5c06ad10 0x7f5c5c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.506+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5c5c06ad10 0x7f5c5c06d1c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5c74000f60 tx=0x7f5c74007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.507+0000 7f5c93a2bb80 1 Processor -- start 2023-06-22T08:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.507+0000 7f5c93a2bb80 1 -- start start 2023-06-22T08:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d3cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d82520 0x565399d3d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d3f7c0 0x565399d6a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x565399c47c60 con 0x565399d3f7c0 2023-06-22T08:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x565399c9a9f0 con 0x565399d82150 2023-06-22T08:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x565399c93e80 con 0x565399d82520 2023-06-22T08:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7e79c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d3f7c0 0x565399d6a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7e79c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d3f7c0 0x565399d6a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33578/0 (socket says 172.21.15.23:33578) 2023-06-22T08:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7e79c700 1 -- 172.21.15.23:0/3616673365 learned_addr learned my addr 172.21.15.23:0/3616673365 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7d79a700 1 --2- 172.21.15.23:0/3616673365 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d82520 0x565399d3d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/3616673365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d3cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7e79c700 1 -- 172.21.15.23:0/3616673365 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d82520 msgr2=0x565399d3d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/3616673365 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d82520 0x565399d3d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7e79c700 1 -- 172.21.15.23:0/3616673365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 msgr2=0x565399d3cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/3616673365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d3cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7e79c700 1 -- 172.21.15.23:0/3616673365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565399b155e0 con 0x565399d3f7c0 2023-06-22T08:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.508+0000 7f5c7d79a700 1 --2- 172.21.15.23:0/3616673365 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d82520 0x565399d3d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:45.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.509+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/3616673365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d3f7c0 0x565399d6a880 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5c7000d010 tx=0x7f5c70059750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ffd1e5e65ec768 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:45.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.509+0000 7f5c5b7fe700 1 -- 172.21.15.23:0/3616673365 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7005ddb0 con 0x565399d3f7c0 2023-06-22T08:00:45.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.509+0000 7f5c5b7fe700 1 -- 172.21.15.23:0/3616673365 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c70062140 con 0x565399d3f7c0 2023-06-22T08:00:45.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.509+0000 7f5c5b7fe700 1 -- 172.21.15.23:0/3616673365 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70058190 con 0x565399d3f7c0 2023-06-22T08:00:45.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.509+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/3616673365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d3f7c0 msgr2=0x565399d6a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:45.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.509+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/3616673365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d3f7c0 0x565399d6a880 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5c7000d010 tx=0x7f5c70059750 comp rx=0 tx=0).stop 2023-06-22T08:00:45.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.510+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/3616673365 shutdown_connections 2023-06-22T08:00:45.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.510+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/3616673365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d3f7c0 0x565399d6a880 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.510+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/3616673365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d3cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.510+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/3616673365 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d82520 0x565399d3d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.510+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/3616673365 >> 172.21.15.23:0/3616673365 conn(0x565399d37fc0 msgr2=0x565399d406a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:45.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.510+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/3616673365 shutdown_connections 2023-06-22T08:00:45.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.510+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/3616673365 wait complete. 2023-06-22T08:00:45.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.510+0000 7f5c93a2bb80 1 -- start start 2023-06-22T08:00:45.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.511+0000 7f5c93a2bb80 1 Processor -- start 2023-06-22T08:00:45.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d935f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d690d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x565399c33a70 con 0x565399d82520 2023-06-22T08:00:45.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x565399c33770 con 0x565399d82150 2023-06-22T08:00:45.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x565399b178d0 con 0x565399d3f7c0 2023-06-22T08:00:45.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c93a2bb80 10 client.0 _ll_register_callbacks cb 0x565399c39210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:45.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7e79c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d690d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7e79c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d690d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33588/0 (socket says 172.21.15.23:33588) 2023-06-22T08:00:45.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1492334150 learned_addr learned my addr 172.21.15.23:0/1492334150 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:45.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7d79a700 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d935f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1492334150 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 msgr2=0x565399d431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:45.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d431c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1492334150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 msgr2=0x565399d935f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:45.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d935f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:45.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1492334150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c68009860 con 0x565399d82520 2023-06-22T08:00:45.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.512+0000 7f5c7d79a700 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d935f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:45.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d690d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5c7002a040 tx=0x7f5c70014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5360232e4dbe96 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:45.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c58ff9700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-06-22T08:00:45.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c58ff9700 1 -- 172.21.15.23:0/1492334150 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7005ddb0 con 0x565399d82520 2023-06-22T08:00:45.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c58ff9700 1 -- 172.21.15.23:0/1492334150 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f5c70061cb0 con 0x565399d82520 2023-06-22T08:00:45.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1492334150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565399d35780 con 0x565399d82520 2023-06-22T08:00:45.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c58ff9700 1 -- 172.21.15.23:0/1492334150 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c70061e70 con 0x565399d82520 2023-06-22T08:00:45.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c93a2bb80 10 client.4848 fetch_fsmap learned FSMap version 19 2023-06-22T08:00:45.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1492334150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x7f5c78157030 con 0x565399d82520 2023-06-22T08:00:45.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c58ff9700 1 -- 172.21.15.23:0/1492334150 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x565399d35780 con 0x565399d82520 2023-06-22T08:00:45.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c58ff9700 1 -- 172.21.15.23:0/1492334150 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 19) v1 ==== 4260+0+0 (secure 0 0 0) 0x7f5c70004690 con 0x565399d82520 2023-06-22T08:00:45.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c93a2bb80 10 client.4848 fetch_fsmap finished waiting for FSMap version 19 2023-06-22T08:00:45.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c93a2bb80 10 client.4848 resolve_mds: resolved name 'e' to daemon mds.e 2023-06-22T08:00:45.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c93a2bb80 20 client.4848 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:45.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.23:6836/1762110734,v1:172.21.15.23:6837/1762110734] conn(0x565399d67e10 0x565399d960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:45.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c93a2bb80 4 client.4848 mds_command: new command op to 4268 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-22T08:00:45.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.513+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1492334150 --> [v2:172.21.15.23:6836/1762110734,v1:172.21.15.23:6837/1762110734] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565399c8ce60 con 0x565399d67e10 2023-06-22T08:00:45.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.514+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.23:6836/1762110734,v1:172.21.15.23:6837/1762110734] conn(0x565399d67e10 0x565399d960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:45.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.514+0000 7f5c58ff9700 10 client.4848 ms_handle_connect on v2:172.21.15.23:6836/1762110734 2023-06-22T08:00:45.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:45.514+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.23:6836/1762110734,v1:172.21.15.23:6837/1762110734] conn(0x565399d67e10 0x565399d960e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=309c26e07508d38e server_cookie=1a665a2b01ba6e29 in_seq=0 out_seq=0 2023-06-22T08:00:45.598 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:00:45.594+0000 7fa5334b2700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-06-22T08:00:46.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.590+0000 7f5c58ff9700 1 -- 172.21.15.23:0/1492334150 <== mds.4 v2:172.21.15.23:6836/1762110734 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x565399c8ce60 con 0x565399d67e10 2023-06-22T08:00:46.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.590+0000 7f5c58ff9700 10 client.4848 handle_command_reply: tid=0 2023-06-22T08:00:46.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.590+0000 7f5c93a2bb80 1 client.4848 shutdown 2023-06-22T08:00:46.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.590+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1492334150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 msgr2=0x565399d690d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.590+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d690d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5c7002a040 tx=0x7f5c70014000 comp rx=0 tx=0).stop 2023-06-22T08:00:46.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.591+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1492334150 shutdown_connections 2023-06-22T08:00:46.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.591+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.23:6836/1762110734,v1:172.21.15.23:6837/1762110734] conn(0x565399d67e10 0x565399d960e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.591+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d690d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.591+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.591+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/1492334150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d935f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.591+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1492334150 >> 172.21.15.23:0/1492334150 conn(0x565399d37fc0 msgr2=0x565399d40ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:46.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.591+0000 7f5c58ff9700 0 client.4848 ms_handle_reset on v2:172.21.15.23:6836/1762110734 2023-06-22T08:00:46.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.591+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1492334150 shutdown_connections 2023-06-22T08:00:46.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.591+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1492334150 wait complete. 2023-06-22T08:00:46.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.591+0000 7f5c93a2bb80 20 client.4848 trim_cache size 0 max 16384 2023-06-22T08:00:46.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.613+0000 7f5c93a2bb80 1 Processor -- start 2023-06-22T08:00:46.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.613+0000 7f5c93a2bb80 1 -- start start 2023-06-22T08:00:46.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d3dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d3e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d8f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x565399d909a0 con 0x565399d82520 2023-06-22T08:00:46.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x565399d7cf10 con 0x565399d82150 2023-06-22T08:00:46.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x565399d873b0 con 0x565399d3f7c0 2023-06-22T08:00:46.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c7e79c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d8f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c7df9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d3dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c7d79a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d3e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c7df9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d3dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:48746/0 (socket says 172.21.15.23:48746) 2023-06-22T08:00:46.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c7df9b700 1 -- 172.21.15.23:0/1931277612 learned_addr learned my addr 172.21.15.23:0/1931277612 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:46.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.614+0000 7f5c7d79a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d3e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50988/0 (socket says 172.21.15.23:50988) 2023-06-22T08:00:46.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.615+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1931277612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 msgr2=0x565399d3dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.615+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/1931277612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d3dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.615+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1931277612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 msgr2=0x565399d3e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.615+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/1931277612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d3e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.615+0000 7f5c7e79c700 1 -- 172.21.15.23:0/1931277612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565399ac1d90 con 0x565399d82520 2023-06-22T08:00:46.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.615+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/1931277612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d3dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:46.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.615+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/1931277612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d8f5f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x565399d43880 tx=0x7f5c70028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35907a8aea79476c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:46.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.615+0000 7f5c5bfff700 1 -- 172.21.15.23:0/1931277612 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70059750 con 0x565399d82520 2023-06-22T08:00:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.615+0000 7f5c5bfff700 1 -- 172.21.15.23:0/1931277612 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7005ddb0 con 0x565399d82520 2023-06-22T08:00:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.615+0000 7f5c5bfff700 1 -- 172.21.15.23:0/1931277612 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c70004690 con 0x565399d82520 2023-06-22T08:00:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.616+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1931277612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 msgr2=0x565399d8f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.616+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/1931277612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d8f5f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x565399d43880 tx=0x7f5c70028000 comp rx=0 tx=0).stop 2023-06-22T08:00:46.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.616+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1931277612 shutdown_connections 2023-06-22T08:00:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.616+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/1931277612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399d8f5f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.616+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/1931277612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d3dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.616+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/1931277612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399d3e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.616+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1931277612 >> 172.21.15.23:0/1931277612 conn(0x565399d37fc0 msgr2=0x565399d7d500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.616+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1931277612 shutdown_connections 2023-06-22T08:00:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.616+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/1931277612 wait complete. 2023-06-22T08:00:46.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.617+0000 7f5c93a2bb80 1 -- start start 2023-06-22T08:00:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.617+0000 7f5c93a2bb80 1 Processor -- start 2023-06-22T08:00:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.618+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d8ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.618+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399dab130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.618+0000 7f5c93a2bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399de3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.618+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x565399d8c480 con 0x565399d82520 2023-06-22T08:00:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.618+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x565399d904d0 con 0x565399d82150 2023-06-22T08:00:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.618+0000 7f5c93a2bb80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x565399ded610 con 0x565399d3f7c0 2023-06-22T08:00:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c93a2bb80 10 client.0 _ll_register_callbacks cb 0x565399c39210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c7df9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d8ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c7df9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d8ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:48762/0 (socket says 172.21.15.23:48762) 2023-06-22T08:00:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c7d79a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399dab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c7df9b700 1 -- 172.21.15.23:0/4142999622 learned_addr learned my addr 172.21.15.23:0/4142999622 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c7e79c700 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399de3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c7df9b700 1 -- 172.21.15.23:0/4142999622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 msgr2=0x565399dab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399dab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c7df9b700 1 -- 172.21.15.23:0/4142999622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 msgr2=0x565399de3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399de3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.619+0000 7f5c7df9b700 1 -- 172.21.15.23:0/4142999622 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c74014a50 con 0x565399d3f7c0 2023-06-22T08:00:46.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.620+0000 7f5c7df9b700 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d8ff90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5c74028810 tx=0x7f5c74012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae135803b7a3fb42 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:46.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.620+0000 7f5c58ff9700 10 client.0 ms_handle_connect on v2:172.21.15.67:3301/0 2023-06-22T08:00:46.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.620+0000 7f5c58ff9700 1 -- 172.21.15.23:0/4142999622 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74028040 con 0x565399d3f7c0 2023-06-22T08:00:46.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.620+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/4142999622 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c70062140 con 0x565399d3f7c0 2023-06-22T08:00:46.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.620+0000 7f5c58ff9700 1 -- 172.21.15.23:0/4142999622 <== mon.2 v2:172.21.15.67:3301/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f5c7401b070 con 0x565399d3f7c0 2023-06-22T08:00:46.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.620+0000 7f5c58ff9700 1 -- 172.21.15.23:0/4142999622 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c74028de0 con 0x565399d3f7c0 2023-06-22T08:00:46.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.620+0000 7f5c93a2bb80 10 client.4688 fetch_fsmap learned FSMap version 19 2023-06-22T08:00:46.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.620+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/4142999622 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x7f5c78157030 con 0x565399d3f7c0 2023-06-22T08:00:46.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.621+0000 7f5c58ff9700 1 -- 172.21.15.23:0/4142999622 <== mon.2 v2:172.21.15.67:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c70062140 con 0x565399d3f7c0 2023-06-22T08:00:46.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.621+0000 7f5c58ff9700 1 -- 172.21.15.23:0/4142999622 <== mon.2 v2:172.21.15.67:3301/0 5 ==== fsmap(e 19) v1 ==== 4260+0+0 (secure 0 0 0) 0x7f5c740281a0 con 0x565399d3f7c0 2023-06-22T08:00:46.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.621+0000 7f5c93a2bb80 10 client.4688 fetch_fsmap finished waiting for FSMap version 19 2023-06-22T08:00:46.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.621+0000 7f5c93a2bb80 10 client.4688 resolve_mds: resolved name 'e' to daemon mds.e 2023-06-22T08:00:46.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.621+0000 7f5c93a2bb80 20 client.4688 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:46.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.621+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.23:6836/1762110734,v1:172.21.15.23:6837/1762110734] conn(0x565399de3b70 0x565399d8ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.621+0000 7f5c93a2bb80 4 client.4688 mds_command: new command op to 4268 tid=0[{"prefix": "status"}] 2023-06-22T08:00:46.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.621+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/4142999622 --> [v2:172.21.15.23:6836/1762110734,v1:172.21.15.23:6837/1762110734] -- command(tid 0: {"prefix": "status"}) v1 -- 0x565399c4f000 con 0x565399de3b70 2023-06-22T08:00:46.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.621+0000 7f5c7d79a700 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.23:6836/1762110734,v1:172.21.15.23:6837/1762110734] conn(0x565399de3b70 0x565399d8ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c7d79a700 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.23:6836/1762110734,v1:172.21.15.23:6837/1762110734] conn(0x565399de3b70 0x565399d8ae80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1dbd94159b947085 server_cookie=ed8c4a99f4be2451 in_seq=0 out_seq=0 2023-06-22T08:00:46.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c58ff9700 10 client.4688 ms_handle_connect on v2:172.21.15.23:6836/1762110734 2023-06-22T08:00:46.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c58ff9700 1 -- 172.21.15.23:0/4142999622 <== mds.4 v2:172.21.15.23:6836/1762110734 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x565399c4f000 con 0x565399de3b70 2023-06-22T08:00:46.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c58ff9700 10 client.4688 handle_command_reply: tid=0 2023-06-22T08:00:46.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c93a2bb80 1 client.4688 shutdown 2023-06-22T08:00:46.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/4142999622 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 msgr2=0x565399d8ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d8ff90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5c74028810 tx=0x7f5c74012000 comp rx=0 tx=0).stop 2023-06-22T08:00:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/4142999622 shutdown_connections 2023-06-22T08:00:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.23:6836/1762110734,v1:172.21.15.23:6837/1762110734] conn(0x565399de3b70 0x565399d8ae80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565399d82520 0x565399de3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x565399d3f7c0 0x565399d8ff90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c93a2bb80 1 --2- 172.21.15.23:0/4142999622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565399d82150 0x565399dab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/4142999622 >> 172.21.15.23:0/4142999622 conn(0x565399d37fc0 msgr2=0x565399d43880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.622+0000 7f5c58ff9700 0 client.4688 ms_handle_reset on v2:172.21.15.23:6836/1762110734 2023-06-22T08:00:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.623+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/4142999622 shutdown_connections 2023-06-22T08:00:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.623+0000 7f5c93a2bb80 1 -- 172.21.15.23:0/4142999622 wait complete. 2023-06-22T08:00:46.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.623+0000 7f5c93a2bb80 20 client.4688 trim_cache size 0 max 16384 2023-06-22T08:00:46.632 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-06-22T08:00:46.632 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "2f053100-95a5-44db-a4cd-e7710a2fac9d", 2023-06-22T08:00:46.633 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 4, 2023-06-22T08:00:46.633 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 4268, 2023-06-22T08:00:46.633 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-06-22T08:00:46.633 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-06-22T08:00:46.633 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-06-22T08:00:46.633 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 10.067943676000001, 2023-06-22T08:00:46.633 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 19, 2023-06-22T08:00:46.634 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 28, 2023-06-22T08:00:46.634 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 28, 2023-06-22T08:00:46.634 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 56.975650510999998 2023-06-22T08:00:46.634 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-06-22T08:00:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.623+0000 7f5c801ff700 1 -- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5c5c06ad10 msgr2=0x7f5c5c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.623+0000 7f5c801ff700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5c5c06ad10 0x7f5c5c06d1c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5c74000f60 tx=0x7f5c74007e70 comp rx=0 tx=0).stop 2023-06-22T08:00:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.623+0000 7f5c801ff700 1 -- 172.21.15.23:0/1700493700 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c78152d70 msgr2=0x7f5c78155020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.623+0000 7f5c801ff700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c78152d70 0x7f5c78155020 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5c7808bcd0 tx=0x7f5c70004ad0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.624+0000 7f5c801ff700 1 -- 172.21.15.23:0/1700493700 shutdown_connections 2023-06-22T08:00:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.624+0000 7f5c801ff700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c78152d70 0x7f5c78155020 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.624+0000 7f5c801ff700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5c780c45f0 0x7f5c7815aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.624+0000 7f5c801ff700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5c5c06ad10 0x7f5c5c06d1c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.624+0000 7f5c801ff700 1 --2- 172.21.15.23:0/1700493700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c780c97d0 0x7f5c78154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.624+0000 7f5c801ff700 1 -- 172.21.15.23:0/1700493700 >> 172.21.15.23:0/1700493700 conn(0x7f5c7806ae30 msgr2=0x7f5c780cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.624+0000 7f5c801ff700 1 -- 172.21.15.23:0/1700493700 shutdown_connections 2023-06-22T08:00:46.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.624+0000 7f5c801ff700 1 -- 172.21.15.23:0/1700493700 wait complete. 2023-06-22T08:00:46.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-06-22T08:00:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.848+0000 7f0b2cdff700 1 Processor -- start 2023-06-22T08:00:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.848+0000 7f0b2cdff700 1 -- start start 2023-06-22T08:00:46.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.849+0000 7f0b2cdff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0b280c9fd0 0x7f0b280cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.849+0000 7f0b2cdff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0b280c40b0 0x7f0b280c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.849+0000 7f0b2cdff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2815c370 con 0x7f0b28152ce0 2023-06-22T08:00:46.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.849+0000 7f0b2cdff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b281530b0 con 0x7f0b280c9fd0 2023-06-22T08:00:46.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.849+0000 7f0b2cdff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b280cf6f0 con 0x7f0b280c40b0 2023-06-22T08:00:46.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.850+0000 7f0b25d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0b280c9fd0 0x7f0b280cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.850+0000 7f0b25d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0b280c9fd0 0x7f0b280cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35378/0 (socket says 172.21.15.23:35378) 2023-06-22T08:00:46.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.850+0000 7f0b25d9b700 1 -- 172.21.15.23:0/158277390 learned_addr learned my addr 172.21.15.23:0/158277390 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:46.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.850+0000 7f0b26d9d700 1 --2- 172.21.15.23:0/158277390 >> v2:172.21.15.67:3300/0 conn(0x7f0b280c40b0 0x7f0b280c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.850+0000 7f0b25d9b700 1 -- 172.21.15.23:0/158277390 >> v2:172.21.15.67:3300/0 conn(0x7f0b280c40b0 msgr2=0x7f0b280c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.850+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/158277390 >> v2:172.21.15.67:3300/0 conn(0x7f0b280c40b0 0x7f0b280c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.850+0000 7f0b25d9b700 1 -- 172.21.15.23:0/158277390 >> v1:172.21.15.67:6789/0 conn(0x7f0b28152ce0 legacy=0x7f0b2815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.850+0000 7f0b25d9b700 1 -- 172.21.15.23:0/158277390 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2815bda0 con 0x7f0b280c9fd0 2023-06-22T08:00:46.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.850+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/158277390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b280c9fd0 0x7f0b280cf1b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0b280b4380 tx=0x7f0b1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1eaf23e67e36961 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:46.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.850+0000 7f0b2559a700 1 -- 172.21.15.23:0/158277390 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c011830 con 0x7f0b280c9fd0 2023-06-22T08:00:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.851+0000 7f0b2559a700 1 -- 172.21.15.23:0/158277390 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c011990 con 0x7f0b280c9fd0 2023-06-22T08:00:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.851+0000 7f0b2559a700 1 -- 172.21.15.23:0/158277390 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c011d20 con 0x7f0b280c9fd0 2023-06-22T08:00:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.851+0000 7f0b2cdff700 1 -- 172.21.15.23:0/158277390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b280c9fd0 msgr2=0x7f0b280cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.851+0000 7f0b2cdff700 1 --2- 172.21.15.23:0/158277390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b280c9fd0 0x7f0b280cf1b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0b280b4380 tx=0x7f0b1c009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.851+0000 7f0b2cdff700 1 -- 172.21.15.23:0/158277390 shutdown_connections 2023-06-22T08:00:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.851+0000 7f0b2cdff700 1 --2- 172.21.15.23:0/158277390 >> v2:172.21.15.67:3300/0 conn(0x7f0b280c40b0 0x7f0b280c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.851+0000 7f0b2cdff700 1 --2- 172.21.15.23:0/158277390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b280c9fd0 0x7f0b280cf1b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.851+0000 7f0b2cdff700 1 -- 172.21.15.23:0/158277390 >> 172.21.15.23:0/158277390 conn(0x7f0b2806ae30 msgr2=0x7f0b2806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.851+0000 7f0b2cdff700 1 -- 172.21.15.23:0/158277390 shutdown_connections 2023-06-22T08:00:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.851+0000 7f0b2cdff700 1 -- 172.21.15.23:0/158277390 wait complete. 2023-06-22T08:00:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.853+0000 7f0b2cdff700 1 Processor -- start 2023-06-22T08:00:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.853+0000 7f0b2cdff700 1 -- start start 2023-06-22T08:00:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.854+0000 7f0b2cdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0b280c40b0 0x7f0b281e2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.854+0000 7f0b2cdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b280c9fd0 0x7f0b28155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.854+0000 7f0b2cdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28152ce0 0x7f0b280cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.854+0000 7f0b2cdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2814fcb0 con 0x7f0b280c9fd0 2023-06-22T08:00:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.854+0000 7f0b2cdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28156010 con 0x7f0b28152ce0 2023-06-22T08:00:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.854+0000 7f0b2cdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b281e27f0 con 0x7f0b280c40b0 2023-06-22T08:00:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.854+0000 7f0b25d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b280c9fd0 0x7f0b28155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.854+0000 7f0b2659c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0b280c40b0 0x7f0b281e2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.854+0000 7f0b26d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28152ce0 0x7f0b280cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.854+0000 7f0b2659c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0b280c40b0 0x7f0b281e2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:48770/0 (socket says 172.21.15.23:48770) 2023-06-22T08:00:46.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b2659c700 1 -- 172.21.15.23:0/3757181612 learned_addr learned my addr 172.21.15.23:0/3757181612 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:46.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b25d9b700 1 -- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0b280c40b0 msgr2=0x7f0b281e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0b280c40b0 0x7f0b281e2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b25d9b700 1 -- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28152ce0 msgr2=0x7f0b280cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28152ce0 0x7f0b280cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b25d9b700 1 -- 172.21.15.23:0/3757181612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c009550 con 0x7f0b280c9fd0 2023-06-22T08:00:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b2659c700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0b280c40b0 0x7f0b281e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b280c9fd0 0x7f0b28155ad0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c016000 tx=0x7f0b1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b16ffd700 1 -- 172.21.15.23:0/3757181612 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c004590 con 0x7f0b280c9fd0 2023-06-22T08:00:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b16ffd700 1 -- 172.21.15.23:0/3757181612 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c0036a0 con 0x7f0b280c9fd0 2023-06-22T08:00:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b2cdff700 1 -- 172.21.15.23:0/3757181612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b281e2ad0 con 0x7f0b280c9fd0 2023-06-22T08:00:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.855+0000 7f0b16ffd700 1 -- 172.21.15.23:0/3757181612 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c003a50 con 0x7f0b280c9fd0 2023-06-22T08:00:46.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.856+0000 7f0b2cdff700 1 -- 172.21.15.23:0/3757181612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b281e3090 con 0x7f0b280c9fd0 2023-06-22T08:00:46.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.857+0000 7f0b16ffd700 1 -- 172.21.15.23:0/3757181612 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0b1c003bb0 con 0x7f0b280c9fd0 2023-06-22T08:00:46.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.858+0000 7f0b16ffd700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0b0806ace0 0x7f0b0806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.859+0000 7f0b16ffd700 1 -- 172.21.15.23:0/3757181612 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0b1c021030 con 0x7f0b280c9fd0 2023-06-22T08:00:46.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.859+0000 7f0b2659c700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0b0806ace0 0x7f0b0806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.859+0000 7f0b2659c700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0b0806ace0 0x7f0b0806d190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0b18001c70 tx=0x7f0b18009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:46.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.861+0000 7f0b406a0b80 1 Processor -- start 2023-06-22T08:00:46.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.861+0000 7f0b406a0b80 1 -- start start 2023-06-22T08:00:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f95662f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 0x562f956634c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95690880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f9556dc60 con 0x562f956657c0 2023-06-22T08:00:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f955c09f0 con 0x562f956a8520 2023-06-22T08:00:46.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x562f955b9e80 con 0x562f956a8150 2023-06-22T08:00:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b26d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95690880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b25d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 0x562f956634c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b2659c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f95662f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b26d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95690880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35396/0 (socket says 172.21.15.23:35396) 2023-06-22T08:00:46.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b26d9d700 1 -- 172.21.15.23:0/3514863342 learned_addr learned my addr 172.21.15.23:0/3514863342 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:46.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b25d9b700 1 -- 172.21.15.23:0/3514863342 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 msgr2=0x562f95662f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3514863342 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f95662f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b25d9b700 1 -- 172.21.15.23:0/3514863342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 msgr2=0x562f95690880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3514863342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95690880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b25d9b700 1 -- 172.21.15.23:0/3514863342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f9543b5e0 con 0x562f956a8520 2023-06-22T08:00:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.862+0000 7f0b26d9d700 1 --2- 172.21.15.23:0/3514863342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95690880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.863+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3514863342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 0x562f956634c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c088df0 tx=0x7f0b1c055b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f125eb6d1c729e00 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:46.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.863+0000 7f0b07fff700 1 -- 172.21.15.23:0/3514863342 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c05d7b0 con 0x562f956a8520 2023-06-22T08:00:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.863+0000 7f0b07fff700 1 -- 172.21.15.23:0/3514863342 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c061b40 con 0x562f956a8520 2023-06-22T08:00:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.863+0000 7f0b07fff700 1 -- 172.21.15.23:0/3514863342 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c05abe0 con 0x562f956a8520 2023-06-22T08:00:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.863+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3514863342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 msgr2=0x562f956634c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.863+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3514863342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 0x562f956634c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c088df0 tx=0x7f0b1c055b30 comp rx=0 tx=0).stop 2023-06-22T08:00:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.864+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3514863342 shutdown_connections 2023-06-22T08:00:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.864+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3514863342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95690880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.864+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3514863342 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f95662f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.864+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3514863342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 0x562f956634c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.864+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3514863342 >> 172.21.15.23:0/3514863342 conn(0x562f9565dfc0 msgr2=0x562f956666a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.864+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3514863342 shutdown_connections 2023-06-22T08:00:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.864+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3514863342 wait complete. 2023-06-22T08:00:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.864+0000 7f0b406a0b80 1 -- start start 2023-06-22T08:00:46.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.865+0000 7f0b406a0b80 1 Processor -- start 2023-06-22T08:00:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 0x562f95692ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f956b9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8520 0x562f95662c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f95559a70 con 0x562f956a8520 2023-06-22T08:00:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f95559770 con 0x562f956657c0 2023-06-22T08:00:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x562f9543d8d0 con 0x562f956a8150 2023-06-22T08:00:46.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b406a0b80 10 client.0 _ll_register_callbacks cb 0x562f9555f210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b26d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8520 0x562f95662c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b2659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 0x562f95692ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b26d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8520 0x562f95662c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35400/0 (socket says 172.21.15.23:35400) 2023-06-22T08:00:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f956b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b26d9d700 1 -- 172.21.15.23:0/3681182140 learned_addr learned my addr 172.21.15.23:0/3681182140 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b26d9d700 1 -- 172.21.15.23:0/3681182140 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 msgr2=0x562f956b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b26d9d700 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f956b9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b26d9d700 1 -- 172.21.15.23:0/3681182140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 msgr2=0x562f95692ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b26d9d700 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 0x562f95692ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b26d9d700 1 -- 172.21.15.23:0/3681182140 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b20011230 con 0x562f956a8520 2023-06-22T08:00:46.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.866+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f956b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:46.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b26d9d700 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8520 0x562f95662c20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b20004690 tx=0x7f0b200049a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376d9d501d302547 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:46.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b057fa700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-06-22T08:00:46.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b057fa700 1 -- 172.21.15.23:0/3681182140 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2000e040 con 0x562f956a8520 2023-06-22T08:00:46.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b057fa700 1 -- 172.21.15.23:0/3681182140 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0b20011c80 con 0x562f956a8520 2023-06-22T08:00:46.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3681182140 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562f9565b780 con 0x562f956a8520 2023-06-22T08:00:46.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b057fa700 1 -- 172.21.15.23:0/3681182140 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b20011e40 con 0x562f956a8520 2023-06-22T08:00:46.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b406a0b80 10 client.4866 fetch_fsmap learned FSMap version 19 2023-06-22T08:00:46.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3681182140 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x7f0b281e3090 con 0x562f956a8520 2023-06-22T08:00:46.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b057fa700 1 -- 172.21.15.23:0/3681182140 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562f9565b780 con 0x562f956a8520 2023-06-22T08:00:46.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b057fa700 1 -- 172.21.15.23:0/3681182140 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 19) v1 ==== 4260+0+0 (secure 0 0 0) 0x7f0b2001c070 con 0x562f956a8520 2023-06-22T08:00:46.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.867+0000 7f0b406a0b80 10 client.4866 fetch_fsmap finished waiting for FSMap version 19 2023-06-22T08:00:46.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.868+0000 7f0b406a0b80 10 client.4866 resolve_mds: resolved name 'c' to daemon mds.c 2023-06-22T08:00:46.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.868+0000 7f0b406a0b80 20 client.4866 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:46.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.868+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.23:6832/3365097525,v1:172.21.15.23:6834/3365097525] conn(0x562f956a2bf0 0x562f9568e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.868+0000 7f0b406a0b80 4 client.4866 mds_command: new command op to 4269 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-22T08:00:46.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.868+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3681182140 --> [v2:172.21.15.23:6832/3365097525,v1:172.21.15.23:6834/3365097525] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562f955b2e60 con 0x562f956a2bf0 2023-06-22T08:00:46.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.868+0000 7f0b2659c700 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.23:6832/3365097525,v1:172.21.15.23:6834/3365097525] conn(0x562f956a2bf0 0x562f9568e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.868+0000 7f0b2659c700 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.23:6832/3365097525,v1:172.21.15.23:6834/3365097525] conn(0x562f956a2bf0 0x562f9568e240 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a2d6fb9914e6b973 server_cookie=a3d3845717df7b62 in_seq=0 out_seq=0 2023-06-22T08:00:46.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.868+0000 7f0b057fa700 10 client.4866 ms_handle_connect on v2:172.21.15.23:6832/3365097525 2023-06-22T08:00:46.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.869+0000 7f0b057fa700 1 -- 172.21.15.23:0/3681182140 <== mds.3 v2:172.21.15.23:6832/3365097525 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562f955b2e60 con 0x562f956a2bf0 2023-06-22T08:00:46.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.869+0000 7f0b057fa700 10 client.4866 handle_command_reply: tid=0 2023-06-22T08:00:46.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.869+0000 7f0b406a0b80 1 client.4866 shutdown 2023-06-22T08:00:46.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.869+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3681182140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8520 msgr2=0x562f95662c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.869+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8520 0x562f95662c20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b20004690 tx=0x7f0b200049a0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.870+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3681182140 shutdown_connections 2023-06-22T08:00:46.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.870+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.23:6832/3365097525,v1:172.21.15.23:6834/3365097525] conn(0x562f956a2bf0 0x562f9568e240 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.870+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8520 0x562f95662c20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.870+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 0x562f95692ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.870+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3681182140 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f956b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.870+0000 7f0b057fa700 0 client.4866 ms_handle_reset on v2:172.21.15.23:6832/3365097525 2023-06-22T08:00:46.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.870+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3681182140 >> 172.21.15.23:0/3681182140 conn(0x562f9565dfc0 msgr2=0x562f95668dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:46.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.870+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3681182140 shutdown_connections 2023-06-22T08:00:46.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.870+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3681182140 wait complete. 2023-06-22T08:00:46.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.870+0000 7f0b406a0b80 20 client.4866 trim_cache size 0 max 16384 2023-06-22T08:00:46.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.888+0000 7f0b406a0b80 1 Processor -- start 2023-06-22T08:00:46.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.888+0000 7f0b406a0b80 1 -- start start 2023-06-22T08:00:46.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.889+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95667ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.889+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f956a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.889+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 0x562f956a3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.889+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f956938a0 con 0x562f956657c0 2023-06-22T08:00:46.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.889+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f95664e80 con 0x562f956a8520 2023-06-22T08:00:46.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.889+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x562f956655b0 con 0x562f956a8150 2023-06-22T08:00:46.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.889+0000 7f0b2659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95667ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.889+0000 7f0b2659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95667ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35404/0 (socket says 172.21.15.23:35404) 2023-06-22T08:00:46.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.889+0000 7f0b2659c700 1 -- 172.21.15.23:0/4050235869 learned_addr learned my addr 172.21.15.23:0/4050235869 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:46.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.890+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/4050235869 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f956a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.890+0000 7f0b26d9d700 1 --2- 172.21.15.23:0/4050235869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 0x562f956a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.890+0000 7f0b2659c700 1 -- 172.21.15.23:0/4050235869 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 msgr2=0x562f956a32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.890+0000 7f0b2659c700 1 --2- 172.21.15.23:0/4050235869 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f956a32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.890+0000 7f0b2659c700 1 -- 172.21.15.23:0/4050235869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 msgr2=0x562f956a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.890+0000 7f0b2659c700 1 --2- 172.21.15.23:0/4050235869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 0x562f956a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.890+0000 7f0b2659c700 1 -- 172.21.15.23:0/4050235869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f9568d150 con 0x562f956657c0 2023-06-22T08:00:46.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.890+0000 7f0b2659c700 1 --2- 172.21.15.23:0/4050235869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95667ad0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800f840 tx=0x7f0b1800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2022fd27b1a9082e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:46.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b14ff9700 1 -- 172.21.15.23:0/4050235869 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18028040 con 0x562f956657c0 2023-06-22T08:00:46.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b14ff9700 1 -- 172.21.15.23:0/4050235869 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b180160d0 con 0x562f956657c0 2023-06-22T08:00:46.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b14ff9700 1 -- 172.21.15.23:0/4050235869 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18029240 con 0x562f956657c0 2023-06-22T08:00:46.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b406a0b80 1 -- 172.21.15.23:0/4050235869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 msgr2=0x562f95667ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/4050235869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95667ad0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800f840 tx=0x7f0b1800a000 comp rx=0 tx=0).stop 2023-06-22T08:00:46.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b406a0b80 1 -- 172.21.15.23:0/4050235869 shutdown_connections 2023-06-22T08:00:46.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/4050235869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956a8520 0x562f956a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/4050235869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956657c0 0x562f95667ad0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/4050235869 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8150 0x562f956a32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b406a0b80 1 -- 172.21.15.23:0/4050235869 >> 172.21.15.23:0/4050235869 conn(0x562f9565dfc0 msgr2=0x562f956cb820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:46.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b406a0b80 1 -- 172.21.15.23:0/4050235869 shutdown_connections 2023-06-22T08:00:46.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.891+0000 7f0b406a0b80 1 -- 172.21.15.23:0/4050235869 wait complete. 2023-06-22T08:00:46.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.892+0000 7f0b406a0b80 1 -- start start 2023-06-22T08:00:46.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.892+0000 7f0b406a0b80 1 Processor -- start 2023-06-22T08:00:46.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.893+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 0x562f95692520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.893+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8150 0x562f95713db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.893+0000 7f0b406a0b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8520 0x562f9570a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.893+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f957147c0 con 0x562f956a8150 2023-06-22T08:00:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.893+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f95668100 con 0x562f956657c0 2023-06-22T08:00:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.893+0000 7f0b406a0b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x562f95691b20 con 0x562f956a8520 2023-06-22T08:00:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.893+0000 7f0b406a0b80 10 client.0 _ll_register_callbacks cb 0x562f9555f210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.893+0000 7f0b2659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 0x562f95692520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b25d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8150 0x562f95713db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b2659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 0x562f95692520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51070/0 (socket says 172.21.15.23:51070) 2023-06-22T08:00:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b2659c700 1 -- 172.21.15.23:0/3161929453 learned_addr learned my addr 172.21.15.23:0/3161929453 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b26d9d700 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8520 0x562f9570a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b25d9b700 1 -- 172.21.15.23:0/3161929453 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8520 msgr2=0x562f9570a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8520 0x562f9570a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b25d9b700 1 -- 172.21.15.23:0/3161929453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 msgr2=0x562f95692520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 0x562f95692520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b25d9b700 1 -- 172.21.15.23:0/3161929453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2815c370 con 0x562f956a8150 2023-06-22T08:00:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b2659c700 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 0x562f95692520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b25d9b700 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8150 0x562f95713db0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c059780 tx=0x7f0b1c05c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a89142e90f46048 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b057fa700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-06-22T08:00:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b057fa700 1 -- 172.21.15.23:0/3161929453 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c05ade0 con 0x562f956a8150 2023-06-22T08:00:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b057fa700 1 -- 172.21.15.23:0/3161929453 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0b1c05d7b0 con 0x562f956a8150 2023-06-22T08:00:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.894+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3161929453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b1c061b40 con 0x562f956a8150 2023-06-22T08:00:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b057fa700 1 -- 172.21.15.23:0/3161929453 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b1c058ce0 con 0x562f956a8150 2023-06-22T08:00:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b406a0b80 10 client.4872 fetch_fsmap learned FSMap version 19 2023-06-22T08:00:46.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3161929453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x562f953e7d90 con 0x562f956a8150 2023-06-22T08:00:46.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b057fa700 1 -- 172.21.15.23:0/3161929453 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c061b40 con 0x562f956a8150 2023-06-22T08:00:46.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b057fa700 1 -- 172.21.15.23:0/3161929453 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 19) v1 ==== 4260+0+0 (secure 0 0 0) 0x7f0b1c0616b0 con 0x562f956a8150 2023-06-22T08:00:46.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b406a0b80 10 client.4872 fetch_fsmap finished waiting for FSMap version 19 2023-06-22T08:00:46.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b406a0b80 10 client.4872 resolve_mds: resolved name 'c' to daemon mds.c 2023-06-22T08:00:46.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b406a0b80 20 client.4872 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:46.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.23:6832/3365097525,v1:172.21.15.23:6834/3365097525] conn(0x562f956a5ca0 0x562f95668a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:46.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b406a0b80 4 client.4872 mds_command: new command op to 4269 tid=0[{"prefix": "status"}] 2023-06-22T08:00:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3161929453 --> [v2:172.21.15.23:6832/3365097525,v1:172.21.15.23:6834/3365097525] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562f95575000 con 0x562f956a5ca0 2023-06-22T08:00:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.895+0000 7f0b2659c700 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.23:6832/3365097525,v1:172.21.15.23:6834/3365097525] conn(0x562f956a5ca0 0x562f95668a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b2659c700 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.23:6832/3365097525,v1:172.21.15.23:6834/3365097525] conn(0x562f956a5ca0 0x562f95668a50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8ae6a4922ce4a239 server_cookie=740f0e05c088df07 in_seq=0 out_seq=0 2023-06-22T08:00:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b057fa700 10 client.4872 ms_handle_connect on v2:172.21.15.23:6832/3365097525 2023-06-22T08:00:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b057fa700 1 -- 172.21.15.23:0/3161929453 <== mds.3 v2:172.21.15.23:6832/3365097525 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x562f95575000 con 0x562f956a5ca0 2023-06-22T08:00:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b057fa700 10 client.4872 handle_command_reply: tid=0 2023-06-22T08:00:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b406a0b80 1 client.4872 shutdown 2023-06-22T08:00:46.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3161929453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8150 msgr2=0x562f95713db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8150 0x562f95713db0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c059780 tx=0x7f0b1c05c000 comp rx=0 tx=0).stop 2023-06-22T08:00:46.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3161929453 shutdown_connections 2023-06-22T08:00:46.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.23:6832/3365097525,v1:172.21.15.23:6834/3365097525] conn(0x562f956a5ca0 0x562f95668a50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x562f956a8520 0x562f9570a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562f956657c0 0x562f95692520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b057fa700 0 client.4872 ms_handle_reset on v2:172.21.15.23:6832/3365097525 2023-06-22T08:00:46.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b406a0b80 1 --2- 172.21.15.23:0/3161929453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562f956a8150 0x562f95713db0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.896+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3161929453 >> 172.21.15.23:0/3161929453 conn(0x562f9565dfc0 msgr2=0x562f95664380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:46.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3161929453 shutdown_connections 2023-06-22T08:00:46.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b406a0b80 1 -- 172.21.15.23:0/3161929453 wait complete. 2023-06-22T08:00:46.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b406a0b80 20 client.4872 trim_cache size 0 max 16384 2023-06-22T08:00:46.907 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-06-22T08:00:46.907 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "2f053100-95a5-44db-a4cd-e7710a2fac9d", 2023-06-22T08:00:46.908 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 3, 2023-06-22T08:00:46.908 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 4269, 2023-06-22T08:00:46.908 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-06-22T08:00:46.908 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-06-22T08:00:46.908 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-06-22T08:00:46.908 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 15.417564622, 2023-06-22T08:00:46.908 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 19, 2023-06-22T08:00:46.909 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 28, 2023-06-22T08:00:46.909 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 28, 2023-06-22T08:00:46.909 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 57.266141482000002 2023-06-22T08:00:46.909 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-06-22T08:00:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 -- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0b0806ace0 msgr2=0x7f0b0806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0b0806ace0 0x7f0b0806d190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0b18001c70 tx=0x7f0b18009040 comp rx=0 tx=0).stop 2023-06-22T08:00:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 -- 172.21.15.23:0/3757181612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b280c9fd0 msgr2=0x7f0b28155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b280c9fd0 0x7f0b28155ad0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c016000 tx=0x7f0b1c004750 comp rx=0 tx=0).stop 2023-06-22T08:00:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 -- 172.21.15.23:0/3757181612 shutdown_connections 2023-06-22T08:00:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0b0806ace0 0x7f0b0806d190 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b28152ce0 0x7f0b280cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0b280c40b0 0x7f0b281e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 --2- 172.21.15.23:0/3757181612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b280c9fd0 0x7f0b28155ad0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 -- 172.21.15.23:0/3757181612 >> 172.21.15.23:0/3757181612 conn(0x7f0b2806ae30 msgr2=0x7f0b2814c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 -- 172.21.15.23:0/3757181612 shutdown_connections 2023-06-22T08:00:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:46.897+0000 7f0b2cdff700 1 -- 172.21.15.23:0/3757181612 wait complete. 2023-06-22T08:00:46.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-06-22T08:00:47.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.113+0000 7ff1087ff700 1 Processor -- start 2023-06-22T08:00:47.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.113+0000 7ff1087ff700 1 -- start start 2023-06-22T08:00:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.115+0000 7ff1087ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1000c9fd0 0x7ff1000cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.115+0000 7ff1087ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10015c370 con 0x7ff1000c40b0 2023-06-22T08:00:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.115+0000 7ff1087ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10015bda0 con 0x7ff100152ce0 2023-06-22T08:00:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.115+0000 7ff1087ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1001530b0 con 0x7ff1000c9fd0 2023-06-22T08:00:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.115+0000 7ff106d9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff1000c40b0 0x7ff1000c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34900/0 (socket says 172.21.15.23:34900) 2023-06-22T08:00:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.115+0000 7ff106d9c700 1 -- 172.21.15.23:0/3488697947 learned_addr learned my addr 172.21.15.23:0/3488697947 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.115+0000 7ff105d9a700 1 --2- 172.21.15.23:0/3488697947 >> v2:172.21.15.67:3300/0 conn(0x7ff1000c9fd0 0x7ff1000cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105599700 1 -- 172.21.15.23:0/3488697947 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851943636 0 0) 0x7ff10015c370 con 0x7ff1000c40b0 2023-06-22T08:00:47.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105599700 1 -- 172.21.15.23:0/3488697947 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0ec0035b0 con 0x7ff1000c40b0 2023-06-22T08:00:47.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105599700 1 -- 172.21.15.23:0/3488697947 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296433831 0 0) 0x7ff10015bda0 con 0x7ff100152ce0 2023-06-22T08:00:47.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105599700 1 -- 172.21.15.23:0/3488697947 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff10015c370 con 0x7ff100152ce0 2023-06-22T08:00:47.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105d9a700 1 -- 172.21.15.23:0/3488697947 >> v1:172.21.15.67:6789/0 conn(0x7ff100152ce0 legacy=0x7ff10015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105d9a700 1 -- 172.21.15.23:0/3488697947 >> v1:172.21.15.23:6789/0 conn(0x7ff1000c40b0 legacy=0x7ff1000c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105d9a700 1 -- 172.21.15.23:0/3488697947 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1000c9460 con 0x7ff1000c9fd0 2023-06-22T08:00:47.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105d9a700 1 --2- 172.21.15.23:0/3488697947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1000c9fd0 0x7ff1000cf1b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009840 tx=0x7ff0f0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fe4bac203b14f28 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105599700 1 -- 172.21.15.23:0/3488697947 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0011830 con 0x7ff1000c9fd0 2023-06-22T08:00:47.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105599700 1 -- 172.21.15.23:0/3488697947 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f0011990 con 0x7ff1000c9fd0 2023-06-22T08:00:47.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.116+0000 7ff105599700 1 -- 172.21.15.23:0/3488697947 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0011d20 con 0x7ff1000c9fd0 2023-06-22T08:00:47.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.117+0000 7ff1087ff700 1 -- 172.21.15.23:0/3488697947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1000c9fd0 msgr2=0x7ff1000cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.117+0000 7ff1087ff700 1 --2- 172.21.15.23:0/3488697947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1000c9fd0 0x7ff1000cf1b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009840 tx=0x7ff0f0009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:47.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.117+0000 7ff1087ff700 1 -- 172.21.15.23:0/3488697947 shutdown_connections 2023-06-22T08:00:47.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.117+0000 7ff1087ff700 1 --2- 172.21.15.23:0/3488697947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1000c9fd0 0x7ff1000cf1b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.117+0000 7ff1087ff700 1 -- 172.21.15.23:0/3488697947 >> 172.21.15.23:0/3488697947 conn(0x7ff10006ae30 msgr2=0x7ff10006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.117+0000 7ff1087ff700 1 -- 172.21.15.23:0/3488697947 shutdown_connections 2023-06-22T08:00:47.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.117+0000 7ff1087ff700 1 -- 172.21.15.23:0/3488697947 wait complete. 2023-06-22T08:00:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.119+0000 7ff1087ff700 1 Processor -- start 2023-06-22T08:00:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.119+0000 7ff1087ff700 1 -- start start 2023-06-22T08:00:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff1087ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1000c40b0 0x7ff1000ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff1087ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1000c9fd0 0x7ff10014a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff1087ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff100152ce0 0x7ff100150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff1087ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10014ea30 con 0x7ff1000c9fd0 2023-06-22T08:00:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff1087ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10014fb90 con 0x7ff1000c40b0 2023-06-22T08:00:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff1087ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000cd8b0 con 0x7ff100152ce0 2023-06-22T08:00:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff105d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1000c9fd0 0x7ff10014a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff105d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1000c9fd0 0x7ff10014a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35412/0 (socket says 172.21.15.23:35412) 2023-06-22T08:00:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff105d9a700 1 -- 172.21.15.23:0/2709404237 learned_addr learned my addr 172.21.15.23:0/2709404237 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff10659b700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1000c40b0 0x7ff1000ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff106d9c700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff100152ce0 0x7ff100150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff105d9a700 1 -- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff100152ce0 msgr2=0x7ff100150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff105d9a700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff100152ce0 0x7ff100150e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff105d9a700 1 -- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1000c40b0 msgr2=0x7ff1000ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff105d9a700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1000c40b0 0x7ff1000ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff105d9a700 1 -- 172.21.15.23:0/2709404237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f00094a0 con 0x7ff1000c9fd0 2023-06-22T08:00:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.120+0000 7ff10659b700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1000c40b0 0x7ff1000ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.121+0000 7ff105d9a700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1000c9fd0 0x7ff10014a580 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009b90 tx=0x7ff0f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.121+0000 7ff0f6ffd700 1 -- 172.21.15.23:0/2709404237 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f001a860 con 0x7ff1000c9fd0 2023-06-22T08:00:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.121+0000 7ff0f6ffd700 1 -- 172.21.15.23:0/2709404237 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f001a9c0 con 0x7ff1000c9fd0 2023-06-22T08:00:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.121+0000 7ff0f6ffd700 1 -- 172.21.15.23:0/2709404237 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f001ad30 con 0x7ff1000c9fd0 2023-06-22T08:00:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.121+0000 7ff1087ff700 1 -- 172.21.15.23:0/2709404237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1001f1b80 con 0x7ff1000c9fd0 2023-06-22T08:00:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.121+0000 7ff1087ff700 1 -- 172.21.15.23:0/2709404237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1000cde30 con 0x7ff1000c9fd0 2023-06-22T08:00:47.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.123+0000 7ff0f6ffd700 1 -- 172.21.15.23:0/2709404237 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff0f0022070 con 0x7ff1000c9fd0 2023-06-22T08:00:47.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.125+0000 7ff0f6ffd700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff0e406ace0 0x7ff0e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.125+0000 7ff0f6ffd700 1 -- 172.21.15.23:0/2709404237 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff0f008a540 con 0x7ff1000c9fd0 2023-06-22T08:00:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.125+0000 7ff10659b700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff0e406ace0 0x7ff0e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.126+0000 7ff10659b700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff0e406ace0 0x7ff0e406d190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0011e0 tx=0x7ff0fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.127+0000 7ff11c1cab80 1 Processor -- start 2023-06-22T08:00:47.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.127+0000 7ff11c1cab80 1 -- start start 2023-06-22T08:00:47.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 0x55e87f64cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f64d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f67a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87f557c60 con 0x55e87f64f7c0 2023-06-22T08:00:47.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87f5aa9f0 con 0x55e87f692150 2023-06-22T08:00:47.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e87f5a3e80 con 0x55e87f692520 2023-06-22T08:00:47.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff106d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f67a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff105d9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f64d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff10659b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 0x55e87f64cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff106d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f67a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35426/0 (socket says 172.21.15.23:35426) 2023-06-22T08:00:47.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff105d9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f64d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:48824/0 (socket says 172.21.15.23:48824) 2023-06-22T08:00:47.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.128+0000 7ff106d9c700 1 -- 172.21.15.23:0/1069061490 learned_addr learned my addr 172.21.15.23:0/1069061490 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff10659b700 1 -- 172.21.15.23:0/1069061490 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 msgr2=0x55e87f64d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff10659b700 1 --2- 172.21.15.23:0/1069061490 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f64d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff10659b700 1 -- 172.21.15.23:0/1069061490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 msgr2=0x55e87f67a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff10659b700 1 --2- 172.21.15.23:0/1069061490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f67a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff10659b700 1 -- 172.21.15.23:0/1069061490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e87f4255e0 con 0x55e87f692150 2023-06-22T08:00:47.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff105d9a700 1 --2- 172.21.15.23:0/1069061490 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f64d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:47.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff106d9c700 1 --2- 172.21.15.23:0/1069061490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f67a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:47.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff10659b700 1 --2- 172.21.15.23:0/1069061490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 0x55e87f64cf80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc005130 tx=0x7ff0fc011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaecd6521a25acd0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff0e3fff700 1 -- 172.21.15.23:0/1069061490 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc003b10 con 0x55e87f692150 2023-06-22T08:00:47.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff0e3fff700 1 -- 172.21.15.23:0/1069061490 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc003c70 con 0x55e87f692150 2023-06-22T08:00:47.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.129+0000 7ff0e3fff700 1 -- 172.21.15.23:0/1069061490 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc00e650 con 0x55e87f692150 2023-06-22T08:00:47.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.130+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1069061490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 msgr2=0x55e87f64cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.130+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/1069061490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 0x55e87f64cf80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc005130 tx=0x7ff0fc011000 comp rx=0 tx=0).stop 2023-06-22T08:00:47.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.130+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1069061490 shutdown_connections 2023-06-22T08:00:47.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.130+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/1069061490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f67a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.130+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/1069061490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 0x55e87f64cf80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.130+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/1069061490 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f64d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.130+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1069061490 >> 172.21.15.23:0/1069061490 conn(0x55e87f647fc0 msgr2=0x55e87f6506a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.130+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1069061490 shutdown_connections 2023-06-22T08:00:47.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.130+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1069061490 wait complete. 2023-06-22T08:00:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.131+0000 7ff11c1cab80 1 -- start start 2023-06-22T08:00:47.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.131+0000 7ff11c1cab80 1 Processor -- start 2023-06-22T08:00:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f64f7c0 0x55e87f6779a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f67b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692520 0x55e87f68c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87f543a70 con 0x55e87f692150 2023-06-22T08:00:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87f543770 con 0x55e87f692520 2023-06-22T08:00:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e87f4278d0 con 0x55e87f64f7c0 2023-06-22T08:00:47.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff11c1cab80 10 client.0 _ll_register_callbacks cb 0x55e87f549210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:47.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff105d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f67b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff10659b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f64f7c0 0x55e87f6779a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff106d9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692520 0x55e87f68c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff105d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f67b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35428/0 (socket says 172.21.15.23:35428) 2023-06-22T08:00:47.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.132+0000 7ff105d9a700 1 -- 172.21.15.23:0/1664048801 learned_addr learned my addr 172.21.15.23:0/1664048801 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff106d9c700 1 -- 172.21.15.23:0/1664048801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f64f7c0 msgr2=0x55e87f6779a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff106d9c700 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f64f7c0 0x55e87f6779a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff106d9c700 1 -- 172.21.15.23:0/1664048801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 msgr2=0x55e87f67b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff106d9c700 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f67b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff106d9c700 1 -- 172.21.15.23:0/1664048801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1000cde30 con 0x55e87f692520 2023-06-22T08:00:47.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff105d9a700 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f67b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:47.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff106d9c700 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692520 0x55e87f68c8d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8001580 tx=0x7ff0f800eab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4ddc09d41ae1356 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff0e17fa700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2023-06-22T08:00:47.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff0e17fa700 1 -- 172.21.15.23:0/1664048801 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8013040 con 0x55e87f692520 2023-06-22T08:00:47.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1664048801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e87f645780 con 0x55e87f692520 2023-06-22T08:00:47.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff0e17fa700 1 -- 172.21.15.23:0/1664048801 <== mon.1 v2:172.21.15.67:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff0f80074e0 con 0x55e87f692520 2023-06-22T08:00:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff0e17fa700 1 -- 172.21.15.23:0/1664048801 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0f80076a0 con 0x55e87f692520 2023-06-22T08:00:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff11c1cab80 10 client.4930 fetch_fsmap learned FSMap version 19 2023-06-22T08:00:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.133+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1664048801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x7ff0fc00f040 con 0x55e87f692520 2023-06-22T08:00:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.134+0000 7ff0e17fa700 1 -- 172.21.15.23:0/1664048801 <== mon.1 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e87f645780 con 0x55e87f692520 2023-06-22T08:00:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.134+0000 7ff0e17fa700 1 -- 172.21.15.23:0/1664048801 <== mon.1 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 4260+0+0 (secure 0 0 0) 0x7ff0f8007800 con 0x55e87f692520 2023-06-22T08:00:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.134+0000 7ff11c1cab80 10 client.4930 fetch_fsmap finished waiting for FSMap version 19 2023-06-22T08:00:47.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.134+0000 7ff11c1cab80 10 client.4930 resolve_mds: resolved name 'a' to daemon mds.a 2023-06-22T08:00:47.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.134+0000 7ff11c1cab80 20 client.4930 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:47.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.134+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.23:6833/3894174037,v1:172.21.15.23:6835/3894174037] conn(0x55e87f68e5a0 0x55e87f6a6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.134+0000 7ff11c1cab80 4 client.4930 mds_command: new command op to 4272 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-22T08:00:47.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.134+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1664048801 --> [v2:172.21.15.23:6833/3894174037,v1:172.21.15.23:6835/3894174037] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e87f59ce60 con 0x55e87f68e5a0 2023-06-22T08:00:47.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.134+0000 7ff10659b700 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.23:6833/3894174037,v1:172.21.15.23:6835/3894174037] conn(0x55e87f68e5a0 0x55e87f6a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.135+0000 7ff10659b700 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.23:6833/3894174037,v1:172.21.15.23:6835/3894174037] conn(0x55e87f68e5a0 0x55e87f6a6030 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=830b5adfb0caf511 server_cookie=a98ede7a1f277861 in_seq=0 out_seq=0 2023-06-22T08:00:47.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.135+0000 7ff0e17fa700 10 client.4930 ms_handle_connect on v2:172.21.15.23:6833/3894174037 2023-06-22T08:00:47.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff0e17fa700 1 -- 172.21.15.23:0/1664048801 <== mds.2 v2:172.21.15.23:6833/3894174037 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e87f59ce60 con 0x55e87f68e5a0 2023-06-22T08:00:47.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff0e17fa700 10 client.4930 handle_command_reply: tid=0 2023-06-22T08:00:47.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 client.4930 shutdown 2023-06-22T08:00:47.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1664048801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692520 msgr2=0x55e87f68c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692520 0x55e87f68c8d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8001580 tx=0x7ff0f800eab0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1664048801 shutdown_connections 2023-06-22T08:00:47.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.23:6833/3894174037,v1:172.21.15.23:6835/3894174037] conn(0x55e87f68e5a0 0x55e87f6a6030 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692520 0x55e87f68c8d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f64f7c0 0x55e87f6779a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/1664048801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f67b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff0e17fa700 0 client.4930 ms_handle_reset on v2:172.21.15.23:6833/3894174037 2023-06-22T08:00:47.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1664048801 >> 172.21.15.23:0/1664048801 conn(0x55e87f647fc0 msgr2=0x55e87f652dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1664048801 shutdown_connections 2023-06-22T08:00:47.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 1 -- 172.21.15.23:0/1664048801 wait complete. 2023-06-22T08:00:47.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.136+0000 7ff11c1cab80 20 client.4930 trim_cache size 0 max 16384 2023-06-22T08:00:47.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.154+0000 7ff11c1cab80 1 Processor -- start 2023-06-22T08:00:47.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.155+0000 7ff11c1cab80 1 -- start start 2023-06-22T08:00:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.155+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f64f7c0 0x55e87f693900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.155+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f693e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.155+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f64cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.155+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87f697c70 con 0x55e87f692150 2023-06-22T08:00:47.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.155+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87f64e940 con 0x55e87f64f7c0 2023-06-22T08:00:47.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.155+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e87f68e070 con 0x55e87f692520 2023-06-22T08:00:47.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff105d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f693e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff105d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f693e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35444/0 (socket says 172.21.15.23:35444) 2023-06-22T08:00:47.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff105d9a700 1 -- 172.21.15.23:0/297698100 learned_addr learned my addr 172.21.15.23:0/297698100 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff10659b700 1 --2- 172.21.15.23:0/297698100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f64f7c0 0x55e87f693900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff106d9c700 1 --2- 172.21.15.23:0/297698100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f64cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff105d9a700 1 -- 172.21.15.23:0/297698100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 msgr2=0x55e87f64cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff105d9a700 1 --2- 172.21.15.23:0/297698100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f64cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff105d9a700 1 -- 172.21.15.23:0/297698100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f64f7c0 msgr2=0x55e87f693900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff105d9a700 1 --2- 172.21.15.23:0/297698100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f64f7c0 0x55e87f693900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff105d9a700 1 -- 172.21.15.23:0/297698100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e87f3d1d90 con 0x55e87f692150 2023-06-22T08:00:47.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff105d9a700 1 --2- 172.21.15.23:0/297698100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f693e40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0058ff0 tx=0x7ff0f008a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecbb42d2c156cd4f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff0f4ff9700 1 -- 172.21.15.23:0/297698100 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0086070 con 0x55e87f692150 2023-06-22T08:00:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff0f4ff9700 1 -- 172.21.15.23:0/297698100 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f005cf70 con 0x55e87f692150 2023-06-22T08:00:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.156+0000 7ff0f4ff9700 1 -- 172.21.15.23:0/297698100 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0061300 con 0x55e87f692150 2023-06-22T08:00:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.157+0000 7ff11c1cab80 1 -- 172.21.15.23:0/297698100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 msgr2=0x55e87f693e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.157+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/297698100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f693e40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0058ff0 tx=0x7ff0f008a040 comp rx=0 tx=0).stop 2023-06-22T08:00:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.157+0000 7ff11c1cab80 1 -- 172.21.15.23:0/297698100 shutdown_connections 2023-06-22T08:00:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.157+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/297698100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f64cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.157+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/297698100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f64f7c0 0x55e87f693900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.157+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/297698100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f692150 0x55e87f693e40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.157+0000 7ff11c1cab80 1 -- 172.21.15.23:0/297698100 >> 172.21.15.23:0/297698100 conn(0x55e87f647fc0 msgr2=0x55e87f6b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.157+0000 7ff11c1cab80 1 -- 172.21.15.23:0/297698100 shutdown_connections 2023-06-22T08:00:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.157+0000 7ff11c1cab80 1 -- 172.21.15.23:0/297698100 wait complete. 2023-06-22T08:00:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.158+0000 7ff11c1cab80 1 -- start start 2023-06-22T08:00:47.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.158+0000 7ff11c1cab80 1 Processor -- start 2023-06-22T08:00:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f64d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 0x55e87f6f9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff11c1cab80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f679850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87f6940e0 con 0x55e87f64f7c0 2023-06-22T08:00:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87f64dc80 con 0x55e87f692150 2023-06-22T08:00:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff11c1cab80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e87f6f9a20 con 0x55e87f692520 2023-06-22T08:00:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff11c1cab80 10 client.0 _ll_register_callbacks cb 0x55e87f549210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff10659b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f64d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff105d9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 0x55e87f6f9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff105d9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 0x55e87f6f9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51122/0 (socket says 172.21.15.23:51122) 2023-06-22T08:00:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff105d9a700 1 -- 172.21.15.23:0/2028255436 learned_addr learned my addr 172.21.15.23:0/2028255436 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff106d9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f679850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff10659b700 1 -- 172.21.15.23:0/2028255436 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 msgr2=0x55e87f679850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff10659b700 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f679850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.159+0000 7ff10659b700 1 -- 172.21.15.23:0/2028255436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 msgr2=0x55e87f6f9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff10659b700 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 0x55e87f6f9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff10659b700 1 -- 172.21.15.23:0/2028255436 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1001f1b80 con 0x55e87f64f7c0 2023-06-22T08:00:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff10659b700 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f64d740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc020000 tx=0x7ff0fc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42260f6918e479cd server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff0e17fa700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-06-22T08:00:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff0e17fa700 1 -- 172.21.15.23:0/2028255436 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc028040 con 0x55e87f64f7c0 2023-06-22T08:00:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff0e17fa700 1 -- 172.21.15.23:0/2028255436 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff0fc02f9d0 con 0x55e87f64f7c0 2023-06-22T08:00:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff11c1cab80 1 -- 172.21.15.23:0/2028255436 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0fc003c70 con 0x55e87f64f7c0 2023-06-22T08:00:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff0e17fa700 1 -- 172.21.15.23:0/2028255436 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0fc02fb90 con 0x55e87f64f7c0 2023-06-22T08:00:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff11c1cab80 10 client.4890 fetch_fsmap learned FSMap version 19 2023-06-22T08:00:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff11c1cab80 1 -- 172.21.15.23:0/2028255436 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55e87f3bac10 con 0x55e87f64f7c0 2023-06-22T08:00:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.160+0000 7ff0e17fa700 1 -- 172.21.15.23:0/2028255436 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc003c70 con 0x55e87f64f7c0 2023-06-22T08:00:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.161+0000 7ff0e17fa700 1 -- 172.21.15.23:0/2028255436 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 19) v1 ==== 4260+0+0 (secure 0 0 0) 0x7ff0fc005160 con 0x55e87f64f7c0 2023-06-22T08:00:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.161+0000 7ff11c1cab80 10 client.4890 fetch_fsmap finished waiting for FSMap version 19 2023-06-22T08:00:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.161+0000 7ff11c1cab80 10 client.4890 resolve_mds: resolved name 'a' to daemon mds.a 2023-06-22T08:00:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.161+0000 7ff11c1cab80 20 client.4890 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:47.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.161+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.23:6833/3894174037,v1:172.21.15.23:6835/3894174037] conn(0x55e87f6fd1d0 0x55e87f6fd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.161+0000 7ff11c1cab80 4 client.4890 mds_command: new command op to 4272 tid=0[{"prefix": "status"}] 2023-06-22T08:00:47.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.161+0000 7ff11c1cab80 1 -- 172.21.15.23:0/2028255436 --> [v2:172.21.15.23:6833/3894174037,v1:172.21.15.23:6835/3894174037] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e87f55f000 con 0x55e87f6fd1d0 2023-06-22T08:00:47.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.161+0000 7ff105d9a700 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.23:6833/3894174037,v1:172.21.15.23:6835/3894174037] conn(0x55e87f6fd1d0 0x55e87f6fd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.161+0000 7ff105d9a700 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.23:6833/3894174037,v1:172.21.15.23:6835/3894174037] conn(0x55e87f6fd1d0 0x55e87f6fd5a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=331354b9a7f221cf server_cookie=c3c7b6e0ca5317b4 in_seq=0 out_seq=0 2023-06-22T08:00:47.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.161+0000 7ff0e17fa700 10 client.4890 ms_handle_connect on v2:172.21.15.23:6833/3894174037 2023-06-22T08:00:47.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff0e17fa700 1 -- 172.21.15.23:0/2028255436 <== mds.2 v2:172.21.15.23:6833/3894174037 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55e87f55f000 con 0x55e87f6fd1d0 2023-06-22T08:00:47.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff0e17fa700 10 client.4890 handle_command_reply: tid=0 2023-06-22T08:00:47.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 client.4890 shutdown 2023-06-22T08:00:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 -- 172.21.15.23:0/2028255436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 msgr2=0x55e87f64d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f64d740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc020000 tx=0x7ff0fc000ed0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 -- 172.21.15.23:0/2028255436 shutdown_connections 2023-06-22T08:00:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.23:6833/3894174037,v1:172.21.15.23:6835/3894174037] conn(0x55e87f6fd1d0 0x55e87f6fd5a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e87f692520 0x55e87f679850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e87f64f7c0 0x55e87f64d740 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff0e17fa700 0 client.4890 ms_handle_reset on v2:172.21.15.23:6833/3894174037 2023-06-22T08:00:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 --2- 172.21.15.23:0/2028255436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e87f692150 0x55e87f6f9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 -- 172.21.15.23:0/2028255436 >> 172.21.15.23:0/2028255436 conn(0x55e87f647fc0 msgr2=0x55e87f68d740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 -- 172.21.15.23:0/2028255436 shutdown_connections 2023-06-22T08:00:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 1 -- 172.21.15.23:0/2028255436 wait complete. 2023-06-22T08:00:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.162+0000 7ff11c1cab80 20 client.4890 trim_cache size 0 max 16384 2023-06-22T08:00:47.173 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-06-22T08:00:47.174 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "2f053100-95a5-44db-a4cd-e7710a2fac9d", 2023-06-22T08:00:47.174 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2023-06-22T08:00:47.174 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 4272, 2023-06-22T08:00:47.174 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-06-22T08:00:47.174 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-06-22T08:00:47.174 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-06-22T08:00:47.174 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 20.776945285, 2023-06-22T08:00:47.174 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 19, 2023-06-22T08:00:47.175 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 28, 2023-06-22T08:00:47.175 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 28, 2023-06-22T08:00:47.175 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 57.529826360000001 2023-06-22T08:00:47.175 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-06-22T08:00:47.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 -- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff0e406ace0 msgr2=0x7ff0e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff0e406ace0 0x7ff0e406d190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0011e0 tx=0x7ff0fc009040 comp rx=0 tx=0).stop 2023-06-22T08:00:47.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 -- 172.21.15.23:0/2709404237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1000c9fd0 msgr2=0x7ff10014a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1000c9fd0 0x7ff10014a580 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009b90 tx=0x7ff0f0004750 comp rx=0 tx=0).stop 2023-06-22T08:00:47.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 -- 172.21.15.23:0/2709404237 shutdown_connections 2023-06-22T08:00:47.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff100152ce0 0x7ff100150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff0e406ace0 0x7ff0e406d190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1000c40b0 0x7ff1000ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 --2- 172.21.15.23:0/2709404237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1000c9fd0 0x7ff10014a580 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 -- 172.21.15.23:0/2709404237 >> 172.21.15.23:0/2709404237 conn(0x7ff10006ae30 msgr2=0x7ff1000cb750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 -- 172.21.15.23:0/2709404237 shutdown_connections 2023-06-22T08:00:47.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.163+0000 7ff1087ff700 1 -- 172.21.15.23:0/2709404237 wait complete. 2023-06-22T08:00:47.183 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-06-22T08:00:47.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.382+0000 7f170e5ff700 1 Processor -- start 2023-06-22T08:00:47.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.382+0000 7f170e5ff700 1 -- start start 2023-06-22T08:00:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f170e5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1708152ce0 0x7f17080cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f170e5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c40b0 0x7f17080c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f170e5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170815c370 con 0x7f17080c9fd0 2023-06-22T08:00:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f170e5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17081530b0 con 0x7f1708152ce0 2023-06-22T08:00:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f170e5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17080cf6f0 con 0x7f17080c40b0 2023-06-22T08:00:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f1707fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1708152ce0 0x7f17080cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f170cb9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c40b0 0x7f17080c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f170cb9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c40b0 0x7f17080c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:48854/0 (socket says 172.21.15.23:48854) 2023-06-22T08:00:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f1707fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1708152ce0 0x7f17080cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51138/0 (socket says 172.21.15.23:51138) 2023-06-22T08:00:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f17077fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f17080c9fd0 0x7f170815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:57736/0 (socket says 172.21.15.23:57736) 2023-06-22T08:00:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.383+0000 7f170cb9c700 1 -- 172.21.15.23:0/3501527484 learned_addr learned my addr 172.21.15.23:0/3501527484 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.384+0000 7f1706ffd700 1 -- 172.21.15.23:0/3501527484 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622603305 0 0) 0x7f170815c370 con 0x7f17080c9fd0 2023-06-22T08:00:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.384+0000 7f170cb9c700 1 -- 172.21.15.23:0/3501527484 >> v2:172.21.15.67:3300/0 conn(0x7f1708152ce0 msgr2=0x7f17080cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.384+0000 7f170cb9c700 1 --2- 172.21.15.23:0/3501527484 >> v2:172.21.15.67:3300/0 conn(0x7f1708152ce0 0x7f17080cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.384+0000 7f170cb9c700 1 -- 172.21.15.23:0/3501527484 >> v1:172.21.15.67:6789/0 conn(0x7f17080c9fd0 legacy=0x7f170815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.384+0000 7f170cb9c700 1 -- 172.21.15.23:0/3501527484 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170815bda0 con 0x7f17080c40b0 2023-06-22T08:00:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.384+0000 7f1707fff700 1 --2- 172.21.15.23:0/3501527484 >> v2:172.21.15.67:3300/0 conn(0x7f1708152ce0 0x7f17080cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.384+0000 7f170cb9c700 1 --2- 172.21.15.23:0/3501527484 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c40b0 0x7f17080c4530 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1700009e50 tx=0x7f170000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2676b26ddbc6067 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.384+0000 7f1706ffd700 1 -- 172.21.15.23:0/3501527484 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170000e070 con 0x7f17080c40b0 2023-06-22T08:00:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.384+0000 7f1706ffd700 1 -- 172.21.15.23:0/3501527484 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170000b910 con 0x7f17080c40b0 2023-06-22T08:00:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.384+0000 7f1706ffd700 1 -- 172.21.15.23:0/3501527484 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170000bcc0 con 0x7f17080c40b0 2023-06-22T08:00:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.385+0000 7f170e5ff700 1 -- 172.21.15.23:0/3501527484 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c40b0 msgr2=0x7f17080c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.385+0000 7f170e5ff700 1 --2- 172.21.15.23:0/3501527484 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c40b0 0x7f17080c4530 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1700009e50 tx=0x7f170000b330 comp rx=0 tx=0).stop 2023-06-22T08:00:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.385+0000 7f170e5ff700 1 -- 172.21.15.23:0/3501527484 shutdown_connections 2023-06-22T08:00:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.385+0000 7f170e5ff700 1 --2- 172.21.15.23:0/3501527484 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c40b0 0x7f17080c4530 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.385+0000 7f170e5ff700 1 --2- 172.21.15.23:0/3501527484 >> v2:172.21.15.67:3300/0 conn(0x7f1708152ce0 0x7f17080cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.385+0000 7f170e5ff700 1 -- 172.21.15.23:0/3501527484 >> 172.21.15.23:0/3501527484 conn(0x7f170806ae30 msgr2=0x7f170806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.385+0000 7f170e5ff700 1 -- 172.21.15.23:0/3501527484 shutdown_connections 2023-06-22T08:00:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.385+0000 7f170e5ff700 1 -- 172.21.15.23:0/3501527484 wait complete. 2023-06-22T08:00:47.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.387+0000 7f170e5ff700 1 Processor -- start 2023-06-22T08:00:47.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.387+0000 7f170e5ff700 1 -- start start 2023-06-22T08:00:47.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f170e5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17080c40b0 0x7f17080c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f170e5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c9fd0 0x7f1708145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f170e5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1708152ce0 0x7f17081456d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f170e5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17080c6650 con 0x7f17080c40b0 2023-06-22T08:00:47.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f170e5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17080c7e10 con 0x7f1708152ce0 2023-06-22T08:00:47.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f170e5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17080c80d0 con 0x7f17080c9fd0 2023-06-22T08:00:47.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f1707fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17080c40b0 0x7f17080c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f1707fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17080c40b0 0x7f17080c6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35456/0 (socket says 172.21.15.23:35456) 2023-06-22T08:00:47.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f1707fff700 1 -- 172.21.15.23:0/3126238167 learned_addr learned my addr 172.21.15.23:0/3126238167 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f170cb9c700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1708152ce0 0x7f17081456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f17077fe700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c9fd0 0x7f1708145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f1707fff700 1 -- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c9fd0 msgr2=0x7f1708145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f1707fff700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c9fd0 0x7f1708145190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f1707fff700 1 -- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1708152ce0 msgr2=0x7f17081456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f1707fff700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1708152ce0 0x7f17081456d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.388+0000 7f1707fff700 1 -- 172.21.15.23:0/3126238167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80094a0 con 0x7f17080c40b0 2023-06-22T08:00:47.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.389+0000 7f1707fff700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17080c40b0 0x7f17080c6a70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f16f800ebe0 tx=0x7f16f800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.389+0000 7f1704ff9700 1 -- 172.21.15.23:0/3126238167 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8009760 con 0x7f17080c40b0 2023-06-22T08:00:47.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.389+0000 7f1704ff9700 1 -- 172.21.15.23:0/3126238167 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f800c850 con 0x7f17080c40b0 2023-06-22T08:00:47.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.389+0000 7f170e5ff700 1 -- 172.21.15.23:0/3126238167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1700009b30 con 0x7f17080c40b0 2023-06-22T08:00:47.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.389+0000 7f1704ff9700 1 -- 172.21.15.23:0/3126238167 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f800cbe0 con 0x7f17080c40b0 2023-06-22T08:00:47.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.389+0000 7f170e5ff700 1 -- 172.21.15.23:0/3126238167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17080c5cb0 con 0x7f17080c40b0 2023-06-22T08:00:47.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.391+0000 7f1704ff9700 1 -- 172.21.15.23:0/3126238167 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f16f800cd40 con 0x7f17080c40b0 2023-06-22T08:00:47.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.393+0000 7f1704ff9700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f16f406ad10 0x7f16f406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.393+0000 7f1704ff9700 1 -- 172.21.15.23:0/3126238167 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f16f8089810 con 0x7f17080c40b0 2023-06-22T08:00:47.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.393+0000 7f17077fe700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f16f406ad10 0x7f16f406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.394+0000 7f17077fe700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f16f406ad10 0x7f16f406d1c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1708150af0 tx=0x7f16fc007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.395+0000 7f1721f73b80 1 Processor -- start 2023-06-22T08:00:47.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.395+0000 7f1721f73b80 1 -- start start 2023-06-22T08:00:47.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.396+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d150 0x5580f9357f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.396+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d520 0x5580f93584c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.396+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 0x5580f9385880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.396+0000 7f1721f73b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f9262c60 con 0x5580f939d150 2023-06-22T08:00:47.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.396+0000 7f1721f73b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f92b59f0 con 0x5580f935a7c0 2023-06-22T08:00:47.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.396+0000 7f1721f73b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580f92aee80 con 0x5580f939d520 2023-06-22T08:00:47.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f17077fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d520 0x5580f93584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f1707fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d150 0x5580f9357f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f1707fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d150 0x5580f9357f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35472/0 (socket says 172.21.15.23:35472) 2023-06-22T08:00:47.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f1707fff700 1 -- 172.21.15.23:0/1298598351 learned_addr learned my addr 172.21.15.23:0/1298598351 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f170cb9c700 1 --2- 172.21.15.23:0/1298598351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 0x5580f9385880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f1707fff700 1 -- 172.21.15.23:0/1298598351 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d520 msgr2=0x5580f93584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f1707fff700 1 --2- 172.21.15.23:0/1298598351 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d520 0x5580f93584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f1707fff700 1 -- 172.21.15.23:0/1298598351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 msgr2=0x5580f9385880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f1707fff700 1 --2- 172.21.15.23:0/1298598351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 0x5580f9385880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f1707fff700 1 -- 172.21.15.23:0/1298598351 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580f91305e0 con 0x5580f939d150 2023-06-22T08:00:47.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f1707fff700 1 --2- 172.21.15.23:0/1298598351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d150 0x5580f9357f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f16f8054660 tx=0x7f16f805c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafdd98af3b6d6c5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f16edffb700 1 -- 172.21.15.23:0/1298598351 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8056500 con 0x5580f939d150 2023-06-22T08:00:47.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f16edffb700 1 -- 172.21.15.23:0/1298598351 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f8057650 con 0x5580f939d150 2023-06-22T08:00:47.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.397+0000 7f16edffb700 1 -- 172.21.15.23:0/1298598351 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f80600b0 con 0x5580f939d150 2023-06-22T08:00:47.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.398+0000 7f1721f73b80 1 -- 172.21.15.23:0/1298598351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d150 msgr2=0x5580f9357f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.398+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1298598351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d150 0x5580f9357f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f16f8054660 tx=0x7f16f805c120 comp rx=0 tx=0).stop 2023-06-22T08:00:47.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.398+0000 7f1721f73b80 1 -- 172.21.15.23:0/1298598351 shutdown_connections 2023-06-22T08:00:47.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.398+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1298598351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 0x5580f9385880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.398+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1298598351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d150 0x5580f9357f80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.398+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1298598351 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d520 0x5580f93584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.398+0000 7f1721f73b80 1 -- 172.21.15.23:0/1298598351 >> 172.21.15.23:0/1298598351 conn(0x5580f9352fc0 msgr2=0x5580f935b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.398+0000 7f1721f73b80 1 -- 172.21.15.23:0/1298598351 shutdown_connections 2023-06-22T08:00:47.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.398+0000 7f1721f73b80 1 -- 172.21.15.23:0/1298598351 wait complete. 2023-06-22T08:00:47.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.399+0000 7f1721f73b80 1 -- start start 2023-06-22T08:00:47.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.399+0000 7f1721f73b80 1 Processor -- start 2023-06-22T08:00:47.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.400+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 0x5580f93a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.400+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d150 0x5580f939fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.400+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9385530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.400+0000 7f1721f73b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f924ea70 con 0x5580f939d520 2023-06-22T08:00:47.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.400+0000 7f1721f73b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f924e770 con 0x5580f935a7c0 2023-06-22T08:00:47.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.400+0000 7f1721f73b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580f91328d0 con 0x5580f939d150 2023-06-22T08:00:47.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.400+0000 7f1721f73b80 10 client.0 _ll_register_callbacks cb 0x5580f9254210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:47.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f170cb9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9385530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f1707fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 0x5580f93a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f17077fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d150 0x5580f939fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f170cb9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9385530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35482/0 (socket says 172.21.15.23:35482) 2023-06-22T08:00:47.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f170cb9c700 1 -- 172.21.15.23:0/1880469652 learned_addr learned my addr 172.21.15.23:0/1880469652 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f170cb9c700 1 -- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d150 msgr2=0x5580f939fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f170cb9c700 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d150 0x5580f939fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f170cb9c700 1 -- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 msgr2=0x5580f93a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f170cb9c700 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 0x5580f93a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f170cb9c700 1 -- 172.21.15.23:0/1880469652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17080c5cb0 con 0x5580f939d520 2023-06-22T08:00:47.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f1707fff700 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 0x5580f93a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:47.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f170cb9c700 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9385530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f170000b680 tx=0x7f1700004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db657f145b61aaf6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.401+0000 7f16e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-06-22T08:00:47.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f16e6ffd700 1 -- 172.21.15.23:0/1880469652 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700022070 con 0x5580f939d520 2023-06-22T08:00:47.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f16e6ffd700 1 -- 172.21.15.23:0/1880469652 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1700004250 con 0x5580f939d520 2023-06-22T08:00:47.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f1721f73b80 1 -- 172.21.15.23:0/1880469652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5580f9350780 con 0x5580f939d520 2023-06-22T08:00:47.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f16e6ffd700 1 -- 172.21.15.23:0/1880469652 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1700003e90 con 0x5580f939d520 2023-06-22T08:00:47.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f1721f73b80 10 client.4899 fetch_fsmap learned FSMap version 19 2023-06-22T08:00:47.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f1721f73b80 1 -- 172.21.15.23:0/1880469652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x5580f90dcd90 con 0x5580f939d520 2023-06-22T08:00:47.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f16e6ffd700 1 -- 172.21.15.23:0/1880469652 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5580f9350780 con 0x5580f939d520 2023-06-22T08:00:47.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f16e6ffd700 1 -- 172.21.15.23:0/1880469652 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 19) v1 ==== 4260+0+0 (secure 0 0 0) 0x7f1700019030 con 0x5580f939d520 2023-06-22T08:00:47.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f1721f73b80 10 client.4899 fetch_fsmap finished waiting for FSMap version 19 2023-06-22T08:00:47.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f1721f73b80 10 client.4899 resolve_mds: resolved name 'b' to daemon mds.b 2023-06-22T08:00:47.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f1721f73b80 20 client.4899 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:47.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:6834/1962314516,v1:172.21.15.67:6836/1962314516] conn(0x5580f93b1300 0x5580f93b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f1721f73b80 4 client.4899 mds_command: new command op to 4279 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-22T08:00:47.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.402+0000 7f1721f73b80 1 -- 172.21.15.23:0/1880469652 --> [v2:172.21.15.67:6834/1962314516,v1:172.21.15.67:6836/1962314516] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5580f92a7e60 con 0x5580f93b1300 2023-06-22T08:00:47.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.403+0000 7f1707fff700 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:6834/1962314516,v1:172.21.15.67:6836/1962314516] conn(0x5580f93b1300 0x5580f93b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.403+0000 7f1707fff700 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:6834/1962314516,v1:172.21.15.67:6836/1962314516] conn(0x5580f93b1300 0x5580f93b16d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=82a1dcbe96c4aff server_cookie=7c508760a48f8ccd in_seq=0 out_seq=0 2023-06-22T08:00:47.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.403+0000 7f16e6ffd700 10 client.4899 ms_handle_connect on v2:172.21.15.67:6834/1962314516 2023-06-22T08:00:47.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f16e6ffd700 1 -- 172.21.15.23:0/1880469652 <== mds.1 v2:172.21.15.67:6834/1962314516 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5580f92a7e60 con 0x5580f93b1300 2023-06-22T08:00:47.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f16e6ffd700 10 client.4899 handle_command_reply: tid=0 2023-06-22T08:00:47.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 client.4899 shutdown 2023-06-22T08:00:47.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 -- 172.21.15.23:0/1880469652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 msgr2=0x5580f9385530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9385530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f170000b680 tx=0x7f1700004b40 comp rx=0 tx=0).stop 2023-06-22T08:00:47.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 -- 172.21.15.23:0/1880469652 shutdown_connections 2023-06-22T08:00:47.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:6834/1962314516,v1:172.21.15.67:6836/1962314516] conn(0x5580f93b1300 0x5580f93b16d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9385530 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f935a7c0 0x5580f93a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f16e6ffd700 0 client.4899 ms_handle_reset on v2:172.21.15.67:6834/1962314516 2023-06-22T08:00:47.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1880469652 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d150 0x5580f939fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 -- 172.21.15.23:0/1880469652 >> 172.21.15.23:0/1880469652 conn(0x5580f9352fc0 msgr2=0x5580f935bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 -- 172.21.15.23:0/1880469652 shutdown_connections 2023-06-22T08:00:47.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.404+0000 7f1721f73b80 1 -- 172.21.15.23:0/1880469652 wait complete. 2023-06-22T08:00:47.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.405+0000 7f1721f73b80 20 client.4899 trim_cache size 0 max 16384 2023-06-22T08:00:47.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.423+0000 7f1721f73b80 1 Processor -- start 2023-06-22T08:00:47.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.423+0000 7f1721f73b80 1 -- start start 2023-06-22T08:00:47.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f935a7c0 0x5580f9386de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d150 0x5580f9387320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9398270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f1721f73b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f9382d60 con 0x5580f939d520 2023-06-22T08:00:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f1721f73b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f939eaa0 con 0x5580f939d150 2023-06-22T08:00:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f1721f73b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580f9384bd0 con 0x5580f935a7c0 2023-06-22T08:00:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f170cb9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9398270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f170cb9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9398270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35484/0 (socket says 172.21.15.23:35484) 2023-06-22T08:00:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f170cb9c700 1 -- 172.21.15.23:0/1726775665 learned_addr learned my addr 172.21.15.23:0/1726775665 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f17077fe700 1 --2- 172.21.15.23:0/1726775665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d150 0x5580f9387320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.424+0000 7f1707fff700 1 --2- 172.21.15.23:0/1726775665 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f935a7c0 0x5580f9386de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.425+0000 7f170cb9c700 1 -- 172.21.15.23:0/1726775665 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f935a7c0 msgr2=0x5580f9386de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.425+0000 7f170cb9c700 1 --2- 172.21.15.23:0/1726775665 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f935a7c0 0x5580f9386de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.425+0000 7f170cb9c700 1 -- 172.21.15.23:0/1726775665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d150 msgr2=0x5580f9387320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.425+0000 7f170cb9c700 1 --2- 172.21.15.23:0/1726775665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d150 0x5580f9387320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.425+0000 7f170cb9c700 1 -- 172.21.15.23:0/1726775665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580f90c5c10 con 0x5580f939d520 2023-06-22T08:00:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.425+0000 7f170cb9c700 1 --2- 172.21.15.23:0/1726775665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9398270 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1700002230 tx=0x7f1700022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ec46e85295d2e6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.425+0000 7f16ee7fc700 1 -- 172.21.15.23:0/1726775665 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700003e90 con 0x5580f939d520 2023-06-22T08:00:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.425+0000 7f16ee7fc700 1 -- 172.21.15.23:0/1726775665 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1700004250 con 0x5580f939d520 2023-06-22T08:00:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.425+0000 7f16ee7fc700 1 -- 172.21.15.23:0/1726775665 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700004880 con 0x5580f939d520 2023-06-22T08:00:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.426+0000 7f1721f73b80 1 -- 172.21.15.23:0/1726775665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 msgr2=0x5580f9398270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.426+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1726775665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9398270 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1700002230 tx=0x7f1700022000 comp rx=0 tx=0).stop 2023-06-22T08:00:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.426+0000 7f1721f73b80 1 -- 172.21.15.23:0/1726775665 shutdown_connections 2023-06-22T08:00:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.426+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1726775665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f939d520 0x5580f9398270 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.426+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1726775665 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f935a7c0 0x5580f9386de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.426+0000 7f1721f73b80 1 --2- 172.21.15.23:0/1726775665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d150 0x5580f9387320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.426+0000 7f1721f73b80 1 -- 172.21.15.23:0/1726775665 >> 172.21.15.23:0/1726775665 conn(0x5580f9352fc0 msgr2=0x5580f935b630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.426+0000 7f1721f73b80 1 -- 172.21.15.23:0/1726775665 shutdown_connections 2023-06-22T08:00:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.426+0000 7f1721f73b80 1 -- 172.21.15.23:0/1726775665 wait complete. 2023-06-22T08:00:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.426+0000 7f1721f73b80 1 -- start start 2023-06-22T08:00:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.427+0000 7f1721f73b80 1 Processor -- start 2023-06-22T08:00:47.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f935a7c0 0x5580f9387140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d150 0x5580f93b7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1721f73b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d520 0x5580f93febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1721f73b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f9387680 con 0x5580f935a7c0 2023-06-22T08:00:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1721f73b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f93b8290 con 0x5580f939d520 2023-06-22T08:00:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1721f73b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580f93a1120 con 0x5580f939d150 2023-06-22T08:00:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1721f73b80 10 client.0 _ll_register_callbacks cb 0x5580f9254210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1707fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f935a7c0 0x5580f9387140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1707fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f935a7c0 0x5580f9387140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35488/0 (socket says 172.21.15.23:35488) 2023-06-22T08:00:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1707fff700 1 -- 172.21.15.23:0/459282465 learned_addr learned my addr 172.21.15.23:0/459282465 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f170cb9c700 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d520 0x5580f93febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f17077fe700 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d150 0x5580f93b7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1707fff700 1 -- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d150 msgr2=0x5580f93b7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1707fff700 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d150 0x5580f93b7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1707fff700 1 -- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d520 msgr2=0x5580f93febf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1707fff700 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d520 0x5580f93febf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f1707fff700 1 -- 172.21.15.23:0/459282465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580f90dcd90 con 0x5580f935a7c0 2023-06-22T08:00:47.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.428+0000 7f170cb9c700 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d520 0x5580f93febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:47.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f1707fff700 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f935a7c0 0x5580f9387140 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f16f8054660 tx=0x7f16f8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e7adcc3ae836a2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f16e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-06-22T08:00:47.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f16e6ffd700 1 -- 172.21.15.23:0/459282465 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8087040 con 0x5580f935a7c0 2023-06-22T08:00:47.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f16e6ffd700 1 -- 172.21.15.23:0/459282465 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f16f8056500 con 0x5580f935a7c0 2023-06-22T08:00:47.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f1721f73b80 1 -- 172.21.15.23:0/459282465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5580f939f320 con 0x5580f935a7c0 2023-06-22T08:00:47.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f16e6ffd700 1 -- 172.21.15.23:0/459282465 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16f80577c0 con 0x5580f935a7c0 2023-06-22T08:00:47.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f1721f73b80 10 client.4905 fetch_fsmap learned FSMap version 19 2023-06-22T08:00:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f1721f73b80 1 -- 172.21.15.23:0/459282465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x5580f9103200 con 0x5580f935a7c0 2023-06-22T08:00:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f16e6ffd700 1 -- 172.21.15.23:0/459282465 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5580f939f320 con 0x5580f935a7c0 2023-06-22T08:00:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f16e6ffd700 1 -- 172.21.15.23:0/459282465 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 19) v1 ==== 4260+0+0 (secure 0 0 0) 0x7f16f806fbc0 con 0x5580f935a7c0 2023-06-22T08:00:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f1721f73b80 10 client.4905 fetch_fsmap finished waiting for FSMap version 19 2023-06-22T08:00:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f1721f73b80 10 client.4905 resolve_mds: resolved name 'b' to daemon mds.b 2023-06-22T08:00:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f1721f73b80 20 client.4905 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f1721f73b80 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:6834/1962314516,v1:172.21.15.67:6836/1962314516] conn(0x5580f93ff3c0 0x5580f935be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f1721f73b80 4 client.4905 mds_command: new command op to 4279 tid=0[{"prefix": "status"}] 2023-06-22T08:00:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.429+0000 7f1721f73b80 1 -- 172.21.15.23:0/459282465 --> [v2:172.21.15.67:6834/1962314516,v1:172.21.15.67:6836/1962314516] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f16f8057650 con 0x5580f93ff3c0 2023-06-22T08:00:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.430+0000 7f17077fe700 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:6834/1962314516,v1:172.21.15.67:6836/1962314516] conn(0x5580f93ff3c0 0x5580f935be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.430+0000 7f16e6ffd700 10 client.4905 ms_handle_connect on v2:172.21.15.67:6834/1962314516 2023-06-22T08:00:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.430+0000 7f17077fe700 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:6834/1962314516,v1:172.21.15.67:6836/1962314516] conn(0x5580f93ff3c0 0x5580f935be10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e451b6ea503c59f9 server_cookie=cbf371fb07b5c53f in_seq=0 out_seq=0 2023-06-22T08:00:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.430+0000 7f16e6ffd700 1 -- 172.21.15.23:0/459282465 <== mds.1 v2:172.21.15.67:6834/1962314516 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7f16f8057650 con 0x5580f93ff3c0 2023-06-22T08:00:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.430+0000 7f16e6ffd700 10 client.4905 handle_command_reply: tid=0 2023-06-22T08:00:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.430+0000 7f1721f73b80 1 client.4905 shutdown 2023-06-22T08:00:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 1 -- 172.21.15.23:0/459282465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f935a7c0 msgr2=0x5580f9387140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f935a7c0 0x5580f9387140 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f16f8054660 tx=0x7f16f8000bc0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 1 -- 172.21.15.23:0/459282465 shutdown_connections 2023-06-22T08:00:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580f939d520 0x5580f93febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:6834/1962314516,v1:172.21.15.67:6836/1962314516] conn(0x5580f93ff3c0 0x5580f935be10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580f935a7c0 0x5580f9387140 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 1 --2- 172.21.15.23:0/459282465 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x5580f939d150 0x5580f93b7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f16e6ffd700 0 client.4905 ms_handle_reset on v2:172.21.15.67:6834/1962314516 2023-06-22T08:00:47.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 1 -- 172.21.15.23:0/459282465 >> 172.21.15.23:0/459282465 conn(0x5580f9352fc0 msgr2=0x5580f93a0990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 1 -- 172.21.15.23:0/459282465 shutdown_connections 2023-06-22T08:00:47.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 1 -- 172.21.15.23:0/459282465 wait complete. 2023-06-22T08:00:47.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.431+0000 7f1721f73b80 20 client.4905 trim_cache size 0 max 16384 2023-06-22T08:00:47.443 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-06-22T08:00:47.443 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "2f053100-95a5-44db-a4cd-e7710a2fac9d", 2023-06-22T08:00:47.443 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2023-06-22T08:00:47.443 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 4279, 2023-06-22T08:00:47.443 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-06-22T08:00:47.443 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-06-22T08:00:47.443 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-06-22T08:00:47.444 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 26.746057795999999, 2023-06-22T08:00:47.444 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 19, 2023-06-22T08:00:47.444 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 28, 2023-06-22T08:00:47.444 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 28, 2023-06-22T08:00:47.444 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 57.861318326000003 2023-06-22T08:00:47.444 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-06-22T08:00:47.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 -- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f16f406ad10 msgr2=0x7f16f406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f16f406ad10 0x7f16f406d1c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1708150af0 tx=0x7f16fc007de0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 -- 172.21.15.23:0/3126238167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17080c40b0 msgr2=0x7f17080c6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17080c40b0 0x7f17080c6a70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f16f800ebe0 tx=0x7f16f800eef0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 -- 172.21.15.23:0/3126238167 shutdown_connections 2023-06-22T08:00:47.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f16f406ad10 0x7f16f406d1c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1708152ce0 0x7f17081456d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17080c40b0 0x7f17080c6a70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 --2- 172.21.15.23:0/3126238167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f17080c9fd0 0x7f1708145190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 -- 172.21.15.23:0/3126238167 >> 172.21.15.23:0/3126238167 conn(0x7f170806ae30 msgr2=0x7f170814adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 -- 172.21.15.23:0/3126238167 shutdown_connections 2023-06-22T08:00:47.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.432+0000 7f170e5ff700 1 -- 172.21.15.23:0/3126238167 wait complete. 2023-06-22T08:00:47.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-06-22T08:00:47.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.650+0000 7fc6ef7ff700 1 Processor -- start 2023-06-22T08:00:47.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.650+0000 7fc6ef7ff700 1 -- start start 2023-06-22T08:00:47.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ef7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc6e80c9fd0 0x7fc6e80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ef7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc6e80c40b0 0x7fc6e80c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ef7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e815c370 con 0x7fc6e8152ce0 2023-06-22T08:00:47.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ef7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e81530b0 con 0x7fc6e80c9fd0 2023-06-22T08:00:47.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ef7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e80cf6f0 con 0x7fc6e80c40b0 2023-06-22T08:00:47.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ed59b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc6e8152ce0 0x7fc6e815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34908/0 (socket says 172.21.15.23:34908) 2023-06-22T08:00:47.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ed59b700 1 -- 172.21.15.23:0/659947742 learned_addr learned my addr 172.21.15.23:0/659947742 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:47.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/659947742 >> v2:172.21.15.23:3300/0 conn(0x7fc6e80c9fd0 0x7fc6e80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6edd9c700 1 --2- 172.21.15.23:0/659947742 >> v2:172.21.15.67:3300/0 conn(0x7fc6e80c40b0 0x7fc6e80c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6dffff700 1 -- 172.21.15.23:0/659947742 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943400199 0 0) 0x7fc6e815c370 con 0x7fc6e8152ce0 2023-06-22T08:00:47.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6dffff700 1 -- 172.21.15.23:0/659947742 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d40035b0 con 0x7fc6e8152ce0 2023-06-22T08:00:47.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/659947742 >> v2:172.21.15.67:3300/0 conn(0x7fc6e80c40b0 msgr2=0x7fc6e80c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/659947742 >> v2:172.21.15.67:3300/0 conn(0x7fc6e80c40b0 0x7fc6e80c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/659947742 >> v1:172.21.15.23:6789/0 conn(0x7fc6e8152ce0 legacy=0x7fc6e815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.652+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/659947742 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e815bda0 con 0x7fc6e80c9fd0 2023-06-22T08:00:47.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.653+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/659947742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e80c9fd0 0x7fc6e80cf1b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80b4380 tx=0x7fc6d8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2723415e5c6eab server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.653+0000 7fc6dffff700 1 -- 172.21.15.23:0/659947742 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8011830 con 0x7fc6e80c9fd0 2023-06-22T08:00:47.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.653+0000 7fc6dffff700 1 -- 172.21.15.23:0/659947742 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d8011990 con 0x7fc6e80c9fd0 2023-06-22T08:00:47.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.653+0000 7fc6dffff700 1 -- 172.21.15.23:0/659947742 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8011d20 con 0x7fc6e80c9fd0 2023-06-22T08:00:47.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.653+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/659947742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e80c9fd0 msgr2=0x7fc6e80cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.653+0000 7fc6ef7ff700 1 --2- 172.21.15.23:0/659947742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e80c9fd0 0x7fc6e80cf1b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80b4380 tx=0x7fc6d8009b20 comp rx=0 tx=0).stop 2023-06-22T08:00:47.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.654+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/659947742 shutdown_connections 2023-06-22T08:00:47.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.654+0000 7fc6ef7ff700 1 --2- 172.21.15.23:0/659947742 >> v2:172.21.15.67:3300/0 conn(0x7fc6e80c40b0 0x7fc6e80c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.654+0000 7fc6ef7ff700 1 --2- 172.21.15.23:0/659947742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e80c9fd0 0x7fc6e80cf1b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.654+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/659947742 >> 172.21.15.23:0/659947742 conn(0x7fc6e806ae30 msgr2=0x7fc6e806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.654+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/659947742 shutdown_connections 2023-06-22T08:00:47.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.654+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/659947742 wait complete. 2023-06-22T08:00:47.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.656+0000 7fc6ef7ff700 1 Processor -- start 2023-06-22T08:00:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.656+0000 7fc6ef7ff700 1 -- start start 2023-06-22T08:00:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.657+0000 7fc6ef7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6e80c40b0 0x7fc6e815b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.657+0000 7fc6ef7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e80c9fd0 0x7fc6e81ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6ef7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e8152ce0 0x7fc6e8149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6ef7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e815b900 con 0x7fc6e8152ce0 2023-06-22T08:00:47.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6ef7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e81eece0 con 0x7fc6e80c9fd0 2023-06-22T08:00:47.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6ef7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e814ae60 con 0x7fc6e80c40b0 2023-06-22T08:00:47.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6ed59b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6e80c40b0 0x7fc6e815b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6ecd9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e80c9fd0 0x7fc6e81ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6edd9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e8152ce0 0x7fc6e8149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6ed59b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6e80c40b0 0x7fc6e815b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:48904/0 (socket says 172.21.15.23:48904) 2023-06-22T08:00:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6ecd9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e80c9fd0 0x7fc6e81ee7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51196/0 (socket says 172.21.15.23:51196) 2023-06-22T08:00:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6ed59b700 1 -- 172.21.15.23:0/1914943019 learned_addr learned my addr 172.21.15.23:0/1914943019 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6edd9c700 1 -- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6e80c40b0 msgr2=0x7fc6e815b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6edd9c700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6e80c40b0 0x7fc6e815b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6edd9c700 1 -- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e80c9fd0 msgr2=0x7fc6e81ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6edd9c700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e80c9fd0 0x7fc6e81ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.658+0000 7fc6edd9c700 1 -- 172.21.15.23:0/1914943019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d80094a0 con 0x7fc6e8152ce0 2023-06-22T08:00:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.659+0000 7fc6edd9c700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e8152ce0 0x7fc6e8149e20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009d90 tx=0x7fc6e000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.659+0000 7fc6ddffb700 1 -- 172.21.15.23:0/1914943019 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0009c00 con 0x7fc6e8152ce0 2023-06-22T08:00:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.659+0000 7fc6ddffb700 1 -- 172.21.15.23:0/1914943019 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e000f040 con 0x7fc6e8152ce0 2023-06-22T08:00:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.659+0000 7fc6ddffb700 1 -- 172.21.15.23:0/1914943019 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e000ce90 con 0x7fc6e8152ce0 2023-06-22T08:00:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.659+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/1914943019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e815af30 con 0x7fc6e8152ce0 2023-06-22T08:00:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.659+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/1914943019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e814b390 con 0x7fc6e8152ce0 2023-06-22T08:00:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.661+0000 7fc6ddffb700 1 -- 172.21.15.23:0/1914943019 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc6e0014480 con 0x7fc6e8152ce0 2023-06-22T08:00:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.662+0000 7fc6ddffb700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6cc06acb0 0x7fc6cc06d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.662+0000 7fc6ddffb700 1 -- 172.21.15.23:0/1914943019 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc6e008a340 con 0x7fc6e8152ce0 2023-06-22T08:00:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.662+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6cc06acb0 0x7fc6cc06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.663+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6cc06acb0 0x7fc6cc06d160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc6e40011e0 tx=0x7fc6e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.664+0000 7fc703031b80 1 Processor -- start 2023-06-22T08:00:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.664+0000 7fc703031b80 1 -- start start 2023-06-22T08:00:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.665+0000 7fc703031b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf97c1150 0x55dcf977bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.665+0000 7fc703031b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.665+0000 7fc703031b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 0x55dcf97a9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.665+0000 7fc703031b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf9686c60 con 0x55dcf97c1520 2023-06-22T08:00:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.665+0000 7fc703031b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf96d99f0 con 0x55dcf97c1150 2023-06-22T08:00:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.665+0000 7fc703031b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dcf96d2e80 con 0x55dcf977e7c0 2023-06-22T08:00:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6ecd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6ecd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35514/0 (socket says 172.21.15.23:35514) 2023-06-22T08:00:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/1727660827 learned_addr learned my addr 172.21.15.23:0/1727660827 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/1727660827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf97c1150 0x55dcf977bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6edd9c700 1 --2- 172.21.15.23:0/1727660827 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 0x55dcf97a9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/1727660827 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 msgr2=0x55dcf97a9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/1727660827 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 0x55dcf97a9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/1727660827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf97c1150 msgr2=0x55dcf977bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/1727660827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf97c1150 0x55dcf977bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/1727660827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcf95545e0 con 0x55dcf97c1520 2023-06-22T08:00:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.666+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/1727660827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977c4c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009e70 tx=0x7fc6d8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511e92c817f41f9e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc6caffd700 1 -- 172.21.15.23:0/1727660827 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8020070 con 0x55dcf97c1520 2023-06-22T08:00:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc6caffd700 1 -- 172.21.15.23:0/1727660827 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d80116b0 con 0x55dcf97c1520 2023-06-22T08:00:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc6caffd700 1 -- 172.21.15.23:0/1727660827 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8011820 con 0x55dcf97c1520 2023-06-22T08:00:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc703031b80 1 -- 172.21.15.23:0/1727660827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 msgr2=0x55dcf977c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc703031b80 1 --2- 172.21.15.23:0/1727660827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977c4c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009e70 tx=0x7fc6d8010000 comp rx=0 tx=0).stop 2023-06-22T08:00:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc703031b80 1 -- 172.21.15.23:0/1727660827 shutdown_connections 2023-06-22T08:00:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc703031b80 1 --2- 172.21.15.23:0/1727660827 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 0x55dcf97a9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc703031b80 1 --2- 172.21.15.23:0/1727660827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf97c1150 0x55dcf977bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc703031b80 1 --2- 172.21.15.23:0/1727660827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977c4c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc703031b80 1 -- 172.21.15.23:0/1727660827 >> 172.21.15.23:0/1727660827 conn(0x55dcf9776fc0 msgr2=0x55dcf977f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc703031b80 1 -- 172.21.15.23:0/1727660827 shutdown_connections 2023-06-22T08:00:47.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.667+0000 7fc703031b80 1 -- 172.21.15.23:0/1727660827 wait complete. 2023-06-22T08:00:47.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.668+0000 7fc703031b80 1 -- start start 2023-06-22T08:00:47.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.668+0000 7fc703031b80 1 Processor -- start 2023-06-22T08:00:47.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.669+0000 7fc703031b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 0x55dcf97d2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.669+0000 7fc703031b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97c4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.669+0000 7fc703031b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf97c1520 0x55dcf97c1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.669+0000 7fc703031b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf9672a70 con 0x55dcf97c1150 2023-06-22T08:00:47.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.669+0000 7fc703031b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf9672770 con 0x55dcf97c1520 2023-06-22T08:00:47.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.669+0000 7fc703031b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dcf95568d0 con 0x55dcf977e7c0 2023-06-22T08:00:47.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.669+0000 7fc703031b80 10 client.0 _ll_register_callbacks cb 0x55dcf9678210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ecd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ed59b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 0x55dcf97d2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ed59b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 0x55dcf97d2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:48924/0 (socket says 172.21.15.23:48924) 2023-06-22T08:00:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ecd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97c4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35520/0 (socket says 172.21.15.23:35520) 2023-06-22T08:00:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ed59b700 1 -- 172.21.15.23:0/3258621801 learned_addr learned my addr 172.21.15.23:0/3258621801 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6edd9c700 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf97c1520 0x55dcf97c1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ed59b700 1 -- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf97c1520 msgr2=0x55dcf97c1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf97c1520 0x55dcf97c1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ed59b700 1 -- 172.21.15.23:0/3258621801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 msgr2=0x55dcf97c4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97c4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ed59b700 1 -- 172.21.15.23:0/3258621801 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e815bda0 con 0x55dcf977e7c0 2023-06-22T08:00:47.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97c4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:47.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 0x55dcf97d2810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc6e40144f0 tx=0x7fc6e4001150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40ea5287abf87ada server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.670+0000 7fc6c3fff700 10 client.0 ms_handle_connect on v2:172.21.15.67:3301/0 2023-06-22T08:00:47.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc6c3fff700 1 -- 172.21.15.23:0/3258621801 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4014bc0 con 0x55dcf977e7c0 2023-06-22T08:00:47.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc703031b80 1 -- 172.21.15.23:0/3258621801 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dcf9774780 con 0x55dcf977e7c0 2023-06-22T08:00:47.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc6c3fff700 1 -- 172.21.15.23:0/3258621801 <== mon.2 v2:172.21.15.67:3301/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc6e401c040 con 0x55dcf977e7c0 2023-06-22T08:00:47.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc6c3fff700 1 -- 172.21.15.23:0/3258621801 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6e402aea0 con 0x55dcf977e7c0 2023-06-22T08:00:47.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc703031b80 10 client.4745 fetch_fsmap learned FSMap version 19 2023-06-22T08:00:47.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc703031b80 1 -- 172.21.15.23:0/3258621801 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55dcf9500d90 con 0x55dcf977e7c0 2023-06-22T08:00:47.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc6c3fff700 1 -- 172.21.15.23:0/3258621801 <== mon.2 v2:172.21.15.67:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55dcf9774780 con 0x55dcf977e7c0 2023-06-22T08:00:47.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc6c3fff700 1 -- 172.21.15.23:0/3258621801 <== mon.2 v2:172.21.15.67:3301/0 5 ==== fsmap(e 19) v1 ==== 4260+0+0 (secure 0 0 0) 0x7fc6e4014d20 con 0x55dcf977e7c0 2023-06-22T08:00:47.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc703031b80 10 client.4745 fetch_fsmap finished waiting for FSMap version 19 2023-06-22T08:00:47.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc703031b80 10 client.4745 resolve_mds: resolved name 'f' to daemon mds.f 2023-06-22T08:00:47.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc703031b80 20 client.4745 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:47.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc703031b80 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55dcf97d5380 0x55dcf97d5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc703031b80 4 client.4745 mds_command: new command op to 4285 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-22T08:00:47.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.671+0000 7fc703031b80 1 -- 172.21.15.23:0/3258621801 --> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dcf96cbe60 con 0x55dcf97d5380 2023-06-22T08:00:47.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.672+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55dcf97d5380 0x55dcf97d5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.672+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55dcf97d5380 0x55dcf97d5750 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=314270123c8e92ad server_cookie=4e51262729fc5334 in_seq=0 out_seq=0 2023-06-22T08:00:47.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.672+0000 7fc6c3fff700 10 client.4745 ms_handle_connect on v2:172.21.15.67:6838/567663545 2023-06-22T08:00:47.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc6c3fff700 1 -- 172.21.15.23:0/3258621801 <== mds.0 v2:172.21.15.67:6838/567663545 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55dcf96cbe60 con 0x55dcf97d5380 2023-06-22T08:00:47.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc6c3fff700 10 client.4745 handle_command_reply: tid=0 2023-06-22T08:00:47.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 client.4745 shutdown 2023-06-22T08:00:47.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 -- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 msgr2=0x55dcf97d2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 0x55dcf97d2810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc6e40144f0 tx=0x7fc6e4001150 comp rx=0 tx=0).stop 2023-06-22T08:00:47.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 -- 172.21.15.23:0/3258621801 shutdown_connections 2023-06-22T08:00:47.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55dcf97d5380 0x55dcf97d5750 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf97c1520 0x55dcf97c1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf977e7c0 0x55dcf97d2810 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 --2- 172.21.15.23:0/3258621801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97c4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc6c3fff700 0 client.4745 ms_handle_reset on v2:172.21.15.67:6838/567663545 2023-06-22T08:00:47.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 -- 172.21.15.23:0/3258621801 >> 172.21.15.23:0/3258621801 conn(0x55dcf9776fc0 msgr2=0x55dcf9781170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 -- 172.21.15.23:0/3258621801 shutdown_connections 2023-06-22T08:00:47.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.673+0000 7fc703031b80 1 -- 172.21.15.23:0/3258621801 wait complete. 2023-06-22T08:00:47.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.674+0000 7fc703031b80 20 client.4745 trim_cache size 0 max 16384 2023-06-22T08:00:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.692+0000 7fc703031b80 1 Processor -- start 2023-06-22T08:00:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.692+0000 7fc703031b80 1 -- start start 2023-06-22T08:00:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc703031b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 0x55dcf97bc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc703031b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc703031b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf97c1520 0x55dcf97d2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc703031b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf977e650 con 0x55dcf97c1150 2023-06-22T08:00:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc703031b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf977d950 con 0x55dcf977e7c0 2023-06-22T08:00:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc703031b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dcf97bf080 con 0x55dcf97c1520 2023-06-22T08:00:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc6ecd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc6ecd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97d1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35536/0 (socket says 172.21.15.23:35536) 2023-06-22T08:00:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/3647133516 learned_addr learned my addr 172.21.15.23:0/3647133516 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/3647133516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 0x55dcf97bc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc6edd9c700 1 --2- 172.21.15.23:0/3647133516 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf97c1520 0x55dcf97d2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/3647133516 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf97c1520 msgr2=0x55dcf97d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.693+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/3647133516 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf97c1520 0x55dcf97d2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.694+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/3647133516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 msgr2=0x55dcf97bc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.694+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/3647133516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 0x55dcf97bc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.694+0000 7fc6ecd9a700 1 -- 172.21.15.23:0/3647133516 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcf94e9c10 con 0x55dcf97c1150 2023-06-22T08:00:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.694+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/3647133516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 0x55dcf97bc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.694+0000 7fc6ecd9a700 1 --2- 172.21.15.23:0/3647133516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97d1d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8011960 tx=0x7fc6d8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65be2ad838e69b0b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.694+0000 7fc6cb7fe700 1 -- 172.21.15.23:0/3647133516 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8020040 con 0x55dcf97c1150 2023-06-22T08:00:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.694+0000 7fc6cb7fe700 1 -- 172.21.15.23:0/3647133516 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d8004240 con 0x55dcf97c1150 2023-06-22T08:00:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.694+0000 7fc6cb7fe700 1 -- 172.21.15.23:0/3647133516 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8011b00 con 0x55dcf97c1150 2023-06-22T08:00:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.695+0000 7fc703031b80 1 -- 172.21.15.23:0/3647133516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 msgr2=0x55dcf97d1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.695+0000 7fc703031b80 1 --2- 172.21.15.23:0/3647133516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97d1d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8011960 tx=0x7fc6d8011000 comp rx=0 tx=0).stop 2023-06-22T08:00:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.695+0000 7fc703031b80 1 -- 172.21.15.23:0/3647133516 shutdown_connections 2023-06-22T08:00:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.695+0000 7fc703031b80 1 --2- 172.21.15.23:0/3647133516 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf97c1520 0x55dcf97d2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.695+0000 7fc703031b80 1 --2- 172.21.15.23:0/3647133516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 0x55dcf97bc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.695+0000 7fc703031b80 1 --2- 172.21.15.23:0/3647133516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1150 0x55dcf97d1d10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.695+0000 7fc703031b80 1 -- 172.21.15.23:0/3647133516 >> 172.21.15.23:0/3647133516 conn(0x55dcf9776fc0 msgr2=0x55dcf9778490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.695+0000 7fc703031b80 1 -- 172.21.15.23:0/3647133516 shutdown_connections 2023-06-22T08:00:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.695+0000 7fc703031b80 1 -- 172.21.15.23:0/3647133516 wait complete. 2023-06-22T08:00:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.695+0000 7fc703031b80 1 -- start start 2023-06-22T08:00:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.696+0000 7fc703031b80 1 Processor -- start 2023-06-22T08:00:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc703031b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 0x55dcf9781260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc703031b80 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf97c1150 0x55dcf97cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc703031b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc703031b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf97817a0 con 0x55dcf97c1520 2023-06-22T08:00:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc703031b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcf97a9b80 con 0x55dcf977e7c0 2023-06-22T08:00:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc703031b80 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dcf97dbad0 con 0x55dcf97c1150 2023-06-22T08:00:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc703031b80 10 client.0 _ll_register_callbacks cb 0x55dcf9678210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-22T08:00:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc6edd9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc6ecd9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf97c1150 0x55dcf97cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc6ed59b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 0x55dcf9781260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc6edd9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35544/0 (socket says 172.21.15.23:35544) 2023-06-22T08:00:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc6edd9c700 1 -- 172.21.15.23:0/4149393373 learned_addr learned my addr 172.21.15.23:0/4149393373 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc6edd9c700 1 -- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf97c1150 msgr2=0x55dcf97cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc6edd9c700 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf97c1150 0x55dcf97cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc6edd9c700 1 -- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 msgr2=0x55dcf9781260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc6edd9c700 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 0x55dcf9781260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.697+0000 7fc6edd9c700 1 -- 172.21.15.23:0/4149393373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dcf95545e0 con 0x55dcf97c1520 2023-06-22T08:00:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 0x55dcf9781260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc6edd9c700 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977ba80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009d60 tx=0x7fc6e0058c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd2dc06398b6016 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc6c3fff700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2023-06-22T08:00:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc6c3fff700 1 -- 172.21.15.23:0/4149393373 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0060de0 con 0x55dcf97c1520 2023-06-22T08:00:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc703031b80 1 -- 172.21.15.23:0/4149393373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dcf97a4d20 con 0x55dcf97c1520 2023-06-22T08:00:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc6c3fff700 1 -- 172.21.15.23:0/4149393373 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc6e0024bd0 con 0x55dcf97c1520 2023-06-22T08:00:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc6c3fff700 1 -- 172.21.15.23:0/4149393373 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6e0024d90 con 0x55dcf97c1520 2023-06-22T08:00:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc703031b80 10 client.4926 fetch_fsmap learned FSMap version 19 2023-06-22T08:00:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc703031b80 1 -- 172.21.15.23:0/4149393373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x55dcf9527200 con 0x55dcf97c1520 2023-06-22T08:00:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc6c3fff700 1 -- 172.21.15.23:0/4149393373 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55dcf97a4d20 con 0x55dcf97c1520 2023-06-22T08:00:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc6c3fff700 1 -- 172.21.15.23:0/4149393373 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 19) v1 ==== 4260+0+0 (secure 0 0 0) 0x7fc6e0054b10 con 0x55dcf97c1520 2023-06-22T08:00:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.698+0000 7fc703031b80 10 client.4926 fetch_fsmap finished waiting for FSMap version 19 2023-06-22T08:00:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.699+0000 7fc703031b80 10 client.4926 resolve_mds: resolved name 'f' to daemon mds.f 2023-06-22T08:00:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.699+0000 7fc703031b80 20 client.4926 populate_metadata read hostname 'smithi023' 2023-06-22T08:00:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.699+0000 7fc703031b80 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55dcf97aa3f0 0x55dcf97aa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.699+0000 7fc703031b80 4 client.4926 mds_command: new command op to 4285 tid=0[{"prefix": "status"}] 2023-06-22T08:00:47.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.699+0000 7fc703031b80 1 -- 172.21.15.23:0/4149393373 --> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fc6d80116b0 con 0x55dcf97aa3f0 2023-06-22T08:00:47.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.699+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55dcf97aa3f0 0x55dcf97aa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:47.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.699+0000 7fc6ed59b700 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55dcf97aa3f0 0x55dcf97aa7c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8c94e640d51335a server_cookie=8ae92321f460e7dc in_seq=0 out_seq=0 2023-06-22T08:00:47.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.699+0000 7fc6c3fff700 10 client.4926 ms_handle_connect on v2:172.21.15.67:6838/567663545 2023-06-22T08:00:47.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc6c3fff700 1 -- 172.21.15.23:0/4149393373 <== mds.0 v2:172.21.15.67:6838/567663545 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7fc6d80116b0 con 0x55dcf97aa3f0 2023-06-22T08:00:47.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc6c3fff700 10 client.4926 handle_command_reply: tid=0 2023-06-22T08:00:47.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 client.4926 shutdown 2023-06-22T08:00:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 -- 172.21.15.23:0/4149393373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 msgr2=0x55dcf977ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977ba80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009d60 tx=0x7fc6e0058c20 comp rx=0 tx=0).stop 2023-06-22T08:00:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 -- 172.21.15.23:0/4149393373 shutdown_connections 2023-06-22T08:00:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:6838/567663545,v1:172.21.15.67:6839/567663545] conn(0x55dcf97aa3f0 0x55dcf97aa7c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dcf97c1520 0x55dcf977ba80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dcf977e7c0 0x55dcf9781260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc6c3fff700 0 client.4926 ms_handle_reset on v2:172.21.15.67:6838/567663545 2023-06-22T08:00:47.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 --2- 172.21.15.23:0/4149393373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55dcf97c1150 0x55dcf97cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 -- 172.21.15.23:0/4149393373 >> 172.21.15.23:0/4149393373 conn(0x55dcf9776fc0 msgr2=0x55dcf97a4430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 -- 172.21.15.23:0/4149393373 shutdown_connections 2023-06-22T08:00:47.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 1 -- 172.21.15.23:0/4149393373 wait complete. 2023-06-22T08:00:47.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.700+0000 7fc703031b80 20 client.4926 trim_cache size 0 max 16384 2023-06-22T08:00:47.712 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2023-06-22T08:00:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "2f053100-95a5-44db-a4cd-e7710a2fac9d", 2023-06-22T08:00:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2023-06-22T08:00:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 4285, 2023-06-22T08:00:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2023-06-22T08:00:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2023-06-22T08:00:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2023-06-22T08:00:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 52.558546651999997, 2023-06-22T08:00:47.714 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 19, 2023-06-22T08:00:47.714 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 28, 2023-06-22T08:00:47.714 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 28, 2023-06-22T08:00:47.714 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 58.128031403999998 2023-06-22T08:00:47.714 INFO:teuthology.orchestra.run.smithi023.stdout:} 2023-06-22T08:00:47.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6cc06acb0 msgr2=0x7fc6cc06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6cc06acb0 0x7fc6cc06d160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc6e40011e0 tx=0x7fc6e4009040 comp rx=0 tx=0).stop 2023-06-22T08:00:47.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/1914943019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e8152ce0 msgr2=0x7fc6e8149e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e8152ce0 0x7fc6e8149e20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009d90 tx=0x7fc6e000c3a0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/1914943019 shutdown_connections 2023-06-22T08:00:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e8152ce0 0x7fc6e8149e20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6e80c40b0 0x7fc6e815b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6cc06acb0 0x7fc6cc06d160 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 --2- 172.21.15.23:0/1914943019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e80c9fd0 0x7fc6e81ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/1914943019 >> 172.21.15.23:0/1914943019 conn(0x7fc6e806ae30 msgr2=0x7fc6e8144210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/1914943019 shutdown_connections 2023-06-22T08:00:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:47.701+0000 7fc6ef7ff700 1 -- 172.21.15.23:0/1914943019 wait complete. 2023-06-22T08:00:47.721 INFO:teuthology.run_tasks:Running task kclient... 2023-06-22T08:00:47.735 INFO:tasks.kclient:Mounting kernel clients... 2023-06-22T08:00:47.736 INFO:tasks.kclient:config is {'client.0': {}} 2023-06-22T08:00:47.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:47.737 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2023-06-22T08:00:47.779 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-06-22T08:00:47.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:47.780 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-06-22T08:00:47.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:47.894 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2023-06-22T08:00:47.999 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-06-22T08:00:48.000 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-06-22T08:00:48.000 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-22T08:00:48.001 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-22T08:00:48.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.200+0000 7fe33afff700 1 Processor -- start 2023-06-22T08:00:48.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.200+0000 7fe33afff700 1 -- start start 2023-06-22T08:00:48.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.202+0000 7fe33afff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe33414f850 0x7fe3340c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:48.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.202+0000 7fe33afff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3340c51a0 con 0x7fe3340c69e0 2023-06-22T08:00:48.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.202+0000 7fe33afff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3340c44e0 con 0x7fe334071070 2023-06-22T08:00:48.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.202+0000 7fe33afff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3340c5440 con 0x7fe33414f850 2023-06-22T08:00:48.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.202+0000 7fe333fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe3340c69e0 0x7fe334155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34912/0 (socket says 172.21.15.23:34912) 2023-06-22T08:00:48.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.202+0000 7fe338d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe33414f850 0x7fe3340c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:48.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.202+0000 7fe333fff700 1 -- 172.21.15.23:0/3847199511 learned_addr learned my addr 172.21.15.23:0/3847199511 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:48.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.202+0000 7fe33959c700 1 --1- 172.21.15.23:0/3847199511 >> v1:172.21.15.67:6789/0 conn(0x7fe334071070 0x7fe33415fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:57744/0 (socket says 172.21.15.23:57744) 2023-06-22T08:00:48.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.203+0000 7fe3337fe700 1 -- 172.21.15.23:0/3847199511 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375872704 0 0) 0x7fe3340c44e0 con 0x7fe334071070 2023-06-22T08:00:48.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.203+0000 7fe3337fe700 1 -- 172.21.15.23:0/3847199511 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3240057c0 con 0x7fe334071070 2023-06-22T08:00:48.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.203+0000 7fe3337fe700 1 -- 172.21.15.23:0/3847199511 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757488934 0 0) 0x7fe3340c51a0 con 0x7fe3340c69e0 2023-06-22T08:00:48.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.203+0000 7fe338d9b700 1 -- 172.21.15.23:0/3847199511 >> v1:172.21.15.67:6789/0 conn(0x7fe334071070 legacy=0x7fe33415fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:48.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.203+0000 7fe338d9b700 1 -- 172.21.15.23:0/3847199511 >> v1:172.21.15.23:6789/0 conn(0x7fe3340c69e0 legacy=0x7fe334155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:48.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.203+0000 7fe338d9b700 1 -- 172.21.15.23:0/3847199511 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3340c5720 con 0x7fe33414f850 2023-06-22T08:00:48.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.203+0000 7fe338d9b700 1 --2- 172.21.15.23:0/3847199511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33414f850 0x7fe3340c3080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe328009840 tx=0x7fe328009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=786be01d82dfa4cb server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:48.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.204+0000 7fe3337fe700 1 -- 172.21.15.23:0/3847199511 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328011830 con 0x7fe33414f850 2023-06-22T08:00:48.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.204+0000 7fe3337fe700 1 -- 172.21.15.23:0/3847199511 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe328011990 con 0x7fe33414f850 2023-06-22T08:00:48.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.204+0000 7fe3337fe700 1 -- 172.21.15.23:0/3847199511 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328011d20 con 0x7fe33414f850 2023-06-22T08:00:48.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.204+0000 7fe33afff700 1 -- 172.21.15.23:0/3847199511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33414f850 msgr2=0x7fe3340c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:48.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.204+0000 7fe33afff700 1 --2- 172.21.15.23:0/3847199511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33414f850 0x7fe3340c3080 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe328009840 tx=0x7fe328009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:48.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.204+0000 7fe33afff700 1 -- 172.21.15.23:0/3847199511 shutdown_connections 2023-06-22T08:00:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.204+0000 7fe33afff700 1 --2- 172.21.15.23:0/3847199511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33414f850 0x7fe3340c3080 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.204+0000 7fe33afff700 1 -- 172.21.15.23:0/3847199511 >> 172.21.15.23:0/3847199511 conn(0x7fe3340ca3f0 msgr2=0x7fe3340ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.204+0000 7fe33afff700 1 -- 172.21.15.23:0/3847199511 shutdown_connections 2023-06-22T08:00:48.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.204+0000 7fe33afff700 1 -- 172.21.15.23:0/3847199511 wait complete. 2023-06-22T08:00:48.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.206+0000 7fe33afff700 1 Processor -- start 2023-06-22T08:00:48.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.206+0000 7fe33afff700 1 -- start start 2023-06-22T08:00:48.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe33afff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe334071070 0x7fe33406fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:48.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe33afff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3340c69e0 0x7fe334153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:48.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe33afff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33414f850 0x7fe334146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:48.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe33afff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3341e0af0 con 0x7fe3340c69e0 2023-06-22T08:00:48.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe33afff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3341561d0 con 0x7fe33414f850 2023-06-22T08:00:48.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe33afff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe334146fb0 con 0x7fe334071070 2023-06-22T08:00:48.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe333fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3340c69e0 0x7fe334153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:48.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe338d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe334071070 0x7fe33406fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:48.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe333fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3340c69e0 0x7fe334153360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35552/0 (socket says 172.21.15.23:35552) 2023-06-22T08:00:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe338d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe334071070 0x7fe33406fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:48944/0 (socket says 172.21.15.23:48944) 2023-06-22T08:00:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe33959c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33414f850 0x7fe334146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.207+0000 7fe333fff700 1 -- 172.21.15.23:0/3742720049 learned_addr learned my addr 172.21.15.23:0/3742720049 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.208+0000 7fe333fff700 1 -- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe334071070 msgr2=0x7fe33406fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.208+0000 7fe333fff700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe334071070 0x7fe33406fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.208+0000 7fe333fff700 1 -- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33414f850 msgr2=0x7fe334146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.208+0000 7fe333fff700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33414f850 0x7fe334146a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.208+0000 7fe333fff700 1 -- 172.21.15.23:0/3742720049 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3280094a0 con 0x7fe3340c69e0 2023-06-22T08:00:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.208+0000 7fe333fff700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3340c69e0 0x7fe334153360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe32000c780 tx=0x7fe32000ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.208+0000 7fe3317fa700 1 -- 172.21.15.23:0/3742720049 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe320004c80 con 0x7fe3340c69e0 2023-06-22T08:00:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.208+0000 7fe3317fa700 1 -- 172.21.15.23:0/3742720049 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe320004e00 con 0x7fe3340c69e0 2023-06-22T08:00:48.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.208+0000 7fe3317fa700 1 -- 172.21.15.23:0/3742720049 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe320007310 con 0x7fe3340c69e0 2023-06-22T08:00:48.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.208+0000 7fe33afff700 1 -- 172.21.15.23:0/3742720049 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33414a2e0 con 0x7fe3340c69e0 2023-06-22T08:00:48.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.209+0000 7fe33afff700 1 -- 172.21.15.23:0/3742720049 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3341dfeb0 con 0x7fe3340c69e0 2023-06-22T08:00:48.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.210+0000 7fe33afff700 1 -- 172.21.15.23:0/3742720049 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3340c1210 con 0x7fe3340c69e0 2023-06-22T08:00:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.210+0000 7fe3317fa700 1 -- 172.21.15.23:0/3742720049 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe320007470 con 0x7fe3340c69e0 2023-06-22T08:00:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.212+0000 7fe3317fa700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe31806ad10 0x7fe31806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.212+0000 7fe3317fa700 1 -- 172.21.15.23:0/3742720049 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe3200893b0 con 0x7fe3340c69e0 2023-06-22T08:00:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.213+0000 7fe338d9b700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe31806ad10 0x7fe31806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.213+0000 7fe338d9b700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe31806ad10 0x7fe31806d1c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe328009fd0 tx=0x7fe32801c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.217+0000 7fe3317fa700 1 -- 172.21.15.23:0/3742720049 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe320054170 con 0x7fe3340c69e0 2023-06-22T08:00:48.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.469+0000 7fe33afff700 1 -- 172.21.15.23:0/3742720049 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe3340717b0 con 0x7fe3340c69e0 2023-06-22T08:00:48.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.469+0000 7fe3317fa700 1 -- 172.21.15.23:0/3742720049 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+88 (secure 0 0 0) 0x7fe320057790 con 0x7fe3340c69e0 2023-06-22T08:00:48.473 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-06-22T08:00:48.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.471+0000 7fe33afff700 1 -- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe31806ad10 msgr2=0x7fe31806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:48.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.471+0000 7fe33afff700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe31806ad10 0x7fe31806d1c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe328009fd0 tx=0x7fe32801c040 comp rx=0 tx=0).stop 2023-06-22T08:00:48.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.471+0000 7fe33afff700 1 -- 172.21.15.23:0/3742720049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3340c69e0 msgr2=0x7fe334153360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:48.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.471+0000 7fe33afff700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3340c69e0 0x7fe334153360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe32000c780 tx=0x7fe32000ca90 comp rx=0 tx=0).stop 2023-06-22T08:00:48.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.471+0000 7fe33afff700 1 -- 172.21.15.23:0/3742720049 shutdown_connections 2023-06-22T08:00:48.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.471+0000 7fe33afff700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe31806ad10 0x7fe31806d1c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:48.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.471+0000 7fe33afff700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe33414f850 0x7fe334146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:48.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.471+0000 7fe33afff700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe334071070 0x7fe33406fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:48.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.471+0000 7fe33afff700 1 --2- 172.21.15.23:0/3742720049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3340c69e0 0x7fe334153360 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:48.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.471+0000 7fe33afff700 1 -- 172.21.15.23:0/3742720049 >> 172.21.15.23:0/3742720049 conn(0x7fe3340ca3f0 msgr2=0x7fe334070190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:48.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.472+0000 7fe33afff700 1 -- 172.21.15.23:0/3742720049 shutdown_connections 2023-06-22T08:00:48.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:48.472+0000 7fe33afff700 1 -- 172.21.15.23:0/3742720049 wait complete. 2023-06-22T08:00:48.493 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-06-22T08:00:48.493 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-22T08:00:48.493 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2023-06-22T08:00:48.494 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-06-22T08:00:48.494 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-06-22T08:00:48.494 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-06-22T08:00:48.494 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-22T08:00:48.494 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-22T08:00:48.495 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-06-22T08:00:48.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:48.495 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2023-06-22T08:00:48.538 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-22T08:00:48.538 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-22T08:00:48.539 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2023-06-22T08:00:48.539 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-06-22T08:00:48.539 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2023-06-22T08:00:48.539 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-06-22T08:00:48.539 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-22T08:00:48.539 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2023-06-22T08:00:48.539 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2023-06-22T08:00:48.540 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-22T08:00:48.540 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2023-06-22T08:00:48.540 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2023-06-22T08:00:48.540 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-22T08:00:48.540 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2023-06-22T08:00:48.540 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2023-06-22T08:00:48.540 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-06-22T08:00:48.541 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 41902sec preferred_lft 41902sec 2023-06-22T08:00:48.541 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2023-06-22T08:00:48.541 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2023-06-22T08:00:48.541 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-22T08:00:48.541 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2023-06-22T08:00:48.541 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2023-06-22T08:00:48.542 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-06-22T08:00:48.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:48.543 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-06-22T08:00:48.543 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-06-22T08:00:48.543 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2023-06-22T08:00:48.543 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2023-06-22T08:00:48.544 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2023-06-22T08:00:48.544 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-06-22T08:00:48.544 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-06-22T08:00:48.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:48.780 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-22T08:00:48.848 INFO:teuthology.orchestra.run.smithi023.stdout:1 2023-06-22T08:00:48.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:48.851 DEBUG:teuthology.orchestra.run.smithi023:> route 2023-06-22T08:00:48.897 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2023-06-22T08:00:48.897 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-22T08:00:48.898 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-06-22T08:00:48.898 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-06-22T08:00:48.898 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-22T08:00:48.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:48.899 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-06-22T08:00:48.899 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-06-22T08:00:48.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-06-22T08:00:48.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-06-22T08:00:48.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-06-22T08:00:48.900 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-06-22T08:00:49.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:49.109 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2023-06-22T08:00:49.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:49.152 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2023-06-22T08:00:49.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:49.234 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-06-22T08:00:49.235 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-06-22T08:00:49.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-22T08:00:49.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-06-22T08:00:49.235 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-06-22T08:00:49.415 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-06-22T08:00:49.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:49.415 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-06-22T08:00:49.416 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-06-22T08:00:49.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-06-22T08:00:49.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-06-22T08:00:49.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-06-22T08:00:49.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-06-22T08:00:49.417 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-06-22T08:00:49.417 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-06-22T08:00:49.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:49.661 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2023-06-22T08:00:49.661 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2023-06-22T08:00:49.662 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2023-06-22T08:00:49.662 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2023-06-22T08:00:49.662 DEBUG:teuthology.orchestra.run.smithi023:> ') 2023-06-22T08:00:49.841 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-22T08:00:49.841 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-06-22T08:00:49.884 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-06-22T08:00:49.884 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-22T08:00:49.885 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-06-22T08:00:49.965 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-06-22T08:00:49.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:49.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-06-22T08:00:50.111 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-06-22T08:00:50.111 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-06-22T08:00:50.111 INFO:teuthology.orchestra.run.smithi023.stdout:invalid new device string format 2023-06-22T08:00:50.524 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-06-22T08:00:50.525 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-06-22T08:00:50.525 INFO:teuthology.orchestra.run.smithi023.stdout:invalid new device string format 2023-06-22T08:00:50.525 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: resolved to: "172.21.15.23:3300,172.21.15.67:3300,172.21.15.67:3301" 2023-06-22T08:00:50.525 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: switching to using v1 address with old syntax 2023-06-22T08:00:50.551 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-06-22T08:00:50.551 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-06-22T08:00:50.551 INFO:teuthology.orchestra.run.smithi023.stdout:invalid new device string format 2023-06-22T08:00:50.551 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: resolved to: "172.21.15.23:3300,172.21.15.67:3300,172.21.15.67:3301" 2023-06-22T08:00:50.551 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: switching to using v1 address with old syntax 2023-06-22T08:00:50.551 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: resolved to: "172.21.15.23:6789,172.21.15.67:6789,172.21.15.67:6790" 2023-06-22T08:00:50.552 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.23:6789,172.21.15.67:6789,172.21.15.67:6790:/ 2023-06-22T08:00:50.552 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=2f053100-95a5-44db-a4cd-e7710a2fac9d" will pass to kernel 2023-06-22T08:00:50.558 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-06-22T08:00:50.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:50.558 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-06-22T08:00:50.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:50.628 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-06-22T08:00:50.714 INFO:teuthology.orchestra.run.smithi023.stdout:client4757 2023-06-22T08:00:50.714 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T08:00:50.715 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-22T08:00:50.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:50.757 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-06-22T08:00:50.880 INFO:teuthology.orchestra.run.smithi023.stdout:client4757 2023-06-22T08:00:50.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:50.881 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-06-22T08:00:50.965 INFO:teuthology.orchestra.run.smithi023.stdout:client4757 2023-06-22T08:00:50.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:00:50.966 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/2f053100-95a5-44db-a4cd-e7710a2fac9d.client4757/status') 2023-06-22T08:00:51.072 INFO:teuthology.orchestra.run.smithi023.stdout:instance: client.4757 (3)172.21.15.23:0/19027214 2023-06-22T08:00:51.073 INFO:teuthology.orchestra.run.smithi023.stdout:blocklisted: false 2023-06-22T08:00:51.073 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2023-06-22T08:00:51.073 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2023-06-22T08:00:51.073 INFO:teuthology.orchestra.run.smithi023.stderr:68 bytes copied, 5.3836e-05 s, 1.3 MB/s 2023-06-22T08:00:51.075 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-06-22T08:00:51.112 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:51.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.309+0000 7f8cd5bff700 1 Processor -- start 2023-06-22T08:00:51.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.309+0000 7f8cd5bff700 1 -- start start 2023-06-22T08:00:51.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.311+0000 7f8cd5bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cd014f850 0x7f8cd00c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.311+0000 7f8cd5bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd0071070 0x7f8cd00c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.311+0000 7f8cd5bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd00c51a0 con 0x7f8cd00c69e0 2023-06-22T08:00:51.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.311+0000 7f8cd5bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd00c44e0 con 0x7f8cd014f850 2023-06-22T08:00:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.311+0000 7f8cd5bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00c5440 con 0x7f8cd0071070 2023-06-22T08:00:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.311+0000 7f8cceffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f8cd00c69e0 0x7f8cd0155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34916/0 (socket says 172.21.15.23:34916) 2023-06-22T08:00:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.311+0000 7f8cceffd700 1 -- 172.21.15.23:0/3543396160 learned_addr learned my addr 172.21.15.23:0/3543396160 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.311+0000 7f8ccf7fe700 1 --2- 172.21.15.23:0/3543396160 >> v2:172.21.15.67:3300/0 conn(0x7f8cd014f850 0x7f8cd00c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.311+0000 7f8ccffff700 1 --2- 172.21.15.23:0/3543396160 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd0071070 0x7f8cd00c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.312+0000 7f8cce7fc700 1 -- 172.21.15.23:0/3543396160 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459468809 0 0) 0x7f8cd00c51a0 con 0x7f8cd00c69e0 2023-06-22T08:00:51.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.312+0000 7f8cce7fc700 1 -- 172.21.15.23:0/3543396160 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb80035b0 con 0x7f8cd00c69e0 2023-06-22T08:00:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.312+0000 7f8ccf7fe700 1 -- 172.21.15.23:0/3543396160 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd0071070 msgr2=0x7f8cd00c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.312+0000 7f8ccf7fe700 1 --2- 172.21.15.23:0/3543396160 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd0071070 0x7f8cd00c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.312+0000 7f8ccf7fe700 1 -- 172.21.15.23:0/3543396160 >> v1:172.21.15.23:6789/0 conn(0x7f8cd00c69e0 legacy=0x7f8cd0155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.312+0000 7f8ccf7fe700 1 -- 172.21.15.23:0/3543396160 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd00c5720 con 0x7f8cd014f850 2023-06-22T08:00:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.312+0000 7f8ccf7fe700 1 --2- 172.21.15.23:0/3543396160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd014f850 0x7f8cd00c3080 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009840 tx=0x7f8cc0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fc572bc38ca023b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.312+0000 7f8cce7fc700 1 -- 172.21.15.23:0/3543396160 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0011830 con 0x7f8cd014f850 2023-06-22T08:00:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.312+0000 7f8cce7fc700 1 -- 172.21.15.23:0/3543396160 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0011990 con 0x7f8cd014f850 2023-06-22T08:00:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.312+0000 7f8cce7fc700 1 -- 172.21.15.23:0/3543396160 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0011d00 con 0x7f8cd014f850 2023-06-22T08:00:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.313+0000 7f8cd5bff700 1 -- 172.21.15.23:0/3543396160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd014f850 msgr2=0x7f8cd00c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.313+0000 7f8cd5bff700 1 --2- 172.21.15.23:0/3543396160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd014f850 0x7f8cd00c3080 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009840 tx=0x7f8cc0009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.313+0000 7f8cd5bff700 1 -- 172.21.15.23:0/3543396160 shutdown_connections 2023-06-22T08:00:51.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.313+0000 7f8cd5bff700 1 --2- 172.21.15.23:0/3543396160 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd0071070 0x7f8cd00c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.313+0000 7f8cd5bff700 1 --2- 172.21.15.23:0/3543396160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd014f850 0x7f8cd00c3080 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.313+0000 7f8cd5bff700 1 -- 172.21.15.23:0/3543396160 >> 172.21.15.23:0/3543396160 conn(0x7f8cd00ca3f0 msgr2=0x7f8cd00ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:51.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.313+0000 7f8cd5bff700 1 -- 172.21.15.23:0/3543396160 shutdown_connections 2023-06-22T08:00:51.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.313+0000 7f8cd5bff700 1 -- 172.21.15.23:0/3543396160 wait complete. 2023-06-22T08:00:51.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.315+0000 7f8cd5bff700 1 Processor -- start 2023-06-22T08:00:51.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.315+0000 7f8cd5bff700 1 -- start start 2023-06-22T08:00:51.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cd5bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd0071070 0x7f8cd01e0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cd5bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd00c69e0 0x7f8cd01ded30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cd5bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd014f850 0x7f8cd01dff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cd5bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0145eb0 con 0x7f8cd00c69e0 2023-06-22T08:00:51.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cd5bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00c10c0 con 0x7f8cd0071070 2023-06-22T08:00:51.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cd5bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00c8b00 con 0x7f8cd014f850 2023-06-22T08:00:51.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cceffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd00c69e0 0x7f8cd01ded30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cceffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd00c69e0 0x7f8cd01ded30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35556/0 (socket says 172.21.15.23:35556) 2023-06-22T08:00:51.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cceffd700 1 -- 172.21.15.23:0/207448928 learned_addr learned my addr 172.21.15.23:0/207448928 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:51.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8ccf7fe700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd0071070 0x7f8cd01e0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8ccffff700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd014f850 0x7f8cd01dff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cceffd700 1 -- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd014f850 msgr2=0x7f8cd01dff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cceffd700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd014f850 0x7f8cd01dff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cceffd700 1 -- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd0071070 msgr2=0x7f8cd01e0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.316+0000 7f8cceffd700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd0071070 0x7f8cd01e0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.317+0000 7f8cceffd700 1 -- 172.21.15.23:0/207448928 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc00094a0 con 0x7f8cd00c69e0 2023-06-22T08:00:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.317+0000 7f8ccffff700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd014f850 0x7f8cd01dff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:00:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.317+0000 7f8cceffd700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd00c69e0 0x7f8cd01ded30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400c4d0 tx=0x7f8cc400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.317+0000 7f8cb7fff700 1 -- 172.21.15.23:0/207448928 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4005d70 con 0x7f8cd00c69e0 2023-06-22T08:00:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.317+0000 7f8cb7fff700 1 -- 172.21.15.23:0/207448928 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc40070d0 con 0x7f8cd00c69e0 2023-06-22T08:00:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.317+0000 7f8cd5bff700 1 -- 172.21.15.23:0/207448928 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd00c0b00 con 0x7f8cd00c69e0 2023-06-22T08:00:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.317+0000 7f8cb7fff700 1 -- 172.21.15.23:0/207448928 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4007440 con 0x7f8cd00c69e0 2023-06-22T08:00:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.317+0000 7f8cd5bff700 1 -- 172.21.15.23:0/207448928 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd0145ad0 con 0x7f8cd00c69e0 2023-06-22T08:00:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.318+0000 7f8cd5bff700 1 -- 172.21.15.23:0/207448928 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd00c2950 con 0x7f8cd00c69e0 2023-06-22T08:00:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.319+0000 7f8cb7fff700 1 -- 172.21.15.23:0/207448928 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8cc40075a0 con 0x7f8cd00c69e0 2023-06-22T08:00:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.321+0000 7f8cb7fff700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cbc06ace0 0x7f8cbc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.321+0000 7f8cb7fff700 1 -- 172.21.15.23:0/207448928 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f8cc4088b80 con 0x7f8cd00c69e0 2023-06-22T08:00:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.321+0000 7f8ccf7fe700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cbc06ace0 0x7f8cbc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.322+0000 7f8ccf7fe700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cbc06ace0 0x7f8cbc06d190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0000c00 tx=0x7f8cc001c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.326+0000 7f8cb7fff700 1 -- 172.21.15.23:0/207448928 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8cc4053950 con 0x7f8cd00c69e0 2023-06-22T08:00:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.534+0000 7f8cd5bff700 1 -- 172.21.15.23:0/207448928 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cd006fc00 con 0x7f8cd00c69e0 2023-06-22T08:00:51.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.535+0000 7f8cb7fff700 1 -- 172.21.15.23:0/207448928 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f8cc4056f70 con 0x7f8cd00c69e0 2023-06-22T08:00:51.539 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:51.539 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 -- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cbc06ace0 msgr2=0x7f8cbc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cbc06ace0 0x7f8cbc06d190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0000c00 tx=0x7f8cc001c040 comp rx=0 tx=0).stop 2023-06-22T08:00:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 -- 172.21.15.23:0/207448928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd00c69e0 msgr2=0x7f8cd01ded30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd00c69e0 0x7f8cd01ded30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400c4d0 tx=0x7f8cc400c7e0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 -- 172.21.15.23:0/207448928 shutdown_connections 2023-06-22T08:00:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cd014f850 0x7f8cd01dff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cbc06ace0 0x7f8cbc06d190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cd0071070 0x7f8cd01e0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 --2- 172.21.15.23:0/207448928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd00c69e0 0x7f8cd01ded30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 -- 172.21.15.23:0/207448928 >> 172.21.15.23:0/207448928 conn(0x7f8cd00ca3f0 msgr2=0x7f8cd01529f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:51.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 -- 172.21.15.23:0/207448928 shutdown_connections 2023-06-22T08:00:51.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.538+0000 7f8cd5bff700 1 -- 172.21.15.23:0/207448928 wait complete. 2023-06-22T08:00:51.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T07:59:40.617747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-22T07:59:48.190528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-22T07:59:53.576494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-22T07:59:54.105472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-22T07:59:57.198896+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-22T08:00:51.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T08:00:51.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.761+0000 7f632e3ff700 1 Processor -- start 2023-06-22T08:00:51.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.761+0000 7f632e3ff700 1 -- start start 2023-06-22T08:00:51.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.762+0000 7f632e3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 0x7f63280ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.762+0000 7f632e3ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63280c9aa0 con 0x7f63281593f0 2023-06-22T08:00:51.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.762+0000 7f632e3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63280ca880 con 0x7f6328152ba0 2023-06-22T08:00:51.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.762+0000 7f632e3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632806aed0 con 0x7f63280c9ee0 2023-06-22T08:00:51.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.762+0000 7f63277fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f63281593f0 0x7f63280c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34924/0 (socket says 172.21.15.23:34924) 2023-06-22T08:00:51.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.762+0000 7f63277fe700 1 -- 172.21.15.23:0/3270288805 learned_addr learned my addr 172.21.15.23:0/3270288805 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:51.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.762+0000 7f632c99c700 1 --2- 172.21.15.23:0/3270288805 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 0x7f63280ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.763+0000 7f6326ffd700 1 -- 172.21.15.23:0/3270288805 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999400948 0 0) 0x7f63280c9aa0 con 0x7f63281593f0 2023-06-22T08:00:51.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.763+0000 7f6326ffd700 1 -- 172.21.15.23:0/3270288805 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63100035b0 con 0x7f63281593f0 2023-06-22T08:00:51.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.763+0000 7f632c99c700 1 -- 172.21.15.23:0/3270288805 >> v1:172.21.15.67:6789/0 conn(0x7f6328152ba0 legacy=0x7f63280c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.763+0000 7f6326ffd700 1 -- 172.21.15.23:0/3270288805 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042923469 0 0) 0x7f63280ca880 con 0x7f6328152ba0 2023-06-22T08:00:51.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.763+0000 7f632c99c700 1 -- 172.21.15.23:0/3270288805 >> v1:172.21.15.23:6789/0 conn(0x7f63281593f0 legacy=0x7f63280c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.763+0000 7f632c99c700 1 -- 172.21.15.23:0/3270288805 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632806b1b0 con 0x7f63280c9ee0 2023-06-22T08:00:51.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.763+0000 7f632c99c700 1 --2- 172.21.15.23:0/3270288805 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 0x7f63280ca340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6320009dc0 tx=0x7f632000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0b2c159eca068fd server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:51.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.763+0000 7f6326ffd700 1 -- 172.21.15.23:0/3270288805 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632000e070 con 0x7f63280c9ee0 2023-06-22T08:00:51.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.763+0000 7f6326ffd700 1 -- 172.21.15.23:0/3270288805 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632000bb20 con 0x7f63280c9ee0 2023-06-22T08:00:51.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.763+0000 7f6326ffd700 1 -- 172.21.15.23:0/3270288805 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632000bed0 con 0x7f63280c9ee0 2023-06-22T08:00:51.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.764+0000 7f632e3ff700 1 -- 172.21.15.23:0/3270288805 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 msgr2=0x7f63280ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.764+0000 7f632e3ff700 1 --2- 172.21.15.23:0/3270288805 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 0x7f63280ca340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6320009dc0 tx=0x7f632000b330 comp rx=0 tx=0).stop 2023-06-22T08:00:51.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.764+0000 7f632e3ff700 1 -- 172.21.15.23:0/3270288805 shutdown_connections 2023-06-22T08:00:51.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.764+0000 7f632e3ff700 1 --2- 172.21.15.23:0/3270288805 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 0x7f63280ca340 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.764+0000 7f632e3ff700 1 -- 172.21.15.23:0/3270288805 >> 172.21.15.23:0/3270288805 conn(0x7f6328065b90 msgr2=0x7f6328065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:51.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.764+0000 7f632e3ff700 1 -- 172.21.15.23:0/3270288805 shutdown_connections 2023-06-22T08:00:51.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.764+0000 7f632e3ff700 1 -- 172.21.15.23:0/3270288805 wait complete. 2023-06-22T08:00:51.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.766+0000 7f632e3ff700 1 Processor -- start 2023-06-22T08:00:51.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.766+0000 7f632e3ff700 1 -- start start 2023-06-22T08:00:51.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f632e3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 0x7f6328158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f632e3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6328152ba0 0x7f63280b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f632e3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63281593f0 0x7f63280b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f632e3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63280c5fd0 con 0x7f6328152ba0 2023-06-22T08:00:51.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f632e3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63280b2380 con 0x7f63281593f0 2023-06-22T08:00:51.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f632e3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63280cecb0 con 0x7f63280c9ee0 2023-06-22T08:00:51.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f63277fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6328152ba0 0x7f63280b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f63277fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6328152ba0 0x7f63280b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35572/0 (socket says 172.21.15.23:35572) 2023-06-22T08:00:51.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f63277fe700 1 -- 172.21.15.23:0/1945661468 learned_addr learned my addr 172.21.15.23:0/1945661468 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:51.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f632c99c700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63281593f0 0x7f63280b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.767+0000 7f6327fff700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 0x7f6328158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f63277fe700 1 -- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 msgr2=0x7f6328158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f63277fe700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 0x7f6328158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f63277fe700 1 -- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63281593f0 msgr2=0x7f63280b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f63277fe700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63281593f0 0x7f63280b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f63277fe700 1 -- 172.21.15.23:0/1945661468 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6320009a70 con 0x7f6328152ba0 2023-06-22T08:00:51.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f63277fe700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6328152ba0 0x7f63280b27d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c4d0 tx=0x7f631c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:51.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f6324ff9700 1 -- 172.21.15.23:0/1945661468 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c005d70 con 0x7f6328152ba0 2023-06-22T08:00:51.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f6324ff9700 1 -- 172.21.15.23:0/1945661468 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c0070d0 con 0x7f6328152ba0 2023-06-22T08:00:51.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f6324ff9700 1 -- 172.21.15.23:0/1945661468 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c007440 con 0x7f6328152ba0 2023-06-22T08:00:51.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f632e3ff700 1 -- 172.21.15.23:0/1945661468 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63280ce9d0 con 0x7f6328152ba0 2023-06-22T08:00:51.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.768+0000 7f632e3ff700 1 -- 172.21.15.23:0/1945661468 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63280b4280 con 0x7f6328152ba0 2023-06-22T08:00:51.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.769+0000 7f632e3ff700 1 -- 172.21.15.23:0/1945661468 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63280cbed0 con 0x7f6328152ba0 2023-06-22T08:00:51.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.770+0000 7f6324ff9700 1 -- 172.21.15.23:0/1945661468 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f631c0075a0 con 0x7f6328152ba0 2023-06-22T08:00:51.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.772+0000 7f6324ff9700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f631406ad10 0x7f631406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:51.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.772+0000 7f6324ff9700 1 -- 172.21.15.23:0/1945661468 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f631c088a20 con 0x7f6328152ba0 2023-06-22T08:00:51.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.772+0000 7f6327fff700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f631406ad10 0x7f631406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:51.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.773+0000 7f6327fff700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f631406ad10 0x7f631406d1c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6318007fd0 tx=0x7f6318009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:51.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.777+0000 7f6324ff9700 1 -- 172.21.15.23:0/1945661468 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f631c053880 con 0x7f6328152ba0 2023-06-22T08:00:51.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.984+0000 7f632e3ff700 1 -- 172.21.15.23:0/1945661468 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f632804ea20 con 0x7f6328152ba0 2023-06-22T08:00:51.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.984+0000 7f6324ff9700 1 -- 172.21.15.23:0/1945661468 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f631c017020 con 0x7f6328152ba0 2023-06-22T08:00:51.988 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-06-22T08:00:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.986+0000 7f632e3ff700 1 -- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f631406ad10 msgr2=0x7f631406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.986+0000 7f632e3ff700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f631406ad10 0x7f631406d1c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6318007fd0 tx=0x7f6318009040 comp rx=0 tx=0).stop 2023-06-22T08:00:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.986+0000 7f632e3ff700 1 -- 172.21.15.23:0/1945661468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6328152ba0 msgr2=0x7f63280b27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.986+0000 7f632e3ff700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6328152ba0 0x7f63280b27d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c4d0 tx=0x7f631c00c7e0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.986+0000 7f632e3ff700 1 -- 172.21.15.23:0/1945661468 shutdown_connections 2023-06-22T08:00:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.986+0000 7f632e3ff700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f631406ad10 0x7f631406d1c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.986+0000 7f632e3ff700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63281593f0 0x7f63280b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.986+0000 7f632e3ff700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63280c9ee0 0x7f6328158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.986+0000 7f632e3ff700 1 --2- 172.21.15.23:0/1945661468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6328152ba0 0x7f63280b27d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.986+0000 7f632e3ff700 1 -- 172.21.15.23:0/1945661468 >> 172.21.15.23:0/1945661468 conn(0x7f6328065b90 msgr2=0x7f6328151df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.987+0000 7f632e3ff700 1 -- 172.21.15.23:0/1945661468 shutdown_connections 2023-06-22T08:00:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:51.987+0000 7f632e3ff700 1 -- 172.21.15.23:0/1945661468 wait complete. 2023-06-22T08:00:52.008 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T08:00:52.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.207+0000 7fb4c0bff700 1 Processor -- start 2023-06-22T08:00:52.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.207+0000 7fb4c0bff700 1 -- start start 2023-06-22T08:00:52.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.208+0000 7fb4c0bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb4bc0c9ee0 0x7fb4bc0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.208+0000 7fb4c0bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4bc0c9aa0 con 0x7fb4bc1593f0 2023-06-22T08:00:52.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4c0bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4bc06aed0 con 0x7fb4bc152ba0 2023-06-22T08:00:52.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4c0bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4bc06b1b0 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4b9d9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb4bc1593f0 0x7fb4bc0c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34930/0 (socket says 172.21.15.23:34930) 2023-06-22T08:00:52.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4bad9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb4bc0c9ee0 0x7fb4bc0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4b9d9b700 1 -- 172.21.15.23:0/2154707489 learned_addr learned my addr 172.21.15.23:0/2154707489 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:52.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4b959a700 1 -- 172.21.15.23:0/2154707489 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318994578 0 0) 0x7fb4bc06aed0 con 0x7fb4bc152ba0 2023-06-22T08:00:52.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4b959a700 1 -- 172.21.15.23:0/2154707489 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a40035b0 con 0x7fb4bc152ba0 2023-06-22T08:00:52.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4bad9d700 1 -- 172.21.15.23:0/2154707489 >> v1:172.21.15.67:6789/0 conn(0x7fb4bc152ba0 legacy=0x7fb4bc0c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4bad9d700 1 -- 172.21.15.23:0/2154707489 >> v1:172.21.15.23:6789/0 conn(0x7fb4bc1593f0 legacy=0x7fb4bc0c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4b959a700 1 -- 172.21.15.23:0/2154707489 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934115028 0 0) 0x7fb4bc0c9aa0 con 0x7fb4bc1593f0 2023-06-22T08:00:52.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.209+0000 7fb4bad9d700 1 -- 172.21.15.23:0/2154707489 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4bc06b490 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.210+0000 7fb4bad9d700 1 --2- 172.21.15.23:0/2154707489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4bc0c9ee0 0x7fb4bc0ca2f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009d90 tx=0x7fb4b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5443e25b1b42e1 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:52.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.210+0000 7fb4b959a700 1 -- 172.21.15.23:0/2154707489 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b400e070 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.210+0000 7fb4b959a700 1 -- 172.21.15.23:0/2154707489 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b400bb20 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.210+0000 7fb4b959a700 1 -- 172.21.15.23:0/2154707489 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b400bed0 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.210+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2154707489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4bc0c9ee0 msgr2=0x7fb4bc0ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.210+0000 7fb4c0bff700 1 --2- 172.21.15.23:0/2154707489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4bc0c9ee0 0x7fb4bc0ca2f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009d90 tx=0x7fb4b400b330 comp rx=0 tx=0).stop 2023-06-22T08:00:52.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.211+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2154707489 shutdown_connections 2023-06-22T08:00:52.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.211+0000 7fb4c0bff700 1 --2- 172.21.15.23:0/2154707489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4bc0c9ee0 0x7fb4bc0ca2f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.211+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2154707489 >> 172.21.15.23:0/2154707489 conn(0x7fb4bc065b90 msgr2=0x7fb4bc065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:52.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.211+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2154707489 shutdown_connections 2023-06-22T08:00:52.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.211+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2154707489 wait complete. 2023-06-22T08:00:52.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.212+0000 7fb4c0bff700 1 Processor -- start 2023-06-22T08:00:52.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.213+0000 7fb4c0bff700 1 -- start start 2023-06-22T08:00:52.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4c0bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4bc0c9ee0 0x7fb4bc0b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4c0bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4bc152ba0 0x7fb4bc0b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4c0bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4bc1593f0 0x7fb4bc0c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4c0bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc0c69a0 con 0x7fb4bc1593f0 2023-06-22T08:00:52.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4c0bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc0c7c70 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4c0bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc151c00 con 0x7fb4bc152ba0 2023-06-22T08:00:52.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4ba59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4bc0c9ee0 0x7fb4bc0b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4b9d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4bc152ba0 0x7fb4bc0b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4bad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4bc1593f0 0x7fb4bc0c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4ba59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4bc0c9ee0 0x7fb4bc0b2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51282/0 (socket says 172.21.15.23:51282) 2023-06-22T08:00:52.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4ba59c700 1 -- 172.21.15.23:0/2041581289 learned_addr learned my addr 172.21.15.23:0/2041581289 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:52.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4b9d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4bc152ba0 0x7fb4bc0b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49000/0 (socket says 172.21.15.23:49000) 2023-06-22T08:00:52.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4ba59c700 1 -- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4bc152ba0 msgr2=0x7fb4bc0b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4ba59c700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4bc152ba0 0x7fb4bc0b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4ba59c700 1 -- 172.21.15.23:0/2041581289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4bc1593f0 msgr2=0x7fb4bc0c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4ba59c700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4bc1593f0 0x7fb4bc0c81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.214+0000 7fb4ba59c700 1 -- 172.21.15.23:0/2041581289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b4009a70 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.215+0000 7fb4ba59c700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4bc0c9ee0 0x7fb4bc0b2350 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00c530 tx=0x7fb4ac00c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:52.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.215+0000 7fb4aaffd700 1 -- 172.21.15.23:0/2041581289 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac0052d0 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.215+0000 7fb4aaffd700 1 -- 172.21.15.23:0/2041581289 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac005450 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.215+0000 7fb4aaffd700 1 -- 172.21.15.23:0/2041581289 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac0038c0 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.215+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2041581289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4bc0c6700 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.215+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2041581289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4bc0b3560 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.216+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2041581289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4bc0b3e90 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.217+0000 7fb4aaffd700 1 -- 172.21.15.23:0/2041581289 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb4ac003a20 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.218+0000 7fb4aaffd700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb49c06ace0 0x7fb49c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.218+0000 7fb4aaffd700 1 -- 172.21.15.23:0/2041581289 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb4ac088be0 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.218+0000 7fb4b9d9b700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb49c06ace0 0x7fb49c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.219+0000 7fb4b9d9b700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb49c06ace0 0x7fb49c06d190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0007fd0 tx=0x7fb4b0007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:52.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.223+0000 7fb4aaffd700 1 -- 172.21.15.23:0/2041581289 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb4ac053900 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.428+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2041581289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb4bc0cbed0 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.429+0000 7fb4aaffd700 1 -- 172.21.15.23:0/2041581289 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb4ac017020 con 0x7fb4bc0c9ee0 2023-06-22T08:00:52.432 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2023-06-22T08:00:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb49c06ace0 msgr2=0x7fb49c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb49c06ace0 0x7fb49c06d190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0007fd0 tx=0x7fb4b0007de0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4bc0c9ee0 msgr2=0x7fb4bc0b2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4bc0c9ee0 0x7fb4bc0b2350 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac00c530 tx=0x7fb4ac00c840 comp rx=0 tx=0).stop 2023-06-22T08:00:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2041581289 shutdown_connections 2023-06-22T08:00:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4bc1593f0 0x7fb4bc0c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb49c06ace0 0x7fb49c06d190 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4bc0c9ee0 0x7fb4bc0b2350 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 --2- 172.21.15.23:0/2041581289 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4bc152ba0 0x7fb4bc0b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2041581289 >> 172.21.15.23:0/2041581289 conn(0x7fb4bc065b90 msgr2=0x7fb4bc150190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.431+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2041581289 shutdown_connections 2023-06-22T08:00:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.432+0000 7fb4c0bff700 1 -- 172.21.15.23:0/2041581289 wait complete. 2023-06-22T08:00:52.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-22T08:00:52.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.653+0000 7fd3423ff700 1 Processor -- start 2023-06-22T08:00:52.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.653+0000 7fd3423ff700 1 -- start start 2023-06-22T08:00:52.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.654+0000 7fd3423ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd33c152ba0 0x7fd33c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.654+0000 7fd3423ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd33c0c9260 0x7fd33c0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.654+0000 7fd3423ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c159930 con 0x7fd33c0c4650 2023-06-22T08:00:52.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.654+0000 7fd3423ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c0c4a60 con 0x7fd33c0c9260 2023-06-22T08:00:52.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.654+0000 7fd3423ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c0c9aa0 con 0x7fd33c152ba0 2023-06-22T08:00:52.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.655+0000 7fd33b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd33c0c9260 0x7fd33c0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.655+0000 7fd33bfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd33c152ba0 0x7fd33c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.655+0000 7fd33b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd33c0c9260 0x7fd33c0c4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35600/0 (socket says 172.21.15.23:35600) 2023-06-22T08:00:52.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.655+0000 7fd33b7fe700 1 -- 172.21.15.23:0/1142712202 learned_addr learned my addr 172.21.15.23:0/1142712202 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:52.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.655+0000 7fd34099c700 1 --1- 172.21.15.23:0/1142712202 >> v1:172.21.15.67:6789/0 conn(0x7fd33c0c4650 0x7fd33c0c9ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:57766/0 (socket says 172.21.15.23:57766) 2023-06-22T08:00:52.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.655+0000 7fd33b7fe700 1 -- 172.21.15.23:0/1142712202 >> v2:172.21.15.67:3300/0 conn(0x7fd33c152ba0 msgr2=0x7fd33c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:52.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.655+0000 7fd33b7fe700 1 --2- 172.21.15.23:0/1142712202 >> v2:172.21.15.67:3300/0 conn(0x7fd33c152ba0 0x7fd33c1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.655+0000 7fd33b7fe700 1 -- 172.21.15.23:0/1142712202 >> v1:172.21.15.67:6789/0 conn(0x7fd33c0c4650 legacy=0x7fd33c0c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.655+0000 7fd33b7fe700 1 -- 172.21.15.23:0/1142712202 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c06aed0 con 0x7fd33c0c9260 2023-06-22T08:00:52.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.655+0000 7fd33b7fe700 1 --2- 172.21.15.23:0/1142712202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c9260 0x7fd33c0c4110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd33c08c670 tx=0x7fd330009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180ff986a14e905c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:52.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd33affd700 1 -- 172.21.15.23:0/1142712202 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330011830 con 0x7fd33c0c9260 2023-06-22T08:00:52.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd33affd700 1 -- 172.21.15.23:0/1142712202 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd330011990 con 0x7fd33c0c9260 2023-06-22T08:00:52.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd33affd700 1 -- 172.21.15.23:0/1142712202 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330011d20 con 0x7fd33c0c9260 2023-06-22T08:00:52.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd3423ff700 1 -- 172.21.15.23:0/1142712202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c9260 msgr2=0x7fd33c0c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd3423ff700 1 --2- 172.21.15.23:0/1142712202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c9260 0x7fd33c0c4110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd33c08c670 tx=0x7fd330009bd0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd3423ff700 1 -- 172.21.15.23:0/1142712202 shutdown_connections 2023-06-22T08:00:52.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd3423ff700 1 --2- 172.21.15.23:0/1142712202 >> v2:172.21.15.67:3300/0 conn(0x7fd33c152ba0 0x7fd33c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd3423ff700 1 --2- 172.21.15.23:0/1142712202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c9260 0x7fd33c0c4110 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd3423ff700 1 -- 172.21.15.23:0/1142712202 >> 172.21.15.23:0/1142712202 conn(0x7fd33c065b90 msgr2=0x7fd33c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:52.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd3423ff700 1 -- 172.21.15.23:0/1142712202 shutdown_connections 2023-06-22T08:00:52.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.656+0000 7fd3423ff700 1 -- 172.21.15.23:0/1142712202 wait complete. 2023-06-22T08:00:52.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.658+0000 7fd3423ff700 1 Processor -- start 2023-06-22T08:00:52.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.658+0000 7fd3423ff700 1 -- start start 2023-06-22T08:00:52.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.659+0000 7fd3423ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c4650 0x7fd33c158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.659+0000 7fd3423ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd33c0c9260 0x7fd33c14d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.659+0000 7fd3423ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd33c152ba0 0x7fd33c0cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.659+0000 7fd3423ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c0cec90 con 0x7fd33c0c4650 2023-06-22T08:00:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.659+0000 7fd3423ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c158750 con 0x7fd33c152ba0 2023-06-22T08:00:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.659+0000 7fd3423ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c157180 con 0x7fd33c0c9260 2023-06-22T08:00:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.659+0000 7fd33bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c4650 0x7fd33c158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.659+0000 7fd33bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c4650 0x7fd33c158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35612/0 (socket says 172.21.15.23:35612) 2023-06-22T08:00:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.659+0000 7fd33bfff700 1 -- 172.21.15.23:0/26527164 learned_addr learned my addr 172.21.15.23:0/26527164 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.659+0000 7fd34099c700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd33c152ba0 0x7fd33c0cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd33b7fe700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd33c0c9260 0x7fd33c14d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd33bfff700 1 -- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd33c0c9260 msgr2=0x7fd33c14d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd33bfff700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd33c0c9260 0x7fd33c14d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd33bfff700 1 -- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd33c152ba0 msgr2=0x7fd33c0cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd33bfff700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd33c152ba0 0x7fd33c0cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd33bfff700 1 -- 172.21.15.23:0/26527164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd330009550 con 0x7fd33c0c4650 2023-06-22T08:00:52.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd33bfff700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c4650 0x7fd33c158210 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00e870 tx=0x7fd32c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:52.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd338ff9700 1 -- 172.21.15.23:0/26527164 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c009590 con 0x7fd33c0c4650 2023-06-22T08:00:52.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd338ff9700 1 -- 172.21.15.23:0/26527164 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c013680 con 0x7fd33c0c4650 2023-06-22T08:00:52.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd3423ff700 1 -- 172.21.15.23:0/26527164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c157460 con 0x7fd33c0c4650 2023-06-22T08:00:52.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.660+0000 7fd338ff9700 1 -- 172.21.15.23:0/26527164 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c0139f0 con 0x7fd33c0c4650 2023-06-22T08:00:52.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.661+0000 7fd3423ff700 1 -- 172.21.15.23:0/26527164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c157a20 con 0x7fd33c0c4650 2023-06-22T08:00:52.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.662+0000 7fd3423ff700 1 -- 172.21.15.23:0/26527164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c0cbed0 con 0x7fd33c0c4650 2023-06-22T08:00:52.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.662+0000 7fd338ff9700 1 -- 172.21.15.23:0/26527164 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd32c013b50 con 0x7fd33c0c4650 2023-06-22T08:00:52.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.663+0000 7fd338ff9700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd32806ace0 0x7fd32806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:52.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.663+0000 7fd338ff9700 1 -- 172.21.15.23:0/26527164 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd32c089730 con 0x7fd33c0c4650 2023-06-22T08:00:52.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.664+0000 7fd33b7fe700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd32806ace0 0x7fd32806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:52.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.664+0000 7fd33b7fe700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd32806ace0 0x7fd32806d190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd330009c10 tx=0x7fd330008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:52.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.668+0000 7fd338ff9700 1 -- 172.21.15.23:0/26527164 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd32c057e50 con 0x7fd33c0c4650 2023-06-22T08:00:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.876+0000 7fd3423ff700 1 -- 172.21.15.23:0/26527164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd33c04ea20 con 0x7fd33c0c4650 2023-06-22T08:00:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.876+0000 7fd338ff9700 1 -- 172.21.15.23:0/26527164 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd32c0579e0 con 0x7fd33c0c4650 2023-06-22T08:00:52.880 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 -- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd32806ace0 msgr2=0x7fd32806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd32806ace0 0x7fd32806d190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd330009c10 tx=0x7fd330008030 comp rx=0 tx=0).stop 2023-06-22T08:00:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 -- 172.21.15.23:0/26527164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c4650 msgr2=0x7fd33c158210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c4650 0x7fd33c158210 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00e870 tx=0x7fd32c00eb80 comp rx=0 tx=0).stop 2023-06-22T08:00:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 -- 172.21.15.23:0/26527164 shutdown_connections 2023-06-22T08:00:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd32806ace0 0x7fd32806d190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd33c152ba0 0x7fd33c0cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd33c0c4650 0x7fd33c158210 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 --2- 172.21.15.23:0/26527164 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd33c0c9260 0x7fd33c14d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 -- 172.21.15.23:0/26527164 >> 172.21.15.23:0/26527164 conn(0x7fd33c065b90 msgr2=0x7fd33c0b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 -- 172.21.15.23:0/26527164 shutdown_connections 2023-06-22T08:00:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:52.879+0000 7fd3423ff700 1 -- 172.21.15.23:0/26527164 wait complete. 2023-06-22T08:00:52.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-22T08:00:53.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.099+0000 7f52effff700 1 Processor -- start 2023-06-22T08:00:53.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.099+0000 7f52effff700 1 -- start start 2023-06-22T08:00:53.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.100+0000 7f52effff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52f0152950 0x7f52f00ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.100+0000 7f52effff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f52f015c160 0x7f52f00ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.100+0000 7f52effff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f00c8980 con 0x7f52f00c8fa0 2023-06-22T08:00:53.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52effff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f0152d20 con 0x7f52f015c160 2023-06-22T08:00:53.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52effff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f00b3fa0 con 0x7f52f0152950 2023-06-22T08:00:53.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52ef7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f52f015c160 0x7f52f00ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52ef7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f52f015c160 0x7f52f00ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35620/0 (socket says 172.21.15.23:35620) 2023-06-22T08:00:53.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52ef7fe700 1 -- 172.21.15.23:0/2209638991 learned_addr learned my addr 172.21.15.23:0/2209638991 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:53.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52e7fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f52f00c8fa0 0x7f52f015ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34934/0 (socket says 172.21.15.23:34934) 2023-06-22T08:00:53.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52eeffd700 1 --2- 172.21.15.23:0/2209638991 >> v2:172.21.15.67:3300/0 conn(0x7f52f0152950 0x7f52f00ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52ef7fe700 1 -- 172.21.15.23:0/2209638991 >> v2:172.21.15.67:3300/0 conn(0x7f52f0152950 msgr2=0x7f52f00ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:53.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52ef7fe700 1 --2- 172.21.15.23:0/2209638991 >> v2:172.21.15.67:3300/0 conn(0x7f52f0152950 0x7f52f00ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52ef7fe700 1 -- 172.21.15.23:0/2209638991 >> v1:172.21.15.23:6789/0 conn(0x7f52f00c8fa0 legacy=0x7f52f015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.101+0000 7f52ef7fe700 1 -- 172.21.15.23:0/2209638991 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00b4240 con 0x7f52f015c160 2023-06-22T08:00:53.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52ef7fe700 1 --2- 172.21.15.23:0/2209638991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f015c160 0x7f52f00ca9f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f52f008b5d0 tx=0x7f52e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e033531d1286414 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:53.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52ee7fc700 1 -- 172.21.15.23:0/2209638991 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e800e070 con 0x7f52f015c160 2023-06-22T08:00:53.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52ee7fc700 1 -- 172.21.15.23:0/2209638991 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e800b910 con 0x7f52f015c160 2023-06-22T08:00:53.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52ee7fc700 1 -- 172.21.15.23:0/2209638991 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e800bcc0 con 0x7f52f015c160 2023-06-22T08:00:53.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52effff700 1 -- 172.21.15.23:0/2209638991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f015c160 msgr2=0x7f52f00ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52effff700 1 --2- 172.21.15.23:0/2209638991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f015c160 0x7f52f00ca9f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f52f008b5d0 tx=0x7f52e800b330 comp rx=0 tx=0).stop 2023-06-22T08:00:53.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52effff700 1 -- 172.21.15.23:0/2209638991 shutdown_connections 2023-06-22T08:00:53.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52effff700 1 --2- 172.21.15.23:0/2209638991 >> v2:172.21.15.67:3300/0 conn(0x7f52f0152950 0x7f52f00ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52effff700 1 --2- 172.21.15.23:0/2209638991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f015c160 0x7f52f00ca9f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52effff700 1 -- 172.21.15.23:0/2209638991 >> 172.21.15.23:0/2209638991 conn(0x7f52f00ce620 msgr2=0x7f52f00cea20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:53.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.102+0000 7f52effff700 1 -- 172.21.15.23:0/2209638991 shutdown_connections 2023-06-22T08:00:53.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.103+0000 7f52effff700 1 -- 172.21.15.23:0/2209638991 wait complete. 2023-06-22T08:00:53.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.104+0000 7f52effff700 1 Processor -- start 2023-06-22T08:00:53.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.104+0000 7f52effff700 1 -- start start 2023-06-22T08:00:53.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.105+0000 7f52effff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52f00c8fa0 0x7f52f014f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.105+0000 7f52effff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f0152950 0x7f52f00c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.105+0000 7f52effff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f015c160 0x7f52f014fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.105+0000 7f52effff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f015a6f0 con 0x7f52f0152950 2023-06-22T08:00:53.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.105+0000 7f52effff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f015a100 con 0x7f52f015c160 2023-06-22T08:00:53.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.105+0000 7f52effff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f01eeed0 con 0x7f52f00c8fa0 2023-06-22T08:00:53.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.105+0000 7f52e7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f0152950 0x7f52f00c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52eeffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52f00c8fa0 0x7f52f014f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52ef7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f015c160 0x7f52f014fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52eeffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52f00c8fa0 0x7f52f014f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49008/0 (socket says 172.21.15.23:49008) 2023-06-22T08:00:53.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52e7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f0152950 0x7f52f00c4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35632/0 (socket says 172.21.15.23:35632) 2023-06-22T08:00:53.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52eeffd700 1 -- 172.21.15.23:0/2996133308 learned_addr learned my addr 172.21.15.23:0/2996133308 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:53.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52e7fff700 1 -- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52f00c8fa0 msgr2=0x7f52f014f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52e7fff700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52f00c8fa0 0x7f52f014f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52e7fff700 1 -- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f015c160 msgr2=0x7f52f014fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52e7fff700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f015c160 0x7f52f014fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52e7fff700 1 -- 172.21.15.23:0/2996133308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e8009b30 con 0x7f52f0152950 2023-06-22T08:00:53.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52ef7fe700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f015c160 0x7f52f014fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:53.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52e7fff700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f0152950 0x7f52f00c4a00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f52d800c450 tx=0x7f52d800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:53.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52e77fe700 1 -- 172.21.15.23:0/2996133308 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d8005cb0 con 0x7f52f0152950 2023-06-22T08:00:53.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52e77fe700 1 -- 172.21.15.23:0/2996133308 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d8005e10 con 0x7f52f0152950 2023-06-22T08:00:53.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.106+0000 7f52effff700 1 -- 172.21.15.23:0/2996133308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f0155c90 con 0x7f52f0152950 2023-06-22T08:00:53.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.107+0000 7f52e77fe700 1 -- 172.21.15.23:0/2996133308 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d80072d0 con 0x7f52f0152950 2023-06-22T08:00:53.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.107+0000 7f52effff700 1 -- 172.21.15.23:0/2996133308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f014f170 con 0x7f52f0152950 2023-06-22T08:00:53.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.108+0000 7f52effff700 1 -- 172.21.15.23:0/2996133308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f015a830 con 0x7f52f0152950 2023-06-22T08:00:53.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.109+0000 7f52e77fe700 1 -- 172.21.15.23:0/2996133308 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f52d8007430 con 0x7f52f0152950 2023-06-22T08:00:53.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.109+0000 7f52e77fe700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f52d006ace0 0x7f52d006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.110+0000 7f52e77fe700 1 -- 172.21.15.23:0/2996133308 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f52d8088940 con 0x7f52f0152950 2023-06-22T08:00:53.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.110+0000 7f52eeffd700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f52d006ace0 0x7f52d006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.111+0000 7f52eeffd700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f52d006ace0 0x7f52d006d190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009ca0 tx=0x7f52e0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:53.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.114+0000 7f52e77fe700 1 -- 172.21.15.23:0/2996133308 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f52f015a830 con 0x7f52f0152950 2023-06-22T08:00:53.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.322+0000 7f52effff700 1 -- 172.21.15.23:0/2996133308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f52f015a830 con 0x7f52f0152950 2023-06-22T08:00:53.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.322+0000 7f52e77fe700 1 -- 172.21.15.23:0/2996133308 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f52d8017020 con 0x7f52f0152950 2023-06-22T08:00:53.326 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-06-22T08:00:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 -- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f52d006ace0 msgr2=0x7f52d006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f52d006ace0 0x7f52d006d190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009ca0 tx=0x7f52e0009210 comp rx=0 tx=0).stop 2023-06-22T08:00:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 -- 172.21.15.23:0/2996133308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f0152950 msgr2=0x7f52f00c4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f0152950 0x7f52f00c4a00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f52d800c450 tx=0x7f52d800c760 comp rx=0 tx=0).stop 2023-06-22T08:00:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 -- 172.21.15.23:0/2996133308 shutdown_connections 2023-06-22T08:00:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f52d006ace0 0x7f52d006d190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52f015c160 0x7f52f014fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52f00c8fa0 0x7f52f014f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 --2- 172.21.15.23:0/2996133308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52f0152950 0x7f52f00c4a00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 -- 172.21.15.23:0/2996133308 >> 172.21.15.23:0/2996133308 conn(0x7f52f00ce620 msgr2=0x7f52f00cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 -- 172.21.15.23:0/2996133308 shutdown_connections 2023-06-22T08:00:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.325+0000 7f52effff700 1 -- 172.21.15.23:0/2996133308 wait complete. 2023-06-22T08:00:53.345 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-22T08:00:53.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.544+0000 7fd5029ff700 1 Processor -- start 2023-06-22T08:00:53.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.544+0000 7fd5029ff700 1 -- start start 2023-06-22T08:00:53.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.545+0000 7fd5029ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4fc152ba0 0x7fd4fc1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.545+0000 7fd5029ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4fc0c9260 0x7fd4fc0c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.545+0000 7fd5029ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc159930 con 0x7fd4fc0c4650 2023-06-22T08:00:53.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.545+0000 7fd5029ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4fc0c9aa0 con 0x7fd4fc152ba0 2023-06-22T08:00:53.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.545+0000 7fd5029ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc06aed0 con 0x7fd4fc0c9260 2023-06-22T08:00:53.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.546+0000 7fd4fbfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4fc152ba0 0x7fd4fc1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.546+0000 7fd4fb7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4fc0c9260 0x7fd4fc0c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.546+0000 7fd500f9c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd4fc0c4650 0x7fd4fc0c9ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:57772/0 (socket says 172.21.15.23:57772) 2023-06-22T08:00:53.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.546+0000 7fd500f9c700 1 -- 172.21.15.23:0/375045009 learned_addr learned my addr 172.21.15.23:0/375045009 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:53.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.546+0000 7fd4faffd700 1 -- 172.21.15.23:0/375045009 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976730043 0 0) 0x7fd4f4001490 con 0x7fd4fc0c4650 2023-06-22T08:00:53.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.546+0000 7fd4fbfff700 1 -- 172.21.15.23:0/375045009 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4fc0c9260 msgr2=0x7fd4fc0c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:53.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.546+0000 7fd4fbfff700 1 --2- 172.21.15.23:0/375045009 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4fc0c9260 0x7fd4fc0c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.546+0000 7fd4fbfff700 1 -- 172.21.15.23:0/375045009 >> v1:172.21.15.67:6789/0 conn(0x7fd4fc0c4650 legacy=0x7fd4fc0c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.546+0000 7fd4fbfff700 1 -- 172.21.15.23:0/375045009 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc06b1b0 con 0x7fd4fc152ba0 2023-06-22T08:00:53.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.547+0000 7fd4fbfff700 1 --2- 172.21.15.23:0/375045009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc152ba0 0x7fd4fc1593f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009840 tx=0x7fd4ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e42f9355ce291369 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:53.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.547+0000 7fd4faffd700 1 -- 172.21.15.23:0/375045009 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec011830 con 0x7fd4fc152ba0 2023-06-22T08:00:53.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.547+0000 7fd4faffd700 1 -- 172.21.15.23:0/375045009 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ec011990 con 0x7fd4fc152ba0 2023-06-22T08:00:53.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.547+0000 7fd4faffd700 1 -- 172.21.15.23:0/375045009 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec011d20 con 0x7fd4fc152ba0 2023-06-22T08:00:53.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.547+0000 7fd5029ff700 1 -- 172.21.15.23:0/375045009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc152ba0 msgr2=0x7fd4fc1593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.547+0000 7fd5029ff700 1 --2- 172.21.15.23:0/375045009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc152ba0 0x7fd4fc1593f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec009840 tx=0x7fd4ec009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:53.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.547+0000 7fd5029ff700 1 -- 172.21.15.23:0/375045009 shutdown_connections 2023-06-22T08:00:53.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.547+0000 7fd5029ff700 1 --2- 172.21.15.23:0/375045009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc152ba0 0x7fd4fc1593f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.547+0000 7fd5029ff700 1 --2- 172.21.15.23:0/375045009 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4fc0c9260 0x7fd4fc0c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.547+0000 7fd5029ff700 1 -- 172.21.15.23:0/375045009 >> 172.21.15.23:0/375045009 conn(0x7fd4fc065b90 msgr2=0x7fd4fc065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.548+0000 7fd5029ff700 1 -- 172.21.15.23:0/375045009 shutdown_connections 2023-06-22T08:00:53.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.548+0000 7fd5029ff700 1 -- 172.21.15.23:0/375045009 wait complete. 2023-06-22T08:00:53.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.549+0000 7fd5029ff700 1 Processor -- start 2023-06-22T08:00:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.549+0000 7fd5029ff700 1 -- start start 2023-06-22T08:00:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.550+0000 7fd5029ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4fc0c4650 0x7fd4fc147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.550+0000 7fd5029ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc0c9260 0x7fd4fc14d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.550+0000 7fd5029ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4fc152ba0 0x7fd4fc14dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.550+0000 7fd5029ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc1ef6f0 con 0x7fd4fc0c4650 2023-06-22T08:00:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.550+0000 7fd5029ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc0b3310 con 0x7fd4fc0c9260 2023-06-22T08:00:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.550+0000 7fd5029ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc0b35f0 con 0x7fd4fc152ba0 2023-06-22T08:00:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4fb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc0c9260 0x7fd4fc14d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4fbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4fc0c4650 0x7fd4fc147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4fb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc0c9260 0x7fd4fc14d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51354/0 (socket says 172.21.15.23:51354) 2023-06-22T08:00:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4fb7fe700 1 -- 172.21.15.23:0/182659651 learned_addr learned my addr 172.21.15.23:0/182659651 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd500f9c700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4fc152ba0 0x7fd4fc14dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4fb7fe700 1 -- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4fc152ba0 msgr2=0x7fd4fc14dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4fb7fe700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4fc152ba0 0x7fd4fc14dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4fb7fe700 1 -- 172.21.15.23:0/182659651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4fc0c4650 msgr2=0x7fd4fc147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4fb7fe700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4fc0c4650 0x7fd4fc147cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4fb7fe700 1 -- 172.21.15.23:0/182659651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec0094a0 con 0x7fd4fc0c9260 2023-06-22T08:00:53.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4fb7fe700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc0c9260 0x7fd4fc14d660 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000e9b0 tx=0x7fd4f000ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:53.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.551+0000 7fd4f8ff9700 1 -- 172.21.15.23:0/182659651 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0009710 con 0x7fd4fc0c9260 2023-06-22T08:00:53.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.552+0000 7fd4f8ff9700 1 -- 172.21.15.23:0/182659651 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f000ee50 con 0x7fd4fc0c9260 2023-06-22T08:00:53.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.552+0000 7fd5029ff700 1 -- 172.21.15.23:0/182659651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4fc0b38d0 con 0x7fd4fc0c9260 2023-06-22T08:00:53.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.552+0000 7fd4f8ff9700 1 -- 172.21.15.23:0/182659651 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f000ca50 con 0x7fd4fc0c9260 2023-06-22T08:00:53.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.552+0000 7fd5029ff700 1 -- 172.21.15.23:0/182659651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4fc0b3e90 con 0x7fd4fc0c9260 2023-06-22T08:00:53.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.553+0000 7fd5029ff700 1 -- 172.21.15.23:0/182659651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4fc0cbed0 con 0x7fd4fc0c9260 2023-06-22T08:00:53.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.554+0000 7fd4f8ff9700 1 -- 172.21.15.23:0/182659651 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd4f000cbb0 con 0x7fd4fc0c9260 2023-06-22T08:00:53.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.555+0000 7fd4f8ff9700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4e806ace0 0x7fd4e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.555+0000 7fd4f8ff9700 1 -- 172.21.15.23:0/182659651 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd4f0089b80 con 0x7fd4fc0c9260 2023-06-22T08:00:53.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.555+0000 7fd4fbfff700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4e806ace0 0x7fd4e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.555+0000 7fd4fbfff700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4e806ace0 0x7fd4e806d190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec0090b0 tx=0x7fd4ec01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:53.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.559+0000 7fd4f8ff9700 1 -- 172.21.15.23:0/182659651 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd4f0058330 con 0x7fd4fc0c9260 2023-06-22T08:00:53.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.767+0000 7fd5029ff700 1 -- 172.21.15.23:0/182659651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd4fc04ea20 con 0x7fd4fc0c9260 2023-06-22T08:00:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.768+0000 7fd4f8ff9700 1 -- 172.21.15.23:0/182659651 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd4f000ce60 con 0x7fd4fc0c9260 2023-06-22T08:00:53.771 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:53.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 -- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4e806ace0 msgr2=0x7fd4e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4e806ace0 0x7fd4e806d190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec0090b0 tx=0x7fd4ec01c040 comp rx=0 tx=0).stop 2023-06-22T08:00:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 -- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc0c9260 msgr2=0x7fd4fc14d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc0c9260 0x7fd4fc14d660 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000e9b0 tx=0x7fd4f000ecc0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 -- 172.21.15.23:0/182659651 shutdown_connections 2023-06-22T08:00:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4fc152ba0 0x7fd4fc14dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4fc0c4650 0x7fd4fc147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4e806ace0 0x7fd4e806d190 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 --2- 172.21.15.23:0/182659651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4fc0c9260 0x7fd4fc14d660 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 -- 172.21.15.23:0/182659651 >> 172.21.15.23:0/182659651 conn(0x7fd4fc065b90 msgr2=0x7fd4fc156580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:53.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 -- 172.21.15.23:0/182659651 shutdown_connections 2023-06-22T08:00:53.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.770+0000 7fd5029ff700 1 -- 172.21.15.23:0/182659651 wait complete. 2023-06-22T08:00:53.791 INFO:tasks.mds_thrash:mds thrasher using random seed: 1687420853 2023-06-22T08:00:53.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:53.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.987+0000 7f8c259ff700 1 Processor -- start 2023-06-22T08:00:53.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.988+0000 7f8c259ff700 1 -- start start 2023-06-22T08:00:53.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.989+0000 7f8c259ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c20152ce0 0x7f8c200cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.989+0000 7f8c259ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8c200c40b0 0x7f8c200c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.989+0000 7f8c259ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2015c370 con 0x7f8c200c9fd0 2023-06-22T08:00:53.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.989+0000 7f8c259ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c201530b0 con 0x7f8c200c40b0 2023-06-22T08:00:53.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.989+0000 7f8c259ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c200cf6f0 con 0x7f8c20152ce0 2023-06-22T08:00:53.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.989+0000 7f8c1f7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8c200c40b0 0x7f8c200c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.989+0000 7f8c1f7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8c200c40b0 0x7f8c200c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35654/0 (socket says 172.21.15.23:35654) 2023-06-22T08:00:53.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.989+0000 7f8c1effd700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c20152ce0 0x7f8c200cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:53.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.989+0000 7f8c1f7fe700 1 -- 172.21.15.23:0/1577504174 learned_addr learned my addr 172.21.15.23:0/1577504174 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:53.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.990+0000 7f8c1effd700 1 -- 172.21.15.23:0/1577504174 >> v2:172.21.15.23:3300/0 conn(0x7f8c200c40b0 msgr2=0x7f8c200c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:53.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.990+0000 7f8c1effd700 1 --2- 172.21.15.23:0/1577504174 >> v2:172.21.15.23:3300/0 conn(0x7f8c200c40b0 0x7f8c200c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.990+0000 7f8c1effd700 1 -- 172.21.15.23:0/1577504174 >> v1:172.21.15.23:6789/0 conn(0x7f8c200c9fd0 legacy=0x7f8c2015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.990+0000 7f8c1effd700 1 -- 172.21.15.23:0/1577504174 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2015bda0 con 0x7f8c20152ce0 2023-06-22T08:00:53.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.990+0000 7f8c1f7fe700 1 --2- 172.21.15.23:0/1577504174 >> v2:172.21.15.23:3300/0 conn(0x7f8c200c40b0 0x7f8c200c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:53.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.990+0000 7f8c1effd700 1 --2- 172.21.15.23:0/1577504174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c20152ce0 0x7f8c200cf1b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8c2008ba10 tx=0x7f8c10009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32e27eb01c4d665e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:53.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.990+0000 7f8c1dffb700 1 -- 172.21.15.23:0/1577504174 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c10011830 con 0x7f8c20152ce0 2023-06-22T08:00:53.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.990+0000 7f8c1dffb700 1 -- 172.21.15.23:0/1577504174 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c10011990 con 0x7f8c20152ce0 2023-06-22T08:00:53.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.990+0000 7f8c1dffb700 1 -- 172.21.15.23:0/1577504174 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c10011d20 con 0x7f8c20152ce0 2023-06-22T08:00:53.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.991+0000 7f8c259ff700 1 -- 172.21.15.23:0/1577504174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c20152ce0 msgr2=0x7f8c200cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:53.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.991+0000 7f8c259ff700 1 --2- 172.21.15.23:0/1577504174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c20152ce0 0x7f8c200cf1b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8c2008ba10 tx=0x7f8c10009b20 comp rx=0 tx=0).stop 2023-06-22T08:00:53.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.991+0000 7f8c259ff700 1 -- 172.21.15.23:0/1577504174 shutdown_connections 2023-06-22T08:00:53.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.991+0000 7f8c259ff700 1 --2- 172.21.15.23:0/1577504174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c20152ce0 0x7f8c200cf1b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.991+0000 7f8c259ff700 1 --2- 172.21.15.23:0/1577504174 >> v2:172.21.15.23:3300/0 conn(0x7f8c200c40b0 0x7f8c200c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:53.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.991+0000 7f8c259ff700 1 -- 172.21.15.23:0/1577504174 >> 172.21.15.23:0/1577504174 conn(0x7f8c2006ae30 msgr2=0x7f8c2006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:53.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.991+0000 7f8c259ff700 1 -- 172.21.15.23:0/1577504174 shutdown_connections 2023-06-22T08:00:53.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.991+0000 7f8c259ff700 1 -- 172.21.15.23:0/1577504174 wait complete. 2023-06-22T08:00:53.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.993+0000 7f8c259ff700 1 Processor -- start 2023-06-22T08:00:53.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.993+0000 7f8c259ff700 1 -- start start 2023-06-22T08:00:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c259ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c200c40b0 0x7f8c200b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c259ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c200c9fd0 0x7f8c2014dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c259ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c20152ce0 0x7f8c200c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c259ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c200c8070 con 0x7f8c20152ce0 2023-06-22T08:00:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c259ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c200b3e40 con 0x7f8c200c9fd0 2023-06-22T08:00:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c259ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c200b46a0 con 0x7f8c200c40b0 2023-06-22T08:00:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c1f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c20152ce0 0x7f8c200c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c1f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c20152ce0 0x7f8c200c64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35664/0 (socket says 172.21.15.23:35664) 2023-06-22T08:00:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c1f7fe700 1 -- 172.21.15.23:0/1940626087 learned_addr learned my addr 172.21.15.23:0/1940626087 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c1effd700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c200c40b0 0x7f8c200b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c1e7fc700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c200c9fd0 0x7f8c2014dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c1f7fe700 1 -- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c200c40b0 msgr2=0x7f8c200b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c1f7fe700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c200c40b0 0x7f8c200b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c1f7fe700 1 -- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c200c9fd0 msgr2=0x7f8c2014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c1f7fe700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c200c9fd0 0x7f8c2014dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.994+0000 7f8c1f7fe700 1 -- 172.21.15.23:0/1940626087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c18009b30 con 0x7f8c20152ce0 2023-06-22T08:00:54.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.995+0000 7f8c1f7fe700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c20152ce0 0x7f8c200c64b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800efc0 tx=0x7f8c1800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:54.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.995+0000 7f8c07fff700 1 -- 172.21.15.23:0/1940626087 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18018070 con 0x7f8c20152ce0 2023-06-22T08:00:54.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.995+0000 7f8c07fff700 1 -- 172.21.15.23:0/1940626087 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1800f040 con 0x7f8c20152ce0 2023-06-22T08:00:54.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.995+0000 7f8c259ff700 1 -- 172.21.15.23:0/1940626087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c100094a0 con 0x7f8c20152ce0 2023-06-22T08:00:54.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.995+0000 7f8c07fff700 1 -- 172.21.15.23:0/1940626087 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18003c20 con 0x7f8c20152ce0 2023-06-22T08:00:54.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.995+0000 7f8c259ff700 1 -- 172.21.15.23:0/1940626087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c200b43c0 con 0x7f8c20152ce0 2023-06-22T08:00:54.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.996+0000 7f8c259ff700 1 -- 172.21.15.23:0/1940626087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2004ea20 con 0x7f8c20152ce0 2023-06-22T08:00:54.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.997+0000 7f8c07fff700 1 -- 172.21.15.23:0/1940626087 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8c1800c880 con 0x7f8c20152ce0 2023-06-22T08:00:54.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.998+0000 7f8c07fff700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c0c06ad10 0x7f8c0c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.998+0000 7f8c07fff700 1 -- 172.21.15.23:0/1940626087 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f8c18089900 con 0x7f8c20152ce0 2023-06-22T08:00:54.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.998+0000 7f8c1effd700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c0c06ad10 0x7f8c0c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:53.999+0000 7f8c1effd700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c0c06ad10 0x7f8c0c06d1c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c10000c00 tx=0x7f8c1001c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:54.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.002+0000 7f8c07fff700 1 -- 172.21.15.23:0/1940626087 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8c180546b0 con 0x7f8c20152ce0 2023-06-22T08:00:54.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.210+0000 7f8c259ff700 1 -- 172.21.15.23:0/1940626087 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c200c9db0 con 0x7f8c20152ce0 2023-06-22T08:00:54.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.211+0000 7f8c07fff700 1 -- 172.21.15.23:0/1940626087 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f8c18057cd0 con 0x7f8c20152ce0 2023-06-22T08:00:54.214 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:54.215 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:54.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 -- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c0c06ad10 msgr2=0x7f8c0c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c0c06ad10 0x7f8c0c06d1c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8c10000c00 tx=0x7f8c1001c040 comp rx=0 tx=0).stop 2023-06-22T08:00:54.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 -- 172.21.15.23:0/1940626087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c20152ce0 msgr2=0x7f8c200c64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c20152ce0 0x7f8c200c64b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800efc0 tx=0x7f8c1800c3a0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 -- 172.21.15.23:0/1940626087 shutdown_connections 2023-06-22T08:00:54.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c20152ce0 0x7f8c200c64b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c200c40b0 0x7f8c200b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c0c06ad10 0x7f8c0c06d1c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 --2- 172.21.15.23:0/1940626087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c200c9fd0 0x7f8c2014dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 -- 172.21.15.23:0/1940626087 >> 172.21.15.23:0/1940626087 conn(0x7f8c2006ae30 msgr2=0x7f8c200cb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.213+0000 7f8c259ff700 1 -- 172.21.15.23:0/1940626087 shutdown_connections 2023-06-22T08:00:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.214+0000 7f8c259ff700 1 -- 172.21.15.23:0/1940626087 wait complete. 2023-06-22T08:00:54.234 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T07:59:40.617747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-22T07:59:48.190528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-22T07:59:53.576494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-22T07:59:54.105472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-22T07:59:57.198896+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-22T08:00:54.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T08:00:54.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.434+0000 7fc8cc3ff700 1 Processor -- start 2023-06-22T08:00:54.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.435+0000 7fc8cc3ff700 1 -- start start 2023-06-22T08:00:54.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.436+0000 7fc8cc3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc8c4152ba0 0x7fc8c41593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.436+0000 7fc8cc3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8c40c9260 0x7fc8c40c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8cc3ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8c4159930 con 0x7fc8c40c4650 2023-06-22T08:00:54.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8cc3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c40c9aa0 con 0x7fc8c4152ba0 2023-06-22T08:00:54.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8cc3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c406aed0 con 0x7fc8c40c9260 2023-06-22T08:00:54.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8ca19b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc8c4152ba0 0x7fc8c41593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8ca99c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc8c40c4650 0x7fc8c40c9ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34952/0 (socket says 172.21.15.23:34952) 2023-06-22T08:00:54.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8ca99c700 1 -- 172.21.15.23:0/3251694964 learned_addr learned my addr 172.21.15.23:0/3251694964 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:54.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8c999a700 1 --2- 172.21.15.23:0/3251694964 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8c40c9260 0x7fc8c40c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8c9199700 1 -- 172.21.15.23:0/3251694964 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343749133 0 0) 0x7fc8bc001490 con 0x7fc8c40c4650 2023-06-22T08:00:54.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8c9199700 1 -- 172.21.15.23:0/3251694964 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b00035b0 con 0x7fc8c40c4650 2023-06-22T08:00:54.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8ca19b700 1 -- 172.21.15.23:0/3251694964 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8c40c9260 msgr2=0x7fc8c40c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:54.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8ca19b700 1 --2- 172.21.15.23:0/3251694964 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8c40c9260 0x7fc8c40c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8ca19b700 1 -- 172.21.15.23:0/3251694964 >> v1:172.21.15.23:6789/0 conn(0x7fc8c40c4650 legacy=0x7fc8c40c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.437+0000 7fc8ca19b700 1 -- 172.21.15.23:0/3251694964 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c406b1b0 con 0x7fc8c4152ba0 2023-06-22T08:00:54.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.438+0000 7fc8ca19b700 1 --2- 172.21.15.23:0/3251694964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c4152ba0 0x7fc8c41593f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009840 tx=0x7fc8c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54d0e74f6af67b1 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:54.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.438+0000 7fc8c9199700 1 -- 172.21.15.23:0/3251694964 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0011830 con 0x7fc8c4152ba0 2023-06-22T08:00:54.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.438+0000 7fc8c9199700 1 -- 172.21.15.23:0/3251694964 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c0011990 con 0x7fc8c4152ba0 2023-06-22T08:00:54.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.438+0000 7fc8c9199700 1 -- 172.21.15.23:0/3251694964 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0011d20 con 0x7fc8c4152ba0 2023-06-22T08:00:54.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.438+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/3251694964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c4152ba0 msgr2=0x7fc8c41593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.438+0000 7fc8cc3ff700 1 --2- 172.21.15.23:0/3251694964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c4152ba0 0x7fc8c41593f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009840 tx=0x7fc8c0009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:54.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.438+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/3251694964 shutdown_connections 2023-06-22T08:00:54.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.438+0000 7fc8cc3ff700 1 --2- 172.21.15.23:0/3251694964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c4152ba0 0x7fc8c41593f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.438+0000 7fc8cc3ff700 1 --2- 172.21.15.23:0/3251694964 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8c40c9260 0x7fc8c40c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.439+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/3251694964 >> 172.21.15.23:0/3251694964 conn(0x7fc8c4065b90 msgr2=0x7fc8c4065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:54.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.439+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/3251694964 shutdown_connections 2023-06-22T08:00:54.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.439+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/3251694964 wait complete. 2023-06-22T08:00:54.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.441+0000 7fc8cc3ff700 1 Processor -- start 2023-06-22T08:00:54.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.441+0000 7fc8cc3ff700 1 -- start start 2023-06-22T08:00:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.442+0000 7fc8cc3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c40c4650 0x7fc8c40c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.442+0000 7fc8cc3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8c40c9260 0x7fc8c4154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.442+0000 7fc8cc3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4152ba0 0x7fc8c4154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.442+0000 7fc8cc3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c41532e0 con 0x7fc8c40c4650 2023-06-22T08:00:54.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.442+0000 7fc8cc3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c4154b80 con 0x7fc8c4152ba0 2023-06-22T08:00:54.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.442+0000 7fc8cc3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c4154df0 con 0x7fc8c40c9260 2023-06-22T08:00:54.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.443+0000 7fc8ca19b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c40c4650 0x7fc8c40c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.443+0000 7fc8ca19b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c40c4650 0x7fc8c40c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35674/0 (socket says 172.21.15.23:35674) 2023-06-22T08:00:54.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.443+0000 7fc8ca19b700 1 -- 172.21.15.23:0/804688887 learned_addr learned my addr 172.21.15.23:0/804688887 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:54.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.443+0000 7fc8ca99c700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4152ba0 0x7fc8c4154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.443+0000 7fc8c999a700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8c40c9260 0x7fc8c4154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.443+0000 7fc8ca19b700 1 -- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8c40c9260 msgr2=0x7fc8c4154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.443+0000 7fc8ca19b700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8c40c9260 0x7fc8c4154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.443+0000 7fc8ca19b700 1 -- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4152ba0 msgr2=0x7fc8c4154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.443+0000 7fc8ca19b700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4152ba0 0x7fc8c4154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.443+0000 7fc8ca19b700 1 -- 172.21.15.23:0/804688887 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c00094a0 con 0x7fc8c40c4650 2023-06-22T08:00:54.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.444+0000 7fc8ca19b700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c40c4650 0x7fc8c40c5c00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009fd0 tx=0x7fc8c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:54.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.444+0000 7fc8baffd700 1 -- 172.21.15.23:0/804688887 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0004590 con 0x7fc8c40c4650 2023-06-22T08:00:54.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.444+0000 7fc8baffd700 1 -- 172.21.15.23:0/804688887 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c001e070 con 0x7fc8c40c4650 2023-06-22T08:00:54.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.444+0000 7fc8baffd700 1 -- 172.21.15.23:0/804688887 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c00038e0 con 0x7fc8c40c4650 2023-06-22T08:00:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.444+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/804688887 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c41550d0 con 0x7fc8c40c4650 2023-06-22T08:00:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.444+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/804688887 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c4153810 con 0x7fc8c40c4650 2023-06-22T08:00:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.445+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/804688887 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c40cbed0 con 0x7fc8c40c4650 2023-06-22T08:00:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.446+0000 7fc8baffd700 1 -- 172.21.15.23:0/804688887 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc8c0003a40 con 0x7fc8c40c4650 2023-06-22T08:00:54.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.447+0000 7fc8baffd700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8a806ace0 0x7fc8a806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.447+0000 7fc8baffd700 1 -- 172.21.15.23:0/804688887 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc8c008a650 con 0x7fc8c40c4650 2023-06-22T08:00:54.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.447+0000 7fc8c999a700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8a806ace0 0x7fc8a806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.448+0000 7fc8c999a700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8a806ace0 0x7fc8a806d190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40c8e10 tx=0x7fc8b4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:54.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.452+0000 7fc8baffd700 1 -- 172.21.15.23:0/804688887 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc8c0055380 con 0x7fc8c40c4650 2023-06-22T08:00:54.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.662+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/804688887 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8c414ffa0 con 0x7fc8c40c4650 2023-06-22T08:00:54.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.662+0000 7fc8baffd700 1 -- 172.21.15.23:0/804688887 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8c00589a0 con 0x7fc8c40c4650 2023-06-22T08:00:54.666 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-06-22T08:00:54.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.664+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8a806ace0 msgr2=0x7fc8a806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.664+0000 7fc8cc3ff700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8a806ace0 0x7fc8a806d190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40c8e10 tx=0x7fc8b4009210 comp rx=0 tx=0).stop 2023-06-22T08:00:54.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.665+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/804688887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c40c4650 msgr2=0x7fc8c40c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.665+0000 7fc8cc3ff700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c40c4650 0x7fc8c40c5c00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0009fd0 tx=0x7fc8c0004750 comp rx=0 tx=0).stop 2023-06-22T08:00:54.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.665+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/804688887 shutdown_connections 2023-06-22T08:00:54.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.665+0000 7fc8cc3ff700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8a806ace0 0x7fc8a806d190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.665+0000 7fc8cc3ff700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8c4152ba0 0x7fc8c4154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.665+0000 7fc8cc3ff700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8c40c4650 0x7fc8c40c5c00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.665+0000 7fc8cc3ff700 1 --2- 172.21.15.23:0/804688887 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8c40c9260 0x7fc8c4154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.665+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/804688887 >> 172.21.15.23:0/804688887 conn(0x7fc8c4065b90 msgr2=0x7fc8c4150e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:54.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.665+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/804688887 shutdown_connections 2023-06-22T08:00:54.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.665+0000 7fc8cc3ff700 1 -- 172.21.15.23:0/804688887 wait complete. 2023-06-22T08:00:54.686 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T08:00:54.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.884+0000 7f52993ff700 1 Processor -- start 2023-06-22T08:00:54.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.884+0000 7f52993ff700 1 -- start start 2023-06-22T08:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.885+0000 7f52993ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f52941593f0 0x7f52940c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.885+0000 7f52993ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52940c4650 0x7f52940c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.885+0000 7f52993ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52940c9aa0 con 0x7f5294152ba0 2023-06-22T08:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f52993ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52940c4a60 con 0x7f52941593f0 2023-06-22T08:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f52993ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f529406aed0 con 0x7f52940c4650 2023-06-22T08:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f52927fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f52941593f0 0x7f52940c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f52937fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52940c4650 0x7f52940c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f5292ffd700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5294152ba0 0x7f52940c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:57782/0 (socket says 172.21.15.23:57782) 2023-06-22T08:00:54.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f5292ffd700 1 -- 172.21.15.23:0/2526777036 learned_addr learned my addr 172.21.15.23:0/2526777036 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:54.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f5291ffb700 1 -- 172.21.15.23:0/2526777036 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443353690 0 0) 0x7f52940c9aa0 con 0x7f5294152ba0 2023-06-22T08:00:54.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f5291ffb700 1 -- 172.21.15.23:0/2526777036 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f527c0035b0 con 0x7f5294152ba0 2023-06-22T08:00:54.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f52937fe700 1 -- 172.21.15.23:0/2526777036 >> v2:172.21.15.23:3300/0 conn(0x7f52941593f0 msgr2=0x7f52940c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:54.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f52937fe700 1 --2- 172.21.15.23:0/2526777036 >> v2:172.21.15.23:3300/0 conn(0x7f52941593f0 0x7f52940c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f52937fe700 1 -- 172.21.15.23:0/2526777036 >> v1:172.21.15.67:6789/0 conn(0x7f5294152ba0 legacy=0x7f52940c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f52937fe700 1 -- 172.21.15.23:0/2526777036 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529406b1b0 con 0x7f52940c4650 2023-06-22T08:00:54.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.886+0000 7f52927fc700 1 --2- 172.21.15.23:0/2526777036 >> v2:172.21.15.23:3300/0 conn(0x7f52941593f0 0x7f52940c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:54.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f52937fe700 1 --2- 172.21.15.23:0/2526777036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52940c4650 0x7f52940c9ee0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f529408b390 tx=0x7f528c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ace9848d7a7ba07 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:54.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f5291ffb700 1 -- 172.21.15.23:0/2526777036 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c00e070 con 0x7f52940c4650 2023-06-22T08:00:54.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f5291ffb700 1 -- 172.21.15.23:0/2526777036 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c00bb20 con 0x7f52940c4650 2023-06-22T08:00:54.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f5291ffb700 1 -- 172.21.15.23:0/2526777036 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c012430 con 0x7f52940c4650 2023-06-22T08:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f52993ff700 1 -- 172.21.15.23:0/2526777036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52940c4650 msgr2=0x7f52940c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f52993ff700 1 --2- 172.21.15.23:0/2526777036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52940c4650 0x7f52940c9ee0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f529408b390 tx=0x7f528c00b330 comp rx=0 tx=0).stop 2023-06-22T08:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f52993ff700 1 -- 172.21.15.23:0/2526777036 shutdown_connections 2023-06-22T08:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f52993ff700 1 --2- 172.21.15.23:0/2526777036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52940c4650 0x7f52940c9ee0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f52993ff700 1 --2- 172.21.15.23:0/2526777036 >> v2:172.21.15.23:3300/0 conn(0x7f52941593f0 0x7f52940c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f52993ff700 1 -- 172.21.15.23:0/2526777036 >> 172.21.15.23:0/2526777036 conn(0x7f5294065b90 msgr2=0x7f5294065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.887+0000 7f52993ff700 1 -- 172.21.15.23:0/2526777036 shutdown_connections 2023-06-22T08:00:54.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.888+0000 7f52993ff700 1 -- 172.21.15.23:0/2526777036 wait complete. 2023-06-22T08:00:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.889+0000 7f52993ff700 1 Processor -- start 2023-06-22T08:00:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.889+0000 7f52993ff700 1 -- start start 2023-06-22T08:00:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.890+0000 7f52993ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52940c4650 0x7f52941eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.890+0000 7f52993ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294152ba0 0x7f52940ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.890+0000 7f52993ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52941593f0 0x7f52940ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.890+0000 7f52993ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52940c8b50 con 0x7f52941593f0 2023-06-22T08:00:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.890+0000 7f52993ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52940c8590 con 0x7f5294152ba0 2023-06-22T08:00:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.890+0000 7f52993ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52940c8870 con 0x7f52940c4650 2023-06-22T08:00:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52937fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52941593f0 0x7f52940ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52927fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294152ba0 0x7f52940ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f5292ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52940c4650 0x7f52941eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52927fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294152ba0 0x7f52940ce1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51382/0 (socket says 172.21.15.23:51382) 2023-06-22T08:00:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52927fc700 1 -- 172.21.15.23:0/3915542505 learned_addr learned my addr 172.21.15.23:0/3915542505 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52927fc700 1 -- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52940c4650 msgr2=0x7f52941eead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52927fc700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52940c4650 0x7f52941eead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52927fc700 1 -- 172.21.15.23:0/3915542505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52941593f0 msgr2=0x7f52940ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52927fc700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52941593f0 0x7f52940ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:54.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52927fc700 1 -- 172.21.15.23:0/3915542505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52880094a0 con 0x7f5294152ba0 2023-06-22T08:00:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52937fe700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52941593f0 0x7f52940ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.891+0000 7f52927fc700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294152ba0 0x7f52940ce1a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f528800ebe0 tx=0x7f528800eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.892+0000 7f527b7fe700 1 -- 172.21.15.23:0/3915542505 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528800c850 con 0x7f5294152ba0 2023-06-22T08:00:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.892+0000 7f527b7fe700 1 -- 172.21.15.23:0/3915542505 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528800c9b0 con 0x7f5294152ba0 2023-06-22T08:00:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.892+0000 7f52993ff700 1 -- 172.21.15.23:0/3915542505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528c009a70 con 0x7f5294152ba0 2023-06-22T08:00:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.892+0000 7f527b7fe700 1 -- 172.21.15.23:0/3915542505 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528800cd40 con 0x7f5294152ba0 2023-06-22T08:00:54.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.892+0000 7f52993ff700 1 -- 172.21.15.23:0/3915542505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52940cd120 con 0x7f5294152ba0 2023-06-22T08:00:54.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.893+0000 7f52993ff700 1 -- 172.21.15.23:0/3915542505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52940cbed0 con 0x7f5294152ba0 2023-06-22T08:00:54.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.894+0000 7f527b7fe700 1 -- 172.21.15.23:0/3915542505 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5288005a20 con 0x7f5294152ba0 2023-06-22T08:00:54.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.895+0000 7f527b7fe700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f528006ad30 0x7f528006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.895+0000 7f527b7fe700 1 -- 172.21.15.23:0/3915542505 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f5288089b60 con 0x7f5294152ba0 2023-06-22T08:00:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.895+0000 7f5292ffd700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f528006ad30 0x7f528006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:54.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.896+0000 7f5292ffd700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f528006ad30 0x7f528006d1e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5284007fd0 tx=0x7f5284007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:54.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:54.899+0000 7f527b7fe700 1 -- 172.21.15.23:0/3915542505 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5288054990 con 0x7f5294152ba0 2023-06-22T08:00:55.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.108+0000 7f52993ff700 1 -- 172.21.15.23:0/3915542505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f529404ea20 con 0x7f5294152ba0 2023-06-22T08:00:55.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.109+0000 7f527b7fe700 1 -- 172.21.15.23:0/3915542505 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5288017020 con 0x7f5294152ba0 2023-06-22T08:00:55.112 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2023-06-22T08:00:55.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 -- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f528006ad30 msgr2=0x7f528006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f528006ad30 0x7f528006d1e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5284007fd0 tx=0x7f5284007dc0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 -- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294152ba0 msgr2=0x7f52940ce1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294152ba0 0x7f52940ce1a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f528800ebe0 tx=0x7f528800eef0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 -- 172.21.15.23:0/3915542505 shutdown_connections 2023-06-22T08:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52941593f0 0x7f52940ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f52940c4650 0x7f52941eead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f528006ad30 0x7f528006d1e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 --2- 172.21.15.23:0/3915542505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294152ba0 0x7f52940ce1a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 -- 172.21.15.23:0/3915542505 >> 172.21.15.23:0/3915542505 conn(0x7f5294065b90 msgr2=0x7f5294151a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:55.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 -- 172.21.15.23:0/3915542505 shutdown_connections 2023-06-22T08:00:55.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.111+0000 7f52993ff700 1 -- 172.21.15.23:0/3915542505 wait complete. 2023-06-22T08:00:55.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-22T08:00:55.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.334+0000 7f432ffff700 1 Processor -- start 2023-06-22T08:00:55.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.334+0000 7f432ffff700 1 -- start start 2023-06-22T08:00:55.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.335+0000 7f432ffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f43281593f0 0x7f43280c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.335+0000 7f432ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f43280c4650 0x7f43280c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.335+0000 7f432ffff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43280c9aa0 con 0x7f4328152ba0 2023-06-22T08:00:55.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.335+0000 7f432ffff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43280c4a60 con 0x7f43280c4650 2023-06-22T08:00:55.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.335+0000 7f432ffff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432806aed0 con 0x7f43281593f0 2023-06-22T08:00:55.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.335+0000 7f432e59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f43280c4650 0x7f43280c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.335+0000 7f432e59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f43280c4650 0x7f43280c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35700/0 (socket says 172.21.15.23:35700) 2023-06-22T08:00:55.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.335+0000 7f432e59c700 1 -- 172.21.15.23:0/1158059642 learned_addr learned my addr 172.21.15.23:0/1158059642 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:55.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.335+0000 7f432d59a700 1 --2- 172.21.15.23:0/1158059642 >> v2:172.21.15.67:3300/0 conn(0x7f43281593f0 0x7f43280c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.336+0000 7f432e59c700 1 -- 172.21.15.23:0/1158059642 >> v2:172.21.15.67:3300/0 conn(0x7f43281593f0 msgr2=0x7f43280c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:55.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.336+0000 7f432e59c700 1 --2- 172.21.15.23:0/1158059642 >> v2:172.21.15.67:3300/0 conn(0x7f43281593f0 0x7f43280c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.336+0000 7f432e59c700 1 -- 172.21.15.23:0/1158059642 >> v1:172.21.15.67:6789/0 conn(0x7f4328152ba0 legacy=0x7f43280c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.336+0000 7f432e59c700 1 -- 172.21.15.23:0/1158059642 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432806b1b0 con 0x7f43280c4650 2023-06-22T08:00:55.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.336+0000 7f432e59c700 1 --2- 172.21.15.23:0/1158059642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43280c4650 0x7f43280c9ee0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f43280b1670 tx=0x7f432000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=755775f82ca803f1 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:55.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.336+0000 7f432cd99700 1 -- 172.21.15.23:0/1158059642 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432000e070 con 0x7f43280c4650 2023-06-22T08:00:55.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.336+0000 7f432cd99700 1 -- 172.21.15.23:0/1158059642 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432000b910 con 0x7f43280c4650 2023-06-22T08:00:55.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.336+0000 7f432cd99700 1 -- 172.21.15.23:0/1158059642 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432000bcc0 con 0x7f43280c4650 2023-06-22T08:00:55.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.337+0000 7f432ffff700 1 -- 172.21.15.23:0/1158059642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43280c4650 msgr2=0x7f43280c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.337+0000 7f432ffff700 1 --2- 172.21.15.23:0/1158059642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43280c4650 0x7f43280c9ee0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f43280b1670 tx=0x7f432000b330 comp rx=0 tx=0).stop 2023-06-22T08:00:55.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.337+0000 7f432ffff700 1 -- 172.21.15.23:0/1158059642 shutdown_connections 2023-06-22T08:00:55.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.337+0000 7f432ffff700 1 --2- 172.21.15.23:0/1158059642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43280c4650 0x7f43280c9ee0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.337+0000 7f432ffff700 1 --2- 172.21.15.23:0/1158059642 >> v2:172.21.15.67:3300/0 conn(0x7f43281593f0 0x7f43280c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.337+0000 7f432ffff700 1 -- 172.21.15.23:0/1158059642 >> 172.21.15.23:0/1158059642 conn(0x7f4328065b90 msgr2=0x7f4328065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:55.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.337+0000 7f432ffff700 1 -- 172.21.15.23:0/1158059642 shutdown_connections 2023-06-22T08:00:55.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.337+0000 7f432ffff700 1 -- 172.21.15.23:0/1158059642 wait complete. 2023-06-22T08:00:55.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.339+0000 7f432ffff700 1 Processor -- start 2023-06-22T08:00:55.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.339+0000 7f432ffff700 1 -- start start 2023-06-22T08:00:55.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43280c4650 0x7f43281dfd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4328152ba0 0x7f43281ef250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43281593f0 0x7f43281e5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432ffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43281ee790 con 0x7f43281593f0 2023-06-22T08:00:55.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432ffff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4328156b30 con 0x7f43280c4650 2023-06-22T08:00:55.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432ffff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432814cc70 con 0x7f4328152ba0 2023-06-22T08:00:55.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43281593f0 0x7f43281e5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43281593f0 0x7f43281e5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35708/0 (socket says 172.21.15.23:35708) 2023-06-22T08:00:55.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432e59c700 1 -- 172.21.15.23:0/1390836830 learned_addr learned my addr 172.21.15.23:0/1390836830 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:55.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432dd9b700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43280c4650 0x7f43281dfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.340+0000 7f432d59a700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4328152ba0 0x7f43281ef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f432e59c700 1 -- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4328152ba0 msgr2=0x7f43281ef250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f432e59c700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4328152ba0 0x7f43281ef250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f432e59c700 1 -- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43280c4650 msgr2=0x7f43281dfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f432e59c700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43280c4650 0x7f43281dfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f432e59c700 1 -- 172.21.15.23:0/1390836830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4320009b30 con 0x7f43281593f0 2023-06-22T08:00:55.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f432e59c700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43281593f0 0x7f43281e5b20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f432808b530 tx=0x7f4320004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:55.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f431e7fc700 1 -- 172.21.15.23:0/1390836830 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320004880 con 0x7f43281593f0 2023-06-22T08:00:55.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f431e7fc700 1 -- 172.21.15.23:0/1390836830 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43200092e0 con 0x7f43281593f0 2023-06-22T08:00:55.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f431e7fc700 1 -- 172.21.15.23:0/1390836830 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432001b650 con 0x7f43281593f0 2023-06-22T08:00:55.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f432ffff700 1 -- 172.21.15.23:0/1390836830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432814cf50 con 0x7f43281593f0 2023-06-22T08:00:55.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.341+0000 7f432ffff700 1 -- 172.21.15.23:0/1390836830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43281eed30 con 0x7f43281593f0 2023-06-22T08:00:55.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.342+0000 7f432ffff700 1 -- 172.21.15.23:0/1390836830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43280cbed0 con 0x7f43281593f0 2023-06-22T08:00:55.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.343+0000 7f431e7fc700 1 -- 172.21.15.23:0/1390836830 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f432001b7b0 con 0x7f43281593f0 2023-06-22T08:00:55.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.344+0000 7f431e7fc700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f430c06ace0 0x7f430c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.344+0000 7f431e7fc700 1 -- 172.21.15.23:0/1390836830 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f432008ac90 con 0x7f43281593f0 2023-06-22T08:00:55.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.345+0000 7f432dd9b700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f430c06ace0 0x7f430c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.345+0000 7f432dd9b700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f430c06ace0 0x7f430c06d190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4324007fd0 tx=0x7f4324009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:55.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.349+0000 7f431e7fc700 1 -- 172.21.15.23:0/1390836830 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4320055a40 con 0x7f43281593f0 2023-06-22T08:00:55.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.557+0000 7f432ffff700 1 -- 172.21.15.23:0/1390836830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f432804ea20 con 0x7f43281593f0 2023-06-22T08:00:55.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.557+0000 7f431e7fc700 1 -- 172.21.15.23:0/1390836830 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4320059060 con 0x7f43281593f0 2023-06-22T08:00:55.560 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:55.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.559+0000 7f432ffff700 1 -- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f430c06ace0 msgr2=0x7f430c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.559+0000 7f432ffff700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f430c06ace0 0x7f430c06d190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4324007fd0 tx=0x7f4324009040 comp rx=0 tx=0).stop 2023-06-22T08:00:55.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.559+0000 7f432ffff700 1 -- 172.21.15.23:0/1390836830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43281593f0 msgr2=0x7f43281e5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.559+0000 7f432ffff700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43281593f0 0x7f43281e5b20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f432808b530 tx=0x7f4320004ad0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.560+0000 7f432ffff700 1 -- 172.21.15.23:0/1390836830 shutdown_connections 2023-06-22T08:00:55.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.560+0000 7f432ffff700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43281593f0 0x7f43281e5b20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.560+0000 7f432ffff700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f430c06ace0 0x7f430c06d190 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.560+0000 7f432ffff700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43280c4650 0x7f43281dfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.560+0000 7f432ffff700 1 --2- 172.21.15.23:0/1390836830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4328152ba0 0x7f43281ef250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.560+0000 7f432ffff700 1 -- 172.21.15.23:0/1390836830 >> 172.21.15.23:0/1390836830 conn(0x7f4328065b90 msgr2=0x7f4328157610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:55.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.560+0000 7f432ffff700 1 -- 172.21.15.23:0/1390836830 shutdown_connections 2023-06-22T08:00:55.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.560+0000 7f432ffff700 1 -- 172.21.15.23:0/1390836830 wait complete. 2023-06-22T08:00:55.580 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-22T08:00:55.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.781+0000 7ffb86fff700 1 Processor -- start 2023-06-22T08:00:55.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.781+0000 7ffb86fff700 1 -- start start 2023-06-22T08:00:55.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.782+0000 7ffb86fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb80152ba0 0x7ffb801593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.782+0000 7ffb86fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 0x7ffb800c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.782+0000 7ffb86fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb80159930 con 0x7ffb800c4650 2023-06-22T08:00:55.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.782+0000 7ffb86fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb800c9aa0 con 0x7ffb80152ba0 2023-06-22T08:00:55.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.782+0000 7ffb86fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8006aed0 con 0x7ffb800c9260 2023-06-22T08:00:55.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.783+0000 7ffb8559c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ffb800c4650 0x7ffb800c9ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34962/0 (socket says 172.21.15.23:34962) 2023-06-22T08:00:55.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.783+0000 7ffb8559c700 1 -- 172.21.15.23:0/3765897401 learned_addr learned my addr 172.21.15.23:0/3765897401 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:55.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.783+0000 7ffb7ffff700 1 --2- 172.21.15.23:0/3765897401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 0x7ffb800c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.783+0000 7ffb84d9b700 1 --2- 172.21.15.23:0/3765897401 >> v2:172.21.15.67:3300/0 conn(0x7ffb80152ba0 0x7ffb801593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.783+0000 7ffb7f7fe700 1 -- 172.21.15.23:0/3765897401 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174588739 0 0) 0x7ffb78001490 con 0x7ffb800c4650 2023-06-22T08:00:55.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.783+0000 7ffb7f7fe700 1 -- 172.21.15.23:0/3765897401 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb700035b0 con 0x7ffb800c4650 2023-06-22T08:00:55.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.783+0000 7ffb7ffff700 1 -- 172.21.15.23:0/3765897401 >> v2:172.21.15.67:3300/0 conn(0x7ffb80152ba0 msgr2=0x7ffb801593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:55.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.783+0000 7ffb7ffff700 1 --2- 172.21.15.23:0/3765897401 >> v2:172.21.15.67:3300/0 conn(0x7ffb80152ba0 0x7ffb801593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.783+0000 7ffb7ffff700 1 -- 172.21.15.23:0/3765897401 >> v1:172.21.15.23:6789/0 conn(0x7ffb800c4650 legacy=0x7ffb800c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.783+0000 7ffb7ffff700 1 -- 172.21.15.23:0/3765897401 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8006b1b0 con 0x7ffb800c9260 2023-06-22T08:00:55.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb7ffff700 1 --2- 172.21.15.23:0/3765897401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 0x7ffb800c4110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009840 tx=0x7ffb6c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb334c4d5ce38ab5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:55.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb7f7fe700 1 -- 172.21.15.23:0/3765897401 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c011830 con 0x7ffb800c9260 2023-06-22T08:00:55.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb7f7fe700 1 -- 172.21.15.23:0/3765897401 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6c011990 con 0x7ffb800c9260 2023-06-22T08:00:55.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb7f7fe700 1 -- 172.21.15.23:0/3765897401 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c011d00 con 0x7ffb800c9260 2023-06-22T08:00:55.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb86fff700 1 -- 172.21.15.23:0/3765897401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 msgr2=0x7ffb800c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb86fff700 1 --2- 172.21.15.23:0/3765897401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 0x7ffb800c4110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009840 tx=0x7ffb6c009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:55.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb86fff700 1 -- 172.21.15.23:0/3765897401 shutdown_connections 2023-06-22T08:00:55.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb86fff700 1 --2- 172.21.15.23:0/3765897401 >> v2:172.21.15.67:3300/0 conn(0x7ffb80152ba0 0x7ffb801593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb86fff700 1 --2- 172.21.15.23:0/3765897401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 0x7ffb800c4110 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb86fff700 1 -- 172.21.15.23:0/3765897401 >> 172.21.15.23:0/3765897401 conn(0x7ffb80065b90 msgr2=0x7ffb80065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:55.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.784+0000 7ffb86fff700 1 -- 172.21.15.23:0/3765897401 shutdown_connections 2023-06-22T08:00:55.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.785+0000 7ffb86fff700 1 -- 172.21.15.23:0/3765897401 wait complete. 2023-06-22T08:00:55.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.786+0000 7ffb86fff700 1 Processor -- start 2023-06-22T08:00:55.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.786+0000 7ffb86fff700 1 -- start start 2023-06-22T08:00:55.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.787+0000 7ffb86fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb800c4650 0x7ffb800c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.787+0000 7ffb86fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 0x7ffb80151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.787+0000 7ffb86fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb80152ba0 0x7ffb8014b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.787+0000 7ffb86fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb800c5d50 con 0x7ffb800c4650 2023-06-22T08:00:55.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.787+0000 7ffb86fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb800c88d0 con 0x7ffb80152ba0 2023-06-22T08:00:55.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.787+0000 7ffb86fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb801f1fb0 con 0x7ffb800c9260 2023-06-22T08:00:55.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb84d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb800c4650 0x7ffb800c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb84d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb800c4650 0x7ffb800c6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35720/0 (socket says 172.21.15.23:35720) 2023-06-22T08:00:55.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb84d9b700 1 -- 172.21.15.23:0/1651903017 learned_addr learned my addr 172.21.15.23:0/1651903017 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:55.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb8559c700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb80152ba0 0x7ffb8014b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb7ffff700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 0x7ffb80151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb84d9b700 1 -- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 msgr2=0x7ffb80151a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb84d9b700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 0x7ffb80151a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb84d9b700 1 -- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb80152ba0 msgr2=0x7ffb8014b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:55.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb84d9b700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb80152ba0 0x7ffb8014b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:55.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb84d9b700 1 -- 172.21.15.23:0/1651903017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c0094a0 con 0x7ffb800c4650 2023-06-22T08:00:55.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb84d9b700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb800c4650 0x7ffb800c6010 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400e9e0 tx=0x7ffb7400ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:55.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.788+0000 7ffb7d7fa700 1 -- 172.21.15.23:0/1651903017 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74009710 con 0x7ffb800c4650 2023-06-22T08:00:55.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.789+0000 7ffb7d7fa700 1 -- 172.21.15.23:0/1651903017 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7400ee80 con 0x7ffb800c4650 2023-06-22T08:00:55.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.789+0000 7ffb7d7fa700 1 -- 172.21.15.23:0/1651903017 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7400ca50 con 0x7ffb800c4650 2023-06-22T08:00:55.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.789+0000 7ffb86fff700 1 -- 172.21.15.23:0/1651903017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb800c6550 con 0x7ffb800c4650 2023-06-22T08:00:55.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.789+0000 7ffb86fff700 1 -- 172.21.15.23:0/1651903017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb800c85f0 con 0x7ffb800c4650 2023-06-22T08:00:55.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.790+0000 7ffb86fff700 1 -- 172.21.15.23:0/1651903017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb800cbed0 con 0x7ffb800c4650 2023-06-22T08:00:55.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.791+0000 7ffb7d7fa700 1 -- 172.21.15.23:0/1651903017 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffb7400cbb0 con 0x7ffb800c4650 2023-06-22T08:00:55.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.792+0000 7ffb7d7fa700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb6406ace0 0x7ffb6406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:55.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.792+0000 7ffb7d7fa700 1 -- 172.21.15.23:0/1651903017 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ffb74089b80 con 0x7ffb800c4650 2023-06-22T08:00:55.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.792+0000 7ffb7ffff700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb6406ace0 0x7ffb6406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:55.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.793+0000 7ffb7ffff700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb6406ace0 0x7ffb6406d190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c000c00 tx=0x7ffb6c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:55.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:55.796+0000 7ffb7d7fa700 1 -- 172.21.15.23:0/1651903017 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffb74058330 con 0x7ffb800c4650 2023-06-22T08:00:56.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.002+0000 7ffb86fff700 1 -- 172.21.15.23:0/1651903017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffb801f2290 con 0x7ffb800c4650 2023-06-22T08:00:56.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.002+0000 7ffb7d7fa700 1 -- 172.21.15.23:0/1651903017 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffb74057ec0 con 0x7ffb800c4650 2023-06-22T08:00:56.006 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-06-22T08:00:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 -- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb6406ace0 msgr2=0x7ffb6406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb6406ace0 0x7ffb6406d190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c000c00 tx=0x7ffb6c008030 comp rx=0 tx=0).stop 2023-06-22T08:00:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 -- 172.21.15.23:0/1651903017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb800c4650 msgr2=0x7ffb800c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb800c4650 0x7ffb800c6010 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400e9e0 tx=0x7ffb7400ecf0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 -- 172.21.15.23:0/1651903017 shutdown_connections 2023-06-22T08:00:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb6406ace0 0x7ffb6406d190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb80152ba0 0x7ffb8014b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb800c4650 0x7ffb800c6010 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 --2- 172.21.15.23:0/1651903017 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb800c9260 0x7ffb80151a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 -- 172.21.15.23:0/1651903017 >> 172.21.15.23:0/1651903017 conn(0x7ffb80065b90 msgr2=0x7ffb80156b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 -- 172.21.15.23:0/1651903017 shutdown_connections 2023-06-22T08:00:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.005+0000 7ffb86fff700 1 -- 172.21.15.23:0/1651903017 wait complete. 2023-06-22T08:00:56.026 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-22T08:00:56.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.226+0000 7f2582fff700 1 Processor -- start 2023-06-22T08:00:56.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.226+0000 7f2582fff700 1 -- start start 2023-06-22T08:00:56.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.227+0000 7f2582fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f257c152ba0 0x7f257c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.227+0000 7f2582fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f257c0c9ee0 0x7f257c0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.227+0000 7f2582fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c159930 con 0x7f257c0c9260 2023-06-22T08:00:56.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.227+0000 7f2582fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c0c9aa0 con 0x7f257c0c9ee0 2023-06-22T08:00:56.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.227+0000 7f2582fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c06aed0 con 0x7f257c152ba0 2023-06-22T08:00:56.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f257bfff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f257c0c9260 0x7f257c0c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:45522/0 (socket says 172.21.15.23:45522) 2023-06-22T08:00:56.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f257bfff700 1 -- 172.21.15.23:0/2469553666 learned_addr learned my addr 172.21.15.23:0/2469553666 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:56.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f258159c700 1 --2- 172.21.15.23:0/2469553666 >> v2:172.21.15.23:3300/0 conn(0x7f257c0c9ee0 0x7f257c0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f2580d9b700 1 --2- 172.21.15.23:0/2469553666 >> v2:172.21.15.67:3300/0 conn(0x7f257c152ba0 0x7f257c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f257b7fe700 1 -- 172.21.15.23:0/2469553666 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147686980 0 0) 0x7f2568001110 con 0x7f257c0c9260 2023-06-22T08:00:56.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f2580d9b700 1 -- 172.21.15.23:0/2469553666 >> v2:172.21.15.23:3300/0 conn(0x7f257c0c9ee0 msgr2=0x7f257c0ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:56.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f2580d9b700 1 --2- 172.21.15.23:0/2469553666 >> v2:172.21.15.23:3300/0 conn(0x7f257c0c9ee0 0x7f257c0ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f2580d9b700 1 -- 172.21.15.23:0/2469553666 >> v1:172.21.15.23:6789/0 conn(0x7f257c0c9260 legacy=0x7f257c0c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f2580d9b700 1 -- 172.21.15.23:0/2469553666 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c06b1b0 con 0x7f257c152ba0 2023-06-22T08:00:56.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f258159c700 1 --2- 172.21.15.23:0/2469553666 >> v2:172.21.15.23:3300/0 conn(0x7f257c0c9ee0 0x7f257c0ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:56.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.228+0000 7f2580d9b700 1 --2- 172.21.15.23:0/2469553666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c152ba0 0x7f257c1593f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f257c08c670 tx=0x7f2570009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d798ae3c734ec0a0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:56.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f257b7fe700 1 -- 172.21.15.23:0/2469553666 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570011830 con 0x7f257c152ba0 2023-06-22T08:00:56.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f257b7fe700 1 -- 172.21.15.23:0/2469553666 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2570011990 con 0x7f257c152ba0 2023-06-22T08:00:56.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f257b7fe700 1 -- 172.21.15.23:0/2469553666 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570011d20 con 0x7f257c152ba0 2023-06-22T08:00:56.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f2582fff700 1 -- 172.21.15.23:0/2469553666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c152ba0 msgr2=0x7f257c1593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f2582fff700 1 --2- 172.21.15.23:0/2469553666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c152ba0 0x7f257c1593f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f257c08c670 tx=0x7f2570009b20 comp rx=0 tx=0).stop 2023-06-22T08:00:56.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f2582fff700 1 -- 172.21.15.23:0/2469553666 shutdown_connections 2023-06-22T08:00:56.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f2582fff700 1 --2- 172.21.15.23:0/2469553666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c152ba0 0x7f257c1593f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f2582fff700 1 --2- 172.21.15.23:0/2469553666 >> v2:172.21.15.23:3300/0 conn(0x7f257c0c9ee0 0x7f257c0ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f2582fff700 1 -- 172.21.15.23:0/2469553666 >> 172.21.15.23:0/2469553666 conn(0x7f257c065b90 msgr2=0x7f257c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:56.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f2582fff700 1 -- 172.21.15.23:0/2469553666 shutdown_connections 2023-06-22T08:00:56.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.229+0000 7f2582fff700 1 -- 172.21.15.23:0/2469553666 wait complete. 2023-06-22T08:00:56.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.231+0000 7f2582fff700 1 Processor -- start 2023-06-22T08:00:56.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.231+0000 7f2582fff700 1 -- start start 2023-06-22T08:00:56.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.232+0000 7f2582fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0c9260 0x7f257c1e4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.232+0000 7f2582fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f257c0c9ee0 0x7f257c1ef370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.232+0000 7f2582fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f257c152ba0 0x7f257c1ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.232+0000 7f2582fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0b3b20 con 0x7f257c152ba0 2023-06-22T08:00:56.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.232+0000 7f2582fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0b2590 con 0x7f257c0c9260 2023-06-22T08:00:56.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.232+0000 7f2582fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0b22f0 con 0x7f257c0c9ee0 2023-06-22T08:00:56.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.232+0000 7f257bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f257c0c9ee0 0x7f257c1ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.232+0000 7f258159c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f257c152ba0 0x7f257c1ef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.232+0000 7f2580d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0c9260 0x7f257c1e4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.233+0000 7f2580d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0c9260 0x7f257c1e4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51620/0 (socket says 172.21.15.23:51620) 2023-06-22T08:00:56.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.233+0000 7f257bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f257c0c9ee0 0x7f257c1ef370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49926/0 (socket says 172.21.15.23:49926) 2023-06-22T08:00:56.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.233+0000 7f2580d9b700 1 -- 172.21.15.23:0/4170893302 learned_addr learned my addr 172.21.15.23:0/4170893302 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:56.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.233+0000 7f258159c700 1 -- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f257c0c9ee0 msgr2=0x7f257c1ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.233+0000 7f258159c700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f257c0c9ee0 0x7f257c1ef370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.233+0000 7f258159c700 1 -- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0c9260 msgr2=0x7f257c1e4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.233+0000 7f258159c700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0c9260 0x7f257c1e4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.233+0000 7f258159c700 1 -- 172.21.15.23:0/4170893302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2574009a70 con 0x7f257c152ba0 2023-06-22T08:00:56.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.233+0000 7f258159c700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f257c152ba0 0x7f257c1ef8b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2574009e50 tx=0x7f257400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:56.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.233+0000 7f25797fa700 1 -- 172.21.15.23:0/4170893302 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257400ca90 con 0x7f257c152ba0 2023-06-22T08:00:56.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.234+0000 7f25797fa700 1 -- 172.21.15.23:0/4170893302 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257400f040 con 0x7f257c152ba0 2023-06-22T08:00:56.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.234+0000 7f2582fff700 1 -- 172.21.15.23:0/4170893302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25700094a0 con 0x7f257c152ba0 2023-06-22T08:00:56.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.234+0000 7f25797fa700 1 -- 172.21.15.23:0/4170893302 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257400ce50 con 0x7f257c152ba0 2023-06-22T08:00:56.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.234+0000 7f2582fff700 1 -- 172.21.15.23:0/4170893302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c0cc7e0 con 0x7f257c152ba0 2023-06-22T08:00:56.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.235+0000 7f2582fff700 1 -- 172.21.15.23:0/4170893302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c0cbed0 con 0x7f257c152ba0 2023-06-22T08:00:56.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.236+0000 7f25797fa700 1 -- 172.21.15.23:0/4170893302 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2574014480 con 0x7f257c152ba0 2023-06-22T08:00:56.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.238+0000 7f25797fa700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f256006ace0 0x7f256006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.238+0000 7f25797fa700 1 -- 172.21.15.23:0/4170893302 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f2574089d00 con 0x7f257c152ba0 2023-06-22T08:00:56.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.238+0000 7f2580d9b700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f256006ace0 0x7f256006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.239+0000 7f2580d9b700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f256006ace0 0x7f256006d190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2570000c00 tx=0x7f257001c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:56.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.242+0000 7f25797fa700 1 -- 172.21.15.23:0/4170893302 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2574054ab0 con 0x7f257c152ba0 2023-06-22T08:00:56.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.450+0000 7f2582fff700 1 -- 172.21.15.23:0/4170893302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f257c0b3cd0 con 0x7f257c152ba0 2023-06-22T08:00:56.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.451+0000 7f25797fa700 1 -- 172.21.15.23:0/4170893302 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f257c0b3cd0 con 0x7f257c152ba0 2023-06-22T08:00:56.454 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:00:56.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 -- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f256006ace0 msgr2=0x7f256006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f256006ace0 0x7f256006d190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2570000c00 tx=0x7f257001c040 comp rx=0 tx=0).stop 2023-06-22T08:00:56.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 -- 172.21.15.23:0/4170893302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f257c152ba0 msgr2=0x7f257c1ef8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f257c152ba0 0x7f257c1ef8b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2574009e50 tx=0x7f257400c3a0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 -- 172.21.15.23:0/4170893302 shutdown_connections 2023-06-22T08:00:56.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f257c152ba0 0x7f257c1ef8b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f256006ace0 0x7f256006d190 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f257c0c9260 0x7f257c1e4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 --2- 172.21.15.23:0/4170893302 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f257c0c9ee0 0x7f257c1ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 -- 172.21.15.23:0/4170893302 >> 172.21.15.23:0/4170893302 conn(0x7f257c065b90 msgr2=0x7f257c0c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:56.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 -- 172.21.15.23:0/4170893302 shutdown_connections 2023-06-22T08:00:56.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.453+0000 7f2582fff700 1 -- 172.21.15.23:0/4170893302 wait complete. 2023-06-22T08:00:56.474 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-06-22T08:00:56.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:00:56.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.670+0000 7f4b28fff700 1 Processor -- start 2023-06-22T08:00:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.671+0000 7f4b28fff700 1 -- start start 2023-06-22T08:00:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.672+0000 7f4b28fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b240c5740 0x7f4b24069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.672+0000 7f4b28fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b24158f60 con 0x7f4b24152e10 2023-06-22T08:00:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.672+0000 7f4b28fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b240cc450 con 0x7f4b240cb6b0 2023-06-22T08:00:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.672+0000 7f4b28fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b2406bd20 con 0x7f4b240c5740 2023-06-22T08:00:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.672+0000 7f4b1bfff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4b24152e10 0x7f4b24161100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:45528/0 (socket says 172.21.15.23:45528) 2023-06-22T08:00:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.672+0000 7f4b1bfff700 1 -- 172.21.15.23:0/1229896580 learned_addr learned my addr 172.21.15.23:0/1229896580 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.672+0000 7f4b22d9d700 1 --2- 172.21.15.23:0/1229896580 >> v2:172.21.15.67:3300/0 conn(0x7f4b240c5740 0x7f4b24069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b21d9b700 1 -- 172.21.15.23:0/1229896580 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193835146 0 0) 0x7f4b24158f60 con 0x7f4b24152e10 2023-06-22T08:00:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b21d9b700 1 -- 172.21.15.23:0/1229896580 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b100035b0 con 0x7f4b24152e10 2023-06-22T08:00:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b21d9b700 1 -- 172.21.15.23:0/1229896580 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939459036 0 0) 0x7f4b240cc450 con 0x7f4b240cb6b0 2023-06-22T08:00:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b21d9b700 1 -- 172.21.15.23:0/1229896580 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b24158f60 con 0x7f4b240cb6b0 2023-06-22T08:00:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b22d9d700 1 -- 172.21.15.23:0/1229896580 >> v1:172.21.15.67:6789/0 conn(0x7f4b240cb6b0 legacy=0x7f4b24152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b22d9d700 1 -- 172.21.15.23:0/1229896580 >> v1:172.21.15.23:6789/0 conn(0x7f4b24152e10 legacy=0x7f4b24161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b22d9d700 1 -- 172.21.15.23:0/1229896580 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b24158980 con 0x7f4b240c5740 2023-06-22T08:00:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b22d9d700 1 --2- 172.21.15.23:0/1229896580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b240c5740 0x7f4b24069eb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009d90 tx=0x7f4b1c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6243cf209a992def server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b21d9b700 1 -- 172.21.15.23:0/1229896580 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c00e070 con 0x7f4b240c5740 2023-06-22T08:00:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b21d9b700 1 -- 172.21.15.23:0/1229896580 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c00bb20 con 0x7f4b240c5740 2023-06-22T08:00:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.673+0000 7f4b21d9b700 1 -- 172.21.15.23:0/1229896580 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c012430 con 0x7f4b240c5740 2023-06-22T08:00:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.674+0000 7f4b28fff700 1 -- 172.21.15.23:0/1229896580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b240c5740 msgr2=0x7f4b24069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.674+0000 7f4b28fff700 1 --2- 172.21.15.23:0/1229896580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b240c5740 0x7f4b24069eb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009d90 tx=0x7f4b1c00b330 comp rx=0 tx=0).stop 2023-06-22T08:00:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.674+0000 7f4b28fff700 1 -- 172.21.15.23:0/1229896580 shutdown_connections 2023-06-22T08:00:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.674+0000 7f4b28fff700 1 --2- 172.21.15.23:0/1229896580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b240c5740 0x7f4b24069eb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.674+0000 7f4b28fff700 1 -- 172.21.15.23:0/1229896580 >> 172.21.15.23:0/1229896580 conn(0x7f4b24070d80 msgr2=0x7f4b24071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.674+0000 7f4b28fff700 1 -- 172.21.15.23:0/1229896580 shutdown_connections 2023-06-22T08:00:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.674+0000 7f4b28fff700 1 -- 172.21.15.23:0/1229896580 wait complete. 2023-06-22T08:00:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.676+0000 7f4b28fff700 1 Processor -- start 2023-06-22T08:00:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.676+0000 7f4b28fff700 1 -- start start 2023-06-22T08:00:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b28fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b240cb6b0 0x7f4b241e1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b28fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b24152e10 0x7f4b24156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b28fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b24156a70 0x7f4b241f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b28fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b240c7900 con 0x7f4b24152e10 2023-06-22T08:00:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b28fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b240c4fd0 con 0x7f4b240cb6b0 2023-06-22T08:00:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b28fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2414bd80 con 0x7f4b24156a70 2023-06-22T08:00:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b1bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b24152e10 0x7f4b24156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b1bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b24152e10 0x7f4b24156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52814/0 (socket says 172.21.15.23:52814) 2023-06-22T08:00:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b1bfff700 1 -- 172.21.15.23:0/2857556626 learned_addr learned my addr 172.21.15.23:0/2857556626 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b2259c700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b240cb6b0 0x7f4b241e1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b22d9d700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b24156a70 0x7f4b241f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b1bfff700 1 -- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b24156a70 msgr2=0x7f4b241f1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b1bfff700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b24156a70 0x7f4b241f1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b1bfff700 1 -- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b240cb6b0 msgr2=0x7f4b241e1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b1bfff700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b240cb6b0 0x7f4b241e1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.677+0000 7f4b1bfff700 1 -- 172.21.15.23:0/2857556626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c009a70 con 0x7f4b24152e10 2023-06-22T08:00:56.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.678+0000 7f4b1bfff700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b24152e10 0x7f4b24156530 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c005910 tx=0x7f4b0c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:56.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.678+0000 7f4b1affd700 1 -- 172.21.15.23:0/2857556626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c005d70 con 0x7f4b24152e10 2023-06-22T08:00:56.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.678+0000 7f4b1affd700 1 -- 172.21.15.23:0/2857556626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0c00ce80 con 0x7f4b24152e10 2023-06-22T08:00:56.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.678+0000 7f4b28fff700 1 -- 172.21.15.23:0/2857556626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b240c4a10 con 0x7f4b24152e10 2023-06-22T08:00:56.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.678+0000 7f4b1affd700 1 -- 172.21.15.23:0/2857556626 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c0072d0 con 0x7f4b24152e10 2023-06-22T08:00:56.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.678+0000 7f4b28fff700 1 -- 172.21.15.23:0/2857556626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2414afc0 con 0x7f4b24152e10 2023-06-22T08:00:56.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.679+0000 7f4b28fff700 1 -- 172.21.15.23:0/2857556626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b2414baa0 con 0x7f4b24152e10 2023-06-22T08:00:56.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.680+0000 7f4b1affd700 1 -- 172.21.15.23:0/2857556626 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4b0c007430 con 0x7f4b24152e10 2023-06-22T08:00:56.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.681+0000 7f4b1affd700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4b0406ace0 0x7f4b0406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:56.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.681+0000 7f4b1affd700 1 -- 172.21.15.23:0/2857556626 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4b0c088a80 con 0x7f4b24152e10 2023-06-22T08:00:56.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.681+0000 7f4b2259c700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4b0406ace0 0x7f4b0406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:56.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.682+0000 7f4b2259c700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4b0406ace0 0x7f4b0406d190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4b140011e0 tx=0x7f4b14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:56.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.685+0000 7f4b1affd700 1 -- 172.21.15.23:0/2857556626 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4b0c053850 con 0x7f4b24152e10 2023-06-22T08:00:56.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.936+0000 7f4b28fff700 1 -- 172.21.15.23:0/2857556626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b2404ea20 con 0x7f4b24152e10 2023-06-22T08:00:56.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.937+0000 7f4b1affd700 1 -- 172.21.15.23:0/2857556626 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6243 (secure 0 0 0) 0x7f4b0c056e70 con 0x7f4b24152e10 2023-06-22T08:00:56.940 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:56.941 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:00:41.586436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:00:56.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 -- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4b0406ace0 msgr2=0x7f4b0406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4b0406ace0 0x7f4b0406d190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4b140011e0 tx=0x7f4b14009040 comp rx=0 tx=0).stop 2023-06-22T08:00:56.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 -- 172.21.15.23:0/2857556626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b24152e10 msgr2=0x7f4b24156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:56.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b24152e10 0x7f4b24156530 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c005910 tx=0x7f4b0c00c7b0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 -- 172.21.15.23:0/2857556626 shutdown_connections 2023-06-22T08:00:56.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b24156a70 0x7f4b241f1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4b0406ace0 0x7f4b0406d190 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b240cb6b0 0x7f4b241e1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 --2- 172.21.15.23:0/2857556626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b24152e10 0x7f4b24156530 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:56.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 -- 172.21.15.23:0/2857556626 >> 172.21.15.23:0/2857556626 conn(0x7f4b24070d80 msgr2=0x7f4b240c7010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:56.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.939+0000 7f4b28fff700 1 -- 172.21.15.23:0/2857556626 shutdown_connections 2023-06-22T08:00:56.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:56.940+0000 7f4b28fff700 1 -- 172.21.15.23:0/2857556626 wait complete. 2023-06-22T08:00:56.944 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 19 2023-06-22T08:00:56.960 INFO:tasks.mds_thrash:Ready to start thrashing 2023-06-22T08:00:56.961 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-06-22T08:00:56.961 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-22T08:00:57.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.156+0000 7f11bf9ff700 1 Processor -- start 2023-06-22T08:00:57.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.156+0000 7f11bf9ff700 1 -- start start 2023-06-22T08:00:57.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.158+0000 7f11bf9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c9fd0 0x7f11b80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.158+0000 7f11bf9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f11b80c40b0 0x7f11b80c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.158+0000 7f11bf9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b815c370 con 0x7f11b8152ce0 2023-06-22T08:00:57.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.158+0000 7f11bf9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b81530b0 con 0x7f11b80c40b0 2023-06-22T08:00:57.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.158+0000 7f11bf9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b80cf6f0 con 0x7f11b80c9fd0 2023-06-22T08:00:57.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.158+0000 7f11bd79b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f11b8152ce0 0x7f11b815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:45544/0 (socket says 172.21.15.23:45544) 2023-06-22T08:00:57.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.158+0000 7f11bd79b700 1 -- 172.21.15.23:0/1188943268 learned_addr learned my addr 172.21.15.23:0/1188943268 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:57.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.158+0000 7f11bdf9c700 1 --2- 172.21.15.23:0/1188943268 >> v2:172.21.15.67:3300/0 conn(0x7f11b80c40b0 0x7f11b80c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.158+0000 7f11bcf9a700 1 --2- 172.21.15.23:0/1188943268 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c9fd0 0x7f11b80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.159+0000 7f11affff700 1 -- 172.21.15.23:0/1188943268 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801173810 0 0) 0x7f11b815c370 con 0x7f11b8152ce0 2023-06-22T08:00:57.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.159+0000 7f11affff700 1 -- 172.21.15.23:0/1188943268 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11a40035b0 con 0x7f11b8152ce0 2023-06-22T08:00:57.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.159+0000 7f11bcf9a700 1 -- 172.21.15.23:0/1188943268 >> v2:172.21.15.67:3300/0 conn(0x7f11b80c40b0 msgr2=0x7f11b80c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:57.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.159+0000 7f11bcf9a700 1 --2- 172.21.15.23:0/1188943268 >> v2:172.21.15.67:3300/0 conn(0x7f11b80c40b0 0x7f11b80c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.159+0000 7f11bcf9a700 1 -- 172.21.15.23:0/1188943268 >> v1:172.21.15.23:6789/0 conn(0x7f11b8152ce0 legacy=0x7f11b815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.159+0000 7f11bcf9a700 1 -- 172.21.15.23:0/1188943268 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b815bda0 con 0x7f11b80c9fd0 2023-06-22T08:00:57.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.159+0000 7f11bcf9a700 1 --2- 172.21.15.23:0/1188943268 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c9fd0 0x7f11b80cf1b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11a8009840 tx=0x7f11a8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa8ca52e65325e14 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:57.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.159+0000 7f11affff700 1 -- 172.21.15.23:0/1188943268 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a8011830 con 0x7f11b80c9fd0 2023-06-22T08:00:57.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.159+0000 7f11affff700 1 -- 172.21.15.23:0/1188943268 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a8011990 con 0x7f11b80c9fd0 2023-06-22T08:00:57.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.159+0000 7f11affff700 1 -- 172.21.15.23:0/1188943268 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a8011d20 con 0x7f11b80c9fd0 2023-06-22T08:00:57.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.160+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1188943268 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c9fd0 msgr2=0x7f11b80cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.160+0000 7f11bf9ff700 1 --2- 172.21.15.23:0/1188943268 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c9fd0 0x7f11b80cf1b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11a8009840 tx=0x7f11a8009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:57.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.160+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1188943268 shutdown_connections 2023-06-22T08:00:57.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.160+0000 7f11bf9ff700 1 --2- 172.21.15.23:0/1188943268 >> v2:172.21.15.67:3300/0 conn(0x7f11b80c40b0 0x7f11b80c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.160+0000 7f11bf9ff700 1 --2- 172.21.15.23:0/1188943268 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c9fd0 0x7f11b80cf1b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.160+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1188943268 >> 172.21.15.23:0/1188943268 conn(0x7f11b806ae30 msgr2=0x7f11b806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:57.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.160+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1188943268 shutdown_connections 2023-06-22T08:00:57.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.160+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1188943268 wait complete. 2023-06-22T08:00:57.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.162+0000 7f11bf9ff700 1 Processor -- start 2023-06-22T08:00:57.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.163+0000 7f11bf9ff700 1 -- start start 2023-06-22T08:00:57.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bf9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c40b0 0x7f11b814e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bf9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11b80c9fd0 0x7f11b815b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bf9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8152ce0 0x7f11b80cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bf9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b80c7850 con 0x7f11b8152ce0 2023-06-22T08:00:57.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bf9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b81ee180 con 0x7f11b80c9fd0 2023-06-22T08:00:57.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bf9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b81ee400 con 0x7f11b80c40b0 2023-06-22T08:00:57.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bdf9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8152ce0 0x7f11b80cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bdf9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8152ce0 0x7f11b80cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52826/0 (socket says 172.21.15.23:52826) 2023-06-22T08:00:57.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bdf9c700 1 -- 172.21.15.23:0/1092282869 learned_addr learned my addr 172.21.15.23:0/1092282869 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:57.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bd79b700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c40b0 0x7f11b814e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.164+0000 7f11bcf9a700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11b80c9fd0 0x7f11b815b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11bdf9c700 1 -- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c40b0 msgr2=0x7f11b814e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11bdf9c700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c40b0 0x7f11b814e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11bdf9c700 1 -- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11b80c9fd0 msgr2=0x7f11b815b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11bdf9c700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11b80c9fd0 0x7f11b815b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11bdf9c700 1 -- 172.21.15.23:0/1092282869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a80094a0 con 0x7f11b8152ce0 2023-06-22T08:00:57.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11bcf9a700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11b80c9fd0 0x7f11b815b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:57.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11bdf9c700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8152ce0 0x7f11b80cec60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f11b000efc0 tx=0x7f11b000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:57.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11adffb700 1 -- 172.21.15.23:0/1092282869 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0009ea0 con 0x7f11b8152ce0 2023-06-22T08:00:57.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11adffb700 1 -- 172.21.15.23:0/1092282869 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b000f040 con 0x7f11b8152ce0 2023-06-22T08:00:57.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1092282869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b81ee6e0 con 0x7f11b8152ce0 2023-06-22T08:00:57.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11adffb700 1 -- 172.21.15.23:0/1092282869 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b000ced0 con 0x7f11b8152ce0 2023-06-22T08:00:57.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.165+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1092282869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b81eeca0 con 0x7f11b8152ce0 2023-06-22T08:00:57.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.167+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1092282869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b804ea20 con 0x7f11b8152ce0 2023-06-22T08:00:57.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.167+0000 7f11adffb700 1 -- 172.21.15.23:0/1092282869 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f11b0014480 con 0x7f11b8152ce0 2023-06-22T08:00:57.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.168+0000 7f11adffb700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f119c06ace0 0x7f119c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.169+0000 7f11adffb700 1 -- 172.21.15.23:0/1092282869 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f11b008a000 con 0x7f11b8152ce0 2023-06-22T08:00:57.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.169+0000 7f11bd79b700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f119c06ace0 0x7f119c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.169+0000 7f11bd79b700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f119c06ace0 0x7f119c06d190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f11b4007fd0 tx=0x7f11b4007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:57.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.173+0000 7f11adffb700 1 -- 172.21.15.23:0/1092282869 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f11b0054d30 con 0x7f11b8152ce0 2023-06-22T08:00:57.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.377+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1092282869 --> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11b80c9740 con 0x7f119c06ace0 2023-06-22T08:00:57.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.390+0000 7f11adffb700 1 -- 172.21.15.23:0/1092282869 <== mgr.4100 v2:172.21.15.67:6832/3180103139 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+392072 (secure 0 0 0) 0x7f11b40058e0 con 0x7f119c06ace0 2023-06-22T08:00:57.397 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:57.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f119c06ace0 msgr2=0x7f119c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f119c06ace0 0x7f119c06d190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f11b4007fd0 tx=0x7f11b4007dc0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1092282869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8152ce0 msgr2=0x7f11b80cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8152ce0 0x7f11b80cec60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f11b000efc0 tx=0x7f11b000c3a0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1092282869 shutdown_connections 2023-06-22T08:00:57.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8152ce0 0x7f11b80cec60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11b80c40b0 0x7f11b814e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f119c06ace0 0x7f119c06d190 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 --2- 172.21.15.23:0/1092282869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11b80c9fd0 0x7f11b815b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1092282869 >> 172.21.15.23:0/1092282869 conn(0x7f11b806ae30 msgr2=0x7f11b80ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:57.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1092282869 shutdown_connections 2023-06-22T08:00:57.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.394+0000 7f11bf9ff700 1 -- 172.21.15.23:0/1092282869 wait complete. 2023-06-22T08:00:57.401 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-06-22T08:00:57.416 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2023-06-22T08:00:56.571499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600650,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":370,"num_write_kb":2485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":3752,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3842048,"data_stored":2189764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1492,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003943"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709823+0000","last_change":"2023-06-22T07:59:55.165740+0000","last_active":"2023-06-22T07:59:59.709823+0000","last_peered":"2023-06-22T07:59:59.709823+0000","last_clean":"2023-06-22T07:59:59.709823+0000","last_became_active":"2023-06-22T07:59:55.165398+0000","last_became_peered":"2023-06-22T07:59:55.165398+0000","last_unstale":"2023-06-22T07:59:59.709823+0000","last_undegraded":"2023-06-22T07:59:59.709823+0000","last_fullsized":"2023-06-22T07:59:59.709823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:37.596656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709026+0000","last_change":"2023-06-22T07:59:55.174603+0000","last_active":"2023-06-22T07:59:59.709026+0000","last_peered":"2023-06-22T07:59:59.709026+0000","last_clean":"2023-06-22T07:59:59.709026+0000","last_became_active":"2023-06-22T07:59:55.174346+0000","last_became_peered":"2023-06-22T07:59:55.174346+0000","last_unstale":"2023-06-22T07:59:59.709026+0000","last_undegraded":"2023-06-22T07:59:59.709026+0000","last_fullsized":"2023-06-22T07:59:59.709026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157321+0000","last_change":"2023-06-22T07:59:55.167525+0000","last_active":"2023-06-22T08:00:00.157321+0000","last_peered":"2023-06-22T08:00:00.157321+0000","last_clean":"2023-06-22T08:00:00.157321+0000","last_became_active":"2023-06-22T07:59:55.167181+0000","last_became_peered":"2023-06-22T07:59:55.167181+0000","last_unstale":"2023-06-22T08:00:00.157321+0000","last_undegraded":"2023-06-22T08:00:00.157321+0000","last_fullsized":"2023-06-22T08:00:00.157321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157927+0000","last_change":"2023-06-22T07:59:55.160549+0000","last_active":"2023-06-22T08:00:00.157927+0000","last_peered":"2023-06-22T08:00:00.157927+0000","last_clean":"2023-06-22T08:00:00.157927+0000","last_became_active":"2023-06-22T07:59:55.160239+0000","last_became_peered":"2023-06-22T07:59:55.160239+0000","last_unstale":"2023-06-22T08:00:00.157927+0000","last_undegraded":"2023-06-22T08:00:00.157927+0000","last_fullsized":"2023-06-22T08:00:00.157927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754390+0000","last_change":"2023-06-22T07:59:55.173006+0000","last_active":"2023-06-22T08:00:00.754390+0000","last_peered":"2023-06-22T08:00:00.754390+0000","last_clean":"2023-06-22T08:00:00.754390+0000","last_became_active":"2023-06-22T07:59:55.172578+0000","last_became_peered":"2023-06-22T07:59:55.172578+0000","last_unstale":"2023-06-22T08:00:00.754390+0000","last_undegraded":"2023-06-22T08:00:00.754390+0000","last_fullsized":"2023-06-22T08:00:00.754390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:12:44.799856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871378+0000","last_change":"2023-06-22T07:59:55.173858+0000","last_active":"2023-06-22T07:59:59.871378+0000","last_peered":"2023-06-22T07:59:59.871378+0000","last_clean":"2023-06-22T07:59:59.871378+0000","last_became_active":"2023-06-22T07:59:55.173599+0000","last_became_peered":"2023-06-22T07:59:55.173599+0000","last_unstale":"2023-06-22T07:59:59.871378+0000","last_undegraded":"2023-06-22T07:59:59.871378+0000","last_fullsized":"2023-06-22T07:59:59.871378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:37.597195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871174+0000","last_change":"2023-06-22T07:59:55.168354+0000","last_active":"2023-06-22T07:59:59.871174+0000","last_peered":"2023-06-22T07:59:59.871174+0000","last_clean":"2023-06-22T07:59:59.871174+0000","last_became_active":"2023-06-22T07:59:55.168050+0000","last_became_peered":"2023-06-22T07:59:55.168050+0000","last_unstale":"2023-06-22T07:59:59.871174+0000","last_undegraded":"2023-06-22T07:59:59.871174+0000","last_fullsized":"2023-06-22T07:59:59.871174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:08:45.847378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155602+0000","last_change":"2023-06-22T07:59:55.159738+0000","last_active":"2023-06-22T08:00:00.155602+0000","last_peered":"2023-06-22T08:00:00.155602+0000","last_clean":"2023-06-22T08:00:00.155602+0000","last_became_active":"2023-06-22T07:59:55.159501+0000","last_became_peered":"2023-06-22T07:59:55.159501+0000","last_unstale":"2023-06-22T08:00:00.155602+0000","last_undegraded":"2023-06-22T08:00:00.155602+0000","last_fullsized":"2023-06-22T08:00:00.155602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:55.160768+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:55.160508+0000","last_became_peered":"2023-06-22T07:59:55.160508+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:55:48.713145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155821+0000","last_change":"2023-06-22T07:59:55.166546+0000","last_active":"2023-06-22T08:00:00.155821+0000","last_peered":"2023-06-22T08:00:00.155821+0000","last_clean":"2023-06-22T08:00:00.155821+0000","last_became_active":"2023-06-22T07:59:55.166311+0000","last_became_peered":"2023-06-22T07:59:55.166311+0000","last_unstale":"2023-06-22T08:00:00.155821+0000","last_undegraded":"2023-06-22T08:00:00.155821+0000","last_fullsized":"2023-06-22T08:00:00.155821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:35.822548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155943+0000","last_change":"2023-06-22T07:59:55.162295+0000","last_active":"2023-06-22T08:00:00.155943+0000","last_peered":"2023-06-22T08:00:00.155943+0000","last_clean":"2023-06-22T08:00:00.155943+0000","last_became_active":"2023-06-22T07:59:55.162010+0000","last_became_peered":"2023-06-22T07:59:55.162010+0000","last_unstale":"2023-06-22T08:00:00.155943+0000","last_undegraded":"2023-06-22T08:00:00.155943+0000","last_fullsized":"2023-06-22T08:00:00.155943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709563+0000","last_change":"2023-06-22T07:59:55.158990+0000","last_active":"2023-06-22T07:59:59.709563+0000","last_peered":"2023-06-22T07:59:59.709563+0000","last_clean":"2023-06-22T07:59:59.709563+0000","last_became_active":"2023-06-22T07:59:55.158614+0000","last_became_peered":"2023-06-22T07:59:55.158614+0000","last_unstale":"2023-06-22T07:59:59.709563+0000","last_undegraded":"2023-06-22T07:59:59.709563+0000","last_fullsized":"2023-06-22T07:59:59.709563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755166+0000","last_change":"2023-06-22T07:59:55.161453+0000","last_active":"2023-06-22T08:00:00.755166+0000","last_peered":"2023-06-22T08:00:00.755166+0000","last_clean":"2023-06-22T08:00:00.755166+0000","last_became_active":"2023-06-22T07:59:55.161168+0000","last_became_peered":"2023-06-22T07:59:55.161168+0000","last_unstale":"2023-06-22T08:00:00.755166+0000","last_undegraded":"2023-06-22T08:00:00.755166+0000","last_fullsized":"2023-06-22T08:00:00.755166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155022+0000","last_change":"2023-06-22T07:59:55.164270+0000","last_active":"2023-06-22T08:00:00.155022+0000","last_peered":"2023-06-22T08:00:00.155022+0000","last_clean":"2023-06-22T08:00:00.155022+0000","last_became_active":"2023-06-22T07:59:55.163993+0000","last_became_peered":"2023-06-22T07:59:55.163993+0000","last_unstale":"2023-06-22T08:00:00.155022+0000","last_undegraded":"2023-06-22T08:00:00.155022+0000","last_fullsized":"2023-06-22T08:00:00.155022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:52.051830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157430+0000","last_change":"2023-06-22T07:59:55.169039+0000","last_active":"2023-06-22T08:00:00.157430+0000","last_peered":"2023-06-22T08:00:00.157430+0000","last_clean":"2023-06-22T08:00:00.157430+0000","last_became_active":"2023-06-22T07:59:55.168698+0000","last_became_peered":"2023-06-22T07:59:55.168698+0000","last_unstale":"2023-06-22T08:00:00.157430+0000","last_undegraded":"2023-06-22T08:00:00.157430+0000","last_fullsized":"2023-06-22T08:00:00.157430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871035+0000","last_change":"2023-06-22T07:59:55.158014+0000","last_active":"2023-06-22T07:59:59.871035+0000","last_peered":"2023-06-22T07:59:59.871035+0000","last_clean":"2023-06-22T07:59:59.871035+0000","last_became_active":"2023-06-22T07:59:55.157562+0000","last_became_peered":"2023-06-22T07:59:55.157562+0000","last_unstale":"2023-06-22T07:59:59.871035+0000","last_undegraded":"2023-06-22T07:59:59.871035+0000","last_fullsized":"2023-06-22T07:59:59.871035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:08:33.856001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754963+0000","last_change":"2023-06-22T07:59:55.165957+0000","last_active":"2023-06-22T08:00:00.754963+0000","last_peered":"2023-06-22T08:00:00.754963+0000","last_clean":"2023-06-22T08:00:00.754963+0000","last_became_active":"2023-06-22T07:59:55.165643+0000","last_became_peered":"2023-06-22T07:59:55.165643+0000","last_unstale":"2023-06-22T08:00:00.754963+0000","last_undegraded":"2023-06-22T08:00:00.754963+0000","last_fullsized":"2023-06-22T08:00:00.754963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:10.134743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157391+0000","last_change":"2023-06-22T07:59:55.159858+0000","last_active":"2023-06-22T08:00:00.157391+0000","last_peered":"2023-06-22T08:00:00.157391+0000","last_clean":"2023-06-22T08:00:00.157391+0000","last_became_active":"2023-06-22T07:59:55.159444+0000","last_became_peered":"2023-06-22T07:59:55.159444+0000","last_unstale":"2023-06-22T08:00:00.157391+0000","last_undegraded":"2023-06-22T08:00:00.157391+0000","last_fullsized":"2023-06-22T08:00:00.157391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709424+0000","last_change":"2023-06-22T07:59:55.157162+0000","last_active":"2023-06-22T07:59:59.709424+0000","last_peered":"2023-06-22T07:59:59.709424+0000","last_clean":"2023-06-22T07:59:59.709424+0000","last_became_active":"2023-06-22T07:59:55.156882+0000","last_became_peered":"2023-06-22T07:59:55.156882+0000","last_unstale":"2023-06-22T07:59:59.709424+0000","last_undegraded":"2023-06-22T07:59:59.709424+0000","last_fullsized":"2023-06-22T07:59:59.709424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155892+0000","last_change":"2023-06-22T07:59:55.163834+0000","last_active":"2023-06-22T08:00:00.155892+0000","last_peered":"2023-06-22T08:00:00.155892+0000","last_clean":"2023-06-22T08:00:00.155892+0000","last_became_active":"2023-06-22T07:59:55.163580+0000","last_became_peered":"2023-06-22T07:59:55.163580+0000","last_unstale":"2023-06-22T08:00:00.155892+0000","last_undegraded":"2023-06-22T08:00:00.155892+0000","last_fullsized":"2023-06-22T08:00:00.155892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155285+0000","last_change":"2023-06-22T07:59:55.164569+0000","last_active":"2023-06-22T08:00:00.155285+0000","last_peered":"2023-06-22T08:00:00.155285+0000","last_clean":"2023-06-22T08:00:00.155285+0000","last_became_active":"2023-06-22T07:59:55.164286+0000","last_became_peered":"2023-06-22T07:59:55.164286+0000","last_unstale":"2023-06-22T08:00:00.155285+0000","last_undegraded":"2023-06-22T08:00:00.155285+0000","last_fullsized":"2023-06-22T08:00:00.155285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:17.386317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754137+0000","last_change":"2023-06-22T07:59:55.159883+0000","last_active":"2023-06-22T08:00:00.754137+0000","last_peered":"2023-06-22T08:00:00.754137+0000","last_clean":"2023-06-22T08:00:00.754137+0000","last_became_active":"2023-06-22T07:59:55.159588+0000","last_became_peered":"2023-06-22T07:59:55.159588+0000","last_unstale":"2023-06-22T08:00:00.754137+0000","last_undegraded":"2023-06-22T08:00:00.754137+0000","last_fullsized":"2023-06-22T08:00:00.754137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754941+0000","last_change":"2023-06-22T07:59:55.160638+0000","last_active":"2023-06-22T08:00:00.754941+0000","last_peered":"2023-06-22T08:00:00.754941+0000","last_clean":"2023-06-22T08:00:00.754941+0000","last_became_active":"2023-06-22T07:59:55.160315+0000","last_became_peered":"2023-06-22T07:59:55.160315+0000","last_unstale":"2023-06-22T08:00:00.754941+0000","last_undegraded":"2023-06-22T08:00:00.754941+0000","last_fullsized":"2023-06-22T08:00:00.754941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:51:09.921626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155504+0000","last_change":"2023-06-22T07:59:55.165950+0000","last_active":"2023-06-22T08:00:00.155504+0000","last_peered":"2023-06-22T08:00:00.155504+0000","last_clean":"2023-06-22T08:00:00.155504+0000","last_became_active":"2023-06-22T07:59:55.165505+0000","last_became_peered":"2023-06-22T07:59:55.165505+0000","last_unstale":"2023-06-22T08:00:00.155504+0000","last_undegraded":"2023-06-22T08:00:00.155504+0000","last_fullsized":"2023-06-22T08:00:00.155504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:13.512588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755640+0000","last_change":"2023-06-22T07:59:55.157572+0000","last_active":"2023-06-22T08:00:00.755640+0000","last_peered":"2023-06-22T08:00:00.755640+0000","last_clean":"2023-06-22T08:00:00.755640+0000","last_became_active":"2023-06-22T07:59:55.157055+0000","last_became_peered":"2023-06-22T07:59:55.157055+0000","last_unstale":"2023-06-22T08:00:00.755640+0000","last_undegraded":"2023-06-22T08:00:00.755640+0000","last_fullsized":"2023-06-22T08:00:00.755640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:34:35.685117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754095+0000","last_change":"2023-06-22T07:59:55.164696+0000","last_active":"2023-06-22T08:00:00.754095+0000","last_peered":"2023-06-22T08:00:00.754095+0000","last_clean":"2023-06-22T08:00:00.754095+0000","last_became_active":"2023-06-22T07:59:55.164179+0000","last_became_peered":"2023-06-22T07:59:55.164179+0000","last_unstale":"2023-06-22T08:00:00.754095+0000","last_undegraded":"2023-06-22T08:00:00.754095+0000","last_fullsized":"2023-06-22T08:00:00.754095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871504+0000","last_change":"2023-06-22T07:59:55.168838+0000","last_active":"2023-06-22T07:59:59.871504+0000","last_peered":"2023-06-22T07:59:59.871504+0000","last_clean":"2023-06-22T07:59:59.871504+0000","last_became_active":"2023-06-22T07:59:55.168551+0000","last_became_peered":"2023-06-22T07:59:55.168551+0000","last_unstale":"2023-06-22T07:59:59.871504+0000","last_undegraded":"2023-06-22T07:59:59.871504+0000","last_fullsized":"2023-06-22T07:59:59.871504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:02:36.615449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708648+0000","last_change":"2023-06-22T07:59:59.195356+0000","last_active":"2023-06-22T07:59:59.708648+0000","last_peered":"2023-06-22T07:59:59.708648+0000","last_clean":"2023-06-22T07:59:59.708648+0000","last_became_active":"2023-06-22T07:59:59.194814+0000","last_became_peered":"2023-06-22T07:59:59.194814+0000","last_unstale":"2023-06-22T07:59:59.708648+0000","last_undegraded":"2023-06-22T07:59:59.708648+0000","last_fullsized":"2023-06-22T07:59:59.708648+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:12.223267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6,0],"acting":[2,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493176+0000","last_change":"2023-06-22T07:59:54.651326+0000","last_active":"2023-06-22T08:00:36.493176+0000","last_peered":"2023-06-22T08:00:36.493176+0000","last_clean":"2023-06-22T08:00:36.493176+0000","last_became_active":"2023-06-22T07:59:54.651010+0000","last_became_peered":"2023-06-22T07:59:54.651010+0000","last_unstale":"2023-06-22T08:00:36.493176+0000","last_undegraded":"2023-06-22T08:00:36.493176+0000","last_fullsized":"2023-06-22T08:00:36.493176+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:17.769117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754538+0000","last_change":"2023-06-22T07:59:55.160345+0000","last_active":"2023-06-22T08:00:00.754538+0000","last_peered":"2023-06-22T08:00:00.754538+0000","last_clean":"2023-06-22T08:00:00.754538+0000","last_became_active":"2023-06-22T07:59:55.160071+0000","last_became_peered":"2023-06-22T07:59:55.160071+0000","last_unstale":"2023-06-22T08:00:00.754538+0000","last_undegraded":"2023-06-22T08:00:00.754538+0000","last_fullsized":"2023-06-22T08:00:00.754538+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708662+0000","last_change":"2023-06-22T07:59:59.196477+0000","last_active":"2023-06-22T07:59:59.708662+0000","last_peered":"2023-06-22T07:59:59.708662+0000","last_clean":"2023-06-22T07:59:59.708662+0000","last_became_active":"2023-06-22T07:59:59.196007+0000","last_became_peered":"2023-06-22T07:59:59.196007+0000","last_unstale":"2023-06-22T07:59:59.708662+0000","last_undegraded":"2023-06-22T07:59:59.708662+0000","last_fullsized":"2023-06-22T07:59:59.708662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,5],"acting":[2,7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:31.397730+0000","last_change":"2023-06-22T07:59:54.653192+0000","last_active":"2023-06-22T08:00:31.397730+0000","last_peered":"2023-06-22T08:00:31.397730+0000","last_clean":"2023-06-22T08:00:31.397730+0000","last_became_active":"2023-06-22T07:59:54.652926+0000","last_became_peered":"2023-06-22T07:59:54.652926+0000","last_unstale":"2023-06-22T08:00:31.397730+0000","last_undegraded":"2023-06-22T08:00:31.397730+0000","last_fullsized":"2023-06-22T08:00:31.397730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:12:59.656555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754609+0000","last_change":"2023-06-22T07:59:55.163171+0000","last_active":"2023-06-22T08:00:00.754609+0000","last_peered":"2023-06-22T08:00:00.754609+0000","last_clean":"2023-06-22T08:00:00.754609+0000","last_became_active":"2023-06-22T07:59:55.162869+0000","last_became_peered":"2023-06-22T07:59:55.162869+0000","last_unstale":"2023-06-22T08:00:00.754609+0000","last_undegraded":"2023-06-22T08:00:00.754609+0000","last_fullsized":"2023-06-22T08:00:00.754609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:37:59.063186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754586+0000","last_change":"2023-06-22T07:59:59.200516+0000","last_active":"2023-06-22T08:00:00.754586+0000","last_peered":"2023-06-22T08:00:00.754586+0000","last_clean":"2023-06-22T08:00:00.754586+0000","last_became_active":"2023-06-22T07:59:59.200096+0000","last_became_peered":"2023-06-22T07:59:59.200096+0000","last_unstale":"2023-06-22T08:00:00.754586+0000","last_undegraded":"2023-06-22T08:00:00.754586+0000","last_fullsized":"2023-06-22T08:00:00.754586+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:06.099105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2,7],"acting":[6,3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.490825+0000","last_change":"2023-06-22T07:59:54.645950+0000","last_active":"2023-06-22T08:00:36.490825+0000","last_peered":"2023-06-22T08:00:36.490825+0000","last_clean":"2023-06-22T08:00:36.490825+0000","last_became_active":"2023-06-22T07:59:54.645520+0000","last_became_peered":"2023-06-22T07:59:54.645520+0000","last_unstale":"2023-06-22T08:00:36.490825+0000","last_undegraded":"2023-06-22T08:00:36.490825+0000","last_fullsized":"2023-06-22T08:00:36.490825+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:34.816556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871992+0000","last_change":"2023-06-22T07:59:55.162767+0000","last_active":"2023-06-22T07:59:59.871992+0000","last_peered":"2023-06-22T07:59:59.871992+0000","last_clean":"2023-06-22T07:59:59.871992+0000","last_became_active":"2023-06-22T07:59:55.162451+0000","last_became_peered":"2023-06-22T07:59:55.162451+0000","last_unstale":"2023-06-22T07:59:59.871992+0000","last_undegraded":"2023-06-22T07:59:59.871992+0000","last_fullsized":"2023-06-22T07:59:59.871992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:38.977413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871071+0000","last_change":"2023-06-22T07:59:59.208215+0000","last_active":"2023-06-22T07:59:59.871071+0000","last_peered":"2023-06-22T07:59:59.871071+0000","last_clean":"2023-06-22T07:59:59.871071+0000","last_became_active":"2023-06-22T07:59:59.207608+0000","last_became_peered":"2023-06-22T07:59:59.207608+0000","last_unstale":"2023-06-22T07:59:59.871071+0000","last_undegraded":"2023-06-22T07:59:59.871071+0000","last_fullsized":"2023-06-22T07:59:59.871071+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:39:48.527076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,5],"acting":[3,2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.699308+0000","last_change":"2023-06-22T07:59:54.652515+0000","last_active":"2023-06-22T08:00:25.699308+0000","last_peered":"2023-06-22T08:00:25.699308+0000","last_clean":"2023-06-22T08:00:25.699308+0000","last_became_active":"2023-06-22T07:59:54.652249+0000","last_became_peered":"2023-06-22T07:59:54.652249+0000","last_unstale":"2023-06-22T08:00:25.699308+0000","last_undegraded":"2023-06-22T08:00:25.699308+0000","last_fullsized":"2023-06-22T08:00:25.699308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:12.506596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755250+0000","last_change":"2023-06-22T07:59:55.158359+0000","last_active":"2023-06-22T08:00:00.755250+0000","last_peered":"2023-06-22T08:00:00.755250+0000","last_clean":"2023-06-22T08:00:00.755250+0000","last_became_active":"2023-06-22T07:59:55.157918+0000","last_became_peered":"2023-06-22T07:59:55.157918+0000","last_unstale":"2023-06-22T08:00:00.755250+0000","last_undegraded":"2023-06-22T08:00:00.755250+0000","last_fullsized":"2023-06-22T08:00:00.755250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708305+0000","last_change":"2023-06-22T07:59:59.201033+0000","last_active":"2023-06-22T07:59:59.708305+0000","last_peered":"2023-06-22T07:59:59.708305+0000","last_clean":"2023-06-22T07:59:59.708305+0000","last_became_active":"2023-06-22T07:59:59.200500+0000","last_became_peered":"2023-06-22T07:59:59.200500+0000","last_unstale":"2023-06-22T07:59:59.708305+0000","last_undegraded":"2023-06-22T07:59:59.708305+0000","last_fullsized":"2023-06-22T07:59:59.708305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:18.174604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,4],"acting":[2,1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.571653+0000","last_change":"2023-06-22T07:59:54.654416+0000","last_active":"2023-06-22T08:00:41.571653+0000","last_peered":"2023-06-22T08:00:41.571653+0000","last_clean":"2023-06-22T08:00:41.571653+0000","last_became_active":"2023-06-22T07:59:54.654139+0000","last_became_peered":"2023-06-22T07:59:54.654139+0000","last_unstale":"2023-06-22T08:00:41.571653+0000","last_undegraded":"2023-06-22T08:00:41.571653+0000","last_fullsized":"2023-06-22T08:00:41.571653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755506+0000","last_change":"2023-06-22T07:59:55.159790+0000","last_active":"2023-06-22T08:00:00.755506+0000","last_peered":"2023-06-22T08:00:00.755506+0000","last_clean":"2023-06-22T08:00:00.755506+0000","last_became_active":"2023-06-22T07:59:55.159489+0000","last_became_peered":"2023-06-22T07:59:55.159489+0000","last_unstale":"2023-06-22T08:00:00.755506+0000","last_undegraded":"2023-06-22T08:00:00.755506+0000","last_fullsized":"2023-06-22T08:00:00.755506+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:15:18.373418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753814+0000","last_change":"2023-06-22T07:59:59.197822+0000","last_active":"2023-06-22T08:00:00.753814+0000","last_peered":"2023-06-22T08:00:00.753814+0000","last_clean":"2023-06-22T08:00:00.753814+0000","last_became_active":"2023-06-22T07:59:59.197431+0000","last_became_peered":"2023-06-22T07:59:59.197431+0000","last_unstale":"2023-06-22T08:00:00.753814+0000","last_undegraded":"2023-06-22T08:00:00.753814+0000","last_fullsized":"2023-06-22T08:00:00.753814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:41.139500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7,1],"acting":[4,6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.698982+0000","last_change":"2023-06-22T07:59:54.644158+0000","last_active":"2023-06-22T08:00:25.698982+0000","last_peered":"2023-06-22T08:00:25.698982+0000","last_clean":"2023-06-22T08:00:25.698982+0000","last_became_active":"2023-06-22T07:59:54.643808+0000","last_became_peered":"2023-06-22T07:59:54.643808+0000","last_unstale":"2023-06-22T08:00:25.698982+0000","last_undegraded":"2023-06-22T08:00:25.698982+0000","last_fullsized":"2023-06-22T08:00:25.698982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:45.438691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754507+0000","last_change":"2023-06-22T07:59:55.157538+0000","last_active":"2023-06-22T08:00:00.754507+0000","last_peered":"2023-06-22T08:00:00.754507+0000","last_clean":"2023-06-22T08:00:00.754507+0000","last_became_active":"2023-06-22T07:59:55.156967+0000","last_became_peered":"2023-06-22T07:59:55.156967+0000","last_unstale":"2023-06-22T08:00:00.754507+0000","last_undegraded":"2023-06-22T08:00:00.754507+0000","last_fullsized":"2023-06-22T08:00:00.754507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:38:44.161245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753821+0000","last_change":"2023-06-22T07:59:59.201671+0000","last_active":"2023-06-22T08:00:00.753821+0000","last_peered":"2023-06-22T08:00:00.753821+0000","last_clean":"2023-06-22T08:00:00.753821+0000","last_became_active":"2023-06-22T07:59:59.201201+0000","last_became_peered":"2023-06-22T07:59:59.201201+0000","last_unstale":"2023-06-22T08:00:00.753821+0000","last_undegraded":"2023-06-22T08:00:00.753821+0000","last_fullsized":"2023-06-22T08:00:00.753821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4,3],"acting":[6,1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"28'16","reported_seq":37,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:50.546385+0000","last_change":"2023-06-22T07:59:54.640470+0000","last_active":"2023-06-22T08:00:50.546385+0000","last_peered":"2023-06-22T08:00:50.546385+0000","last_clean":"2023-06-22T08:00:50.546385+0000","last_became_active":"2023-06-22T07:59:54.640124+0000","last_became_peered":"2023-06-22T07:59:54.640124+0000","last_unstale":"2023-06-22T08:00:50.546385+0000","last_undegraded":"2023-06-22T08:00:50.546385+0000","last_fullsized":"2023-06-22T08:00:50.546385+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:06:41.502702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4052,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708599+0000","last_change":"2023-06-22T07:59:55.165550+0000","last_active":"2023-06-22T07:59:59.708599+0000","last_peered":"2023-06-22T07:59:59.708599+0000","last_clean":"2023-06-22T07:59:59.708599+0000","last_became_active":"2023-06-22T07:59:55.164998+0000","last_became_peered":"2023-06-22T07:59:55.164998+0000","last_unstale":"2023-06-22T07:59:59.708599+0000","last_undegraded":"2023-06-22T07:59:59.708599+0000","last_fullsized":"2023-06-22T07:59:59.708599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872133+0000","last_change":"2023-06-22T07:59:59.207103+0000","last_active":"2023-06-22T07:59:59.872133+0000","last_peered":"2023-06-22T07:59:59.872133+0000","last_clean":"2023-06-22T07:59:59.872133+0000","last_became_active":"2023-06-22T07:59:59.206531+0000","last_became_peered":"2023-06-22T07:59:59.206531+0000","last_unstale":"2023-06-22T07:59:59.872133+0000","last_undegraded":"2023-06-22T07:59:59.872133+0000","last_fullsized":"2023-06-22T07:59:59.872133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:42:48.571228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0,6],"acting":[3,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"28'9","reported_seq":30,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.570482+0000","last_change":"2023-06-22T07:59:54.644301+0000","last_active":"2023-06-22T08:00:41.570482+0000","last_peered":"2023-06-22T08:00:41.570482+0000","last_clean":"2023-06-22T08:00:41.570482+0000","last_became_active":"2023-06-22T07:59:54.643939+0000","last_became_peered":"2023-06-22T07:59:54.643939+0000","last_unstale":"2023-06-22T08:00:41.570482+0000","last_undegraded":"2023-06-22T08:00:41.570482+0000","last_fullsized":"2023-06-22T08:00:41.570482+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:07:42.118767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753877+0000","last_change":"2023-06-22T07:59:55.163311+0000","last_active":"2023-06-22T08:00:00.753877+0000","last_peered":"2023-06-22T08:00:00.753877+0000","last_clean":"2023-06-22T08:00:00.753877+0000","last_became_active":"2023-06-22T07:59:55.162991+0000","last_became_peered":"2023-06-22T07:59:55.162991+0000","last_unstale":"2023-06-22T08:00:00.753877+0000","last_undegraded":"2023-06-22T08:00:00.753877+0000","last_fullsized":"2023-06-22T08:00:00.753877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754227+0000","last_change":"2023-06-22T07:59:59.201324+0000","last_active":"2023-06-22T08:00:00.754227+0000","last_peered":"2023-06-22T08:00:00.754227+0000","last_clean":"2023-06-22T08:00:00.754227+0000","last_became_active":"2023-06-22T07:59:59.200877+0000","last_became_peered":"2023-06-22T07:59:59.200877+0000","last_unstale":"2023-06-22T08:00:00.754227+0000","last_undegraded":"2023-06-22T08:00:00.754227+0000","last_fullsized":"2023-06-22T08:00:00.754227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:14:17.094426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5,7],"acting":[4,2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493994+0000","last_change":"2023-06-22T07:59:54.651527+0000","last_active":"2023-06-22T08:00:36.493994+0000","last_peered":"2023-06-22T08:00:36.493994+0000","last_clean":"2023-06-22T08:00:36.493994+0000","last_became_active":"2023-06-22T07:59:54.651196+0000","last_became_peered":"2023-06-22T07:59:54.651196+0000","last_unstale":"2023-06-22T08:00:36.493994+0000","last_undegraded":"2023-06-22T08:00:36.493994+0000","last_fullsized":"2023-06-22T08:00:36.493994+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872142+0000","last_change":"2023-06-22T07:59:55.162610+0000","last_active":"2023-06-22T07:59:59.872142+0000","last_peered":"2023-06-22T07:59:59.872142+0000","last_clean":"2023-06-22T07:59:59.872142+0000","last_became_active":"2023-06-22T07:59:55.162334+0000","last_became_peered":"2023-06-22T07:59:55.162334+0000","last_unstale":"2023-06-22T07:59:59.872142+0000","last_undegraded":"2023-06-22T07:59:59.872142+0000","last_fullsized":"2023-06-22T07:59:59.872142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:42.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:59.199402+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:59.198979+0000","last_became_peered":"2023-06-22T07:59:59.198979+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:17:07.122732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1,6],"acting":[4,5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493688+0000","last_change":"2023-06-22T07:59:54.649236+0000","last_active":"2023-06-22T08:00:36.493688+0000","last_peered":"2023-06-22T08:00:36.493688+0000","last_clean":"2023-06-22T08:00:36.493688+0000","last_became_active":"2023-06-22T07:59:54.648852+0000","last_became_peered":"2023-06-22T07:59:54.648852+0000","last_unstale":"2023-06-22T08:00:36.493688+0000","last_undegraded":"2023-06-22T08:00:36.493688+0000","last_fullsized":"2023-06-22T08:00:36.493688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:16:59.865638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709134+0000","last_change":"2023-06-22T07:59:55.165859+0000","last_active":"2023-06-22T07:59:59.709134+0000","last_peered":"2023-06-22T07:59:59.709134+0000","last_clean":"2023-06-22T07:59:59.709134+0000","last_became_active":"2023-06-22T07:59:55.165542+0000","last_became_peered":"2023-06-22T07:59:55.165542+0000","last_unstale":"2023-06-22T07:59:59.709134+0000","last_undegraded":"2023-06-22T07:59:59.709134+0000","last_fullsized":"2023-06-22T07:59:59.709134+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:32.147969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871894+0000","last_change":"2023-06-22T07:59:59.207794+0000","last_active":"2023-06-22T07:59:59.871894+0000","last_peered":"2023-06-22T07:59:59.871894+0000","last_clean":"2023-06-22T07:59:59.871894+0000","last_became_active":"2023-06-22T07:59:59.207364+0000","last_became_peered":"2023-06-22T07:59:59.207364+0000","last_unstale":"2023-06-22T07:59:59.871894+0000","last_undegraded":"2023-06-22T07:59:59.871894+0000","last_fullsized":"2023-06-22T07:59:59.871894+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:30.718491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4,0],"acting":[3,6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"28'17","reported_seq":38,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.491476+0000","last_change":"2023-06-22T07:59:54.651423+0000","last_active":"2023-06-22T08:00:36.491476+0000","last_peered":"2023-06-22T08:00:36.491476+0000","last_clean":"2023-06-22T08:00:36.491476+0000","last_became_active":"2023-06-22T07:59:54.651167+0000","last_became_peered":"2023-06-22T07:59:54.651167+0000","last_unstale":"2023-06-22T08:00:36.491476+0000","last_undegraded":"2023-06-22T08:00:36.491476+0000","last_fullsized":"2023-06-22T08:00:36.491476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:31.141977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709649+0000","last_change":"2023-06-22T07:59:55.164478+0000","last_active":"2023-06-22T07:59:59.709649+0000","last_peered":"2023-06-22T07:59:59.709649+0000","last_clean":"2023-06-22T07:59:59.709649+0000","last_became_active":"2023-06-22T07:59:55.163922+0000","last_became_peered":"2023-06-22T07:59:55.163922+0000","last_unstale":"2023-06-22T07:59:59.709649+0000","last_undegraded":"2023-06-22T07:59:59.709649+0000","last_fullsized":"2023-06-22T07:59:59.709649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:43.772399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155085+0000","last_change":"2023-06-22T07:59:59.198496+0000","last_active":"2023-06-22T08:00:00.155085+0000","last_peered":"2023-06-22T08:00:00.155085+0000","last_clean":"2023-06-22T08:00:00.155085+0000","last_became_active":"2023-06-22T07:59:59.198067+0000","last_became_peered":"2023-06-22T07:59:59.198067+0000","last_unstale":"2023-06-22T08:00:00.155085+0000","last_undegraded":"2023-06-22T08:00:00.155085+0000","last_fullsized":"2023-06-22T08:00:00.155085+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:24.865757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,7],"acting":[1,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158445+0000","last_change":"2023-06-22T07:59:54.651366+0000","last_active":"2023-06-22T08:00:00.158445+0000","last_peered":"2023-06-22T08:00:00.158445+0000","last_clean":"2023-06-22T08:00:00.158445+0000","last_became_active":"2023-06-22T07:59:54.651059+0000","last_became_peered":"2023-06-22T07:59:54.651059+0000","last_unstale":"2023-06-22T08:00:00.158445+0000","last_undegraded":"2023-06-22T08:00:00.158445+0000","last_fullsized":"2023-06-22T08:00:00.158445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:37.183143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871131+0000","last_change":"2023-06-22T07:59:55.162820+0000","last_active":"2023-06-22T07:59:59.871131+0000","last_peered":"2023-06-22T07:59:59.871131+0000","last_clean":"2023-06-22T07:59:59.871131+0000","last_became_active":"2023-06-22T07:59:55.162505+0000","last_became_peered":"2023-06-22T07:59:55.162505+0000","last_unstale":"2023-06-22T07:59:59.871131+0000","last_undegraded":"2023-06-22T07:59:59.871131+0000","last_fullsized":"2023-06-22T07:59:59.871131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:46:38.312778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753744+0000","last_change":"2023-06-22T07:59:59.201059+0000","last_active":"2023-06-22T08:00:00.753744+0000","last_peered":"2023-06-22T08:00:00.753744+0000","last_clean":"2023-06-22T08:00:00.753744+0000","last_became_active":"2023-06-22T07:59:59.200460+0000","last_became_peered":"2023-06-22T07:59:59.200460+0000","last_unstale":"2023-06-22T08:00:00.753744+0000","last_undegraded":"2023-06-22T08:00:00.753744+0000","last_fullsized":"2023-06-22T08:00:00.753744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,0],"acting":[6,4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.565345+0000","last_change":"2023-06-22T07:59:54.646017+0000","last_active":"2023-06-22T08:00:41.565345+0000","last_peered":"2023-06-22T08:00:41.565345+0000","last_clean":"2023-06-22T08:00:41.565345+0000","last_became_active":"2023-06-22T07:59:54.645554+0000","last_became_peered":"2023-06-22T07:59:54.645554+0000","last_unstale":"2023-06-22T08:00:41.565345+0000","last_undegraded":"2023-06-22T08:00:41.565345+0000","last_fullsized":"2023-06-22T08:00:41.565345+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:01.550269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154806+0000","last_change":"2023-06-22T07:59:55.166283+0000","last_active":"2023-06-22T08:00:00.154806+0000","last_peered":"2023-06-22T08:00:00.154806+0000","last_clean":"2023-06-22T08:00:00.154806+0000","last_became_active":"2023-06-22T07:59:55.165977+0000","last_became_peered":"2023-06-22T07:59:55.165977+0000","last_unstale":"2023-06-22T08:00:00.154806+0000","last_undegraded":"2023-06-22T08:00:00.154806+0000","last_fullsized":"2023-06-22T08:00:00.154806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:21:29.484640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709141+0000","last_change":"2023-06-22T07:59:59.205411+0000","last_active":"2023-06-22T07:59:59.709141+0000","last_peered":"2023-06-22T07:59:59.709141+0000","last_clean":"2023-06-22T07:59:59.709141+0000","last_became_active":"2023-06-22T07:59:59.204980+0000","last_became_peered":"2023-06-22T07:59:59.204980+0000","last_unstale":"2023-06-22T07:59:59.709141+0000","last_undegraded":"2023-06-22T07:59:59.709141+0000","last_fullsized":"2023-06-22T07:59:59.709141+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:22.317952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493022+0000","last_change":"2023-06-22T07:59:54.654491+0000","last_active":"2023-06-22T08:00:36.493022+0000","last_peered":"2023-06-22T08:00:36.493022+0000","last_clean":"2023-06-22T08:00:36.493022+0000","last_became_active":"2023-06-22T07:59:54.654197+0000","last_became_peered":"2023-06-22T07:59:54.654197+0000","last_unstale":"2023-06-22T08:00:36.493022+0000","last_undegraded":"2023-06-22T08:00:36.493022+0000","last_fullsized":"2023-06-22T08:00:36.493022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:42:59.725455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708444+0000","last_change":"2023-06-22T07:59:55.172053+0000","last_active":"2023-06-22T07:59:59.708444+0000","last_peered":"2023-06-22T07:59:59.708444+0000","last_clean":"2023-06-22T07:59:59.708444+0000","last_became_active":"2023-06-22T07:59:55.165501+0000","last_became_peered":"2023-06-22T07:59:55.165501+0000","last_unstale":"2023-06-22T07:59:59.708444+0000","last_undegraded":"2023-06-22T07:59:59.708444+0000","last_fullsized":"2023-06-22T07:59:59.708444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753990+0000","last_change":"2023-06-22T07:59:59.199179+0000","last_active":"2023-06-22T08:00:00.753990+0000","last_peered":"2023-06-22T08:00:00.753990+0000","last_clean":"2023-06-22T08:00:00.753990+0000","last_became_active":"2023-06-22T07:59:59.198740+0000","last_became_peered":"2023-06-22T07:59:59.198740+0000","last_unstale":"2023-06-22T08:00:00.753990+0000","last_undegraded":"2023-06-22T08:00:00.753990+0000","last_fullsized":"2023-06-22T08:00:00.753990+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.491850+0000","last_change":"2023-06-22T07:59:54.644520+0000","last_active":"2023-06-22T08:00:36.491850+0000","last_peered":"2023-06-22T08:00:36.491850+0000","last_clean":"2023-06-22T08:00:36.491850+0000","last_became_active":"2023-06-22T07:59:54.644204+0000","last_became_peered":"2023-06-22T07:59:54.644204+0000","last_unstale":"2023-06-22T08:00:36.491850+0000","last_undegraded":"2023-06-22T08:00:36.491850+0000","last_fullsized":"2023-06-22T08:00:36.491850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:07.674431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709657+0000","last_change":"2023-06-22T07:59:55.164111+0000","last_active":"2023-06-22T07:59:59.709657+0000","last_peered":"2023-06-22T07:59:59.709657+0000","last_clean":"2023-06-22T07:59:59.709657+0000","last_became_active":"2023-06-22T07:59:55.163802+0000","last_became_peered":"2023-06-22T07:59:55.163802+0000","last_unstale":"2023-06-22T07:59:59.709657+0000","last_undegraded":"2023-06-22T07:59:59.709657+0000","last_fullsized":"2023-06-22T07:59:59.709657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755252+0000","last_change":"2023-06-22T07:59:59.197721+0000","last_active":"2023-06-22T08:00:00.755252+0000","last_peered":"2023-06-22T08:00:00.755252+0000","last_clean":"2023-06-22T08:00:00.755252+0000","last_became_active":"2023-06-22T07:59:59.197259+0000","last_became_peered":"2023-06-22T07:59:59.197259+0000","last_unstale":"2023-06-22T08:00:00.755252+0000","last_undegraded":"2023-06-22T08:00:00.755252+0000","last_fullsized":"2023-06-22T08:00:00.755252+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5,1],"acting":[7,0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.697877+0000","last_change":"2023-06-22T07:59:54.643124+0000","last_active":"2023-06-22T08:00:25.697877+0000","last_peered":"2023-06-22T08:00:25.697877+0000","last_clean":"2023-06-22T08:00:25.697877+0000","last_became_active":"2023-06-22T07:59:54.642833+0000","last_became_peered":"2023-06-22T07:59:54.642833+0000","last_unstale":"2023-06-22T08:00:25.697877+0000","last_undegraded":"2023-06-22T08:00:25.697877+0000","last_fullsized":"2023-06-22T08:00:25.697877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158064+0000","last_change":"2023-06-22T07:59:55.160551+0000","last_active":"2023-06-22T08:00:00.158064+0000","last_peered":"2023-06-22T08:00:00.158064+0000","last_clean":"2023-06-22T08:00:00.158064+0000","last_became_active":"2023-06-22T07:59:55.160221+0000","last_became_peered":"2023-06-22T07:59:55.160221+0000","last_unstale":"2023-06-22T08:00:00.158064+0000","last_undegraded":"2023-06-22T08:00:00.158064+0000","last_fullsized":"2023-06-22T08:00:00.158064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708974+0000","last_change":"2023-06-22T07:59:59.204643+0000","last_active":"2023-06-22T07:59:59.708974+0000","last_peered":"2023-06-22T07:59:59.708974+0000","last_clean":"2023-06-22T07:59:59.708974+0000","last_became_active":"2023-06-22T07:59:59.204182+0000","last_became_peered":"2023-06-22T07:59:59.204182+0000","last_unstale":"2023-06-22T07:59:59.708974+0000","last_undegraded":"2023-06-22T07:59:59.708974+0000","last_fullsized":"2023-06-22T07:59:59.708974+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4,2],"acting":[5,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493681+0000","last_change":"2023-06-22T07:59:54.654036+0000","last_active":"2023-06-22T08:00:36.493681+0000","last_peered":"2023-06-22T08:00:36.493681+0000","last_clean":"2023-06-22T08:00:36.493681+0000","last_became_active":"2023-06-22T07:59:54.653736+0000","last_became_peered":"2023-06-22T07:59:54.653736+0000","last_unstale":"2023-06-22T08:00:36.493681+0000","last_undegraded":"2023-06-22T08:00:36.493681+0000","last_fullsized":"2023-06-22T08:00:36.493681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:44:09.331748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871999+0000","last_change":"2023-06-22T07:59:55.163181+0000","last_active":"2023-06-22T07:59:59.871999+0000","last_peered":"2023-06-22T07:59:59.871999+0000","last_clean":"2023-06-22T07:59:59.871999+0000","last_became_active":"2023-06-22T07:59:55.162860+0000","last_became_peered":"2023-06-22T07:59:55.162860+0000","last_unstale":"2023-06-22T07:59:59.871999+0000","last_undegraded":"2023-06-22T07:59:59.871999+0000","last_fullsized":"2023-06-22T07:59:59.871999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:59:25.037938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155510+0000","last_change":"2023-06-22T07:59:59.195710+0000","last_active":"2023-06-22T08:00:00.155510+0000","last_peered":"2023-06-22T08:00:00.155510+0000","last_clean":"2023-06-22T08:00:00.155510+0000","last_became_active":"2023-06-22T07:59:59.195246+0000","last_became_peered":"2023-06-22T07:59:59.195246+0000","last_unstale":"2023-06-22T08:00:00.155510+0000","last_undegraded":"2023-06-22T08:00:00.155510+0000","last_fullsized":"2023-06-22T08:00:00.155510+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:36.457493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,5],"acting":[1,0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.570855+0000","last_change":"2023-06-22T07:59:54.649692+0000","last_active":"2023-06-22T08:00:41.570855+0000","last_peered":"2023-06-22T08:00:41.570855+0000","last_clean":"2023-06-22T08:00:41.570855+0000","last_became_active":"2023-06-22T07:59:54.649416+0000","last_became_peered":"2023-06-22T07:59:54.649416+0000","last_unstale":"2023-06-22T08:00:41.570855+0000","last_undegraded":"2023-06-22T08:00:41.570855+0000","last_fullsized":"2023-06-22T08:00:41.570855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:54:02.923433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708909+0000","last_change":"2023-06-22T07:59:55.175392+0000","last_active":"2023-06-22T07:59:59.708909+0000","last_peered":"2023-06-22T07:59:59.708909+0000","last_clean":"2023-06-22T07:59:59.708909+0000","last_became_active":"2023-06-22T07:59:55.175112+0000","last_became_peered":"2023-06-22T07:59:55.175112+0000","last_unstale":"2023-06-22T07:59:59.708909+0000","last_undegraded":"2023-06-22T07:59:59.708909+0000","last_fullsized":"2023-06-22T07:59:59.708909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871873+0000","last_change":"2023-06-22T07:59:59.206671+0000","last_active":"2023-06-22T07:59:59.871873+0000","last_peered":"2023-06-22T07:59:59.871873+0000","last_clean":"2023-06-22T07:59:59.871873+0000","last_became_active":"2023-06-22T07:59:59.206179+0000","last_became_peered":"2023-06-22T07:59:59.206179+0000","last_unstale":"2023-06-22T07:59:59.871873+0000","last_undegraded":"2023-06-22T07:59:59.871873+0000","last_fullsized":"2023-06-22T07:59:59.871873+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:00.427518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4,5],"acting":[3,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.698074+0000","last_change":"2023-06-22T07:59:54.640508+0000","last_active":"2023-06-22T08:00:25.698074+0000","last_peered":"2023-06-22T08:00:25.698074+0000","last_clean":"2023-06-22T08:00:25.698074+0000","last_became_active":"2023-06-22T07:59:54.640108+0000","last_became_peered":"2023-06-22T07:59:54.640108+0000","last_unstale":"2023-06-22T08:00:25.698074+0000","last_undegraded":"2023-06-22T08:00:25.698074+0000","last_fullsized":"2023-06-22T08:00:25.698074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:19.193231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754822+0000","last_change":"2023-06-22T07:59:55.164398+0000","last_active":"2023-06-22T08:00:00.754822+0000","last_peered":"2023-06-22T08:00:00.754822+0000","last_clean":"2023-06-22T08:00:00.754822+0000","last_became_active":"2023-06-22T07:59:55.164086+0000","last_became_peered":"2023-06-22T07:59:55.164086+0000","last_unstale":"2023-06-22T08:00:00.754822+0000","last_undegraded":"2023-06-22T08:00:00.754822+0000","last_fullsized":"2023-06-22T08:00:00.754822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:38.189135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753997+0000","last_change":"2023-06-22T07:59:59.207752+0000","last_active":"2023-06-22T08:00:00.753997+0000","last_peered":"2023-06-22T08:00:00.753997+0000","last_clean":"2023-06-22T08:00:00.753997+0000","last_became_active":"2023-06-22T07:59:59.207360+0000","last_became_peered":"2023-06-22T07:59:59.207360+0000","last_unstale":"2023-06-22T08:00:00.753997+0000","last_undegraded":"2023-06-22T08:00:00.753997+0000","last_fullsized":"2023-06-22T08:00:00.753997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:08:25.633009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3,1],"acting":[4,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709114+0000","last_change":"2023-06-22T07:59:49.652772+0000","last_active":"2023-06-22T07:59:59.709114+0000","last_peered":"2023-06-22T07:59:59.709114+0000","last_clean":"2023-06-22T07:59:59.709114+0000","last_became_active":"2023-06-22T07:59:49.652487+0000","last_became_peered":"2023-06-22T07:59:49.652487+0000","last_unstale":"2023-06-22T07:59:59.709114+0000","last_undegraded":"2023-06-22T07:59:59.709114+0000","last_fullsized":"2023-06-22T07:59:59.709114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:11.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493495+0000","last_change":"2023-06-22T07:59:54.644538+0000","last_active":"2023-06-22T08:00:36.493495+0000","last_peered":"2023-06-22T08:00:36.493495+0000","last_clean":"2023-06-22T08:00:36.493495+0000","last_became_active":"2023-06-22T07:59:54.644253+0000","last_became_peered":"2023-06-22T07:59:54.644253+0000","last_unstale":"2023-06-22T08:00:36.493495+0000","last_undegraded":"2023-06-22T08:00:36.493495+0000","last_fullsized":"2023-06-22T08:00:36.493495+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:09.128751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754228+0000","last_change":"2023-06-22T07:59:55.163802+0000","last_active":"2023-06-22T08:00:00.754228+0000","last_peered":"2023-06-22T08:00:00.754228+0000","last_clean":"2023-06-22T08:00:00.754228+0000","last_became_active":"2023-06-22T07:59:55.163481+0000","last_became_peered":"2023-06-22T07:59:55.163481+0000","last_unstale":"2023-06-22T08:00:00.754228+0000","last_undegraded":"2023-06-22T08:00:00.754228+0000","last_fullsized":"2023-06-22T08:00:00.754228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:16.916350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755246+0000","last_change":"2023-06-22T07:59:59.207344+0000","last_active":"2023-06-22T08:00:00.755246+0000","last_peered":"2023-06-22T08:00:00.755246+0000","last_clean":"2023-06-22T08:00:00.755246+0000","last_became_active":"2023-06-22T07:59:59.206975+0000","last_became_peered":"2023-06-22T07:59:59.206975+0000","last_unstale":"2023-06-22T08:00:00.755246+0000","last_undegraded":"2023-06-22T08:00:00.755246+0000","last_fullsized":"2023-06-22T08:00:00.755246+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3,1],"acting":[7,4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871513+0000","last_change":"2023-06-22T07:59:49.653053+0000","last_active":"2023-06-22T07:59:59.871513+0000","last_peered":"2023-06-22T07:59:59.871513+0000","last_clean":"2023-06-22T07:59:59.871513+0000","last_became_active":"2023-06-22T07:59:49.652861+0000","last_became_peered":"2023-06-22T07:59:49.652861+0000","last_unstale":"2023-06-22T07:59:59.871513+0000","last_undegraded":"2023-06-22T07:59:59.871513+0000","last_fullsized":"2023-06-22T07:59:59.871513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:28:40.388367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"28'6","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493566+0000","last_change":"2023-06-22T07:59:54.645403+0000","last_active":"2023-06-22T08:00:36.493566+0000","last_peered":"2023-06-22T08:00:36.493566+0000","last_clean":"2023-06-22T08:00:36.493566+0000","last_became_active":"2023-06-22T07:59:54.644973+0000","last_became_peered":"2023-06-22T07:59:54.644973+0000","last_unstale":"2023-06-22T08:00:36.493566+0000","last_undegraded":"2023-06-22T08:00:36.493566+0000","last_fullsized":"2023-06-22T08:00:36.493566+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:15.910358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754101+0000","last_change":"2023-06-22T07:59:55.161484+0000","last_active":"2023-06-22T08:00:00.754101+0000","last_peered":"2023-06-22T08:00:00.754101+0000","last_clean":"2023-06-22T08:00:00.754101+0000","last_became_active":"2023-06-22T07:59:55.161194+0000","last_became_peered":"2023-06-22T07:59:55.161194+0000","last_unstale":"2023-06-22T08:00:00.754101+0000","last_undegraded":"2023-06-22T08:00:00.754101+0000","last_fullsized":"2023-06-22T08:00:00.754101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754988+0000","last_change":"2023-06-22T07:59:59.198241+0000","last_active":"2023-06-22T08:00:00.754988+0000","last_peered":"2023-06-22T08:00:00.754988+0000","last_clean":"2023-06-22T08:00:00.754988+0000","last_became_active":"2023-06-22T07:59:59.197689+0000","last_became_peered":"2023-06-22T07:59:59.197689+0000","last_unstale":"2023-06-22T08:00:00.754988+0000","last_undegraded":"2023-06-22T08:00:00.754988+0000","last_fullsized":"2023-06-22T08:00:00.754988+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:39.513874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4,1],"acting":[7,0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753886+0000","last_change":"2023-06-22T07:59:49.450044+0000","last_active":"2023-06-22T08:00:00.753886+0000","last_peered":"2023-06-22T08:00:00.753886+0000","last_clean":"2023-06-22T08:00:00.753886+0000","last_became_active":"2023-06-22T07:59:49.449731+0000","last_became_peered":"2023-06-22T07:59:49.449731+0000","last_unstale":"2023-06-22T08:00:00.753886+0000","last_undegraded":"2023-06-22T08:00:00.753886+0000","last_fullsized":"2023-06-22T08:00:00.753886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:40.262337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:31.395849+0000","last_change":"2023-06-22T07:59:54.640462+0000","last_active":"2023-06-22T08:00:31.395849+0000","last_peered":"2023-06-22T08:00:31.395849+0000","last_clean":"2023-06-22T08:00:31.395849+0000","last_became_active":"2023-06-22T07:59:54.640123+0000","last_became_peered":"2023-06-22T07:59:54.640123+0000","last_unstale":"2023-06-22T08:00:31.395849+0000","last_undegraded":"2023-06-22T08:00:31.395849+0000","last_fullsized":"2023-06-22T08:00:31.395849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:20.316921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755350+0000","last_change":"2023-06-22T07:59:55.164651+0000","last_active":"2023-06-22T08:00:00.755350+0000","last_peered":"2023-06-22T08:00:00.755350+0000","last_clean":"2023-06-22T08:00:00.755350+0000","last_became_active":"2023-06-22T07:59:55.164235+0000","last_became_peered":"2023-06-22T07:59:55.164235+0000","last_unstale":"2023-06-22T08:00:00.755350+0000","last_undegraded":"2023-06-22T08:00:00.755350+0000","last_fullsized":"2023-06-22T08:00:00.755350+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:32.924136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157741+0000","last_change":"2023-06-22T07:59:59.204779+0000","last_active":"2023-06-22T08:00:00.157741+0000","last_peered":"2023-06-22T08:00:00.157741+0000","last_clean":"2023-06-22T08:00:00.157741+0000","last_became_active":"2023-06-22T07:59:59.204288+0000","last_became_peered":"2023-06-22T07:59:59.204288+0000","last_unstale":"2023-06-22T08:00:00.157741+0000","last_undegraded":"2023-06-22T08:00:00.157741+0000","last_fullsized":"2023-06-22T08:00:00.157741+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871497+0000","last_change":"2023-06-22T07:59:49.657008+0000","last_active":"2023-06-22T07:59:59.871497+0000","last_peered":"2023-06-22T07:59:59.871497+0000","last_clean":"2023-06-22T07:59:59.871497+0000","last_became_active":"2023-06-22T07:59:49.656752+0000","last_became_peered":"2023-06-22T07:59:49.656752+0000","last_unstale":"2023-06-22T07:59:59.871497+0000","last_undegraded":"2023-06-22T07:59:59.871497+0000","last_fullsized":"2023-06-22T07:59:59.871497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:50:37.639134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":185,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155134+0000","last_change":"2023-06-22T07:59:42.501363+0000","last_active":"2023-06-22T08:00:00.155134+0000","last_peered":"2023-06-22T08:00:00.155134+0000","last_clean":"2023-06-22T08:00:00.155134+0000","last_became_active":"2023-06-22T07:59:42.500320+0000","last_became_peered":"2023-06-22T07:59:42.500320+0000","last_unstale":"2023-06-22T08:00:00.155134+0000","last_undegraded":"2023-06-22T08:00:00.155134+0000","last_fullsized":"2023-06-22T08:00:00.155134+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_clean_scrub_stamp":"2023-06-22T07:59:41.381475+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:08:53.784030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.569235+0000","last_change":"2023-06-22T07:59:54.645031+0000","last_active":"2023-06-22T08:00:41.569235+0000","last_peered":"2023-06-22T08:00:41.569235+0000","last_clean":"2023-06-22T08:00:41.569235+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:41.569235+0000","last_undegraded":"2023-06-22T08:00:41.569235+0000","last_fullsized":"2023-06-22T08:00:41.569235+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:42.766407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872074+0000","last_change":"2023-06-22T07:59:55.162799+0000","last_active":"2023-06-22T07:59:59.872074+0000","last_peered":"2023-06-22T07:59:59.872074+0000","last_clean":"2023-06-22T07:59:59.872074+0000","last_became_active":"2023-06-22T07:59:55.162501+0000","last_became_peered":"2023-06-22T07:59:55.162501+0000","last_unstale":"2023-06-22T07:59:59.872074+0000","last_undegraded":"2023-06-22T07:59:59.872074+0000","last_fullsized":"2023-06-22T07:59:59.872074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:57:14.305897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871805+0000","last_change":"2023-06-22T07:59:59.205097+0000","last_active":"2023-06-22T07:59:59.871805+0000","last_peered":"2023-06-22T07:59:59.871805+0000","last_clean":"2023-06-22T07:59:59.871805+0000","last_became_active":"2023-06-22T07:59:59.201465+0000","last_became_peered":"2023-06-22T07:59:59.201465+0000","last_unstale":"2023-06-22T07:59:59.871805+0000","last_undegraded":"2023-06-22T07:59:59.871805+0000","last_fullsized":"2023-06-22T07:59:59.871805+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:17:45.352998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,0],"acting":[3,7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871056+0000","last_change":"2023-06-22T07:59:49.442558+0000","last_active":"2023-06-22T07:59:59.871056+0000","last_peered":"2023-06-22T07:59:59.871056+0000","last_clean":"2023-06-22T07:59:59.871056+0000","last_became_active":"2023-06-22T07:59:49.442154+0000","last_became_peered":"2023-06-22T07:59:49.442154+0000","last_unstale":"2023-06-22T07:59:59.871056+0000","last_undegraded":"2023-06-22T07:59:59.871056+0000","last_fullsized":"2023-06-22T07:59:59.871056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:46:16.233204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:31.399798+0000","last_change":"2023-06-22T07:59:54.641746+0000","last_active":"2023-06-22T08:00:31.399798+0000","last_peered":"2023-06-22T08:00:31.399798+0000","last_clean":"2023-06-22T08:00:31.399798+0000","last_became_active":"2023-06-22T07:59:54.641431+0000","last_became_peered":"2023-06-22T07:59:54.641431+0000","last_unstale":"2023-06-22T08:00:31.399798+0000","last_undegraded":"2023-06-22T08:00:31.399798+0000","last_fullsized":"2023-06-22T08:00:31.399798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709468+0000","last_change":"2023-06-22T07:59:55.163214+0000","last_active":"2023-06-22T07:59:59.709468+0000","last_peered":"2023-06-22T07:59:59.709468+0000","last_clean":"2023-06-22T07:59:59.709468+0000","last_became_active":"2023-06-22T07:59:55.162907+0000","last_became_peered":"2023-06-22T07:59:55.162907+0000","last_unstale":"2023-06-22T07:59:59.709468+0000","last_undegraded":"2023-06-22T07:59:59.709468+0000","last_fullsized":"2023-06-22T07:59:59.709468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:46.988703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157450+0000","last_change":"2023-06-22T07:59:59.198945+0000","last_active":"2023-06-22T08:00:00.157450+0000","last_peered":"2023-06-22T08:00:00.157450+0000","last_clean":"2023-06-22T08:00:00.157450+0000","last_became_active":"2023-06-22T07:59:59.198561+0000","last_became_peered":"2023-06-22T07:59:59.198561+0000","last_unstale":"2023-06-22T08:00:00.157450+0000","last_undegraded":"2023-06-22T08:00:00.157450+0000","last_fullsized":"2023-06-22T08:00:00.157450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:06.135651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2,5],"acting":[0,7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871842+0000","last_change":"2023-06-22T07:59:49.442246+0000","last_active":"2023-06-22T07:59:59.871842+0000","last_peered":"2023-06-22T07:59:59.871842+0000","last_clean":"2023-06-22T07:59:59.871842+0000","last_became_active":"2023-06-22T07:59:49.442001+0000","last_became_peered":"2023-06-22T07:59:49.442001+0000","last_unstale":"2023-06-22T07:59:59.871842+0000","last_undegraded":"2023-06-22T07:59:59.871842+0000","last_fullsized":"2023-06-22T07:59:59.871842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:27:24.457978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754302+0000","last_change":"2023-06-22T07:59:54.643272+0000","last_active":"2023-06-22T08:00:00.754302+0000","last_peered":"2023-06-22T08:00:00.754302+0000","last_clean":"2023-06-22T08:00:00.754302+0000","last_became_active":"2023-06-22T07:59:54.642925+0000","last_became_peered":"2023-06-22T07:59:54.642925+0000","last_unstale":"2023-06-22T08:00:00.754302+0000","last_undegraded":"2023-06-22T08:00:00.754302+0000","last_fullsized":"2023-06-22T08:00:00.754302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:13.625768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754231+0000","last_change":"2023-06-22T07:59:55.167307+0000","last_active":"2023-06-22T08:00:00.754231+0000","last_peered":"2023-06-22T08:00:00.754231+0000","last_clean":"2023-06-22T08:00:00.754231+0000","last_became_active":"2023-06-22T07:59:55.167026+0000","last_became_peered":"2023-06-22T07:59:55.167026+0000","last_unstale":"2023-06-22T08:00:00.754231+0000","last_undegraded":"2023-06-22T08:00:00.754231+0000","last_fullsized":"2023-06-22T08:00:00.754231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871379+0000","last_change":"2023-06-22T07:59:59.208263+0000","last_active":"2023-06-22T07:59:59.871379+0000","last_peered":"2023-06-22T07:59:59.871379+0000","last_clean":"2023-06-22T07:59:59.871379+0000","last_became_active":"2023-06-22T07:59:59.207691+0000","last_became_peered":"2023-06-22T07:59:59.207691+0000","last_unstale":"2023-06-22T07:59:59.871379+0000","last_undegraded":"2023-06-22T07:59:59.871379+0000","last_fullsized":"2023-06-22T07:59:59.871379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:37:57.170857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,2],"acting":[3,7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.564652+0000","last_change":"2023-06-22T07:59:54.644570+0000","last_active":"2023-06-22T08:00:41.564652+0000","last_peered":"2023-06-22T08:00:41.564652+0000","last_clean":"2023-06-22T08:00:41.564652+0000","last_became_active":"2023-06-22T07:59:54.644218+0000","last_became_peered":"2023-06-22T07:59:54.644218+0000","last_unstale":"2023-06-22T08:00:41.564652+0000","last_undegraded":"2023-06-22T08:00:41.564652+0000","last_fullsized":"2023-06-22T08:00:41.564652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:05:00.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754571+0000","last_change":"2023-06-22T07:59:49.655601+0000","last_active":"2023-06-22T08:00:00.754571+0000","last_peered":"2023-06-22T08:00:00.754571+0000","last_clean":"2023-06-22T08:00:00.754571+0000","last_became_active":"2023-06-22T07:59:49.655328+0000","last_became_peered":"2023-06-22T07:59:49.655328+0000","last_unstale":"2023-06-22T08:00:00.754571+0000","last_undegraded":"2023-06-22T08:00:00.754571+0000","last_fullsized":"2023-06-22T08:00:00.754571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:29:09.967964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755018+0000","last_change":"2023-06-22T07:59:55.163241+0000","last_active":"2023-06-22T08:00:00.755018+0000","last_peered":"2023-06-22T08:00:00.755018+0000","last_clean":"2023-06-22T08:00:00.755018+0000","last_became_active":"2023-06-22T07:59:55.162949+0000","last_became_peered":"2023-06-22T07:59:55.162949+0000","last_unstale":"2023-06-22T08:00:00.755018+0000","last_undegraded":"2023-06-22T08:00:00.755018+0000","last_fullsized":"2023-06-22T08:00:00.755018+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:51.267726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755485+0000","last_change":"2023-06-22T07:59:59.201187+0000","last_active":"2023-06-22T08:00:00.755485+0000","last_peered":"2023-06-22T08:00:00.755485+0000","last_clean":"2023-06-22T08:00:00.755485+0000","last_became_active":"2023-06-22T07:59:59.200688+0000","last_became_peered":"2023-06-22T07:59:59.200688+0000","last_unstale":"2023-06-22T08:00:00.755485+0000","last_undegraded":"2023-06-22T08:00:00.755485+0000","last_fullsized":"2023-06-22T08:00:00.755485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:52:15.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2,6],"acting":[4,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871926+0000","last_change":"2023-06-22T07:59:49.655839+0000","last_active":"2023-06-22T07:59:59.871926+0000","last_peered":"2023-06-22T07:59:59.871926+0000","last_clean":"2023-06-22T07:59:59.871926+0000","last_became_active":"2023-06-22T07:59:49.655494+0000","last_became_peered":"2023-06-22T07:59:49.655494+0000","last_unstale":"2023-06-22T07:59:59.871926+0000","last_undegraded":"2023-06-22T07:59:59.871926+0000","last_fullsized":"2023-06-22T07:59:59.871926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:01:23.950347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.490383+0000","last_change":"2023-06-22T07:59:54.641464+0000","last_active":"2023-06-22T08:00:36.490383+0000","last_peered":"2023-06-22T08:00:36.490383+0000","last_clean":"2023-06-22T08:00:36.490383+0000","last_became_active":"2023-06-22T07:59:54.641161+0000","last_became_peered":"2023-06-22T07:59:54.641161+0000","last_unstale":"2023-06-22T08:00:36.490383+0000","last_undegraded":"2023-06-22T08:00:36.490383+0000","last_fullsized":"2023-06-22T08:00:36.490383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:46.346592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708316+0000","last_change":"2023-06-22T07:59:55.164629+0000","last_active":"2023-06-22T07:59:59.708316+0000","last_peered":"2023-06-22T07:59:59.708316+0000","last_clean":"2023-06-22T07:59:59.708316+0000","last_became_active":"2023-06-22T07:59:55.164329+0000","last_became_peered":"2023-06-22T07:59:55.164329+0000","last_unstale":"2023-06-22T07:59:59.708316+0000","last_undegraded":"2023-06-22T07:59:59.708316+0000","last_fullsized":"2023-06-22T07:59:59.708316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754950+0000","last_change":"2023-06-22T07:59:59.201778+0000","last_active":"2023-06-22T08:00:00.754950+0000","last_peered":"2023-06-22T08:00:00.754950+0000","last_clean":"2023-06-22T08:00:00.754950+0000","last_became_active":"2023-06-22T07:59:59.201326+0000","last_became_peered":"2023-06-22T07:59:59.201326+0000","last_unstale":"2023-06-22T08:00:00.754950+0000","last_undegraded":"2023-06-22T08:00:00.754950+0000","last_fullsized":"2023-06-22T08:00:00.754950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:17:31.817024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.703139+0000","last_change":"2023-06-22T07:59:54.644816+0000","last_active":"2023-06-22T08:00:25.703139+0000","last_peered":"2023-06-22T08:00:25.703139+0000","last_clean":"2023-06-22T08:00:25.703139+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:25.703139+0000","last_undegraded":"2023-06-22T08:00:25.703139+0000","last_fullsized":"2023-06-22T08:00:25.703139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:33.819054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753735+0000","last_change":"2023-06-22T07:59:55.167979+0000","last_active":"2023-06-22T08:00:00.753735+0000","last_peered":"2023-06-22T08:00:00.753735+0000","last_clean":"2023-06-22T08:00:00.753735+0000","last_became_active":"2023-06-22T07:59:55.163775+0000","last_became_peered":"2023-06-22T07:59:55.163775+0000","last_unstale":"2023-06-22T08:00:00.753735+0000","last_undegraded":"2023-06-22T08:00:00.753735+0000","last_fullsized":"2023-06-22T08:00:00.753735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155135+0000","last_change":"2023-06-22T07:59:59.194581+0000","last_active":"2023-06-22T08:00:00.155135+0000","last_peered":"2023-06-22T08:00:00.155135+0000","last_clean":"2023-06-22T08:00:00.155135+0000","last_became_active":"2023-06-22T07:59:59.194135+0000","last_became_peered":"2023-06-22T07:59:59.194135+0000","last_unstale":"2023-06-22T08:00:00.155135+0000","last_undegraded":"2023-06-22T08:00:00.155135+0000","last_fullsized":"2023-06-22T08:00:00.155135+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.565859+0000","last_change":"2023-06-22T07:59:54.651138+0000","last_active":"2023-06-22T08:00:41.565859+0000","last_peered":"2023-06-22T08:00:41.565859+0000","last_clean":"2023-06-22T08:00:41.565859+0000","last_became_active":"2023-06-22T07:59:54.650803+0000","last_became_peered":"2023-06-22T07:59:54.650803+0000","last_unstale":"2023-06-22T08:00:41.565859+0000","last_undegraded":"2023-06-22T08:00:41.565859+0000","last_fullsized":"2023-06-22T08:00:41.565859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:45.982711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708617+0000","last_change":"2023-06-22T07:59:55.165814+0000","last_active":"2023-06-22T07:59:59.708617+0000","last_peered":"2023-06-22T07:59:59.708617+0000","last_clean":"2023-06-22T07:59:59.708617+0000","last_became_active":"2023-06-22T07:59:55.165270+0000","last_became_peered":"2023-06-22T07:59:55.165270+0000","last_unstale":"2023-06-22T07:59:59.708617+0000","last_undegraded":"2023-06-22T07:59:59.708617+0000","last_fullsized":"2023-06-22T07:59:59.708617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708465+0000","last_change":"2023-06-22T07:59:59.212042+0000","last_active":"2023-06-22T07:59:59.708465+0000","last_peered":"2023-06-22T07:59:59.708465+0000","last_clean":"2023-06-22T07:59:59.708465+0000","last_became_active":"2023-06-22T07:59:59.211666+0000","last_became_peered":"2023-06-22T07:59:59.211666+0000","last_unstale":"2023-06-22T07:59:59.708465+0000","last_undegraded":"2023-06-22T07:59:59.708465+0000","last_fullsized":"2023-06-22T07:59:59.708465+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,4],"acting":[2,7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"28'12","reported_seq":33,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.571232+0000","last_change":"2023-06-22T07:59:54.650251+0000","last_active":"2023-06-22T08:00:41.571232+0000","last_peered":"2023-06-22T08:00:41.571232+0000","last_clean":"2023-06-22T08:00:41.571232+0000","last_became_active":"2023-06-22T07:59:54.649993+0000","last_became_peered":"2023-06-22T07:59:54.649993+0000","last_unstale":"2023-06-22T08:00:41.571232+0000","last_undegraded":"2023-06-22T08:00:41.571232+0000","last_fullsized":"2023-06-22T08:00:41.571232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:20:20.733896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708375+0000","last_change":"2023-06-22T07:59:55.172595+0000","last_active":"2023-06-22T07:59:59.708375+0000","last_peered":"2023-06-22T07:59:59.708375+0000","last_clean":"2023-06-22T07:59:59.708375+0000","last_became_active":"2023-06-22T07:59:55.165873+0000","last_became_peered":"2023-06-22T07:59:55.165873+0000","last_unstale":"2023-06-22T07:59:59.708375+0000","last_undegraded":"2023-06-22T07:59:59.708375+0000","last_fullsized":"2023-06-22T07:59:59.708375+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:02.592807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154969+0000","last_change":"2023-06-22T07:59:59.192708+0000","last_active":"2023-06-22T08:00:00.154969+0000","last_peered":"2023-06-22T08:00:00.154969+0000","last_clean":"2023-06-22T08:00:00.154969+0000","last_became_active":"2023-06-22T07:59:59.191693+0000","last_became_peered":"2023-06-22T07:59:59.191693+0000","last_unstale":"2023-06-22T08:00:00.154969+0000","last_undegraded":"2023-06-22T08:00:00.154969+0000","last_fullsized":"2023-06-22T08:00:00.154969+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T12:59:33.454962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5,7],"acting":[1,0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.151569+0000","last_change":"2023-06-22T07:59:54.651179+0000","last_active":"2023-06-22T08:00:00.151569+0000","last_peered":"2023-06-22T08:00:00.151569+0000","last_clean":"2023-06-22T08:00:00.151569+0000","last_became_active":"2023-06-22T07:59:54.650852+0000","last_became_peered":"2023-06-22T07:59:54.650852+0000","last_unstale":"2023-06-22T08:00:00.151569+0000","last_undegraded":"2023-06-22T08:00:00.151569+0000","last_fullsized":"2023-06-22T08:00:00.151569+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:01.351861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754043+0000","last_change":"2023-06-22T07:59:55.163050+0000","last_active":"2023-06-22T08:00:00.754043+0000","last_peered":"2023-06-22T08:00:00.754043+0000","last_clean":"2023-06-22T08:00:00.754043+0000","last_became_active":"2023-06-22T07:59:55.162754+0000","last_became_peered":"2023-06-22T07:59:55.162754+0000","last_unstale":"2023-06-22T08:00:00.754043+0000","last_undegraded":"2023-06-22T08:00:00.754043+0000","last_fullsized":"2023-06-22T08:00:00.754043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754797+0000","last_change":"2023-06-22T07:59:59.209558+0000","last_active":"2023-06-22T08:00:00.754797+0000","last_peered":"2023-06-22T08:00:00.754797+0000","last_clean":"2023-06-22T08:00:00.754797+0000","last_became_active":"2023-06-22T07:59:59.209129+0000","last_became_peered":"2023-06-22T07:59:59.209129+0000","last_unstale":"2023-06-22T08:00:00.754797+0000","last_undegraded":"2023-06-22T08:00:00.754797+0000","last_fullsized":"2023-06-22T08:00:00.754797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:21.581063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1,0],"acting":[4,3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"28'13","reported_seq":34,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.566206+0000","last_change":"2023-06-22T07:59:54.652059+0000","last_active":"2023-06-22T08:00:41.566206+0000","last_peered":"2023-06-22T08:00:41.566206+0000","last_clean":"2023-06-22T08:00:41.566206+0000","last_became_active":"2023-06-22T07:59:54.651720+0000","last_became_peered":"2023-06-22T07:59:54.651720+0000","last_unstale":"2023-06-22T08:00:41.566206+0000","last_undegraded":"2023-06-22T08:00:41.566206+0000","last_fullsized":"2023-06-22T08:00:41.566206+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:31.908657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708158+0000","last_change":"2023-06-22T07:59:55.172794+0000","last_active":"2023-06-22T07:59:59.708158+0000","last_peered":"2023-06-22T07:59:59.708158+0000","last_clean":"2023-06-22T07:59:59.708158+0000","last_became_active":"2023-06-22T07:59:55.172527+0000","last_became_peered":"2023-06-22T07:59:55.172527+0000","last_unstale":"2023-06-22T07:59:59.708158+0000","last_undegraded":"2023-06-22T07:59:59.708158+0000","last_fullsized":"2023-06-22T07:59:59.708158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155457+0000","last_change":"2023-06-22T07:59:55.165758+0000","last_active":"2023-06-22T08:00:00.155457+0000","last_peered":"2023-06-22T08:00:00.155457+0000","last_clean":"2023-06-22T08:00:00.155457+0000","last_became_active":"2023-06-22T07:59:55.165117+0000","last_became_peered":"2023-06-22T07:59:55.165117+0000","last_unstale":"2023-06-22T08:00:00.155457+0000","last_undegraded":"2023-06-22T08:00:00.155457+0000","last_fullsized":"2023-06-22T08:00:00.155457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:02.357853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753486+0000","last_change":"2023-06-22T07:59:55.162439+0000","last_active":"2023-06-22T08:00:00.753486+0000","last_peered":"2023-06-22T08:00:00.753486+0000","last_clean":"2023-06-22T08:00:00.753486+0000","last_became_active":"2023-06-22T07:59:55.162072+0000","last_became_peered":"2023-06-22T07:59:55.162072+0000","last_unstale":"2023-06-22T08:00:00.753486+0000","last_undegraded":"2023-06-22T08:00:00.753486+0000","last_fullsized":"2023-06-22T08:00:00.753486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157229+0000","last_change":"2023-06-22T07:59:55.159299+0000","last_active":"2023-06-22T08:00:00.157229+0000","last_peered":"2023-06-22T08:00:00.157229+0000","last_clean":"2023-06-22T08:00:00.157229+0000","last_became_active":"2023-06-22T07:59:55.158893+0000","last_became_peered":"2023-06-22T07:59:55.158893+0000","last_unstale":"2023-06-22T08:00:00.157229+0000","last_undegraded":"2023-06-22T08:00:00.157229+0000","last_fullsized":"2023-06-22T08:00:00.157229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709716+0000","last_change":"2023-06-22T07:59:55.162520+0000","last_active":"2023-06-22T07:59:59.709716+0000","last_peered":"2023-06-22T07:59:59.709716+0000","last_clean":"2023-06-22T07:59:59.709716+0000","last_became_active":"2023-06-22T07:59:55.162211+0000","last_became_peered":"2023-06-22T07:59:55.162211+0000","last_unstale":"2023-06-22T07:59:59.709716+0000","last_undegraded":"2023-06-22T07:59:59.709716+0000","last_fullsized":"2023-06-22T07:59:59.709716+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:13:00.662547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708947+0000","last_change":"2023-06-22T07:59:55.165364+0000","last_active":"2023-06-22T07:59:59.708947+0000","last_peered":"2023-06-22T07:59:59.708947+0000","last_clean":"2023-06-22T07:59:59.708947+0000","last_became_active":"2023-06-22T07:59:55.165021+0000","last_became_peered":"2023-06-22T07:59:55.165021+0000","last_unstale":"2023-06-22T07:59:59.708947+0000","last_undegraded":"2023-06-22T07:59:59.708947+0000","last_fullsized":"2023-06-22T07:59:59.708947+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:09:07.008945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":10282,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":211,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476755,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":344064,"data_stored":128982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.64400000000000002}]},{"osd":1,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.437},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.55000000000000004}]},{"osd":3,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.49099999999999999}]},{"osd":6,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.46400000000000002}]},{"osd":7,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.58499999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542162,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":344064,"data_stored":128648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.503}]},{"osd":4,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.73599999999999999}]},{"osd":6,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.748}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":319488,"data_stored":124784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.4989999999999997,"5min":6.4989999999999997,"15min":6.4989999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":10.401,"5min":10.401,"15min":10.401},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":10.173999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":10.291,"5min":10.291,"15min":10.291},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":10.064,"5min":10.064,"15min":10.064},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.39900000000000002}]},{"osd":6,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.378},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.64100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":921600,"data_stored":715384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.312},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.82699999999999996}]},{"osd":2,"last update":"Thu Jun 22 08:00:47 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57499999999999996}]},{"osd":3,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.53800000000000003}]},{"osd":4,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67000000000000004}]},{"osd":6,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.73999999999999999}]},{"osd":7,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.64400000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":921600,"data_stored":715472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.623}]},{"osd":2,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.74199999999999999}]},{"osd":4,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.79000000000000004}]},{"osd":6,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.753}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":335872,"data_stored":125148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.7000000000000002,"5min":7.7000000000000002,"15min":7.7000000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.8010000000000002,"5min":3.8010000000000002,"15min":3.8010000000000002},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jun 22 08:00:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":9.8460000000000001,"5min":9.8460000000000001,"15min":9.8460000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.7210000000000001,"5min":2.7210000000000001,"15min":2.7210000000000001},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.867,"5min":1.867,"15min":1.867},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":9.9019999999999992,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":0.502}]},{"osd":4,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48599999999999999}]}]},{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":307200,"data_stored":124680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51500000000000001}]},{"osd":3,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.496},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.36499999999999999}]},{"osd":5,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.47099999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":348160,"data_stored":126666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.917,"5min":1.917,"15min":1.917},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.8099999999999996,"5min":6.8099999999999996,"15min":6.8099999999999996},"last":0.32600000000000001}]},{"osd":2,"last update":"Thu Jun 22 08:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.6589999999999998},"last":0.438},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-22T08:00:57.425 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-22T08:00:57.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.652+0000 7f20ca7ff700 1 Processor -- start 2023-06-22T08:00:57.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.652+0000 7f20ca7ff700 1 -- start start 2023-06-22T08:00:57.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.654+0000 7f20ca7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f20c40c8bd0 0x7f20c40aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.654+0000 7f20ca7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f20c40cc640 0x7f20c40cca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.654+0000 7f20ca7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c40c7400 con 0x7f20c4152dd0 2023-06-22T08:00:57.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.654+0000 7f20ca7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c40c81c0 con 0x7f20c40c8bd0 2023-06-22T08:00:57.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.654+0000 7f20ca7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c40c84a0 con 0x7f20c40cc640 2023-06-22T08:00:57.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.654+0000 7f20c3fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f20c4152dd0 0x7f20c406afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:45558/0 (socket says 172.21.15.23:45558) 2023-06-22T08:00:57.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.654+0000 7f20c3fff700 1 -- 172.21.15.23:0/2932357284 learned_addr learned my addr 172.21.15.23:0/2932357284 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:57.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.654+0000 7f20c37fe700 1 --2- 172.21.15.23:0/2932357284 >> v2:172.21.15.23:3300/0 conn(0x7f20c40c8bd0 0x7f20c40aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.654+0000 7f20c8d9c700 1 --2- 172.21.15.23:0/2932357284 >> v2:172.21.15.67:3300/0 conn(0x7f20c40cc640 0x7f20c40cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.655+0000 7f20c8d9c700 1 -- 172.21.15.23:0/2932357284 >> v2:172.21.15.23:3300/0 conn(0x7f20c40c8bd0 msgr2=0x7f20c40aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.655+0000 7f20c8d9c700 1 --2- 172.21.15.23:0/2932357284 >> v2:172.21.15.23:3300/0 conn(0x7f20c40c8bd0 0x7f20c40aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.655+0000 7f20c8d9c700 1 -- 172.21.15.23:0/2932357284 >> v1:172.21.15.23:6789/0 conn(0x7f20c4152dd0 legacy=0x7f20c406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.655+0000 7f20c8d9c700 1 -- 172.21.15.23:0/2932357284 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40cd370 con 0x7f20c40cc640 2023-06-22T08:00:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.655+0000 7f20c37fe700 1 --2- 172.21.15.23:0/2932357284 >> v2:172.21.15.23:3300/0 conn(0x7f20c40c8bd0 0x7f20c40aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.655+0000 7f20c2ffd700 1 -- 172.21.15.23:0/2932357284 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591987707 0 0) 0x7f20c40c7400 con 0x7f20c4152dd0 2023-06-22T08:00:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.655+0000 7f20c8d9c700 1 --2- 172.21.15.23:0/2932357284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c40cc640 0x7f20c40cca80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f20c40737d0 tx=0x7f20bc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=416a90f5f0fcd129 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.655+0000 7f20c2ffd700 1 -- 172.21.15.23:0/2932357284 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc00e070 con 0x7f20c40cc640 2023-06-22T08:00:57.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.655+0000 7f20c2ffd700 1 -- 172.21.15.23:0/2932357284 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20bc00bb20 con 0x7f20c40cc640 2023-06-22T08:00:57.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.655+0000 7f20c2ffd700 1 -- 172.21.15.23:0/2932357284 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc012430 con 0x7f20c40cc640 2023-06-22T08:00:57.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.656+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2932357284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c40cc640 msgr2=0x7f20c40cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.656+0000 7f20ca7ff700 1 --2- 172.21.15.23:0/2932357284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c40cc640 0x7f20c40cca80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f20c40737d0 tx=0x7f20bc00b330 comp rx=0 tx=0).stop 2023-06-22T08:00:57.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.656+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2932357284 shutdown_connections 2023-06-22T08:00:57.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.656+0000 7f20ca7ff700 1 --2- 172.21.15.23:0/2932357284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c40cc640 0x7f20c40cca80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.656+0000 7f20ca7ff700 1 --2- 172.21.15.23:0/2932357284 >> v2:172.21.15.23:3300/0 conn(0x7f20c40c8bd0 0x7f20c40aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.656+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2932357284 >> 172.21.15.23:0/2932357284 conn(0x7f20c406fb70 msgr2=0x7f20c406ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:57.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.656+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2932357284 shutdown_connections 2023-06-22T08:00:57.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.656+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2932357284 wait complete. 2023-06-22T08:00:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.658+0000 7f20ca7ff700 1 Processor -- start 2023-06-22T08:00:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.658+0000 7f20ca7ff700 1 -- start start 2023-06-22T08:00:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20ca7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c40c8bd0 0x7f20c406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20ca7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c40cc640 0x7f20c40b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20ca7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20c4152dd0 0x7f20c41f12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20ca7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40b2770 con 0x7f20c4152dd0 2023-06-22T08:00:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20ca7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40c2c40 con 0x7f20c40c8bd0 2023-06-22T08:00:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20ca7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40b1f60 con 0x7f20c40cc640 2023-06-22T08:00:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20c3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c40c8bd0 0x7f20c406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20c8d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20c4152dd0 0x7f20c41f12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20c3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c40c8bd0 0x7f20c406e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51654/0 (socket says 172.21.15.23:51654) 2023-06-22T08:00:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20c8d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20c4152dd0 0x7f20c41f12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52844/0 (socket says 172.21.15.23:52844) 2023-06-22T08:00:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20c3fff700 1 -- 172.21.15.23:0/2219838226 learned_addr learned my addr 172.21.15.23:0/2219838226 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:57.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.659+0000 7f20c37fe700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c40cc640 0x7f20c40b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20c37fe700 1 -- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c40c8bd0 msgr2=0x7f20c406e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20c37fe700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c40c8bd0 0x7f20c406e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20c37fe700 1 -- 172.21.15.23:0/2219838226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20c4152dd0 msgr2=0x7f20c41f12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20c37fe700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20c4152dd0 0x7f20c41f12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20c37fe700 1 -- 172.21.15.23:0/2219838226 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b80094a0 con 0x7f20c40cc640 2023-06-22T08:00:57.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20c37fe700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c40cc640 0x7f20c40b0180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f20b800e9e0 tx=0x7f20b800ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:57.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20c0ff9700 1 -- 172.21.15.23:0/2219838226 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b800ee80 con 0x7f20c40cc640 2023-06-22T08:00:57.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20c0ff9700 1 -- 172.21.15.23:0/2219838226 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b800c850 con 0x7f20c40cc640 2023-06-22T08:00:57.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2219838226 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20bc009a70 con 0x7f20c40cc640 2023-06-22T08:00:57.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20c0ff9700 1 -- 172.21.15.23:0/2219838226 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b800cc00 con 0x7f20c40cc640 2023-06-22T08:00:57.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.660+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2219838226 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c4158a00 con 0x7f20c40cc640 2023-06-22T08:00:57.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.662+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2219838226 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c4158b90 con 0x7f20c40cc640 2023-06-22T08:00:57.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.663+0000 7f20c0ff9700 1 -- 172.21.15.23:0/2219838226 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f20b800cd60 con 0x7f20c40cc640 2023-06-22T08:00:57.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.663+0000 7f20c0ff9700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f20b006ace0 0x7f20b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:57.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.664+0000 7f20c0ff9700 1 -- 172.21.15.23:0/2219838226 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f20b8089b90 con 0x7f20c40cc640 2023-06-22T08:00:57.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.664+0000 7f20c3fff700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f20b006ace0 0x7f20b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:57.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.664+0000 7f20c3fff700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f20b006ace0 0x7f20b006d190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f20b4007fd0 tx=0x7f20b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:57.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.668+0000 7f20c0ff9700 1 -- 172.21.15.23:0/2219838226 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f20b80548b0 con 0x7f20c40cc640 2023-06-22T08:00:57.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.872+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2219838226 --> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20c40c7650 con 0x7f20b006ace0 2023-06-22T08:00:57.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.884+0000 7f20c0ff9700 1 -- 172.21.15.23:0/2219838226 <== mgr.4100 v2:172.21.15.67:6832/3180103139 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+392072 (secure 0 0 0) 0x7f20b4001580 con 0x7f20b006ace0 2023-06-22T08:00:57.891 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:57.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f20b006ace0 msgr2=0x7f20b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f20b006ace0 0x7f20b006d190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f20b4007fd0 tx=0x7f20b4009040 comp rx=0 tx=0).stop 2023-06-22T08:00:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c40cc640 msgr2=0x7f20c40b0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c40cc640 0x7f20c40b0180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f20b800e9e0 tx=0x7f20b800ecf0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2219838226 shutdown_connections 2023-06-22T08:00:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20c4152dd0 0x7f20c41f12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f20b006ace0 0x7f20b006d190 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c40c8bd0 0x7f20c406e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 --2- 172.21.15.23:0/2219838226 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c40cc640 0x7f20c40b0180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2219838226 >> 172.21.15.23:0/2219838226 conn(0x7f20c406fb70 msgr2=0x7f20c40ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2219838226 shutdown_connections 2023-06-22T08:00:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:57.889+0000 7f20ca7ff700 1 -- 172.21.15.23:0/2219838226 wait complete. 2023-06-22T08:00:57.895 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-06-22T08:00:57.911 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2023-06-22T08:00:56.571499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600650,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":370,"num_write_kb":2485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":3752,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3842048,"data_stored":2189764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1492,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003943"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709823+0000","last_change":"2023-06-22T07:59:55.165740+0000","last_active":"2023-06-22T07:59:59.709823+0000","last_peered":"2023-06-22T07:59:59.709823+0000","last_clean":"2023-06-22T07:59:59.709823+0000","last_became_active":"2023-06-22T07:59:55.165398+0000","last_became_peered":"2023-06-22T07:59:55.165398+0000","last_unstale":"2023-06-22T07:59:59.709823+0000","last_undegraded":"2023-06-22T07:59:59.709823+0000","last_fullsized":"2023-06-22T07:59:59.709823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:37.596656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709026+0000","last_change":"2023-06-22T07:59:55.174603+0000","last_active":"2023-06-22T07:59:59.709026+0000","last_peered":"2023-06-22T07:59:59.709026+0000","last_clean":"2023-06-22T07:59:59.709026+0000","last_became_active":"2023-06-22T07:59:55.174346+0000","last_became_peered":"2023-06-22T07:59:55.174346+0000","last_unstale":"2023-06-22T07:59:59.709026+0000","last_undegraded":"2023-06-22T07:59:59.709026+0000","last_fullsized":"2023-06-22T07:59:59.709026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157321+0000","last_change":"2023-06-22T07:59:55.167525+0000","last_active":"2023-06-22T08:00:00.157321+0000","last_peered":"2023-06-22T08:00:00.157321+0000","last_clean":"2023-06-22T08:00:00.157321+0000","last_became_active":"2023-06-22T07:59:55.167181+0000","last_became_peered":"2023-06-22T07:59:55.167181+0000","last_unstale":"2023-06-22T08:00:00.157321+0000","last_undegraded":"2023-06-22T08:00:00.157321+0000","last_fullsized":"2023-06-22T08:00:00.157321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157927+0000","last_change":"2023-06-22T07:59:55.160549+0000","last_active":"2023-06-22T08:00:00.157927+0000","last_peered":"2023-06-22T08:00:00.157927+0000","last_clean":"2023-06-22T08:00:00.157927+0000","last_became_active":"2023-06-22T07:59:55.160239+0000","last_became_peered":"2023-06-22T07:59:55.160239+0000","last_unstale":"2023-06-22T08:00:00.157927+0000","last_undegraded":"2023-06-22T08:00:00.157927+0000","last_fullsized":"2023-06-22T08:00:00.157927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754390+0000","last_change":"2023-06-22T07:59:55.173006+0000","last_active":"2023-06-22T08:00:00.754390+0000","last_peered":"2023-06-22T08:00:00.754390+0000","last_clean":"2023-06-22T08:00:00.754390+0000","last_became_active":"2023-06-22T07:59:55.172578+0000","last_became_peered":"2023-06-22T07:59:55.172578+0000","last_unstale":"2023-06-22T08:00:00.754390+0000","last_undegraded":"2023-06-22T08:00:00.754390+0000","last_fullsized":"2023-06-22T08:00:00.754390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:12:44.799856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871378+0000","last_change":"2023-06-22T07:59:55.173858+0000","last_active":"2023-06-22T07:59:59.871378+0000","last_peered":"2023-06-22T07:59:59.871378+0000","last_clean":"2023-06-22T07:59:59.871378+0000","last_became_active":"2023-06-22T07:59:55.173599+0000","last_became_peered":"2023-06-22T07:59:55.173599+0000","last_unstale":"2023-06-22T07:59:59.871378+0000","last_undegraded":"2023-06-22T07:59:59.871378+0000","last_fullsized":"2023-06-22T07:59:59.871378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:37.597195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871174+0000","last_change":"2023-06-22T07:59:55.168354+0000","last_active":"2023-06-22T07:59:59.871174+0000","last_peered":"2023-06-22T07:59:59.871174+0000","last_clean":"2023-06-22T07:59:59.871174+0000","last_became_active":"2023-06-22T07:59:55.168050+0000","last_became_peered":"2023-06-22T07:59:55.168050+0000","last_unstale":"2023-06-22T07:59:59.871174+0000","last_undegraded":"2023-06-22T07:59:59.871174+0000","last_fullsized":"2023-06-22T07:59:59.871174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:08:45.847378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155602+0000","last_change":"2023-06-22T07:59:55.159738+0000","last_active":"2023-06-22T08:00:00.155602+0000","last_peered":"2023-06-22T08:00:00.155602+0000","last_clean":"2023-06-22T08:00:00.155602+0000","last_became_active":"2023-06-22T07:59:55.159501+0000","last_became_peered":"2023-06-22T07:59:55.159501+0000","last_unstale":"2023-06-22T08:00:00.155602+0000","last_undegraded":"2023-06-22T08:00:00.155602+0000","last_fullsized":"2023-06-22T08:00:00.155602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:55.160768+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:55.160508+0000","last_became_peered":"2023-06-22T07:59:55.160508+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:55:48.713145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155821+0000","last_change":"2023-06-22T07:59:55.166546+0000","last_active":"2023-06-22T08:00:00.155821+0000","last_peered":"2023-06-22T08:00:00.155821+0000","last_clean":"2023-06-22T08:00:00.155821+0000","last_became_active":"2023-06-22T07:59:55.166311+0000","last_became_peered":"2023-06-22T07:59:55.166311+0000","last_unstale":"2023-06-22T08:00:00.155821+0000","last_undegraded":"2023-06-22T08:00:00.155821+0000","last_fullsized":"2023-06-22T08:00:00.155821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:35.822548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155943+0000","last_change":"2023-06-22T07:59:55.162295+0000","last_active":"2023-06-22T08:00:00.155943+0000","last_peered":"2023-06-22T08:00:00.155943+0000","last_clean":"2023-06-22T08:00:00.155943+0000","last_became_active":"2023-06-22T07:59:55.162010+0000","last_became_peered":"2023-06-22T07:59:55.162010+0000","last_unstale":"2023-06-22T08:00:00.155943+0000","last_undegraded":"2023-06-22T08:00:00.155943+0000","last_fullsized":"2023-06-22T08:00:00.155943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709563+0000","last_change":"2023-06-22T07:59:55.158990+0000","last_active":"2023-06-22T07:59:59.709563+0000","last_peered":"2023-06-22T07:59:59.709563+0000","last_clean":"2023-06-22T07:59:59.709563+0000","last_became_active":"2023-06-22T07:59:55.158614+0000","last_became_peered":"2023-06-22T07:59:55.158614+0000","last_unstale":"2023-06-22T07:59:59.709563+0000","last_undegraded":"2023-06-22T07:59:59.709563+0000","last_fullsized":"2023-06-22T07:59:59.709563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755166+0000","last_change":"2023-06-22T07:59:55.161453+0000","last_active":"2023-06-22T08:00:00.755166+0000","last_peered":"2023-06-22T08:00:00.755166+0000","last_clean":"2023-06-22T08:00:00.755166+0000","last_became_active":"2023-06-22T07:59:55.161168+0000","last_became_peered":"2023-06-22T07:59:55.161168+0000","last_unstale":"2023-06-22T08:00:00.755166+0000","last_undegraded":"2023-06-22T08:00:00.755166+0000","last_fullsized":"2023-06-22T08:00:00.755166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155022+0000","last_change":"2023-06-22T07:59:55.164270+0000","last_active":"2023-06-22T08:00:00.155022+0000","last_peered":"2023-06-22T08:00:00.155022+0000","last_clean":"2023-06-22T08:00:00.155022+0000","last_became_active":"2023-06-22T07:59:55.163993+0000","last_became_peered":"2023-06-22T07:59:55.163993+0000","last_unstale":"2023-06-22T08:00:00.155022+0000","last_undegraded":"2023-06-22T08:00:00.155022+0000","last_fullsized":"2023-06-22T08:00:00.155022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:52.051830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157430+0000","last_change":"2023-06-22T07:59:55.169039+0000","last_active":"2023-06-22T08:00:00.157430+0000","last_peered":"2023-06-22T08:00:00.157430+0000","last_clean":"2023-06-22T08:00:00.157430+0000","last_became_active":"2023-06-22T07:59:55.168698+0000","last_became_peered":"2023-06-22T07:59:55.168698+0000","last_unstale":"2023-06-22T08:00:00.157430+0000","last_undegraded":"2023-06-22T08:00:00.157430+0000","last_fullsized":"2023-06-22T08:00:00.157430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871035+0000","last_change":"2023-06-22T07:59:55.158014+0000","last_active":"2023-06-22T07:59:59.871035+0000","last_peered":"2023-06-22T07:59:59.871035+0000","last_clean":"2023-06-22T07:59:59.871035+0000","last_became_active":"2023-06-22T07:59:55.157562+0000","last_became_peered":"2023-06-22T07:59:55.157562+0000","last_unstale":"2023-06-22T07:59:59.871035+0000","last_undegraded":"2023-06-22T07:59:59.871035+0000","last_fullsized":"2023-06-22T07:59:59.871035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:08:33.856001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754963+0000","last_change":"2023-06-22T07:59:55.165957+0000","last_active":"2023-06-22T08:00:00.754963+0000","last_peered":"2023-06-22T08:00:00.754963+0000","last_clean":"2023-06-22T08:00:00.754963+0000","last_became_active":"2023-06-22T07:59:55.165643+0000","last_became_peered":"2023-06-22T07:59:55.165643+0000","last_unstale":"2023-06-22T08:00:00.754963+0000","last_undegraded":"2023-06-22T08:00:00.754963+0000","last_fullsized":"2023-06-22T08:00:00.754963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:10.134743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157391+0000","last_change":"2023-06-22T07:59:55.159858+0000","last_active":"2023-06-22T08:00:00.157391+0000","last_peered":"2023-06-22T08:00:00.157391+0000","last_clean":"2023-06-22T08:00:00.157391+0000","last_became_active":"2023-06-22T07:59:55.159444+0000","last_became_peered":"2023-06-22T07:59:55.159444+0000","last_unstale":"2023-06-22T08:00:00.157391+0000","last_undegraded":"2023-06-22T08:00:00.157391+0000","last_fullsized":"2023-06-22T08:00:00.157391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709424+0000","last_change":"2023-06-22T07:59:55.157162+0000","last_active":"2023-06-22T07:59:59.709424+0000","last_peered":"2023-06-22T07:59:59.709424+0000","last_clean":"2023-06-22T07:59:59.709424+0000","last_became_active":"2023-06-22T07:59:55.156882+0000","last_became_peered":"2023-06-22T07:59:55.156882+0000","last_unstale":"2023-06-22T07:59:59.709424+0000","last_undegraded":"2023-06-22T07:59:59.709424+0000","last_fullsized":"2023-06-22T07:59:59.709424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155892+0000","last_change":"2023-06-22T07:59:55.163834+0000","last_active":"2023-06-22T08:00:00.155892+0000","last_peered":"2023-06-22T08:00:00.155892+0000","last_clean":"2023-06-22T08:00:00.155892+0000","last_became_active":"2023-06-22T07:59:55.163580+0000","last_became_peered":"2023-06-22T07:59:55.163580+0000","last_unstale":"2023-06-22T08:00:00.155892+0000","last_undegraded":"2023-06-22T08:00:00.155892+0000","last_fullsized":"2023-06-22T08:00:00.155892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155285+0000","last_change":"2023-06-22T07:59:55.164569+0000","last_active":"2023-06-22T08:00:00.155285+0000","last_peered":"2023-06-22T08:00:00.155285+0000","last_clean":"2023-06-22T08:00:00.155285+0000","last_became_active":"2023-06-22T07:59:55.164286+0000","last_became_peered":"2023-06-22T07:59:55.164286+0000","last_unstale":"2023-06-22T08:00:00.155285+0000","last_undegraded":"2023-06-22T08:00:00.155285+0000","last_fullsized":"2023-06-22T08:00:00.155285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:17.386317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754137+0000","last_change":"2023-06-22T07:59:55.159883+0000","last_active":"2023-06-22T08:00:00.754137+0000","last_peered":"2023-06-22T08:00:00.754137+0000","last_clean":"2023-06-22T08:00:00.754137+0000","last_became_active":"2023-06-22T07:59:55.159588+0000","last_became_peered":"2023-06-22T07:59:55.159588+0000","last_unstale":"2023-06-22T08:00:00.754137+0000","last_undegraded":"2023-06-22T08:00:00.754137+0000","last_fullsized":"2023-06-22T08:00:00.754137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754941+0000","last_change":"2023-06-22T07:59:55.160638+0000","last_active":"2023-06-22T08:00:00.754941+0000","last_peered":"2023-06-22T08:00:00.754941+0000","last_clean":"2023-06-22T08:00:00.754941+0000","last_became_active":"2023-06-22T07:59:55.160315+0000","last_became_peered":"2023-06-22T07:59:55.160315+0000","last_unstale":"2023-06-22T08:00:00.754941+0000","last_undegraded":"2023-06-22T08:00:00.754941+0000","last_fullsized":"2023-06-22T08:00:00.754941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:51:09.921626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155504+0000","last_change":"2023-06-22T07:59:55.165950+0000","last_active":"2023-06-22T08:00:00.155504+0000","last_peered":"2023-06-22T08:00:00.155504+0000","last_clean":"2023-06-22T08:00:00.155504+0000","last_became_active":"2023-06-22T07:59:55.165505+0000","last_became_peered":"2023-06-22T07:59:55.165505+0000","last_unstale":"2023-06-22T08:00:00.155504+0000","last_undegraded":"2023-06-22T08:00:00.155504+0000","last_fullsized":"2023-06-22T08:00:00.155504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:13.512588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755640+0000","last_change":"2023-06-22T07:59:55.157572+0000","last_active":"2023-06-22T08:00:00.755640+0000","last_peered":"2023-06-22T08:00:00.755640+0000","last_clean":"2023-06-22T08:00:00.755640+0000","last_became_active":"2023-06-22T07:59:55.157055+0000","last_became_peered":"2023-06-22T07:59:55.157055+0000","last_unstale":"2023-06-22T08:00:00.755640+0000","last_undegraded":"2023-06-22T08:00:00.755640+0000","last_fullsized":"2023-06-22T08:00:00.755640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:34:35.685117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754095+0000","last_change":"2023-06-22T07:59:55.164696+0000","last_active":"2023-06-22T08:00:00.754095+0000","last_peered":"2023-06-22T08:00:00.754095+0000","last_clean":"2023-06-22T08:00:00.754095+0000","last_became_active":"2023-06-22T07:59:55.164179+0000","last_became_peered":"2023-06-22T07:59:55.164179+0000","last_unstale":"2023-06-22T08:00:00.754095+0000","last_undegraded":"2023-06-22T08:00:00.754095+0000","last_fullsized":"2023-06-22T08:00:00.754095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871504+0000","last_change":"2023-06-22T07:59:55.168838+0000","last_active":"2023-06-22T07:59:59.871504+0000","last_peered":"2023-06-22T07:59:59.871504+0000","last_clean":"2023-06-22T07:59:59.871504+0000","last_became_active":"2023-06-22T07:59:55.168551+0000","last_became_peered":"2023-06-22T07:59:55.168551+0000","last_unstale":"2023-06-22T07:59:59.871504+0000","last_undegraded":"2023-06-22T07:59:59.871504+0000","last_fullsized":"2023-06-22T07:59:59.871504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:02:36.615449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708648+0000","last_change":"2023-06-22T07:59:59.195356+0000","last_active":"2023-06-22T07:59:59.708648+0000","last_peered":"2023-06-22T07:59:59.708648+0000","last_clean":"2023-06-22T07:59:59.708648+0000","last_became_active":"2023-06-22T07:59:59.194814+0000","last_became_peered":"2023-06-22T07:59:59.194814+0000","last_unstale":"2023-06-22T07:59:59.708648+0000","last_undegraded":"2023-06-22T07:59:59.708648+0000","last_fullsized":"2023-06-22T07:59:59.708648+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:12.223267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6,0],"acting":[2,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493176+0000","last_change":"2023-06-22T07:59:54.651326+0000","last_active":"2023-06-22T08:00:36.493176+0000","last_peered":"2023-06-22T08:00:36.493176+0000","last_clean":"2023-06-22T08:00:36.493176+0000","last_became_active":"2023-06-22T07:59:54.651010+0000","last_became_peered":"2023-06-22T07:59:54.651010+0000","last_unstale":"2023-06-22T08:00:36.493176+0000","last_undegraded":"2023-06-22T08:00:36.493176+0000","last_fullsized":"2023-06-22T08:00:36.493176+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:17.769117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754538+0000","last_change":"2023-06-22T07:59:55.160345+0000","last_active":"2023-06-22T08:00:00.754538+0000","last_peered":"2023-06-22T08:00:00.754538+0000","last_clean":"2023-06-22T08:00:00.754538+0000","last_became_active":"2023-06-22T07:59:55.160071+0000","last_became_peered":"2023-06-22T07:59:55.160071+0000","last_unstale":"2023-06-22T08:00:00.754538+0000","last_undegraded":"2023-06-22T08:00:00.754538+0000","last_fullsized":"2023-06-22T08:00:00.754538+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708662+0000","last_change":"2023-06-22T07:59:59.196477+0000","last_active":"2023-06-22T07:59:59.708662+0000","last_peered":"2023-06-22T07:59:59.708662+0000","last_clean":"2023-06-22T07:59:59.708662+0000","last_became_active":"2023-06-22T07:59:59.196007+0000","last_became_peered":"2023-06-22T07:59:59.196007+0000","last_unstale":"2023-06-22T07:59:59.708662+0000","last_undegraded":"2023-06-22T07:59:59.708662+0000","last_fullsized":"2023-06-22T07:59:59.708662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,5],"acting":[2,7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:31.397730+0000","last_change":"2023-06-22T07:59:54.653192+0000","last_active":"2023-06-22T08:00:31.397730+0000","last_peered":"2023-06-22T08:00:31.397730+0000","last_clean":"2023-06-22T08:00:31.397730+0000","last_became_active":"2023-06-22T07:59:54.652926+0000","last_became_peered":"2023-06-22T07:59:54.652926+0000","last_unstale":"2023-06-22T08:00:31.397730+0000","last_undegraded":"2023-06-22T08:00:31.397730+0000","last_fullsized":"2023-06-22T08:00:31.397730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:12:59.656555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754609+0000","last_change":"2023-06-22T07:59:55.163171+0000","last_active":"2023-06-22T08:00:00.754609+0000","last_peered":"2023-06-22T08:00:00.754609+0000","last_clean":"2023-06-22T08:00:00.754609+0000","last_became_active":"2023-06-22T07:59:55.162869+0000","last_became_peered":"2023-06-22T07:59:55.162869+0000","last_unstale":"2023-06-22T08:00:00.754609+0000","last_undegraded":"2023-06-22T08:00:00.754609+0000","last_fullsized":"2023-06-22T08:00:00.754609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:37:59.063186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754586+0000","last_change":"2023-06-22T07:59:59.200516+0000","last_active":"2023-06-22T08:00:00.754586+0000","last_peered":"2023-06-22T08:00:00.754586+0000","last_clean":"2023-06-22T08:00:00.754586+0000","last_became_active":"2023-06-22T07:59:59.200096+0000","last_became_peered":"2023-06-22T07:59:59.200096+0000","last_unstale":"2023-06-22T08:00:00.754586+0000","last_undegraded":"2023-06-22T08:00:00.754586+0000","last_fullsized":"2023-06-22T08:00:00.754586+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:06.099105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2,7],"acting":[6,3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.490825+0000","last_change":"2023-06-22T07:59:54.645950+0000","last_active":"2023-06-22T08:00:36.490825+0000","last_peered":"2023-06-22T08:00:36.490825+0000","last_clean":"2023-06-22T08:00:36.490825+0000","last_became_active":"2023-06-22T07:59:54.645520+0000","last_became_peered":"2023-06-22T07:59:54.645520+0000","last_unstale":"2023-06-22T08:00:36.490825+0000","last_undegraded":"2023-06-22T08:00:36.490825+0000","last_fullsized":"2023-06-22T08:00:36.490825+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:34.816556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871992+0000","last_change":"2023-06-22T07:59:55.162767+0000","last_active":"2023-06-22T07:59:59.871992+0000","last_peered":"2023-06-22T07:59:59.871992+0000","last_clean":"2023-06-22T07:59:59.871992+0000","last_became_active":"2023-06-22T07:59:55.162451+0000","last_became_peered":"2023-06-22T07:59:55.162451+0000","last_unstale":"2023-06-22T07:59:59.871992+0000","last_undegraded":"2023-06-22T07:59:59.871992+0000","last_fullsized":"2023-06-22T07:59:59.871992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:38.977413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871071+0000","last_change":"2023-06-22T07:59:59.208215+0000","last_active":"2023-06-22T07:59:59.871071+0000","last_peered":"2023-06-22T07:59:59.871071+0000","last_clean":"2023-06-22T07:59:59.871071+0000","last_became_active":"2023-06-22T07:59:59.207608+0000","last_became_peered":"2023-06-22T07:59:59.207608+0000","last_unstale":"2023-06-22T07:59:59.871071+0000","last_undegraded":"2023-06-22T07:59:59.871071+0000","last_fullsized":"2023-06-22T07:59:59.871071+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:39:48.527076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,5],"acting":[3,2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.699308+0000","last_change":"2023-06-22T07:59:54.652515+0000","last_active":"2023-06-22T08:00:25.699308+0000","last_peered":"2023-06-22T08:00:25.699308+0000","last_clean":"2023-06-22T08:00:25.699308+0000","last_became_active":"2023-06-22T07:59:54.652249+0000","last_became_peered":"2023-06-22T07:59:54.652249+0000","last_unstale":"2023-06-22T08:00:25.699308+0000","last_undegraded":"2023-06-22T08:00:25.699308+0000","last_fullsized":"2023-06-22T08:00:25.699308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:12.506596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755250+0000","last_change":"2023-06-22T07:59:55.158359+0000","last_active":"2023-06-22T08:00:00.755250+0000","last_peered":"2023-06-22T08:00:00.755250+0000","last_clean":"2023-06-22T08:00:00.755250+0000","last_became_active":"2023-06-22T07:59:55.157918+0000","last_became_peered":"2023-06-22T07:59:55.157918+0000","last_unstale":"2023-06-22T08:00:00.755250+0000","last_undegraded":"2023-06-22T08:00:00.755250+0000","last_fullsized":"2023-06-22T08:00:00.755250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708305+0000","last_change":"2023-06-22T07:59:59.201033+0000","last_active":"2023-06-22T07:59:59.708305+0000","last_peered":"2023-06-22T07:59:59.708305+0000","last_clean":"2023-06-22T07:59:59.708305+0000","last_became_active":"2023-06-22T07:59:59.200500+0000","last_became_peered":"2023-06-22T07:59:59.200500+0000","last_unstale":"2023-06-22T07:59:59.708305+0000","last_undegraded":"2023-06-22T07:59:59.708305+0000","last_fullsized":"2023-06-22T07:59:59.708305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:18.174604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,4],"acting":[2,1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.571653+0000","last_change":"2023-06-22T07:59:54.654416+0000","last_active":"2023-06-22T08:00:41.571653+0000","last_peered":"2023-06-22T08:00:41.571653+0000","last_clean":"2023-06-22T08:00:41.571653+0000","last_became_active":"2023-06-22T07:59:54.654139+0000","last_became_peered":"2023-06-22T07:59:54.654139+0000","last_unstale":"2023-06-22T08:00:41.571653+0000","last_undegraded":"2023-06-22T08:00:41.571653+0000","last_fullsized":"2023-06-22T08:00:41.571653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755506+0000","last_change":"2023-06-22T07:59:55.159790+0000","last_active":"2023-06-22T08:00:00.755506+0000","last_peered":"2023-06-22T08:00:00.755506+0000","last_clean":"2023-06-22T08:00:00.755506+0000","last_became_active":"2023-06-22T07:59:55.159489+0000","last_became_peered":"2023-06-22T07:59:55.159489+0000","last_unstale":"2023-06-22T08:00:00.755506+0000","last_undegraded":"2023-06-22T08:00:00.755506+0000","last_fullsized":"2023-06-22T08:00:00.755506+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:15:18.373418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753814+0000","last_change":"2023-06-22T07:59:59.197822+0000","last_active":"2023-06-22T08:00:00.753814+0000","last_peered":"2023-06-22T08:00:00.753814+0000","last_clean":"2023-06-22T08:00:00.753814+0000","last_became_active":"2023-06-22T07:59:59.197431+0000","last_became_peered":"2023-06-22T07:59:59.197431+0000","last_unstale":"2023-06-22T08:00:00.753814+0000","last_undegraded":"2023-06-22T08:00:00.753814+0000","last_fullsized":"2023-06-22T08:00:00.753814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:41.139500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7,1],"acting":[4,6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.698982+0000","last_change":"2023-06-22T07:59:54.644158+0000","last_active":"2023-06-22T08:00:25.698982+0000","last_peered":"2023-06-22T08:00:25.698982+0000","last_clean":"2023-06-22T08:00:25.698982+0000","last_became_active":"2023-06-22T07:59:54.643808+0000","last_became_peered":"2023-06-22T07:59:54.643808+0000","last_unstale":"2023-06-22T08:00:25.698982+0000","last_undegraded":"2023-06-22T08:00:25.698982+0000","last_fullsized":"2023-06-22T08:00:25.698982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:45.438691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754507+0000","last_change":"2023-06-22T07:59:55.157538+0000","last_active":"2023-06-22T08:00:00.754507+0000","last_peered":"2023-06-22T08:00:00.754507+0000","last_clean":"2023-06-22T08:00:00.754507+0000","last_became_active":"2023-06-22T07:59:55.156967+0000","last_became_peered":"2023-06-22T07:59:55.156967+0000","last_unstale":"2023-06-22T08:00:00.754507+0000","last_undegraded":"2023-06-22T08:00:00.754507+0000","last_fullsized":"2023-06-22T08:00:00.754507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:38:44.161245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753821+0000","last_change":"2023-06-22T07:59:59.201671+0000","last_active":"2023-06-22T08:00:00.753821+0000","last_peered":"2023-06-22T08:00:00.753821+0000","last_clean":"2023-06-22T08:00:00.753821+0000","last_became_active":"2023-06-22T07:59:59.201201+0000","last_became_peered":"2023-06-22T07:59:59.201201+0000","last_unstale":"2023-06-22T08:00:00.753821+0000","last_undegraded":"2023-06-22T08:00:00.753821+0000","last_fullsized":"2023-06-22T08:00:00.753821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4,3],"acting":[6,1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"28'16","reported_seq":37,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:50.546385+0000","last_change":"2023-06-22T07:59:54.640470+0000","last_active":"2023-06-22T08:00:50.546385+0000","last_peered":"2023-06-22T08:00:50.546385+0000","last_clean":"2023-06-22T08:00:50.546385+0000","last_became_active":"2023-06-22T07:59:54.640124+0000","last_became_peered":"2023-06-22T07:59:54.640124+0000","last_unstale":"2023-06-22T08:00:50.546385+0000","last_undegraded":"2023-06-22T08:00:50.546385+0000","last_fullsized":"2023-06-22T08:00:50.546385+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:06:41.502702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4052,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708599+0000","last_change":"2023-06-22T07:59:55.165550+0000","last_active":"2023-06-22T07:59:59.708599+0000","last_peered":"2023-06-22T07:59:59.708599+0000","last_clean":"2023-06-22T07:59:59.708599+0000","last_became_active":"2023-06-22T07:59:55.164998+0000","last_became_peered":"2023-06-22T07:59:55.164998+0000","last_unstale":"2023-06-22T07:59:59.708599+0000","last_undegraded":"2023-06-22T07:59:59.708599+0000","last_fullsized":"2023-06-22T07:59:59.708599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872133+0000","last_change":"2023-06-22T07:59:59.207103+0000","last_active":"2023-06-22T07:59:59.872133+0000","last_peered":"2023-06-22T07:59:59.872133+0000","last_clean":"2023-06-22T07:59:59.872133+0000","last_became_active":"2023-06-22T07:59:59.206531+0000","last_became_peered":"2023-06-22T07:59:59.206531+0000","last_unstale":"2023-06-22T07:59:59.872133+0000","last_undegraded":"2023-06-22T07:59:59.872133+0000","last_fullsized":"2023-06-22T07:59:59.872133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:42:48.571228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0,6],"acting":[3,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"28'9","reported_seq":30,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.570482+0000","last_change":"2023-06-22T07:59:54.644301+0000","last_active":"2023-06-22T08:00:41.570482+0000","last_peered":"2023-06-22T08:00:41.570482+0000","last_clean":"2023-06-22T08:00:41.570482+0000","last_became_active":"2023-06-22T07:59:54.643939+0000","last_became_peered":"2023-06-22T07:59:54.643939+0000","last_unstale":"2023-06-22T08:00:41.570482+0000","last_undegraded":"2023-06-22T08:00:41.570482+0000","last_fullsized":"2023-06-22T08:00:41.570482+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:07:42.118767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753877+0000","last_change":"2023-06-22T07:59:55.163311+0000","last_active":"2023-06-22T08:00:00.753877+0000","last_peered":"2023-06-22T08:00:00.753877+0000","last_clean":"2023-06-22T08:00:00.753877+0000","last_became_active":"2023-06-22T07:59:55.162991+0000","last_became_peered":"2023-06-22T07:59:55.162991+0000","last_unstale":"2023-06-22T08:00:00.753877+0000","last_undegraded":"2023-06-22T08:00:00.753877+0000","last_fullsized":"2023-06-22T08:00:00.753877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754227+0000","last_change":"2023-06-22T07:59:59.201324+0000","last_active":"2023-06-22T08:00:00.754227+0000","last_peered":"2023-06-22T08:00:00.754227+0000","last_clean":"2023-06-22T08:00:00.754227+0000","last_became_active":"2023-06-22T07:59:59.200877+0000","last_became_peered":"2023-06-22T07:59:59.200877+0000","last_unstale":"2023-06-22T08:00:00.754227+0000","last_undegraded":"2023-06-22T08:00:00.754227+0000","last_fullsized":"2023-06-22T08:00:00.754227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:14:17.094426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5,7],"acting":[4,2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493994+0000","last_change":"2023-06-22T07:59:54.651527+0000","last_active":"2023-06-22T08:00:36.493994+0000","last_peered":"2023-06-22T08:00:36.493994+0000","last_clean":"2023-06-22T08:00:36.493994+0000","last_became_active":"2023-06-22T07:59:54.651196+0000","last_became_peered":"2023-06-22T07:59:54.651196+0000","last_unstale":"2023-06-22T08:00:36.493994+0000","last_undegraded":"2023-06-22T08:00:36.493994+0000","last_fullsized":"2023-06-22T08:00:36.493994+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872142+0000","last_change":"2023-06-22T07:59:55.162610+0000","last_active":"2023-06-22T07:59:59.872142+0000","last_peered":"2023-06-22T07:59:59.872142+0000","last_clean":"2023-06-22T07:59:59.872142+0000","last_became_active":"2023-06-22T07:59:55.162334+0000","last_became_peered":"2023-06-22T07:59:55.162334+0000","last_unstale":"2023-06-22T07:59:59.872142+0000","last_undegraded":"2023-06-22T07:59:59.872142+0000","last_fullsized":"2023-06-22T07:59:59.872142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:42.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:59.199402+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:59.198979+0000","last_became_peered":"2023-06-22T07:59:59.198979+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:17:07.122732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1,6],"acting":[4,5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493688+0000","last_change":"2023-06-22T07:59:54.649236+0000","last_active":"2023-06-22T08:00:36.493688+0000","last_peered":"2023-06-22T08:00:36.493688+0000","last_clean":"2023-06-22T08:00:36.493688+0000","last_became_active":"2023-06-22T07:59:54.648852+0000","last_became_peered":"2023-06-22T07:59:54.648852+0000","last_unstale":"2023-06-22T08:00:36.493688+0000","last_undegraded":"2023-06-22T08:00:36.493688+0000","last_fullsized":"2023-06-22T08:00:36.493688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:16:59.865638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709134+0000","last_change":"2023-06-22T07:59:55.165859+0000","last_active":"2023-06-22T07:59:59.709134+0000","last_peered":"2023-06-22T07:59:59.709134+0000","last_clean":"2023-06-22T07:59:59.709134+0000","last_became_active":"2023-06-22T07:59:55.165542+0000","last_became_peered":"2023-06-22T07:59:55.165542+0000","last_unstale":"2023-06-22T07:59:59.709134+0000","last_undegraded":"2023-06-22T07:59:59.709134+0000","last_fullsized":"2023-06-22T07:59:59.709134+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:32.147969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871894+0000","last_change":"2023-06-22T07:59:59.207794+0000","last_active":"2023-06-22T07:59:59.871894+0000","last_peered":"2023-06-22T07:59:59.871894+0000","last_clean":"2023-06-22T07:59:59.871894+0000","last_became_active":"2023-06-22T07:59:59.207364+0000","last_became_peered":"2023-06-22T07:59:59.207364+0000","last_unstale":"2023-06-22T07:59:59.871894+0000","last_undegraded":"2023-06-22T07:59:59.871894+0000","last_fullsized":"2023-06-22T07:59:59.871894+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:30.718491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4,0],"acting":[3,6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"28'17","reported_seq":38,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.491476+0000","last_change":"2023-06-22T07:59:54.651423+0000","last_active":"2023-06-22T08:00:36.491476+0000","last_peered":"2023-06-22T08:00:36.491476+0000","last_clean":"2023-06-22T08:00:36.491476+0000","last_became_active":"2023-06-22T07:59:54.651167+0000","last_became_peered":"2023-06-22T07:59:54.651167+0000","last_unstale":"2023-06-22T08:00:36.491476+0000","last_undegraded":"2023-06-22T08:00:36.491476+0000","last_fullsized":"2023-06-22T08:00:36.491476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:31.141977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709649+0000","last_change":"2023-06-22T07:59:55.164478+0000","last_active":"2023-06-22T07:59:59.709649+0000","last_peered":"2023-06-22T07:59:59.709649+0000","last_clean":"2023-06-22T07:59:59.709649+0000","last_became_active":"2023-06-22T07:59:55.163922+0000","last_became_peered":"2023-06-22T07:59:55.163922+0000","last_unstale":"2023-06-22T07:59:59.709649+0000","last_undegraded":"2023-06-22T07:59:59.709649+0000","last_fullsized":"2023-06-22T07:59:59.709649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:43.772399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155085+0000","last_change":"2023-06-22T07:59:59.198496+0000","last_active":"2023-06-22T08:00:00.155085+0000","last_peered":"2023-06-22T08:00:00.155085+0000","last_clean":"2023-06-22T08:00:00.155085+0000","last_became_active":"2023-06-22T07:59:59.198067+0000","last_became_peered":"2023-06-22T07:59:59.198067+0000","last_unstale":"2023-06-22T08:00:00.155085+0000","last_undegraded":"2023-06-22T08:00:00.155085+0000","last_fullsized":"2023-06-22T08:00:00.155085+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:24.865757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,7],"acting":[1,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158445+0000","last_change":"2023-06-22T07:59:54.651366+0000","last_active":"2023-06-22T08:00:00.158445+0000","last_peered":"2023-06-22T08:00:00.158445+0000","last_clean":"2023-06-22T08:00:00.158445+0000","last_became_active":"2023-06-22T07:59:54.651059+0000","last_became_peered":"2023-06-22T07:59:54.651059+0000","last_unstale":"2023-06-22T08:00:00.158445+0000","last_undegraded":"2023-06-22T08:00:00.158445+0000","last_fullsized":"2023-06-22T08:00:00.158445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:37.183143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871131+0000","last_change":"2023-06-22T07:59:55.162820+0000","last_active":"2023-06-22T07:59:59.871131+0000","last_peered":"2023-06-22T07:59:59.871131+0000","last_clean":"2023-06-22T07:59:59.871131+0000","last_became_active":"2023-06-22T07:59:55.162505+0000","last_became_peered":"2023-06-22T07:59:55.162505+0000","last_unstale":"2023-06-22T07:59:59.871131+0000","last_undegraded":"2023-06-22T07:59:59.871131+0000","last_fullsized":"2023-06-22T07:59:59.871131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:46:38.312778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753744+0000","last_change":"2023-06-22T07:59:59.201059+0000","last_active":"2023-06-22T08:00:00.753744+0000","last_peered":"2023-06-22T08:00:00.753744+0000","last_clean":"2023-06-22T08:00:00.753744+0000","last_became_active":"2023-06-22T07:59:59.200460+0000","last_became_peered":"2023-06-22T07:59:59.200460+0000","last_unstale":"2023-06-22T08:00:00.753744+0000","last_undegraded":"2023-06-22T08:00:00.753744+0000","last_fullsized":"2023-06-22T08:00:00.753744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,0],"acting":[6,4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.565345+0000","last_change":"2023-06-22T07:59:54.646017+0000","last_active":"2023-06-22T08:00:41.565345+0000","last_peered":"2023-06-22T08:00:41.565345+0000","last_clean":"2023-06-22T08:00:41.565345+0000","last_became_active":"2023-06-22T07:59:54.645554+0000","last_became_peered":"2023-06-22T07:59:54.645554+0000","last_unstale":"2023-06-22T08:00:41.565345+0000","last_undegraded":"2023-06-22T08:00:41.565345+0000","last_fullsized":"2023-06-22T08:00:41.565345+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:01.550269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154806+0000","last_change":"2023-06-22T07:59:55.166283+0000","last_active":"2023-06-22T08:00:00.154806+0000","last_peered":"2023-06-22T08:00:00.154806+0000","last_clean":"2023-06-22T08:00:00.154806+0000","last_became_active":"2023-06-22T07:59:55.165977+0000","last_became_peered":"2023-06-22T07:59:55.165977+0000","last_unstale":"2023-06-22T08:00:00.154806+0000","last_undegraded":"2023-06-22T08:00:00.154806+0000","last_fullsized":"2023-06-22T08:00:00.154806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:21:29.484640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709141+0000","last_change":"2023-06-22T07:59:59.205411+0000","last_active":"2023-06-22T07:59:59.709141+0000","last_peered":"2023-06-22T07:59:59.709141+0000","last_clean":"2023-06-22T07:59:59.709141+0000","last_became_active":"2023-06-22T07:59:59.204980+0000","last_became_peered":"2023-06-22T07:59:59.204980+0000","last_unstale":"2023-06-22T07:59:59.709141+0000","last_undegraded":"2023-06-22T07:59:59.709141+0000","last_fullsized":"2023-06-22T07:59:59.709141+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:22.317952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493022+0000","last_change":"2023-06-22T07:59:54.654491+0000","last_active":"2023-06-22T08:00:36.493022+0000","last_peered":"2023-06-22T08:00:36.493022+0000","last_clean":"2023-06-22T08:00:36.493022+0000","last_became_active":"2023-06-22T07:59:54.654197+0000","last_became_peered":"2023-06-22T07:59:54.654197+0000","last_unstale":"2023-06-22T08:00:36.493022+0000","last_undegraded":"2023-06-22T08:00:36.493022+0000","last_fullsized":"2023-06-22T08:00:36.493022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:42:59.725455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708444+0000","last_change":"2023-06-22T07:59:55.172053+0000","last_active":"2023-06-22T07:59:59.708444+0000","last_peered":"2023-06-22T07:59:59.708444+0000","last_clean":"2023-06-22T07:59:59.708444+0000","last_became_active":"2023-06-22T07:59:55.165501+0000","last_became_peered":"2023-06-22T07:59:55.165501+0000","last_unstale":"2023-06-22T07:59:59.708444+0000","last_undegraded":"2023-06-22T07:59:59.708444+0000","last_fullsized":"2023-06-22T07:59:59.708444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753990+0000","last_change":"2023-06-22T07:59:59.199179+0000","last_active":"2023-06-22T08:00:00.753990+0000","last_peered":"2023-06-22T08:00:00.753990+0000","last_clean":"2023-06-22T08:00:00.753990+0000","last_became_active":"2023-06-22T07:59:59.198740+0000","last_became_peered":"2023-06-22T07:59:59.198740+0000","last_unstale":"2023-06-22T08:00:00.753990+0000","last_undegraded":"2023-06-22T08:00:00.753990+0000","last_fullsized":"2023-06-22T08:00:00.753990+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.491850+0000","last_change":"2023-06-22T07:59:54.644520+0000","last_active":"2023-06-22T08:00:36.491850+0000","last_peered":"2023-06-22T08:00:36.491850+0000","last_clean":"2023-06-22T08:00:36.491850+0000","last_became_active":"2023-06-22T07:59:54.644204+0000","last_became_peered":"2023-06-22T07:59:54.644204+0000","last_unstale":"2023-06-22T08:00:36.491850+0000","last_undegraded":"2023-06-22T08:00:36.491850+0000","last_fullsized":"2023-06-22T08:00:36.491850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:07.674431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709657+0000","last_change":"2023-06-22T07:59:55.164111+0000","last_active":"2023-06-22T07:59:59.709657+0000","last_peered":"2023-06-22T07:59:59.709657+0000","last_clean":"2023-06-22T07:59:59.709657+0000","last_became_active":"2023-06-22T07:59:55.163802+0000","last_became_peered":"2023-06-22T07:59:55.163802+0000","last_unstale":"2023-06-22T07:59:59.709657+0000","last_undegraded":"2023-06-22T07:59:59.709657+0000","last_fullsized":"2023-06-22T07:59:59.709657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755252+0000","last_change":"2023-06-22T07:59:59.197721+0000","last_active":"2023-06-22T08:00:00.755252+0000","last_peered":"2023-06-22T08:00:00.755252+0000","last_clean":"2023-06-22T08:00:00.755252+0000","last_became_active":"2023-06-22T07:59:59.197259+0000","last_became_peered":"2023-06-22T07:59:59.197259+0000","last_unstale":"2023-06-22T08:00:00.755252+0000","last_undegraded":"2023-06-22T08:00:00.755252+0000","last_fullsized":"2023-06-22T08:00:00.755252+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5,1],"acting":[7,0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.697877+0000","last_change":"2023-06-22T07:59:54.643124+0000","last_active":"2023-06-22T08:00:25.697877+0000","last_peered":"2023-06-22T08:00:25.697877+0000","last_clean":"2023-06-22T08:00:25.697877+0000","last_became_active":"2023-06-22T07:59:54.642833+0000","last_became_peered":"2023-06-22T07:59:54.642833+0000","last_unstale":"2023-06-22T08:00:25.697877+0000","last_undegraded":"2023-06-22T08:00:25.697877+0000","last_fullsized":"2023-06-22T08:00:25.697877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158064+0000","last_change":"2023-06-22T07:59:55.160551+0000","last_active":"2023-06-22T08:00:00.158064+0000","last_peered":"2023-06-22T08:00:00.158064+0000","last_clean":"2023-06-22T08:00:00.158064+0000","last_became_active":"2023-06-22T07:59:55.160221+0000","last_became_peered":"2023-06-22T07:59:55.160221+0000","last_unstale":"2023-06-22T08:00:00.158064+0000","last_undegraded":"2023-06-22T08:00:00.158064+0000","last_fullsized":"2023-06-22T08:00:00.158064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708974+0000","last_change":"2023-06-22T07:59:59.204643+0000","last_active":"2023-06-22T07:59:59.708974+0000","last_peered":"2023-06-22T07:59:59.708974+0000","last_clean":"2023-06-22T07:59:59.708974+0000","last_became_active":"2023-06-22T07:59:59.204182+0000","last_became_peered":"2023-06-22T07:59:59.204182+0000","last_unstale":"2023-06-22T07:59:59.708974+0000","last_undegraded":"2023-06-22T07:59:59.708974+0000","last_fullsized":"2023-06-22T07:59:59.708974+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4,2],"acting":[5,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493681+0000","last_change":"2023-06-22T07:59:54.654036+0000","last_active":"2023-06-22T08:00:36.493681+0000","last_peered":"2023-06-22T08:00:36.493681+0000","last_clean":"2023-06-22T08:00:36.493681+0000","last_became_active":"2023-06-22T07:59:54.653736+0000","last_became_peered":"2023-06-22T07:59:54.653736+0000","last_unstale":"2023-06-22T08:00:36.493681+0000","last_undegraded":"2023-06-22T08:00:36.493681+0000","last_fullsized":"2023-06-22T08:00:36.493681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:44:09.331748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871999+0000","last_change":"2023-06-22T07:59:55.163181+0000","last_active":"2023-06-22T07:59:59.871999+0000","last_peered":"2023-06-22T07:59:59.871999+0000","last_clean":"2023-06-22T07:59:59.871999+0000","last_became_active":"2023-06-22T07:59:55.162860+0000","last_became_peered":"2023-06-22T07:59:55.162860+0000","last_unstale":"2023-06-22T07:59:59.871999+0000","last_undegraded":"2023-06-22T07:59:59.871999+0000","last_fullsized":"2023-06-22T07:59:59.871999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:59:25.037938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155510+0000","last_change":"2023-06-22T07:59:59.195710+0000","last_active":"2023-06-22T08:00:00.155510+0000","last_peered":"2023-06-22T08:00:00.155510+0000","last_clean":"2023-06-22T08:00:00.155510+0000","last_became_active":"2023-06-22T07:59:59.195246+0000","last_became_peered":"2023-06-22T07:59:59.195246+0000","last_unstale":"2023-06-22T08:00:00.155510+0000","last_undegraded":"2023-06-22T08:00:00.155510+0000","last_fullsized":"2023-06-22T08:00:00.155510+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:36.457493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,5],"acting":[1,0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.570855+0000","last_change":"2023-06-22T07:59:54.649692+0000","last_active":"2023-06-22T08:00:41.570855+0000","last_peered":"2023-06-22T08:00:41.570855+0000","last_clean":"2023-06-22T08:00:41.570855+0000","last_became_active":"2023-06-22T07:59:54.649416+0000","last_became_peered":"2023-06-22T07:59:54.649416+0000","last_unstale":"2023-06-22T08:00:41.570855+0000","last_undegraded":"2023-06-22T08:00:41.570855+0000","last_fullsized":"2023-06-22T08:00:41.570855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:54:02.923433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708909+0000","last_change":"2023-06-22T07:59:55.175392+0000","last_active":"2023-06-22T07:59:59.708909+0000","last_peered":"2023-06-22T07:59:59.708909+0000","last_clean":"2023-06-22T07:59:59.708909+0000","last_became_active":"2023-06-22T07:59:55.175112+0000","last_became_peered":"2023-06-22T07:59:55.175112+0000","last_unstale":"2023-06-22T07:59:59.708909+0000","last_undegraded":"2023-06-22T07:59:59.708909+0000","last_fullsized":"2023-06-22T07:59:59.708909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871873+0000","last_change":"2023-06-22T07:59:59.206671+0000","last_active":"2023-06-22T07:59:59.871873+0000","last_peered":"2023-06-22T07:59:59.871873+0000","last_clean":"2023-06-22T07:59:59.871873+0000","last_became_active":"2023-06-22T07:59:59.206179+0000","last_became_peered":"2023-06-22T07:59:59.206179+0000","last_unstale":"2023-06-22T07:59:59.871873+0000","last_undegraded":"2023-06-22T07:59:59.871873+0000","last_fullsized":"2023-06-22T07:59:59.871873+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:00.427518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4,5],"acting":[3,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.698074+0000","last_change":"2023-06-22T07:59:54.640508+0000","last_active":"2023-06-22T08:00:25.698074+0000","last_peered":"2023-06-22T08:00:25.698074+0000","last_clean":"2023-06-22T08:00:25.698074+0000","last_became_active":"2023-06-22T07:59:54.640108+0000","last_became_peered":"2023-06-22T07:59:54.640108+0000","last_unstale":"2023-06-22T08:00:25.698074+0000","last_undegraded":"2023-06-22T08:00:25.698074+0000","last_fullsized":"2023-06-22T08:00:25.698074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:19.193231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754822+0000","last_change":"2023-06-22T07:59:55.164398+0000","last_active":"2023-06-22T08:00:00.754822+0000","last_peered":"2023-06-22T08:00:00.754822+0000","last_clean":"2023-06-22T08:00:00.754822+0000","last_became_active":"2023-06-22T07:59:55.164086+0000","last_became_peered":"2023-06-22T07:59:55.164086+0000","last_unstale":"2023-06-22T08:00:00.754822+0000","last_undegraded":"2023-06-22T08:00:00.754822+0000","last_fullsized":"2023-06-22T08:00:00.754822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:38.189135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753997+0000","last_change":"2023-06-22T07:59:59.207752+0000","last_active":"2023-06-22T08:00:00.753997+0000","last_peered":"2023-06-22T08:00:00.753997+0000","last_clean":"2023-06-22T08:00:00.753997+0000","last_became_active":"2023-06-22T07:59:59.207360+0000","last_became_peered":"2023-06-22T07:59:59.207360+0000","last_unstale":"2023-06-22T08:00:00.753997+0000","last_undegraded":"2023-06-22T08:00:00.753997+0000","last_fullsized":"2023-06-22T08:00:00.753997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:08:25.633009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3,1],"acting":[4,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709114+0000","last_change":"2023-06-22T07:59:49.652772+0000","last_active":"2023-06-22T07:59:59.709114+0000","last_peered":"2023-06-22T07:59:59.709114+0000","last_clean":"2023-06-22T07:59:59.709114+0000","last_became_active":"2023-06-22T07:59:49.652487+0000","last_became_peered":"2023-06-22T07:59:49.652487+0000","last_unstale":"2023-06-22T07:59:59.709114+0000","last_undegraded":"2023-06-22T07:59:59.709114+0000","last_fullsized":"2023-06-22T07:59:59.709114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:11.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493495+0000","last_change":"2023-06-22T07:59:54.644538+0000","last_active":"2023-06-22T08:00:36.493495+0000","last_peered":"2023-06-22T08:00:36.493495+0000","last_clean":"2023-06-22T08:00:36.493495+0000","last_became_active":"2023-06-22T07:59:54.644253+0000","last_became_peered":"2023-06-22T07:59:54.644253+0000","last_unstale":"2023-06-22T08:00:36.493495+0000","last_undegraded":"2023-06-22T08:00:36.493495+0000","last_fullsized":"2023-06-22T08:00:36.493495+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:09.128751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754228+0000","last_change":"2023-06-22T07:59:55.163802+0000","last_active":"2023-06-22T08:00:00.754228+0000","last_peered":"2023-06-22T08:00:00.754228+0000","last_clean":"2023-06-22T08:00:00.754228+0000","last_became_active":"2023-06-22T07:59:55.163481+0000","last_became_peered":"2023-06-22T07:59:55.163481+0000","last_unstale":"2023-06-22T08:00:00.754228+0000","last_undegraded":"2023-06-22T08:00:00.754228+0000","last_fullsized":"2023-06-22T08:00:00.754228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:16.916350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755246+0000","last_change":"2023-06-22T07:59:59.207344+0000","last_active":"2023-06-22T08:00:00.755246+0000","last_peered":"2023-06-22T08:00:00.755246+0000","last_clean":"2023-06-22T08:00:00.755246+0000","last_became_active":"2023-06-22T07:59:59.206975+0000","last_became_peered":"2023-06-22T07:59:59.206975+0000","last_unstale":"2023-06-22T08:00:00.755246+0000","last_undegraded":"2023-06-22T08:00:00.755246+0000","last_fullsized":"2023-06-22T08:00:00.755246+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3,1],"acting":[7,4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871513+0000","last_change":"2023-06-22T07:59:49.653053+0000","last_active":"2023-06-22T07:59:59.871513+0000","last_peered":"2023-06-22T07:59:59.871513+0000","last_clean":"2023-06-22T07:59:59.871513+0000","last_became_active":"2023-06-22T07:59:49.652861+0000","last_became_peered":"2023-06-22T07:59:49.652861+0000","last_unstale":"2023-06-22T07:59:59.871513+0000","last_undegraded":"2023-06-22T07:59:59.871513+0000","last_fullsized":"2023-06-22T07:59:59.871513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:28:40.388367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"28'6","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493566+0000","last_change":"2023-06-22T07:59:54.645403+0000","last_active":"2023-06-22T08:00:36.493566+0000","last_peered":"2023-06-22T08:00:36.493566+0000","last_clean":"2023-06-22T08:00:36.493566+0000","last_became_active":"2023-06-22T07:59:54.644973+0000","last_became_peered":"2023-06-22T07:59:54.644973+0000","last_unstale":"2023-06-22T08:00:36.493566+0000","last_undegraded":"2023-06-22T08:00:36.493566+0000","last_fullsized":"2023-06-22T08:00:36.493566+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:15.910358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754101+0000","last_change":"2023-06-22T07:59:55.161484+0000","last_active":"2023-06-22T08:00:00.754101+0000","last_peered":"2023-06-22T08:00:00.754101+0000","last_clean":"2023-06-22T08:00:00.754101+0000","last_became_active":"2023-06-22T07:59:55.161194+0000","last_became_peered":"2023-06-22T07:59:55.161194+0000","last_unstale":"2023-06-22T08:00:00.754101+0000","last_undegraded":"2023-06-22T08:00:00.754101+0000","last_fullsized":"2023-06-22T08:00:00.754101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754988+0000","last_change":"2023-06-22T07:59:59.198241+0000","last_active":"2023-06-22T08:00:00.754988+0000","last_peered":"2023-06-22T08:00:00.754988+0000","last_clean":"2023-06-22T08:00:00.754988+0000","last_became_active":"2023-06-22T07:59:59.197689+0000","last_became_peered":"2023-06-22T07:59:59.197689+0000","last_unstale":"2023-06-22T08:00:00.754988+0000","last_undegraded":"2023-06-22T08:00:00.754988+0000","last_fullsized":"2023-06-22T08:00:00.754988+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:39.513874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4,1],"acting":[7,0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753886+0000","last_change":"2023-06-22T07:59:49.450044+0000","last_active":"2023-06-22T08:00:00.753886+0000","last_peered":"2023-06-22T08:00:00.753886+0000","last_clean":"2023-06-22T08:00:00.753886+0000","last_became_active":"2023-06-22T07:59:49.449731+0000","last_became_peered":"2023-06-22T07:59:49.449731+0000","last_unstale":"2023-06-22T08:00:00.753886+0000","last_undegraded":"2023-06-22T08:00:00.753886+0000","last_fullsized":"2023-06-22T08:00:00.753886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:40.262337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:31.395849+0000","last_change":"2023-06-22T07:59:54.640462+0000","last_active":"2023-06-22T08:00:31.395849+0000","last_peered":"2023-06-22T08:00:31.395849+0000","last_clean":"2023-06-22T08:00:31.395849+0000","last_became_active":"2023-06-22T07:59:54.640123+0000","last_became_peered":"2023-06-22T07:59:54.640123+0000","last_unstale":"2023-06-22T08:00:31.395849+0000","last_undegraded":"2023-06-22T08:00:31.395849+0000","last_fullsized":"2023-06-22T08:00:31.395849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:20.316921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755350+0000","last_change":"2023-06-22T07:59:55.164651+0000","last_active":"2023-06-22T08:00:00.755350+0000","last_peered":"2023-06-22T08:00:00.755350+0000","last_clean":"2023-06-22T08:00:00.755350+0000","last_became_active":"2023-06-22T07:59:55.164235+0000","last_became_peered":"2023-06-22T07:59:55.164235+0000","last_unstale":"2023-06-22T08:00:00.755350+0000","last_undegraded":"2023-06-22T08:00:00.755350+0000","last_fullsized":"2023-06-22T08:00:00.755350+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:32.924136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157741+0000","last_change":"2023-06-22T07:59:59.204779+0000","last_active":"2023-06-22T08:00:00.157741+0000","last_peered":"2023-06-22T08:00:00.157741+0000","last_clean":"2023-06-22T08:00:00.157741+0000","last_became_active":"2023-06-22T07:59:59.204288+0000","last_became_peered":"2023-06-22T07:59:59.204288+0000","last_unstale":"2023-06-22T08:00:00.157741+0000","last_undegraded":"2023-06-22T08:00:00.157741+0000","last_fullsized":"2023-06-22T08:00:00.157741+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871497+0000","last_change":"2023-06-22T07:59:49.657008+0000","last_active":"2023-06-22T07:59:59.871497+0000","last_peered":"2023-06-22T07:59:59.871497+0000","last_clean":"2023-06-22T07:59:59.871497+0000","last_became_active":"2023-06-22T07:59:49.656752+0000","last_became_peered":"2023-06-22T07:59:49.656752+0000","last_unstale":"2023-06-22T07:59:59.871497+0000","last_undegraded":"2023-06-22T07:59:59.871497+0000","last_fullsized":"2023-06-22T07:59:59.871497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:50:37.639134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":185,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155134+0000","last_change":"2023-06-22T07:59:42.501363+0000","last_active":"2023-06-22T08:00:00.155134+0000","last_peered":"2023-06-22T08:00:00.155134+0000","last_clean":"2023-06-22T08:00:00.155134+0000","last_became_active":"2023-06-22T07:59:42.500320+0000","last_became_peered":"2023-06-22T07:59:42.500320+0000","last_unstale":"2023-06-22T08:00:00.155134+0000","last_undegraded":"2023-06-22T08:00:00.155134+0000","last_fullsized":"2023-06-22T08:00:00.155134+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_clean_scrub_stamp":"2023-06-22T07:59:41.381475+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:08:53.784030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.569235+0000","last_change":"2023-06-22T07:59:54.645031+0000","last_active":"2023-06-22T08:00:41.569235+0000","last_peered":"2023-06-22T08:00:41.569235+0000","last_clean":"2023-06-22T08:00:41.569235+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:41.569235+0000","last_undegraded":"2023-06-22T08:00:41.569235+0000","last_fullsized":"2023-06-22T08:00:41.569235+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:42.766407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872074+0000","last_change":"2023-06-22T07:59:55.162799+0000","last_active":"2023-06-22T07:59:59.872074+0000","last_peered":"2023-06-22T07:59:59.872074+0000","last_clean":"2023-06-22T07:59:59.872074+0000","last_became_active":"2023-06-22T07:59:55.162501+0000","last_became_peered":"2023-06-22T07:59:55.162501+0000","last_unstale":"2023-06-22T07:59:59.872074+0000","last_undegraded":"2023-06-22T07:59:59.872074+0000","last_fullsized":"2023-06-22T07:59:59.872074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:57:14.305897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871805+0000","last_change":"2023-06-22T07:59:59.205097+0000","last_active":"2023-06-22T07:59:59.871805+0000","last_peered":"2023-06-22T07:59:59.871805+0000","last_clean":"2023-06-22T07:59:59.871805+0000","last_became_active":"2023-06-22T07:59:59.201465+0000","last_became_peered":"2023-06-22T07:59:59.201465+0000","last_unstale":"2023-06-22T07:59:59.871805+0000","last_undegraded":"2023-06-22T07:59:59.871805+0000","last_fullsized":"2023-06-22T07:59:59.871805+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:17:45.352998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,0],"acting":[3,7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871056+0000","last_change":"2023-06-22T07:59:49.442558+0000","last_active":"2023-06-22T07:59:59.871056+0000","last_peered":"2023-06-22T07:59:59.871056+0000","last_clean":"2023-06-22T07:59:59.871056+0000","last_became_active":"2023-06-22T07:59:49.442154+0000","last_became_peered":"2023-06-22T07:59:49.442154+0000","last_unstale":"2023-06-22T07:59:59.871056+0000","last_undegraded":"2023-06-22T07:59:59.871056+0000","last_fullsized":"2023-06-22T07:59:59.871056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:46:16.233204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:31.399798+0000","last_change":"2023-06-22T07:59:54.641746+0000","last_active":"2023-06-22T08:00:31.399798+0000","last_peered":"2023-06-22T08:00:31.399798+0000","last_clean":"2023-06-22T08:00:31.399798+0000","last_became_active":"2023-06-22T07:59:54.641431+0000","last_became_peered":"2023-06-22T07:59:54.641431+0000","last_unstale":"2023-06-22T08:00:31.399798+0000","last_undegraded":"2023-06-22T08:00:31.399798+0000","last_fullsized":"2023-06-22T08:00:31.399798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709468+0000","last_change":"2023-06-22T07:59:55.163214+0000","last_active":"2023-06-22T07:59:59.709468+0000","last_peered":"2023-06-22T07:59:59.709468+0000","last_clean":"2023-06-22T07:59:59.709468+0000","last_became_active":"2023-06-22T07:59:55.162907+0000","last_became_peered":"2023-06-22T07:59:55.162907+0000","last_unstale":"2023-06-22T07:59:59.709468+0000","last_undegraded":"2023-06-22T07:59:59.709468+0000","last_fullsized":"2023-06-22T07:59:59.709468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:46.988703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157450+0000","last_change":"2023-06-22T07:59:59.198945+0000","last_active":"2023-06-22T08:00:00.157450+0000","last_peered":"2023-06-22T08:00:00.157450+0000","last_clean":"2023-06-22T08:00:00.157450+0000","last_became_active":"2023-06-22T07:59:59.198561+0000","last_became_peered":"2023-06-22T07:59:59.198561+0000","last_unstale":"2023-06-22T08:00:00.157450+0000","last_undegraded":"2023-06-22T08:00:00.157450+0000","last_fullsized":"2023-06-22T08:00:00.157450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:06.135651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2,5],"acting":[0,7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871842+0000","last_change":"2023-06-22T07:59:49.442246+0000","last_active":"2023-06-22T07:59:59.871842+0000","last_peered":"2023-06-22T07:59:59.871842+0000","last_clean":"2023-06-22T07:59:59.871842+0000","last_became_active":"2023-06-22T07:59:49.442001+0000","last_became_peered":"2023-06-22T07:59:49.442001+0000","last_unstale":"2023-06-22T07:59:59.871842+0000","last_undegraded":"2023-06-22T07:59:59.871842+0000","last_fullsized":"2023-06-22T07:59:59.871842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:27:24.457978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754302+0000","last_change":"2023-06-22T07:59:54.643272+0000","last_active":"2023-06-22T08:00:00.754302+0000","last_peered":"2023-06-22T08:00:00.754302+0000","last_clean":"2023-06-22T08:00:00.754302+0000","last_became_active":"2023-06-22T07:59:54.642925+0000","last_became_peered":"2023-06-22T07:59:54.642925+0000","last_unstale":"2023-06-22T08:00:00.754302+0000","last_undegraded":"2023-06-22T08:00:00.754302+0000","last_fullsized":"2023-06-22T08:00:00.754302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:13.625768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754231+0000","last_change":"2023-06-22T07:59:55.167307+0000","last_active":"2023-06-22T08:00:00.754231+0000","last_peered":"2023-06-22T08:00:00.754231+0000","last_clean":"2023-06-22T08:00:00.754231+0000","last_became_active":"2023-06-22T07:59:55.167026+0000","last_became_peered":"2023-06-22T07:59:55.167026+0000","last_unstale":"2023-06-22T08:00:00.754231+0000","last_undegraded":"2023-06-22T08:00:00.754231+0000","last_fullsized":"2023-06-22T08:00:00.754231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871379+0000","last_change":"2023-06-22T07:59:59.208263+0000","last_active":"2023-06-22T07:59:59.871379+0000","last_peered":"2023-06-22T07:59:59.871379+0000","last_clean":"2023-06-22T07:59:59.871379+0000","last_became_active":"2023-06-22T07:59:59.207691+0000","last_became_peered":"2023-06-22T07:59:59.207691+0000","last_unstale":"2023-06-22T07:59:59.871379+0000","last_undegraded":"2023-06-22T07:59:59.871379+0000","last_fullsized":"2023-06-22T07:59:59.871379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:37:57.170857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,2],"acting":[3,7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.564652+0000","last_change":"2023-06-22T07:59:54.644570+0000","last_active":"2023-06-22T08:00:41.564652+0000","last_peered":"2023-06-22T08:00:41.564652+0000","last_clean":"2023-06-22T08:00:41.564652+0000","last_became_active":"2023-06-22T07:59:54.644218+0000","last_became_peered":"2023-06-22T07:59:54.644218+0000","last_unstale":"2023-06-22T08:00:41.564652+0000","last_undegraded":"2023-06-22T08:00:41.564652+0000","last_fullsized":"2023-06-22T08:00:41.564652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:05:00.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754571+0000","last_change":"2023-06-22T07:59:49.655601+0000","last_active":"2023-06-22T08:00:00.754571+0000","last_peered":"2023-06-22T08:00:00.754571+0000","last_clean":"2023-06-22T08:00:00.754571+0000","last_became_active":"2023-06-22T07:59:49.655328+0000","last_became_peered":"2023-06-22T07:59:49.655328+0000","last_unstale":"2023-06-22T08:00:00.754571+0000","last_undegraded":"2023-06-22T08:00:00.754571+0000","last_fullsized":"2023-06-22T08:00:00.754571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:29:09.967964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755018+0000","last_change":"2023-06-22T07:59:55.163241+0000","last_active":"2023-06-22T08:00:00.755018+0000","last_peered":"2023-06-22T08:00:00.755018+0000","last_clean":"2023-06-22T08:00:00.755018+0000","last_became_active":"2023-06-22T07:59:55.162949+0000","last_became_peered":"2023-06-22T07:59:55.162949+0000","last_unstale":"2023-06-22T08:00:00.755018+0000","last_undegraded":"2023-06-22T08:00:00.755018+0000","last_fullsized":"2023-06-22T08:00:00.755018+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:51.267726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755485+0000","last_change":"2023-06-22T07:59:59.201187+0000","last_active":"2023-06-22T08:00:00.755485+0000","last_peered":"2023-06-22T08:00:00.755485+0000","last_clean":"2023-06-22T08:00:00.755485+0000","last_became_active":"2023-06-22T07:59:59.200688+0000","last_became_peered":"2023-06-22T07:59:59.200688+0000","last_unstale":"2023-06-22T08:00:00.755485+0000","last_undegraded":"2023-06-22T08:00:00.755485+0000","last_fullsized":"2023-06-22T08:00:00.755485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:52:15.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2,6],"acting":[4,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871926+0000","last_change":"2023-06-22T07:59:49.655839+0000","last_active":"2023-06-22T07:59:59.871926+0000","last_peered":"2023-06-22T07:59:59.871926+0000","last_clean":"2023-06-22T07:59:59.871926+0000","last_became_active":"2023-06-22T07:59:49.655494+0000","last_became_peered":"2023-06-22T07:59:49.655494+0000","last_unstale":"2023-06-22T07:59:59.871926+0000","last_undegraded":"2023-06-22T07:59:59.871926+0000","last_fullsized":"2023-06-22T07:59:59.871926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:01:23.950347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.490383+0000","last_change":"2023-06-22T07:59:54.641464+0000","last_active":"2023-06-22T08:00:36.490383+0000","last_peered":"2023-06-22T08:00:36.490383+0000","last_clean":"2023-06-22T08:00:36.490383+0000","last_became_active":"2023-06-22T07:59:54.641161+0000","last_became_peered":"2023-06-22T07:59:54.641161+0000","last_unstale":"2023-06-22T08:00:36.490383+0000","last_undegraded":"2023-06-22T08:00:36.490383+0000","last_fullsized":"2023-06-22T08:00:36.490383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:46.346592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708316+0000","last_change":"2023-06-22T07:59:55.164629+0000","last_active":"2023-06-22T07:59:59.708316+0000","last_peered":"2023-06-22T07:59:59.708316+0000","last_clean":"2023-06-22T07:59:59.708316+0000","last_became_active":"2023-06-22T07:59:55.164329+0000","last_became_peered":"2023-06-22T07:59:55.164329+0000","last_unstale":"2023-06-22T07:59:59.708316+0000","last_undegraded":"2023-06-22T07:59:59.708316+0000","last_fullsized":"2023-06-22T07:59:59.708316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754950+0000","last_change":"2023-06-22T07:59:59.201778+0000","last_active":"2023-06-22T08:00:00.754950+0000","last_peered":"2023-06-22T08:00:00.754950+0000","last_clean":"2023-06-22T08:00:00.754950+0000","last_became_active":"2023-06-22T07:59:59.201326+0000","last_became_peered":"2023-06-22T07:59:59.201326+0000","last_unstale":"2023-06-22T08:00:00.754950+0000","last_undegraded":"2023-06-22T08:00:00.754950+0000","last_fullsized":"2023-06-22T08:00:00.754950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:17:31.817024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.703139+0000","last_change":"2023-06-22T07:59:54.644816+0000","last_active":"2023-06-22T08:00:25.703139+0000","last_peered":"2023-06-22T08:00:25.703139+0000","last_clean":"2023-06-22T08:00:25.703139+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:25.703139+0000","last_undegraded":"2023-06-22T08:00:25.703139+0000","last_fullsized":"2023-06-22T08:00:25.703139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:33.819054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753735+0000","last_change":"2023-06-22T07:59:55.167979+0000","last_active":"2023-06-22T08:00:00.753735+0000","last_peered":"2023-06-22T08:00:00.753735+0000","last_clean":"2023-06-22T08:00:00.753735+0000","last_became_active":"2023-06-22T07:59:55.163775+0000","last_became_peered":"2023-06-22T07:59:55.163775+0000","last_unstale":"2023-06-22T08:00:00.753735+0000","last_undegraded":"2023-06-22T08:00:00.753735+0000","last_fullsized":"2023-06-22T08:00:00.753735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155135+0000","last_change":"2023-06-22T07:59:59.194581+0000","last_active":"2023-06-22T08:00:00.155135+0000","last_peered":"2023-06-22T08:00:00.155135+0000","last_clean":"2023-06-22T08:00:00.155135+0000","last_became_active":"2023-06-22T07:59:59.194135+0000","last_became_peered":"2023-06-22T07:59:59.194135+0000","last_unstale":"2023-06-22T08:00:00.155135+0000","last_undegraded":"2023-06-22T08:00:00.155135+0000","last_fullsized":"2023-06-22T08:00:00.155135+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.565859+0000","last_change":"2023-06-22T07:59:54.651138+0000","last_active":"2023-06-22T08:00:41.565859+0000","last_peered":"2023-06-22T08:00:41.565859+0000","last_clean":"2023-06-22T08:00:41.565859+0000","last_became_active":"2023-06-22T07:59:54.650803+0000","last_became_peered":"2023-06-22T07:59:54.650803+0000","last_unstale":"2023-06-22T08:00:41.565859+0000","last_undegraded":"2023-06-22T08:00:41.565859+0000","last_fullsized":"2023-06-22T08:00:41.565859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:45.982711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708617+0000","last_change":"2023-06-22T07:59:55.165814+0000","last_active":"2023-06-22T07:59:59.708617+0000","last_peered":"2023-06-22T07:59:59.708617+0000","last_clean":"2023-06-22T07:59:59.708617+0000","last_became_active":"2023-06-22T07:59:55.165270+0000","last_became_peered":"2023-06-22T07:59:55.165270+0000","last_unstale":"2023-06-22T07:59:59.708617+0000","last_undegraded":"2023-06-22T07:59:59.708617+0000","last_fullsized":"2023-06-22T07:59:59.708617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708465+0000","last_change":"2023-06-22T07:59:59.212042+0000","last_active":"2023-06-22T07:59:59.708465+0000","last_peered":"2023-06-22T07:59:59.708465+0000","last_clean":"2023-06-22T07:59:59.708465+0000","last_became_active":"2023-06-22T07:59:59.211666+0000","last_became_peered":"2023-06-22T07:59:59.211666+0000","last_unstale":"2023-06-22T07:59:59.708465+0000","last_undegraded":"2023-06-22T07:59:59.708465+0000","last_fullsized":"2023-06-22T07:59:59.708465+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,4],"acting":[2,7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"28'12","reported_seq":33,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.571232+0000","last_change":"2023-06-22T07:59:54.650251+0000","last_active":"2023-06-22T08:00:41.571232+0000","last_peered":"2023-06-22T08:00:41.571232+0000","last_clean":"2023-06-22T08:00:41.571232+0000","last_became_active":"2023-06-22T07:59:54.649993+0000","last_became_peered":"2023-06-22T07:59:54.649993+0000","last_unstale":"2023-06-22T08:00:41.571232+0000","last_undegraded":"2023-06-22T08:00:41.571232+0000","last_fullsized":"2023-06-22T08:00:41.571232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:20:20.733896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708375+0000","last_change":"2023-06-22T07:59:55.172595+0000","last_active":"2023-06-22T07:59:59.708375+0000","last_peered":"2023-06-22T07:59:59.708375+0000","last_clean":"2023-06-22T07:59:59.708375+0000","last_became_active":"2023-06-22T07:59:55.165873+0000","last_became_peered":"2023-06-22T07:59:55.165873+0000","last_unstale":"2023-06-22T07:59:59.708375+0000","last_undegraded":"2023-06-22T07:59:59.708375+0000","last_fullsized":"2023-06-22T07:59:59.708375+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:02.592807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154969+0000","last_change":"2023-06-22T07:59:59.192708+0000","last_active":"2023-06-22T08:00:00.154969+0000","last_peered":"2023-06-22T08:00:00.154969+0000","last_clean":"2023-06-22T08:00:00.154969+0000","last_became_active":"2023-06-22T07:59:59.191693+0000","last_became_peered":"2023-06-22T07:59:59.191693+0000","last_unstale":"2023-06-22T08:00:00.154969+0000","last_undegraded":"2023-06-22T08:00:00.154969+0000","last_fullsized":"2023-06-22T08:00:00.154969+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T12:59:33.454962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5,7],"acting":[1,0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.151569+0000","last_change":"2023-06-22T07:59:54.651179+0000","last_active":"2023-06-22T08:00:00.151569+0000","last_peered":"2023-06-22T08:00:00.151569+0000","last_clean":"2023-06-22T08:00:00.151569+0000","last_became_active":"2023-06-22T07:59:54.650852+0000","last_became_peered":"2023-06-22T07:59:54.650852+0000","last_unstale":"2023-06-22T08:00:00.151569+0000","last_undegraded":"2023-06-22T08:00:00.151569+0000","last_fullsized":"2023-06-22T08:00:00.151569+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:01.351861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754043+0000","last_change":"2023-06-22T07:59:55.163050+0000","last_active":"2023-06-22T08:00:00.754043+0000","last_peered":"2023-06-22T08:00:00.754043+0000","last_clean":"2023-06-22T08:00:00.754043+0000","last_became_active":"2023-06-22T07:59:55.162754+0000","last_became_peered":"2023-06-22T07:59:55.162754+0000","last_unstale":"2023-06-22T08:00:00.754043+0000","last_undegraded":"2023-06-22T08:00:00.754043+0000","last_fullsized":"2023-06-22T08:00:00.754043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754797+0000","last_change":"2023-06-22T07:59:59.209558+0000","last_active":"2023-06-22T08:00:00.754797+0000","last_peered":"2023-06-22T08:00:00.754797+0000","last_clean":"2023-06-22T08:00:00.754797+0000","last_became_active":"2023-06-22T07:59:59.209129+0000","last_became_peered":"2023-06-22T07:59:59.209129+0000","last_unstale":"2023-06-22T08:00:00.754797+0000","last_undegraded":"2023-06-22T08:00:00.754797+0000","last_fullsized":"2023-06-22T08:00:00.754797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:21.581063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1,0],"acting":[4,3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"28'13","reported_seq":34,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.566206+0000","last_change":"2023-06-22T07:59:54.652059+0000","last_active":"2023-06-22T08:00:41.566206+0000","last_peered":"2023-06-22T08:00:41.566206+0000","last_clean":"2023-06-22T08:00:41.566206+0000","last_became_active":"2023-06-22T07:59:54.651720+0000","last_became_peered":"2023-06-22T07:59:54.651720+0000","last_unstale":"2023-06-22T08:00:41.566206+0000","last_undegraded":"2023-06-22T08:00:41.566206+0000","last_fullsized":"2023-06-22T08:00:41.566206+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:31.908657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708158+0000","last_change":"2023-06-22T07:59:55.172794+0000","last_active":"2023-06-22T07:59:59.708158+0000","last_peered":"2023-06-22T07:59:59.708158+0000","last_clean":"2023-06-22T07:59:59.708158+0000","last_became_active":"2023-06-22T07:59:55.172527+0000","last_became_peered":"2023-06-22T07:59:55.172527+0000","last_unstale":"2023-06-22T07:59:59.708158+0000","last_undegraded":"2023-06-22T07:59:59.708158+0000","last_fullsized":"2023-06-22T07:59:59.708158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155457+0000","last_change":"2023-06-22T07:59:55.165758+0000","last_active":"2023-06-22T08:00:00.155457+0000","last_peered":"2023-06-22T08:00:00.155457+0000","last_clean":"2023-06-22T08:00:00.155457+0000","last_became_active":"2023-06-22T07:59:55.165117+0000","last_became_peered":"2023-06-22T07:59:55.165117+0000","last_unstale":"2023-06-22T08:00:00.155457+0000","last_undegraded":"2023-06-22T08:00:00.155457+0000","last_fullsized":"2023-06-22T08:00:00.155457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:02.357853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753486+0000","last_change":"2023-06-22T07:59:55.162439+0000","last_active":"2023-06-22T08:00:00.753486+0000","last_peered":"2023-06-22T08:00:00.753486+0000","last_clean":"2023-06-22T08:00:00.753486+0000","last_became_active":"2023-06-22T07:59:55.162072+0000","last_became_peered":"2023-06-22T07:59:55.162072+0000","last_unstale":"2023-06-22T08:00:00.753486+0000","last_undegraded":"2023-06-22T08:00:00.753486+0000","last_fullsized":"2023-06-22T08:00:00.753486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157229+0000","last_change":"2023-06-22T07:59:55.159299+0000","last_active":"2023-06-22T08:00:00.157229+0000","last_peered":"2023-06-22T08:00:00.157229+0000","last_clean":"2023-06-22T08:00:00.157229+0000","last_became_active":"2023-06-22T07:59:55.158893+0000","last_became_peered":"2023-06-22T07:59:55.158893+0000","last_unstale":"2023-06-22T08:00:00.157229+0000","last_undegraded":"2023-06-22T08:00:00.157229+0000","last_fullsized":"2023-06-22T08:00:00.157229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709716+0000","last_change":"2023-06-22T07:59:55.162520+0000","last_active":"2023-06-22T07:59:59.709716+0000","last_peered":"2023-06-22T07:59:59.709716+0000","last_clean":"2023-06-22T07:59:59.709716+0000","last_became_active":"2023-06-22T07:59:55.162211+0000","last_became_peered":"2023-06-22T07:59:55.162211+0000","last_unstale":"2023-06-22T07:59:59.709716+0000","last_undegraded":"2023-06-22T07:59:59.709716+0000","last_fullsized":"2023-06-22T07:59:59.709716+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:13:00.662547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708947+0000","last_change":"2023-06-22T07:59:55.165364+0000","last_active":"2023-06-22T07:59:59.708947+0000","last_peered":"2023-06-22T07:59:59.708947+0000","last_clean":"2023-06-22T07:59:59.708947+0000","last_became_active":"2023-06-22T07:59:55.165021+0000","last_became_peered":"2023-06-22T07:59:55.165021+0000","last_unstale":"2023-06-22T07:59:59.708947+0000","last_undegraded":"2023-06-22T07:59:59.708947+0000","last_fullsized":"2023-06-22T07:59:59.708947+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:09:07.008945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":10282,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":211,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476755,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":344064,"data_stored":128982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.64400000000000002}]},{"osd":1,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.437},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.55000000000000004}]},{"osd":3,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.49099999999999999}]},{"osd":6,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.46400000000000002}]},{"osd":7,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.58499999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542162,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":344064,"data_stored":128648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.503}]},{"osd":4,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.73599999999999999}]},{"osd":6,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.748}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":319488,"data_stored":124784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.4989999999999997,"5min":6.4989999999999997,"15min":6.4989999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":10.401,"5min":10.401,"15min":10.401},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":10.173999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":10.291,"5min":10.291,"15min":10.291},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":10.064,"5min":10.064,"15min":10.064},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.39900000000000002}]},{"osd":6,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.378},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.64100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":921600,"data_stored":715384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.312},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.82699999999999996}]},{"osd":2,"last update":"Thu Jun 22 08:00:47 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57499999999999996}]},{"osd":3,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.53800000000000003}]},{"osd":4,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67000000000000004}]},{"osd":6,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.73999999999999999}]},{"osd":7,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.64400000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":921600,"data_stored":715472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.623}]},{"osd":2,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.74199999999999999}]},{"osd":4,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.79000000000000004}]},{"osd":6,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.753}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":335872,"data_stored":125148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.7000000000000002,"5min":7.7000000000000002,"15min":7.7000000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.8010000000000002,"5min":3.8010000000000002,"15min":3.8010000000000002},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jun 22 08:00:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":9.8460000000000001,"5min":9.8460000000000001,"15min":9.8460000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.7210000000000001,"5min":2.7210000000000001,"15min":2.7210000000000001},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.867,"5min":1.867,"15min":1.867},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":9.9019999999999992,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":0.502}]},{"osd":4,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48599999999999999}]}]},{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":307200,"data_stored":124680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51500000000000001}]},{"osd":3,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.496},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.36499999999999999}]},{"osd":5,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.47099999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":348160,"data_stored":126666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.917,"5min":1.917,"15min":1.917},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.8099999999999996,"5min":6.8099999999999996,"15min":6.8099999999999996},"last":0.32600000000000001}]},{"osd":2,"last update":"Thu Jun 22 08:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.6589999999999998},"last":0.438},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-22T08:00:57.919 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-06-22T08:00:57.920 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-22T08:00:58.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.147+0000 7fe3f81ff700 1 Processor -- start 2023-06-22T08:00:58.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.147+0000 7fe3f81ff700 1 -- start start 2023-06-22T08:00:58.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f81ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3f0152d70 0x7fe3f00c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f81ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe3f0159330 0x7fe3f00cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f81ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f0159790 con 0x7fe3f00cb800 2023-06-22T08:00:58.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f81ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f00c95e0 con 0x7fe3f0159330 2023-06-22T08:00:58.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f81ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f015bea0 con 0x7fe3f0152d70 2023-06-22T08:00:58.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f579a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe3f0159330 0x7fe3f00cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f5f9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3f0152d70 0x7fe3f00c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f679c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe3f00cb800 0x7fe3f00c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:51462/0 (socket says 172.21.15.23:51462) 2023-06-22T08:00:58.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f679c700 1 -- 172.21.15.23:0/2698518894 learned_addr learned my addr 172.21.15.23:0/2698518894 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:58.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f579a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe3f0159330 0x7fe3f00cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52850/0 (socket says 172.21.15.23:52850) 2023-06-22T08:00:58.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f579a700 1 -- 172.21.15.23:0/2698518894 >> v2:172.21.15.67:3300/0 conn(0x7fe3f0152d70 msgr2=0x7fe3f00c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:58.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f579a700 1 --2- 172.21.15.23:0/2698518894 >> v2:172.21.15.67:3300/0 conn(0x7fe3f0152d70 0x7fe3f00c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f579a700 1 -- 172.21.15.23:0/2698518894 >> v1:172.21.15.67:6789/0 conn(0x7fe3f00cb800 legacy=0x7fe3f00c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f579a700 1 -- 172.21.15.23:0/2698518894 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f015c160 con 0x7fe3f0159330 2023-06-22T08:00:58.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.149+0000 7fe3f5f9b700 1 --2- 172.21.15.23:0/2698518894 >> v2:172.21.15.67:3300/0 conn(0x7fe3f0152d70 0x7fe3f00c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:58.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.150+0000 7fe3f579a700 1 --2- 172.21.15.23:0/2698518894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f0159330 0x7fe3f00cb2c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe3f00c8ef0 tx=0x7fe3e0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2b51c979621d18 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:58.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.150+0000 7fe3f4f99700 1 -- 172.21.15.23:0/2698518894 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0011830 con 0x7fe3f0159330 2023-06-22T08:00:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.150+0000 7fe3f4f99700 1 -- 172.21.15.23:0/2698518894 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e0011990 con 0x7fe3f0159330 2023-06-22T08:00:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.150+0000 7fe3f4f99700 1 -- 172.21.15.23:0/2698518894 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0011d20 con 0x7fe3f0159330 2023-06-22T08:00:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.150+0000 7fe3f81ff700 1 -- 172.21.15.23:0/2698518894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f0159330 msgr2=0x7fe3f00cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.150+0000 7fe3f81ff700 1 --2- 172.21.15.23:0/2698518894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f0159330 0x7fe3f00cb2c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe3f00c8ef0 tx=0x7fe3e0009b20 comp rx=0 tx=0).stop 2023-06-22T08:00:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.151+0000 7fe3f81ff700 1 -- 172.21.15.23:0/2698518894 shutdown_connections 2023-06-22T08:00:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.151+0000 7fe3f81ff700 1 --2- 172.21.15.23:0/2698518894 >> v2:172.21.15.67:3300/0 conn(0x7fe3f0152d70 0x7fe3f00c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.151+0000 7fe3f81ff700 1 --2- 172.21.15.23:0/2698518894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f0159330 0x7fe3f00cb2c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.151+0000 7fe3f81ff700 1 -- 172.21.15.23:0/2698518894 >> 172.21.15.23:0/2698518894 conn(0x7fe3f006ae30 msgr2=0x7fe3f006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:58.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.151+0000 7fe3f81ff700 1 -- 172.21.15.23:0/2698518894 shutdown_connections 2023-06-22T08:00:58.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.151+0000 7fe3f81ff700 1 -- 172.21.15.23:0/2698518894 wait complete. 2023-06-22T08:00:58.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.152+0000 7fe3f81ff700 1 Processor -- start 2023-06-22T08:00:58.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.153+0000 7fe3f81ff700 1 -- start start 2023-06-22T08:00:58.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.153+0000 7fe3f81ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f00cb800 0x7fe3f0151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.153+0000 7fe3f81ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3f0152d70 0x7fe3f01ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f81ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f0159330 0x7fe3f01efe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f81ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f0150f70 con 0x7fe3f00cb800 2023-06-22T08:00:58.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f81ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f01ef1f0 con 0x7fe3f0159330 2023-06-22T08:00:58.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f81ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f014bfd0 con 0x7fe3f0152d70 2023-06-22T08:00:58.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f5f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f00cb800 0x7fe3f0151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f5f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f00cb800 0x7fe3f0151d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52858/0 (socket says 172.21.15.23:52858) 2023-06-22T08:00:58.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f5f9b700 1 -- 172.21.15.23:0/1705534729 learned_addr learned my addr 172.21.15.23:0/1705534729 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:58.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f579a700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3f0152d70 0x7fe3f01ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f679c700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f0159330 0x7fe3f01efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f5f9b700 1 -- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3f0152d70 msgr2=0x7fe3f01ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f5f9b700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3f0152d70 0x7fe3f01ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f5f9b700 1 -- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f0159330 msgr2=0x7fe3f01efe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f5f9b700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f0159330 0x7fe3f01efe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f5f9b700 1 -- 172.21.15.23:0/1705534729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec0094a0 con 0x7fe3f00cb800 2023-06-22T08:00:58.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.154+0000 7fe3f5f9b700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f00cb800 0x7fe3f0151d00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00ebe0 tx=0x7fe3ec00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:58.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.155+0000 7fe3e67fc700 1 -- 172.21.15.23:0/1705534729 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec00c850 con 0x7fe3f00cb800 2023-06-22T08:00:58.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.155+0000 7fe3e67fc700 1 -- 172.21.15.23:0/1705534729 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ec00c9b0 con 0x7fe3f00cb800 2023-06-22T08:00:58.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.155+0000 7fe3e67fc700 1 -- 172.21.15.23:0/1705534729 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec00cd40 con 0x7fe3f00cb800 2023-06-22T08:00:58.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.155+0000 7fe3f81ff700 1 -- 172.21.15.23:0/1705534729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e00094a0 con 0x7fe3f00cb800 2023-06-22T08:00:58.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.155+0000 7fe3f81ff700 1 -- 172.21.15.23:0/1705534729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f014c220 con 0x7fe3f00cb800 2023-06-22T08:00:58.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.156+0000 7fe3f81ff700 1 -- 172.21.15.23:0/1705534729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f004ea20 con 0x7fe3f00cb800 2023-06-22T08:00:58.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.157+0000 7fe3e67fc700 1 -- 172.21.15.23:0/1705534729 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe3ec013470 con 0x7fe3f00cb800 2023-06-22T08:00:58.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.158+0000 7fe3e67fc700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3d406ace0 0x7fe3d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.158+0000 7fe3e67fc700 1 -- 172.21.15.23:0/1705534729 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fe3ec089bd0 con 0x7fe3f00cb800 2023-06-22T08:00:58.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.158+0000 7fe3f579a700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3d406ace0 0x7fe3d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.159+0000 7fe3f579a700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3d406ace0 0x7fe3d406d190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00090b0 tx=0x7fe3e0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:58.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.162+0000 7fe3e67fc700 1 -- 172.21.15.23:0/1705534729 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe3ec054980 con 0x7fe3f00cb800 2023-06-22T08:00:58.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.366+0000 7fe3f81ff700 1 -- 172.21.15.23:0/1705534729 --> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3f015c720 con 0x7fe3d406ace0 2023-06-22T08:00:58.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.379+0000 7fe3e67fc700 1 -- 172.21.15.23:0/1705534729 <== mgr.4100 v2:172.21.15.67:6832/3180103139 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+392072 (secure 0 0 0) 0x7fe3e0011c30 con 0x7fe3d406ace0 2023-06-22T08:00:58.386 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:58.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 -- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3d406ace0 msgr2=0x7fe3d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3d406ace0 0x7fe3d406d190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00090b0 tx=0x7fe3e0008030 comp rx=0 tx=0).stop 2023-06-22T08:00:58.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 -- 172.21.15.23:0/1705534729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f00cb800 msgr2=0x7fe3f0151d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f00cb800 0x7fe3f0151d00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00ebe0 tx=0x7fe3ec00eef0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 -- 172.21.15.23:0/1705534729 shutdown_connections 2023-06-22T08:00:58.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3d406ace0 0x7fe3d406d190 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3f0159330 0x7fe3f01efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f00cb800 0x7fe3f0151d00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 --2- 172.21.15.23:0/1705534729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3f0152d70 0x7fe3f01ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 -- 172.21.15.23:0/1705534729 >> 172.21.15.23:0/1705534729 conn(0x7fe3f006ae30 msgr2=0x7fe3f014f820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.383+0000 7fe3f81ff700 1 -- 172.21.15.23:0/1705534729 shutdown_connections 2023-06-22T08:00:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.384+0000 7fe3f81ff700 1 -- 172.21.15.23:0/1705534729 wait complete. 2023-06-22T08:00:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2023-06-22T08:00:58.405 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2023-06-22T08:00:56.571499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600650,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":370,"num_write_kb":2485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219368,"kb_used_data":3752,"kb_used_omap":0,"kb_used_meta":215552,"kb_avail":754755352,"statfs":{"total":773094113280,"available":772869480448,"internally_reserved":0,"allocated":3842048,"data_stored":2189764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":220725248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1492,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003943"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709823+0000","last_change":"2023-06-22T07:59:55.165740+0000","last_active":"2023-06-22T07:59:59.709823+0000","last_peered":"2023-06-22T07:59:59.709823+0000","last_clean":"2023-06-22T07:59:59.709823+0000","last_became_active":"2023-06-22T07:59:55.165398+0000","last_became_peered":"2023-06-22T07:59:55.165398+0000","last_unstale":"2023-06-22T07:59:59.709823+0000","last_undegraded":"2023-06-22T07:59:59.709823+0000","last_fullsized":"2023-06-22T07:59:59.709823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:37.596656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709026+0000","last_change":"2023-06-22T07:59:55.174603+0000","last_active":"2023-06-22T07:59:59.709026+0000","last_peered":"2023-06-22T07:59:59.709026+0000","last_clean":"2023-06-22T07:59:59.709026+0000","last_became_active":"2023-06-22T07:59:55.174346+0000","last_became_peered":"2023-06-22T07:59:55.174346+0000","last_unstale":"2023-06-22T07:59:59.709026+0000","last_undegraded":"2023-06-22T07:59:59.709026+0000","last_fullsized":"2023-06-22T07:59:59.709026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157321+0000","last_change":"2023-06-22T07:59:55.167525+0000","last_active":"2023-06-22T08:00:00.157321+0000","last_peered":"2023-06-22T08:00:00.157321+0000","last_clean":"2023-06-22T08:00:00.157321+0000","last_became_active":"2023-06-22T07:59:55.167181+0000","last_became_peered":"2023-06-22T07:59:55.167181+0000","last_unstale":"2023-06-22T08:00:00.157321+0000","last_undegraded":"2023-06-22T08:00:00.157321+0000","last_fullsized":"2023-06-22T08:00:00.157321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157927+0000","last_change":"2023-06-22T07:59:55.160549+0000","last_active":"2023-06-22T08:00:00.157927+0000","last_peered":"2023-06-22T08:00:00.157927+0000","last_clean":"2023-06-22T08:00:00.157927+0000","last_became_active":"2023-06-22T07:59:55.160239+0000","last_became_peered":"2023-06-22T07:59:55.160239+0000","last_unstale":"2023-06-22T08:00:00.157927+0000","last_undegraded":"2023-06-22T08:00:00.157927+0000","last_fullsized":"2023-06-22T08:00:00.157927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754390+0000","last_change":"2023-06-22T07:59:55.173006+0000","last_active":"2023-06-22T08:00:00.754390+0000","last_peered":"2023-06-22T08:00:00.754390+0000","last_clean":"2023-06-22T08:00:00.754390+0000","last_became_active":"2023-06-22T07:59:55.172578+0000","last_became_peered":"2023-06-22T07:59:55.172578+0000","last_unstale":"2023-06-22T08:00:00.754390+0000","last_undegraded":"2023-06-22T08:00:00.754390+0000","last_fullsized":"2023-06-22T08:00:00.754390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:12:44.799856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871378+0000","last_change":"2023-06-22T07:59:55.173858+0000","last_active":"2023-06-22T07:59:59.871378+0000","last_peered":"2023-06-22T07:59:59.871378+0000","last_clean":"2023-06-22T07:59:59.871378+0000","last_became_active":"2023-06-22T07:59:55.173599+0000","last_became_peered":"2023-06-22T07:59:55.173599+0000","last_unstale":"2023-06-22T07:59:59.871378+0000","last_undegraded":"2023-06-22T07:59:59.871378+0000","last_fullsized":"2023-06-22T07:59:59.871378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:37.597195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871174+0000","last_change":"2023-06-22T07:59:55.168354+0000","last_active":"2023-06-22T07:59:59.871174+0000","last_peered":"2023-06-22T07:59:59.871174+0000","last_clean":"2023-06-22T07:59:59.871174+0000","last_became_active":"2023-06-22T07:59:55.168050+0000","last_became_peered":"2023-06-22T07:59:55.168050+0000","last_unstale":"2023-06-22T07:59:59.871174+0000","last_undegraded":"2023-06-22T07:59:59.871174+0000","last_fullsized":"2023-06-22T07:59:59.871174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:08:45.847378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155602+0000","last_change":"2023-06-22T07:59:55.159738+0000","last_active":"2023-06-22T08:00:00.155602+0000","last_peered":"2023-06-22T08:00:00.155602+0000","last_clean":"2023-06-22T08:00:00.155602+0000","last_became_active":"2023-06-22T07:59:55.159501+0000","last_became_peered":"2023-06-22T07:59:55.159501+0000","last_unstale":"2023-06-22T08:00:00.155602+0000","last_undegraded":"2023-06-22T08:00:00.155602+0000","last_fullsized":"2023-06-22T08:00:00.155602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:55.160768+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:55.160508+0000","last_became_peered":"2023-06-22T07:59:55.160508+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:55:48.713145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155821+0000","last_change":"2023-06-22T07:59:55.166546+0000","last_active":"2023-06-22T08:00:00.155821+0000","last_peered":"2023-06-22T08:00:00.155821+0000","last_clean":"2023-06-22T08:00:00.155821+0000","last_became_active":"2023-06-22T07:59:55.166311+0000","last_became_peered":"2023-06-22T07:59:55.166311+0000","last_unstale":"2023-06-22T08:00:00.155821+0000","last_undegraded":"2023-06-22T08:00:00.155821+0000","last_fullsized":"2023-06-22T08:00:00.155821+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:35.822548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155943+0000","last_change":"2023-06-22T07:59:55.162295+0000","last_active":"2023-06-22T08:00:00.155943+0000","last_peered":"2023-06-22T08:00:00.155943+0000","last_clean":"2023-06-22T08:00:00.155943+0000","last_became_active":"2023-06-22T07:59:55.162010+0000","last_became_peered":"2023-06-22T07:59:55.162010+0000","last_unstale":"2023-06-22T08:00:00.155943+0000","last_undegraded":"2023-06-22T08:00:00.155943+0000","last_fullsized":"2023-06-22T08:00:00.155943+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709563+0000","last_change":"2023-06-22T07:59:55.158990+0000","last_active":"2023-06-22T07:59:59.709563+0000","last_peered":"2023-06-22T07:59:59.709563+0000","last_clean":"2023-06-22T07:59:59.709563+0000","last_became_active":"2023-06-22T07:59:55.158614+0000","last_became_peered":"2023-06-22T07:59:55.158614+0000","last_unstale":"2023-06-22T07:59:59.709563+0000","last_undegraded":"2023-06-22T07:59:59.709563+0000","last_fullsized":"2023-06-22T07:59:59.709563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755166+0000","last_change":"2023-06-22T07:59:55.161453+0000","last_active":"2023-06-22T08:00:00.755166+0000","last_peered":"2023-06-22T08:00:00.755166+0000","last_clean":"2023-06-22T08:00:00.755166+0000","last_became_active":"2023-06-22T07:59:55.161168+0000","last_became_peered":"2023-06-22T07:59:55.161168+0000","last_unstale":"2023-06-22T08:00:00.755166+0000","last_undegraded":"2023-06-22T08:00:00.755166+0000","last_fullsized":"2023-06-22T08:00:00.755166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155022+0000","last_change":"2023-06-22T07:59:55.164270+0000","last_active":"2023-06-22T08:00:00.155022+0000","last_peered":"2023-06-22T08:00:00.155022+0000","last_clean":"2023-06-22T08:00:00.155022+0000","last_became_active":"2023-06-22T07:59:55.163993+0000","last_became_peered":"2023-06-22T07:59:55.163993+0000","last_unstale":"2023-06-22T08:00:00.155022+0000","last_undegraded":"2023-06-22T08:00:00.155022+0000","last_fullsized":"2023-06-22T08:00:00.155022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:52.051830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157430+0000","last_change":"2023-06-22T07:59:55.169039+0000","last_active":"2023-06-22T08:00:00.157430+0000","last_peered":"2023-06-22T08:00:00.157430+0000","last_clean":"2023-06-22T08:00:00.157430+0000","last_became_active":"2023-06-22T07:59:55.168698+0000","last_became_peered":"2023-06-22T07:59:55.168698+0000","last_unstale":"2023-06-22T08:00:00.157430+0000","last_undegraded":"2023-06-22T08:00:00.157430+0000","last_fullsized":"2023-06-22T08:00:00.157430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871035+0000","last_change":"2023-06-22T07:59:55.158014+0000","last_active":"2023-06-22T07:59:59.871035+0000","last_peered":"2023-06-22T07:59:59.871035+0000","last_clean":"2023-06-22T07:59:59.871035+0000","last_became_active":"2023-06-22T07:59:55.157562+0000","last_became_peered":"2023-06-22T07:59:55.157562+0000","last_unstale":"2023-06-22T07:59:59.871035+0000","last_undegraded":"2023-06-22T07:59:59.871035+0000","last_fullsized":"2023-06-22T07:59:59.871035+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:08:33.856001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754963+0000","last_change":"2023-06-22T07:59:55.165957+0000","last_active":"2023-06-22T08:00:00.754963+0000","last_peered":"2023-06-22T08:00:00.754963+0000","last_clean":"2023-06-22T08:00:00.754963+0000","last_became_active":"2023-06-22T07:59:55.165643+0000","last_became_peered":"2023-06-22T07:59:55.165643+0000","last_unstale":"2023-06-22T08:00:00.754963+0000","last_undegraded":"2023-06-22T08:00:00.754963+0000","last_fullsized":"2023-06-22T08:00:00.754963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:10.134743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157391+0000","last_change":"2023-06-22T07:59:55.159858+0000","last_active":"2023-06-22T08:00:00.157391+0000","last_peered":"2023-06-22T08:00:00.157391+0000","last_clean":"2023-06-22T08:00:00.157391+0000","last_became_active":"2023-06-22T07:59:55.159444+0000","last_became_peered":"2023-06-22T07:59:55.159444+0000","last_unstale":"2023-06-22T08:00:00.157391+0000","last_undegraded":"2023-06-22T08:00:00.157391+0000","last_fullsized":"2023-06-22T08:00:00.157391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709424+0000","last_change":"2023-06-22T07:59:55.157162+0000","last_active":"2023-06-22T07:59:59.709424+0000","last_peered":"2023-06-22T07:59:59.709424+0000","last_clean":"2023-06-22T07:59:59.709424+0000","last_became_active":"2023-06-22T07:59:55.156882+0000","last_became_peered":"2023-06-22T07:59:55.156882+0000","last_unstale":"2023-06-22T07:59:59.709424+0000","last_undegraded":"2023-06-22T07:59:59.709424+0000","last_fullsized":"2023-06-22T07:59:59.709424+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:22:00.173854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155892+0000","last_change":"2023-06-22T07:59:55.163834+0000","last_active":"2023-06-22T08:00:00.155892+0000","last_peered":"2023-06-22T08:00:00.155892+0000","last_clean":"2023-06-22T08:00:00.155892+0000","last_became_active":"2023-06-22T07:59:55.163580+0000","last_became_peered":"2023-06-22T07:59:55.163580+0000","last_unstale":"2023-06-22T08:00:00.155892+0000","last_undegraded":"2023-06-22T08:00:00.155892+0000","last_fullsized":"2023-06-22T08:00:00.155892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:20.199223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155285+0000","last_change":"2023-06-22T07:59:55.164569+0000","last_active":"2023-06-22T08:00:00.155285+0000","last_peered":"2023-06-22T08:00:00.155285+0000","last_clean":"2023-06-22T08:00:00.155285+0000","last_became_active":"2023-06-22T07:59:55.164286+0000","last_became_peered":"2023-06-22T07:59:55.164286+0000","last_unstale":"2023-06-22T08:00:00.155285+0000","last_undegraded":"2023-06-22T08:00:00.155285+0000","last_fullsized":"2023-06-22T08:00:00.155285+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:17.386317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754137+0000","last_change":"2023-06-22T07:59:55.159883+0000","last_active":"2023-06-22T08:00:00.754137+0000","last_peered":"2023-06-22T08:00:00.754137+0000","last_clean":"2023-06-22T08:00:00.754137+0000","last_became_active":"2023-06-22T07:59:55.159588+0000","last_became_peered":"2023-06-22T07:59:55.159588+0000","last_unstale":"2023-06-22T08:00:00.754137+0000","last_undegraded":"2023-06-22T08:00:00.754137+0000","last_fullsized":"2023-06-22T08:00:00.754137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754941+0000","last_change":"2023-06-22T07:59:55.160638+0000","last_active":"2023-06-22T08:00:00.754941+0000","last_peered":"2023-06-22T08:00:00.754941+0000","last_clean":"2023-06-22T08:00:00.754941+0000","last_became_active":"2023-06-22T07:59:55.160315+0000","last_became_peered":"2023-06-22T07:59:55.160315+0000","last_unstale":"2023-06-22T08:00:00.754941+0000","last_undegraded":"2023-06-22T08:00:00.754941+0000","last_fullsized":"2023-06-22T08:00:00.754941+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:51:09.921626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155504+0000","last_change":"2023-06-22T07:59:55.165950+0000","last_active":"2023-06-22T08:00:00.155504+0000","last_peered":"2023-06-22T08:00:00.155504+0000","last_clean":"2023-06-22T08:00:00.155504+0000","last_became_active":"2023-06-22T07:59:55.165505+0000","last_became_peered":"2023-06-22T07:59:55.165505+0000","last_unstale":"2023-06-22T08:00:00.155504+0000","last_undegraded":"2023-06-22T08:00:00.155504+0000","last_fullsized":"2023-06-22T08:00:00.155504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:13.512588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755640+0000","last_change":"2023-06-22T07:59:55.157572+0000","last_active":"2023-06-22T08:00:00.755640+0000","last_peered":"2023-06-22T08:00:00.755640+0000","last_clean":"2023-06-22T08:00:00.755640+0000","last_became_active":"2023-06-22T07:59:55.157055+0000","last_became_peered":"2023-06-22T07:59:55.157055+0000","last_unstale":"2023-06-22T08:00:00.755640+0000","last_undegraded":"2023-06-22T08:00:00.755640+0000","last_fullsized":"2023-06-22T08:00:00.755640+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:34:35.685117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754095+0000","last_change":"2023-06-22T07:59:55.164696+0000","last_active":"2023-06-22T08:00:00.754095+0000","last_peered":"2023-06-22T08:00:00.754095+0000","last_clean":"2023-06-22T08:00:00.754095+0000","last_became_active":"2023-06-22T07:59:55.164179+0000","last_became_peered":"2023-06-22T07:59:55.164179+0000","last_unstale":"2023-06-22T08:00:00.754095+0000","last_undegraded":"2023-06-22T08:00:00.754095+0000","last_fullsized":"2023-06-22T08:00:00.754095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871504+0000","last_change":"2023-06-22T07:59:55.168838+0000","last_active":"2023-06-22T07:59:59.871504+0000","last_peered":"2023-06-22T07:59:59.871504+0000","last_clean":"2023-06-22T07:59:59.871504+0000","last_became_active":"2023-06-22T07:59:55.168551+0000","last_became_peered":"2023-06-22T07:59:55.168551+0000","last_unstale":"2023-06-22T07:59:59.871504+0000","last_undegraded":"2023-06-22T07:59:59.871504+0000","last_fullsized":"2023-06-22T07:59:59.871504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:02:36.615449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708648+0000","last_change":"2023-06-22T07:59:59.195356+0000","last_active":"2023-06-22T07:59:59.708648+0000","last_peered":"2023-06-22T07:59:59.708648+0000","last_clean":"2023-06-22T07:59:59.708648+0000","last_became_active":"2023-06-22T07:59:59.194814+0000","last_became_peered":"2023-06-22T07:59:59.194814+0000","last_unstale":"2023-06-22T07:59:59.708648+0000","last_undegraded":"2023-06-22T07:59:59.708648+0000","last_fullsized":"2023-06-22T07:59:59.708648+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:12.223267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6,0],"acting":[2,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493176+0000","last_change":"2023-06-22T07:59:54.651326+0000","last_active":"2023-06-22T08:00:36.493176+0000","last_peered":"2023-06-22T08:00:36.493176+0000","last_clean":"2023-06-22T08:00:36.493176+0000","last_became_active":"2023-06-22T07:59:54.651010+0000","last_became_peered":"2023-06-22T07:59:54.651010+0000","last_unstale":"2023-06-22T08:00:36.493176+0000","last_undegraded":"2023-06-22T08:00:36.493176+0000","last_fullsized":"2023-06-22T08:00:36.493176+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:17.769117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754538+0000","last_change":"2023-06-22T07:59:55.160345+0000","last_active":"2023-06-22T08:00:00.754538+0000","last_peered":"2023-06-22T08:00:00.754538+0000","last_clean":"2023-06-22T08:00:00.754538+0000","last_became_active":"2023-06-22T07:59:55.160071+0000","last_became_peered":"2023-06-22T07:59:55.160071+0000","last_unstale":"2023-06-22T08:00:00.754538+0000","last_undegraded":"2023-06-22T08:00:00.754538+0000","last_fullsized":"2023-06-22T08:00:00.754538+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708662+0000","last_change":"2023-06-22T07:59:59.196477+0000","last_active":"2023-06-22T07:59:59.708662+0000","last_peered":"2023-06-22T07:59:59.708662+0000","last_clean":"2023-06-22T07:59:59.708662+0000","last_became_active":"2023-06-22T07:59:59.196007+0000","last_became_peered":"2023-06-22T07:59:59.196007+0000","last_unstale":"2023-06-22T07:59:59.708662+0000","last_undegraded":"2023-06-22T07:59:59.708662+0000","last_fullsized":"2023-06-22T07:59:59.708662+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,5],"acting":[2,7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:31.397730+0000","last_change":"2023-06-22T07:59:54.653192+0000","last_active":"2023-06-22T08:00:31.397730+0000","last_peered":"2023-06-22T08:00:31.397730+0000","last_clean":"2023-06-22T08:00:31.397730+0000","last_became_active":"2023-06-22T07:59:54.652926+0000","last_became_peered":"2023-06-22T07:59:54.652926+0000","last_unstale":"2023-06-22T08:00:31.397730+0000","last_undegraded":"2023-06-22T08:00:31.397730+0000","last_fullsized":"2023-06-22T08:00:31.397730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:12:59.656555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754609+0000","last_change":"2023-06-22T07:59:55.163171+0000","last_active":"2023-06-22T08:00:00.754609+0000","last_peered":"2023-06-22T08:00:00.754609+0000","last_clean":"2023-06-22T08:00:00.754609+0000","last_became_active":"2023-06-22T07:59:55.162869+0000","last_became_peered":"2023-06-22T07:59:55.162869+0000","last_unstale":"2023-06-22T08:00:00.754609+0000","last_undegraded":"2023-06-22T08:00:00.754609+0000","last_fullsized":"2023-06-22T08:00:00.754609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:37:59.063186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754586+0000","last_change":"2023-06-22T07:59:59.200516+0000","last_active":"2023-06-22T08:00:00.754586+0000","last_peered":"2023-06-22T08:00:00.754586+0000","last_clean":"2023-06-22T08:00:00.754586+0000","last_became_active":"2023-06-22T07:59:59.200096+0000","last_became_peered":"2023-06-22T07:59:59.200096+0000","last_unstale":"2023-06-22T08:00:00.754586+0000","last_undegraded":"2023-06-22T08:00:00.754586+0000","last_fullsized":"2023-06-22T08:00:00.754586+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:06.099105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2,7],"acting":[6,3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.490825+0000","last_change":"2023-06-22T07:59:54.645950+0000","last_active":"2023-06-22T08:00:36.490825+0000","last_peered":"2023-06-22T08:00:36.490825+0000","last_clean":"2023-06-22T08:00:36.490825+0000","last_became_active":"2023-06-22T07:59:54.645520+0000","last_became_peered":"2023-06-22T07:59:54.645520+0000","last_unstale":"2023-06-22T08:00:36.490825+0000","last_undegraded":"2023-06-22T08:00:36.490825+0000","last_fullsized":"2023-06-22T08:00:36.490825+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:17:34.816556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871992+0000","last_change":"2023-06-22T07:59:55.162767+0000","last_active":"2023-06-22T07:59:59.871992+0000","last_peered":"2023-06-22T07:59:59.871992+0000","last_clean":"2023-06-22T07:59:59.871992+0000","last_became_active":"2023-06-22T07:59:55.162451+0000","last_became_peered":"2023-06-22T07:59:55.162451+0000","last_unstale":"2023-06-22T07:59:59.871992+0000","last_undegraded":"2023-06-22T07:59:59.871992+0000","last_fullsized":"2023-06-22T07:59:59.871992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:38.977413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871071+0000","last_change":"2023-06-22T07:59:59.208215+0000","last_active":"2023-06-22T07:59:59.871071+0000","last_peered":"2023-06-22T07:59:59.871071+0000","last_clean":"2023-06-22T07:59:59.871071+0000","last_became_active":"2023-06-22T07:59:59.207608+0000","last_became_peered":"2023-06-22T07:59:59.207608+0000","last_unstale":"2023-06-22T07:59:59.871071+0000","last_undegraded":"2023-06-22T07:59:59.871071+0000","last_fullsized":"2023-06-22T07:59:59.871071+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:39:48.527076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,5],"acting":[3,2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.699308+0000","last_change":"2023-06-22T07:59:54.652515+0000","last_active":"2023-06-22T08:00:25.699308+0000","last_peered":"2023-06-22T08:00:25.699308+0000","last_clean":"2023-06-22T08:00:25.699308+0000","last_became_active":"2023-06-22T07:59:54.652249+0000","last_became_peered":"2023-06-22T07:59:54.652249+0000","last_unstale":"2023-06-22T08:00:25.699308+0000","last_undegraded":"2023-06-22T08:00:25.699308+0000","last_fullsized":"2023-06-22T08:00:25.699308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:22:12.506596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755250+0000","last_change":"2023-06-22T07:59:55.158359+0000","last_active":"2023-06-22T08:00:00.755250+0000","last_peered":"2023-06-22T08:00:00.755250+0000","last_clean":"2023-06-22T08:00:00.755250+0000","last_became_active":"2023-06-22T07:59:55.157918+0000","last_became_peered":"2023-06-22T07:59:55.157918+0000","last_unstale":"2023-06-22T08:00:00.755250+0000","last_undegraded":"2023-06-22T08:00:00.755250+0000","last_fullsized":"2023-06-22T08:00:00.755250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708305+0000","last_change":"2023-06-22T07:59:59.201033+0000","last_active":"2023-06-22T07:59:59.708305+0000","last_peered":"2023-06-22T07:59:59.708305+0000","last_clean":"2023-06-22T07:59:59.708305+0000","last_became_active":"2023-06-22T07:59:59.200500+0000","last_became_peered":"2023-06-22T07:59:59.200500+0000","last_unstale":"2023-06-22T07:59:59.708305+0000","last_undegraded":"2023-06-22T07:59:59.708305+0000","last_fullsized":"2023-06-22T07:59:59.708305+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:18.174604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,4],"acting":[2,1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.571653+0000","last_change":"2023-06-22T07:59:54.654416+0000","last_active":"2023-06-22T08:00:41.571653+0000","last_peered":"2023-06-22T08:00:41.571653+0000","last_clean":"2023-06-22T08:00:41.571653+0000","last_became_active":"2023-06-22T07:59:54.654139+0000","last_became_peered":"2023-06-22T07:59:54.654139+0000","last_unstale":"2023-06-22T08:00:41.571653+0000","last_undegraded":"2023-06-22T08:00:41.571653+0000","last_fullsized":"2023-06-22T08:00:41.571653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755506+0000","last_change":"2023-06-22T07:59:55.159790+0000","last_active":"2023-06-22T08:00:00.755506+0000","last_peered":"2023-06-22T08:00:00.755506+0000","last_clean":"2023-06-22T08:00:00.755506+0000","last_became_active":"2023-06-22T07:59:55.159489+0000","last_became_peered":"2023-06-22T07:59:55.159489+0000","last_unstale":"2023-06-22T08:00:00.755506+0000","last_undegraded":"2023-06-22T08:00:00.755506+0000","last_fullsized":"2023-06-22T08:00:00.755506+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:15:18.373418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753814+0000","last_change":"2023-06-22T07:59:59.197822+0000","last_active":"2023-06-22T08:00:00.753814+0000","last_peered":"2023-06-22T08:00:00.753814+0000","last_clean":"2023-06-22T08:00:00.753814+0000","last_became_active":"2023-06-22T07:59:59.197431+0000","last_became_peered":"2023-06-22T07:59:59.197431+0000","last_unstale":"2023-06-22T08:00:00.753814+0000","last_undegraded":"2023-06-22T08:00:00.753814+0000","last_fullsized":"2023-06-22T08:00:00.753814+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:48:41.139500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7,1],"acting":[4,6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.698982+0000","last_change":"2023-06-22T07:59:54.644158+0000","last_active":"2023-06-22T08:00:25.698982+0000","last_peered":"2023-06-22T08:00:25.698982+0000","last_clean":"2023-06-22T08:00:25.698982+0000","last_became_active":"2023-06-22T07:59:54.643808+0000","last_became_peered":"2023-06-22T07:59:54.643808+0000","last_unstale":"2023-06-22T08:00:25.698982+0000","last_undegraded":"2023-06-22T08:00:25.698982+0000","last_fullsized":"2023-06-22T08:00:25.698982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:45.438691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754507+0000","last_change":"2023-06-22T07:59:55.157538+0000","last_active":"2023-06-22T08:00:00.754507+0000","last_peered":"2023-06-22T08:00:00.754507+0000","last_clean":"2023-06-22T08:00:00.754507+0000","last_became_active":"2023-06-22T07:59:55.156967+0000","last_became_peered":"2023-06-22T07:59:55.156967+0000","last_unstale":"2023-06-22T08:00:00.754507+0000","last_undegraded":"2023-06-22T08:00:00.754507+0000","last_fullsized":"2023-06-22T08:00:00.754507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:38:44.161245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753821+0000","last_change":"2023-06-22T07:59:59.201671+0000","last_active":"2023-06-22T08:00:00.753821+0000","last_peered":"2023-06-22T08:00:00.753821+0000","last_clean":"2023-06-22T08:00:00.753821+0000","last_became_active":"2023-06-22T07:59:59.201201+0000","last_became_peered":"2023-06-22T07:59:59.201201+0000","last_unstale":"2023-06-22T08:00:00.753821+0000","last_undegraded":"2023-06-22T08:00:00.753821+0000","last_fullsized":"2023-06-22T08:00:00.753821+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4,3],"acting":[6,1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"28'16","reported_seq":37,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:50.546385+0000","last_change":"2023-06-22T07:59:54.640470+0000","last_active":"2023-06-22T08:00:50.546385+0000","last_peered":"2023-06-22T08:00:50.546385+0000","last_clean":"2023-06-22T08:00:50.546385+0000","last_became_active":"2023-06-22T07:59:54.640124+0000","last_became_peered":"2023-06-22T07:59:54.640124+0000","last_unstale":"2023-06-22T08:00:50.546385+0000","last_undegraded":"2023-06-22T08:00:50.546385+0000","last_fullsized":"2023-06-22T08:00:50.546385+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:06:41.502702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4052,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708599+0000","last_change":"2023-06-22T07:59:55.165550+0000","last_active":"2023-06-22T07:59:59.708599+0000","last_peered":"2023-06-22T07:59:59.708599+0000","last_clean":"2023-06-22T07:59:59.708599+0000","last_became_active":"2023-06-22T07:59:55.164998+0000","last_became_peered":"2023-06-22T07:59:55.164998+0000","last_unstale":"2023-06-22T07:59:59.708599+0000","last_undegraded":"2023-06-22T07:59:59.708599+0000","last_fullsized":"2023-06-22T07:59:59.708599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872133+0000","last_change":"2023-06-22T07:59:59.207103+0000","last_active":"2023-06-22T07:59:59.872133+0000","last_peered":"2023-06-22T07:59:59.872133+0000","last_clean":"2023-06-22T07:59:59.872133+0000","last_became_active":"2023-06-22T07:59:59.206531+0000","last_became_peered":"2023-06-22T07:59:59.206531+0000","last_unstale":"2023-06-22T07:59:59.872133+0000","last_undegraded":"2023-06-22T07:59:59.872133+0000","last_fullsized":"2023-06-22T07:59:59.872133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:42:48.571228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0,6],"acting":[3,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"28'9","reported_seq":30,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.570482+0000","last_change":"2023-06-22T07:59:54.644301+0000","last_active":"2023-06-22T08:00:41.570482+0000","last_peered":"2023-06-22T08:00:41.570482+0000","last_clean":"2023-06-22T08:00:41.570482+0000","last_became_active":"2023-06-22T07:59:54.643939+0000","last_became_peered":"2023-06-22T07:59:54.643939+0000","last_unstale":"2023-06-22T08:00:41.570482+0000","last_undegraded":"2023-06-22T08:00:41.570482+0000","last_fullsized":"2023-06-22T08:00:41.570482+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:07:42.118767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753877+0000","last_change":"2023-06-22T07:59:55.163311+0000","last_active":"2023-06-22T08:00:00.753877+0000","last_peered":"2023-06-22T08:00:00.753877+0000","last_clean":"2023-06-22T08:00:00.753877+0000","last_became_active":"2023-06-22T07:59:55.162991+0000","last_became_peered":"2023-06-22T07:59:55.162991+0000","last_unstale":"2023-06-22T08:00:00.753877+0000","last_undegraded":"2023-06-22T08:00:00.753877+0000","last_fullsized":"2023-06-22T08:00:00.753877+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:35.971030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754227+0000","last_change":"2023-06-22T07:59:59.201324+0000","last_active":"2023-06-22T08:00:00.754227+0000","last_peered":"2023-06-22T08:00:00.754227+0000","last_clean":"2023-06-22T08:00:00.754227+0000","last_became_active":"2023-06-22T07:59:59.200877+0000","last_became_peered":"2023-06-22T07:59:59.200877+0000","last_unstale":"2023-06-22T08:00:00.754227+0000","last_undegraded":"2023-06-22T08:00:00.754227+0000","last_fullsized":"2023-06-22T08:00:00.754227+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:14:17.094426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5,7],"acting":[4,2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493994+0000","last_change":"2023-06-22T07:59:54.651527+0000","last_active":"2023-06-22T08:00:36.493994+0000","last_peered":"2023-06-22T08:00:36.493994+0000","last_clean":"2023-06-22T08:00:36.493994+0000","last_became_active":"2023-06-22T07:59:54.651196+0000","last_became_peered":"2023-06-22T07:59:54.651196+0000","last_unstale":"2023-06-22T08:00:36.493994+0000","last_undegraded":"2023-06-22T08:00:36.493994+0000","last_fullsized":"2023-06-22T08:00:36.493994+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:47.493751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872142+0000","last_change":"2023-06-22T07:59:55.162610+0000","last_active":"2023-06-22T07:59:59.872142+0000","last_peered":"2023-06-22T07:59:59.872142+0000","last_clean":"2023-06-22T07:59:59.872142+0000","last_became_active":"2023-06-22T07:59:55.162334+0000","last_became_peered":"2023-06-22T07:59:55.162334+0000","last_unstale":"2023-06-22T07:59:59.872142+0000","last_undegraded":"2023-06-22T07:59:59.872142+0000","last_fullsized":"2023-06-22T07:59:59.872142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:42.490565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755371+0000","last_change":"2023-06-22T07:59:59.199402+0000","last_active":"2023-06-22T08:00:00.755371+0000","last_peered":"2023-06-22T08:00:00.755371+0000","last_clean":"2023-06-22T08:00:00.755371+0000","last_became_active":"2023-06-22T07:59:59.198979+0000","last_became_peered":"2023-06-22T07:59:59.198979+0000","last_unstale":"2023-06-22T08:00:00.755371+0000","last_undegraded":"2023-06-22T08:00:00.755371+0000","last_fullsized":"2023-06-22T08:00:00.755371+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:17:07.122732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1,6],"acting":[4,5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493688+0000","last_change":"2023-06-22T07:59:54.649236+0000","last_active":"2023-06-22T08:00:36.493688+0000","last_peered":"2023-06-22T08:00:36.493688+0000","last_clean":"2023-06-22T08:00:36.493688+0000","last_became_active":"2023-06-22T07:59:54.648852+0000","last_became_peered":"2023-06-22T07:59:54.648852+0000","last_unstale":"2023-06-22T08:00:36.493688+0000","last_undegraded":"2023-06-22T08:00:36.493688+0000","last_fullsized":"2023-06-22T08:00:36.493688+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:16:59.865638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709134+0000","last_change":"2023-06-22T07:59:55.165859+0000","last_active":"2023-06-22T07:59:59.709134+0000","last_peered":"2023-06-22T07:59:59.709134+0000","last_clean":"2023-06-22T07:59:59.709134+0000","last_became_active":"2023-06-22T07:59:55.165542+0000","last_became_peered":"2023-06-22T07:59:55.165542+0000","last_unstale":"2023-06-22T07:59:59.709134+0000","last_undegraded":"2023-06-22T07:59:59.709134+0000","last_fullsized":"2023-06-22T07:59:59.709134+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:32.147969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871894+0000","last_change":"2023-06-22T07:59:59.207794+0000","last_active":"2023-06-22T07:59:59.871894+0000","last_peered":"2023-06-22T07:59:59.871894+0000","last_clean":"2023-06-22T07:59:59.871894+0000","last_became_active":"2023-06-22T07:59:59.207364+0000","last_became_peered":"2023-06-22T07:59:59.207364+0000","last_unstale":"2023-06-22T07:59:59.871894+0000","last_undegraded":"2023-06-22T07:59:59.871894+0000","last_fullsized":"2023-06-22T07:59:59.871894+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:23:30.718491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4,0],"acting":[3,6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"28'17","reported_seq":38,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.491476+0000","last_change":"2023-06-22T07:59:54.651423+0000","last_active":"2023-06-22T08:00:36.491476+0000","last_peered":"2023-06-22T08:00:36.491476+0000","last_clean":"2023-06-22T08:00:36.491476+0000","last_became_active":"2023-06-22T07:59:54.651167+0000","last_became_peered":"2023-06-22T07:59:54.651167+0000","last_unstale":"2023-06-22T08:00:36.491476+0000","last_undegraded":"2023-06-22T08:00:36.491476+0000","last_fullsized":"2023-06-22T08:00:36.491476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:10:31.141977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709649+0000","last_change":"2023-06-22T07:59:55.164478+0000","last_active":"2023-06-22T07:59:59.709649+0000","last_peered":"2023-06-22T07:59:59.709649+0000","last_clean":"2023-06-22T07:59:59.709649+0000","last_became_active":"2023-06-22T07:59:55.163922+0000","last_became_peered":"2023-06-22T07:59:55.163922+0000","last_unstale":"2023-06-22T07:59:59.709649+0000","last_undegraded":"2023-06-22T07:59:59.709649+0000","last_fullsized":"2023-06-22T07:59:59.709649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:43.772399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155085+0000","last_change":"2023-06-22T07:59:59.198496+0000","last_active":"2023-06-22T08:00:00.155085+0000","last_peered":"2023-06-22T08:00:00.155085+0000","last_clean":"2023-06-22T08:00:00.155085+0000","last_became_active":"2023-06-22T07:59:59.198067+0000","last_became_peered":"2023-06-22T07:59:59.198067+0000","last_unstale":"2023-06-22T08:00:00.155085+0000","last_undegraded":"2023-06-22T08:00:00.155085+0000","last_fullsized":"2023-06-22T08:00:00.155085+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:24.865757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,7],"acting":[1,0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158445+0000","last_change":"2023-06-22T07:59:54.651366+0000","last_active":"2023-06-22T08:00:00.158445+0000","last_peered":"2023-06-22T08:00:00.158445+0000","last_clean":"2023-06-22T08:00:00.158445+0000","last_became_active":"2023-06-22T07:59:54.651059+0000","last_became_peered":"2023-06-22T07:59:54.651059+0000","last_unstale":"2023-06-22T08:00:00.158445+0000","last_undegraded":"2023-06-22T08:00:00.158445+0000","last_fullsized":"2023-06-22T08:00:00.158445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:37.183143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871131+0000","last_change":"2023-06-22T07:59:55.162820+0000","last_active":"2023-06-22T07:59:59.871131+0000","last_peered":"2023-06-22T07:59:59.871131+0000","last_clean":"2023-06-22T07:59:59.871131+0000","last_became_active":"2023-06-22T07:59:55.162505+0000","last_became_peered":"2023-06-22T07:59:55.162505+0000","last_unstale":"2023-06-22T07:59:59.871131+0000","last_undegraded":"2023-06-22T07:59:59.871131+0000","last_fullsized":"2023-06-22T07:59:59.871131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:46:38.312778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753744+0000","last_change":"2023-06-22T07:59:59.201059+0000","last_active":"2023-06-22T08:00:00.753744+0000","last_peered":"2023-06-22T08:00:00.753744+0000","last_clean":"2023-06-22T08:00:00.753744+0000","last_became_active":"2023-06-22T07:59:59.200460+0000","last_became_peered":"2023-06-22T07:59:59.200460+0000","last_unstale":"2023-06-22T08:00:00.753744+0000","last_undegraded":"2023-06-22T08:00:00.753744+0000","last_fullsized":"2023-06-22T08:00:00.753744+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,0],"acting":[6,4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.565345+0000","last_change":"2023-06-22T07:59:54.646017+0000","last_active":"2023-06-22T08:00:41.565345+0000","last_peered":"2023-06-22T08:00:41.565345+0000","last_clean":"2023-06-22T08:00:41.565345+0000","last_became_active":"2023-06-22T07:59:54.645554+0000","last_became_peered":"2023-06-22T07:59:54.645554+0000","last_unstale":"2023-06-22T08:00:41.565345+0000","last_undegraded":"2023-06-22T08:00:41.565345+0000","last_fullsized":"2023-06-22T08:00:41.565345+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:39:01.550269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154806+0000","last_change":"2023-06-22T07:59:55.166283+0000","last_active":"2023-06-22T08:00:00.154806+0000","last_peered":"2023-06-22T08:00:00.154806+0000","last_clean":"2023-06-22T08:00:00.154806+0000","last_became_active":"2023-06-22T07:59:55.165977+0000","last_became_peered":"2023-06-22T07:59:55.165977+0000","last_unstale":"2023-06-22T08:00:00.154806+0000","last_undegraded":"2023-06-22T08:00:00.154806+0000","last_fullsized":"2023-06-22T08:00:00.154806+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:21:29.484640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709141+0000","last_change":"2023-06-22T07:59:59.205411+0000","last_active":"2023-06-22T07:59:59.709141+0000","last_peered":"2023-06-22T07:59:59.709141+0000","last_clean":"2023-06-22T07:59:59.709141+0000","last_became_active":"2023-06-22T07:59:59.204980+0000","last_became_peered":"2023-06-22T07:59:59.204980+0000","last_unstale":"2023-06-22T07:59:59.709141+0000","last_undegraded":"2023-06-22T07:59:59.709141+0000","last_fullsized":"2023-06-22T07:59:59.709141+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:42:22.317952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493022+0000","last_change":"2023-06-22T07:59:54.654491+0000","last_active":"2023-06-22T08:00:36.493022+0000","last_peered":"2023-06-22T08:00:36.493022+0000","last_clean":"2023-06-22T08:00:36.493022+0000","last_became_active":"2023-06-22T07:59:54.654197+0000","last_became_peered":"2023-06-22T07:59:54.654197+0000","last_unstale":"2023-06-22T08:00:36.493022+0000","last_undegraded":"2023-06-22T08:00:36.493022+0000","last_fullsized":"2023-06-22T08:00:36.493022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:42:59.725455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708444+0000","last_change":"2023-06-22T07:59:55.172053+0000","last_active":"2023-06-22T07:59:59.708444+0000","last_peered":"2023-06-22T07:59:59.708444+0000","last_clean":"2023-06-22T07:59:59.708444+0000","last_became_active":"2023-06-22T07:59:55.165501+0000","last_became_peered":"2023-06-22T07:59:55.165501+0000","last_unstale":"2023-06-22T07:59:59.708444+0000","last_undegraded":"2023-06-22T07:59:59.708444+0000","last_fullsized":"2023-06-22T07:59:59.708444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753990+0000","last_change":"2023-06-22T07:59:59.199179+0000","last_active":"2023-06-22T08:00:00.753990+0000","last_peered":"2023-06-22T08:00:00.753990+0000","last_clean":"2023-06-22T08:00:00.753990+0000","last_became_active":"2023-06-22T07:59:59.198740+0000","last_became_peered":"2023-06-22T07:59:59.198740+0000","last_unstale":"2023-06-22T08:00:00.753990+0000","last_undegraded":"2023-06-22T08:00:00.753990+0000","last_fullsized":"2023-06-22T08:00:00.753990+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4,7],"acting":[6,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.491850+0000","last_change":"2023-06-22T07:59:54.644520+0000","last_active":"2023-06-22T08:00:36.491850+0000","last_peered":"2023-06-22T08:00:36.491850+0000","last_clean":"2023-06-22T08:00:36.491850+0000","last_became_active":"2023-06-22T07:59:54.644204+0000","last_became_peered":"2023-06-22T07:59:54.644204+0000","last_unstale":"2023-06-22T08:00:36.491850+0000","last_undegraded":"2023-06-22T08:00:36.491850+0000","last_fullsized":"2023-06-22T08:00:36.491850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:07.674431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709657+0000","last_change":"2023-06-22T07:59:55.164111+0000","last_active":"2023-06-22T07:59:59.709657+0000","last_peered":"2023-06-22T07:59:59.709657+0000","last_clean":"2023-06-22T07:59:59.709657+0000","last_became_active":"2023-06-22T07:59:55.163802+0000","last_became_peered":"2023-06-22T07:59:55.163802+0000","last_unstale":"2023-06-22T07:59:59.709657+0000","last_undegraded":"2023-06-22T07:59:59.709657+0000","last_fullsized":"2023-06-22T07:59:59.709657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755252+0000","last_change":"2023-06-22T07:59:59.197721+0000","last_active":"2023-06-22T08:00:00.755252+0000","last_peered":"2023-06-22T08:00:00.755252+0000","last_clean":"2023-06-22T08:00:00.755252+0000","last_became_active":"2023-06-22T07:59:59.197259+0000","last_became_peered":"2023-06-22T07:59:59.197259+0000","last_unstale":"2023-06-22T08:00:00.755252+0000","last_undegraded":"2023-06-22T08:00:00.755252+0000","last_fullsized":"2023-06-22T08:00:00.755252+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:20.459194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5,1],"acting":[7,0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.697877+0000","last_change":"2023-06-22T07:59:54.643124+0000","last_active":"2023-06-22T08:00:25.697877+0000","last_peered":"2023-06-22T08:00:25.697877+0000","last_clean":"2023-06-22T08:00:25.697877+0000","last_became_active":"2023-06-22T07:59:54.642833+0000","last_became_peered":"2023-06-22T07:59:54.642833+0000","last_unstale":"2023-06-22T08:00:25.697877+0000","last_undegraded":"2023-06-22T08:00:25.697877+0000","last_fullsized":"2023-06-22T08:00:25.697877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.158064+0000","last_change":"2023-06-22T07:59:55.160551+0000","last_active":"2023-06-22T08:00:00.158064+0000","last_peered":"2023-06-22T08:00:00.158064+0000","last_clean":"2023-06-22T08:00:00.158064+0000","last_became_active":"2023-06-22T07:59:55.160221+0000","last_became_peered":"2023-06-22T07:59:55.160221+0000","last_unstale":"2023-06-22T08:00:00.158064+0000","last_undegraded":"2023-06-22T08:00:00.158064+0000","last_fullsized":"2023-06-22T08:00:00.158064+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708974+0000","last_change":"2023-06-22T07:59:59.204643+0000","last_active":"2023-06-22T07:59:59.708974+0000","last_peered":"2023-06-22T07:59:59.708974+0000","last_clean":"2023-06-22T07:59:59.708974+0000","last_became_active":"2023-06-22T07:59:59.204182+0000","last_became_peered":"2023-06-22T07:59:59.204182+0000","last_unstale":"2023-06-22T07:59:59.708974+0000","last_undegraded":"2023-06-22T07:59:59.708974+0000","last_fullsized":"2023-06-22T07:59:59.708974+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4,2],"acting":[5,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"28'3","reported_seq":24,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493681+0000","last_change":"2023-06-22T07:59:54.654036+0000","last_active":"2023-06-22T08:00:36.493681+0000","last_peered":"2023-06-22T08:00:36.493681+0000","last_clean":"2023-06-22T08:00:36.493681+0000","last_became_active":"2023-06-22T07:59:54.653736+0000","last_became_peered":"2023-06-22T07:59:54.653736+0000","last_unstale":"2023-06-22T08:00:36.493681+0000","last_undegraded":"2023-06-22T08:00:36.493681+0000","last_fullsized":"2023-06-22T08:00:36.493681+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:44:09.331748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871999+0000","last_change":"2023-06-22T07:59:55.163181+0000","last_active":"2023-06-22T07:59:59.871999+0000","last_peered":"2023-06-22T07:59:59.871999+0000","last_clean":"2023-06-22T07:59:59.871999+0000","last_became_active":"2023-06-22T07:59:55.162860+0000","last_became_peered":"2023-06-22T07:59:55.162860+0000","last_unstale":"2023-06-22T07:59:59.871999+0000","last_undegraded":"2023-06-22T07:59:59.871999+0000","last_fullsized":"2023-06-22T07:59:59.871999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:59:25.037938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155510+0000","last_change":"2023-06-22T07:59:59.195710+0000","last_active":"2023-06-22T08:00:00.155510+0000","last_peered":"2023-06-22T08:00:00.155510+0000","last_clean":"2023-06-22T08:00:00.155510+0000","last_became_active":"2023-06-22T07:59:59.195246+0000","last_became_peered":"2023-06-22T07:59:59.195246+0000","last_unstale":"2023-06-22T08:00:00.155510+0000","last_undegraded":"2023-06-22T08:00:00.155510+0000","last_fullsized":"2023-06-22T08:00:00.155510+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:36.457493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,5],"acting":[1,0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.570855+0000","last_change":"2023-06-22T07:59:54.649692+0000","last_active":"2023-06-22T08:00:41.570855+0000","last_peered":"2023-06-22T08:00:41.570855+0000","last_clean":"2023-06-22T08:00:41.570855+0000","last_became_active":"2023-06-22T07:59:54.649416+0000","last_became_peered":"2023-06-22T07:59:54.649416+0000","last_unstale":"2023-06-22T08:00:41.570855+0000","last_undegraded":"2023-06-22T08:00:41.570855+0000","last_fullsized":"2023-06-22T08:00:41.570855+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:54:02.923433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708909+0000","last_change":"2023-06-22T07:59:55.175392+0000","last_active":"2023-06-22T07:59:59.708909+0000","last_peered":"2023-06-22T07:59:59.708909+0000","last_clean":"2023-06-22T07:59:59.708909+0000","last_became_active":"2023-06-22T07:59:55.175112+0000","last_became_peered":"2023-06-22T07:59:55.175112+0000","last_unstale":"2023-06-22T07:59:59.708909+0000","last_undegraded":"2023-06-22T07:59:59.708909+0000","last_fullsized":"2023-06-22T07:59:59.708909+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:01:35.069734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871873+0000","last_change":"2023-06-22T07:59:59.206671+0000","last_active":"2023-06-22T07:59:59.871873+0000","last_peered":"2023-06-22T07:59:59.871873+0000","last_clean":"2023-06-22T07:59:59.871873+0000","last_became_active":"2023-06-22T07:59:59.206179+0000","last_became_peered":"2023-06-22T07:59:59.206179+0000","last_unstale":"2023-06-22T07:59:59.871873+0000","last_undegraded":"2023-06-22T07:59:59.871873+0000","last_fullsized":"2023-06-22T07:59:59.871873+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:05:00.427518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4,5],"acting":[3,2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.698074+0000","last_change":"2023-06-22T07:59:54.640508+0000","last_active":"2023-06-22T08:00:25.698074+0000","last_peered":"2023-06-22T08:00:25.698074+0000","last_clean":"2023-06-22T08:00:25.698074+0000","last_became_active":"2023-06-22T07:59:54.640108+0000","last_became_peered":"2023-06-22T07:59:54.640108+0000","last_unstale":"2023-06-22T08:00:25.698074+0000","last_undegraded":"2023-06-22T08:00:25.698074+0000","last_fullsized":"2023-06-22T08:00:25.698074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:11:19.193231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754822+0000","last_change":"2023-06-22T07:59:55.164398+0000","last_active":"2023-06-22T08:00:00.754822+0000","last_peered":"2023-06-22T08:00:00.754822+0000","last_clean":"2023-06-22T08:00:00.754822+0000","last_became_active":"2023-06-22T07:59:55.164086+0000","last_became_peered":"2023-06-22T07:59:55.164086+0000","last_unstale":"2023-06-22T08:00:00.754822+0000","last_undegraded":"2023-06-22T08:00:00.754822+0000","last_fullsized":"2023-06-22T08:00:00.754822+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:38.189135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753997+0000","last_change":"2023-06-22T07:59:59.207752+0000","last_active":"2023-06-22T08:00:00.753997+0000","last_peered":"2023-06-22T08:00:00.753997+0000","last_clean":"2023-06-22T08:00:00.753997+0000","last_became_active":"2023-06-22T07:59:59.207360+0000","last_became_peered":"2023-06-22T07:59:59.207360+0000","last_unstale":"2023-06-22T08:00:00.753997+0000","last_undegraded":"2023-06-22T08:00:00.753997+0000","last_fullsized":"2023-06-22T08:00:00.753997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:08:25.633009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3,1],"acting":[4,6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709114+0000","last_change":"2023-06-22T07:59:49.652772+0000","last_active":"2023-06-22T07:59:59.709114+0000","last_peered":"2023-06-22T07:59:59.709114+0000","last_clean":"2023-06-22T07:59:59.709114+0000","last_became_active":"2023-06-22T07:59:49.652487+0000","last_became_peered":"2023-06-22T07:59:49.652487+0000","last_unstale":"2023-06-22T07:59:59.709114+0000","last_undegraded":"2023-06-22T07:59:59.709114+0000","last_fullsized":"2023-06-22T07:59:59.709114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:05:11.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493495+0000","last_change":"2023-06-22T07:59:54.644538+0000","last_active":"2023-06-22T08:00:36.493495+0000","last_peered":"2023-06-22T08:00:36.493495+0000","last_clean":"2023-06-22T08:00:36.493495+0000","last_became_active":"2023-06-22T07:59:54.644253+0000","last_became_peered":"2023-06-22T07:59:54.644253+0000","last_unstale":"2023-06-22T08:00:36.493495+0000","last_undegraded":"2023-06-22T08:00:36.493495+0000","last_fullsized":"2023-06-22T08:00:36.493495+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:09.128751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754228+0000","last_change":"2023-06-22T07:59:55.163802+0000","last_active":"2023-06-22T08:00:00.754228+0000","last_peered":"2023-06-22T08:00:00.754228+0000","last_clean":"2023-06-22T08:00:00.754228+0000","last_became_active":"2023-06-22T07:59:55.163481+0000","last_became_peered":"2023-06-22T07:59:55.163481+0000","last_unstale":"2023-06-22T08:00:00.754228+0000","last_undegraded":"2023-06-22T08:00:00.754228+0000","last_fullsized":"2023-06-22T08:00:00.754228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:16.916350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755246+0000","last_change":"2023-06-22T07:59:59.207344+0000","last_active":"2023-06-22T08:00:00.755246+0000","last_peered":"2023-06-22T08:00:00.755246+0000","last_clean":"2023-06-22T08:00:00.755246+0000","last_became_active":"2023-06-22T07:59:59.206975+0000","last_became_peered":"2023-06-22T07:59:59.206975+0000","last_unstale":"2023-06-22T08:00:00.755246+0000","last_undegraded":"2023-06-22T08:00:00.755246+0000","last_fullsized":"2023-06-22T08:00:00.755246+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:55.965255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3,1],"acting":[7,4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871513+0000","last_change":"2023-06-22T07:59:49.653053+0000","last_active":"2023-06-22T07:59:59.871513+0000","last_peered":"2023-06-22T07:59:59.871513+0000","last_clean":"2023-06-22T07:59:59.871513+0000","last_became_active":"2023-06-22T07:59:49.652861+0000","last_became_peered":"2023-06-22T07:59:49.652861+0000","last_unstale":"2023-06-22T07:59:59.871513+0000","last_undegraded":"2023-06-22T07:59:59.871513+0000","last_fullsized":"2023-06-22T07:59:59.871513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:28:40.388367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"28'6","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.493566+0000","last_change":"2023-06-22T07:59:54.645403+0000","last_active":"2023-06-22T08:00:36.493566+0000","last_peered":"2023-06-22T08:00:36.493566+0000","last_clean":"2023-06-22T08:00:36.493566+0000","last_became_active":"2023-06-22T07:59:54.644973+0000","last_became_peered":"2023-06-22T07:59:54.644973+0000","last_unstale":"2023-06-22T08:00:36.493566+0000","last_undegraded":"2023-06-22T08:00:36.493566+0000","last_fullsized":"2023-06-22T08:00:36.493566+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:55:15.910358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754101+0000","last_change":"2023-06-22T07:59:55.161484+0000","last_active":"2023-06-22T08:00:00.754101+0000","last_peered":"2023-06-22T08:00:00.754101+0000","last_clean":"2023-06-22T08:00:00.754101+0000","last_became_active":"2023-06-22T07:59:55.161194+0000","last_became_peered":"2023-06-22T07:59:55.161194+0000","last_unstale":"2023-06-22T08:00:00.754101+0000","last_undegraded":"2023-06-22T08:00:00.754101+0000","last_fullsized":"2023-06-22T08:00:00.754101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:53:08.680423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754988+0000","last_change":"2023-06-22T07:59:59.198241+0000","last_active":"2023-06-22T08:00:00.754988+0000","last_peered":"2023-06-22T08:00:00.754988+0000","last_clean":"2023-06-22T08:00:00.754988+0000","last_became_active":"2023-06-22T07:59:59.197689+0000","last_became_peered":"2023-06-22T07:59:59.197689+0000","last_unstale":"2023-06-22T08:00:00.754988+0000","last_undegraded":"2023-06-22T08:00:00.754988+0000","last_fullsized":"2023-06-22T08:00:00.754988+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:11:39.513874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4,1],"acting":[7,0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753886+0000","last_change":"2023-06-22T07:59:49.450044+0000","last_active":"2023-06-22T08:00:00.753886+0000","last_peered":"2023-06-22T08:00:00.753886+0000","last_clean":"2023-06-22T08:00:00.753886+0000","last_became_active":"2023-06-22T07:59:49.449731+0000","last_became_peered":"2023-06-22T07:59:49.449731+0000","last_unstale":"2023-06-22T08:00:00.753886+0000","last_undegraded":"2023-06-22T08:00:00.753886+0000","last_fullsized":"2023-06-22T08:00:00.753886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:17:40.262337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"28'4","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:31.395849+0000","last_change":"2023-06-22T07:59:54.640462+0000","last_active":"2023-06-22T08:00:31.395849+0000","last_peered":"2023-06-22T08:00:31.395849+0000","last_clean":"2023-06-22T08:00:31.395849+0000","last_became_active":"2023-06-22T07:59:54.640123+0000","last_became_peered":"2023-06-22T07:59:54.640123+0000","last_unstale":"2023-06-22T08:00:31.395849+0000","last_undegraded":"2023-06-22T08:00:31.395849+0000","last_fullsized":"2023-06-22T08:00:31.395849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:39:20.316921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755350+0000","last_change":"2023-06-22T07:59:55.164651+0000","last_active":"2023-06-22T08:00:00.755350+0000","last_peered":"2023-06-22T08:00:00.755350+0000","last_clean":"2023-06-22T08:00:00.755350+0000","last_became_active":"2023-06-22T07:59:55.164235+0000","last_became_peered":"2023-06-22T07:59:55.164235+0000","last_unstale":"2023-06-22T08:00:00.755350+0000","last_undegraded":"2023-06-22T08:00:00.755350+0000","last_fullsized":"2023-06-22T08:00:00.755350+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:11:32.924136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157741+0000","last_change":"2023-06-22T07:59:59.204779+0000","last_active":"2023-06-22T08:00:00.157741+0000","last_peered":"2023-06-22T08:00:00.157741+0000","last_clean":"2023-06-22T08:00:00.157741+0000","last_became_active":"2023-06-22T07:59:59.204288+0000","last_became_peered":"2023-06-22T07:59:59.204288+0000","last_unstale":"2023-06-22T08:00:00.157741+0000","last_undegraded":"2023-06-22T08:00:00.157741+0000","last_fullsized":"2023-06-22T08:00:00.157741+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:49:13.677587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871497+0000","last_change":"2023-06-22T07:59:49.657008+0000","last_active":"2023-06-22T07:59:59.871497+0000","last_peered":"2023-06-22T07:59:59.871497+0000","last_clean":"2023-06-22T07:59:59.871497+0000","last_became_active":"2023-06-22T07:59:49.656752+0000","last_became_peered":"2023-06-22T07:59:49.656752+0000","last_unstale":"2023-06-22T07:59:59.871497+0000","last_undegraded":"2023-06-22T07:59:59.871497+0000","last_fullsized":"2023-06-22T07:59:59.871497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:50:37.639134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":185,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155134+0000","last_change":"2023-06-22T07:59:42.501363+0000","last_active":"2023-06-22T08:00:00.155134+0000","last_peered":"2023-06-22T08:00:00.155134+0000","last_clean":"2023-06-22T08:00:00.155134+0000","last_became_active":"2023-06-22T07:59:42.500320+0000","last_became_peered":"2023-06-22T07:59:42.500320+0000","last_unstale":"2023-06-22T08:00:00.155134+0000","last_undegraded":"2023-06-22T08:00:00.155134+0000","last_fullsized":"2023-06-22T08:00:00.155134+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:41.381475+0000","last_clean_scrub_stamp":"2023-06-22T07:59:41.381475+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:08:53.784030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.569235+0000","last_change":"2023-06-22T07:59:54.645031+0000","last_active":"2023-06-22T08:00:41.569235+0000","last_peered":"2023-06-22T08:00:41.569235+0000","last_clean":"2023-06-22T08:00:41.569235+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:41.569235+0000","last_undegraded":"2023-06-22T08:00:41.569235+0000","last_fullsized":"2023-06-22T08:00:41.569235+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:56:42.766407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.872074+0000","last_change":"2023-06-22T07:59:55.162799+0000","last_active":"2023-06-22T07:59:59.872074+0000","last_peered":"2023-06-22T07:59:59.872074+0000","last_clean":"2023-06-22T07:59:59.872074+0000","last_became_active":"2023-06-22T07:59:55.162501+0000","last_became_peered":"2023-06-22T07:59:55.162501+0000","last_unstale":"2023-06-22T07:59:59.872074+0000","last_undegraded":"2023-06-22T07:59:59.872074+0000","last_fullsized":"2023-06-22T07:59:59.872074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:57:14.305897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871805+0000","last_change":"2023-06-22T07:59:59.205097+0000","last_active":"2023-06-22T07:59:59.871805+0000","last_peered":"2023-06-22T07:59:59.871805+0000","last_clean":"2023-06-22T07:59:59.871805+0000","last_became_active":"2023-06-22T07:59:59.201465+0000","last_became_peered":"2023-06-22T07:59:59.201465+0000","last_unstale":"2023-06-22T07:59:59.871805+0000","last_undegraded":"2023-06-22T07:59:59.871805+0000","last_fullsized":"2023-06-22T07:59:59.871805+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:17:45.352998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,0],"acting":[3,7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871056+0000","last_change":"2023-06-22T07:59:49.442558+0000","last_active":"2023-06-22T07:59:59.871056+0000","last_peered":"2023-06-22T07:59:59.871056+0000","last_clean":"2023-06-22T07:59:59.871056+0000","last_became_active":"2023-06-22T07:59:49.442154+0000","last_became_peered":"2023-06-22T07:59:49.442154+0000","last_unstale":"2023-06-22T07:59:59.871056+0000","last_undegraded":"2023-06-22T07:59:59.871056+0000","last_fullsized":"2023-06-22T07:59:59.871056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:46:16.233204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:31.399798+0000","last_change":"2023-06-22T07:59:54.641746+0000","last_active":"2023-06-22T08:00:31.399798+0000","last_peered":"2023-06-22T08:00:31.399798+0000","last_clean":"2023-06-22T08:00:31.399798+0000","last_became_active":"2023-06-22T07:59:54.641431+0000","last_became_peered":"2023-06-22T07:59:54.641431+0000","last_unstale":"2023-06-22T08:00:31.399798+0000","last_undegraded":"2023-06-22T08:00:31.399798+0000","last_fullsized":"2023-06-22T08:00:31.399798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:01.586815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709468+0000","last_change":"2023-06-22T07:59:55.163214+0000","last_active":"2023-06-22T07:59:59.709468+0000","last_peered":"2023-06-22T07:59:59.709468+0000","last_clean":"2023-06-22T07:59:59.709468+0000","last_became_active":"2023-06-22T07:59:55.162907+0000","last_became_peered":"2023-06-22T07:59:55.162907+0000","last_unstale":"2023-06-22T07:59:59.709468+0000","last_undegraded":"2023-06-22T07:59:59.709468+0000","last_fullsized":"2023-06-22T07:59:59.709468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:46.988703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157450+0000","last_change":"2023-06-22T07:59:59.198945+0000","last_active":"2023-06-22T08:00:00.157450+0000","last_peered":"2023-06-22T08:00:00.157450+0000","last_clean":"2023-06-22T08:00:00.157450+0000","last_became_active":"2023-06-22T07:59:59.198561+0000","last_became_peered":"2023-06-22T07:59:59.198561+0000","last_unstale":"2023-06-22T08:00:00.157450+0000","last_undegraded":"2023-06-22T08:00:00.157450+0000","last_fullsized":"2023-06-22T08:00:00.157450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:06.135651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2,5],"acting":[0,7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871842+0000","last_change":"2023-06-22T07:59:49.442246+0000","last_active":"2023-06-22T07:59:59.871842+0000","last_peered":"2023-06-22T07:59:59.871842+0000","last_clean":"2023-06-22T07:59:59.871842+0000","last_became_active":"2023-06-22T07:59:49.442001+0000","last_became_peered":"2023-06-22T07:59:49.442001+0000","last_unstale":"2023-06-22T07:59:59.871842+0000","last_undegraded":"2023-06-22T07:59:59.871842+0000","last_fullsized":"2023-06-22T07:59:59.871842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:27:24.457978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"28'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754302+0000","last_change":"2023-06-22T07:59:54.643272+0000","last_active":"2023-06-22T08:00:00.754302+0000","last_peered":"2023-06-22T08:00:00.754302+0000","last_clean":"2023-06-22T08:00:00.754302+0000","last_became_active":"2023-06-22T07:59:54.642925+0000","last_became_peered":"2023-06-22T07:59:54.642925+0000","last_unstale":"2023-06-22T08:00:00.754302+0000","last_undegraded":"2023-06-22T08:00:00.754302+0000","last_fullsized":"2023-06-22T08:00:00.754302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:49:13.625768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754231+0000","last_change":"2023-06-22T07:59:55.167307+0000","last_active":"2023-06-22T08:00:00.754231+0000","last_peered":"2023-06-22T08:00:00.754231+0000","last_clean":"2023-06-22T08:00:00.754231+0000","last_became_active":"2023-06-22T07:59:55.167026+0000","last_became_peered":"2023-06-22T07:59:55.167026+0000","last_unstale":"2023-06-22T08:00:00.754231+0000","last_undegraded":"2023-06-22T08:00:00.754231+0000","last_fullsized":"2023-06-22T08:00:00.754231+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:54:51.321010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871379+0000","last_change":"2023-06-22T07:59:59.208263+0000","last_active":"2023-06-22T07:59:59.871379+0000","last_peered":"2023-06-22T07:59:59.871379+0000","last_clean":"2023-06-22T07:59:59.871379+0000","last_became_active":"2023-06-22T07:59:59.207691+0000","last_became_peered":"2023-06-22T07:59:59.207691+0000","last_unstale":"2023-06-22T07:59:59.871379+0000","last_undegraded":"2023-06-22T07:59:59.871379+0000","last_fullsized":"2023-06-22T07:59:59.871379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:37:57.170857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,2],"acting":[3,7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.564652+0000","last_change":"2023-06-22T07:59:54.644570+0000","last_active":"2023-06-22T08:00:41.564652+0000","last_peered":"2023-06-22T08:00:41.564652+0000","last_clean":"2023-06-22T08:00:41.564652+0000","last_became_active":"2023-06-22T07:59:54.644218+0000","last_became_peered":"2023-06-22T07:59:54.644218+0000","last_unstale":"2023-06-22T08:00:41.564652+0000","last_undegraded":"2023-06-22T08:00:41.564652+0000","last_fullsized":"2023-06-22T08:00:41.564652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:05:00.109708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754571+0000","last_change":"2023-06-22T07:59:49.655601+0000","last_active":"2023-06-22T08:00:00.754571+0000","last_peered":"2023-06-22T08:00:00.754571+0000","last_clean":"2023-06-22T08:00:00.754571+0000","last_became_active":"2023-06-22T07:59:49.655328+0000","last_became_peered":"2023-06-22T07:59:49.655328+0000","last_unstale":"2023-06-22T08:00:00.754571+0000","last_undegraded":"2023-06-22T08:00:00.754571+0000","last_fullsized":"2023-06-22T08:00:00.754571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:29:09.967964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755018+0000","last_change":"2023-06-22T07:59:55.163241+0000","last_active":"2023-06-22T08:00:00.755018+0000","last_peered":"2023-06-22T08:00:00.755018+0000","last_clean":"2023-06-22T08:00:00.755018+0000","last_became_active":"2023-06-22T07:59:55.162949+0000","last_became_peered":"2023-06-22T07:59:55.162949+0000","last_unstale":"2023-06-22T08:00:00.755018+0000","last_undegraded":"2023-06-22T08:00:00.755018+0000","last_fullsized":"2023-06-22T08:00:00.755018+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T16:57:51.267726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.755485+0000","last_change":"2023-06-22T07:59:59.201187+0000","last_active":"2023-06-22T08:00:00.755485+0000","last_peered":"2023-06-22T08:00:00.755485+0000","last_clean":"2023-06-22T08:00:00.755485+0000","last_became_active":"2023-06-22T07:59:59.200688+0000","last_became_peered":"2023-06-22T07:59:59.200688+0000","last_unstale":"2023-06-22T08:00:00.755485+0000","last_undegraded":"2023-06-22T08:00:00.755485+0000","last_fullsized":"2023-06-22T08:00:00.755485+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:52:15.059222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2,6],"acting":[4,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.871926+0000","last_change":"2023-06-22T07:59:49.655839+0000","last_active":"2023-06-22T07:59:59.871926+0000","last_peered":"2023-06-22T07:59:59.871926+0000","last_clean":"2023-06-22T07:59:59.871926+0000","last_became_active":"2023-06-22T07:59:49.655494+0000","last_became_peered":"2023-06-22T07:59:49.655494+0000","last_unstale":"2023-06-22T07:59:59.871926+0000","last_undegraded":"2023-06-22T07:59:59.871926+0000","last_fullsized":"2023-06-22T07:59:59.871926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:48.424045+0000","last_clean_scrub_stamp":"2023-06-22T07:59:48.424045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T11:01:23.950347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"28'7","reported_seq":28,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:36.490383+0000","last_change":"2023-06-22T07:59:54.641464+0000","last_active":"2023-06-22T08:00:36.490383+0000","last_peered":"2023-06-22T08:00:36.490383+0000","last_clean":"2023-06-22T08:00:36.490383+0000","last_became_active":"2023-06-22T07:59:54.641161+0000","last_became_peered":"2023-06-22T07:59:54.641161+0000","last_unstale":"2023-06-22T08:00:36.490383+0000","last_undegraded":"2023-06-22T08:00:36.490383+0000","last_fullsized":"2023-06-22T08:00:36.490383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:46.346592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708316+0000","last_change":"2023-06-22T07:59:55.164629+0000","last_active":"2023-06-22T07:59:59.708316+0000","last_peered":"2023-06-22T07:59:59.708316+0000","last_clean":"2023-06-22T07:59:59.708316+0000","last_became_active":"2023-06-22T07:59:55.164329+0000","last_became_peered":"2023-06-22T07:59:55.164329+0000","last_unstale":"2023-06-22T07:59:59.708316+0000","last_undegraded":"2023-06-22T07:59:59.708316+0000","last_fullsized":"2023-06-22T07:59:59.708316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:02:52.058586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754950+0000","last_change":"2023-06-22T07:59:59.201778+0000","last_active":"2023-06-22T08:00:00.754950+0000","last_peered":"2023-06-22T08:00:00.754950+0000","last_clean":"2023-06-22T08:00:00.754950+0000","last_became_active":"2023-06-22T07:59:59.201326+0000","last_became_peered":"2023-06-22T07:59:59.201326+0000","last_unstale":"2023-06-22T08:00:00.754950+0000","last_undegraded":"2023-06-22T08:00:00.754950+0000","last_fullsized":"2023-06-22T08:00:00.754950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:17:31.817024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:25.703139+0000","last_change":"2023-06-22T07:59:54.644816+0000","last_active":"2023-06-22T08:00:25.703139+0000","last_peered":"2023-06-22T08:00:25.703139+0000","last_clean":"2023-06-22T08:00:25.703139+0000","last_became_active":"2023-06-22T07:59:54.644508+0000","last_became_peered":"2023-06-22T07:59:54.644508+0000","last_unstale":"2023-06-22T08:00:25.703139+0000","last_undegraded":"2023-06-22T08:00:25.703139+0000","last_fullsized":"2023-06-22T08:00:25.703139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T17:20:33.819054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753735+0000","last_change":"2023-06-22T07:59:55.167979+0000","last_active":"2023-06-22T08:00:00.753735+0000","last_peered":"2023-06-22T08:00:00.753735+0000","last_clean":"2023-06-22T08:00:00.753735+0000","last_became_active":"2023-06-22T07:59:55.163775+0000","last_became_peered":"2023-06-22T07:59:55.163775+0000","last_unstale":"2023-06-22T08:00:00.753735+0000","last_undegraded":"2023-06-22T08:00:00.753735+0000","last_fullsized":"2023-06-22T08:00:00.753735+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:53:40.882056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155135+0000","last_change":"2023-06-22T07:59:59.194581+0000","last_active":"2023-06-22T08:00:00.155135+0000","last_peered":"2023-06-22T08:00:00.155135+0000","last_clean":"2023-06-22T08:00:00.155135+0000","last_became_active":"2023-06-22T07:59:59.194135+0000","last_became_peered":"2023-06-22T07:59:59.194135+0000","last_unstale":"2023-06-22T08:00:00.155135+0000","last_undegraded":"2023-06-22T08:00:00.155135+0000","last_fullsized":"2023-06-22T08:00:00.155135+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T19:58:12.942764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"28'5","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.565859+0000","last_change":"2023-06-22T07:59:54.651138+0000","last_active":"2023-06-22T08:00:41.565859+0000","last_peered":"2023-06-22T08:00:41.565859+0000","last_clean":"2023-06-22T08:00:41.565859+0000","last_became_active":"2023-06-22T07:59:54.650803+0000","last_became_peered":"2023-06-22T07:59:54.650803+0000","last_unstale":"2023-06-22T08:00:41.565859+0000","last_undegraded":"2023-06-22T08:00:41.565859+0000","last_fullsized":"2023-06-22T08:00:41.565859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:47:45.982711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708617+0000","last_change":"2023-06-22T07:59:55.165814+0000","last_active":"2023-06-22T07:59:59.708617+0000","last_peered":"2023-06-22T07:59:59.708617+0000","last_clean":"2023-06-22T07:59:59.708617+0000","last_became_active":"2023-06-22T07:59:55.165270+0000","last_became_peered":"2023-06-22T07:59:55.165270+0000","last_unstale":"2023-06-22T07:59:59.708617+0000","last_undegraded":"2023-06-22T07:59:59.708617+0000","last_fullsized":"2023-06-22T07:59:59.708617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T13:46:52.422411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708465+0000","last_change":"2023-06-22T07:59:59.212042+0000","last_active":"2023-06-22T07:59:59.708465+0000","last_peered":"2023-06-22T07:59:59.708465+0000","last_clean":"2023-06-22T07:59:59.708465+0000","last_became_active":"2023-06-22T07:59:59.211666+0000","last_became_peered":"2023-06-22T07:59:59.211666+0000","last_unstale":"2023-06-22T07:59:59.708465+0000","last_undegraded":"2023-06-22T07:59:59.708465+0000","last_fullsized":"2023-06-22T07:59:59.708465+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:52:41.731979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3,4],"acting":[2,7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"28'12","reported_seq":33,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.571232+0000","last_change":"2023-06-22T07:59:54.650251+0000","last_active":"2023-06-22T08:00:41.571232+0000","last_peered":"2023-06-22T08:00:41.571232+0000","last_clean":"2023-06-22T08:00:41.571232+0000","last_became_active":"2023-06-22T07:59:54.649993+0000","last_became_peered":"2023-06-22T07:59:54.649993+0000","last_unstale":"2023-06-22T08:00:41.571232+0000","last_undegraded":"2023-06-22T08:00:41.571232+0000","last_fullsized":"2023-06-22T08:00:41.571232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T08:20:20.733896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708375+0000","last_change":"2023-06-22T07:59:55.172595+0000","last_active":"2023-06-22T07:59:59.708375+0000","last_peered":"2023-06-22T07:59:59.708375+0000","last_clean":"2023-06-22T07:59:59.708375+0000","last_became_active":"2023-06-22T07:59:55.165873+0000","last_became_peered":"2023-06-22T07:59:55.165873+0000","last_unstale":"2023-06-22T07:59:59.708375+0000","last_undegraded":"2023-06-22T07:59:59.708375+0000","last_fullsized":"2023-06-22T07:59:59.708375+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:07:02.592807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.154969+0000","last_change":"2023-06-22T07:59:59.192708+0000","last_active":"2023-06-22T08:00:00.154969+0000","last_peered":"2023-06-22T08:00:00.154969+0000","last_clean":"2023-06-22T08:00:00.154969+0000","last_became_active":"2023-06-22T07:59:59.191693+0000","last_became_peered":"2023-06-22T07:59:59.191693+0000","last_unstale":"2023-06-22T08:00:00.154969+0000","last_undegraded":"2023-06-22T08:00:00.154969+0000","last_fullsized":"2023-06-22T08:00:00.154969+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T12:59:33.454962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5,7],"acting":[1,0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.151569+0000","last_change":"2023-06-22T07:59:54.651179+0000","last_active":"2023-06-22T08:00:00.151569+0000","last_peered":"2023-06-22T08:00:00.151569+0000","last_clean":"2023-06-22T08:00:00.151569+0000","last_became_active":"2023-06-22T07:59:54.650852+0000","last_became_peered":"2023-06-22T07:59:54.650852+0000","last_unstale":"2023-06-22T08:00:00.151569+0000","last_undegraded":"2023-06-22T08:00:00.151569+0000","last_fullsized":"2023-06-22T08:00:00.151569+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:01.351861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754043+0000","last_change":"2023-06-22T07:59:55.163050+0000","last_active":"2023-06-22T08:00:00.754043+0000","last_peered":"2023-06-22T08:00:00.754043+0000","last_clean":"2023-06-22T08:00:00.754043+0000","last_became_active":"2023-06-22T07:59:55.162754+0000","last_became_peered":"2023-06-22T07:59:55.162754+0000","last_unstale":"2023-06-22T08:00:00.754043+0000","last_undegraded":"2023-06-22T08:00:00.754043+0000","last_fullsized":"2023-06-22T08:00:00.754043+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.754797+0000","last_change":"2023-06-22T07:59:59.209558+0000","last_active":"2023-06-22T08:00:00.754797+0000","last_peered":"2023-06-22T08:00:00.754797+0000","last_clean":"2023-06-22T08:00:00.754797+0000","last_became_active":"2023-06-22T07:59:59.209129+0000","last_became_peered":"2023-06-22T07:59:59.209129+0000","last_unstale":"2023-06-22T08:00:00.754797+0000","last_undegraded":"2023-06-22T08:00:00.754797+0000","last_fullsized":"2023-06-22T08:00:00.754797+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:58.149234+0000","last_clean_scrub_stamp":"2023-06-22T07:59:58.149234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:22:21.581063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1,0],"acting":[4,3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"28'13","reported_seq":34,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:41.566206+0000","last_change":"2023-06-22T07:59:54.652059+0000","last_active":"2023-06-22T08:00:41.566206+0000","last_peered":"2023-06-22T08:00:41.566206+0000","last_clean":"2023-06-22T08:00:41.566206+0000","last_became_active":"2023-06-22T07:59:54.651720+0000","last_became_peered":"2023-06-22T07:59:54.651720+0000","last_unstale":"2023-06-22T08:00:41.566206+0000","last_undegraded":"2023-06-22T08:00:41.566206+0000","last_fullsized":"2023-06-22T08:00:41.566206+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:53.600398+0000","last_clean_scrub_stamp":"2023-06-22T07:59:53.600398+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T09:54:31.908657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708158+0000","last_change":"2023-06-22T07:59:55.172794+0000","last_active":"2023-06-22T07:59:59.708158+0000","last_peered":"2023-06-22T07:59:59.708158+0000","last_clean":"2023-06-22T07:59:59.708158+0000","last_became_active":"2023-06-22T07:59:55.172527+0000","last_became_peered":"2023-06-22T07:59:55.172527+0000","last_unstale":"2023-06-22T07:59:59.708158+0000","last_undegraded":"2023-06-22T07:59:59.708158+0000","last_fullsized":"2023-06-22T07:59:59.708158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T18:11:09.563525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.155457+0000","last_change":"2023-06-22T07:59:55.165758+0000","last_active":"2023-06-22T08:00:00.155457+0000","last_peered":"2023-06-22T08:00:00.155457+0000","last_clean":"2023-06-22T08:00:00.155457+0000","last_became_active":"2023-06-22T07:59:55.165117+0000","last_became_peered":"2023-06-22T07:59:55.165117+0000","last_unstale":"2023-06-22T08:00:00.155457+0000","last_undegraded":"2023-06-22T08:00:00.155457+0000","last_fullsized":"2023-06-22T08:00:00.155457+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:35:02.357853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.753486+0000","last_change":"2023-06-22T07:59:55.162439+0000","last_active":"2023-06-22T08:00:00.753486+0000","last_peered":"2023-06-22T08:00:00.753486+0000","last_clean":"2023-06-22T08:00:00.753486+0000","last_became_active":"2023-06-22T07:59:55.162072+0000","last_became_peered":"2023-06-22T07:59:55.162072+0000","last_unstale":"2023-06-22T08:00:00.753486+0000","last_undegraded":"2023-06-22T08:00:00.753486+0000","last_fullsized":"2023-06-22T08:00:00.753486+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T10:01:19.745394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T08:00:00.157229+0000","last_change":"2023-06-22T07:59:55.159299+0000","last_active":"2023-06-22T08:00:00.157229+0000","last_peered":"2023-06-22T08:00:00.157229+0000","last_clean":"2023-06-22T08:00:00.157229+0000","last_became_active":"2023-06-22T07:59:55.158893+0000","last_became_peered":"2023-06-22T07:59:55.158893+0000","last_unstale":"2023-06-22T08:00:00.157229+0000","last_undegraded":"2023-06-22T08:00:00.157229+0000","last_fullsized":"2023-06-22T08:00:00.157229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:58:34.094192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.709716+0000","last_change":"2023-06-22T07:59:55.162520+0000","last_active":"2023-06-22T07:59:59.709716+0000","last_peered":"2023-06-22T07:59:59.709716+0000","last_clean":"2023-06-22T07:59:59.709716+0000","last_became_active":"2023-06-22T07:59:55.162211+0000","last_became_peered":"2023-06-22T07:59:55.162211+0000","last_unstale":"2023-06-22T07:59:59.709716+0000","last_undegraded":"2023-06-22T07:59:59.709716+0000","last_fullsized":"2023-06-22T07:59:59.709716+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T14:13:00.662547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-06-22T07:59:59.708947+0000","last_change":"2023-06-22T07:59:55.165364+0000","last_active":"2023-06-22T07:59:59.708947+0000","last_peered":"2023-06-22T07:59:59.708947+0000","last_clean":"2023-06-22T07:59:59.708947+0000","last_became_active":"2023-06-22T07:59:55.165021+0000","last_became_peered":"2023-06-22T07:59:55.165021+0000","last_unstale":"2023-06-22T07:59:59.708947+0000","last_undegraded":"2023-06-22T07:59:59.708947+0000","last_fullsized":"2023-06-22T07:59:59.708947+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-22T07:59:54.606390+0000","last_clean_scrub_stamp":"2023-06-22T07:59:54.606390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-06-23T15:09:07.008945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":10282,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":211,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":185,"ondisk_log_size":185,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":2,"up_from":16,"seq":68719476755,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":344064,"data_stored":128982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.64400000000000002}]},{"osd":1,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.437},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.55000000000000004}]},{"osd":3,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.49099999999999999}]},{"osd":6,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.46400000000000002}]},{"osd":7,"last update":"Thu Jun 22 08:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.58499999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542162,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":344064,"data_stored":128648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.503}]},{"osd":4,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.73599999999999999}]},{"osd":6,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.748}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":319488,"data_stored":124784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.4989999999999997,"5min":6.4989999999999997,"15min":6.4989999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":10.401,"5min":10.401,"15min":10.401},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":10.173999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":10.291,"5min":10.291,"15min":10.291},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":10.064,"5min":10.064,"15min":10.064},"last":0.61499999999999999}]},{"osd":5,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.39900000000000002}]},{"osd":6,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.378},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53900000000000003}]},{"osd":7,"last update":"Thu Jun 22 08:00:40 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.64100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":921600,"data_stored":715384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.312},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.82699999999999996}]},{"osd":2,"last update":"Thu Jun 22 08:00:47 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57499999999999996}]},{"osd":3,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.53800000000000003}]},{"osd":4,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60199999999999998}]},{"osd":5,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67000000000000004}]},{"osd":6,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.73999999999999999}]},{"osd":7,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.64400000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":921600,"data_stored":715472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.623}]},{"osd":2,"last update":"Thu Jun 22 08:00:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.74199999999999999}]},{"osd":4,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.79000000000000004}]},{"osd":6,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.67400000000000004}]},{"osd":7,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.753}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":335872,"data_stored":125148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":7.7000000000000002,"5min":7.7000000000000002,"15min":7.7000000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":3.8010000000000002,"5min":3.8010000000000002,"15min":3.8010000000000002},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":7.7960000000000003,"5min":7.7960000000000003,"15min":7.7960000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Jun 22 08:00:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":9.8460000000000001,"5min":9.8460000000000001,"15min":9.8460000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":2.7210000000000001,"5min":2.7210000000000001,"15min":2.7210000000000001},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.867,"5min":1.867,"15min":1.867},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":9.9019999999999992,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":0.502}]},{"osd":4,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48599999999999999}]}]},{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":307200,"data_stored":124680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jun 22 08:00:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51500000000000001}]},{"osd":3,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.496},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.36499999999999999}]},{"osd":5,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jun 22 08:00:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.47099999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":348160,"data_stored":126666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.917,"5min":1.917,"15min":1.917},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":6.8099999999999996,"5min":6.8099999999999996,"15min":6.8099999999999996},"last":0.32600000000000001}]},{"osd":2,"last update":"Thu Jun 22 08:00:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.6589999999999998},"last":0.438},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.46200000000000002}]},{"osd":3,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48499999999999999}]},{"osd":4,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Thu Jun 22 08:00:42 2023","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-06-22T08:00:58.414 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:00:58.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.641+0000 7f7be5bff700 1 Processor -- start 2023-06-22T08:00:58.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.641+0000 7f7be5bff700 1 -- start start 2023-06-22T08:00:58.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.642+0000 7f7be5bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7be0152d70 0x7f7be00c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.642+0000 7f7be5bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be0159330 0x7f7be00cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.642+0000 7f7be5bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7be00cb800 0x7f7be00c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.642+0000 7f7be5bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be00c95e0 con 0x7f7be00cb800 2023-06-22T08:00:58.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.642+0000 7f7be5bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be015bea0 con 0x7f7be0152d70 2023-06-22T08:00:58.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.642+0000 7f7be5bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be015c160 con 0x7f7be0159330 2023-06-22T08:00:58.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7be00cb800 0x7f7be00c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdeffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be0159330 0x7f7be00cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdf7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7be0152d70 0x7f7be00c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7be00cb800 0x7f7be00c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52868/0 (socket says 172.21.15.23:52868) 2023-06-22T08:00:58.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdeffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be0159330 0x7f7be00cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49976/0 (socket says 172.21.15.23:49976) 2023-06-22T08:00:58.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdffff700 1 -- 172.21.15.23:0/2031952941 learned_addr learned my addr 172.21.15.23:0/2031952941 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:58.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdffff700 1 -- 172.21.15.23:0/2031952941 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be0159330 msgr2=0x7f7be00cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:58.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdffff700 1 --2- 172.21.15.23:0/2031952941 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be0159330 0x7f7be00cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdffff700 1 -- 172.21.15.23:0/2031952941 >> v2:172.21.15.67:3300/0 conn(0x7f7be0152d70 msgr2=0x7f7be00c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:58.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdffff700 1 --2- 172.21.15.23:0/2031952941 >> v2:172.21.15.67:3300/0 conn(0x7f7be0152d70 0x7f7be00c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdffff700 1 -- 172.21.15.23:0/2031952941 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be015c440 con 0x7f7be00cb800 2023-06-22T08:00:58.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.643+0000 7f7bdffff700 1 --2- 172.21.15.23:0/2031952941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be00cb800 0x7f7be00c45f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7be00c8ef0 tx=0x7f7bd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dce6ab33e45129d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:58.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7bde7fc700 1 -- 172.21.15.23:0/2031952941 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd800e070 con 0x7f7be00cb800 2023-06-22T08:00:58.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7bde7fc700 1 -- 172.21.15.23:0/2031952941 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd800b910 con 0x7f7be00cb800 2023-06-22T08:00:58.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7bde7fc700 1 -- 172.21.15.23:0/2031952941 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd800bcc0 con 0x7f7be00cb800 2023-06-22T08:00:58.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7be5bff700 1 -- 172.21.15.23:0/2031952941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be00cb800 msgr2=0x7f7be00c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7be5bff700 1 --2- 172.21.15.23:0/2031952941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be00cb800 0x7f7be00c45f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7be00c8ef0 tx=0x7f7bd800b330 comp rx=0 tx=0).stop 2023-06-22T08:00:58.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7be5bff700 1 -- 172.21.15.23:0/2031952941 shutdown_connections 2023-06-22T08:00:58.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7be5bff700 1 --2- 172.21.15.23:0/2031952941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be00cb800 0x7f7be00c45f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7be5bff700 1 --2- 172.21.15.23:0/2031952941 >> v2:172.21.15.67:3300/0 conn(0x7f7be0152d70 0x7f7be00c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7be5bff700 1 --2- 172.21.15.23:0/2031952941 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be0159330 0x7f7be00cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7be5bff700 1 -- 172.21.15.23:0/2031952941 >> 172.21.15.23:0/2031952941 conn(0x7f7be006ae30 msgr2=0x7f7be006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:58.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7be5bff700 1 -- 172.21.15.23:0/2031952941 shutdown_connections 2023-06-22T08:00:58.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.644+0000 7f7be5bff700 1 -- 172.21.15.23:0/2031952941 wait complete. 2023-06-22T08:00:58.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.647+0000 7f7be5bff700 1 Processor -- start 2023-06-22T08:00:58.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.647+0000 7f7be5bff700 1 -- start start 2023-06-22T08:00:58.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7be5bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be00cb800 0x7f7be01e4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7be5bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be0152d70 0x7f7be014be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7be5bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be0159330 0x7f7be01f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7be5bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be01e4c20 con 0x7f7be0159330 2023-06-22T08:00:58.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7be5bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be01e4980 con 0x7f7be0152d70 2023-06-22T08:00:58.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7be5bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be00cf680 con 0x7f7be00cb800 2023-06-22T08:00:58.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7bdf7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be00cb800 0x7f7be01e4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7bdf7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be00cb800 0x7f7be01e4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49988/0 (socket says 172.21.15.23:49988) 2023-06-22T08:00:58.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7bdf7fe700 1 -- 172.21.15.23:0/1852350685 learned_addr learned my addr 172.21.15.23:0/1852350685 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:58.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be0152d70 0x7f7be014be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.648+0000 7f7bdffff700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be0159330 0x7f7be01f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7bdf7fe700 1 -- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be0152d70 msgr2=0x7f7be014be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7bdf7fe700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be0152d70 0x7f7be014be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7bdf7fe700 1 -- 172.21.15.23:0/1852350685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be0159330 msgr2=0x7f7be01f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7bdf7fe700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be0159330 0x7f7be01f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7bdf7fe700 1 -- 172.21.15.23:0/1852350685 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd8009b30 con 0x7f7be00cb800 2023-06-22T08:00:58.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7bdf7fe700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be00cb800 0x7f7be01e4350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000e930 tx=0x7f7bd000ec40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:58.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7bc7fff700 1 -- 172.21.15.23:0/1852350685 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0013680 con 0x7f7be00cb800 2023-06-22T08:00:58.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7bc7fff700 1 -- 172.21.15.23:0/1852350685 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd00137e0 con 0x7f7be00cb800 2023-06-22T08:00:58.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7bc7fff700 1 -- 172.21.15.23:0/1852350685 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd0013b70 con 0x7f7be00cb800 2023-06-22T08:00:58.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7be5bff700 1 -- 172.21.15.23:0/1852350685 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be00cf3a0 con 0x7f7be00cb800 2023-06-22T08:00:58.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.649+0000 7f7be5bff700 1 -- 172.21.15.23:0/1852350685 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be01f2090 con 0x7f7be00cb800 2023-06-22T08:00:58.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.650+0000 7f7be5bff700 1 -- 172.21.15.23:0/1852350685 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be00742e0 con 0x7f7be00cb800 2023-06-22T08:00:58.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.652+0000 7f7bc7fff700 1 -- 172.21.15.23:0/1852350685 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7bd0013cd0 con 0x7f7be00cb800 2023-06-22T08:00:58.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.653+0000 7f7bc7fff700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bcc06ace0 0x7f7bcc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:58.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.653+0000 7f7bc7fff700 1 -- 172.21.15.23:0/1852350685 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f7bd00897e0 con 0x7f7be00cb800 2023-06-22T08:00:58.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.653+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bcc06ace0 0x7f7bcc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:58.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.654+0000 7f7bdeffd700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bcc06ace0 0x7f7bcc06d190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7be00b4810 tx=0x7f7bd4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:58.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.656+0000 7f7bc7fff700 1 -- 172.21.15.23:0/1852350685 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7bd0054580 con 0x7f7be00cb800 2023-06-22T08:00:58.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.870+0000 7f7be5bff700 1 -- 172.21.15.23:0/1852350685 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7be014cde0 con 0x7f7be00cb800 2023-06-22T08:00:58.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.872+0000 7f7bc7fff700 1 -- 172.21.15.23:0/1852350685 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f7bd0057ba0 con 0x7f7be00cb800 2023-06-22T08:00:58.876 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:00:58.876 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:00:58.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 -- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bcc06ace0 msgr2=0x7f7bcc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bcc06ace0 0x7f7bcc06d190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7be00b4810 tx=0x7f7bd4009210 comp rx=0 tx=0).stop 2023-06-22T08:00:58.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 -- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be00cb800 msgr2=0x7f7be01e4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:58.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be00cb800 0x7f7be01e4350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000e930 tx=0x7f7bd000ec40 comp rx=0 tx=0).stop 2023-06-22T08:00:58.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 -- 172.21.15.23:0/1852350685 shutdown_connections 2023-06-22T08:00:58.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7be0159330 0x7f7be01f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7be00cb800 0x7f7be01e4350 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bcc06ace0 0x7f7bcc06d190 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 --2- 172.21.15.23:0/1852350685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7be0152d70 0x7f7be014be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:58.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 -- 172.21.15.23:0/1852350685 >> 172.21.15.23:0/1852350685 conn(0x7f7be006ae30 msgr2=0x7f7be00ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:58.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 -- 172.21.15.23:0/1852350685 shutdown_connections 2023-06-22T08:00:58.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:58.875+0000 7f7be5bff700 1 -- 172.21.15.23:0/1852350685 wait complete. 2023-06-22T08:00:58.896 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T07:59:40.617747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-22T07:59:48.190528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-22T07:59:53.576494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-22T07:59:54.105472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-22T07:59:57.198896+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-22T08:00:58.897 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T08:00:59.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.096+0000 7f88f97ff700 1 Processor -- start 2023-06-22T08:00:59.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.096+0000 7f88f97ff700 1 -- start start 2023-06-22T08:00:59.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.098+0000 7f88f97ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f88f4152ba0 0x7f88f41593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.098+0000 7f88f97ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f88f40c9260 0x7f88f40c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.098+0000 7f88f97ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f4159930 con 0x7f88f40c4650 2023-06-22T08:00:59.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.098+0000 7f88f97ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f40c9aa0 con 0x7f88f4152ba0 2023-06-22T08:00:59.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.098+0000 7f88f97ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f406aed0 con 0x7f88f40c9260 2023-06-22T08:00:59.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.098+0000 7f88f2ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f88f4152ba0 0x7f88f41593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.098+0000 7f88f2ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f88f4152ba0 0x7f88f41593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52884/0 (socket says 172.21.15.23:52884) 2023-06-22T08:00:59.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.098+0000 7f88f2ffd700 1 -- 172.21.15.23:0/2647681382 learned_addr learned my addr 172.21.15.23:0/2647681382 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:59.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.098+0000 7f88f27fc700 1 --2- 172.21.15.23:0/2647681382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f88f40c9260 0x7f88f40c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.099+0000 7f88f2ffd700 1 -- 172.21.15.23:0/2647681382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f88f40c9260 msgr2=0x7f88f40c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:59.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.099+0000 7f88f2ffd700 1 --2- 172.21.15.23:0/2647681382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f88f40c9260 0x7f88f40c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.099+0000 7f88f2ffd700 1 -- 172.21.15.23:0/2647681382 >> v1:172.21.15.67:6789/0 conn(0x7f88f40c4650 legacy=0x7f88f40c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.099+0000 7f88f2ffd700 1 -- 172.21.15.23:0/2647681382 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f406b1b0 con 0x7f88f4152ba0 2023-06-22T08:00:59.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.099+0000 7f88f2ffd700 1 --2- 172.21.15.23:0/2647681382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f4152ba0 0x7f88f41593f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f88e40098f0 tx=0x7f88e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911679def0deb7d4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:59.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.099+0000 7f88f1ffb700 1 -- 172.21.15.23:0/2647681382 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e4011830 con 0x7f88f4152ba0 2023-06-22T08:00:59.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.099+0000 7f88f1ffb700 1 -- 172.21.15.23:0/2647681382 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e4011990 con 0x7f88f4152ba0 2023-06-22T08:00:59.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.099+0000 7f88f1ffb700 1 -- 172.21.15.23:0/2647681382 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e4011d20 con 0x7f88f4152ba0 2023-06-22T08:00:59.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.100+0000 7f88f97ff700 1 -- 172.21.15.23:0/2647681382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f4152ba0 msgr2=0x7f88f41593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.100+0000 7f88f97ff700 1 --2- 172.21.15.23:0/2647681382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f4152ba0 0x7f88f41593f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f88e40098f0 tx=0x7f88e4009c00 comp rx=0 tx=0).stop 2023-06-22T08:00:59.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.100+0000 7f88f97ff700 1 -- 172.21.15.23:0/2647681382 shutdown_connections 2023-06-22T08:00:59.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.100+0000 7f88f97ff700 1 --2- 172.21.15.23:0/2647681382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f4152ba0 0x7f88f41593f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.100+0000 7f88f97ff700 1 --2- 172.21.15.23:0/2647681382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f88f40c9260 0x7f88f40c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.100+0000 7f88f97ff700 1 -- 172.21.15.23:0/2647681382 >> 172.21.15.23:0/2647681382 conn(0x7f88f4065b90 msgr2=0x7f88f4065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:59.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.100+0000 7f88f97ff700 1 -- 172.21.15.23:0/2647681382 shutdown_connections 2023-06-22T08:00:59.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.100+0000 7f88f97ff700 1 -- 172.21.15.23:0/2647681382 wait complete. 2023-06-22T08:00:59.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.101+0000 7f88f97ff700 1 Processor -- start 2023-06-22T08:00:59.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.102+0000 7f88f97ff700 1 -- start start 2023-06-22T08:00:59.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.102+0000 7f88f97ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f40c4650 0x7f88f40b3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.102+0000 7f88f97ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f88f40c9260 0x7f88f40b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.102+0000 7f88f97ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f4152ba0 0x7f88f41efbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.102+0000 7f88f97ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f41f17c0 con 0x7f88f4152ba0 2023-06-22T08:00:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.102+0000 7f88f97ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f41f2300 con 0x7f88f40c4650 2023-06-22T08:00:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.102+0000 7f88f97ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f41f1a60 con 0x7f88f40c9260 2023-06-22T08:00:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f40c4650 0x7f88f40b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f4152ba0 0x7f88f41efbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f27fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f88f40c9260 0x7f88f40b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f40c4650 0x7f88f40b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51704/0 (socket says 172.21.15.23:51704) 2023-06-22T08:00:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f2ffd700 1 -- 172.21.15.23:0/3015587387 learned_addr learned my addr 172.21.15.23:0/3015587387 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f2ffd700 1 -- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f88f40c9260 msgr2=0x7f88f40b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f2ffd700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f88f40c9260 0x7f88f40b38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f2ffd700 1 -- 172.21.15.23:0/3015587387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f4152ba0 msgr2=0x7f88f41efbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f2ffd700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f4152ba0 0x7f88f41efbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f2ffd700 1 -- 172.21.15.23:0/3015587387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e4009550 con 0x7f88f40c4650 2023-06-22T08:00:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f37fe700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f4152ba0 0x7f88f41efbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.103+0000 7f88f2ffd700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f40c4650 0x7f88f40b3360 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009c40 tx=0x7f88e4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:59.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.104+0000 7f88db7fe700 1 -- 172.21.15.23:0/3015587387 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e4004510 con 0x7f88f40c4650 2023-06-22T08:00:59.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.104+0000 7f88db7fe700 1 -- 172.21.15.23:0/3015587387 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e401e070 con 0x7f88f40c4650 2023-06-22T08:00:59.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.104+0000 7f88db7fe700 1 -- 172.21.15.23:0/3015587387 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e40038e0 con 0x7f88f40c4650 2023-06-22T08:00:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.104+0000 7f88f97ff700 1 -- 172.21.15.23:0/3015587387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f41f1d40 con 0x7f88f40c4650 2023-06-22T08:00:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.104+0000 7f88f97ff700 1 -- 172.21.15.23:0/3015587387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f41ef3e0 con 0x7f88f40c4650 2023-06-22T08:00:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.105+0000 7f88f97ff700 1 -- 172.21.15.23:0/3015587387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f40cbed0 con 0x7f88f40c4650 2023-06-22T08:00:59.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.106+0000 7f88db7fe700 1 -- 172.21.15.23:0/3015587387 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f88e4003a40 con 0x7f88f40c4650 2023-06-22T08:00:59.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.107+0000 7f88db7fe700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006ad10 0x7f88e006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.107+0000 7f88db7fe700 1 -- 172.21.15.23:0/3015587387 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f88e408a6d0 con 0x7f88f40c4650 2023-06-22T08:00:59.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.107+0000 7f88f27fc700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006ad10 0x7f88e006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.108+0000 7f88f27fc700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006ad10 0x7f88e006d1c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f88f414e890 tx=0x7f88e8009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:59.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.112+0000 7f88db7fe700 1 -- 172.21.15.23:0/3015587387 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f88e4055400 con 0x7f88f40c4650 2023-06-22T08:00:59.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.320+0000 7f88f97ff700 1 -- 172.21.15.23:0/3015587387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88f404ea20 con 0x7f88f40c4650 2023-06-22T08:00:59.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.321+0000 7f88db7fe700 1 -- 172.21.15.23:0/3015587387 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88e4058a20 con 0x7f88f40c4650 2023-06-22T08:00:59.324 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-06-22T08:00:59.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 -- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006ad10 msgr2=0x7f88e006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006ad10 0x7f88e006d1c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f88f414e890 tx=0x7f88e8009210 comp rx=0 tx=0).stop 2023-06-22T08:00:59.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 -- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f40c4650 msgr2=0x7f88f40b3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f40c4650 0x7f88f40b3360 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009c40 tx=0x7f88e4004750 comp rx=0 tx=0).stop 2023-06-22T08:00:59.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 -- 172.21.15.23:0/3015587387 shutdown_connections 2023-06-22T08:00:59.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88f4152ba0 0x7f88f41efbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f88e006ad10 0x7f88e006d1c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88f40c4650 0x7f88f40b3360 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 --2- 172.21.15.23:0/3015587387 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f88f40c9260 0x7f88f40b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 -- 172.21.15.23:0/3015587387 >> 172.21.15.23:0/3015587387 conn(0x7f88f4065b90 msgr2=0x7f88f4150c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:59.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 -- 172.21.15.23:0/3015587387 shutdown_connections 2023-06-22T08:00:59.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.323+0000 7f88f97ff700 1 -- 172.21.15.23:0/3015587387 wait complete. 2023-06-22T08:00:59.345 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T08:00:59.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.542+0000 7f34a9dff700 1 Processor -- start 2023-06-22T08:00:59.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.543+0000 7f34a9dff700 1 -- start start 2023-06-22T08:00:59.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.544+0000 7f34a9dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34a4150c60 0x7f34a415a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.544+0000 7f34a9dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a40c6160 0x7f34a40c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.544+0000 7f34a9dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a40bdc40 con 0x7f34a4150890 2023-06-22T08:00:59.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.544+0000 7f34a9dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a4157710 con 0x7f34a4150c60 2023-06-22T08:00:59.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.544+0000 7f34a9dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4156b90 con 0x7f34a40c6160 2023-06-22T08:00:59.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.544+0000 7f34a2ffd700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34a4150c60 0x7f34a415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.544+0000 7f34a3fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a40c6160 0x7f34a40c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.544+0000 7f34a37fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f34a4150890 0x7f34a40c7570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:51472/0 (socket says 172.21.15.23:51472) 2023-06-22T08:00:59.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.544+0000 7f34a37fe700 1 -- 172.21.15.23:0/2953705103 learned_addr learned my addr 172.21.15.23:0/2953705103 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:59.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a27fc700 1 -- 172.21.15.23:0/2953705103 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370910450 0 0) 0x7f34a40bdc40 con 0x7f34a4150890 2023-06-22T08:00:59.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a27fc700 1 -- 172.21.15.23:0/2953705103 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f348c0035b0 con 0x7f34a4150890 2023-06-22T08:00:59.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a2ffd700 1 -- 172.21.15.23:0/2953705103 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a40c6160 msgr2=0x7f34a40c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:59.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a2ffd700 1 --2- 172.21.15.23:0/2953705103 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a40c6160 0x7f34a40c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a2ffd700 1 -- 172.21.15.23:0/2953705103 >> v1:172.21.15.67:6789/0 conn(0x7f34a4150890 legacy=0x7f34a40c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a2ffd700 1 -- 172.21.15.23:0/2953705103 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a4156e70 con 0x7f34a4150c60 2023-06-22T08:00:59.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a3fff700 1 --2- 172.21.15.23:0/2953705103 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a40c6160 0x7f34a40c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:00:59.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a2ffd700 1 --2- 172.21.15.23:0/2953705103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a4150c60 0x7f34a415a890 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3498009840 tx=0x7f3498009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2dbb0ecff0d1fc0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:59.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a27fc700 1 -- 172.21.15.23:0/2953705103 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3498011830 con 0x7f34a4150c60 2023-06-22T08:00:59.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a27fc700 1 -- 172.21.15.23:0/2953705103 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3498011990 con 0x7f34a4150c60 2023-06-22T08:00:59.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.545+0000 7f34a27fc700 1 -- 172.21.15.23:0/2953705103 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3498011d20 con 0x7f34a4150c60 2023-06-22T08:00:59.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.546+0000 7f34a9dff700 1 -- 172.21.15.23:0/2953705103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a4150c60 msgr2=0x7f34a415a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.546+0000 7f34a9dff700 1 --2- 172.21.15.23:0/2953705103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a4150c60 0x7f34a415a890 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3498009840 tx=0x7f3498009b50 comp rx=0 tx=0).stop 2023-06-22T08:00:59.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.546+0000 7f34a9dff700 1 -- 172.21.15.23:0/2953705103 shutdown_connections 2023-06-22T08:00:59.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.546+0000 7f34a9dff700 1 --2- 172.21.15.23:0/2953705103 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a40c6160 0x7f34a40c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.546+0000 7f34a9dff700 1 --2- 172.21.15.23:0/2953705103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a4150c60 0x7f34a415a890 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.546+0000 7f34a9dff700 1 -- 172.21.15.23:0/2953705103 >> 172.21.15.23:0/2953705103 conn(0x7f34a40cb070 msgr2=0x7f34a40c6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:59.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.546+0000 7f34a9dff700 1 -- 172.21.15.23:0/2953705103 shutdown_connections 2023-06-22T08:00:59.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.546+0000 7f34a9dff700 1 -- 172.21.15.23:0/2953705103 wait complete. 2023-06-22T08:00:59.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.548+0000 7f34a9dff700 1 Processor -- start 2023-06-22T08:00:59.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.548+0000 7f34a9dff700 1 -- start start 2023-06-22T08:00:59.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a9dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40c6160 0x7f34a4148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a9dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34a4150890 0x7f34a41587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a9dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a4150c60 0x7f34a4158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a9dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a41580c0 con 0x7f34a4150890 2023-06-22T08:00:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a9dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4157de0 con 0x7f34a40c6160 2023-06-22T08:00:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a9dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4070db0 con 0x7f34a4150c60 2023-06-22T08:00:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34a4150890 0x7f34a41587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34a4150890 0x7f34a41587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52908/0 (socket says 172.21.15.23:52908) 2023-06-22T08:00:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a2ffd700 1 -- 172.21.15.23:0/3623554335 learned_addr learned my addr 172.21.15.23:0/3623554335 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:00:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a37fe700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40c6160 0x7f34a4148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a3fff700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a4150c60 0x7f34a4158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a2ffd700 1 -- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a4150c60 msgr2=0x7f34a4158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a2ffd700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a4150c60 0x7f34a4158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a2ffd700 1 -- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40c6160 msgr2=0x7f34a4148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a2ffd700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40c6160 0x7f34a4148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.549+0000 7f34a2ffd700 1 -- 172.21.15.23:0/3623554335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c009a70 con 0x7f34a4150890 2023-06-22T08:00:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.550+0000 7f34a2ffd700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34a4150890 0x7f34a41587b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3498009b90 tx=0x7f3498004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.550+0000 7f348bfff700 1 -- 172.21.15.23:0/3623554335 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980044d0 con 0x7f34a4150890 2023-06-22T08:00:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.550+0000 7f348bfff700 1 -- 172.21.15.23:0/3623554335 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34980036a0 con 0x7f34a4150890 2023-06-22T08:00:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.550+0000 7f348bfff700 1 -- 172.21.15.23:0/3623554335 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3498003a30 con 0x7f34a4150890 2023-06-22T08:00:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.550+0000 7f34a9dff700 1 -- 172.21.15.23:0/3623554335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34980094a0 con 0x7f34a4150890 2023-06-22T08:00:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.550+0000 7f34a9dff700 1 -- 172.21.15.23:0/3623554335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a4071370 con 0x7f34a4150890 2023-06-22T08:00:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.551+0000 7f34a9dff700 1 -- 172.21.15.23:0/3623554335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a404ea20 con 0x7f34a4150890 2023-06-22T08:00:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.552+0000 7f348bfff700 1 -- 172.21.15.23:0/3623554335 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3498003b90 con 0x7f34a4150890 2023-06-22T08:00:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.554+0000 7f348bfff700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f349006ace0 0x7f349006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.554+0000 7f348bfff700 1 -- 172.21.15.23:0/3623554335 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f349808ab80 con 0x7f34a4150890 2023-06-22T08:00:59.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.554+0000 7f34a37fe700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f349006ace0 0x7f349006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.555+0000 7f34a37fe700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f349006ace0 0x7f349006d190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3494007fd0 tx=0x7f3494007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:59.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.559+0000 7f348bfff700 1 -- 172.21.15.23:0/3623554335 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3498055af0 con 0x7f34a4150890 2023-06-22T08:00:59.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.765+0000 7f34a9dff700 1 -- 172.21.15.23:0/3623554335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f34a4158320 con 0x7f34a4150890 2023-06-22T08:00:59.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.766+0000 7f348bfff700 1 -- 172.21.15.23:0/3623554335 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v28) v1 ==== 92+0+10 (secure 0 0 0) 0x7f34980150a0 con 0x7f34a4150890 2023-06-22T08:00:59.769 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2023-06-22T08:00:59.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.768+0000 7f34a9dff700 1 -- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f349006ace0 msgr2=0x7f349006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.768+0000 7f34a9dff700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f349006ace0 0x7f349006d190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3494007fd0 tx=0x7f3494007dc0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.768+0000 7f34a9dff700 1 -- 172.21.15.23:0/3623554335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34a4150890 msgr2=0x7f34a41587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.768+0000 7f34a9dff700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34a4150890 0x7f34a41587b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3498009b90 tx=0x7f3498004750 comp rx=0 tx=0).stop 2023-06-22T08:00:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.768+0000 7f34a9dff700 1 -- 172.21.15.23:0/3623554335 shutdown_connections 2023-06-22T08:00:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.768+0000 7f34a9dff700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f34a4150c60 0x7f34a4158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.768+0000 7f34a9dff700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f349006ace0 0x7f349006d190 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.768+0000 7f34a9dff700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34a40c6160 0x7f34a4148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.768+0000 7f34a9dff700 1 --2- 172.21.15.23:0/3623554335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34a4150890 0x7f34a41587b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.768+0000 7f34a9dff700 1 -- 172.21.15.23:0/3623554335 >> 172.21.15.23:0/3623554335 conn(0x7f34a40cb070 msgr2=0x7f34a414e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.769+0000 7f34a9dff700 1 -- 172.21.15.23:0/3623554335 shutdown_connections 2023-06-22T08:00:59.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.769+0000 7f34a9dff700 1 -- 172.21.15.23:0/3623554335 wait complete. 2023-06-22T08:00:59.789 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-22T08:00:59.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.988+0000 7f19085ff700 1 Processor -- start 2023-06-22T08:00:59.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.988+0000 7f19085ff700 1 -- start start 2023-06-22T08:00:59.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.989+0000 7f19085ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1900152ba0 0x7f19001593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.989+0000 7f19085ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f19000c9ee0 0x7f19000ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.989+0000 7f19085ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1900159930 con 0x7f19000c9260 2023-06-22T08:00:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.989+0000 7f19085ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19000c9aa0 con 0x7f19000c9ee0 2023-06-22T08:00:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.989+0000 7f19085ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19000ca880 con 0x7f1900152ba0 2023-06-22T08:00:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.989+0000 7f1906b9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f19000c9ee0 0x7f19000ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.989+0000 7f1905b9a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f19000c9260 0x7f19000c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:45568/0 (socket says 172.21.15.23:45568) 2023-06-22T08:00:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.989+0000 7f1905b9a700 1 -- 172.21.15.23:0/2186873878 learned_addr learned my addr 172.21.15.23:0/2186873878 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:00:59.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.989+0000 7f190639b700 1 --2- 172.21.15.23:0/2186873878 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1900152ba0 0x7f19001593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:00:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.990+0000 7f1906b9c700 1 -- 172.21.15.23:0/2186873878 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1900152ba0 msgr2=0x7f19001593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:00:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.990+0000 7f1906b9c700 1 --2- 172.21.15.23:0/2186873878 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1900152ba0 0x7f19001593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.990+0000 7f1906b9c700 1 -- 172.21.15.23:0/2186873878 >> v1:172.21.15.23:6789/0 conn(0x7f19000c9260 legacy=0x7f19000c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.990+0000 7f1906b9c700 1 -- 172.21.15.23:0/2186873878 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190006aed0 con 0x7f19000c9ee0 2023-06-22T08:00:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.990+0000 7f1906b9c700 1 --2- 172.21.15.23:0/2186873878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19000c9ee0 0x7f19000ca340 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009d90 tx=0x7f18f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1cbef5bc0720829 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:00:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f1905399700 1 -- 172.21.15.23:0/2186873878 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f800e070 con 0x7f19000c9ee0 2023-06-22T08:00:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f1905399700 1 -- 172.21.15.23:0/2186873878 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f800bb20 con 0x7f19000c9ee0 2023-06-22T08:00:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f1905399700 1 -- 172.21.15.23:0/2186873878 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8012430 con 0x7f19000c9ee0 2023-06-22T08:00:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f19085ff700 1 -- 172.21.15.23:0/2186873878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19000c9ee0 msgr2=0x7f19000ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:00:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f19085ff700 1 --2- 172.21.15.23:0/2186873878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19000c9ee0 0x7f19000ca340 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009d90 tx=0x7f18f800b330 comp rx=0 tx=0).stop 2023-06-22T08:00:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f19085ff700 1 -- 172.21.15.23:0/2186873878 shutdown_connections 2023-06-22T08:00:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f19085ff700 1 --2- 172.21.15.23:0/2186873878 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1900152ba0 0x7f19001593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f19085ff700 1 --2- 172.21.15.23:0/2186873878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19000c9ee0 0x7f19000ca340 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:00:59.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f19085ff700 1 -- 172.21.15.23:0/2186873878 >> 172.21.15.23:0/2186873878 conn(0x7f1900065b90 msgr2=0x7f1900065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:00:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f19085ff700 1 -- 172.21.15.23:0/2186873878 shutdown_connections 2023-06-22T08:00:59.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.991+0000 7f19085ff700 1 -- 172.21.15.23:0/2186873878 wait complete. 2023-06-22T08:00:59.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.993+0000 7f19085ff700 1 Processor -- start 2023-06-22T08:00:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.993+0000 7f19085ff700 1 -- start start 2023-06-22T08:00:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f19085ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19000c9260 0x7f19000cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f19085ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19000c9ee0 0x7f19001534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f19085ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1900152ba0 0x7f19001ef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:00:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f19085ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19001546e0 con 0x7f1900152ba0 2023-06-22T08:00:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f19085ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19001ef2d0 con 0x7f19000c9ee0 2023-06-22T08:00:59.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f19085ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1900154930 con 0x7f19000c9260 2023-06-22T08:01:00.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f1906b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1900152ba0 0x7f19001ef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f190639b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19000c9260 0x7f19000cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f190639b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19000c9260 0x7f19000cf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:50042/0 (socket says 172.21.15.23:50042) 2023-06-22T08:01:00.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f190639b700 1 -- 172.21.15.23:0/1042308492 learned_addr learned my addr 172.21.15.23:0/1042308492 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:01:00.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f1906b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1900152ba0 0x7f19001ef7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52940/0 (socket says 172.21.15.23:52940) 2023-06-22T08:01:00.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.994+0000 7f1905b9a700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19000c9ee0 0x7f19001534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.995+0000 7f1906b9c700 1 -- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19000c9260 msgr2=0x7f19000cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.995+0000 7f1906b9c700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19000c9260 0x7f19000cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.995+0000 7f1906b9c700 1 -- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19000c9ee0 msgr2=0x7f19001534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.995+0000 7f1906b9c700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19000c9ee0 0x7f19001534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.995+0000 7f1906b9c700 1 -- 172.21.15.23:0/1042308492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f8009a70 con 0x7f1900152ba0 2023-06-22T08:01:00.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.995+0000 7f1906b9c700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1900152ba0 0x7f19001ef7d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f18f8016000 tx=0x7f18f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:00.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.995+0000 7f18f6ffd700 1 -- 172.21.15.23:0/1042308492 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8012430 con 0x7f1900152ba0 2023-06-22T08:01:00.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.995+0000 7f18f6ffd700 1 -- 172.21.15.23:0/1042308492 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f8020070 con 0x7f1900152ba0 2023-06-22T08:01:00.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.995+0000 7f18f6ffd700 1 -- 172.21.15.23:0/1042308492 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f80098f0 con 0x7f1900152ba0 2023-06-22T08:01:00.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.995+0000 7f19085ff700 1 -- 172.21.15.23:0/1042308492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1900154c10 con 0x7f1900152ba0 2023-06-22T08:01:00.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.996+0000 7f19085ff700 1 -- 172.21.15.23:0/1042308492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19001551d0 con 0x7f1900152ba0 2023-06-22T08:01:00.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.997+0000 7f19085ff700 1 -- 172.21.15.23:0/1042308492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19000cbe90 con 0x7f1900152ba0 2023-06-22T08:01:00.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.997+0000 7f18f6ffd700 1 -- 172.21.15.23:0/1042308492 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f18f8004040 con 0x7f1900152ba0 2023-06-22T08:01:00.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.999+0000 7f18f6ffd700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f18e406ace0 0x7f18e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.999+0000 7f18f6ffd700 1 -- 172.21.15.23:0/1042308492 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f18f8022070 con 0x7f1900152ba0 2023-06-22T08:01:00.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:00:59.999+0000 7f190639b700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f18e406ace0 0x7f18e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.000+0000 7f190639b700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f18e406ace0 0x7f18e406d190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009c00 tx=0x7f18fc009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:00.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.004+0000 7f18f6ffd700 1 -- 172.21.15.23:0/1042308492 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f18f80560f0 con 0x7f1900152ba0 2023-06-22T08:01:00.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.211+0000 7f19085ff700 1 -- 172.21.15.23:0/1042308492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f190004ea20 con 0x7f1900152ba0 2023-06-22T08:01:00.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.212+0000 7f18f6ffd700 1 -- 172.21.15.23:0/1042308492 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f18f8059710 con 0x7f1900152ba0 2023-06-22T08:01:00.215 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:01:00.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 -- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f18e406ace0 msgr2=0x7f18e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f18e406ace0 0x7f18e406d190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009c00 tx=0x7f18fc009210 comp rx=0 tx=0).stop 2023-06-22T08:01:00.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 -- 172.21.15.23:0/1042308492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1900152ba0 msgr2=0x7f19001ef7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1900152ba0 0x7f19001ef7d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f18f8016000 tx=0x7f18f8004ad0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 -- 172.21.15.23:0/1042308492 shutdown_connections 2023-06-22T08:01:00.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1900152ba0 0x7f19001ef7d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19000c9260 0x7f19000cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f18e406ace0 0x7f18e406d190 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 --2- 172.21.15.23:0/1042308492 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19000c9ee0 0x7f19001534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 -- 172.21.15.23:0/1042308492 >> 172.21.15.23:0/1042308492 conn(0x7f1900065b90 msgr2=0x7f19000cb540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:01:00.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 -- 172.21.15.23:0/1042308492 shutdown_connections 2023-06-22T08:01:00.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.214+0000 7f19085ff700 1 -- 172.21.15.23:0/1042308492 wait complete. 2023-06-22T08:01:00.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-22T08:01:00.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.434+0000 7f3cff5ff700 1 Processor -- start 2023-06-22T08:01:00.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.434+0000 7f3cff5ff700 1 -- start start 2023-06-22T08:01:00.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.435+0000 7f3cff5ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3cf8152ba0 0x7f3cf81593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.435+0000 7f3cff5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf80c9ee0 0x7f3cf80ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cff5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf8159930 con 0x7f3cf80c9260 2023-06-22T08:01:00.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cff5ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf80c9aa0 con 0x7f3cf8152ba0 2023-06-22T08:01:00.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cff5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf80ca880 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cfd39b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3cf8152ba0 0x7f3cf81593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cfd39b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3cf8152ba0 0x7f3cf81593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52942/0 (socket says 172.21.15.23:52942) 2023-06-22T08:01:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cfdb9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf80c9ee0 0x7f3cf80ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cfd39b700 1 -- 172.21.15.23:0/3895320178 learned_addr learned my addr 172.21.15.23:0/3895320178 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:01:00.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cfd39b700 1 -- 172.21.15.23:0/3895320178 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf80c9ee0 msgr2=0x7f3cf80ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:01:00.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cfd39b700 1 --2- 172.21.15.23:0/3895320178 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf80c9ee0 0x7f3cf80ca340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cfd39b700 1 -- 172.21.15.23:0/3895320178 >> v1:172.21.15.67:6789/0 conn(0x7f3cf80c9260 legacy=0x7f3cf80c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3cfd39b700 1 -- 172.21.15.23:0/3895320178 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf806aed0 con 0x7f3cf8152ba0 2023-06-22T08:01:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.436+0000 7f3ceffff700 1 -- 172.21.15.23:0/3895320178 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194883882 0 0) 0x7f3ce8001110 con 0x7f3cf80c9260 2023-06-22T08:01:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.437+0000 7f3cfd39b700 1 --2- 172.21.15.23:0/3895320178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf8152ba0 0x7f3cf81593f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3cf40098f0 tx=0x7f3cf4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5137e3c113e9c65 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.437+0000 7f3ceffff700 1 -- 172.21.15.23:0/3895320178 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4011830 con 0x7f3cf8152ba0 2023-06-22T08:01:00.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.437+0000 7f3ceffff700 1 -- 172.21.15.23:0/3895320178 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf4011990 con 0x7f3cf8152ba0 2023-06-22T08:01:00.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.437+0000 7f3ceffff700 1 -- 172.21.15.23:0/3895320178 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4011d20 con 0x7f3cf8152ba0 2023-06-22T08:01:00.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.437+0000 7f3cff5ff700 1 -- 172.21.15.23:0/3895320178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf8152ba0 msgr2=0x7f3cf81593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.437+0000 7f3cff5ff700 1 --2- 172.21.15.23:0/3895320178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf8152ba0 0x7f3cf81593f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3cf40098f0 tx=0x7f3cf4009c00 comp rx=0 tx=0).stop 2023-06-22T08:01:00.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.438+0000 7f3cff5ff700 1 -- 172.21.15.23:0/3895320178 shutdown_connections 2023-06-22T08:01:00.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.438+0000 7f3cff5ff700 1 --2- 172.21.15.23:0/3895320178 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf80c9ee0 0x7f3cf80ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.438+0000 7f3cff5ff700 1 --2- 172.21.15.23:0/3895320178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf8152ba0 0x7f3cf81593f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.438+0000 7f3cff5ff700 1 -- 172.21.15.23:0/3895320178 >> 172.21.15.23:0/3895320178 conn(0x7f3cf8065b90 msgr2=0x7f3cf8065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:01:00.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.438+0000 7f3cff5ff700 1 -- 172.21.15.23:0/3895320178 shutdown_connections 2023-06-22T08:01:00.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.438+0000 7f3cff5ff700 1 -- 172.21.15.23:0/3895320178 wait complete. 2023-06-22T08:01:00.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.440+0000 7f3cff5ff700 1 Processor -- start 2023-06-22T08:01:00.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.440+0000 7f3cff5ff700 1 -- start start 2023-06-22T08:01:00.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.441+0000 7f3cff5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf80c9260 0x7f3cf8157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.441+0000 7f3cff5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf80c9ee0 0x7f3cf80c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.441+0000 7f3cff5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf8152ba0 0x7f3cf80cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cff5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf81f1400 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cff5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf80cf310 con 0x7f3cf80c9260 2023-06-22T08:01:00.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cff5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf80cf5f0 con 0x7f3cf8152ba0 2023-06-22T08:01:00.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfcb9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf80c9ee0 0x7f3cf80c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfd39b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf80c9260 0x7f3cf8157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfcb9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf80c9ee0 0x7f3cf80c5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52948/0 (socket says 172.21.15.23:52948) 2023-06-22T08:01:00.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfcb9a700 1 -- 172.21.15.23:0/2326240876 learned_addr learned my addr 172.21.15.23:0/2326240876 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:01:00.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfdb9c700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf8152ba0 0x7f3cf80cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfcb9a700 1 -- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf8152ba0 msgr2=0x7f3cf80cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfcb9a700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf8152ba0 0x7f3cf80cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfcb9a700 1 -- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf80c9260 msgr2=0x7f3cf8157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfcb9a700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf80c9260 0x7f3cf8157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfcb9a700 1 -- 172.21.15.23:0/2326240876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf4009550 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.442+0000 7f3cfcb9a700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf80c9ee0 0x7f3cf80c5c00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800c750 tx=0x7f3ce800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.443+0000 7f3cedffb700 1 -- 172.21.15.23:0/2326240876 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce80054e0 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.443+0000 7f3cedffb700 1 -- 172.21.15.23:0/2326240876 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce8003680 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.443+0000 7f3cedffb700 1 -- 172.21.15.23:0/2326240876 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8003a30 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.443+0000 7f3cff5ff700 1 -- 172.21.15.23:0/2326240876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf81f16c0 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.443+0000 7f3cff5ff700 1 -- 172.21.15.23:0/2326240876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf81f1c80 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.444+0000 7f3cff5ff700 1 -- 172.21.15.23:0/2326240876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf80cbed0 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.445+0000 7f3cedffb700 1 -- 172.21.15.23:0/2326240876 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3ce8003b90 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.446+0000 7f3cedffb700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cdc06ace0 0x7f3cdc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.446+0000 7f3cedffb700 1 -- 172.21.15.23:0/2326240876 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3ce8088ad0 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.446+0000 7f3cfd39b700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cdc06ace0 0x7f3cdc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.447+0000 7f3cfd39b700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cdc06ace0 0x7f3cdc06d190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4014040 tx=0x7f3cf4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:00.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.450+0000 7f3cedffb700 1 -- 172.21.15.23:0/2326240876 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3ce8057280 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.660+0000 7f3cff5ff700 1 -- 172.21.15.23:0/2326240876 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3cf8159930 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.661+0000 7f3cedffb700 1 -- 172.21.15.23:0/2326240876 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3ce8017020 con 0x7f3cf80c9ee0 2023-06-22T08:01:00.664 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-06-22T08:01:00.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 -- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cdc06ace0 msgr2=0x7f3cdc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cdc06ace0 0x7f3cdc06d190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4014040 tx=0x7f3cf4008030 comp rx=0 tx=0).stop 2023-06-22T08:01:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 -- 172.21.15.23:0/2326240876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf80c9ee0 msgr2=0x7f3cf80c5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf80c9ee0 0x7f3cf80c5c00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800c750 tx=0x7f3ce800ca60 comp rx=0 tx=0).stop 2023-06-22T08:01:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 -- 172.21.15.23:0/2326240876 shutdown_connections 2023-06-22T08:01:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf8152ba0 0x7f3cf80cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cdc06ace0 0x7f3cdc06d190 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf80c9260 0x7f3cf8157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 --2- 172.21.15.23:0/2326240876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf80c9ee0 0x7f3cf80c5c00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 -- 172.21.15.23:0/2326240876 >> 172.21.15.23:0/2326240876 conn(0x7f3cf8065b90 msgr2=0x7f3cf80c49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:01:00.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 -- 172.21.15.23:0/2326240876 shutdown_connections 2023-06-22T08:01:00.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.663+0000 7f3cff5ff700 1 -- 172.21.15.23:0/2326240876 wait complete. 2023-06-22T08:01:00.685 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-22T08:01:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.884+0000 7f1e3edff700 1 Processor -- start 2023-06-22T08:01:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.884+0000 7f1e3edff700 1 -- start start 2023-06-22T08:01:00.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.885+0000 7f1e3edff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1e381593f0 0x7f1e380c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.885+0000 7f1e3edff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e380c9aa0 con 0x7f1e38152ba0 2023-06-22T08:01:00.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.885+0000 7f1e3edff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3806aed0 con 0x7f1e380c4650 2023-06-22T08:01:00.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.885+0000 7f1e3edff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e380c4a60 con 0x7f1e381593f0 2023-06-22T08:01:00.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.886+0000 7f1e3cb9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1e38152ba0 0x7f1e380c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:45578/0 (socket says 172.21.15.23:45578) 2023-06-22T08:01:00.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.886+0000 7f1e3cb9b700 1 -- 172.21.15.23:0/2414197019 learned_addr learned my addr 172.21.15.23:0/2414197019 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:01:00.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.886+0000 7f1e37fff700 1 --2- 172.21.15.23:0/2414197019 >> v2:172.21.15.67:3300/0 conn(0x7f1e381593f0 0x7f1e380c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.886+0000 7f1e377fe700 1 -- 172.21.15.23:0/2414197019 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101138779 0 0) 0x7f1e380c9aa0 con 0x7f1e38152ba0 2023-06-22T08:01:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.886+0000 7f1e377fe700 1 -- 172.21.15.23:0/2414197019 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e280035b0 con 0x7f1e38152ba0 2023-06-22T08:01:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.886+0000 7f1e377fe700 1 -- 172.21.15.23:0/2414197019 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909746232 0 0) 0x7f1e3806aed0 con 0x7f1e380c4650 2023-06-22T08:01:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.886+0000 7f1e377fe700 1 -- 172.21.15.23:0/2414197019 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e380c9aa0 con 0x7f1e380c4650 2023-06-22T08:01:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.886+0000 7f1e37fff700 1 -- 172.21.15.23:0/2414197019 >> v1:172.21.15.67:6789/0 conn(0x7f1e380c4650 legacy=0x7f1e380c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.886+0000 7f1e37fff700 1 -- 172.21.15.23:0/2414197019 >> v1:172.21.15.23:6789/0 conn(0x7f1e38152ba0 legacy=0x7f1e380c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.886+0000 7f1e37fff700 1 -- 172.21.15.23:0/2414197019 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3806b1b0 con 0x7f1e381593f0 2023-06-22T08:01:00.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.887+0000 7f1e37fff700 1 --2- 172.21.15.23:0/2414197019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e381593f0 0x7f1e380c4110 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009840 tx=0x7f1e24009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f3d81f2a1475b5a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:00.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.887+0000 7f1e377fe700 1 -- 172.21.15.23:0/2414197019 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24011830 con 0x7f1e381593f0 2023-06-22T08:01:00.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.887+0000 7f1e377fe700 1 -- 172.21.15.23:0/2414197019 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e24011990 con 0x7f1e381593f0 2023-06-22T08:01:00.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.887+0000 7f1e377fe700 1 -- 172.21.15.23:0/2414197019 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e24011d20 con 0x7f1e381593f0 2023-06-22T08:01:00.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.887+0000 7f1e3edff700 1 -- 172.21.15.23:0/2414197019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e381593f0 msgr2=0x7f1e380c4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.887+0000 7f1e3edff700 1 --2- 172.21.15.23:0/2414197019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e381593f0 0x7f1e380c4110 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009840 tx=0x7f1e24009b50 comp rx=0 tx=0).stop 2023-06-22T08:01:00.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.887+0000 7f1e3edff700 1 -- 172.21.15.23:0/2414197019 shutdown_connections 2023-06-22T08:01:00.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.887+0000 7f1e3edff700 1 --2- 172.21.15.23:0/2414197019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e381593f0 0x7f1e380c4110 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.887+0000 7f1e3edff700 1 -- 172.21.15.23:0/2414197019 >> 172.21.15.23:0/2414197019 conn(0x7f1e38065b90 msgr2=0x7f1e38065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:01:00.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.887+0000 7f1e3edff700 1 -- 172.21.15.23:0/2414197019 shutdown_connections 2023-06-22T08:01:00.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.888+0000 7f1e3edff700 1 -- 172.21.15.23:0/2414197019 wait complete. 2023-06-22T08:01:00.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.889+0000 7f1e3edff700 1 Processor -- start 2023-06-22T08:01:00.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.889+0000 7f1e3edff700 1 -- start start 2023-06-22T08:01:00.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.890+0000 7f1e3edff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e380c4650 0x7f1e381f13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.890+0000 7f1e3edff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e38152ba0 0x7f1e381f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.890+0000 7f1e3edff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e381593f0 0x7f1e380c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.890+0000 7f1e3edff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380b3ec0 con 0x7f1e38152ba0 2023-06-22T08:01:00.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.890+0000 7f1e3edff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380cf730 con 0x7f1e381593f0 2023-06-22T08:01:00.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.890+0000 7f1e3edff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380b3990 con 0x7f1e380c4650 2023-06-22T08:01:00.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e37fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e38152ba0 0x7f1e381f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e37fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e38152ba0 0x7f1e381f1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52958/0 (socket says 172.21.15.23:52958) 2023-06-22T08:01:00.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e37fff700 1 -- 172.21.15.23:0/396684861 learned_addr learned my addr 172.21.15.23:0/396684861 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:01:00.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e3cb9b700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e380c4650 0x7f1e381f13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e3d39c700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e381593f0 0x7f1e380c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e37fff700 1 -- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e380c4650 msgr2=0x7f1e381f13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e37fff700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e380c4650 0x7f1e381f13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e37fff700 1 -- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e381593f0 msgr2=0x7f1e380c6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:00.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e37fff700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e381593f0 0x7f1e380c6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:00.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e37fff700 1 -- 172.21.15.23:0/396684861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e240094a0 con 0x7f1e38152ba0 2023-06-22T08:01:00.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e3d39c700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e381593f0 0x7f1e380c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:01:00.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e3cb9b700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e380c4650 0x7f1e381f13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:01:00.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.891+0000 7f1e37fff700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e38152ba0 0x7f1e381f1910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009fd0 tx=0x7f1e24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:00.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.892+0000 7f1e357fa700 1 -- 172.21.15.23:0/396684861 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2401aa70 con 0x7f1e38152ba0 2023-06-22T08:01:00.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.892+0000 7f1e357fa700 1 -- 172.21.15.23:0/396684861 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2401abd0 con 0x7f1e38152ba0 2023-06-22T08:01:00.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.892+0000 7f1e357fa700 1 -- 172.21.15.23:0/396684861 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e240243f0 con 0x7f1e38152ba0 2023-06-22T08:01:00.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.892+0000 7f1e3edff700 1 -- 172.21.15.23:0/396684861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e380c62b0 con 0x7f1e38152ba0 2023-06-22T08:01:00.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.892+0000 7f1e3edff700 1 -- 172.21.15.23:0/396684861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e380c5c60 con 0x7f1e38152ba0 2023-06-22T08:01:00.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.893+0000 7f1e3edff700 1 -- 172.21.15.23:0/396684861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e380cbed0 con 0x7f1e38152ba0 2023-06-22T08:01:00.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.894+0000 7f1e357fa700 1 -- 172.21.15.23:0/396684861 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1e24024550 con 0x7f1e38152ba0 2023-06-22T08:01:00.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.895+0000 7f1e357fa700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e1c06ace0 0x7f1e1c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:00.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.895+0000 7f1e357fa700 1 -- 172.21.15.23:0/396684861 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1e2408ab40 con 0x7f1e38152ba0 2023-06-22T08:01:00.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.895+0000 7f1e3cb9b700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e1c06ace0 0x7f1e1c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:00.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.896+0000 7f1e3cb9b700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e1c06ace0 0x7f1e1c06d190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c000f60 tx=0x7f1e2c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:00.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:00.899+0000 7f1e357fa700 1 -- 172.21.15.23:0/396684861 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1e24055ab0 con 0x7f1e38152ba0 2023-06-22T08:01:01.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.106+0000 7f1e3edff700 1 -- 172.21.15.23:0/396684861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1e380c5ac0 con 0x7f1e38152ba0 2023-06-22T08:01:01.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.107+0000 7f1e357fa700 1 -- 172.21.15.23:0/396684861 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1e380c5ac0 con 0x7f1e38152ba0 2023-06-22T08:01:01.110 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:01:01.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 -- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e1c06ace0 msgr2=0x7f1e1c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e1c06ace0 0x7f1e1c06d190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c000f60 tx=0x7f1e2c009040 comp rx=0 tx=0).stop 2023-06-22T08:01:01.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 -- 172.21.15.23:0/396684861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e38152ba0 msgr2=0x7f1e381f1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e38152ba0 0x7f1e381f1910 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009fd0 tx=0x7f1e24004750 comp rx=0 tx=0).stop 2023-06-22T08:01:01.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 -- 172.21.15.23:0/396684861 shutdown_connections 2023-06-22T08:01:01.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e1c06ace0 0x7f1e1c06d190 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e381593f0 0x7f1e380c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e380c4650 0x7f1e381f13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 --2- 172.21.15.23:0/396684861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e38152ba0 0x7f1e381f1910 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 -- 172.21.15.23:0/396684861 >> 172.21.15.23:0/396684861 conn(0x7f1e38065b90 msgr2=0x7f1e380c9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:01:01.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 -- 172.21.15.23:0/396684861 shutdown_connections 2023-06-22T08:01:01.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.109+0000 7f1e3edff700 1 -- 172.21.15.23:0/396684861 wait complete. 2023-06-22T08:01:01.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:01:01.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.327+0000 7f59f41ff700 1 Processor -- start 2023-06-22T08:01:01.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.327+0000 7f59f41ff700 1 -- start start 2023-06-22T08:01:01.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.328+0000 7f59f41ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f59ec152a10 0x7f59ec0c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.328+0000 7f59f41ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f59ec0c9200 0x7f59ec0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.328+0000 7f59f41ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ec159730 con 0x7f59ec0cb7a0 2023-06-22T08:01:01.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.328+0000 7f59f41ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59ec0c5470 con 0x7f59ec0c9200 2023-06-22T08:01:01.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.328+0000 7f59f41ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59ec158910 con 0x7f59ec152a10 2023-06-22T08:01:01.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.328+0000 7f59f1f9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f59ec152a10 0x7f59ec0c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.328+0000 7f59f279c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f59ec0cb7a0 0x7f59ec15c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:51490/0 (socket says 172.21.15.23:51490) 2023-06-22T08:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.328+0000 7f59f279c700 1 -- 172.21.15.23:0/2295446503 learned_addr learned my addr 172.21.15.23:0/2295446503 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.328+0000 7f59f179a700 1 --2- 172.21.15.23:0/2295446503 >> v2:172.21.15.23:3300/0 conn(0x7f59ec0c9200 0x7f59ec0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.329+0000 7f59f1f9b700 1 -- 172.21.15.23:0/2295446503 >> v2:172.21.15.23:3300/0 conn(0x7f59ec0c9200 msgr2=0x7f59ec0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.329+0000 7f59f0f99700 1 -- 172.21.15.23:0/2295446503 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366116876 0 0) 0x7f59ec159730 con 0x7f59ec0cb7a0 2023-06-22T08:01:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.329+0000 7f59f1f9b700 1 --2- 172.21.15.23:0/2295446503 >> v2:172.21.15.23:3300/0 conn(0x7f59ec0c9200 0x7f59ec0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.329+0000 7f59f1f9b700 1 -- 172.21.15.23:0/2295446503 >> v1:172.21.15.67:6789/0 conn(0x7f59ec0cb7a0 legacy=0x7f59ec15c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.329+0000 7f59f1f9b700 1 -- 172.21.15.23:0/2295446503 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec158bb0 con 0x7f59ec152a10 2023-06-22T08:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.329+0000 7f59f179a700 1 --2- 172.21.15.23:0/2295446503 >> v2:172.21.15.23:3300/0 conn(0x7f59ec0c9200 0x7f59ec0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.329+0000 7f59f1f9b700 1 --2- 172.21.15.23:0/2295446503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec152a10 0x7f59ec0c4a30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f59ec08b320 tx=0x7f59e8009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20a1bd4c32a0cab server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.329+0000 7f59f0f99700 1 -- 172.21.15.23:0/2295446503 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e8011830 con 0x7f59ec152a10 2023-06-22T08:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.329+0000 7f59f0f99700 1 -- 172.21.15.23:0/2295446503 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e8011990 con 0x7f59ec152a10 2023-06-22T08:01:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.330+0000 7f59f0f99700 1 -- 172.21.15.23:0/2295446503 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e8011d20 con 0x7f59ec152a10 2023-06-22T08:01:01.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.330+0000 7f59f41ff700 1 -- 172.21.15.23:0/2295446503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec152a10 msgr2=0x7f59ec0c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.330+0000 7f59f41ff700 1 --2- 172.21.15.23:0/2295446503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec152a10 0x7f59ec0c4a30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f59ec08b320 tx=0x7f59e8009b20 comp rx=0 tx=0).stop 2023-06-22T08:01:01.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.330+0000 7f59f41ff700 1 -- 172.21.15.23:0/2295446503 shutdown_connections 2023-06-22T08:01:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.330+0000 7f59f41ff700 1 --2- 172.21.15.23:0/2295446503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec152a10 0x7f59ec0c4a30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.330+0000 7f59f41ff700 1 --2- 172.21.15.23:0/2295446503 >> v2:172.21.15.23:3300/0 conn(0x7f59ec0c9200 0x7f59ec0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.330+0000 7f59f41ff700 1 -- 172.21.15.23:0/2295446503 >> 172.21.15.23:0/2295446503 conn(0x7f59ec065b90 msgr2=0x7f59ec065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:01:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.330+0000 7f59f41ff700 1 -- 172.21.15.23:0/2295446503 shutdown_connections 2023-06-22T08:01:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.330+0000 7f59f41ff700 1 -- 172.21.15.23:0/2295446503 wait complete. 2023-06-22T08:01:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.332+0000 7f59f41ff700 1 Processor -- start 2023-06-22T08:01:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.332+0000 7f59f41ff700 1 -- start start 2023-06-22T08:01:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f41ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0c9200 0x7f59ec1514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f41ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59ec0cb7a0 0x7f59ec0b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f41ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59ec152a10 0x7f59ec0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f41ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0b3270 con 0x7f59ec152a10 2023-06-22T08:01:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f41ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec151ea0 con 0x7f59ec0c9200 2023-06-22T08:01:01.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f41ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec1ef2c0 con 0x7f59ec0cb7a0 2023-06-22T08:01:01.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f1f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0c9200 0x7f59ec1514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f179a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59ec0cb7a0 0x7f59ec0b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f279c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59ec152a10 0x7f59ec0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f1f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0c9200 0x7f59ec1514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51778/0 (socket says 172.21.15.23:51778) 2023-06-22T08:01:01.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.333+0000 7f59f1f9b700 1 -- 172.21.15.23:0/3076126232 learned_addr learned my addr 172.21.15.23:0/3076126232 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:01:01.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59f1f9b700 1 -- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59ec0cb7a0 msgr2=0x7f59ec0b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59f1f9b700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59ec0cb7a0 0x7f59ec0b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59f1f9b700 1 -- 172.21.15.23:0/3076126232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59ec152a10 msgr2=0x7f59ec0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59f1f9b700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59ec152a10 0x7f59ec0b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59f1f9b700 1 -- 172.21.15.23:0/3076126232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59dc0094a0 con 0x7f59ec0c9200 2023-06-22T08:01:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59f1f9b700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0c9200 0x7f59ec1514f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f59e8000c00 tx=0x7f59e8004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59e27fc700 1 -- 172.21.15.23:0/3076126232 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e8004510 con 0x7f59ec0c9200 2023-06-22T08:01:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59e27fc700 1 -- 172.21.15.23:0/3076126232 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e801e070 con 0x7f59ec0c9200 2023-06-22T08:01:01.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59f41ff700 1 -- 172.21.15.23:0/3076126232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e80094a0 con 0x7f59ec0c9200 2023-06-22T08:01:01.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59e27fc700 1 -- 172.21.15.23:0/3076126232 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e80038e0 con 0x7f59ec0c9200 2023-06-22T08:01:01.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.334+0000 7f59f41ff700 1 -- 172.21.15.23:0/3076126232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ec0b2440 con 0x7f59ec0c9200 2023-06-22T08:01:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.335+0000 7f59f41ff700 1 -- 172.21.15.23:0/3076126232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ec0b35c0 con 0x7f59ec0c9200 2023-06-22T08:01:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.337+0000 7f59e27fc700 1 -- 172.21.15.23:0/3076126232 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f59e8003a40 con 0x7f59ec0c9200 2023-06-22T08:01:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.338+0000 7f59e27fc700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59d006ace0 0x7f59d006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.338+0000 7f59e27fc700 1 -- 172.21.15.23:0/3076126232 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f59e808a980 con 0x7f59ec0c9200 2023-06-22T08:01:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.338+0000 7f59f179a700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59d006ace0 0x7f59d006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.338+0000 7f59f179a700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59d006ace0 0x7f59d006d190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f59ec1ec340 tx=0x7f59dc009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.341+0000 7f59e27fc700 1 -- 172.21.15.23:0/3076126232 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f59e80557b0 con 0x7f59ec0c9200 2023-06-22T08:01:01.598 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.595+0000 7f59f41ff700 1 -- 172.21.15.23:0/3076126232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59ec06c360 con 0x7f59ec0c9200 2023-06-22T08:01:01.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.596+0000 7f59e27fc700 1 -- 172.21.15.23:0/3076126232 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+6243 (secure 0 0 0) 0x7f59e8020090 con 0x7f59ec0c9200 2023-06-22T08:01:01.599 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:01:01.599 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:00:41.586436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:01:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.598+0000 7f59f41ff700 1 -- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59d006ace0 msgr2=0x7f59d006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.598+0000 7f59f41ff700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59d006ace0 0x7f59d006d190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f59ec1ec340 tx=0x7f59dc009210 comp rx=0 tx=0).stop 2023-06-22T08:01:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.598+0000 7f59f41ff700 1 -- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0c9200 msgr2=0x7f59ec1514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.598+0000 7f59f41ff700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0c9200 0x7f59ec1514f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f59e8000c00 tx=0x7f59e8004750 comp rx=0 tx=0).stop 2023-06-22T08:01:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.599+0000 7f59f41ff700 1 -- 172.21.15.23:0/3076126232 shutdown_connections 2023-06-22T08:01:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.599+0000 7f59f41ff700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59ec152a10 0x7f59ec0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.599+0000 7f59f41ff700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59d006ace0 0x7f59d006d190 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.599+0000 7f59f41ff700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59ec0c9200 0x7f59ec1514f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.599+0000 7f59f41ff700 1 --2- 172.21.15.23:0/3076126232 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59ec0cb7a0 0x7f59ec0b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.599+0000 7f59f41ff700 1 -- 172.21.15.23:0/3076126232 >> 172.21.15.23:0/3076126232 conn(0x7f59ec065b90 msgr2=0x7f59ec065f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:01:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.599+0000 7f59f41ff700 1 -- 172.21.15.23:0/3076126232 shutdown_connections 2023-06-22T08:01:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.599+0000 7f59f41ff700 1 -- 172.21.15.23:0/3076126232 wait complete. 2023-06-22T08:01:01.603 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 19 2023-06-22T08:01:01.620 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:01:01.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.817+0000 7f7e367ff700 1 Processor -- start 2023-06-22T08:01:01.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.818+0000 7f7e367ff700 1 -- start start 2023-06-22T08:01:01.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.819+0000 7f7e367ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e30152d70 0x7f7e300c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.819+0000 7f7e367ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7e300c45f0 0x7f7e300c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.819+0000 7f7e367ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e300c4f90 con 0x7f7e30159330 2023-06-22T08:01:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.819+0000 7f7e367ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e300c95e0 con 0x7f7e30152d70 2023-06-22T08:01:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.819+0000 7f7e367ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e30159790 con 0x7f7e300c45f0 2023-06-22T08:01:01.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.820+0000 7f7e2ffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e30152d70 0x7f7e300c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.820+0000 7f7e34d9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7e300c45f0 0x7f7e300c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.820+0000 7f7e2ffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e30152d70 0x7f7e300c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51788/0 (socket says 172.21.15.23:51788) 2023-06-22T08:01:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.820+0000 7f7e2ffff700 1 -- 172.21.15.23:0/358109487 learned_addr learned my addr 172.21.15.23:0/358109487 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:01:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.820+0000 7f7e2ffff700 1 -- 172.21.15.23:0/358109487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7e300c45f0 msgr2=0x7f7e300c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:01:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.820+0000 7f7e2ffff700 1 --2- 172.21.15.23:0/358109487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7e300c45f0 0x7f7e300c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.820+0000 7f7e2ffff700 1 -- 172.21.15.23:0/358109487 >> v1:172.21.15.67:6789/0 conn(0x7f7e30159330 legacy=0x7f7e300cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.820+0000 7f7e2ffff700 1 -- 172.21.15.23:0/358109487 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3015bea0 con 0x7f7e30152d70 2023-06-22T08:01:01.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.820+0000 7f7e2ffff700 1 --2- 172.21.15.23:0/358109487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30152d70 0x7f7e300c97d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7e200098f0 tx=0x7f7e20009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47ca2d7ab5b82525 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:01.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.820+0000 7f7e2effd700 1 -- 172.21.15.23:0/358109487 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20011830 con 0x7f7e30152d70 2023-06-22T08:01:01.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.821+0000 7f7e2effd700 1 -- 172.21.15.23:0/358109487 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e20011990 con 0x7f7e30152d70 2023-06-22T08:01:01.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.821+0000 7f7e2effd700 1 -- 172.21.15.23:0/358109487 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20011d20 con 0x7f7e30152d70 2023-06-22T08:01:01.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.821+0000 7f7e367ff700 1 -- 172.21.15.23:0/358109487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30152d70 msgr2=0x7f7e300c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.821+0000 7f7e367ff700 1 --2- 172.21.15.23:0/358109487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30152d70 0x7f7e300c97d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7e200098f0 tx=0x7f7e20009c00 comp rx=0 tx=0).stop 2023-06-22T08:01:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.821+0000 7f7e367ff700 1 -- 172.21.15.23:0/358109487 shutdown_connections 2023-06-22T08:01:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.821+0000 7f7e367ff700 1 --2- 172.21.15.23:0/358109487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7e300c45f0 0x7f7e300c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.821+0000 7f7e367ff700 1 --2- 172.21.15.23:0/358109487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e30152d70 0x7f7e300c97d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.821+0000 7f7e367ff700 1 -- 172.21.15.23:0/358109487 >> 172.21.15.23:0/358109487 conn(0x7f7e3006ae30 msgr2=0x7f7e3006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:01:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.821+0000 7f7e367ff700 1 -- 172.21.15.23:0/358109487 shutdown_connections 2023-06-22T08:01:01.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.821+0000 7f7e367ff700 1 -- 172.21.15.23:0/358109487 wait complete. 2023-06-22T08:01:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.823+0000 7f7e367ff700 1 Processor -- start 2023-06-22T08:01:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.823+0000 7f7e367ff700 1 -- start start 2023-06-22T08:01:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e367ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300c45f0 0x7f7e301ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e367ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7e30152d70 0x7f7e301eeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e367ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e30159330 0x7f7e301e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e367ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e30159b70 con 0x7f7e30159330 2023-06-22T08:01:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e367ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3015b8f0 con 0x7f7e300c45f0 2023-06-22T08:01:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e367ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e300b2490 con 0x7f7e30152d70 2023-06-22T08:01:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e2ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300c45f0 0x7f7e301ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e2f7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7e30152d70 0x7f7e301eeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e2ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300c45f0 0x7f7e301ee360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51804/0 (socket says 172.21.15.23:51804) 2023-06-22T08:01:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e2ffff700 1 -- 172.21.15.23:0/2178718061 learned_addr learned my addr 172.21.15.23:0/2178718061 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:01:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.824+0000 7f7e34d9c700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e30159330 0x7f7e301e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e2ffff700 1 -- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7e30152d70 msgr2=0x7f7e301eeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e2ffff700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7e30152d70 0x7f7e301eeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e2ffff700 1 -- 172.21.15.23:0/2178718061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e30159330 msgr2=0x7f7e301e4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e2ffff700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e30159330 0x7f7e301e4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e2ffff700 1 -- 172.21.15.23:0/2178718061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e20009550 con 0x7f7e300c45f0 2023-06-22T08:01:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e2ffff700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300c45f0 0x7f7e301ee360 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7e20000c00 tx=0x7f7e20004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e2cff9700 1 -- 172.21.15.23:0/2178718061 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20004400 con 0x7f7e300c45f0 2023-06-22T08:01:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e2cff9700 1 -- 172.21.15.23:0/2178718061 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2001f070 con 0x7f7e300c45f0 2023-06-22T08:01:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e2cff9700 1 -- 172.21.15.23:0/2178718061 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e200038e0 con 0x7f7e300c45f0 2023-06-22T08:01:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e367ff700 1 -- 172.21.15.23:0/2178718061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e300b2750 con 0x7f7e300c45f0 2023-06-22T08:01:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.825+0000 7f7e367ff700 1 -- 172.21.15.23:0/2178718061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e300b2d10 con 0x7f7e300c45f0 2023-06-22T08:01:01.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.826+0000 7f7e367ff700 1 -- 172.21.15.23:0/2178718061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3008bf10 con 0x7f7e300c45f0 2023-06-22T08:01:01.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.828+0000 7f7e2cff9700 1 -- 172.21.15.23:0/2178718061 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7e20021030 con 0x7f7e300c45f0 2023-06-22T08:01:01.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.828+0000 7f7e2cff9700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7e1c06ad10 0x7f7e1c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:01:01.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.829+0000 7f7e2cff9700 1 -- 172.21.15.23:0/2178718061 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f7e2008abb0 con 0x7f7e300c45f0 2023-06-22T08:01:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.829+0000 7f7e2f7fe700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7e1c06ad10 0x7f7e1c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:01:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.829+0000 7f7e2f7fe700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7e1c06ad10 0x7f7e1c06d1c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7e30152160 tx=0x7f7e24009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:01:01.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:01.833+0000 7f7e2cff9700 1 -- 172.21.15.23:0/2178718061 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7e20055a00 con 0x7f7e300c45f0 2023-06-22T08:01:02.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.045+0000 7f7e367ff700 1 -- 172.21.15.23:0/2178718061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e3004ea20 con 0x7f7e300c45f0 2023-06-22T08:01:02.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.046+0000 7f7e2cff9700 1 -- 172.21.15.23:0/2178718061 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+20146 (secure 0 0 0) 0x7f7e20015090 con 0x7f7e300c45f0 2023-06-22T08:01:02.050 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:01:02.050 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T07:59:59.700075+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:01:02.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 -- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7e1c06ad10 msgr2=0x7f7e1c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:02.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7e1c06ad10 0x7f7e1c06d1c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7e30152160 tx=0x7f7e24009040 comp rx=0 tx=0).stop 2023-06-22T08:01:02.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 -- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300c45f0 msgr2=0x7f7e301ee360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:01:02.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300c45f0 0x7f7e301ee360 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7e20000c00 tx=0x7f7e20004750 comp rx=0 tx=0).stop 2023-06-22T08:01:02.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 -- 172.21.15.23:0/2178718061 shutdown_connections 2023-06-22T08:01:02.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e30159330 0x7f7e301e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:02.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7e1c06ad10 0x7f7e1c06d1c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:02.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e300c45f0 0x7f7e301ee360 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:02.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 --2- 172.21.15.23:0/2178718061 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7e30152d70 0x7f7e301eeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:01:02.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 -- 172.21.15.23:0/2178718061 >> 172.21.15.23:0/2178718061 conn(0x7f7e3006ae30 msgr2=0x7f7e300ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:01:02.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 -- 172.21.15.23:0/2178718061 shutdown_connections 2023-06-22T08:01:02.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:01:02.049+0000 7f7e367ff700 1 -- 172.21.15.23:0/2178718061 wait complete. 2023-06-22T08:01:02.071 DEBUG:tasks.mds_thrash:Yielding 2023-06-22T08:01:02.071 INFO:teuthology.run_tasks:Running task workunit... 2023-06-22T08:01:02.086 INFO:tasks.workunit:Pulling workunits from ref 8627af3e0adcb765a3c249fcc209cba9f4873e1b 2023-06-22T08:01:02.087 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-06-22T08:01:02.088 INFO:tasks.workunit:timeout=6h 2023-06-22T08:01:02.088 INFO:tasks.workunit:cleanup=True 2023-06-22T08:01:02.089 DEBUG:teuthology.orchestra.run.smithi023:> stat -- /home/ubuntu/cephtest/mnt.0 2023-06-22T08:01:02.090 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-06-22T08:01:02.090 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 108 secs before thrashing 2023-06-22T08:01:02.133 INFO:teuthology.orchestra.run.smithi023.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-06-22T08:01:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-06-22T08:01:02.134 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 2bh/43d Inode: 1 Links: 2 2023-06-22T08:01:02.134 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T08:01:02.134 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:cephfs_t:s0 2023-06-22T08:01:02.134 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-06-22T08:01:02.134 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2023-06-22 08:00:00.143350021 +0000 2023-06-22T08:01:02.135 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2023-06-22 08:00:11.150396545 +0000 2023-06-22T08:01:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2023-06-22 08:00:00.143350021 +0000 2023-06-22T08:01:02.135 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-06-22T08:01:02.136 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-06-22T08:01:02.250 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8627af3e0adcb765a3c249fcc209cba9f4873e1b 2023-06-22T08:01:02.296 INFO:tasks.workunit.client.0.smithi023.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-06-22T08:01:05.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:01:05.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:01:36.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:01:36.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:02:01.097 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 62% (7339/11710) Updating files: 63% (7378/11710) Updating files: 64% (7495/11710) Updating files: 65% (7612/11710) Updating files: 66% (7729/11710) Updating files: 67% (7846/11710) Updating files: 68% (7963/11710) Updating files: 69% (8080/11710) Updating files: 70% (8197/11710) Updating files: 71% (8315/11710) Updating files: 72% (8432/11710) Updating files: 73% (8549/11710) Updating files: 74% (8666/11710) Updating files: 75% (8783/11710) Updating files: 76% (8900/11710) Updating files: 77% (9017/11710) Updating files: 78% (9134/11710) Updating files: 79% (9251/11710) Updating files: 80% (9368/11710) Updating files: 81% (9486/11710) Updating files: 82% (9603/11710) Updating files: 83% (9720/11710) Updating files: 84% (9837/11710) Updating files: 85% (9954/11710) Updating files: 86% (10071/11710) Updating files: 87% (10188/11710) Updating files: 88% (10305/11710) Updating files: 89% (10422/11710) Updating files: 90% (10539/11710) Updating files: 91% (10657/11710) Updating files: 92% (10774/11710) Updating files: 93% (10891/11710) Updating files: 94% (11008/11710) Updating files: 95% (11125/11710) Updating files: 96% (11242/11710) Updating files: 97% (11359/11710) Updating files: 98% (11476/11710) Updating files: 99% (11593/11710) Updating files: 99% (11701/11710) Updating files: 100% (11710/11710) Updating files: 100% (11710/11710), done. 2023-06-22T08:02:01.660 INFO:tasks.workunit.client.0.smithi023.stderr:Note: switching to '8627af3e0adcb765a3c249fcc209cba9f4873e1b'. 2023-06-22T08:02:01.660 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-06-22T08:02:01.660 INFO:tasks.workunit.client.0.smithi023.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-06-22T08:02:01.661 INFO:tasks.workunit.client.0.smithi023.stderr:changes and commit them, and you can discard any commits you make in this 2023-06-22T08:02:01.661 INFO:tasks.workunit.client.0.smithi023.stderr:state without impacting any branches by switching back to a branch. 2023-06-22T08:02:01.661 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-06-22T08:02:01.661 INFO:tasks.workunit.client.0.smithi023.stderr:If you want to create a new branch to retain commits you create, you may 2023-06-22T08:02:01.661 INFO:tasks.workunit.client.0.smithi023.stderr:do so (now or later) by using -c with the switch command. Example: 2023-06-22T08:02:01.661 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-06-22T08:02:01.662 INFO:tasks.workunit.client.0.smithi023.stderr: git switch -c 2023-06-22T08:02:01.662 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-06-22T08:02:01.662 INFO:tasks.workunit.client.0.smithi023.stderr:Or undo this operation with: 2023-06-22T08:02:01.662 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-06-22T08:02:01.662 INFO:tasks.workunit.client.0.smithi023.stderr: git switch - 2023-06-22T08:02:01.662 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-06-22T08:02:01.662 INFO:tasks.workunit.client.0.smithi023.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-06-22T08:02:01.662 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-06-22T08:02:01.663 INFO:tasks.workunit.client.0.smithi023.stderr:HEAD is now at 8627af3e0ad AuthMonitor: refactor valid_caps() 2023-06-22T08:02:01.675 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-06-22T08:02:01.729 INFO:tasks.workunit.client.0.smithi023.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-06-22T08:02:01.734 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-06-22T08:02:01.734 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-06-22T08:02:02.276 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-06-22T08:02:02.358 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-06-22T08:02:02.421 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-06-22T08:02:02.425 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-06-22T08:02:02.426 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-06-22T08:02:02.489 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-06-22T08:02:02.496 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T08:02:02.497 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-06-22T08:02:02.539 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-06-22T08:02:02.541 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-06-22T08:02:02.541 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-06-22T08:02:02.637 INFO:tasks.workunit.client.0.smithi023.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-06-22T08:02:05.001 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir foo 2023-06-22T08:02:05.010 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-06-22T08:02:05.116 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir foo/.snap/barsnap 2023-06-22T08:02:05.128 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-06-22T08:02:05.146 INFO:tasks.workunit.client.0.smithi023.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-06-22T08:02:05.440 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'passed: files are identical' 2023-06-22T08:02:05.440 INFO:tasks.workunit.client.0.smithi023.stdout:passed: files are identical 2023-06-22T08:02:05.441 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir foo/.snap/barsnap 2023-06-22T08:02:05.461 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:02:05.462 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:02:05.462 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:02:05.462 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:02:05.546 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-06-22T08:02:05.546 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-06-22T08:02:05.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 1 2023-06-22T08:02:05.735 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 2 2023-06-22T08:02:05.736 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-06-22T08:02:05.736 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' -e file4 ']' 2023-06-22T08:02:05.736 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/snap1 2023-06-22T08:02:05.755 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 4 2023-06-22T08:02:05.758 INFO:tasks.workunit.client.0.smithi023.stderr:++ ls 2023-06-22T08:02:05.770 INFO:tasks.workunit.client.0.smithi023.stderr:+ now='file1 2023-06-22T08:02:05.771 INFO:tasks.workunit.client.0.smithi023.stderr:file2 2023-06-22T08:02:05.771 INFO:tasks.workunit.client.0.smithi023.stderr:file3 2023-06-22T08:02:05.771 INFO:tasks.workunit.client.0.smithi023.stderr:file4' 2023-06-22T08:02:05.771 INFO:tasks.workunit.client.0.smithi023.stderr:++ ls .snap/snap1 2023-06-22T08:02:05.787 INFO:tasks.workunit.client.0.smithi023.stderr:+ then='file1 2023-06-22T08:02:05.787 INFO:tasks.workunit.client.0.smithi023.stderr:file2 2023-06-22T08:02:05.787 INFO:tasks.workunit.client.0.smithi023.stderr:file3' 2023-06-22T08:02:05.787 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/snap1 2023-06-22T08:02:05.815 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 'file1 2023-06-22T08:02:05.816 INFO:tasks.workunit.client.0.smithi023.stderr:file2 2023-06-22T08:02:05.816 INFO:tasks.workunit.client.0.smithi023.stderr:file3 2023-06-22T08:02:05.816 INFO:tasks.workunit.client.0.smithi023.stderr:file4' = 'file1 2023-06-22T08:02:05.816 INFO:tasks.workunit.client.0.smithi023.stderr:file2 2023-06-22T08:02:05.816 INFO:tasks.workunit.client.0.smithi023.stderr:file3' ']' 2023-06-22T08:02:05.817 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 1 2023-06-22T08:02:05.824 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 2 2023-06-22T08:02:05.831 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-06-22T08:02:05.839 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/snap1 2023-06-22T08:02:05.847 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 4 2023-06-22T08:02:05.854 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/snap1 2023-06-22T08:02:05.861 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm file1 file2 file3 file4 2023-06-22T08:02:05.869 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:02:05.870 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:02:05.870 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:02:05.870 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:02:05.948 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-06-22T08:02:05.948 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-06-22T08:02:06.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:02:06.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:02:08.371 INFO:tasks.workunit.client.0.smithi023.stdout:Create dir 100 to 199 ... 2023-06-22T08:02:08.381 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 100 2023-06-22T08:02:08.390 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 101 2023-06-22T08:02:08.403 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 102 2023-06-22T08:02:08.416 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 103 2023-06-22T08:02:08.429 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 104 2023-06-22T08:02:08.441 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 105 2023-06-22T08:02:08.453 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 106 2023-06-22T08:02:08.464 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 107 2023-06-22T08:02:08.476 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 108 2023-06-22T08:02:08.488 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 109 2023-06-22T08:02:08.499 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 110 2023-06-22T08:02:08.510 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 111 2023-06-22T08:02:08.522 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 112 2023-06-22T08:02:08.533 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 113 2023-06-22T08:02:08.545 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 114 2023-06-22T08:02:08.555 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 115 2023-06-22T08:02:08.566 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 116 2023-06-22T08:02:08.577 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 117 2023-06-22T08:02:08.588 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 118 2023-06-22T08:02:08.599 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 119 2023-06-22T08:02:08.610 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 120 2023-06-22T08:02:08.621 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 121 2023-06-22T08:02:08.631 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 122 2023-06-22T08:02:08.642 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 123 2023-06-22T08:02:08.653 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 124 2023-06-22T08:02:08.664 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 125 2023-06-22T08:02:08.675 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 126 2023-06-22T08:02:08.685 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 127 2023-06-22T08:02:08.696 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 128 2023-06-22T08:02:08.707 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 129 2023-06-22T08:02:08.718 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 130 2023-06-22T08:02:08.729 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 131 2023-06-22T08:02:08.739 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 132 2023-06-22T08:02:08.750 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 133 2023-06-22T08:02:08.761 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 134 2023-06-22T08:02:08.772 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 135 2023-06-22T08:02:08.783 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 136 2023-06-22T08:02:08.794 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 137 2023-06-22T08:02:08.805 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 138 2023-06-22T08:02:08.816 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 139 2023-06-22T08:02:08.826 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 140 2023-06-22T08:02:08.837 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 141 2023-06-22T08:02:08.847 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 142 2023-06-22T08:02:08.858 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 143 2023-06-22T08:02:08.869 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 144 2023-06-22T08:02:08.885 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 145 2023-06-22T08:02:08.891 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 146 2023-06-22T08:02:08.902 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 147 2023-06-22T08:02:08.914 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 148 2023-06-22T08:02:08.925 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 149 2023-06-22T08:02:08.935 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 150 2023-06-22T08:02:08.946 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 151 2023-06-22T08:02:08.958 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 152 2023-06-22T08:02:08.969 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 153 2023-06-22T08:02:08.980 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 154 2023-06-22T08:02:08.990 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 155 2023-06-22T08:02:09.002 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 156 2023-06-22T08:02:09.013 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 157 2023-06-22T08:02:09.024 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 158 2023-06-22T08:02:09.035 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 159 2023-06-22T08:02:09.046 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 160 2023-06-22T08:02:09.057 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 161 2023-06-22T08:02:09.068 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 162 2023-06-22T08:02:09.078 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 163 2023-06-22T08:02:09.089 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 164 2023-06-22T08:02:09.100 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 165 2023-06-22T08:02:09.111 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 166 2023-06-22T08:02:09.122 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 167 2023-06-22T08:02:09.132 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 168 2023-06-22T08:02:09.142 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 169 2023-06-22T08:02:09.153 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 170 2023-06-22T08:02:09.163 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 171 2023-06-22T08:02:09.174 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 172 2023-06-22T08:02:09.185 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 173 2023-06-22T08:02:09.195 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 174 2023-06-22T08:02:09.206 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 175 2023-06-22T08:02:09.217 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 176 2023-06-22T08:02:09.228 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 177 2023-06-22T08:02:09.238 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 178 2023-06-22T08:02:09.249 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 179 2023-06-22T08:02:09.259 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 180 2023-06-22T08:02:09.271 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 181 2023-06-22T08:02:09.282 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 182 2023-06-22T08:02:09.295 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 183 2023-06-22T08:02:09.306 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 184 2023-06-22T08:02:09.317 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 185 2023-06-22T08:02:09.327 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 186 2023-06-22T08:02:09.338 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 187 2023-06-22T08:02:09.349 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 188 2023-06-22T08:02:09.361 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 189 2023-06-22T08:02:09.380 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 190 2023-06-22T08:02:09.395 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 191 2023-06-22T08:02:09.410 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 192 2023-06-22T08:02:09.425 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 193 2023-06-22T08:02:09.441 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 194 2023-06-22T08:02:09.459 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 195 2023-06-22T08:02:09.475 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 196 2023-06-22T08:02:09.492 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 197 2023-06-22T08:02:09.508 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 198 2023-06-22T08:02:09.523 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 199 2023-06-22T08:02:09.539 INFO:tasks.workunit.client.0.smithi023.stdout:Take first snapshot .snap/test1 2023-06-22T08:02:09.999 INFO:tasks.workunit.client.0.smithi023.stdout:Create dir 200 to 299 ... 2023-06-22T08:02:10.001 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 200 2023-06-22T08:02:10.009 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 201 2023-06-22T08:02:10.019 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 202 2023-06-22T08:02:10.030 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 203 2023-06-22T08:02:10.040 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 204 2023-06-22T08:02:10.050 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 205 2023-06-22T08:02:10.060 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 206 2023-06-22T08:02:10.071 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 207 2023-06-22T08:02:10.081 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 208 2023-06-22T08:02:10.092 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 209 2023-06-22T08:02:10.101 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 210 2023-06-22T08:02:10.112 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 211 2023-06-22T08:02:10.123 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 212 2023-06-22T08:02:10.134 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 213 2023-06-22T08:02:10.145 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 214 2023-06-22T08:02:10.156 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 215 2023-06-22T08:02:10.166 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 216 2023-06-22T08:02:10.177 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 217 2023-06-22T08:02:10.187 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 218 2023-06-22T08:02:10.198 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 219 2023-06-22T08:02:10.209 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 220 2023-06-22T08:02:10.219 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 221 2023-06-22T08:02:10.230 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 222 2023-06-22T08:02:10.240 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 223 2023-06-22T08:02:10.251 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 224 2023-06-22T08:02:10.262 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 225 2023-06-22T08:02:10.273 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 226 2023-06-22T08:02:10.283 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 227 2023-06-22T08:02:10.294 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 228 2023-06-22T08:02:10.305 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 229 2023-06-22T08:02:10.316 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 230 2023-06-22T08:02:10.327 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 231 2023-06-22T08:02:10.341 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 232 2023-06-22T08:02:10.356 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 233 2023-06-22T08:02:10.366 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 234 2023-06-22T08:02:10.377 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 235 2023-06-22T08:02:10.387 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 236 2023-06-22T08:02:10.398 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 237 2023-06-22T08:02:10.409 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 238 2023-06-22T08:02:10.419 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 239 2023-06-22T08:02:10.430 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 240 2023-06-22T08:02:10.441 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 241 2023-06-22T08:02:10.451 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 242 2023-06-22T08:02:10.463 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 243 2023-06-22T08:02:10.473 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 244 2023-06-22T08:02:10.484 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 245 2023-06-22T08:02:10.495 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 246 2023-06-22T08:02:10.506 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 247 2023-06-22T08:02:10.517 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 248 2023-06-22T08:02:10.527 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 249 2023-06-22T08:02:10.538 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 250 2023-06-22T08:02:10.569 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 251 2023-06-22T08:02:10.855 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 252 2023-06-22T08:02:10.869 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 253 2023-06-22T08:02:10.879 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 254 2023-06-22T08:02:10.890 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 255 2023-06-22T08:02:10.900 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 256 2023-06-22T08:02:10.910 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 257 2023-06-22T08:02:10.920 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 258 2023-06-22T08:02:10.931 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 259 2023-06-22T08:02:10.941 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 260 2023-06-22T08:02:10.950 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 261 2023-06-22T08:02:10.960 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 262 2023-06-22T08:02:10.971 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 263 2023-06-22T08:02:10.982 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 264 2023-06-22T08:02:10.993 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 265 2023-06-22T08:02:11.002 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 266 2023-06-22T08:02:11.013 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 267 2023-06-22T08:02:11.023 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 268 2023-06-22T08:02:11.034 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 269 2023-06-22T08:02:11.045 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 270 2023-06-22T08:02:11.056 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 271 2023-06-22T08:02:11.065 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 272 2023-06-22T08:02:11.075 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 273 2023-06-22T08:02:11.086 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 274 2023-06-22T08:02:11.097 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 275 2023-06-22T08:02:11.107 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 276 2023-06-22T08:02:11.117 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 277 2023-06-22T08:02:11.128 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 278 2023-06-22T08:02:11.138 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 279 2023-06-22T08:02:11.149 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 280 2023-06-22T08:02:11.159 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 281 2023-06-22T08:02:11.171 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 282 2023-06-22T08:02:11.186 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 283 2023-06-22T08:02:11.204 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 284 2023-06-22T08:02:11.218 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 285 2023-06-22T08:02:11.233 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 286 2023-06-22T08:02:11.248 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 287 2023-06-22T08:02:11.263 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 288 2023-06-22T08:02:11.279 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 289 2023-06-22T08:02:11.292 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 290 2023-06-22T08:02:11.307 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 291 2023-06-22T08:02:11.322 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 292 2023-06-22T08:02:11.338 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 293 2023-06-22T08:02:11.353 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 294 2023-06-22T08:02:11.367 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 295 2023-06-22T08:02:11.381 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 296 2023-06-22T08:02:11.396 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 297 2023-06-22T08:02:11.412 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 298 2023-06-22T08:02:11.426 INFO:tasks.workunit.client.0.smithi023.stdout: create dir 299 2023-06-22T08:02:11.440 INFO:tasks.workunit.client.0.smithi023.stdout:Create a snapshot in every first level dir ... 2023-06-22T08:02:11.580 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/.snap/snap-subdir-test 2023-06-22T08:02:11.602 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/30 file after the snapshot 2023-06-22T08:02:11.604 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/31 file after the snapshot 2023-06-22T08:02:11.604 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/32 file after the snapshot 2023-06-22T08:02:11.604 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/33 file after the snapshot 2023-06-22T08:02:11.604 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/34 file after the snapshot 2023-06-22T08:02:11.605 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/35 file after the snapshot 2023-06-22T08:02:11.605 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/36 file after the snapshot 2023-06-22T08:02:11.605 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/37 file after the snapshot 2023-06-22T08:02:11.605 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/38 file after the snapshot 2023-06-22T08:02:11.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 100/39 file after the snapshot 2023-06-22T08:02:11.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/.snap/snap-subdir-test 2023-06-22T08:02:11.637 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/30 file after the snapshot 2023-06-22T08:02:11.639 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/31 file after the snapshot 2023-06-22T08:02:11.639 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/32 file after the snapshot 2023-06-22T08:02:11.640 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/33 file after the snapshot 2023-06-22T08:02:11.640 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/34 file after the snapshot 2023-06-22T08:02:11.640 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/35 file after the snapshot 2023-06-22T08:02:11.641 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/36 file after the snapshot 2023-06-22T08:02:11.641 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/37 file after the snapshot 2023-06-22T08:02:11.641 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/38 file after the snapshot 2023-06-22T08:02:11.641 INFO:tasks.workunit.client.0.smithi023.stdout: create 101/39 file after the snapshot 2023-06-22T08:02:11.642 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/.snap/snap-subdir-test 2023-06-22T08:02:11.671 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/30 file after the snapshot 2023-06-22T08:02:11.672 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/31 file after the snapshot 2023-06-22T08:02:11.673 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/32 file after the snapshot 2023-06-22T08:02:11.673 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/33 file after the snapshot 2023-06-22T08:02:11.673 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/34 file after the snapshot 2023-06-22T08:02:11.673 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/35 file after the snapshot 2023-06-22T08:02:11.673 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/36 file after the snapshot 2023-06-22T08:02:11.673 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/37 file after the snapshot 2023-06-22T08:02:11.674 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/38 file after the snapshot 2023-06-22T08:02:11.674 INFO:tasks.workunit.client.0.smithi023.stdout: create 102/39 file after the snapshot 2023-06-22T08:02:11.674 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/.snap/snap-subdir-test 2023-06-22T08:02:11.706 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/30 file after the snapshot 2023-06-22T08:02:11.708 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/31 file after the snapshot 2023-06-22T08:02:11.708 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/32 file after the snapshot 2023-06-22T08:02:11.709 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/33 file after the snapshot 2023-06-22T08:02:11.709 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/34 file after the snapshot 2023-06-22T08:02:11.709 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/35 file after the snapshot 2023-06-22T08:02:11.709 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/36 file after the snapshot 2023-06-22T08:02:11.709 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/37 file after the snapshot 2023-06-22T08:02:11.710 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/38 file after the snapshot 2023-06-22T08:02:11.710 INFO:tasks.workunit.client.0.smithi023.stdout: create 103/39 file after the snapshot 2023-06-22T08:02:11.710 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/.snap/snap-subdir-test 2023-06-22T08:02:11.740 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/30 file after the snapshot 2023-06-22T08:02:11.742 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/31 file after the snapshot 2023-06-22T08:02:11.742 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/32 file after the snapshot 2023-06-22T08:02:11.743 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/33 file after the snapshot 2023-06-22T08:02:11.743 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/34 file after the snapshot 2023-06-22T08:02:11.743 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/35 file after the snapshot 2023-06-22T08:02:11.743 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/36 file after the snapshot 2023-06-22T08:02:11.743 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/37 file after the snapshot 2023-06-22T08:02:11.743 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/38 file after the snapshot 2023-06-22T08:02:11.743 INFO:tasks.workunit.client.0.smithi023.stdout: create 104/39 file after the snapshot 2023-06-22T08:02:11.743 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/.snap/snap-subdir-test 2023-06-22T08:02:11.779 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/30 file after the snapshot 2023-06-22T08:02:11.781 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/31 file after the snapshot 2023-06-22T08:02:11.781 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/32 file after the snapshot 2023-06-22T08:02:11.781 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/33 file after the snapshot 2023-06-22T08:02:11.782 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/34 file after the snapshot 2023-06-22T08:02:11.782 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/35 file after the snapshot 2023-06-22T08:02:11.782 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/36 file after the snapshot 2023-06-22T08:02:11.782 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/37 file after the snapshot 2023-06-22T08:02:11.783 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/38 file after the snapshot 2023-06-22T08:02:11.783 INFO:tasks.workunit.client.0.smithi023.stdout: create 105/39 file after the snapshot 2023-06-22T08:02:11.783 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/.snap/snap-subdir-test 2023-06-22T08:02:11.814 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/30 file after the snapshot 2023-06-22T08:02:11.815 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/31 file after the snapshot 2023-06-22T08:02:11.816 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/32 file after the snapshot 2023-06-22T08:02:11.816 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/33 file after the snapshot 2023-06-22T08:02:11.816 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/34 file after the snapshot 2023-06-22T08:02:11.817 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/35 file after the snapshot 2023-06-22T08:02:11.817 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/36 file after the snapshot 2023-06-22T08:02:11.817 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/37 file after the snapshot 2023-06-22T08:02:11.817 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/38 file after the snapshot 2023-06-22T08:02:11.817 INFO:tasks.workunit.client.0.smithi023.stdout: create 106/39 file after the snapshot 2023-06-22T08:02:11.818 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/.snap/snap-subdir-test 2023-06-22T08:02:11.851 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/30 file after the snapshot 2023-06-22T08:02:11.852 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/31 file after the snapshot 2023-06-22T08:02:11.853 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/32 file after the snapshot 2023-06-22T08:02:11.853 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/33 file after the snapshot 2023-06-22T08:02:11.853 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/34 file after the snapshot 2023-06-22T08:02:11.853 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/35 file after the snapshot 2023-06-22T08:02:11.853 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/36 file after the snapshot 2023-06-22T08:02:11.854 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/37 file after the snapshot 2023-06-22T08:02:11.854 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/38 file after the snapshot 2023-06-22T08:02:11.854 INFO:tasks.workunit.client.0.smithi023.stdout: create 107/39 file after the snapshot 2023-06-22T08:02:11.854 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/.snap/snap-subdir-test 2023-06-22T08:02:11.886 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/30 file after the snapshot 2023-06-22T08:02:11.888 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/31 file after the snapshot 2023-06-22T08:02:11.888 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/32 file after the snapshot 2023-06-22T08:02:11.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/33 file after the snapshot 2023-06-22T08:02:11.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/34 file after the snapshot 2023-06-22T08:02:11.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/35 file after the snapshot 2023-06-22T08:02:11.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/36 file after the snapshot 2023-06-22T08:02:11.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/37 file after the snapshot 2023-06-22T08:02:11.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/38 file after the snapshot 2023-06-22T08:02:11.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 108/39 file after the snapshot 2023-06-22T08:02:11.890 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/.snap/snap-subdir-test 2023-06-22T08:02:11.923 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/30 file after the snapshot 2023-06-22T08:02:11.924 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/31 file after the snapshot 2023-06-22T08:02:11.924 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/32 file after the snapshot 2023-06-22T08:02:11.925 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/33 file after the snapshot 2023-06-22T08:02:11.925 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/34 file after the snapshot 2023-06-22T08:02:11.925 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/35 file after the snapshot 2023-06-22T08:02:11.925 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/36 file after the snapshot 2023-06-22T08:02:11.925 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/37 file after the snapshot 2023-06-22T08:02:11.925 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/38 file after the snapshot 2023-06-22T08:02:11.925 INFO:tasks.workunit.client.0.smithi023.stdout: create 109/39 file after the snapshot 2023-06-22T08:02:11.926 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/.snap/snap-subdir-test 2023-06-22T08:02:11.957 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/30 file after the snapshot 2023-06-22T08:02:11.959 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/31 file after the snapshot 2023-06-22T08:02:11.959 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/32 file after the snapshot 2023-06-22T08:02:11.959 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/33 file after the snapshot 2023-06-22T08:02:11.959 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/34 file after the snapshot 2023-06-22T08:02:11.959 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/35 file after the snapshot 2023-06-22T08:02:11.960 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/36 file after the snapshot 2023-06-22T08:02:11.960 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/37 file after the snapshot 2023-06-22T08:02:11.960 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/38 file after the snapshot 2023-06-22T08:02:11.960 INFO:tasks.workunit.client.0.smithi023.stdout: create 110/39 file after the snapshot 2023-06-22T08:02:11.960 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/.snap/snap-subdir-test 2023-06-22T08:02:11.990 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/30 file after the snapshot 2023-06-22T08:02:11.991 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/31 file after the snapshot 2023-06-22T08:02:11.992 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/32 file after the snapshot 2023-06-22T08:02:11.992 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/33 file after the snapshot 2023-06-22T08:02:11.993 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/34 file after the snapshot 2023-06-22T08:02:11.993 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/35 file after the snapshot 2023-06-22T08:02:11.993 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/36 file after the snapshot 2023-06-22T08:02:11.993 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/37 file after the snapshot 2023-06-22T08:02:11.993 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/38 file after the snapshot 2023-06-22T08:02:11.993 INFO:tasks.workunit.client.0.smithi023.stdout: create 111/39 file after the snapshot 2023-06-22T08:02:11.993 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/.snap/snap-subdir-test 2023-06-22T08:02:12.029 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/30 file after the snapshot 2023-06-22T08:02:12.031 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/31 file after the snapshot 2023-06-22T08:02:12.031 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/32 file after the snapshot 2023-06-22T08:02:12.031 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/33 file after the snapshot 2023-06-22T08:02:12.031 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/34 file after the snapshot 2023-06-22T08:02:12.031 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/35 file after the snapshot 2023-06-22T08:02:12.032 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/36 file after the snapshot 2023-06-22T08:02:12.032 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/37 file after the snapshot 2023-06-22T08:02:12.032 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/38 file after the snapshot 2023-06-22T08:02:12.032 INFO:tasks.workunit.client.0.smithi023.stdout: create 112/39 file after the snapshot 2023-06-22T08:02:12.032 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/.snap/snap-subdir-test 2023-06-22T08:02:12.062 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/30 file after the snapshot 2023-06-22T08:02:12.063 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/31 file after the snapshot 2023-06-22T08:02:12.063 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/32 file after the snapshot 2023-06-22T08:02:12.063 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/33 file after the snapshot 2023-06-22T08:02:12.064 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/34 file after the snapshot 2023-06-22T08:02:12.064 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/35 file after the snapshot 2023-06-22T08:02:12.065 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/36 file after the snapshot 2023-06-22T08:02:12.065 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/37 file after the snapshot 2023-06-22T08:02:12.065 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/38 file after the snapshot 2023-06-22T08:02:12.065 INFO:tasks.workunit.client.0.smithi023.stdout: create 113/39 file after the snapshot 2023-06-22T08:02:12.065 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/.snap/snap-subdir-test 2023-06-22T08:02:12.095 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/30 file after the snapshot 2023-06-22T08:02:12.096 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/31 file after the snapshot 2023-06-22T08:02:12.097 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/32 file after the snapshot 2023-06-22T08:02:12.097 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/33 file after the snapshot 2023-06-22T08:02:12.097 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/34 file after the snapshot 2023-06-22T08:02:12.097 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/35 file after the snapshot 2023-06-22T08:02:12.097 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/36 file after the snapshot 2023-06-22T08:02:12.097 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/37 file after the snapshot 2023-06-22T08:02:12.098 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/38 file after the snapshot 2023-06-22T08:02:12.098 INFO:tasks.workunit.client.0.smithi023.stdout: create 114/39 file after the snapshot 2023-06-22T08:02:12.098 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/.snap/snap-subdir-test 2023-06-22T08:02:12.131 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/30 file after the snapshot 2023-06-22T08:02:12.132 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/31 file after the snapshot 2023-06-22T08:02:12.133 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/32 file after the snapshot 2023-06-22T08:02:12.133 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/33 file after the snapshot 2023-06-22T08:02:12.133 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/34 file after the snapshot 2023-06-22T08:02:12.133 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/35 file after the snapshot 2023-06-22T08:02:12.133 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/36 file after the snapshot 2023-06-22T08:02:12.133 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/37 file after the snapshot 2023-06-22T08:02:12.134 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/38 file after the snapshot 2023-06-22T08:02:12.134 INFO:tasks.workunit.client.0.smithi023.stdout: create 115/39 file after the snapshot 2023-06-22T08:02:12.134 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/.snap/snap-subdir-test 2023-06-22T08:02:12.164 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/30 file after the snapshot 2023-06-22T08:02:12.166 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/31 file after the snapshot 2023-06-22T08:02:12.166 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/32 file after the snapshot 2023-06-22T08:02:12.166 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/33 file after the snapshot 2023-06-22T08:02:12.166 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/34 file after the snapshot 2023-06-22T08:02:12.166 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/35 file after the snapshot 2023-06-22T08:02:12.166 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/36 file after the snapshot 2023-06-22T08:02:12.167 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/37 file after the snapshot 2023-06-22T08:02:12.167 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/38 file after the snapshot 2023-06-22T08:02:12.167 INFO:tasks.workunit.client.0.smithi023.stdout: create 116/39 file after the snapshot 2023-06-22T08:02:12.167 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/.snap/snap-subdir-test 2023-06-22T08:02:12.198 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/30 file after the snapshot 2023-06-22T08:02:12.199 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/31 file after the snapshot 2023-06-22T08:02:12.199 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/32 file after the snapshot 2023-06-22T08:02:12.200 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/33 file after the snapshot 2023-06-22T08:02:12.200 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/34 file after the snapshot 2023-06-22T08:02:12.200 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/35 file after the snapshot 2023-06-22T08:02:12.200 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/36 file after the snapshot 2023-06-22T08:02:12.200 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/37 file after the snapshot 2023-06-22T08:02:12.201 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/38 file after the snapshot 2023-06-22T08:02:12.201 INFO:tasks.workunit.client.0.smithi023.stdout: create 117/39 file after the snapshot 2023-06-22T08:02:12.201 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/.snap/snap-subdir-test 2023-06-22T08:02:12.231 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/30 file after the snapshot 2023-06-22T08:02:12.232 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/31 file after the snapshot 2023-06-22T08:02:12.232 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/32 file after the snapshot 2023-06-22T08:02:12.233 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/33 file after the snapshot 2023-06-22T08:02:12.233 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/34 file after the snapshot 2023-06-22T08:02:12.234 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/35 file after the snapshot 2023-06-22T08:02:12.234 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/36 file after the snapshot 2023-06-22T08:02:12.234 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/37 file after the snapshot 2023-06-22T08:02:12.234 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/38 file after the snapshot 2023-06-22T08:02:12.234 INFO:tasks.workunit.client.0.smithi023.stdout: create 118/39 file after the snapshot 2023-06-22T08:02:12.234 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/.snap/snap-subdir-test 2023-06-22T08:02:12.267 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/30 file after the snapshot 2023-06-22T08:02:12.269 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/31 file after the snapshot 2023-06-22T08:02:12.269 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/32 file after the snapshot 2023-06-22T08:02:12.269 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/33 file after the snapshot 2023-06-22T08:02:12.269 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/34 file after the snapshot 2023-06-22T08:02:12.270 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/35 file after the snapshot 2023-06-22T08:02:12.270 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/36 file after the snapshot 2023-06-22T08:02:12.270 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/37 file after the snapshot 2023-06-22T08:02:12.270 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/38 file after the snapshot 2023-06-22T08:02:12.270 INFO:tasks.workunit.client.0.smithi023.stdout: create 119/39 file after the snapshot 2023-06-22T08:02:12.270 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/.snap/snap-subdir-test 2023-06-22T08:02:12.303 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/30 file after the snapshot 2023-06-22T08:02:12.305 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/31 file after the snapshot 2023-06-22T08:02:12.305 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/32 file after the snapshot 2023-06-22T08:02:12.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/33 file after the snapshot 2023-06-22T08:02:12.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/34 file after the snapshot 2023-06-22T08:02:12.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/35 file after the snapshot 2023-06-22T08:02:12.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/36 file after the snapshot 2023-06-22T08:02:12.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/37 file after the snapshot 2023-06-22T08:02:12.307 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/38 file after the snapshot 2023-06-22T08:02:12.307 INFO:tasks.workunit.client.0.smithi023.stdout: create 120/39 file after the snapshot 2023-06-22T08:02:12.307 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/.snap/snap-subdir-test 2023-06-22T08:02:12.339 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/30 file after the snapshot 2023-06-22T08:02:12.342 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/31 file after the snapshot 2023-06-22T08:02:12.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/32 file after the snapshot 2023-06-22T08:02:12.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/33 file after the snapshot 2023-06-22T08:02:12.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/34 file after the snapshot 2023-06-22T08:02:12.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/35 file after the snapshot 2023-06-22T08:02:12.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/36 file after the snapshot 2023-06-22T08:02:12.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/37 file after the snapshot 2023-06-22T08:02:12.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/38 file after the snapshot 2023-06-22T08:02:12.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 121/39 file after the snapshot 2023-06-22T08:02:12.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/.snap/snap-subdir-test 2023-06-22T08:02:12.378 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/30 file after the snapshot 2023-06-22T08:02:12.379 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/31 file after the snapshot 2023-06-22T08:02:12.379 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/32 file after the snapshot 2023-06-22T08:02:12.379 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/33 file after the snapshot 2023-06-22T08:02:12.379 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/34 file after the snapshot 2023-06-22T08:02:12.380 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/35 file after the snapshot 2023-06-22T08:02:12.380 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/36 file after the snapshot 2023-06-22T08:02:12.380 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/37 file after the snapshot 2023-06-22T08:02:12.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/38 file after the snapshot 2023-06-22T08:02:12.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 122/39 file after the snapshot 2023-06-22T08:02:12.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/.snap/snap-subdir-test 2023-06-22T08:02:12.411 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/30 file after the snapshot 2023-06-22T08:02:12.413 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/31 file after the snapshot 2023-06-22T08:02:12.413 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/32 file after the snapshot 2023-06-22T08:02:12.413 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/33 file after the snapshot 2023-06-22T08:02:12.413 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/34 file after the snapshot 2023-06-22T08:02:12.413 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/35 file after the snapshot 2023-06-22T08:02:12.413 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/36 file after the snapshot 2023-06-22T08:02:12.414 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/37 file after the snapshot 2023-06-22T08:02:12.414 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/38 file after the snapshot 2023-06-22T08:02:12.414 INFO:tasks.workunit.client.0.smithi023.stdout: create 123/39 file after the snapshot 2023-06-22T08:02:12.415 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/.snap/snap-subdir-test 2023-06-22T08:02:12.451 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/30 file after the snapshot 2023-06-22T08:02:12.454 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/31 file after the snapshot 2023-06-22T08:02:12.454 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/32 file after the snapshot 2023-06-22T08:02:12.454 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/33 file after the snapshot 2023-06-22T08:02:12.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/34 file after the snapshot 2023-06-22T08:02:12.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/35 file after the snapshot 2023-06-22T08:02:12.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/36 file after the snapshot 2023-06-22T08:02:12.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/37 file after the snapshot 2023-06-22T08:02:12.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/38 file after the snapshot 2023-06-22T08:02:12.456 INFO:tasks.workunit.client.0.smithi023.stdout: create 124/39 file after the snapshot 2023-06-22T08:02:12.456 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/.snap/snap-subdir-test 2023-06-22T08:02:12.491 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/30 file after the snapshot 2023-06-22T08:02:12.493 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/31 file after the snapshot 2023-06-22T08:02:12.493 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/32 file after the snapshot 2023-06-22T08:02:12.493 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/33 file after the snapshot 2023-06-22T08:02:12.493 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/34 file after the snapshot 2023-06-22T08:02:12.494 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/35 file after the snapshot 2023-06-22T08:02:12.494 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/36 file after the snapshot 2023-06-22T08:02:12.494 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/37 file after the snapshot 2023-06-22T08:02:12.494 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/38 file after the snapshot 2023-06-22T08:02:12.494 INFO:tasks.workunit.client.0.smithi023.stdout: create 125/39 file after the snapshot 2023-06-22T08:02:12.494 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/.snap/snap-subdir-test 2023-06-22T08:02:12.524 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/30 file after the snapshot 2023-06-22T08:02:12.525 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/31 file after the snapshot 2023-06-22T08:02:12.525 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/32 file after the snapshot 2023-06-22T08:02:12.525 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/33 file after the snapshot 2023-06-22T08:02:12.525 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/34 file after the snapshot 2023-06-22T08:02:12.526 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/35 file after the snapshot 2023-06-22T08:02:12.526 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/36 file after the snapshot 2023-06-22T08:02:12.526 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/37 file after the snapshot 2023-06-22T08:02:12.527 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/38 file after the snapshot 2023-06-22T08:02:12.527 INFO:tasks.workunit.client.0.smithi023.stdout: create 126/39 file after the snapshot 2023-06-22T08:02:12.527 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/.snap/snap-subdir-test 2023-06-22T08:02:12.572 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/30 file after the snapshot 2023-06-22T08:02:12.573 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/31 file after the snapshot 2023-06-22T08:02:12.574 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/32 file after the snapshot 2023-06-22T08:02:12.574 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/33 file after the snapshot 2023-06-22T08:02:12.574 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/34 file after the snapshot 2023-06-22T08:02:12.574 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/35 file after the snapshot 2023-06-22T08:02:12.574 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/36 file after the snapshot 2023-06-22T08:02:12.574 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/37 file after the snapshot 2023-06-22T08:02:12.575 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/38 file after the snapshot 2023-06-22T08:02:12.575 INFO:tasks.workunit.client.0.smithi023.stdout: create 127/39 file after the snapshot 2023-06-22T08:02:12.575 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/.snap/snap-subdir-test 2023-06-22T08:02:12.604 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/30 file after the snapshot 2023-06-22T08:02:12.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/31 file after the snapshot 2023-06-22T08:02:12.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/32 file after the snapshot 2023-06-22T08:02:12.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/33 file after the snapshot 2023-06-22T08:02:12.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/34 file after the snapshot 2023-06-22T08:02:12.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/35 file after the snapshot 2023-06-22T08:02:12.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/36 file after the snapshot 2023-06-22T08:02:12.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/37 file after the snapshot 2023-06-22T08:02:12.607 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/38 file after the snapshot 2023-06-22T08:02:12.607 INFO:tasks.workunit.client.0.smithi023.stdout: create 128/39 file after the snapshot 2023-06-22T08:02:12.607 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/.snap/snap-subdir-test 2023-06-22T08:02:12.642 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/30 file after the snapshot 2023-06-22T08:02:12.643 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/31 file after the snapshot 2023-06-22T08:02:12.643 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/32 file after the snapshot 2023-06-22T08:02:12.643 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/33 file after the snapshot 2023-06-22T08:02:12.644 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/34 file after the snapshot 2023-06-22T08:02:12.644 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/35 file after the snapshot 2023-06-22T08:02:12.644 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/36 file after the snapshot 2023-06-22T08:02:12.644 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/37 file after the snapshot 2023-06-22T08:02:12.644 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/38 file after the snapshot 2023-06-22T08:02:12.645 INFO:tasks.workunit.client.0.smithi023.stdout: create 129/39 file after the snapshot 2023-06-22T08:02:12.645 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/.snap/snap-subdir-test 2023-06-22T08:02:12.675 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/30 file after the snapshot 2023-06-22T08:02:12.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/31 file after the snapshot 2023-06-22T08:02:12.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/32 file after the snapshot 2023-06-22T08:02:12.677 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/33 file after the snapshot 2023-06-22T08:02:12.677 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/34 file after the snapshot 2023-06-22T08:02:12.677 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/35 file after the snapshot 2023-06-22T08:02:12.677 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/36 file after the snapshot 2023-06-22T08:02:12.677 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/37 file after the snapshot 2023-06-22T08:02:12.677 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/38 file after the snapshot 2023-06-22T08:02:12.677 INFO:tasks.workunit.client.0.smithi023.stdout: create 130/39 file after the snapshot 2023-06-22T08:02:12.678 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/.snap/snap-subdir-test 2023-06-22T08:02:12.715 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/30 file after the snapshot 2023-06-22T08:02:12.716 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/31 file after the snapshot 2023-06-22T08:02:12.716 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/32 file after the snapshot 2023-06-22T08:02:12.717 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/33 file after the snapshot 2023-06-22T08:02:12.717 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/34 file after the snapshot 2023-06-22T08:02:12.717 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/35 file after the snapshot 2023-06-22T08:02:12.717 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/36 file after the snapshot 2023-06-22T08:02:12.717 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/37 file after the snapshot 2023-06-22T08:02:12.717 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/38 file after the snapshot 2023-06-22T08:02:12.718 INFO:tasks.workunit.client.0.smithi023.stdout: create 131/39 file after the snapshot 2023-06-22T08:02:12.718 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/.snap/snap-subdir-test 2023-06-22T08:02:12.747 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/30 file after the snapshot 2023-06-22T08:02:12.748 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/31 file after the snapshot 2023-06-22T08:02:12.748 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/32 file after the snapshot 2023-06-22T08:02:12.748 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/33 file after the snapshot 2023-06-22T08:02:12.749 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/34 file after the snapshot 2023-06-22T08:02:12.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/35 file after the snapshot 2023-06-22T08:02:12.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/36 file after the snapshot 2023-06-22T08:02:12.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/37 file after the snapshot 2023-06-22T08:02:12.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/38 file after the snapshot 2023-06-22T08:02:12.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 132/39 file after the snapshot 2023-06-22T08:02:12.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/.snap/snap-subdir-test 2023-06-22T08:02:12.780 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/30 file after the snapshot 2023-06-22T08:02:12.782 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/31 file after the snapshot 2023-06-22T08:02:12.782 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/32 file after the snapshot 2023-06-22T08:02:12.782 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/33 file after the snapshot 2023-06-22T08:02:12.782 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/34 file after the snapshot 2023-06-22T08:02:12.782 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/35 file after the snapshot 2023-06-22T08:02:12.782 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/36 file after the snapshot 2023-06-22T08:02:12.783 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/37 file after the snapshot 2023-06-22T08:02:12.783 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/38 file after the snapshot 2023-06-22T08:02:12.783 INFO:tasks.workunit.client.0.smithi023.stdout: create 133/39 file after the snapshot 2023-06-22T08:02:12.783 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/.snap/snap-subdir-test 2023-06-22T08:02:12.813 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/30 file after the snapshot 2023-06-22T08:02:12.815 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/31 file after the snapshot 2023-06-22T08:02:12.816 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/32 file after the snapshot 2023-06-22T08:02:12.816 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/33 file after the snapshot 2023-06-22T08:02:12.816 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/34 file after the snapshot 2023-06-22T08:02:12.816 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/35 file after the snapshot 2023-06-22T08:02:12.816 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/36 file after the snapshot 2023-06-22T08:02:12.816 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/37 file after the snapshot 2023-06-22T08:02:12.817 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/38 file after the snapshot 2023-06-22T08:02:12.817 INFO:tasks.workunit.client.0.smithi023.stdout: create 134/39 file after the snapshot 2023-06-22T08:02:12.817 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/.snap/snap-subdir-test 2023-06-22T08:02:12.847 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/30 file after the snapshot 2023-06-22T08:02:12.848 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/31 file after the snapshot 2023-06-22T08:02:12.849 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/32 file after the snapshot 2023-06-22T08:02:12.849 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/33 file after the snapshot 2023-06-22T08:02:12.849 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/34 file after the snapshot 2023-06-22T08:02:12.849 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/35 file after the snapshot 2023-06-22T08:02:12.849 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/36 file after the snapshot 2023-06-22T08:02:12.850 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/37 file after the snapshot 2023-06-22T08:02:12.850 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/38 file after the snapshot 2023-06-22T08:02:12.850 INFO:tasks.workunit.client.0.smithi023.stdout: create 135/39 file after the snapshot 2023-06-22T08:02:12.850 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/.snap/snap-subdir-test 2023-06-22T08:02:12.879 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/30 file after the snapshot 2023-06-22T08:02:12.881 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/31 file after the snapshot 2023-06-22T08:02:12.881 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/32 file after the snapshot 2023-06-22T08:02:12.881 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/33 file after the snapshot 2023-06-22T08:02:12.882 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/34 file after the snapshot 2023-06-22T08:02:12.882 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/35 file after the snapshot 2023-06-22T08:02:12.882 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/36 file after the snapshot 2023-06-22T08:02:12.882 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/37 file after the snapshot 2023-06-22T08:02:12.882 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/38 file after the snapshot 2023-06-22T08:02:12.882 INFO:tasks.workunit.client.0.smithi023.stdout: create 136/39 file after the snapshot 2023-06-22T08:02:12.883 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/.snap/snap-subdir-test 2023-06-22T08:02:12.912 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/30 file after the snapshot 2023-06-22T08:02:12.914 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/31 file after the snapshot 2023-06-22T08:02:12.914 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/32 file after the snapshot 2023-06-22T08:02:12.914 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/33 file after the snapshot 2023-06-22T08:02:12.914 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/34 file after the snapshot 2023-06-22T08:02:12.914 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/35 file after the snapshot 2023-06-22T08:02:12.915 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/36 file after the snapshot 2023-06-22T08:02:12.915 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/37 file after the snapshot 2023-06-22T08:02:12.915 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/38 file after the snapshot 2023-06-22T08:02:12.915 INFO:tasks.workunit.client.0.smithi023.stdout: create 137/39 file after the snapshot 2023-06-22T08:02:12.916 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/.snap/snap-subdir-test 2023-06-22T08:02:12.948 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/30 file after the snapshot 2023-06-22T08:02:12.949 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/31 file after the snapshot 2023-06-22T08:02:12.949 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/32 file after the snapshot 2023-06-22T08:02:12.950 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/33 file after the snapshot 2023-06-22T08:02:12.950 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/34 file after the snapshot 2023-06-22T08:02:12.950 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/35 file after the snapshot 2023-06-22T08:02:12.951 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/36 file after the snapshot 2023-06-22T08:02:12.952 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/37 file after the snapshot 2023-06-22T08:02:12.952 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/38 file after the snapshot 2023-06-22T08:02:12.952 INFO:tasks.workunit.client.0.smithi023.stdout: create 138/39 file after the snapshot 2023-06-22T08:02:12.952 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/.snap/snap-subdir-test 2023-06-22T08:02:12.980 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/30 file after the snapshot 2023-06-22T08:02:12.981 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/31 file after the snapshot 2023-06-22T08:02:12.982 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/32 file after the snapshot 2023-06-22T08:02:12.982 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/33 file after the snapshot 2023-06-22T08:02:12.982 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/34 file after the snapshot 2023-06-22T08:02:12.983 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/35 file after the snapshot 2023-06-22T08:02:12.983 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/36 file after the snapshot 2023-06-22T08:02:12.983 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/37 file after the snapshot 2023-06-22T08:02:12.983 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/38 file after the snapshot 2023-06-22T08:02:12.984 INFO:tasks.workunit.client.0.smithi023.stdout: create 139/39 file after the snapshot 2023-06-22T08:02:12.984 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/.snap/snap-subdir-test 2023-06-22T08:02:13.013 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/30 file after the snapshot 2023-06-22T08:02:13.014 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/31 file after the snapshot 2023-06-22T08:02:13.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/32 file after the snapshot 2023-06-22T08:02:13.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/33 file after the snapshot 2023-06-22T08:02:13.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/34 file after the snapshot 2023-06-22T08:02:13.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/35 file after the snapshot 2023-06-22T08:02:13.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/36 file after the snapshot 2023-06-22T08:02:13.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/37 file after the snapshot 2023-06-22T08:02:13.016 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/38 file after the snapshot 2023-06-22T08:02:13.016 INFO:tasks.workunit.client.0.smithi023.stdout: create 140/39 file after the snapshot 2023-06-22T08:02:13.016 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/.snap/snap-subdir-test 2023-06-22T08:02:13.058 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/30 file after the snapshot 2023-06-22T08:02:13.060 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/31 file after the snapshot 2023-06-22T08:02:13.060 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/32 file after the snapshot 2023-06-22T08:02:13.060 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/33 file after the snapshot 2023-06-22T08:02:13.060 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/34 file after the snapshot 2023-06-22T08:02:13.060 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/35 file after the snapshot 2023-06-22T08:02:13.061 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/36 file after the snapshot 2023-06-22T08:02:13.061 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/37 file after the snapshot 2023-06-22T08:02:13.061 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/38 file after the snapshot 2023-06-22T08:02:13.061 INFO:tasks.workunit.client.0.smithi023.stdout: create 141/39 file after the snapshot 2023-06-22T08:02:13.061 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/.snap/snap-subdir-test 2023-06-22T08:02:13.100 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/30 file after the snapshot 2023-06-22T08:02:13.102 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/31 file after the snapshot 2023-06-22T08:02:13.102 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/32 file after the snapshot 2023-06-22T08:02:13.103 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/33 file after the snapshot 2023-06-22T08:02:13.103 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/34 file after the snapshot 2023-06-22T08:02:13.103 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/35 file after the snapshot 2023-06-22T08:02:13.103 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/36 file after the snapshot 2023-06-22T08:02:13.103 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/37 file after the snapshot 2023-06-22T08:02:13.103 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/38 file after the snapshot 2023-06-22T08:02:13.104 INFO:tasks.workunit.client.0.smithi023.stdout: create 142/39 file after the snapshot 2023-06-22T08:02:13.104 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/.snap/snap-subdir-test 2023-06-22T08:02:13.133 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/30 file after the snapshot 2023-06-22T08:02:13.135 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/31 file after the snapshot 2023-06-22T08:02:13.136 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/32 file after the snapshot 2023-06-22T08:02:13.136 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/33 file after the snapshot 2023-06-22T08:02:13.136 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/34 file after the snapshot 2023-06-22T08:02:13.136 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/35 file after the snapshot 2023-06-22T08:02:13.136 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/36 file after the snapshot 2023-06-22T08:02:13.137 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/37 file after the snapshot 2023-06-22T08:02:13.137 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/38 file after the snapshot 2023-06-22T08:02:13.137 INFO:tasks.workunit.client.0.smithi023.stdout: create 143/39 file after the snapshot 2023-06-22T08:02:13.137 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/.snap/snap-subdir-test 2023-06-22T08:02:13.167 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/30 file after the snapshot 2023-06-22T08:02:13.169 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/31 file after the snapshot 2023-06-22T08:02:13.169 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/32 file after the snapshot 2023-06-22T08:02:13.169 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/33 file after the snapshot 2023-06-22T08:02:13.170 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/34 file after the snapshot 2023-06-22T08:02:13.170 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/35 file after the snapshot 2023-06-22T08:02:13.170 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/36 file after the snapshot 2023-06-22T08:02:13.170 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/37 file after the snapshot 2023-06-22T08:02:13.170 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/38 file after the snapshot 2023-06-22T08:02:13.170 INFO:tasks.workunit.client.0.smithi023.stdout: create 144/39 file after the snapshot 2023-06-22T08:02:13.170 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/.snap/snap-subdir-test 2023-06-22T08:02:13.201 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/30 file after the snapshot 2023-06-22T08:02:13.202 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/31 file after the snapshot 2023-06-22T08:02:13.202 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/32 file after the snapshot 2023-06-22T08:02:13.203 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/33 file after the snapshot 2023-06-22T08:02:13.204 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/34 file after the snapshot 2023-06-22T08:02:13.204 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/35 file after the snapshot 2023-06-22T08:02:13.204 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/36 file after the snapshot 2023-06-22T08:02:13.204 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/37 file after the snapshot 2023-06-22T08:02:13.204 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/38 file after the snapshot 2023-06-22T08:02:13.204 INFO:tasks.workunit.client.0.smithi023.stdout: create 145/39 file after the snapshot 2023-06-22T08:02:13.205 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/.snap/snap-subdir-test 2023-06-22T08:02:13.236 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/30 file after the snapshot 2023-06-22T08:02:13.238 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/31 file after the snapshot 2023-06-22T08:02:13.238 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/32 file after the snapshot 2023-06-22T08:02:13.239 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/33 file after the snapshot 2023-06-22T08:02:13.239 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/34 file after the snapshot 2023-06-22T08:02:13.239 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/35 file after the snapshot 2023-06-22T08:02:13.239 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/36 file after the snapshot 2023-06-22T08:02:13.239 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/37 file after the snapshot 2023-06-22T08:02:13.239 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/38 file after the snapshot 2023-06-22T08:02:13.240 INFO:tasks.workunit.client.0.smithi023.stdout: create 146/39 file after the snapshot 2023-06-22T08:02:13.240 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/.snap/snap-subdir-test 2023-06-22T08:02:13.269 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/30 file after the snapshot 2023-06-22T08:02:13.271 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/31 file after the snapshot 2023-06-22T08:02:13.271 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/32 file after the snapshot 2023-06-22T08:02:13.271 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/33 file after the snapshot 2023-06-22T08:02:13.271 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/34 file after the snapshot 2023-06-22T08:02:13.271 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/35 file after the snapshot 2023-06-22T08:02:13.271 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/36 file after the snapshot 2023-06-22T08:02:13.272 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/37 file after the snapshot 2023-06-22T08:02:13.272 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/38 file after the snapshot 2023-06-22T08:02:13.272 INFO:tasks.workunit.client.0.smithi023.stdout: create 147/39 file after the snapshot 2023-06-22T08:02:13.272 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/.snap/snap-subdir-test 2023-06-22T08:02:13.303 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/30 file after the snapshot 2023-06-22T08:02:13.304 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/31 file after the snapshot 2023-06-22T08:02:13.304 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/32 file after the snapshot 2023-06-22T08:02:13.304 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/33 file after the snapshot 2023-06-22T08:02:13.305 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/34 file after the snapshot 2023-06-22T08:02:13.305 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/35 file after the snapshot 2023-06-22T08:02:13.305 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/36 file after the snapshot 2023-06-22T08:02:13.305 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/37 file after the snapshot 2023-06-22T08:02:13.305 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/38 file after the snapshot 2023-06-22T08:02:13.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 148/39 file after the snapshot 2023-06-22T08:02:13.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/.snap/snap-subdir-test 2023-06-22T08:02:13.334 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/30 file after the snapshot 2023-06-22T08:02:13.338 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/31 file after the snapshot 2023-06-22T08:02:13.339 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/32 file after the snapshot 2023-06-22T08:02:13.339 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/33 file after the snapshot 2023-06-22T08:02:13.339 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/34 file after the snapshot 2023-06-22T08:02:13.339 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/35 file after the snapshot 2023-06-22T08:02:13.339 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/36 file after the snapshot 2023-06-22T08:02:13.340 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/37 file after the snapshot 2023-06-22T08:02:13.340 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/38 file after the snapshot 2023-06-22T08:02:13.340 INFO:tasks.workunit.client.0.smithi023.stdout: create 149/39 file after the snapshot 2023-06-22T08:02:13.340 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/.snap/snap-subdir-test 2023-06-22T08:02:13.374 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/30 file after the snapshot 2023-06-22T08:02:13.375 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/31 file after the snapshot 2023-06-22T08:02:13.376 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/32 file after the snapshot 2023-06-22T08:02:13.376 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/33 file after the snapshot 2023-06-22T08:02:13.376 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/34 file after the snapshot 2023-06-22T08:02:13.376 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/35 file after the snapshot 2023-06-22T08:02:13.376 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/36 file after the snapshot 2023-06-22T08:02:13.377 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/37 file after the snapshot 2023-06-22T08:02:13.377 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/38 file after the snapshot 2023-06-22T08:02:13.377 INFO:tasks.workunit.client.0.smithi023.stdout: create 150/39 file after the snapshot 2023-06-22T08:02:13.377 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/.snap/snap-subdir-test 2023-06-22T08:02:13.407 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/30 file after the snapshot 2023-06-22T08:02:13.408 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/31 file after the snapshot 2023-06-22T08:02:13.409 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/32 file after the snapshot 2023-06-22T08:02:13.409 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/33 file after the snapshot 2023-06-22T08:02:13.409 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/34 file after the snapshot 2023-06-22T08:02:13.409 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/35 file after the snapshot 2023-06-22T08:02:13.409 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/36 file after the snapshot 2023-06-22T08:02:13.409 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/37 file after the snapshot 2023-06-22T08:02:13.410 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/38 file after the snapshot 2023-06-22T08:02:13.410 INFO:tasks.workunit.client.0.smithi023.stdout: create 151/39 file after the snapshot 2023-06-22T08:02:13.410 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/.snap/snap-subdir-test 2023-06-22T08:02:13.440 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/30 file after the snapshot 2023-06-22T08:02:13.442 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/31 file after the snapshot 2023-06-22T08:02:13.442 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/32 file after the snapshot 2023-06-22T08:02:13.442 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/33 file after the snapshot 2023-06-22T08:02:13.443 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/34 file after the snapshot 2023-06-22T08:02:13.443 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/35 file after the snapshot 2023-06-22T08:02:13.443 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/36 file after the snapshot 2023-06-22T08:02:13.443 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/37 file after the snapshot 2023-06-22T08:02:13.443 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/38 file after the snapshot 2023-06-22T08:02:13.443 INFO:tasks.workunit.client.0.smithi023.stdout: create 152/39 file after the snapshot 2023-06-22T08:02:13.444 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/.snap/snap-subdir-test 2023-06-22T08:02:13.476 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/30 file after the snapshot 2023-06-22T08:02:13.479 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/31 file after the snapshot 2023-06-22T08:02:13.479 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/32 file after the snapshot 2023-06-22T08:02:13.479 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/33 file after the snapshot 2023-06-22T08:02:13.479 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/34 file after the snapshot 2023-06-22T08:02:13.479 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/35 file after the snapshot 2023-06-22T08:02:13.479 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/36 file after the snapshot 2023-06-22T08:02:13.480 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/37 file after the snapshot 2023-06-22T08:02:13.480 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/38 file after the snapshot 2023-06-22T08:02:13.480 INFO:tasks.workunit.client.0.smithi023.stdout: create 153/39 file after the snapshot 2023-06-22T08:02:13.480 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/.snap/snap-subdir-test 2023-06-22T08:02:13.509 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/30 file after the snapshot 2023-06-22T08:02:13.511 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/31 file after the snapshot 2023-06-22T08:02:13.511 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/32 file after the snapshot 2023-06-22T08:02:13.511 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/33 file after the snapshot 2023-06-22T08:02:13.511 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/34 file after the snapshot 2023-06-22T08:02:13.511 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/35 file after the snapshot 2023-06-22T08:02:13.512 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/36 file after the snapshot 2023-06-22T08:02:13.512 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/37 file after the snapshot 2023-06-22T08:02:13.512 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/38 file after the snapshot 2023-06-22T08:02:13.512 INFO:tasks.workunit.client.0.smithi023.stdout: create 154/39 file after the snapshot 2023-06-22T08:02:13.513 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/.snap/snap-subdir-test 2023-06-22T08:02:13.543 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/30 file after the snapshot 2023-06-22T08:02:13.545 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/31 file after the snapshot 2023-06-22T08:02:13.546 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/32 file after the snapshot 2023-06-22T08:02:13.546 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/33 file after the snapshot 2023-06-22T08:02:13.546 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/34 file after the snapshot 2023-06-22T08:02:13.546 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/35 file after the snapshot 2023-06-22T08:02:13.547 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/36 file after the snapshot 2023-06-22T08:02:13.547 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/37 file after the snapshot 2023-06-22T08:02:13.548 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/38 file after the snapshot 2023-06-22T08:02:13.548 INFO:tasks.workunit.client.0.smithi023.stdout: create 155/39 file after the snapshot 2023-06-22T08:02:13.548 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/.snap/snap-subdir-test 2023-06-22T08:02:13.582 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/30 file after the snapshot 2023-06-22T08:02:13.583 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/31 file after the snapshot 2023-06-22T08:02:13.583 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/32 file after the snapshot 2023-06-22T08:02:13.583 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/33 file after the snapshot 2023-06-22T08:02:13.583 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/34 file after the snapshot 2023-06-22T08:02:13.584 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/35 file after the snapshot 2023-06-22T08:02:13.584 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/36 file after the snapshot 2023-06-22T08:02:13.584 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/37 file after the snapshot 2023-06-22T08:02:13.584 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/38 file after the snapshot 2023-06-22T08:02:13.585 INFO:tasks.workunit.client.0.smithi023.stdout: create 156/39 file after the snapshot 2023-06-22T08:02:13.585 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/.snap/snap-subdir-test 2023-06-22T08:02:13.614 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/30 file after the snapshot 2023-06-22T08:02:13.616 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/31 file after the snapshot 2023-06-22T08:02:13.616 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/32 file after the snapshot 2023-06-22T08:02:13.616 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/33 file after the snapshot 2023-06-22T08:02:13.617 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/34 file after the snapshot 2023-06-22T08:02:13.617 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/35 file after the snapshot 2023-06-22T08:02:13.617 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/36 file after the snapshot 2023-06-22T08:02:13.617 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/37 file after the snapshot 2023-06-22T08:02:13.617 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/38 file after the snapshot 2023-06-22T08:02:13.617 INFO:tasks.workunit.client.0.smithi023.stdout: create 157/39 file after the snapshot 2023-06-22T08:02:13.617 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/.snap/snap-subdir-test 2023-06-22T08:02:13.648 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/30 file after the snapshot 2023-06-22T08:02:13.650 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/31 file after the snapshot 2023-06-22T08:02:13.650 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/32 file after the snapshot 2023-06-22T08:02:13.650 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/33 file after the snapshot 2023-06-22T08:02:13.650 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/34 file after the snapshot 2023-06-22T08:02:13.651 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/35 file after the snapshot 2023-06-22T08:02:13.651 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/36 file after the snapshot 2023-06-22T08:02:13.651 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/37 file after the snapshot 2023-06-22T08:02:13.651 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/38 file after the snapshot 2023-06-22T08:02:13.651 INFO:tasks.workunit.client.0.smithi023.stdout: create 158/39 file after the snapshot 2023-06-22T08:02:13.651 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/.snap/snap-subdir-test 2023-06-22T08:02:13.681 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/30 file after the snapshot 2023-06-22T08:02:13.682 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/31 file after the snapshot 2023-06-22T08:02:13.683 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/32 file after the snapshot 2023-06-22T08:02:13.683 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/33 file after the snapshot 2023-06-22T08:02:13.684 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/34 file after the snapshot 2023-06-22T08:02:13.684 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/35 file after the snapshot 2023-06-22T08:02:13.684 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/36 file after the snapshot 2023-06-22T08:02:13.685 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/37 file after the snapshot 2023-06-22T08:02:13.685 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/38 file after the snapshot 2023-06-22T08:02:13.685 INFO:tasks.workunit.client.0.smithi023.stdout: create 159/39 file after the snapshot 2023-06-22T08:02:13.685 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/.snap/snap-subdir-test 2023-06-22T08:02:13.714 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/30 file after the snapshot 2023-06-22T08:02:13.716 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/31 file after the snapshot 2023-06-22T08:02:13.716 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/32 file after the snapshot 2023-06-22T08:02:13.716 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/33 file after the snapshot 2023-06-22T08:02:13.717 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/34 file after the snapshot 2023-06-22T08:02:13.717 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/35 file after the snapshot 2023-06-22T08:02:13.718 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/36 file after the snapshot 2023-06-22T08:02:13.719 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/37 file after the snapshot 2023-06-22T08:02:13.719 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/38 file after the snapshot 2023-06-22T08:02:13.719 INFO:tasks.workunit.client.0.smithi023.stdout: create 160/39 file after the snapshot 2023-06-22T08:02:13.719 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/.snap/snap-subdir-test 2023-06-22T08:02:13.754 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/30 file after the snapshot 2023-06-22T08:02:13.756 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/31 file after the snapshot 2023-06-22T08:02:13.756 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/32 file after the snapshot 2023-06-22T08:02:13.756 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/33 file after the snapshot 2023-06-22T08:02:13.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/34 file after the snapshot 2023-06-22T08:02:13.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/35 file after the snapshot 2023-06-22T08:02:13.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/36 file after the snapshot 2023-06-22T08:02:13.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/37 file after the snapshot 2023-06-22T08:02:13.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/38 file after the snapshot 2023-06-22T08:02:13.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 161/39 file after the snapshot 2023-06-22T08:02:13.758 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/.snap/snap-subdir-test 2023-06-22T08:02:13.794 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/30 file after the snapshot 2023-06-22T08:02:13.795 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/31 file after the snapshot 2023-06-22T08:02:13.795 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/32 file after the snapshot 2023-06-22T08:02:13.795 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/33 file after the snapshot 2023-06-22T08:02:13.796 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/34 file after the snapshot 2023-06-22T08:02:13.796 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/35 file after the snapshot 2023-06-22T08:02:13.796 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/36 file after the snapshot 2023-06-22T08:02:13.796 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/37 file after the snapshot 2023-06-22T08:02:13.797 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/38 file after the snapshot 2023-06-22T08:02:13.797 INFO:tasks.workunit.client.0.smithi023.stdout: create 162/39 file after the snapshot 2023-06-22T08:02:13.797 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/.snap/snap-subdir-test 2023-06-22T08:02:13.834 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/30 file after the snapshot 2023-06-22T08:02:13.835 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/31 file after the snapshot 2023-06-22T08:02:13.836 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/32 file after the snapshot 2023-06-22T08:02:13.836 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/33 file after the snapshot 2023-06-22T08:02:13.836 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/34 file after the snapshot 2023-06-22T08:02:13.836 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/35 file after the snapshot 2023-06-22T08:02:13.836 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/36 file after the snapshot 2023-06-22T08:02:13.837 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/37 file after the snapshot 2023-06-22T08:02:13.837 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/38 file after the snapshot 2023-06-22T08:02:13.837 INFO:tasks.workunit.client.0.smithi023.stdout: create 163/39 file after the snapshot 2023-06-22T08:02:13.837 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/.snap/snap-subdir-test 2023-06-22T08:02:13.866 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/30 file after the snapshot 2023-06-22T08:02:13.867 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/31 file after the snapshot 2023-06-22T08:02:13.867 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/32 file after the snapshot 2023-06-22T08:02:13.867 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/33 file after the snapshot 2023-06-22T08:02:13.867 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/34 file after the snapshot 2023-06-22T08:02:13.867 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/35 file after the snapshot 2023-06-22T08:02:13.868 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/36 file after the snapshot 2023-06-22T08:02:13.868 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/37 file after the snapshot 2023-06-22T08:02:13.868 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/38 file after the snapshot 2023-06-22T08:02:13.869 INFO:tasks.workunit.client.0.smithi023.stdout: create 164/39 file after the snapshot 2023-06-22T08:02:13.869 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/.snap/snap-subdir-test 2023-06-22T08:02:13.897 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/30 file after the snapshot 2023-06-22T08:02:13.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/31 file after the snapshot 2023-06-22T08:02:13.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/32 file after the snapshot 2023-06-22T08:02:13.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/33 file after the snapshot 2023-06-22T08:02:13.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/34 file after the snapshot 2023-06-22T08:02:13.900 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/35 file after the snapshot 2023-06-22T08:02:13.900 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/36 file after the snapshot 2023-06-22T08:02:13.900 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/37 file after the snapshot 2023-06-22T08:02:13.900 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/38 file after the snapshot 2023-06-22T08:02:13.901 INFO:tasks.workunit.client.0.smithi023.stdout: create 165/39 file after the snapshot 2023-06-22T08:02:13.901 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/.snap/snap-subdir-test 2023-06-22T08:02:13.936 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/30 file after the snapshot 2023-06-22T08:02:13.938 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/31 file after the snapshot 2023-06-22T08:02:13.938 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/32 file after the snapshot 2023-06-22T08:02:13.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/33 file after the snapshot 2023-06-22T08:02:13.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/34 file after the snapshot 2023-06-22T08:02:13.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/35 file after the snapshot 2023-06-22T08:02:13.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/36 file after the snapshot 2023-06-22T08:02:13.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/37 file after the snapshot 2023-06-22T08:02:13.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/38 file after the snapshot 2023-06-22T08:02:13.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 166/39 file after the snapshot 2023-06-22T08:02:13.940 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/.snap/snap-subdir-test 2023-06-22T08:02:13.969 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/30 file after the snapshot 2023-06-22T08:02:13.970 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/31 file after the snapshot 2023-06-22T08:02:13.970 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/32 file after the snapshot 2023-06-22T08:02:13.970 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/33 file after the snapshot 2023-06-22T08:02:13.971 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/34 file after the snapshot 2023-06-22T08:02:13.971 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/35 file after the snapshot 2023-06-22T08:02:13.971 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/36 file after the snapshot 2023-06-22T08:02:13.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/37 file after the snapshot 2023-06-22T08:02:13.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/38 file after the snapshot 2023-06-22T08:02:13.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 167/39 file after the snapshot 2023-06-22T08:02:13.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/.snap/snap-subdir-test 2023-06-22T08:02:14.001 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/30 file after the snapshot 2023-06-22T08:02:14.003 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/31 file after the snapshot 2023-06-22T08:02:14.003 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/32 file after the snapshot 2023-06-22T08:02:14.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/33 file after the snapshot 2023-06-22T08:02:14.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/34 file after the snapshot 2023-06-22T08:02:14.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/35 file after the snapshot 2023-06-22T08:02:14.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/36 file after the snapshot 2023-06-22T08:02:14.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/37 file after the snapshot 2023-06-22T08:02:14.005 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/38 file after the snapshot 2023-06-22T08:02:14.005 INFO:tasks.workunit.client.0.smithi023.stdout: create 168/39 file after the snapshot 2023-06-22T08:02:14.005 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/.snap/snap-subdir-test 2023-06-22T08:02:14.034 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/30 file after the snapshot 2023-06-22T08:02:14.035 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/31 file after the snapshot 2023-06-22T08:02:14.035 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/32 file after the snapshot 2023-06-22T08:02:14.035 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/33 file after the snapshot 2023-06-22T08:02:14.036 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/34 file after the snapshot 2023-06-22T08:02:14.036 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/35 file after the snapshot 2023-06-22T08:02:14.036 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/36 file after the snapshot 2023-06-22T08:02:14.036 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/37 file after the snapshot 2023-06-22T08:02:14.037 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/38 file after the snapshot 2023-06-22T08:02:14.037 INFO:tasks.workunit.client.0.smithi023.stdout: create 169/39 file after the snapshot 2023-06-22T08:02:14.037 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/.snap/snap-subdir-test 2023-06-22T08:02:14.067 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/30 file after the snapshot 2023-06-22T08:02:14.068 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/31 file after the snapshot 2023-06-22T08:02:14.068 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/32 file after the snapshot 2023-06-22T08:02:14.069 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/33 file after the snapshot 2023-06-22T08:02:14.069 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/34 file after the snapshot 2023-06-22T08:02:14.069 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/35 file after the snapshot 2023-06-22T08:02:14.069 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/36 file after the snapshot 2023-06-22T08:02:14.069 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/37 file after the snapshot 2023-06-22T08:02:14.069 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/38 file after the snapshot 2023-06-22T08:02:14.070 INFO:tasks.workunit.client.0.smithi023.stdout: create 170/39 file after the snapshot 2023-06-22T08:02:14.070 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/.snap/snap-subdir-test 2023-06-22T08:02:14.100 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/30 file after the snapshot 2023-06-22T08:02:14.101 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/31 file after the snapshot 2023-06-22T08:02:14.101 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/32 file after the snapshot 2023-06-22T08:02:14.101 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/33 file after the snapshot 2023-06-22T08:02:14.101 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/34 file after the snapshot 2023-06-22T08:02:14.102 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/35 file after the snapshot 2023-06-22T08:02:14.102 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/36 file after the snapshot 2023-06-22T08:02:14.102 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/37 file after the snapshot 2023-06-22T08:02:14.103 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/38 file after the snapshot 2023-06-22T08:02:14.103 INFO:tasks.workunit.client.0.smithi023.stdout: create 171/39 file after the snapshot 2023-06-22T08:02:14.103 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/.snap/snap-subdir-test 2023-06-22T08:02:14.133 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/30 file after the snapshot 2023-06-22T08:02:14.134 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/31 file after the snapshot 2023-06-22T08:02:14.134 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/32 file after the snapshot 2023-06-22T08:02:14.134 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/33 file after the snapshot 2023-06-22T08:02:14.135 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/34 file after the snapshot 2023-06-22T08:02:14.135 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/35 file after the snapshot 2023-06-22T08:02:14.136 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/36 file after the snapshot 2023-06-22T08:02:14.136 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/37 file after the snapshot 2023-06-22T08:02:14.136 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/38 file after the snapshot 2023-06-22T08:02:14.136 INFO:tasks.workunit.client.0.smithi023.stdout: create 172/39 file after the snapshot 2023-06-22T08:02:14.136 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/.snap/snap-subdir-test 2023-06-22T08:02:14.165 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/30 file after the snapshot 2023-06-22T08:02:14.166 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/31 file after the snapshot 2023-06-22T08:02:14.166 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/32 file after the snapshot 2023-06-22T08:02:14.167 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/33 file after the snapshot 2023-06-22T08:02:14.167 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/34 file after the snapshot 2023-06-22T08:02:14.167 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/35 file after the snapshot 2023-06-22T08:02:14.168 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/36 file after the snapshot 2023-06-22T08:02:14.168 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/37 file after the snapshot 2023-06-22T08:02:14.168 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/38 file after the snapshot 2023-06-22T08:02:14.168 INFO:tasks.workunit.client.0.smithi023.stdout: create 173/39 file after the snapshot 2023-06-22T08:02:14.168 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/.snap/snap-subdir-test 2023-06-22T08:02:14.200 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/30 file after the snapshot 2023-06-22T08:02:14.202 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/31 file after the snapshot 2023-06-22T08:02:14.202 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/32 file after the snapshot 2023-06-22T08:02:14.202 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/33 file after the snapshot 2023-06-22T08:02:14.202 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/34 file after the snapshot 2023-06-22T08:02:14.203 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/35 file after the snapshot 2023-06-22T08:02:14.203 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/36 file after the snapshot 2023-06-22T08:02:14.203 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/37 file after the snapshot 2023-06-22T08:02:14.203 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/38 file after the snapshot 2023-06-22T08:02:14.203 INFO:tasks.workunit.client.0.smithi023.stdout: create 174/39 file after the snapshot 2023-06-22T08:02:14.203 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/.snap/snap-subdir-test 2023-06-22T08:02:14.229 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/30 file after the snapshot 2023-06-22T08:02:14.231 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/31 file after the snapshot 2023-06-22T08:02:14.231 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/32 file after the snapshot 2023-06-22T08:02:14.231 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/33 file after the snapshot 2023-06-22T08:02:14.231 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/34 file after the snapshot 2023-06-22T08:02:14.232 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/35 file after the snapshot 2023-06-22T08:02:14.232 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/36 file after the snapshot 2023-06-22T08:02:14.232 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/37 file after the snapshot 2023-06-22T08:02:14.232 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/38 file after the snapshot 2023-06-22T08:02:14.232 INFO:tasks.workunit.client.0.smithi023.stdout: create 175/39 file after the snapshot 2023-06-22T08:02:14.232 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/.snap/snap-subdir-test 2023-06-22T08:02:14.261 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/30 file after the snapshot 2023-06-22T08:02:14.263 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/31 file after the snapshot 2023-06-22T08:02:14.263 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/32 file after the snapshot 2023-06-22T08:02:14.263 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/33 file after the snapshot 2023-06-22T08:02:14.263 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/34 file after the snapshot 2023-06-22T08:02:14.263 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/35 file after the snapshot 2023-06-22T08:02:14.264 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/36 file after the snapshot 2023-06-22T08:02:14.264 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/37 file after the snapshot 2023-06-22T08:02:14.264 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/38 file after the snapshot 2023-06-22T08:02:14.264 INFO:tasks.workunit.client.0.smithi023.stdout: create 176/39 file after the snapshot 2023-06-22T08:02:14.265 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/.snap/snap-subdir-test 2023-06-22T08:02:14.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/30 file after the snapshot 2023-06-22T08:02:14.308 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/31 file after the snapshot 2023-06-22T08:02:14.308 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/32 file after the snapshot 2023-06-22T08:02:14.308 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/33 file after the snapshot 2023-06-22T08:02:14.308 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/34 file after the snapshot 2023-06-22T08:02:14.308 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/35 file after the snapshot 2023-06-22T08:02:14.309 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/36 file after the snapshot 2023-06-22T08:02:14.309 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/37 file after the snapshot 2023-06-22T08:02:14.309 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/38 file after the snapshot 2023-06-22T08:02:14.309 INFO:tasks.workunit.client.0.smithi023.stdout: create 177/39 file after the snapshot 2023-06-22T08:02:14.310 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/.snap/snap-subdir-test 2023-06-22T08:02:14.339 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/30 file after the snapshot 2023-06-22T08:02:14.341 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/31 file after the snapshot 2023-06-22T08:02:14.342 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/32 file after the snapshot 2023-06-22T08:02:14.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/33 file after the snapshot 2023-06-22T08:02:14.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/34 file after the snapshot 2023-06-22T08:02:14.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/35 file after the snapshot 2023-06-22T08:02:14.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/36 file after the snapshot 2023-06-22T08:02:14.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/37 file after the snapshot 2023-06-22T08:02:14.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/38 file after the snapshot 2023-06-22T08:02:14.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 178/39 file after the snapshot 2023-06-22T08:02:14.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/.snap/snap-subdir-test 2023-06-22T08:02:14.372 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/30 file after the snapshot 2023-06-22T08:02:14.375 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/31 file after the snapshot 2023-06-22T08:02:14.375 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/32 file after the snapshot 2023-06-22T08:02:14.375 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/33 file after the snapshot 2023-06-22T08:02:14.375 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/34 file after the snapshot 2023-06-22T08:02:14.376 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/35 file after the snapshot 2023-06-22T08:02:14.376 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/36 file after the snapshot 2023-06-22T08:02:14.376 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/37 file after the snapshot 2023-06-22T08:02:14.376 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/38 file after the snapshot 2023-06-22T08:02:14.377 INFO:tasks.workunit.client.0.smithi023.stdout: create 179/39 file after the snapshot 2023-06-22T08:02:14.377 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/.snap/snap-subdir-test 2023-06-22T08:02:14.407 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/30 file after the snapshot 2023-06-22T08:02:14.410 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/31 file after the snapshot 2023-06-22T08:02:14.410 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/32 file after the snapshot 2023-06-22T08:02:14.410 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/33 file after the snapshot 2023-06-22T08:02:14.410 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/34 file after the snapshot 2023-06-22T08:02:14.410 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/35 file after the snapshot 2023-06-22T08:02:14.410 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/36 file after the snapshot 2023-06-22T08:02:14.411 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/37 file after the snapshot 2023-06-22T08:02:14.411 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/38 file after the snapshot 2023-06-22T08:02:14.411 INFO:tasks.workunit.client.0.smithi023.stdout: create 180/39 file after the snapshot 2023-06-22T08:02:14.411 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/.snap/snap-subdir-test 2023-06-22T08:02:14.439 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/30 file after the snapshot 2023-06-22T08:02:14.441 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/31 file after the snapshot 2023-06-22T08:02:14.441 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/32 file after the snapshot 2023-06-22T08:02:14.441 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/33 file after the snapshot 2023-06-22T08:02:14.442 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/34 file after the snapshot 2023-06-22T08:02:14.442 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/35 file after the snapshot 2023-06-22T08:02:14.442 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/36 file after the snapshot 2023-06-22T08:02:14.442 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/37 file after the snapshot 2023-06-22T08:02:14.442 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/38 file after the snapshot 2023-06-22T08:02:14.442 INFO:tasks.workunit.client.0.smithi023.stdout: create 181/39 file after the snapshot 2023-06-22T08:02:14.442 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/.snap/snap-subdir-test 2023-06-22T08:02:14.471 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/30 file after the snapshot 2023-06-22T08:02:14.474 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/31 file after the snapshot 2023-06-22T08:02:14.474 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/32 file after the snapshot 2023-06-22T08:02:14.475 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/33 file after the snapshot 2023-06-22T08:02:14.475 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/34 file after the snapshot 2023-06-22T08:02:14.475 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/35 file after the snapshot 2023-06-22T08:02:14.476 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/36 file after the snapshot 2023-06-22T08:02:14.476 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/37 file after the snapshot 2023-06-22T08:02:14.476 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/38 file after the snapshot 2023-06-22T08:02:14.476 INFO:tasks.workunit.client.0.smithi023.stdout: create 182/39 file after the snapshot 2023-06-22T08:02:14.477 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/.snap/snap-subdir-test 2023-06-22T08:02:14.503 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/30 file after the snapshot 2023-06-22T08:02:14.507 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/31 file after the snapshot 2023-06-22T08:02:14.507 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/32 file after the snapshot 2023-06-22T08:02:14.507 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/33 file after the snapshot 2023-06-22T08:02:14.508 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/34 file after the snapshot 2023-06-22T08:02:14.508 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/35 file after the snapshot 2023-06-22T08:02:14.508 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/36 file after the snapshot 2023-06-22T08:02:14.508 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/37 file after the snapshot 2023-06-22T08:02:14.508 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/38 file after the snapshot 2023-06-22T08:02:14.509 INFO:tasks.workunit.client.0.smithi023.stdout: create 183/39 file after the snapshot 2023-06-22T08:02:14.509 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/.snap/snap-subdir-test 2023-06-22T08:02:14.543 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/30 file after the snapshot 2023-06-22T08:02:14.545 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/31 file after the snapshot 2023-06-22T08:02:14.546 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/32 file after the snapshot 2023-06-22T08:02:14.546 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/33 file after the snapshot 2023-06-22T08:02:14.546 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/34 file after the snapshot 2023-06-22T08:02:14.547 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/35 file after the snapshot 2023-06-22T08:02:14.547 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/36 file after the snapshot 2023-06-22T08:02:14.547 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/37 file after the snapshot 2023-06-22T08:02:14.547 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/38 file after the snapshot 2023-06-22T08:02:14.547 INFO:tasks.workunit.client.0.smithi023.stdout: create 184/39 file after the snapshot 2023-06-22T08:02:14.547 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/.snap/snap-subdir-test 2023-06-22T08:02:14.574 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/30 file after the snapshot 2023-06-22T08:02:14.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/31 file after the snapshot 2023-06-22T08:02:14.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/32 file after the snapshot 2023-06-22T08:02:14.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/33 file after the snapshot 2023-06-22T08:02:14.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/34 file after the snapshot 2023-06-22T08:02:14.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/35 file after the snapshot 2023-06-22T08:02:14.577 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/36 file after the snapshot 2023-06-22T08:02:14.577 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/37 file after the snapshot 2023-06-22T08:02:14.577 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/38 file after the snapshot 2023-06-22T08:02:14.577 INFO:tasks.workunit.client.0.smithi023.stdout: create 185/39 file after the snapshot 2023-06-22T08:02:14.578 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/.snap/snap-subdir-test 2023-06-22T08:02:14.610 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/30 file after the snapshot 2023-06-22T08:02:14.611 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/31 file after the snapshot 2023-06-22T08:02:14.611 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/32 file after the snapshot 2023-06-22T08:02:14.611 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/33 file after the snapshot 2023-06-22T08:02:14.612 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/34 file after the snapshot 2023-06-22T08:02:14.612 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/35 file after the snapshot 2023-06-22T08:02:14.612 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/36 file after the snapshot 2023-06-22T08:02:14.612 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/37 file after the snapshot 2023-06-22T08:02:14.612 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/38 file after the snapshot 2023-06-22T08:02:14.613 INFO:tasks.workunit.client.0.smithi023.stdout: create 186/39 file after the snapshot 2023-06-22T08:02:14.613 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/.snap/snap-subdir-test 2023-06-22T08:02:14.641 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/30 file after the snapshot 2023-06-22T08:02:14.643 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/31 file after the snapshot 2023-06-22T08:02:14.643 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/32 file after the snapshot 2023-06-22T08:02:14.643 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/33 file after the snapshot 2023-06-22T08:02:14.643 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/34 file after the snapshot 2023-06-22T08:02:14.644 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/35 file after the snapshot 2023-06-22T08:02:14.644 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/36 file after the snapshot 2023-06-22T08:02:14.644 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/37 file after the snapshot 2023-06-22T08:02:14.644 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/38 file after the snapshot 2023-06-22T08:02:14.645 INFO:tasks.workunit.client.0.smithi023.stdout: create 187/39 file after the snapshot 2023-06-22T08:02:14.645 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/.snap/snap-subdir-test 2023-06-22T08:02:14.673 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/30 file after the snapshot 2023-06-22T08:02:14.675 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/31 file after the snapshot 2023-06-22T08:02:14.675 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/32 file after the snapshot 2023-06-22T08:02:14.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/33 file after the snapshot 2023-06-22T08:02:14.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/34 file after the snapshot 2023-06-22T08:02:14.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/35 file after the snapshot 2023-06-22T08:02:14.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/36 file after the snapshot 2023-06-22T08:02:14.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/37 file after the snapshot 2023-06-22T08:02:14.677 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/38 file after the snapshot 2023-06-22T08:02:14.677 INFO:tasks.workunit.client.0.smithi023.stdout: create 188/39 file after the snapshot 2023-06-22T08:02:14.677 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/.snap/snap-subdir-test 2023-06-22T08:02:14.705 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/30 file after the snapshot 2023-06-22T08:02:14.707 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/31 file after the snapshot 2023-06-22T08:02:14.707 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/32 file after the snapshot 2023-06-22T08:02:14.708 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/33 file after the snapshot 2023-06-22T08:02:14.708 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/34 file after the snapshot 2023-06-22T08:02:14.708 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/35 file after the snapshot 2023-06-22T08:02:14.708 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/36 file after the snapshot 2023-06-22T08:02:14.708 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/37 file after the snapshot 2023-06-22T08:02:14.709 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/38 file after the snapshot 2023-06-22T08:02:14.709 INFO:tasks.workunit.client.0.smithi023.stdout: create 189/39 file after the snapshot 2023-06-22T08:02:14.709 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/.snap/snap-subdir-test 2023-06-22T08:02:14.737 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/30 file after the snapshot 2023-06-22T08:02:14.738 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/31 file after the snapshot 2023-06-22T08:02:14.738 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/32 file after the snapshot 2023-06-22T08:02:14.739 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/33 file after the snapshot 2023-06-22T08:02:14.739 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/34 file after the snapshot 2023-06-22T08:02:14.739 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/35 file after the snapshot 2023-06-22T08:02:14.739 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/36 file after the snapshot 2023-06-22T08:02:14.739 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/37 file after the snapshot 2023-06-22T08:02:14.739 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/38 file after the snapshot 2023-06-22T08:02:14.739 INFO:tasks.workunit.client.0.smithi023.stdout: create 190/39 file after the snapshot 2023-06-22T08:02:14.740 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/.snap/snap-subdir-test 2023-06-22T08:02:14.768 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/30 file after the snapshot 2023-06-22T08:02:14.770 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/31 file after the snapshot 2023-06-22T08:02:14.770 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/32 file after the snapshot 2023-06-22T08:02:14.770 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/33 file after the snapshot 2023-06-22T08:02:14.770 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/34 file after the snapshot 2023-06-22T08:02:14.771 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/35 file after the snapshot 2023-06-22T08:02:14.771 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/36 file after the snapshot 2023-06-22T08:02:14.771 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/37 file after the snapshot 2023-06-22T08:02:14.771 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/38 file after the snapshot 2023-06-22T08:02:14.771 INFO:tasks.workunit.client.0.smithi023.stdout: create 191/39 file after the snapshot 2023-06-22T08:02:14.771 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/.snap/snap-subdir-test 2023-06-22T08:02:14.802 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/30 file after the snapshot 2023-06-22T08:02:14.804 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/31 file after the snapshot 2023-06-22T08:02:14.804 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/32 file after the snapshot 2023-06-22T08:02:14.804 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/33 file after the snapshot 2023-06-22T08:02:14.804 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/34 file after the snapshot 2023-06-22T08:02:14.804 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/35 file after the snapshot 2023-06-22T08:02:14.805 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/36 file after the snapshot 2023-06-22T08:02:14.805 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/37 file after the snapshot 2023-06-22T08:02:14.805 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/38 file after the snapshot 2023-06-22T08:02:14.805 INFO:tasks.workunit.client.0.smithi023.stdout: create 192/39 file after the snapshot 2023-06-22T08:02:14.805 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/.snap/snap-subdir-test 2023-06-22T08:02:14.833 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/30 file after the snapshot 2023-06-22T08:02:14.835 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/31 file after the snapshot 2023-06-22T08:02:14.835 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/32 file after the snapshot 2023-06-22T08:02:14.835 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/33 file after the snapshot 2023-06-22T08:02:14.835 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/34 file after the snapshot 2023-06-22T08:02:14.836 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/35 file after the snapshot 2023-06-22T08:02:14.836 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/36 file after the snapshot 2023-06-22T08:02:14.836 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/37 file after the snapshot 2023-06-22T08:02:14.836 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/38 file after the snapshot 2023-06-22T08:02:14.836 INFO:tasks.workunit.client.0.smithi023.stdout: create 193/39 file after the snapshot 2023-06-22T08:02:14.837 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/.snap/snap-subdir-test 2023-06-22T08:02:14.864 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/30 file after the snapshot 2023-06-22T08:02:14.865 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/31 file after the snapshot 2023-06-22T08:02:14.866 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/32 file after the snapshot 2023-06-22T08:02:14.866 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/33 file after the snapshot 2023-06-22T08:02:14.866 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/34 file after the snapshot 2023-06-22T08:02:14.866 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/35 file after the snapshot 2023-06-22T08:02:14.866 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/36 file after the snapshot 2023-06-22T08:02:14.866 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/37 file after the snapshot 2023-06-22T08:02:14.867 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/38 file after the snapshot 2023-06-22T08:02:14.867 INFO:tasks.workunit.client.0.smithi023.stdout: create 194/39 file after the snapshot 2023-06-22T08:02:14.867 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/.snap/snap-subdir-test 2023-06-22T08:02:14.895 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/30 file after the snapshot 2023-06-22T08:02:14.896 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/31 file after the snapshot 2023-06-22T08:02:14.897 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/32 file after the snapshot 2023-06-22T08:02:14.897 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/33 file after the snapshot 2023-06-22T08:02:14.897 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/34 file after the snapshot 2023-06-22T08:02:14.898 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/35 file after the snapshot 2023-06-22T08:02:14.898 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/36 file after the snapshot 2023-06-22T08:02:14.898 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/37 file after the snapshot 2023-06-22T08:02:14.898 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/38 file after the snapshot 2023-06-22T08:02:14.898 INFO:tasks.workunit.client.0.smithi023.stdout: create 195/39 file after the snapshot 2023-06-22T08:02:14.898 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/.snap/snap-subdir-test 2023-06-22T08:02:14.928 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/30 file after the snapshot 2023-06-22T08:02:14.930 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/31 file after the snapshot 2023-06-22T08:02:14.930 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/32 file after the snapshot 2023-06-22T08:02:14.930 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/33 file after the snapshot 2023-06-22T08:02:14.930 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/34 file after the snapshot 2023-06-22T08:02:14.930 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/35 file after the snapshot 2023-06-22T08:02:14.931 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/36 file after the snapshot 2023-06-22T08:02:14.931 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/37 file after the snapshot 2023-06-22T08:02:14.931 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/38 file after the snapshot 2023-06-22T08:02:14.931 INFO:tasks.workunit.client.0.smithi023.stdout: create 196/39 file after the snapshot 2023-06-22T08:02:14.932 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/.snap/snap-subdir-test 2023-06-22T08:02:14.967 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/30 file after the snapshot 2023-06-22T08:02:14.970 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/31 file after the snapshot 2023-06-22T08:02:14.971 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/32 file after the snapshot 2023-06-22T08:02:14.971 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/33 file after the snapshot 2023-06-22T08:02:14.971 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/34 file after the snapshot 2023-06-22T08:02:14.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/35 file after the snapshot 2023-06-22T08:02:14.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/36 file after the snapshot 2023-06-22T08:02:14.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/37 file after the snapshot 2023-06-22T08:02:14.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/38 file after the snapshot 2023-06-22T08:02:14.973 INFO:tasks.workunit.client.0.smithi023.stdout: create 197/39 file after the snapshot 2023-06-22T08:02:14.973 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/.snap/snap-subdir-test 2023-06-22T08:02:15.005 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/30 file after the snapshot 2023-06-22T08:02:15.009 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/31 file after the snapshot 2023-06-22T08:02:15.009 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/32 file after the snapshot 2023-06-22T08:02:15.010 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/33 file after the snapshot 2023-06-22T08:02:15.010 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/34 file after the snapshot 2023-06-22T08:02:15.010 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/35 file after the snapshot 2023-06-22T08:02:15.010 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/36 file after the snapshot 2023-06-22T08:02:15.010 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/37 file after the snapshot 2023-06-22T08:02:15.010 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/38 file after the snapshot 2023-06-22T08:02:15.010 INFO:tasks.workunit.client.0.smithi023.stdout: create 198/39 file after the snapshot 2023-06-22T08:02:15.011 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/.snap/snap-subdir-test 2023-06-22T08:02:15.045 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/30 file after the snapshot 2023-06-22T08:02:15.049 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/31 file after the snapshot 2023-06-22T08:02:15.049 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/32 file after the snapshot 2023-06-22T08:02:15.050 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/33 file after the snapshot 2023-06-22T08:02:15.050 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/34 file after the snapshot 2023-06-22T08:02:15.050 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/35 file after the snapshot 2023-06-22T08:02:15.050 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/36 file after the snapshot 2023-06-22T08:02:15.050 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/37 file after the snapshot 2023-06-22T08:02:15.051 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/38 file after the snapshot 2023-06-22T08:02:15.051 INFO:tasks.workunit.client.0.smithi023.stdout: create 199/39 file after the snapshot 2023-06-22T08:02:15.051 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/.snap/snap-subdir-test 2023-06-22T08:02:15.084 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/30 file after the snapshot 2023-06-22T08:02:15.086 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/31 file after the snapshot 2023-06-22T08:02:15.086 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/32 file after the snapshot 2023-06-22T08:02:15.086 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/33 file after the snapshot 2023-06-22T08:02:15.086 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/34 file after the snapshot 2023-06-22T08:02:15.087 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/35 file after the snapshot 2023-06-22T08:02:15.087 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/36 file after the snapshot 2023-06-22T08:02:15.087 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/37 file after the snapshot 2023-06-22T08:02:15.087 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/38 file after the snapshot 2023-06-22T08:02:15.087 INFO:tasks.workunit.client.0.smithi023.stdout: create 200/39 file after the snapshot 2023-06-22T08:02:15.088 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/.snap/snap-subdir-test 2023-06-22T08:02:15.114 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/30 file after the snapshot 2023-06-22T08:02:15.116 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/31 file after the snapshot 2023-06-22T08:02:15.116 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/32 file after the snapshot 2023-06-22T08:02:15.116 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/33 file after the snapshot 2023-06-22T08:02:15.116 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/34 file after the snapshot 2023-06-22T08:02:15.117 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/35 file after the snapshot 2023-06-22T08:02:15.117 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/36 file after the snapshot 2023-06-22T08:02:15.117 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/37 file after the snapshot 2023-06-22T08:02:15.117 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/38 file after the snapshot 2023-06-22T08:02:15.117 INFO:tasks.workunit.client.0.smithi023.stdout: create 201/39 file after the snapshot 2023-06-22T08:02:15.117 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/.snap/snap-subdir-test 2023-06-22T08:02:15.143 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/30 file after the snapshot 2023-06-22T08:02:15.144 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/31 file after the snapshot 2023-06-22T08:02:15.145 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/32 file after the snapshot 2023-06-22T08:02:15.145 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/33 file after the snapshot 2023-06-22T08:02:15.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/34 file after the snapshot 2023-06-22T08:02:15.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/35 file after the snapshot 2023-06-22T08:02:15.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/36 file after the snapshot 2023-06-22T08:02:15.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/37 file after the snapshot 2023-06-22T08:02:15.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/38 file after the snapshot 2023-06-22T08:02:15.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 202/39 file after the snapshot 2023-06-22T08:02:15.147 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/.snap/snap-subdir-test 2023-06-22T08:02:15.173 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/30 file after the snapshot 2023-06-22T08:02:15.175 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/31 file after the snapshot 2023-06-22T08:02:15.175 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/32 file after the snapshot 2023-06-22T08:02:15.175 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/33 file after the snapshot 2023-06-22T08:02:15.175 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/34 file after the snapshot 2023-06-22T08:02:15.176 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/35 file after the snapshot 2023-06-22T08:02:15.176 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/36 file after the snapshot 2023-06-22T08:02:15.176 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/37 file after the snapshot 2023-06-22T08:02:15.177 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/38 file after the snapshot 2023-06-22T08:02:15.177 INFO:tasks.workunit.client.0.smithi023.stdout: create 203/39 file after the snapshot 2023-06-22T08:02:15.177 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/.snap/snap-subdir-test 2023-06-22T08:02:15.204 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/30 file after the snapshot 2023-06-22T08:02:15.205 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/31 file after the snapshot 2023-06-22T08:02:15.205 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/32 file after the snapshot 2023-06-22T08:02:15.206 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/33 file after the snapshot 2023-06-22T08:02:15.206 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/34 file after the snapshot 2023-06-22T08:02:15.206 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/35 file after the snapshot 2023-06-22T08:02:15.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/36 file after the snapshot 2023-06-22T08:02:15.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/37 file after the snapshot 2023-06-22T08:02:15.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/38 file after the snapshot 2023-06-22T08:02:15.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 204/39 file after the snapshot 2023-06-22T08:02:15.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/.snap/snap-subdir-test 2023-06-22T08:02:15.235 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/30 file after the snapshot 2023-06-22T08:02:15.238 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/31 file after the snapshot 2023-06-22T08:02:15.238 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/32 file after the snapshot 2023-06-22T08:02:15.239 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/33 file after the snapshot 2023-06-22T08:02:15.240 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/34 file after the snapshot 2023-06-22T08:02:15.240 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/35 file after the snapshot 2023-06-22T08:02:15.240 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/36 file after the snapshot 2023-06-22T08:02:15.241 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/37 file after the snapshot 2023-06-22T08:02:15.241 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/38 file after the snapshot 2023-06-22T08:02:15.241 INFO:tasks.workunit.client.0.smithi023.stdout: create 205/39 file after the snapshot 2023-06-22T08:02:15.241 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/.snap/snap-subdir-test 2023-06-22T08:02:15.272 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/30 file after the snapshot 2023-06-22T08:02:15.274 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/31 file after the snapshot 2023-06-22T08:02:15.274 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/32 file after the snapshot 2023-06-22T08:02:15.274 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/33 file after the snapshot 2023-06-22T08:02:15.274 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/34 file after the snapshot 2023-06-22T08:02:15.274 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/35 file after the snapshot 2023-06-22T08:02:15.274 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/36 file after the snapshot 2023-06-22T08:02:15.274 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/37 file after the snapshot 2023-06-22T08:02:15.275 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/38 file after the snapshot 2023-06-22T08:02:15.275 INFO:tasks.workunit.client.0.smithi023.stdout: create 206/39 file after the snapshot 2023-06-22T08:02:15.275 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/.snap/snap-subdir-test 2023-06-22T08:02:15.303 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/30 file after the snapshot 2023-06-22T08:02:15.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/31 file after the snapshot 2023-06-22T08:02:15.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/32 file after the snapshot 2023-06-22T08:02:15.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/33 file after the snapshot 2023-06-22T08:02:15.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/34 file after the snapshot 2023-06-22T08:02:15.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/35 file after the snapshot 2023-06-22T08:02:15.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/36 file after the snapshot 2023-06-22T08:02:15.306 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/37 file after the snapshot 2023-06-22T08:02:15.307 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/38 file after the snapshot 2023-06-22T08:02:15.307 INFO:tasks.workunit.client.0.smithi023.stdout: create 207/39 file after the snapshot 2023-06-22T08:02:15.307 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/.snap/snap-subdir-test 2023-06-22T08:02:15.340 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/30 file after the snapshot 2023-06-22T08:02:15.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/31 file after the snapshot 2023-06-22T08:02:15.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/32 file after the snapshot 2023-06-22T08:02:15.343 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/33 file after the snapshot 2023-06-22T08:02:15.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/34 file after the snapshot 2023-06-22T08:02:15.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/35 file after the snapshot 2023-06-22T08:02:15.344 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/36 file after the snapshot 2023-06-22T08:02:15.345 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/37 file after the snapshot 2023-06-22T08:02:15.345 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/38 file after the snapshot 2023-06-22T08:02:15.345 INFO:tasks.workunit.client.0.smithi023.stdout: create 208/39 file after the snapshot 2023-06-22T08:02:15.345 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/.snap/snap-subdir-test 2023-06-22T08:02:15.377 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/30 file after the snapshot 2023-06-22T08:02:15.379 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/31 file after the snapshot 2023-06-22T08:02:15.380 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/32 file after the snapshot 2023-06-22T08:02:15.380 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/33 file after the snapshot 2023-06-22T08:02:15.380 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/34 file after the snapshot 2023-06-22T08:02:15.380 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/35 file after the snapshot 2023-06-22T08:02:15.380 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/36 file after the snapshot 2023-06-22T08:02:15.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/37 file after the snapshot 2023-06-22T08:02:15.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/38 file after the snapshot 2023-06-22T08:02:15.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 209/39 file after the snapshot 2023-06-22T08:02:15.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/.snap/snap-subdir-test 2023-06-22T08:02:15.417 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/30 file after the snapshot 2023-06-22T08:02:15.420 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/31 file after the snapshot 2023-06-22T08:02:15.420 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/32 file after the snapshot 2023-06-22T08:02:15.421 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/33 file after the snapshot 2023-06-22T08:02:15.421 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/34 file after the snapshot 2023-06-22T08:02:15.421 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/35 file after the snapshot 2023-06-22T08:02:15.422 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/36 file after the snapshot 2023-06-22T08:02:15.422 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/37 file after the snapshot 2023-06-22T08:02:15.422 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/38 file after the snapshot 2023-06-22T08:02:15.422 INFO:tasks.workunit.client.0.smithi023.stdout: create 210/39 file after the snapshot 2023-06-22T08:02:15.423 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/.snap/snap-subdir-test 2023-06-22T08:02:15.453 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/30 file after the snapshot 2023-06-22T08:02:15.454 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/31 file after the snapshot 2023-06-22T08:02:15.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/32 file after the snapshot 2023-06-22T08:02:15.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/33 file after the snapshot 2023-06-22T08:02:15.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/34 file after the snapshot 2023-06-22T08:02:15.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/35 file after the snapshot 2023-06-22T08:02:15.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/36 file after the snapshot 2023-06-22T08:02:15.455 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/37 file after the snapshot 2023-06-22T08:02:15.456 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/38 file after the snapshot 2023-06-22T08:02:15.456 INFO:tasks.workunit.client.0.smithi023.stdout: create 211/39 file after the snapshot 2023-06-22T08:02:15.456 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/.snap/snap-subdir-test 2023-06-22T08:02:15.486 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/30 file after the snapshot 2023-06-22T08:02:15.487 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/31 file after the snapshot 2023-06-22T08:02:15.487 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/32 file after the snapshot 2023-06-22T08:02:15.487 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/33 file after the snapshot 2023-06-22T08:02:15.488 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/34 file after the snapshot 2023-06-22T08:02:15.488 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/35 file after the snapshot 2023-06-22T08:02:15.488 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/36 file after the snapshot 2023-06-22T08:02:15.489 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/37 file after the snapshot 2023-06-22T08:02:15.489 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/38 file after the snapshot 2023-06-22T08:02:15.489 INFO:tasks.workunit.client.0.smithi023.stdout: create 212/39 file after the snapshot 2023-06-22T08:02:15.489 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/.snap/snap-subdir-test 2023-06-22T08:02:15.512 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/30 file after the snapshot 2023-06-22T08:02:15.514 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/31 file after the snapshot 2023-06-22T08:02:15.515 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/32 file after the snapshot 2023-06-22T08:02:15.515 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/33 file after the snapshot 2023-06-22T08:02:15.515 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/34 file after the snapshot 2023-06-22T08:02:15.515 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/35 file after the snapshot 2023-06-22T08:02:15.516 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/36 file after the snapshot 2023-06-22T08:02:15.516 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/37 file after the snapshot 2023-06-22T08:02:15.516 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/38 file after the snapshot 2023-06-22T08:02:15.517 INFO:tasks.workunit.client.0.smithi023.stdout: create 213/39 file after the snapshot 2023-06-22T08:02:15.517 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/.snap/snap-subdir-test 2023-06-22T08:02:15.541 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/30 file after the snapshot 2023-06-22T08:02:15.543 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/31 file after the snapshot 2023-06-22T08:02:15.543 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/32 file after the snapshot 2023-06-22T08:02:15.543 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/33 file after the snapshot 2023-06-22T08:02:15.543 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/34 file after the snapshot 2023-06-22T08:02:15.543 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/35 file after the snapshot 2023-06-22T08:02:15.544 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/36 file after the snapshot 2023-06-22T08:02:15.544 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/37 file after the snapshot 2023-06-22T08:02:15.544 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/38 file after the snapshot 2023-06-22T08:02:15.544 INFO:tasks.workunit.client.0.smithi023.stdout: create 214/39 file after the snapshot 2023-06-22T08:02:15.544 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/.snap/snap-subdir-test 2023-06-22T08:02:15.573 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/30 file after the snapshot 2023-06-22T08:02:15.575 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/31 file after the snapshot 2023-06-22T08:02:15.575 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/32 file after the snapshot 2023-06-22T08:02:15.575 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/33 file after the snapshot 2023-06-22T08:02:15.575 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/34 file after the snapshot 2023-06-22T08:02:15.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/35 file after the snapshot 2023-06-22T08:02:15.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/36 file after the snapshot 2023-06-22T08:02:15.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/37 file after the snapshot 2023-06-22T08:02:15.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/38 file after the snapshot 2023-06-22T08:02:15.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 215/39 file after the snapshot 2023-06-22T08:02:15.576 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/.snap/snap-subdir-test 2023-06-22T08:02:15.604 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/30 file after the snapshot 2023-06-22T08:02:15.605 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/31 file after the snapshot 2023-06-22T08:02:15.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/32 file after the snapshot 2023-06-22T08:02:15.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/33 file after the snapshot 2023-06-22T08:02:15.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/34 file after the snapshot 2023-06-22T08:02:15.606 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/35 file after the snapshot 2023-06-22T08:02:15.607 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/36 file after the snapshot 2023-06-22T08:02:15.607 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/37 file after the snapshot 2023-06-22T08:02:15.607 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/38 file after the snapshot 2023-06-22T08:02:15.607 INFO:tasks.workunit.client.0.smithi023.stdout: create 216/39 file after the snapshot 2023-06-22T08:02:15.607 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/.snap/snap-subdir-test 2023-06-22T08:02:15.634 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/30 file after the snapshot 2023-06-22T08:02:15.636 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/31 file after the snapshot 2023-06-22T08:02:15.636 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/32 file after the snapshot 2023-06-22T08:02:15.636 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/33 file after the snapshot 2023-06-22T08:02:15.636 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/34 file after the snapshot 2023-06-22T08:02:15.636 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/35 file after the snapshot 2023-06-22T08:02:15.636 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/36 file after the snapshot 2023-06-22T08:02:15.636 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/37 file after the snapshot 2023-06-22T08:02:15.639 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/38 file after the snapshot 2023-06-22T08:02:15.639 INFO:tasks.workunit.client.0.smithi023.stdout: create 217/39 file after the snapshot 2023-06-22T08:02:15.639 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/.snap/snap-subdir-test 2023-06-22T08:02:15.663 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/30 file after the snapshot 2023-06-22T08:02:15.665 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/31 file after the snapshot 2023-06-22T08:02:15.665 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/32 file after the snapshot 2023-06-22T08:02:15.665 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/33 file after the snapshot 2023-06-22T08:02:15.665 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/34 file after the snapshot 2023-06-22T08:02:15.666 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/35 file after the snapshot 2023-06-22T08:02:15.666 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/36 file after the snapshot 2023-06-22T08:02:15.666 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/37 file after the snapshot 2023-06-22T08:02:15.666 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/38 file after the snapshot 2023-06-22T08:02:15.667 INFO:tasks.workunit.client.0.smithi023.stdout: create 218/39 file after the snapshot 2023-06-22T08:02:15.667 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/.snap/snap-subdir-test 2023-06-22T08:02:15.692 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/30 file after the snapshot 2023-06-22T08:02:15.694 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/31 file after the snapshot 2023-06-22T08:02:15.694 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/32 file after the snapshot 2023-06-22T08:02:15.694 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/33 file after the snapshot 2023-06-22T08:02:15.694 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/34 file after the snapshot 2023-06-22T08:02:15.694 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/35 file after the snapshot 2023-06-22T08:02:15.695 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/36 file after the snapshot 2023-06-22T08:02:15.695 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/37 file after the snapshot 2023-06-22T08:02:15.695 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/38 file after the snapshot 2023-06-22T08:02:15.695 INFO:tasks.workunit.client.0.smithi023.stdout: create 219/39 file after the snapshot 2023-06-22T08:02:15.695 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/.snap/snap-subdir-test 2023-06-22T08:02:15.723 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/30 file after the snapshot 2023-06-22T08:02:15.728 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/31 file after the snapshot 2023-06-22T08:02:15.728 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/32 file after the snapshot 2023-06-22T08:02:15.728 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/33 file after the snapshot 2023-06-22T08:02:15.729 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/34 file after the snapshot 2023-06-22T08:02:15.729 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/35 file after the snapshot 2023-06-22T08:02:15.729 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/36 file after the snapshot 2023-06-22T08:02:15.729 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/37 file after the snapshot 2023-06-22T08:02:15.730 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/38 file after the snapshot 2023-06-22T08:02:15.730 INFO:tasks.workunit.client.0.smithi023.stdout: create 220/39 file after the snapshot 2023-06-22T08:02:15.730 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/.snap/snap-subdir-test 2023-06-22T08:02:15.757 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/30 file after the snapshot 2023-06-22T08:02:15.758 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/31 file after the snapshot 2023-06-22T08:02:15.759 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/32 file after the snapshot 2023-06-22T08:02:15.759 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/33 file after the snapshot 2023-06-22T08:02:15.759 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/34 file after the snapshot 2023-06-22T08:02:15.759 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/35 file after the snapshot 2023-06-22T08:02:15.759 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/36 file after the snapshot 2023-06-22T08:02:15.759 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/37 file after the snapshot 2023-06-22T08:02:15.759 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/38 file after the snapshot 2023-06-22T08:02:15.760 INFO:tasks.workunit.client.0.smithi023.stdout: create 221/39 file after the snapshot 2023-06-22T08:02:15.760 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/.snap/snap-subdir-test 2023-06-22T08:02:15.791 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/30 file after the snapshot 2023-06-22T08:02:15.793 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/31 file after the snapshot 2023-06-22T08:02:15.793 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/32 file after the snapshot 2023-06-22T08:02:15.793 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/33 file after the snapshot 2023-06-22T08:02:15.793 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/34 file after the snapshot 2023-06-22T08:02:15.793 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/35 file after the snapshot 2023-06-22T08:02:15.794 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/36 file after the snapshot 2023-06-22T08:02:15.794 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/37 file after the snapshot 2023-06-22T08:02:15.794 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/38 file after the snapshot 2023-06-22T08:02:15.794 INFO:tasks.workunit.client.0.smithi023.stdout: create 222/39 file after the snapshot 2023-06-22T08:02:15.794 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/.snap/snap-subdir-test 2023-06-22T08:02:15.828 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/30 file after the snapshot 2023-06-22T08:02:15.831 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/31 file after the snapshot 2023-06-22T08:02:15.831 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/32 file after the snapshot 2023-06-22T08:02:15.832 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/33 file after the snapshot 2023-06-22T08:02:15.832 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/34 file after the snapshot 2023-06-22T08:02:15.832 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/35 file after the snapshot 2023-06-22T08:02:15.833 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/36 file after the snapshot 2023-06-22T08:02:15.833 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/37 file after the snapshot 2023-06-22T08:02:15.833 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/38 file after the snapshot 2023-06-22T08:02:15.833 INFO:tasks.workunit.client.0.smithi023.stdout: create 223/39 file after the snapshot 2023-06-22T08:02:15.834 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/.snap/snap-subdir-test 2023-06-22T08:02:15.865 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/30 file after the snapshot 2023-06-22T08:02:15.868 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/31 file after the snapshot 2023-06-22T08:02:15.868 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/32 file after the snapshot 2023-06-22T08:02:15.869 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/33 file after the snapshot 2023-06-22T08:02:15.869 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/34 file after the snapshot 2023-06-22T08:02:15.869 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/35 file after the snapshot 2023-06-22T08:02:15.870 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/36 file after the snapshot 2023-06-22T08:02:15.870 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/37 file after the snapshot 2023-06-22T08:02:15.870 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/38 file after the snapshot 2023-06-22T08:02:15.871 INFO:tasks.workunit.client.0.smithi023.stdout: create 224/39 file after the snapshot 2023-06-22T08:02:15.871 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/.snap/snap-subdir-test 2023-06-22T08:02:15.897 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/30 file after the snapshot 2023-06-22T08:02:15.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/31 file after the snapshot 2023-06-22T08:02:15.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/32 file after the snapshot 2023-06-22T08:02:15.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/33 file after the snapshot 2023-06-22T08:02:15.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/34 file after the snapshot 2023-06-22T08:02:15.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/35 file after the snapshot 2023-06-22T08:02:15.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/36 file after the snapshot 2023-06-22T08:02:15.899 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/37 file after the snapshot 2023-06-22T08:02:15.900 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/38 file after the snapshot 2023-06-22T08:02:15.900 INFO:tasks.workunit.client.0.smithi023.stdout: create 225/39 file after the snapshot 2023-06-22T08:02:15.900 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/.snap/snap-subdir-test 2023-06-22T08:02:15.926 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/30 file after the snapshot 2023-06-22T08:02:15.927 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/31 file after the snapshot 2023-06-22T08:02:15.928 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/32 file after the snapshot 2023-06-22T08:02:15.928 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/33 file after the snapshot 2023-06-22T08:02:15.928 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/34 file after the snapshot 2023-06-22T08:02:15.928 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/35 file after the snapshot 2023-06-22T08:02:15.928 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/36 file after the snapshot 2023-06-22T08:02:15.929 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/37 file after the snapshot 2023-06-22T08:02:15.929 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/38 file after the snapshot 2023-06-22T08:02:15.929 INFO:tasks.workunit.client.0.smithi023.stdout: create 226/39 file after the snapshot 2023-06-22T08:02:15.929 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/.snap/snap-subdir-test 2023-06-22T08:02:15.955 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/30 file after the snapshot 2023-06-22T08:02:15.956 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/31 file after the snapshot 2023-06-22T08:02:15.957 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/32 file after the snapshot 2023-06-22T08:02:15.957 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/33 file after the snapshot 2023-06-22T08:02:15.957 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/34 file after the snapshot 2023-06-22T08:02:15.957 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/35 file after the snapshot 2023-06-22T08:02:15.957 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/36 file after the snapshot 2023-06-22T08:02:15.957 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/37 file after the snapshot 2023-06-22T08:02:15.958 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/38 file after the snapshot 2023-06-22T08:02:15.958 INFO:tasks.workunit.client.0.smithi023.stdout: create 227/39 file after the snapshot 2023-06-22T08:02:15.958 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/.snap/snap-subdir-test 2023-06-22T08:02:15.984 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/30 file after the snapshot 2023-06-22T08:02:15.986 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/31 file after the snapshot 2023-06-22T08:02:15.986 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/32 file after the snapshot 2023-06-22T08:02:15.987 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/33 file after the snapshot 2023-06-22T08:02:15.987 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/34 file after the snapshot 2023-06-22T08:02:15.987 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/35 file after the snapshot 2023-06-22T08:02:15.987 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/36 file after the snapshot 2023-06-22T08:02:15.987 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/37 file after the snapshot 2023-06-22T08:02:15.988 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/38 file after the snapshot 2023-06-22T08:02:15.988 INFO:tasks.workunit.client.0.smithi023.stdout: create 228/39 file after the snapshot 2023-06-22T08:02:15.988 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/.snap/snap-subdir-test 2023-06-22T08:02:16.013 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/30 file after the snapshot 2023-06-22T08:02:16.014 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/31 file after the snapshot 2023-06-22T08:02:16.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/32 file after the snapshot 2023-06-22T08:02:16.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/33 file after the snapshot 2023-06-22T08:02:16.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/34 file after the snapshot 2023-06-22T08:02:16.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/35 file after the snapshot 2023-06-22T08:02:16.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/36 file after the snapshot 2023-06-22T08:02:16.015 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/37 file after the snapshot 2023-06-22T08:02:16.016 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/38 file after the snapshot 2023-06-22T08:02:16.016 INFO:tasks.workunit.client.0.smithi023.stdout: create 229/39 file after the snapshot 2023-06-22T08:02:16.016 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/.snap/snap-subdir-test 2023-06-22T08:02:16.049 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/30 file after the snapshot 2023-06-22T08:02:16.051 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/31 file after the snapshot 2023-06-22T08:02:16.051 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/32 file after the snapshot 2023-06-22T08:02:16.051 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/33 file after the snapshot 2023-06-22T08:02:16.051 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/34 file after the snapshot 2023-06-22T08:02:16.051 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/35 file after the snapshot 2023-06-22T08:02:16.052 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/36 file after the snapshot 2023-06-22T08:02:16.052 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/37 file after the snapshot 2023-06-22T08:02:16.052 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/38 file after the snapshot 2023-06-22T08:02:16.052 INFO:tasks.workunit.client.0.smithi023.stdout: create 230/39 file after the snapshot 2023-06-22T08:02:16.052 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/.snap/snap-subdir-test 2023-06-22T08:02:16.086 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/30 file after the snapshot 2023-06-22T08:02:16.087 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/31 file after the snapshot 2023-06-22T08:02:16.087 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/32 file after the snapshot 2023-06-22T08:02:16.087 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/33 file after the snapshot 2023-06-22T08:02:16.088 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/34 file after the snapshot 2023-06-22T08:02:16.088 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/35 file after the snapshot 2023-06-22T08:02:16.088 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/36 file after the snapshot 2023-06-22T08:02:16.088 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/37 file after the snapshot 2023-06-22T08:02:16.089 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/38 file after the snapshot 2023-06-22T08:02:16.089 INFO:tasks.workunit.client.0.smithi023.stdout: create 231/39 file after the snapshot 2023-06-22T08:02:16.089 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/.snap/snap-subdir-test 2023-06-22T08:02:16.118 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/30 file after the snapshot 2023-06-22T08:02:16.120 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/31 file after the snapshot 2023-06-22T08:02:16.120 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/32 file after the snapshot 2023-06-22T08:02:16.120 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/33 file after the snapshot 2023-06-22T08:02:16.120 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/34 file after the snapshot 2023-06-22T08:02:16.120 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/35 file after the snapshot 2023-06-22T08:02:16.121 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/36 file after the snapshot 2023-06-22T08:02:16.121 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/37 file after the snapshot 2023-06-22T08:02:16.121 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/38 file after the snapshot 2023-06-22T08:02:16.121 INFO:tasks.workunit.client.0.smithi023.stdout: create 232/39 file after the snapshot 2023-06-22T08:02:16.122 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/.snap/snap-subdir-test 2023-06-22T08:02:16.148 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/30 file after the snapshot 2023-06-22T08:02:16.149 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/31 file after the snapshot 2023-06-22T08:02:16.149 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/32 file after the snapshot 2023-06-22T08:02:16.149 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/33 file after the snapshot 2023-06-22T08:02:16.149 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/34 file after the snapshot 2023-06-22T08:02:16.150 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/35 file after the snapshot 2023-06-22T08:02:16.150 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/36 file after the snapshot 2023-06-22T08:02:16.150 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/37 file after the snapshot 2023-06-22T08:02:16.150 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/38 file after the snapshot 2023-06-22T08:02:16.151 INFO:tasks.workunit.client.0.smithi023.stdout: create 233/39 file after the snapshot 2023-06-22T08:02:16.151 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/.snap/snap-subdir-test 2023-06-22T08:02:16.177 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/30 file after the snapshot 2023-06-22T08:02:16.178 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/31 file after the snapshot 2023-06-22T08:02:16.178 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/32 file after the snapshot 2023-06-22T08:02:16.178 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/33 file after the snapshot 2023-06-22T08:02:16.179 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/34 file after the snapshot 2023-06-22T08:02:16.179 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/35 file after the snapshot 2023-06-22T08:02:16.179 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/36 file after the snapshot 2023-06-22T08:02:16.179 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/37 file after the snapshot 2023-06-22T08:02:16.180 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/38 file after the snapshot 2023-06-22T08:02:16.180 INFO:tasks.workunit.client.0.smithi023.stdout: create 234/39 file after the snapshot 2023-06-22T08:02:16.180 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/.snap/snap-subdir-test 2023-06-22T08:02:16.205 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/30 file after the snapshot 2023-06-22T08:02:16.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/31 file after the snapshot 2023-06-22T08:02:16.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/32 file after the snapshot 2023-06-22T08:02:16.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/33 file after the snapshot 2023-06-22T08:02:16.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/34 file after the snapshot 2023-06-22T08:02:16.208 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/35 file after the snapshot 2023-06-22T08:02:16.208 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/36 file after the snapshot 2023-06-22T08:02:16.208 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/37 file after the snapshot 2023-06-22T08:02:16.208 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/38 file after the snapshot 2023-06-22T08:02:16.209 INFO:tasks.workunit.client.0.smithi023.stdout: create 235/39 file after the snapshot 2023-06-22T08:02:16.209 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/.snap/snap-subdir-test 2023-06-22T08:02:16.234 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/30 file after the snapshot 2023-06-22T08:02:16.235 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/31 file after the snapshot 2023-06-22T08:02:16.235 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/32 file after the snapshot 2023-06-22T08:02:16.235 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/33 file after the snapshot 2023-06-22T08:02:16.236 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/34 file after the snapshot 2023-06-22T08:02:16.236 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/35 file after the snapshot 2023-06-22T08:02:16.237 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/36 file after the snapshot 2023-06-22T08:02:16.237 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/37 file after the snapshot 2023-06-22T08:02:16.237 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/38 file after the snapshot 2023-06-22T08:02:16.237 INFO:tasks.workunit.client.0.smithi023.stdout: create 236/39 file after the snapshot 2023-06-22T08:02:16.238 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/.snap/snap-subdir-test 2023-06-22T08:02:16.263 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/30 file after the snapshot 2023-06-22T08:02:16.265 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/31 file after the snapshot 2023-06-22T08:02:16.265 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/32 file after the snapshot 2023-06-22T08:02:16.266 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/33 file after the snapshot 2023-06-22T08:02:16.266 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/34 file after the snapshot 2023-06-22T08:02:16.266 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/35 file after the snapshot 2023-06-22T08:02:16.266 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/36 file after the snapshot 2023-06-22T08:02:16.267 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/37 file after the snapshot 2023-06-22T08:02:16.267 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/38 file after the snapshot 2023-06-22T08:02:16.267 INFO:tasks.workunit.client.0.smithi023.stdout: create 237/39 file after the snapshot 2023-06-22T08:02:16.267 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/.snap/snap-subdir-test 2023-06-22T08:02:16.294 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/30 file after the snapshot 2023-06-22T08:02:16.296 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/31 file after the snapshot 2023-06-22T08:02:16.296 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/32 file after the snapshot 2023-06-22T08:02:16.297 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/33 file after the snapshot 2023-06-22T08:02:16.297 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/34 file after the snapshot 2023-06-22T08:02:16.297 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/35 file after the snapshot 2023-06-22T08:02:16.297 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/36 file after the snapshot 2023-06-22T08:02:16.297 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/37 file after the snapshot 2023-06-22T08:02:16.297 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/38 file after the snapshot 2023-06-22T08:02:16.297 INFO:tasks.workunit.client.0.smithi023.stdout: create 238/39 file after the snapshot 2023-06-22T08:02:16.298 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/.snap/snap-subdir-test 2023-06-22T08:02:16.322 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/30 file after the snapshot 2023-06-22T08:02:16.324 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/31 file after the snapshot 2023-06-22T08:02:16.324 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/32 file after the snapshot 2023-06-22T08:02:16.325 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/33 file after the snapshot 2023-06-22T08:02:16.325 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/34 file after the snapshot 2023-06-22T08:02:16.325 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/35 file after the snapshot 2023-06-22T08:02:16.325 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/36 file after the snapshot 2023-06-22T08:02:16.325 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/37 file after the snapshot 2023-06-22T08:02:16.325 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/38 file after the snapshot 2023-06-22T08:02:16.326 INFO:tasks.workunit.client.0.smithi023.stdout: create 239/39 file after the snapshot 2023-06-22T08:02:16.326 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/.snap/snap-subdir-test 2023-06-22T08:02:16.352 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/30 file after the snapshot 2023-06-22T08:02:16.354 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/31 file after the snapshot 2023-06-22T08:02:16.354 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/32 file after the snapshot 2023-06-22T08:02:16.354 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/33 file after the snapshot 2023-06-22T08:02:16.354 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/34 file after the snapshot 2023-06-22T08:02:16.354 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/35 file after the snapshot 2023-06-22T08:02:16.354 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/36 file after the snapshot 2023-06-22T08:02:16.355 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/37 file after the snapshot 2023-06-22T08:02:16.355 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/38 file after the snapshot 2023-06-22T08:02:16.355 INFO:tasks.workunit.client.0.smithi023.stdout: create 240/39 file after the snapshot 2023-06-22T08:02:16.355 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/.snap/snap-subdir-test 2023-06-22T08:02:16.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/30 file after the snapshot 2023-06-22T08:02:16.383 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/31 file after the snapshot 2023-06-22T08:02:16.383 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/32 file after the snapshot 2023-06-22T08:02:16.383 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/33 file after the snapshot 2023-06-22T08:02:16.383 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/34 file after the snapshot 2023-06-22T08:02:16.383 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/35 file after the snapshot 2023-06-22T08:02:16.383 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/36 file after the snapshot 2023-06-22T08:02:16.383 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/37 file after the snapshot 2023-06-22T08:02:16.384 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/38 file after the snapshot 2023-06-22T08:02:16.384 INFO:tasks.workunit.client.0.smithi023.stdout: create 241/39 file after the snapshot 2023-06-22T08:02:16.384 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/.snap/snap-subdir-test 2023-06-22T08:02:16.417 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/30 file after the snapshot 2023-06-22T08:02:16.419 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/31 file after the snapshot 2023-06-22T08:02:16.419 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/32 file after the snapshot 2023-06-22T08:02:16.420 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/33 file after the snapshot 2023-06-22T08:02:16.420 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/34 file after the snapshot 2023-06-22T08:02:16.420 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/35 file after the snapshot 2023-06-22T08:02:16.420 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/36 file after the snapshot 2023-06-22T08:02:16.420 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/37 file after the snapshot 2023-06-22T08:02:16.420 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/38 file after the snapshot 2023-06-22T08:02:16.421 INFO:tasks.workunit.client.0.smithi023.stdout: create 242/39 file after the snapshot 2023-06-22T08:02:16.421 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/.snap/snap-subdir-test 2023-06-22T08:02:16.453 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/30 file after the snapshot 2023-06-22T08:02:16.456 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/31 file after the snapshot 2023-06-22T08:02:16.456 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/32 file after the snapshot 2023-06-22T08:02:16.456 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/33 file after the snapshot 2023-06-22T08:02:16.456 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/34 file after the snapshot 2023-06-22T08:02:16.456 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/35 file after the snapshot 2023-06-22T08:02:16.457 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/36 file after the snapshot 2023-06-22T08:02:16.457 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/37 file after the snapshot 2023-06-22T08:02:16.457 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/38 file after the snapshot 2023-06-22T08:02:16.457 INFO:tasks.workunit.client.0.smithi023.stdout: create 243/39 file after the snapshot 2023-06-22T08:02:16.457 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/.snap/snap-subdir-test 2023-06-22T08:02:16.490 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/30 file after the snapshot 2023-06-22T08:02:16.492 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/31 file after the snapshot 2023-06-22T08:02:16.492 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/32 file after the snapshot 2023-06-22T08:02:16.493 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/33 file after the snapshot 2023-06-22T08:02:16.493 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/34 file after the snapshot 2023-06-22T08:02:16.493 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/35 file after the snapshot 2023-06-22T08:02:16.493 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/36 file after the snapshot 2023-06-22T08:02:16.493 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/37 file after the snapshot 2023-06-22T08:02:16.494 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/38 file after the snapshot 2023-06-22T08:02:16.494 INFO:tasks.workunit.client.0.smithi023.stdout: create 244/39 file after the snapshot 2023-06-22T08:02:16.494 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/.snap/snap-subdir-test 2023-06-22T08:02:16.526 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/30 file after the snapshot 2023-06-22T08:02:16.528 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/31 file after the snapshot 2023-06-22T08:02:16.529 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/32 file after the snapshot 2023-06-22T08:02:16.529 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/33 file after the snapshot 2023-06-22T08:02:16.529 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/34 file after the snapshot 2023-06-22T08:02:16.529 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/35 file after the snapshot 2023-06-22T08:02:16.529 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/36 file after the snapshot 2023-06-22T08:02:16.530 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/37 file after the snapshot 2023-06-22T08:02:16.530 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/38 file after the snapshot 2023-06-22T08:02:16.530 INFO:tasks.workunit.client.0.smithi023.stdout: create 245/39 file after the snapshot 2023-06-22T08:02:16.530 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/.snap/snap-subdir-test 2023-06-22T08:02:16.562 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/30 file after the snapshot 2023-06-22T08:02:16.565 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/31 file after the snapshot 2023-06-22T08:02:16.565 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/32 file after the snapshot 2023-06-22T08:02:16.565 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/33 file after the snapshot 2023-06-22T08:02:16.565 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/34 file after the snapshot 2023-06-22T08:02:16.565 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/35 file after the snapshot 2023-06-22T08:02:16.566 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/36 file after the snapshot 2023-06-22T08:02:16.566 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/37 file after the snapshot 2023-06-22T08:02:16.566 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/38 file after the snapshot 2023-06-22T08:02:16.566 INFO:tasks.workunit.client.0.smithi023.stdout: create 246/39 file after the snapshot 2023-06-22T08:02:16.566 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/.snap/snap-subdir-test 2023-06-22T08:02:16.592 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/30 file after the snapshot 2023-06-22T08:02:16.594 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/31 file after the snapshot 2023-06-22T08:02:16.595 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/32 file after the snapshot 2023-06-22T08:02:16.595 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/33 file after the snapshot 2023-06-22T08:02:16.595 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/34 file after the snapshot 2023-06-22T08:02:16.596 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/35 file after the snapshot 2023-06-22T08:02:16.596 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/36 file after the snapshot 2023-06-22T08:02:16.596 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/37 file after the snapshot 2023-06-22T08:02:16.596 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/38 file after the snapshot 2023-06-22T08:02:16.597 INFO:tasks.workunit.client.0.smithi023.stdout: create 247/39 file after the snapshot 2023-06-22T08:02:16.597 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/.snap/snap-subdir-test 2023-06-22T08:02:16.623 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/30 file after the snapshot 2023-06-22T08:02:16.625 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/31 file after the snapshot 2023-06-22T08:02:16.626 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/32 file after the snapshot 2023-06-22T08:02:16.626 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/33 file after the snapshot 2023-06-22T08:02:16.626 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/34 file after the snapshot 2023-06-22T08:02:16.627 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/35 file after the snapshot 2023-06-22T08:02:16.627 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/36 file after the snapshot 2023-06-22T08:02:16.627 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/37 file after the snapshot 2023-06-22T08:02:16.628 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/38 file after the snapshot 2023-06-22T08:02:16.628 INFO:tasks.workunit.client.0.smithi023.stdout: create 248/39 file after the snapshot 2023-06-22T08:02:16.628 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/.snap/snap-subdir-test 2023-06-22T08:02:16.658 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/30 file after the snapshot 2023-06-22T08:02:16.659 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/31 file after the snapshot 2023-06-22T08:02:16.660 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/32 file after the snapshot 2023-06-22T08:02:16.660 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/33 file after the snapshot 2023-06-22T08:02:16.660 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/34 file after the snapshot 2023-06-22T08:02:16.660 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/35 file after the snapshot 2023-06-22T08:02:16.660 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/36 file after the snapshot 2023-06-22T08:02:16.660 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/37 file after the snapshot 2023-06-22T08:02:16.661 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/38 file after the snapshot 2023-06-22T08:02:16.661 INFO:tasks.workunit.client.0.smithi023.stdout: create 249/39 file after the snapshot 2023-06-22T08:02:16.661 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/.snap/snap-subdir-test 2023-06-22T08:02:16.687 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/30 file after the snapshot 2023-06-22T08:02:16.689 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/31 file after the snapshot 2023-06-22T08:02:16.689 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/32 file after the snapshot 2023-06-22T08:02:16.690 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/33 file after the snapshot 2023-06-22T08:02:16.690 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/34 file after the snapshot 2023-06-22T08:02:16.690 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/35 file after the snapshot 2023-06-22T08:02:16.690 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/36 file after the snapshot 2023-06-22T08:02:16.691 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/37 file after the snapshot 2023-06-22T08:02:16.691 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/38 file after the snapshot 2023-06-22T08:02:16.691 INFO:tasks.workunit.client.0.smithi023.stdout: create 250/39 file after the snapshot 2023-06-22T08:02:16.692 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/.snap/snap-subdir-test 2023-06-22T08:02:16.716 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/30 file after the snapshot 2023-06-22T08:02:16.718 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/31 file after the snapshot 2023-06-22T08:02:16.718 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/32 file after the snapshot 2023-06-22T08:02:16.718 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/33 file after the snapshot 2023-06-22T08:02:16.719 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/34 file after the snapshot 2023-06-22T08:02:16.719 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/35 file after the snapshot 2023-06-22T08:02:16.719 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/36 file after the snapshot 2023-06-22T08:02:16.719 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/37 file after the snapshot 2023-06-22T08:02:16.720 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/38 file after the snapshot 2023-06-22T08:02:16.720 INFO:tasks.workunit.client.0.smithi023.stdout: create 251/39 file after the snapshot 2023-06-22T08:02:16.720 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/.snap/snap-subdir-test 2023-06-22T08:02:16.747 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/30 file after the snapshot 2023-06-22T08:02:16.748 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/31 file after the snapshot 2023-06-22T08:02:16.748 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/32 file after the snapshot 2023-06-22T08:02:16.748 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/33 file after the snapshot 2023-06-22T08:02:16.749 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/34 file after the snapshot 2023-06-22T08:02:16.749 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/35 file after the snapshot 2023-06-22T08:02:16.749 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/36 file after the snapshot 2023-06-22T08:02:16.749 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/37 file after the snapshot 2023-06-22T08:02:16.750 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/38 file after the snapshot 2023-06-22T08:02:16.750 INFO:tasks.workunit.client.0.smithi023.stdout: create 252/39 file after the snapshot 2023-06-22T08:02:16.750 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/.snap/snap-subdir-test 2023-06-22T08:02:16.776 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/30 file after the snapshot 2023-06-22T08:02:16.778 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/31 file after the snapshot 2023-06-22T08:02:16.778 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/32 file after the snapshot 2023-06-22T08:02:16.778 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/33 file after the snapshot 2023-06-22T08:02:16.778 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/34 file after the snapshot 2023-06-22T08:02:16.778 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/35 file after the snapshot 2023-06-22T08:02:16.778 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/36 file after the snapshot 2023-06-22T08:02:16.779 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/37 file after the snapshot 2023-06-22T08:02:16.779 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/38 file after the snapshot 2023-06-22T08:02:16.779 INFO:tasks.workunit.client.0.smithi023.stdout: create 253/39 file after the snapshot 2023-06-22T08:02:16.779 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/.snap/snap-subdir-test 2023-06-22T08:02:16.807 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/30 file after the snapshot 2023-06-22T08:02:16.808 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/31 file after the snapshot 2023-06-22T08:02:16.809 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/32 file after the snapshot 2023-06-22T08:02:16.809 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/33 file after the snapshot 2023-06-22T08:02:16.809 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/34 file after the snapshot 2023-06-22T08:02:16.809 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/35 file after the snapshot 2023-06-22T08:02:16.809 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/36 file after the snapshot 2023-06-22T08:02:16.810 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/37 file after the snapshot 2023-06-22T08:02:16.810 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/38 file after the snapshot 2023-06-22T08:02:16.810 INFO:tasks.workunit.client.0.smithi023.stdout: create 254/39 file after the snapshot 2023-06-22T08:02:16.810 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/.snap/snap-subdir-test 2023-06-22T08:02:16.843 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/30 file after the snapshot 2023-06-22T08:02:16.845 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/31 file after the snapshot 2023-06-22T08:02:16.845 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/32 file after the snapshot 2023-06-22T08:02:16.845 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/33 file after the snapshot 2023-06-22T08:02:16.846 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/34 file after the snapshot 2023-06-22T08:02:16.846 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/35 file after the snapshot 2023-06-22T08:02:16.846 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/36 file after the snapshot 2023-06-22T08:02:16.847 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/37 file after the snapshot 2023-06-22T08:02:16.847 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/38 file after the snapshot 2023-06-22T08:02:16.847 INFO:tasks.workunit.client.0.smithi023.stdout: create 255/39 file after the snapshot 2023-06-22T08:02:16.847 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/.snap/snap-subdir-test 2023-06-22T08:02:16.878 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/30 file after the snapshot 2023-06-22T08:02:16.881 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/31 file after the snapshot 2023-06-22T08:02:16.882 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/32 file after the snapshot 2023-06-22T08:02:16.882 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/33 file after the snapshot 2023-06-22T08:02:16.882 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/34 file after the snapshot 2023-06-22T08:02:16.882 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/35 file after the snapshot 2023-06-22T08:02:16.883 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/36 file after the snapshot 2023-06-22T08:02:16.883 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/37 file after the snapshot 2023-06-22T08:02:16.883 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/38 file after the snapshot 2023-06-22T08:02:16.883 INFO:tasks.workunit.client.0.smithi023.stdout: create 256/39 file after the snapshot 2023-06-22T08:02:16.884 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/.snap/snap-subdir-test 2023-06-22T08:02:16.909 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/30 file after the snapshot 2023-06-22T08:02:16.910 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/31 file after the snapshot 2023-06-22T08:02:16.910 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/32 file after the snapshot 2023-06-22T08:02:16.910 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/33 file after the snapshot 2023-06-22T08:02:16.911 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/34 file after the snapshot 2023-06-22T08:02:16.911 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/35 file after the snapshot 2023-06-22T08:02:16.911 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/36 file after the snapshot 2023-06-22T08:02:16.911 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/37 file after the snapshot 2023-06-22T08:02:16.912 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/38 file after the snapshot 2023-06-22T08:02:16.912 INFO:tasks.workunit.client.0.smithi023.stdout: create 257/39 file after the snapshot 2023-06-22T08:02:16.912 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/.snap/snap-subdir-test 2023-06-22T08:02:16.937 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/30 file after the snapshot 2023-06-22T08:02:16.938 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/31 file after the snapshot 2023-06-22T08:02:16.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/32 file after the snapshot 2023-06-22T08:02:16.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/33 file after the snapshot 2023-06-22T08:02:16.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/34 file after the snapshot 2023-06-22T08:02:16.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/35 file after the snapshot 2023-06-22T08:02:16.939 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/36 file after the snapshot 2023-06-22T08:02:16.940 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/37 file after the snapshot 2023-06-22T08:02:16.940 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/38 file after the snapshot 2023-06-22T08:02:16.940 INFO:tasks.workunit.client.0.smithi023.stdout: create 258/39 file after the snapshot 2023-06-22T08:02:16.941 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/.snap/snap-subdir-test 2023-06-22T08:02:16.966 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/30 file after the snapshot 2023-06-22T08:02:16.968 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/31 file after the snapshot 2023-06-22T08:02:16.968 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/32 file after the snapshot 2023-06-22T08:02:16.968 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/33 file after the snapshot 2023-06-22T08:02:16.968 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/34 file after the snapshot 2023-06-22T08:02:16.968 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/35 file after the snapshot 2023-06-22T08:02:16.968 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/36 file after the snapshot 2023-06-22T08:02:16.969 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/37 file after the snapshot 2023-06-22T08:02:16.969 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/38 file after the snapshot 2023-06-22T08:02:16.969 INFO:tasks.workunit.client.0.smithi023.stdout: create 259/39 file after the snapshot 2023-06-22T08:02:16.969 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/.snap/snap-subdir-test 2023-06-22T08:02:17.001 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/30 file after the snapshot 2023-06-22T08:02:17.003 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/31 file after the snapshot 2023-06-22T08:02:17.003 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/32 file after the snapshot 2023-06-22T08:02:17.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/33 file after the snapshot 2023-06-22T08:02:17.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/34 file after the snapshot 2023-06-22T08:02:17.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/35 file after the snapshot 2023-06-22T08:02:17.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/36 file after the snapshot 2023-06-22T08:02:17.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/37 file after the snapshot 2023-06-22T08:02:17.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/38 file after the snapshot 2023-06-22T08:02:17.004 INFO:tasks.workunit.client.0.smithi023.stdout: create 260/39 file after the snapshot 2023-06-22T08:02:17.005 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/.snap/snap-subdir-test 2023-06-22T08:02:17.037 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/30 file after the snapshot 2023-06-22T08:02:17.039 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/31 file after the snapshot 2023-06-22T08:02:17.039 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/32 file after the snapshot 2023-06-22T08:02:17.039 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/33 file after the snapshot 2023-06-22T08:02:17.039 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/34 file after the snapshot 2023-06-22T08:02:17.039 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/35 file after the snapshot 2023-06-22T08:02:17.039 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/36 file after the snapshot 2023-06-22T08:02:17.039 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/37 file after the snapshot 2023-06-22T08:02:17.040 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/38 file after the snapshot 2023-06-22T08:02:17.040 INFO:tasks.workunit.client.0.smithi023.stdout: create 261/39 file after the snapshot 2023-06-22T08:02:17.040 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/.snap/snap-subdir-test 2023-06-22T08:02:17.072 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/30 file after the snapshot 2023-06-22T08:02:17.074 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/31 file after the snapshot 2023-06-22T08:02:17.074 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/32 file after the snapshot 2023-06-22T08:02:17.074 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/33 file after the snapshot 2023-06-22T08:02:17.074 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/34 file after the snapshot 2023-06-22T08:02:17.074 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/35 file after the snapshot 2023-06-22T08:02:17.075 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/36 file after the snapshot 2023-06-22T08:02:17.075 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/37 file after the snapshot 2023-06-22T08:02:17.075 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/38 file after the snapshot 2023-06-22T08:02:17.075 INFO:tasks.workunit.client.0.smithi023.stdout: create 262/39 file after the snapshot 2023-06-22T08:02:17.076 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/.snap/snap-subdir-test 2023-06-22T08:02:17.107 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/30 file after the snapshot 2023-06-22T08:02:17.109 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/31 file after the snapshot 2023-06-22T08:02:17.109 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/32 file after the snapshot 2023-06-22T08:02:17.109 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/33 file after the snapshot 2023-06-22T08:02:17.110 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/34 file after the snapshot 2023-06-22T08:02:17.110 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/35 file after the snapshot 2023-06-22T08:02:17.110 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/36 file after the snapshot 2023-06-22T08:02:17.110 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/37 file after the snapshot 2023-06-22T08:02:17.110 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/38 file after the snapshot 2023-06-22T08:02:17.110 INFO:tasks.workunit.client.0.smithi023.stdout: create 263/39 file after the snapshot 2023-06-22T08:02:17.111 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/.snap/snap-subdir-test 2023-06-22T08:02:17.143 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/30 file after the snapshot 2023-06-22T08:02:17.145 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/31 file after the snapshot 2023-06-22T08:02:17.145 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/32 file after the snapshot 2023-06-22T08:02:17.145 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/33 file after the snapshot 2023-06-22T08:02:17.145 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/34 file after the snapshot 2023-06-22T08:02:17.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/35 file after the snapshot 2023-06-22T08:02:17.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/36 file after the snapshot 2023-06-22T08:02:17.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/37 file after the snapshot 2023-06-22T08:02:17.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/38 file after the snapshot 2023-06-22T08:02:17.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 264/39 file after the snapshot 2023-06-22T08:02:17.146 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/.snap/snap-subdir-test 2023-06-22T08:02:17.179 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/30 file after the snapshot 2023-06-22T08:02:17.180 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/31 file after the snapshot 2023-06-22T08:02:17.180 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/32 file after the snapshot 2023-06-22T08:02:17.180 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/33 file after the snapshot 2023-06-22T08:02:17.180 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/34 file after the snapshot 2023-06-22T08:02:17.180 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/35 file after the snapshot 2023-06-22T08:02:17.180 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/36 file after the snapshot 2023-06-22T08:02:17.181 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/37 file after the snapshot 2023-06-22T08:02:17.181 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/38 file after the snapshot 2023-06-22T08:02:17.181 INFO:tasks.workunit.client.0.smithi023.stdout: create 265/39 file after the snapshot 2023-06-22T08:02:17.181 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/.snap/snap-subdir-test 2023-06-22T08:02:17.205 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/30 file after the snapshot 2023-06-22T08:02:17.206 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/31 file after the snapshot 2023-06-22T08:02:17.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/32 file after the snapshot 2023-06-22T08:02:17.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/33 file after the snapshot 2023-06-22T08:02:17.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/34 file after the snapshot 2023-06-22T08:02:17.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/35 file after the snapshot 2023-06-22T08:02:17.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/36 file after the snapshot 2023-06-22T08:02:17.207 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/37 file after the snapshot 2023-06-22T08:02:17.208 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/38 file after the snapshot 2023-06-22T08:02:17.208 INFO:tasks.workunit.client.0.smithi023.stdout: create 266/39 file after the snapshot 2023-06-22T08:02:17.208 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/.snap/snap-subdir-test 2023-06-22T08:02:17.233 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/30 file after the snapshot 2023-06-22T08:02:17.234 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/31 file after the snapshot 2023-06-22T08:02:17.235 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/32 file after the snapshot 2023-06-22T08:02:17.235 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/33 file after the snapshot 2023-06-22T08:02:17.235 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/34 file after the snapshot 2023-06-22T08:02:17.235 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/35 file after the snapshot 2023-06-22T08:02:17.235 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/36 file after the snapshot 2023-06-22T08:02:17.235 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/37 file after the snapshot 2023-06-22T08:02:17.236 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/38 file after the snapshot 2023-06-22T08:02:17.236 INFO:tasks.workunit.client.0.smithi023.stdout: create 267/39 file after the snapshot 2023-06-22T08:02:17.236 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/.snap/snap-subdir-test 2023-06-22T08:02:17.262 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/30 file after the snapshot 2023-06-22T08:02:17.263 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/31 file after the snapshot 2023-06-22T08:02:17.263 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/32 file after the snapshot 2023-06-22T08:02:17.264 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/33 file after the snapshot 2023-06-22T08:02:17.264 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/34 file after the snapshot 2023-06-22T08:02:17.264 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/35 file after the snapshot 2023-06-22T08:02:17.264 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/36 file after the snapshot 2023-06-22T08:02:17.265 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/37 file after the snapshot 2023-06-22T08:02:17.265 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/38 file after the snapshot 2023-06-22T08:02:17.265 INFO:tasks.workunit.client.0.smithi023.stdout: create 268/39 file after the snapshot 2023-06-22T08:02:17.265 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/.snap/snap-subdir-test 2023-06-22T08:02:17.291 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/30 file after the snapshot 2023-06-22T08:02:17.292 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/31 file after the snapshot 2023-06-22T08:02:17.293 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/32 file after the snapshot 2023-06-22T08:02:17.293 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/33 file after the snapshot 2023-06-22T08:02:17.293 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/34 file after the snapshot 2023-06-22T08:02:17.293 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/35 file after the snapshot 2023-06-22T08:02:17.293 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/36 file after the snapshot 2023-06-22T08:02:17.293 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/37 file after the snapshot 2023-06-22T08:02:17.293 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/38 file after the snapshot 2023-06-22T08:02:17.294 INFO:tasks.workunit.client.0.smithi023.stdout: create 269/39 file after the snapshot 2023-06-22T08:02:17.294 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/.snap/snap-subdir-test 2023-06-22T08:02:17.320 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/30 file after the snapshot 2023-06-22T08:02:17.322 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/31 file after the snapshot 2023-06-22T08:02:17.322 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/32 file after the snapshot 2023-06-22T08:02:17.323 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/33 file after the snapshot 2023-06-22T08:02:17.323 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/34 file after the snapshot 2023-06-22T08:02:17.323 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/35 file after the snapshot 2023-06-22T08:02:17.323 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/36 file after the snapshot 2023-06-22T08:02:17.324 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/37 file after the snapshot 2023-06-22T08:02:17.324 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/38 file after the snapshot 2023-06-22T08:02:17.324 INFO:tasks.workunit.client.0.smithi023.stdout: create 270/39 file after the snapshot 2023-06-22T08:02:17.325 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/.snap/snap-subdir-test 2023-06-22T08:02:17.349 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/30 file after the snapshot 2023-06-22T08:02:17.351 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/31 file after the snapshot 2023-06-22T08:02:17.352 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/32 file after the snapshot 2023-06-22T08:02:17.352 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/33 file after the snapshot 2023-06-22T08:02:17.352 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/34 file after the snapshot 2023-06-22T08:02:17.352 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/35 file after the snapshot 2023-06-22T08:02:17.353 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/36 file after the snapshot 2023-06-22T08:02:17.353 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/37 file after the snapshot 2023-06-22T08:02:17.353 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/38 file after the snapshot 2023-06-22T08:02:17.354 INFO:tasks.workunit.client.0.smithi023.stdout: create 271/39 file after the snapshot 2023-06-22T08:02:17.354 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/.snap/snap-subdir-test 2023-06-22T08:02:17.379 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/30 file after the snapshot 2023-06-22T08:02:17.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/31 file after the snapshot 2023-06-22T08:02:17.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/32 file after the snapshot 2023-06-22T08:02:17.381 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/33 file after the snapshot 2023-06-22T08:02:17.382 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/34 file after the snapshot 2023-06-22T08:02:17.382 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/35 file after the snapshot 2023-06-22T08:02:17.382 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/36 file after the snapshot 2023-06-22T08:02:17.382 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/37 file after the snapshot 2023-06-22T08:02:17.382 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/38 file after the snapshot 2023-06-22T08:02:17.382 INFO:tasks.workunit.client.0.smithi023.stdout: create 272/39 file after the snapshot 2023-06-22T08:02:17.382 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/.snap/snap-subdir-test 2023-06-22T08:02:17.414 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/30 file after the snapshot 2023-06-22T08:02:17.416 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/31 file after the snapshot 2023-06-22T08:02:17.416 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/32 file after the snapshot 2023-06-22T08:02:17.416 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/33 file after the snapshot 2023-06-22T08:02:17.417 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/34 file after the snapshot 2023-06-22T08:02:17.417 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/35 file after the snapshot 2023-06-22T08:02:17.417 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/36 file after the snapshot 2023-06-22T08:02:17.418 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/37 file after the snapshot 2023-06-22T08:02:17.418 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/38 file after the snapshot 2023-06-22T08:02:17.419 INFO:tasks.workunit.client.0.smithi023.stdout: create 273/39 file after the snapshot 2023-06-22T08:02:17.419 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/.snap/snap-subdir-test 2023-06-22T08:02:17.448 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/30 file after the snapshot 2023-06-22T08:02:17.450 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/31 file after the snapshot 2023-06-22T08:02:17.450 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/32 file after the snapshot 2023-06-22T08:02:17.451 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/33 file after the snapshot 2023-06-22T08:02:17.451 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/34 file after the snapshot 2023-06-22T08:02:17.451 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/35 file after the snapshot 2023-06-22T08:02:17.451 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/36 file after the snapshot 2023-06-22T08:02:17.451 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/37 file after the snapshot 2023-06-22T08:02:17.451 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/38 file after the snapshot 2023-06-22T08:02:17.451 INFO:tasks.workunit.client.0.smithi023.stdout: create 274/39 file after the snapshot 2023-06-22T08:02:17.452 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/.snap/snap-subdir-test 2023-06-22T08:02:17.476 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/30 file after the snapshot 2023-06-22T08:02:17.478 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/31 file after the snapshot 2023-06-22T08:02:17.478 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/32 file after the snapshot 2023-06-22T08:02:17.478 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/33 file after the snapshot 2023-06-22T08:02:17.478 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/34 file after the snapshot 2023-06-22T08:02:17.478 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/35 file after the snapshot 2023-06-22T08:02:17.478 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/36 file after the snapshot 2023-06-22T08:02:17.479 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/37 file after the snapshot 2023-06-22T08:02:17.479 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/38 file after the snapshot 2023-06-22T08:02:17.479 INFO:tasks.workunit.client.0.smithi023.stdout: create 275/39 file after the snapshot 2023-06-22T08:02:17.479 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/.snap/snap-subdir-test 2023-06-22T08:02:17.504 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/30 file after the snapshot 2023-06-22T08:02:17.506 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/31 file after the snapshot 2023-06-22T08:02:17.506 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/32 file after the snapshot 2023-06-22T08:02:17.506 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/33 file after the snapshot 2023-06-22T08:02:17.506 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/34 file after the snapshot 2023-06-22T08:02:17.507 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/35 file after the snapshot 2023-06-22T08:02:17.507 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/36 file after the snapshot 2023-06-22T08:02:17.507 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/37 file after the snapshot 2023-06-22T08:02:17.507 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/38 file after the snapshot 2023-06-22T08:02:17.508 INFO:tasks.workunit.client.0.smithi023.stdout: create 276/39 file after the snapshot 2023-06-22T08:02:17.508 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/.snap/snap-subdir-test 2023-06-22T08:02:17.533 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/30 file after the snapshot 2023-06-22T08:02:17.534 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/31 file after the snapshot 2023-06-22T08:02:17.535 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/32 file after the snapshot 2023-06-22T08:02:17.535 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/33 file after the snapshot 2023-06-22T08:02:17.535 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/34 file after the snapshot 2023-06-22T08:02:17.535 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/35 file after the snapshot 2023-06-22T08:02:17.535 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/36 file after the snapshot 2023-06-22T08:02:17.535 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/37 file after the snapshot 2023-06-22T08:02:17.536 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/38 file after the snapshot 2023-06-22T08:02:17.536 INFO:tasks.workunit.client.0.smithi023.stdout: create 277/39 file after the snapshot 2023-06-22T08:02:17.536 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/.snap/snap-subdir-test 2023-06-22T08:02:17.561 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/30 file after the snapshot 2023-06-22T08:02:17.562 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/31 file after the snapshot 2023-06-22T08:02:17.563 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/32 file after the snapshot 2023-06-22T08:02:17.563 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/33 file after the snapshot 2023-06-22T08:02:17.563 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/34 file after the snapshot 2023-06-22T08:02:17.563 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/35 file after the snapshot 2023-06-22T08:02:17.563 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/36 file after the snapshot 2023-06-22T08:02:17.563 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/37 file after the snapshot 2023-06-22T08:02:17.564 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/38 file after the snapshot 2023-06-22T08:02:17.564 INFO:tasks.workunit.client.0.smithi023.stdout: create 278/39 file after the snapshot 2023-06-22T08:02:17.564 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/.snap/snap-subdir-test 2023-06-22T08:02:17.589 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/30 file after the snapshot 2023-06-22T08:02:17.590 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/31 file after the snapshot 2023-06-22T08:02:17.590 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/32 file after the snapshot 2023-06-22T08:02:17.590 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/33 file after the snapshot 2023-06-22T08:02:17.591 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/34 file after the snapshot 2023-06-22T08:02:17.591 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/35 file after the snapshot 2023-06-22T08:02:17.591 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/36 file after the snapshot 2023-06-22T08:02:17.591 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/37 file after the snapshot 2023-06-22T08:02:17.591 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/38 file after the snapshot 2023-06-22T08:02:17.592 INFO:tasks.workunit.client.0.smithi023.stdout: create 279/39 file after the snapshot 2023-06-22T08:02:17.592 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/.snap/snap-subdir-test 2023-06-22T08:02:17.617 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/30 file after the snapshot 2023-06-22T08:02:17.618 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/31 file after the snapshot 2023-06-22T08:02:17.618 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/32 file after the snapshot 2023-06-22T08:02:17.619 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/33 file after the snapshot 2023-06-22T08:02:17.619 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/34 file after the snapshot 2023-06-22T08:02:17.619 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/35 file after the snapshot 2023-06-22T08:02:17.619 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/36 file after the snapshot 2023-06-22T08:02:17.619 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/37 file after the snapshot 2023-06-22T08:02:17.619 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/38 file after the snapshot 2023-06-22T08:02:17.620 INFO:tasks.workunit.client.0.smithi023.stdout: create 280/39 file after the snapshot 2023-06-22T08:02:17.620 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/.snap/snap-subdir-test 2023-06-22T08:02:17.645 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/30 file after the snapshot 2023-06-22T08:02:17.646 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/31 file after the snapshot 2023-06-22T08:02:17.646 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/32 file after the snapshot 2023-06-22T08:02:17.646 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/33 file after the snapshot 2023-06-22T08:02:17.647 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/34 file after the snapshot 2023-06-22T08:02:17.647 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/35 file after the snapshot 2023-06-22T08:02:17.647 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/36 file after the snapshot 2023-06-22T08:02:17.647 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/37 file after the snapshot 2023-06-22T08:02:17.647 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/38 file after the snapshot 2023-06-22T08:02:17.647 INFO:tasks.workunit.client.0.smithi023.stdout: create 281/39 file after the snapshot 2023-06-22T08:02:17.647 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/.snap/snap-subdir-test 2023-06-22T08:02:17.673 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/30 file after the snapshot 2023-06-22T08:02:17.675 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/31 file after the snapshot 2023-06-22T08:02:17.675 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/32 file after the snapshot 2023-06-22T08:02:17.675 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/33 file after the snapshot 2023-06-22T08:02:17.675 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/34 file after the snapshot 2023-06-22T08:02:17.675 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/35 file after the snapshot 2023-06-22T08:02:17.675 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/36 file after the snapshot 2023-06-22T08:02:17.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/37 file after the snapshot 2023-06-22T08:02:17.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/38 file after the snapshot 2023-06-22T08:02:17.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 282/39 file after the snapshot 2023-06-22T08:02:17.676 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/.snap/snap-subdir-test 2023-06-22T08:02:17.701 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/30 file after the snapshot 2023-06-22T08:02:17.703 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/31 file after the snapshot 2023-06-22T08:02:17.703 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/32 file after the snapshot 2023-06-22T08:02:17.703 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/33 file after the snapshot 2023-06-22T08:02:17.703 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/34 file after the snapshot 2023-06-22T08:02:17.703 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/35 file after the snapshot 2023-06-22T08:02:17.704 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/36 file after the snapshot 2023-06-22T08:02:17.704 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/37 file after the snapshot 2023-06-22T08:02:17.704 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/38 file after the snapshot 2023-06-22T08:02:17.704 INFO:tasks.workunit.client.0.smithi023.stdout: create 283/39 file after the snapshot 2023-06-22T08:02:17.704 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/.snap/snap-subdir-test 2023-06-22T08:02:17.731 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/30 file after the snapshot 2023-06-22T08:02:17.733 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/31 file after the snapshot 2023-06-22T08:02:17.733 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/32 file after the snapshot 2023-06-22T08:02:17.733 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/33 file after the snapshot 2023-06-22T08:02:17.733 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/34 file after the snapshot 2023-06-22T08:02:17.734 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/35 file after the snapshot 2023-06-22T08:02:17.734 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/36 file after the snapshot 2023-06-22T08:02:17.734 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/37 file after the snapshot 2023-06-22T08:02:17.734 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/38 file after the snapshot 2023-06-22T08:02:17.734 INFO:tasks.workunit.client.0.smithi023.stdout: create 284/39 file after the snapshot 2023-06-22T08:02:17.734 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/.snap/snap-subdir-test 2023-06-22T08:02:17.766 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/30 file after the snapshot 2023-06-22T08:02:17.768 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/31 file after the snapshot 2023-06-22T08:02:17.768 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/32 file after the snapshot 2023-06-22T08:02:17.768 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/33 file after the snapshot 2023-06-22T08:02:17.769 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/34 file after the snapshot 2023-06-22T08:02:17.769 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/35 file after the snapshot 2023-06-22T08:02:17.769 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/36 file after the snapshot 2023-06-22T08:02:17.769 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/37 file after the snapshot 2023-06-22T08:02:17.769 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/38 file after the snapshot 2023-06-22T08:02:17.769 INFO:tasks.workunit.client.0.smithi023.stdout: create 285/39 file after the snapshot 2023-06-22T08:02:17.770 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/.snap/snap-subdir-test 2023-06-22T08:02:17.794 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/30 file after the snapshot 2023-06-22T08:02:17.796 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/31 file after the snapshot 2023-06-22T08:02:17.796 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/32 file after the snapshot 2023-06-22T08:02:17.796 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/33 file after the snapshot 2023-06-22T08:02:17.797 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/34 file after the snapshot 2023-06-22T08:02:17.797 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/35 file after the snapshot 2023-06-22T08:02:17.797 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/36 file after the snapshot 2023-06-22T08:02:17.797 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/37 file after the snapshot 2023-06-22T08:02:17.798 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/38 file after the snapshot 2023-06-22T08:02:17.798 INFO:tasks.workunit.client.0.smithi023.stdout: create 286/39 file after the snapshot 2023-06-22T08:02:17.798 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/.snap/snap-subdir-test 2023-06-22T08:02:17.823 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/30 file after the snapshot 2023-06-22T08:02:17.825 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/31 file after the snapshot 2023-06-22T08:02:17.825 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/32 file after the snapshot 2023-06-22T08:02:17.825 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/33 file after the snapshot 2023-06-22T08:02:17.826 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/34 file after the snapshot 2023-06-22T08:02:17.826 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/35 file after the snapshot 2023-06-22T08:02:17.826 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/36 file after the snapshot 2023-06-22T08:02:17.826 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/37 file after the snapshot 2023-06-22T08:02:17.826 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/38 file after the snapshot 2023-06-22T08:02:17.826 INFO:tasks.workunit.client.0.smithi023.stdout: create 287/39 file after the snapshot 2023-06-22T08:02:17.827 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/.snap/snap-subdir-test 2023-06-22T08:02:17.859 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/30 file after the snapshot 2023-06-22T08:02:17.861 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/31 file after the snapshot 2023-06-22T08:02:17.861 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/32 file after the snapshot 2023-06-22T08:02:17.862 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/33 file after the snapshot 2023-06-22T08:02:17.862 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/34 file after the snapshot 2023-06-22T08:02:17.862 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/35 file after the snapshot 2023-06-22T08:02:17.862 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/36 file after the snapshot 2023-06-22T08:02:17.862 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/37 file after the snapshot 2023-06-22T08:02:17.862 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/38 file after the snapshot 2023-06-22T08:02:17.862 INFO:tasks.workunit.client.0.smithi023.stdout: create 288/39 file after the snapshot 2023-06-22T08:02:17.863 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/.snap/snap-subdir-test 2023-06-22T08:02:17.886 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/30 file after the snapshot 2023-06-22T08:02:17.888 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/31 file after the snapshot 2023-06-22T08:02:17.888 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/32 file after the snapshot 2023-06-22T08:02:17.888 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/33 file after the snapshot 2023-06-22T08:02:17.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/34 file after the snapshot 2023-06-22T08:02:17.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/35 file after the snapshot 2023-06-22T08:02:17.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/36 file after the snapshot 2023-06-22T08:02:17.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/37 file after the snapshot 2023-06-22T08:02:17.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/38 file after the snapshot 2023-06-22T08:02:17.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 289/39 file after the snapshot 2023-06-22T08:02:17.889 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/.snap/snap-subdir-test 2023-06-22T08:02:17.913 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/30 file after the snapshot 2023-06-22T08:02:17.915 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/31 file after the snapshot 2023-06-22T08:02:17.915 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/32 file after the snapshot 2023-06-22T08:02:17.915 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/33 file after the snapshot 2023-06-22T08:02:17.916 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/34 file after the snapshot 2023-06-22T08:02:17.916 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/35 file after the snapshot 2023-06-22T08:02:17.916 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/36 file after the snapshot 2023-06-22T08:02:17.916 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/37 file after the snapshot 2023-06-22T08:02:17.916 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/38 file after the snapshot 2023-06-22T08:02:17.916 INFO:tasks.workunit.client.0.smithi023.stdout: create 290/39 file after the snapshot 2023-06-22T08:02:17.916 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/.snap/snap-subdir-test 2023-06-22T08:02:17.941 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/30 file after the snapshot 2023-06-22T08:02:17.943 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/31 file after the snapshot 2023-06-22T08:02:17.944 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/32 file after the snapshot 2023-06-22T08:02:17.944 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/33 file after the snapshot 2023-06-22T08:02:17.944 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/34 file after the snapshot 2023-06-22T08:02:17.945 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/35 file after the snapshot 2023-06-22T08:02:17.945 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/36 file after the snapshot 2023-06-22T08:02:17.945 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/37 file after the snapshot 2023-06-22T08:02:17.945 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/38 file after the snapshot 2023-06-22T08:02:17.946 INFO:tasks.workunit.client.0.smithi023.stdout: create 291/39 file after the snapshot 2023-06-22T08:02:17.946 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/.snap/snap-subdir-test 2023-06-22T08:02:17.968 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/30 file after the snapshot 2023-06-22T08:02:17.970 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/31 file after the snapshot 2023-06-22T08:02:17.971 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/32 file after the snapshot 2023-06-22T08:02:17.971 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/33 file after the snapshot 2023-06-22T08:02:17.971 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/34 file after the snapshot 2023-06-22T08:02:17.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/35 file after the snapshot 2023-06-22T08:02:17.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/36 file after the snapshot 2023-06-22T08:02:17.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/37 file after the snapshot 2023-06-22T08:02:17.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/38 file after the snapshot 2023-06-22T08:02:17.972 INFO:tasks.workunit.client.0.smithi023.stdout: create 292/39 file after the snapshot 2023-06-22T08:02:17.973 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/.snap/snap-subdir-test 2023-06-22T08:02:17.997 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/30 file after the snapshot 2023-06-22T08:02:17.998 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/31 file after the snapshot 2023-06-22T08:02:17.998 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/32 file after the snapshot 2023-06-22T08:02:17.999 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/33 file after the snapshot 2023-06-22T08:02:17.999 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/34 file after the snapshot 2023-06-22T08:02:17.999 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/35 file after the snapshot 2023-06-22T08:02:17.999 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/36 file after the snapshot 2023-06-22T08:02:18.000 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/37 file after the snapshot 2023-06-22T08:02:18.000 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/38 file after the snapshot 2023-06-22T08:02:18.000 INFO:tasks.workunit.client.0.smithi023.stdout: create 293/39 file after the snapshot 2023-06-22T08:02:18.000 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/.snap/snap-subdir-test 2023-06-22T08:02:18.022 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/30 file after the snapshot 2023-06-22T08:02:18.024 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/31 file after the snapshot 2023-06-22T08:02:18.024 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/32 file after the snapshot 2023-06-22T08:02:18.024 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/33 file after the snapshot 2023-06-22T08:02:18.024 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/34 file after the snapshot 2023-06-22T08:02:18.024 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/35 file after the snapshot 2023-06-22T08:02:18.025 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/36 file after the snapshot 2023-06-22T08:02:18.025 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/37 file after the snapshot 2023-06-22T08:02:18.025 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/38 file after the snapshot 2023-06-22T08:02:18.025 INFO:tasks.workunit.client.0.smithi023.stdout: create 294/39 file after the snapshot 2023-06-22T08:02:18.025 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/.snap/snap-subdir-test 2023-06-22T08:02:18.050 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/30 file after the snapshot 2023-06-22T08:02:18.052 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/31 file after the snapshot 2023-06-22T08:02:18.052 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/32 file after the snapshot 2023-06-22T08:02:18.052 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/33 file after the snapshot 2023-06-22T08:02:18.053 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/34 file after the snapshot 2023-06-22T08:02:18.053 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/35 file after the snapshot 2023-06-22T08:02:18.053 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/36 file after the snapshot 2023-06-22T08:02:18.053 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/37 file after the snapshot 2023-06-22T08:02:18.053 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/38 file after the snapshot 2023-06-22T08:02:18.053 INFO:tasks.workunit.client.0.smithi023.stdout: create 295/39 file after the snapshot 2023-06-22T08:02:18.053 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/.snap/snap-subdir-test 2023-06-22T08:02:18.077 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/30 file after the snapshot 2023-06-22T08:02:18.079 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/31 file after the snapshot 2023-06-22T08:02:18.079 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/32 file after the snapshot 2023-06-22T08:02:18.079 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/33 file after the snapshot 2023-06-22T08:02:18.079 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/34 file after the snapshot 2023-06-22T08:02:18.079 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/35 file after the snapshot 2023-06-22T08:02:18.080 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/36 file after the snapshot 2023-06-22T08:02:18.080 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/37 file after the snapshot 2023-06-22T08:02:18.080 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/38 file after the snapshot 2023-06-22T08:02:18.080 INFO:tasks.workunit.client.0.smithi023.stdout: create 296/39 file after the snapshot 2023-06-22T08:02:18.080 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/.snap/snap-subdir-test 2023-06-22T08:02:18.108 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/30 file after the snapshot 2023-06-22T08:02:18.110 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/31 file after the snapshot 2023-06-22T08:02:18.110 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/32 file after the snapshot 2023-06-22T08:02:18.110 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/33 file after the snapshot 2023-06-22T08:02:18.111 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/34 file after the snapshot 2023-06-22T08:02:18.111 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/35 file after the snapshot 2023-06-22T08:02:18.111 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/36 file after the snapshot 2023-06-22T08:02:18.111 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/37 file after the snapshot 2023-06-22T08:02:18.111 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/38 file after the snapshot 2023-06-22T08:02:18.112 INFO:tasks.workunit.client.0.smithi023.stdout: create 297/39 file after the snapshot 2023-06-22T08:02:18.112 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/.snap/snap-subdir-test 2023-06-22T08:02:18.135 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/30 file after the snapshot 2023-06-22T08:02:18.137 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/31 file after the snapshot 2023-06-22T08:02:18.137 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/32 file after the snapshot 2023-06-22T08:02:18.137 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/33 file after the snapshot 2023-06-22T08:02:18.138 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/34 file after the snapshot 2023-06-22T08:02:18.138 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/35 file after the snapshot 2023-06-22T08:02:18.138 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/36 file after the snapshot 2023-06-22T08:02:18.138 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/37 file after the snapshot 2023-06-22T08:02:18.138 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/38 file after the snapshot 2023-06-22T08:02:18.138 INFO:tasks.workunit.client.0.smithi023.stdout: create 298/39 file after the snapshot 2023-06-22T08:02:18.138 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/.snap/snap-subdir-test 2023-06-22T08:02:18.163 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/30 file after the snapshot 2023-06-22T08:02:18.164 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/31 file after the snapshot 2023-06-22T08:02:18.164 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/32 file after the snapshot 2023-06-22T08:02:18.165 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/33 file after the snapshot 2023-06-22T08:02:18.165 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/34 file after the snapshot 2023-06-22T08:02:18.165 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/35 file after the snapshot 2023-06-22T08:02:18.165 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/36 file after the snapshot 2023-06-22T08:02:18.165 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/37 file after the snapshot 2023-06-22T08:02:18.165 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/38 file after the snapshot 2023-06-22T08:02:18.165 INFO:tasks.workunit.client.0.smithi023.stdout: create 299/39 file after the snapshot 2023-06-22T08:02:18.166 INFO:tasks.workunit.client.0.smithi023.stdout:Take second snapshot .snap/test2 2023-06-22T08:02:18.242 INFO:tasks.workunit.client.0.smithi023.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-06-22T08:02:35.792 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-06-22T08:02:35.792 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-06-22T08:02:35.793 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-06-22T08:02:35.793 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-06-22T08:02:35.793 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-06-22T08:02:35.793 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-06-22T08:02:35.793 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-06-22T08:02:35.793 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-06-22T08:02:35.793 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-06-22T08:02:35.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-06-22T08:02:35.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-06-22T08:02:35.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-06-22T08:02:35.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-06-22T08:02:35.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-06-22T08:02:35.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-06-22T08:02:35.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-06-22T08:02:35.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-06-22T08:02:35.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-06-22T08:02:35.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-06-22T08:02:35.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-06-22T08:02:35.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-06-22T08:02:35.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-06-22T08:02:35.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-06-22T08:02:35.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-06-22T08:02:35.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-06-22T08:02:35.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-06-22T08:02:35.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-06-22T08:02:35.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-06-22T08:02:35.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-06-22T08:02:35.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-06-22T08:02:35.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-06-22T08:02:35.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-06-22T08:02:35.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-06-22T08:02:35.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-06-22T08:02:35.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-06-22T08:02:35.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-06-22T08:02:35.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-06-22T08:02:35.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-06-22T08:02:35.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-06-22T08:02:35.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-06-22T08:02:35.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-06-22T08:02:35.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-06-22T08:02:35.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-06-22T08:02:35.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-06-22T08:02:35.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-06-22T08:02:35.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-06-22T08:02:35.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-06-22T08:02:35.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-06-22T08:02:35.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-06-22T08:02:35.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-06-22T08:02:35.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-06-22T08:02:35.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-06-22T08:02:35.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-06-22T08:02:35.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-06-22T08:02:35.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-06-22T08:02:35.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-06-22T08:02:35.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-06-22T08:02:35.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-06-22T08:02:35.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-06-22T08:02:35.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-06-22T08:02:35.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-06-22T08:02:35.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-06-22T08:02:35.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-06-22T08:02:35.801 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-06-22T08:02:35.801 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-06-22T08:02:35.801 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-06-22T08:02:35.801 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-06-22T08:02:35.801 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-06-22T08:02:35.801 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-06-22T08:02:35.801 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-06-22T08:02:35.801 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-06-22T08:02:35.802 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-06-22T08:02:35.802 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-06-22T08:02:35.802 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-06-22T08:02:35.802 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-06-22T08:02:35.802 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-06-22T08:02:35.802 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-06-22T08:02:35.802 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-06-22T08:02:35.802 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-06-22T08:02:35.803 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-06-22T08:02:35.803 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-06-22T08:02:36.013 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-06-22T08:02:36.014 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-06-22T08:02:36.014 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-06-22T08:02:36.014 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-06-22T08:02:36.014 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-06-22T08:02:36.014 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-06-22T08:02:36.014 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-06-22T08:02:36.014 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-06-22T08:02:36.015 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-06-22T08:02:36.015 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-06-22T08:02:36.015 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-06-22T08:02:36.015 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-06-22T08:02:36.015 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-06-22T08:02:36.015 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-06-22T08:02:36.015 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-06-22T08:02:36.016 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-06-22T08:02:36.016 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-06-22T08:02:36.016 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-06-22T08:02:36.016 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-06-22T08:02:36.016 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-06-22T08:02:36.017 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-06-22T08:02:36.017 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-06-22T08:02:36.017 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-06-22T08:02:36.017 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-06-22T08:02:36.017 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-06-22T08:02:36.017 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-06-22T08:02:36.017 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-06-22T08:02:36.018 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-06-22T08:02:36.018 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-06-22T08:02:36.018 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-06-22T08:02:36.018 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-06-22T08:02:36.018 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-06-22T08:02:36.018 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-06-22T08:02:36.018 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-06-22T08:02:36.018 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-06-22T08:02:36.019 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-06-22T08:02:36.019 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-06-22T08:02:36.019 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-06-22T08:02:36.019 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-06-22T08:02:36.019 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-06-22T08:02:36.019 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-06-22T08:02:36.019 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-06-22T08:02:36.019 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-06-22T08:02:36.020 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-06-22T08:02:36.020 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-06-22T08:02:36.020 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-06-22T08:02:36.020 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-06-22T08:02:36.020 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-06-22T08:02:36.020 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-06-22T08:02:36.020 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-06-22T08:02:36.021 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-06-22T08:02:36.021 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-06-22T08:02:36.021 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-06-22T08:02:36.021 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-06-22T08:02:36.021 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-06-22T08:02:36.021 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-06-22T08:02:36.022 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-06-22T08:02:36.022 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-06-22T08:02:36.022 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-06-22T08:02:36.022 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-06-22T08:02:36.022 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-06-22T08:02:36.022 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-06-22T08:02:36.022 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-06-22T08:02:36.023 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-06-22T08:02:36.023 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-06-22T08:02:36.023 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-06-22T08:02:36.023 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-06-22T08:02:36.023 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-06-22T08:02:36.023 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-06-22T08:02:36.023 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-06-22T08:02:36.023 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-06-22T08:02:36.024 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-06-22T08:02:36.024 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-06-22T08:02:36.024 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-06-22T08:02:36.024 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-06-22T08:02:36.024 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-06-22T08:02:36.024 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-06-22T08:02:36.025 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-06-22T08:02:36.025 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-06-22T08:02:36.025 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-06-22T08:02:36.025 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-06-22T08:02:36.213 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-06-22T08:02:36.214 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-06-22T08:02:36.214 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-06-22T08:02:36.214 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-06-22T08:02:36.214 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-06-22T08:02:36.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-06-22T08:02:36.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-06-22T08:02:36.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-06-22T08:02:36.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-06-22T08:02:36.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-06-22T08:02:36.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-06-22T08:02:36.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-06-22T08:02:36.216 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-06-22T08:02:36.216 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-06-22T08:02:36.216 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-06-22T08:02:36.216 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-06-22T08:02:36.216 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-06-22T08:02:36.216 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-06-22T08:02:36.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-06-22T08:02:36.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-06-22T08:02:36.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-06-22T08:02:36.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-06-22T08:02:36.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-06-22T08:02:36.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-06-22T08:02:36.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-06-22T08:02:36.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-06-22T08:02:36.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-06-22T08:02:36.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-06-22T08:02:36.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-06-22T08:02:36.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-06-22T08:02:36.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-06-22T08:02:36.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-06-22T08:02:36.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-06-22T08:02:36.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-06-22T08:02:36.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-06-22T08:02:36.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-06-22T08:02:36.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-06-22T08:02:36.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-06-22T08:02:36.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-06-22T08:02:36.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-06-22T08:02:36.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-06-22T08:02:36.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-06-22T08:02:36.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-06-22T08:02:36.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-06-22T08:02:36.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-06-22T08:02:36.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-06-22T08:02:36.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-06-22T08:02:36.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-06-22T08:02:36.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-06-22T08:02:36.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-06-22T08:02:36.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-06-22T08:02:36.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-06-22T08:02:36.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-06-22T08:02:36.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-06-22T08:02:36.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-06-22T08:02:36.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-06-22T08:02:36.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-06-22T08:02:36.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-06-22T08:02:36.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-06-22T08:02:36.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-06-22T08:02:36.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-06-22T08:02:36.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-06-22T08:02:36.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-06-22T08:02:36.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-06-22T08:02:36.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-06-22T08:02:36.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-06-22T08:02:36.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-06-22T08:02:36.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-06-22T08:02:36.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-06-22T08:02:36.224 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-06-22T08:02:36.224 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-06-22T08:02:36.224 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-06-22T08:02:36.224 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-06-22T08:02:36.224 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-06-22T08:02:36.225 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-06-22T08:02:36.225 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-06-22T08:02:36.225 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-06-22T08:02:36.225 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-06-22T08:02:36.225 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-06-22T08:02:36.225 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-06-22T08:02:36.225 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-06-22T08:02:36.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:02:36.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:02:36.716 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-06-22T08:02:36.717 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-06-22T08:02:36.717 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-06-22T08:02:36.717 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-06-22T08:02:36.718 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-06-22T08:02:36.718 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-06-22T08:02:36.718 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-06-22T08:02:36.718 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-06-22T08:02:36.719 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-06-22T08:02:36.719 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-06-22T08:02:36.719 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-06-22T08:02:36.720 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-06-22T08:02:36.720 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-06-22T08:02:36.720 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-06-22T08:02:36.720 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-06-22T08:02:36.720 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-06-22T08:02:36.721 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-06-22T08:02:36.721 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-06-22T08:02:36.721 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-06-22T08:02:36.721 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-06-22T08:02:36.722 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-06-22T08:02:36.722 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-06-22T08:02:36.722 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-06-22T08:02:36.722 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-06-22T08:02:36.723 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-06-22T08:02:36.723 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-06-22T08:02:36.723 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-06-22T08:02:36.723 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-06-22T08:02:36.723 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-06-22T08:02:36.723 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-06-22T08:02:36.723 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-06-22T08:02:36.723 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-06-22T08:02:36.724 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-06-22T08:02:36.724 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-06-22T08:02:36.724 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-06-22T08:02:36.724 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-06-22T08:02:36.724 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-06-22T08:02:36.725 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-06-22T08:02:36.725 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-06-22T08:02:36.725 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-06-22T08:02:36.725 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-06-22T08:02:36.725 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-06-22T08:02:36.725 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-06-22T08:02:36.725 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-06-22T08:02:36.726 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-06-22T08:02:36.726 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-06-22T08:02:36.726 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-06-22T08:02:36.726 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-06-22T08:02:36.726 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-06-22T08:02:36.726 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-06-22T08:02:36.726 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-06-22T08:02:36.727 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-06-22T08:02:36.727 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-06-22T08:02:36.727 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-06-22T08:02:36.727 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-06-22T08:02:36.727 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-06-22T08:02:36.727 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-06-22T08:02:36.727 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-06-22T08:02:36.728 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-06-22T08:02:36.728 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-06-22T08:02:36.728 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-06-22T08:02:36.728 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-06-22T08:02:36.728 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-06-22T08:02:36.728 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-06-22T08:02:36.729 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-06-22T08:02:36.729 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-06-22T08:02:36.729 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-06-22T08:02:36.729 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-06-22T08:02:36.729 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-06-22T08:02:36.729 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-06-22T08:02:36.729 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-06-22T08:02:36.729 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-06-22T08:02:36.730 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-06-22T08:02:36.730 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-06-22T08:02:36.730 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-06-22T08:02:36.730 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-06-22T08:02:36.730 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-06-22T08:02:36.730 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-06-22T08:02:36.730 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-06-22T08:02:36.731 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-06-22T08:02:36.731 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-06-22T08:02:36.923 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-06-22T08:02:36.924 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-06-22T08:02:36.924 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-06-22T08:02:36.924 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-06-22T08:02:36.924 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-06-22T08:02:36.925 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-06-22T08:02:36.925 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-06-22T08:02:36.925 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-06-22T08:02:36.925 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-06-22T08:02:36.925 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-06-22T08:02:36.925 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-06-22T08:02:36.926 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-06-22T08:02:36.926 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-06-22T08:02:36.926 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-06-22T08:02:36.926 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-06-22T08:02:36.926 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-06-22T08:02:36.926 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-06-22T08:02:36.926 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-06-22T08:02:36.927 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-06-22T08:02:36.927 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-06-22T08:02:36.927 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-06-22T08:02:36.927 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-06-22T08:02:36.927 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-06-22T08:02:36.927 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-06-22T08:02:36.927 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-06-22T08:02:36.928 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-06-22T08:02:36.928 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-06-22T08:02:36.928 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-06-22T08:02:36.928 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-06-22T08:02:36.928 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-06-22T08:02:36.928 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-06-22T08:02:36.928 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-06-22T08:02:36.929 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-06-22T08:02:36.929 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-06-22T08:02:36.929 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-06-22T08:02:36.929 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-06-22T08:02:36.929 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-06-22T08:02:36.929 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-06-22T08:02:36.929 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-06-22T08:02:36.929 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-06-22T08:02:36.930 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-06-22T08:02:36.930 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-06-22T08:02:36.930 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-06-22T08:02:36.930 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-06-22T08:02:36.930 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-06-22T08:02:36.930 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-06-22T08:02:36.930 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-06-22T08:02:36.930 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-06-22T08:02:36.931 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-06-22T08:02:36.931 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-06-22T08:02:36.931 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-06-22T08:02:36.931 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-06-22T08:02:36.932 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-06-22T08:02:36.932 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-06-22T08:02:36.932 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-06-22T08:02:36.932 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-06-22T08:02:36.932 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-06-22T08:02:36.932 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-06-22T08:02:36.933 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-06-22T08:02:36.933 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-06-22T08:02:36.933 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-06-22T08:02:36.933 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-06-22T08:02:36.933 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-06-22T08:02:36.933 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-06-22T08:02:36.933 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-06-22T08:02:36.933 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-06-22T08:02:36.934 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-06-22T08:02:36.934 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-06-22T08:02:36.934 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-06-22T08:02:36.934 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-06-22T08:02:36.934 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-06-22T08:02:36.934 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-06-22T08:02:36.934 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-06-22T08:02:36.935 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-06-22T08:02:36.935 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-06-22T08:02:36.935 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-06-22T08:02:36.935 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-06-22T08:02:36.935 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-06-22T08:02:36.935 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-06-22T08:02:36.935 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-06-22T08:02:36.935 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-06-22T08:02:37.142 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-06-22T08:02:37.143 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-06-22T08:02:37.143 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-06-22T08:02:37.143 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-06-22T08:02:37.143 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-06-22T08:02:37.143 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-06-22T08:02:37.143 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-06-22T08:02:37.144 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-06-22T08:02:37.144 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-06-22T08:02:37.144 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-06-22T08:02:37.144 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-06-22T08:02:37.144 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-06-22T08:02:37.144 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-06-22T08:02:37.144 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-06-22T08:02:37.145 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-06-22T08:02:37.145 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-06-22T08:02:37.145 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-06-22T08:02:37.145 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-06-22T08:02:37.145 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-06-22T08:02:37.145 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-06-22T08:02:37.145 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-06-22T08:02:37.145 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-06-22T08:02:37.146 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-06-22T08:02:37.146 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-06-22T08:02:37.146 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-06-22T08:02:37.146 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-06-22T08:02:37.146 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-06-22T08:02:37.146 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-06-22T08:02:37.146 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-06-22T08:02:37.146 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-06-22T08:02:37.147 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-06-22T08:02:37.147 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-06-22T08:02:37.147 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-06-22T08:02:37.147 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-06-22T08:02:37.147 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-06-22T08:02:37.147 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-06-22T08:02:37.147 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-06-22T08:02:37.147 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-06-22T08:02:37.148 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-06-22T08:02:37.148 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-06-22T08:02:37.148 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-06-22T08:02:37.148 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-06-22T08:02:37.148 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-06-22T08:02:37.148 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-06-22T08:02:37.148 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-06-22T08:02:37.148 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-06-22T08:02:37.149 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-06-22T08:02:37.149 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-06-22T08:02:37.149 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-06-22T08:02:37.149 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-06-22T08:02:37.149 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-06-22T08:02:37.149 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-06-22T08:02:37.149 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-06-22T08:02:37.150 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-06-22T08:02:37.150 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-06-22T08:02:37.150 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-06-22T08:02:37.150 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-06-22T08:02:37.150 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-06-22T08:02:37.151 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-06-22T08:02:37.151 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-06-22T08:02:37.151 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-06-22T08:02:37.151 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-06-22T08:02:37.151 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-06-22T08:02:37.151 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-06-22T08:02:37.151 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-06-22T08:02:37.151 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-06-22T08:02:37.152 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-06-22T08:02:37.152 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-06-22T08:02:37.152 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-06-22T08:02:37.152 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-06-22T08:02:37.152 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-06-22T08:02:37.152 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-06-22T08:02:37.155 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-06-22T08:02:37.155 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-06-22T08:02:37.155 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-06-22T08:02:37.155 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-06-22T08:02:37.155 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-06-22T08:02:37.155 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-06-22T08:02:37.156 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-06-22T08:02:37.156 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-06-22T08:02:37.156 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-06-22T08:02:37.374 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-06-22T08:02:37.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-06-22T08:02:37.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-06-22T08:02:37.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-06-22T08:02:37.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-06-22T08:02:37.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-06-22T08:02:37.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-06-22T08:02:37.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-06-22T08:02:37.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-06-22T08:02:37.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-06-22T08:02:37.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-06-22T08:02:37.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-06-22T08:02:37.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-06-22T08:02:37.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-06-22T08:02:37.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-06-22T08:02:37.377 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-06-22T08:02:37.377 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-06-22T08:02:37.377 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-06-22T08:02:37.377 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-06-22T08:02:37.377 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-06-22T08:02:37.377 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-06-22T08:02:37.377 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-06-22T08:02:37.377 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-06-22T08:02:37.378 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-06-22T08:02:37.378 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-06-22T08:02:37.378 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-06-22T08:02:37.378 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-06-22T08:02:37.378 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-06-22T08:02:37.378 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-06-22T08:02:37.378 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-06-22T08:02:37.378 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-06-22T08:02:37.379 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-06-22T08:02:37.379 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-06-22T08:02:37.379 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-06-22T08:02:37.379 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-06-22T08:02:37.379 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-06-22T08:02:37.379 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-06-22T08:02:37.379 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-06-22T08:02:37.380 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-06-22T08:02:37.380 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-06-22T08:02:37.380 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-06-22T08:02:37.380 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-06-22T08:02:37.380 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-06-22T08:02:37.380 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-06-22T08:02:37.380 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-06-22T08:02:37.380 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-06-22T08:02:37.381 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-06-22T08:02:37.381 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-06-22T08:02:37.381 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-06-22T08:02:37.381 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-06-22T08:02:37.381 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-06-22T08:02:37.381 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-06-22T08:02:37.381 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-06-22T08:02:37.381 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-06-22T08:02:37.382 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-06-22T08:02:37.382 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-06-22T08:02:37.382 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-06-22T08:02:37.382 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-06-22T08:02:37.382 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-06-22T08:02:37.382 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-06-22T08:02:37.382 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-06-22T08:02:37.382 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-06-22T08:02:37.383 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-06-22T08:02:37.383 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-06-22T08:02:37.383 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-06-22T08:02:37.383 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-06-22T08:02:37.383 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-06-22T08:02:37.383 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-06-22T08:02:37.383 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-06-22T08:02:37.383 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-06-22T08:02:37.384 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-06-22T08:02:37.384 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-06-22T08:02:37.384 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-06-22T08:02:37.384 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-06-22T08:02:37.384 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-06-22T08:02:37.384 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-06-22T08:02:37.384 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-06-22T08:02:37.384 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-06-22T08:02:37.385 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-06-22T08:02:37.385 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-06-22T08:02:37.385 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-06-22T08:02:37.385 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-06-22T08:02:37.588 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-06-22T08:02:37.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-06-22T08:02:37.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-06-22T08:02:37.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-06-22T08:02:37.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-06-22T08:02:37.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-06-22T08:02:37.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-06-22T08:02:37.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-06-22T08:02:37.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-06-22T08:02:37.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-06-22T08:02:37.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-06-22T08:02:37.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-06-22T08:02:37.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-06-22T08:02:37.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-06-22T08:02:37.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-06-22T08:02:37.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-06-22T08:02:37.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-06-22T08:02:37.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-06-22T08:02:37.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-06-22T08:02:37.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-06-22T08:02:37.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-06-22T08:02:37.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-06-22T08:02:37.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-06-22T08:02:37.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-06-22T08:02:37.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-06-22T08:02:37.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-06-22T08:02:37.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-06-22T08:02:37.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-06-22T08:02:37.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-06-22T08:02:37.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-06-22T08:02:37.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-06-22T08:02:37.593 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-06-22T08:02:37.593 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-06-22T08:02:37.593 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-06-22T08:02:37.593 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-06-22T08:02:37.593 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-06-22T08:02:37.593 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-06-22T08:02:37.593 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-06-22T08:02:37.593 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-06-22T08:02:37.594 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-06-22T08:02:37.594 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-06-22T08:02:37.594 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-06-22T08:02:37.594 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-06-22T08:02:37.594 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-06-22T08:02:37.594 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-06-22T08:02:37.594 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-06-22T08:02:37.594 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-06-22T08:02:37.595 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-06-22T08:02:37.595 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-06-22T08:02:37.595 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-06-22T08:02:37.595 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-06-22T08:02:37.595 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-06-22T08:02:37.595 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-06-22T08:02:37.595 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-06-22T08:02:37.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-06-22T08:02:37.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-06-22T08:02:37.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-06-22T08:02:37.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-06-22T08:02:37.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-06-22T08:02:37.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-06-22T08:02:37.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-06-22T08:02:37.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-06-22T08:02:37.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-06-22T08:02:37.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-06-22T08:02:37.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-06-22T08:02:37.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-06-22T08:02:37.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-06-22T08:02:37.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-06-22T08:02:37.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-06-22T08:02:37.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-06-22T08:02:37.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-06-22T08:02:37.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-06-22T08:02:37.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-06-22T08:02:37.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-06-22T08:02:37.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-06-22T08:02:37.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-06-22T08:02:37.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-06-22T08:02:37.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-06-22T08:02:37.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-06-22T08:02:37.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-06-22T08:02:37.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-06-22T08:02:37.781 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-06-22T08:02:37.781 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-06-22T08:02:37.782 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-06-22T08:02:37.782 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-06-22T08:02:37.782 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-06-22T08:02:37.782 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-06-22T08:02:37.782 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-06-22T08:02:37.782 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-06-22T08:02:37.782 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-06-22T08:02:37.783 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-06-22T08:02:37.783 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-06-22T08:02:37.783 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-06-22T08:02:37.783 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-06-22T08:02:37.783 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-06-22T08:02:37.783 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-06-22T08:02:37.783 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-06-22T08:02:37.783 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-06-22T08:02:37.784 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-06-22T08:02:37.784 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-06-22T08:02:37.784 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-06-22T08:02:37.784 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-06-22T08:02:37.784 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-06-22T08:02:37.784 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-06-22T08:02:37.784 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-06-22T08:02:37.785 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-06-22T08:02:37.785 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-06-22T08:02:37.785 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-06-22T08:02:37.785 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-06-22T08:02:37.785 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-06-22T08:02:37.785 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-06-22T08:02:37.785 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-06-22T08:02:37.785 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-06-22T08:02:37.786 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-06-22T08:02:37.786 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-06-22T08:02:37.786 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-06-22T08:02:37.786 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-06-22T08:02:37.786 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-06-22T08:02:37.786 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-06-22T08:02:37.786 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-06-22T08:02:37.786 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-06-22T08:02:37.787 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-06-22T08:02:37.787 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-06-22T08:02:37.787 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-06-22T08:02:37.787 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-06-22T08:02:37.787 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-06-22T08:02:37.787 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-06-22T08:02:37.787 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-06-22T08:02:37.787 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-06-22T08:02:37.788 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-06-22T08:02:37.788 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-06-22T08:02:37.788 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-06-22T08:02:37.788 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-06-22T08:02:37.788 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-06-22T08:02:37.788 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-06-22T08:02:37.788 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-06-22T08:02:37.788 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-06-22T08:02:37.789 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-06-22T08:02:37.789 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-06-22T08:02:37.789 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-06-22T08:02:37.789 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-06-22T08:02:37.789 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-06-22T08:02:37.789 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-06-22T08:02:37.789 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-06-22T08:02:37.790 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-06-22T08:02:37.790 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-06-22T08:02:37.790 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-06-22T08:02:37.790 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-06-22T08:02:37.790 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-06-22T08:02:37.790 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-06-22T08:02:37.790 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-06-22T08:02:37.790 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-06-22T08:02:37.791 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-06-22T08:02:37.791 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-06-22T08:02:37.791 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-06-22T08:02:37.791 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-06-22T08:02:37.791 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-06-22T08:02:37.791 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-06-22T08:02:37.791 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-06-22T08:02:37.791 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-06-22T08:02:37.792 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-06-22T08:02:37.792 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-06-22T08:02:37.982 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-06-22T08:02:37.983 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-06-22T08:02:37.983 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-06-22T08:02:37.983 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-06-22T08:02:37.983 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-06-22T08:02:37.983 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-06-22T08:02:37.983 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-06-22T08:02:37.984 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-06-22T08:02:37.984 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-06-22T08:02:37.984 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-06-22T08:02:37.984 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-06-22T08:02:37.984 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-06-22T08:02:37.984 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-06-22T08:02:37.984 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-06-22T08:02:37.985 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-06-22T08:02:37.985 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-06-22T08:02:37.985 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-06-22T08:02:37.985 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-06-22T08:02:37.985 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-06-22T08:02:37.985 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-06-22T08:02:37.985 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-06-22T08:02:37.986 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-06-22T08:02:37.986 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-06-22T08:02:37.986 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-06-22T08:02:37.986 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-06-22T08:02:37.986 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-06-22T08:02:37.986 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-06-22T08:02:37.986 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-06-22T08:02:37.986 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-06-22T08:02:37.987 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-06-22T08:02:37.987 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-06-22T08:02:37.987 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-06-22T08:02:37.987 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-06-22T08:02:37.987 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-06-22T08:02:37.987 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-06-22T08:02:37.987 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-06-22T08:02:37.987 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-06-22T08:02:37.988 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-06-22T08:02:37.988 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-06-22T08:02:37.988 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-06-22T08:02:37.988 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-06-22T08:02:37.988 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-06-22T08:02:37.988 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-06-22T08:02:37.988 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-06-22T08:02:37.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-06-22T08:02:37.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-06-22T08:02:37.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-06-22T08:02:37.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-06-22T08:02:37.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-06-22T08:02:37.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-06-22T08:02:37.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-06-22T08:02:37.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-06-22T08:02:37.990 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-06-22T08:02:37.990 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-06-22T08:02:37.990 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-06-22T08:02:37.990 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-06-22T08:02:37.990 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-06-22T08:02:37.990 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-06-22T08:02:37.990 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-06-22T08:02:37.990 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-06-22T08:02:37.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-06-22T08:02:37.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-06-22T08:02:37.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-06-22T08:02:37.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-06-22T08:02:37.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-06-22T08:02:37.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-06-22T08:02:37.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-06-22T08:02:37.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-06-22T08:02:37.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-06-22T08:02:37.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-06-22T08:02:37.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-06-22T08:02:37.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-06-22T08:02:37.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-06-22T08:02:37.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-06-22T08:02:37.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-06-22T08:02:37.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-06-22T08:02:37.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-06-22T08:02:37.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-06-22T08:02:37.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-06-22T08:02:37.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-06-22T08:02:37.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-06-22T08:02:38.199 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-06-22T08:02:38.199 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-06-22T08:02:38.199 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-06-22T08:02:38.199 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-06-22T08:02:38.199 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-06-22T08:02:38.200 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-06-22T08:02:38.200 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-06-22T08:02:38.200 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-06-22T08:02:38.200 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-06-22T08:02:38.200 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-06-22T08:02:38.200 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-06-22T08:02:38.200 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-06-22T08:02:38.201 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-06-22T08:02:38.201 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-06-22T08:02:38.201 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-06-22T08:02:38.201 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-06-22T08:02:38.201 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-06-22T08:02:38.201 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-06-22T08:02:38.202 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-06-22T08:02:38.202 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-06-22T08:02:38.202 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-06-22T08:02:38.202 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-06-22T08:02:38.202 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-06-22T08:02:38.202 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-06-22T08:02:38.203 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-06-22T08:02:38.203 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-06-22T08:02:38.203 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-06-22T08:02:38.203 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-06-22T08:02:38.203 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-06-22T08:02:38.203 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-06-22T08:02:38.203 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-06-22T08:02:38.203 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-06-22T08:02:38.204 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-06-22T08:02:38.204 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-06-22T08:02:38.204 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-06-22T08:02:38.204 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-06-22T08:02:38.204 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-06-22T08:02:38.204 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-06-22T08:02:38.204 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-06-22T08:02:38.205 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-06-22T08:02:38.205 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-06-22T08:02:38.205 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-06-22T08:02:38.205 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-06-22T08:02:38.205 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-06-22T08:02:38.205 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-06-22T08:02:38.205 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-06-22T08:02:38.205 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-06-22T08:02:38.206 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-06-22T08:02:38.206 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-06-22T08:02:38.206 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-06-22T08:02:38.206 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-06-22T08:02:38.206 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-06-22T08:02:38.206 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-06-22T08:02:38.206 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-06-22T08:02:38.206 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-06-22T08:02:38.207 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-06-22T08:02:38.207 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-06-22T08:02:38.207 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-06-22T08:02:38.207 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-06-22T08:02:38.207 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-06-22T08:02:38.207 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-06-22T08:02:38.207 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-06-22T08:02:38.208 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-06-22T08:02:38.208 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-06-22T08:02:38.208 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-06-22T08:02:38.208 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-06-22T08:02:38.208 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-06-22T08:02:38.208 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-06-22T08:02:38.208 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-06-22T08:02:38.208 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-06-22T08:02:38.209 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-06-22T08:02:38.209 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-06-22T08:02:38.209 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-06-22T08:02:38.209 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-06-22T08:02:38.209 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-06-22T08:02:38.209 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-06-22T08:02:38.209 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-06-22T08:02:38.209 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-06-22T08:02:38.210 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-06-22T08:02:38.210 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-06-22T08:02:38.210 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-06-22T08:02:38.415 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-06-22T08:02:38.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-06-22T08:02:38.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-06-22T08:02:38.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-06-22T08:02:38.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-06-22T08:02:38.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-06-22T08:02:38.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-06-22T08:02:38.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-06-22T08:02:38.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-06-22T08:02:38.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-06-22T08:02:38.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-06-22T08:02:38.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-06-22T08:02:38.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-06-22T08:02:38.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-06-22T08:02:38.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-06-22T08:02:38.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-06-22T08:02:38.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-06-22T08:02:38.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-06-22T08:02:38.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-06-22T08:02:38.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-06-22T08:02:38.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-06-22T08:02:38.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-06-22T08:02:38.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-06-22T08:02:38.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-06-22T08:02:38.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-06-22T08:02:38.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-06-22T08:02:38.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-06-22T08:02:38.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-06-22T08:02:38.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-06-22T08:02:38.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-06-22T08:02:38.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-06-22T08:02:38.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-06-22T08:02:38.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-06-22T08:02:38.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-06-22T08:02:38.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-06-22T08:02:38.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-06-22T08:02:38.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-06-22T08:02:38.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-06-22T08:02:38.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-06-22T08:02:38.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-06-22T08:02:38.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-06-22T08:02:38.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-06-22T08:02:38.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-06-22T08:02:38.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-06-22T08:02:38.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-06-22T08:02:38.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-06-22T08:02:38.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-06-22T08:02:38.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-06-22T08:02:38.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-06-22T08:02:38.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-06-22T08:02:38.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-06-22T08:02:38.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-06-22T08:02:38.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-06-22T08:02:38.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-06-22T08:02:38.424 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-06-22T08:02:38.424 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-06-22T08:02:38.424 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-06-22T08:02:38.424 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-06-22T08:02:38.424 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-06-22T08:02:38.424 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-06-22T08:02:38.424 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-06-22T08:02:38.425 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-06-22T08:02:38.425 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-06-22T08:02:38.425 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-06-22T08:02:38.425 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-06-22T08:02:38.425 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-06-22T08:02:38.425 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-06-22T08:02:38.425 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-06-22T08:02:38.426 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-06-22T08:02:38.426 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-06-22T08:02:38.426 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-06-22T08:02:38.426 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-06-22T08:02:38.426 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-06-22T08:02:38.426 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-06-22T08:02:38.427 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-06-22T08:02:38.427 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-06-22T08:02:38.427 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-06-22T08:02:38.427 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-06-22T08:02:38.427 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-06-22T08:02:38.427 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-06-22T08:02:38.427 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-06-22T08:02:38.613 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-06-22T08:02:38.614 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-06-22T08:02:38.614 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-06-22T08:02:38.614 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-06-22T08:02:38.614 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-06-22T08:02:38.614 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-06-22T08:02:38.614 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-06-22T08:02:38.614 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-06-22T08:02:38.615 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-06-22T08:02:38.615 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-06-22T08:02:38.615 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-06-22T08:02:38.615 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-06-22T08:02:38.615 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-06-22T08:02:38.615 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-06-22T08:02:38.615 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-06-22T08:02:38.615 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-06-22T08:02:38.616 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-06-22T08:02:38.616 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-06-22T08:02:38.616 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-06-22T08:02:38.616 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-06-22T08:02:38.616 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-06-22T08:02:38.616 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-06-22T08:02:38.616 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-06-22T08:02:38.617 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-06-22T08:02:38.617 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-06-22T08:02:38.617 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-06-22T08:02:38.617 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-06-22T08:02:38.617 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-06-22T08:02:38.617 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-06-22T08:02:38.617 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-06-22T08:02:38.617 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-06-22T08:02:38.618 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-06-22T08:02:38.618 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-06-22T08:02:38.618 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-06-22T08:02:38.618 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-06-22T08:02:38.618 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-06-22T08:02:38.618 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-06-22T08:02:38.618 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-06-22T08:02:38.618 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-06-22T08:02:38.619 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-06-22T08:02:38.619 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-06-22T08:02:38.619 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-06-22T08:02:38.619 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-06-22T08:02:38.619 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-06-22T08:02:38.619 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-06-22T08:02:38.619 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-06-22T08:02:38.620 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-06-22T08:02:38.620 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-06-22T08:02:38.620 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-06-22T08:02:38.620 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-06-22T08:02:38.620 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-06-22T08:02:38.620 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-06-22T08:02:38.620 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-06-22T08:02:38.621 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-06-22T08:02:38.621 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-06-22T08:02:38.621 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-06-22T08:02:38.621 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-06-22T08:02:38.621 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-06-22T08:02:38.621 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-06-22T08:02:38.621 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-06-22T08:02:38.621 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-06-22T08:02:38.622 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-06-22T08:02:38.622 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-06-22T08:02:38.622 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-06-22T08:02:38.622 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-06-22T08:02:38.623 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-06-22T08:02:38.623 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-06-22T08:02:38.623 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-06-22T08:02:38.623 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-06-22T08:02:38.624 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-06-22T08:02:38.624 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-06-22T08:02:38.624 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-06-22T08:02:38.625 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-06-22T08:02:38.625 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-06-22T08:02:38.625 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-06-22T08:02:38.625 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-06-22T08:02:38.626 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-06-22T08:02:38.626 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-06-22T08:02:38.626 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-06-22T08:02:38.627 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-06-22T08:02:38.627 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-06-22T08:02:38.818 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-06-22T08:02:38.818 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-06-22T08:02:38.818 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-06-22T08:02:38.818 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-06-22T08:02:38.818 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-06-22T08:02:38.818 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-06-22T08:02:38.819 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-06-22T08:02:38.819 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-06-22T08:02:38.819 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-06-22T08:02:38.819 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-06-22T08:02:38.819 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-06-22T08:02:38.819 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-06-22T08:02:38.819 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-06-22T08:02:38.820 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-06-22T08:02:38.820 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-06-22T08:02:38.820 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-06-22T08:02:38.820 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-06-22T08:02:38.820 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-06-22T08:02:38.821 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-06-22T08:02:38.821 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-06-22T08:02:38.821 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-06-22T08:02:38.821 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-06-22T08:02:38.821 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-06-22T08:02:38.821 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-06-22T08:02:38.821 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-06-22T08:02:38.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-06-22T08:02:38.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-06-22T08:02:38.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-06-22T08:02:38.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-06-22T08:02:38.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-06-22T08:02:38.823 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-06-22T08:02:38.823 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-06-22T08:02:38.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-06-22T08:02:38.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-06-22T08:02:38.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-06-22T08:02:38.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-06-22T08:02:38.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-06-22T08:02:38.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-06-22T08:02:38.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-06-22T08:02:38.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-06-22T08:02:38.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-06-22T08:02:38.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-06-22T08:02:38.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-06-22T08:02:38.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-06-22T08:02:38.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-06-22T08:02:38.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-06-22T08:02:38.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-06-22T08:02:38.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-06-22T08:02:38.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-06-22T08:02:38.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-06-22T08:02:38.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-06-22T08:02:38.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-06-22T08:02:38.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-06-22T08:02:38.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-06-22T08:02:38.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-06-22T08:02:38.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-06-22T08:02:38.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-06-22T08:02:38.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-06-22T08:02:38.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-06-22T08:02:38.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-06-22T08:02:38.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-06-22T08:02:38.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-06-22T08:02:38.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-06-22T08:02:38.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-06-22T08:02:38.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-06-22T08:02:38.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-06-22T08:02:38.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-06-22T08:02:38.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-06-22T08:02:38.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-06-22T08:02:38.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-06-22T08:02:38.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-06-22T08:02:38.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-06-22T08:02:38.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-06-22T08:02:38.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-06-22T08:02:38.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-06-22T08:02:38.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-06-22T08:02:38.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-06-22T08:02:38.831 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-06-22T08:02:38.831 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-06-22T08:02:38.831 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-06-22T08:02:38.831 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-06-22T08:02:38.987 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-06-22T08:02:38.988 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-06-22T08:02:38.988 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-06-22T08:02:38.988 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-06-22T08:02:38.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-06-22T08:02:38.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-06-22T08:02:38.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-06-22T08:02:38.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-06-22T08:02:38.989 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-06-22T08:02:38.990 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-06-22T08:02:38.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-06-22T08:02:38.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-06-22T08:02:38.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-06-22T08:02:38.991 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-06-22T08:02:38.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-06-22T08:02:38.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-06-22T08:02:38.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-06-22T08:02:38.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-06-22T08:02:38.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-06-22T08:02:38.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-06-22T08:02:38.992 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-06-22T08:02:38.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-06-22T08:02:38.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-06-22T08:02:38.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-06-22T08:02:38.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-06-22T08:02:38.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-06-22T08:02:38.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-06-22T08:02:38.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-06-22T08:02:38.993 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-06-22T08:02:38.994 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-06-22T08:02:38.994 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-06-22T08:02:38.994 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-06-22T08:02:38.994 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-06-22T08:02:38.994 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-06-22T08:02:38.994 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-06-22T08:02:38.994 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-06-22T08:02:38.994 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-06-22T08:02:38.995 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-06-22T08:02:38.995 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-06-22T08:02:38.995 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-06-22T08:02:38.995 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-06-22T08:02:38.995 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-06-22T08:02:38.995 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-06-22T08:02:38.995 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-06-22T08:02:38.995 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-06-22T08:02:38.996 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-06-22T08:02:38.996 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-06-22T08:02:38.996 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-06-22T08:02:38.996 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-06-22T08:02:38.996 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-06-22T08:02:38.996 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-06-22T08:02:38.996 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-06-22T08:02:38.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-06-22T08:02:38.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-06-22T08:02:38.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-06-22T08:02:38.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-06-22T08:02:38.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-06-22T08:02:38.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-06-22T08:02:38.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-06-22T08:02:38.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-06-22T08:02:38.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-06-22T08:02:38.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-06-22T08:02:38.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-06-22T08:02:38.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-06-22T08:02:38.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-06-22T08:02:38.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-06-22T08:02:38.998 INFO:tasks.workunit.client.0.smithi023.stdout:Take third snapshot .snap/test3 2023-06-22T08:02:39.003 INFO:tasks.workunit.client.0.smithi023.stdout:Delete the snapshots... 2023-06-22T08:02:39.362 INFO:tasks.workunit.client.0.smithi023.stdout:.//.snap/test1 2023-06-22T08:02:39.363 INFO:tasks.workunit.client.0.smithi023.stdout:.//.snap/test2 2023-06-22T08:02:39.363 INFO:tasks.workunit.client.0.smithi023.stdout:.//.snap/test3 2023-06-22T08:02:39.372 INFO:tasks.workunit.client.0.smithi023.stdout:./274/.snap/snap-subdir-test 2023-06-22T08:02:39.381 INFO:tasks.workunit.client.0.smithi023.stdout:./166/.snap/snap-subdir-test 2023-06-22T08:02:39.389 INFO:tasks.workunit.client.0.smithi023.stdout:./256/.snap/snap-subdir-test 2023-06-22T08:02:39.397 INFO:tasks.workunit.client.0.smithi023.stdout:./247/.snap/snap-subdir-test 2023-06-22T08:02:39.407 INFO:tasks.workunit.client.0.smithi023.stdout:./248/.snap/snap-subdir-test 2023-06-22T08:02:39.418 INFO:tasks.workunit.client.0.smithi023.stdout:./147/.snap/snap-subdir-test 2023-06-22T08:02:39.427 INFO:tasks.workunit.client.0.smithi023.stdout:./151/.snap/snap-subdir-test 2023-06-22T08:02:39.437 INFO:tasks.workunit.client.0.smithi023.stdout:./170/.snap/snap-subdir-test 2023-06-22T08:02:39.447 INFO:tasks.workunit.client.0.smithi023.stdout:./167/.snap/snap-subdir-test 2023-06-22T08:02:39.457 INFO:tasks.workunit.client.0.smithi023.stdout:./140/.snap/snap-subdir-test 2023-06-22T08:02:39.466 INFO:tasks.workunit.client.0.smithi023.stdout:./142/.snap/snap-subdir-test 2023-06-22T08:02:39.475 INFO:tasks.workunit.client.0.smithi023.stdout:./150/.snap/snap-subdir-test 2023-06-22T08:02:39.483 INFO:tasks.workunit.client.0.smithi023.stdout:./273/.snap/snap-subdir-test 2023-06-22T08:02:39.492 INFO:tasks.workunit.client.0.smithi023.stdout:./269/.snap/snap-subdir-test 2023-06-22T08:02:39.501 INFO:tasks.workunit.client.0.smithi023.stdout:./105/.snap/snap-subdir-test 2023-06-22T08:02:39.509 INFO:tasks.workunit.client.0.smithi023.stdout:./210/.snap/snap-subdir-test 2023-06-22T08:02:39.518 INFO:tasks.workunit.client.0.smithi023.stdout:./265/.snap/snap-subdir-test 2023-06-22T08:02:39.527 INFO:tasks.workunit.client.0.smithi023.stdout:./218/.snap/snap-subdir-test 2023-06-22T08:02:39.536 INFO:tasks.workunit.client.0.smithi023.stdout:./290/.snap/snap-subdir-test 2023-06-22T08:02:39.545 INFO:tasks.workunit.client.0.smithi023.stdout:./149/.snap/snap-subdir-test 2023-06-22T08:02:39.554 INFO:tasks.workunit.client.0.smithi023.stdout:./109/.snap/snap-subdir-test 2023-06-22T08:02:39.563 INFO:tasks.workunit.client.0.smithi023.stdout:./292/.snap/snap-subdir-test 2023-06-22T08:02:39.573 INFO:tasks.workunit.client.0.smithi023.stdout:./242/.snap/snap-subdir-test 2023-06-22T08:02:39.583 INFO:tasks.workunit.client.0.smithi023.stdout:./158/.snap/snap-subdir-test 2023-06-22T08:02:39.592 INFO:tasks.workunit.client.0.smithi023.stdout:./243/.snap/snap-subdir-test 2023-06-22T08:02:39.601 INFO:tasks.workunit.client.0.smithi023.stdout:./106/.snap/snap-subdir-test 2023-06-22T08:02:39.613 INFO:tasks.workunit.client.0.smithi023.stdout:./250/.snap/snap-subdir-test 2023-06-22T08:02:39.623 INFO:tasks.workunit.client.0.smithi023.stdout:./221/.snap/snap-subdir-test 2023-06-22T08:02:39.633 INFO:tasks.workunit.client.0.smithi023.stdout:./244/.snap/snap-subdir-test 2023-06-22T08:02:39.642 INFO:tasks.workunit.client.0.smithi023.stdout:./258/.snap/snap-subdir-test 2023-06-22T08:02:39.652 INFO:tasks.workunit.client.0.smithi023.stdout:./268/.snap/snap-subdir-test 2023-06-22T08:02:39.661 INFO:tasks.workunit.client.0.smithi023.stdout:./189/.snap/snap-subdir-test 2023-06-22T08:02:39.670 INFO:tasks.workunit.client.0.smithi023.stdout:./285/.snap/snap-subdir-test 2023-06-22T08:02:39.679 INFO:tasks.workunit.client.0.smithi023.stdout:./208/.snap/snap-subdir-test 2023-06-22T08:02:39.693 INFO:tasks.workunit.client.0.smithi023.stdout:./288/.snap/snap-subdir-test 2023-06-22T08:02:39.702 INFO:tasks.workunit.client.0.smithi023.stdout:./209/.snap/snap-subdir-test 2023-06-22T08:02:39.712 INFO:tasks.workunit.client.0.smithi023.stdout:./253/.snap/snap-subdir-test 2023-06-22T08:02:39.722 INFO:tasks.workunit.client.0.smithi023.stdout:./249/.snap/snap-subdir-test 2023-06-22T08:02:39.730 INFO:tasks.workunit.client.0.smithi023.stdout:./133/.snap/snap-subdir-test 2023-06-22T08:02:39.739 INFO:tasks.workunit.client.0.smithi023.stdout:./291/.snap/snap-subdir-test 2023-06-22T08:02:39.748 INFO:tasks.workunit.client.0.smithi023.stdout:./226/.snap/snap-subdir-test 2023-06-22T08:02:39.759 INFO:tasks.workunit.client.0.smithi023.stdout:./173/.snap/snap-subdir-test 2023-06-22T08:02:39.770 INFO:tasks.workunit.client.0.smithi023.stdout:./174/.snap/snap-subdir-test 2023-06-22T08:02:39.779 INFO:tasks.workunit.client.0.smithi023.stdout:./299/.snap/snap-subdir-test 2023-06-22T08:02:39.787 INFO:tasks.workunit.client.0.smithi023.stdout:./262/.snap/snap-subdir-test 2023-06-22T08:02:39.796 INFO:tasks.workunit.client.0.smithi023.stdout:./231/.snap/snap-subdir-test 2023-06-22T08:02:39.805 INFO:tasks.workunit.client.0.smithi023.stdout:./214/.snap/snap-subdir-test 2023-06-22T08:02:39.813 INFO:tasks.workunit.client.0.smithi023.stdout:./281/.snap/snap-subdir-test 2023-06-22T08:02:39.820 INFO:tasks.workunit.client.0.smithi023.stdout:./289/.snap/snap-subdir-test 2023-06-22T08:02:39.829 INFO:tasks.workunit.client.0.smithi023.stdout:./283/.snap/snap-subdir-test 2023-06-22T08:02:39.837 INFO:tasks.workunit.client.0.smithi023.stdout:./232/.snap/snap-subdir-test 2023-06-22T08:02:39.846 INFO:tasks.workunit.client.0.smithi023.stdout:./298/.snap/snap-subdir-test 2023-06-22T08:02:39.858 INFO:tasks.workunit.client.0.smithi023.stdout:./233/.snap/snap-subdir-test 2023-06-22T08:02:39.868 INFO:tasks.workunit.client.0.smithi023.stdout:./155/.snap/snap-subdir-test 2023-06-22T08:02:39.877 INFO:tasks.workunit.client.0.smithi023.stdout:./114/.snap/snap-subdir-test 2023-06-22T08:02:39.890 INFO:tasks.workunit.client.0.smithi023.stdout:./159/.snap/snap-subdir-test 2023-06-22T08:02:39.898 INFO:tasks.workunit.client.0.smithi023.stdout:./187/.snap/snap-subdir-test 2023-06-22T08:02:39.907 INFO:tasks.workunit.client.0.smithi023.stdout:./185/.snap/snap-subdir-test 2023-06-22T08:02:39.915 INFO:tasks.workunit.client.0.smithi023.stdout:./293/.snap/snap-subdir-test 2023-06-22T08:02:39.922 INFO:tasks.workunit.client.0.smithi023.stdout:./228/.snap/snap-subdir-test 2023-06-22T08:02:39.931 INFO:tasks.workunit.client.0.smithi023.stdout:./120/.snap/snap-subdir-test 2023-06-22T08:02:39.940 INFO:tasks.workunit.client.0.smithi023.stdout:./110/.snap/snap-subdir-test 2023-06-22T08:02:39.949 INFO:tasks.workunit.client.0.smithi023.stdout:./267/.snap/snap-subdir-test 2023-06-22T08:02:39.957 INFO:tasks.workunit.client.0.smithi023.stdout:./202/.snap/snap-subdir-test 2023-06-22T08:02:39.967 INFO:tasks.workunit.client.0.smithi023.stdout:./137/.snap/snap-subdir-test 2023-06-22T08:02:39.978 INFO:tasks.workunit.client.0.smithi023.stdout:./246/.snap/snap-subdir-test 2023-06-22T08:02:39.987 INFO:tasks.workunit.client.0.smithi023.stdout:./219/.snap/snap-subdir-test 2023-06-22T08:02:39.995 INFO:tasks.workunit.client.0.smithi023.stdout:./198/.snap/snap-subdir-test 2023-06-22T08:02:40.004 INFO:tasks.workunit.client.0.smithi023.stdout:./136/.snap/snap-subdir-test 2023-06-22T08:02:40.011 INFO:tasks.workunit.client.0.smithi023.stdout:./270/.snap/snap-subdir-test 2023-06-22T08:02:40.022 INFO:tasks.workunit.client.0.smithi023.stdout:./125/.snap/snap-subdir-test 2023-06-22T08:02:40.031 INFO:tasks.workunit.client.0.smithi023.stdout:./194/.snap/snap-subdir-test 2023-06-22T08:02:40.038 INFO:tasks.workunit.client.0.smithi023.stdout:./119/.snap/snap-subdir-test 2023-06-22T08:02:40.047 INFO:tasks.workunit.client.0.smithi023.stdout:./297/.snap/snap-subdir-test 2023-06-22T08:02:40.056 INFO:tasks.workunit.client.0.smithi023.stdout:./230/.snap/snap-subdir-test 2023-06-22T08:02:40.064 INFO:tasks.workunit.client.0.smithi023.stdout:./260/.snap/snap-subdir-test 2023-06-22T08:02:40.073 INFO:tasks.workunit.client.0.smithi023.stdout:./157/.snap/snap-subdir-test 2023-06-22T08:02:40.084 INFO:tasks.workunit.client.0.smithi023.stdout:./172/.snap/snap-subdir-test 2023-06-22T08:02:40.092 INFO:tasks.workunit.client.0.smithi023.stdout:./277/.snap/snap-subdir-test 2023-06-22T08:02:40.101 INFO:tasks.workunit.client.0.smithi023.stdout:./186/.snap/snap-subdir-test 2023-06-22T08:02:40.110 INFO:tasks.workunit.client.0.smithi023.stdout:./237/.snap/snap-subdir-test 2023-06-22T08:02:40.120 INFO:tasks.workunit.client.0.smithi023.stdout:./190/.snap/snap-subdir-test 2023-06-22T08:02:40.130 INFO:tasks.workunit.client.0.smithi023.stdout:./213/.snap/snap-subdir-test 2023-06-22T08:02:40.139 INFO:tasks.workunit.client.0.smithi023.stdout:./104/.snap/snap-subdir-test 2023-06-22T08:02:40.150 INFO:tasks.workunit.client.0.smithi023.stdout:./164/.snap/snap-subdir-test 2023-06-22T08:02:40.161 INFO:tasks.workunit.client.0.smithi023.stdout:./112/.snap/snap-subdir-test 2023-06-22T08:02:40.170 INFO:tasks.workunit.client.0.smithi023.stdout:./295/.snap/snap-subdir-test 2023-06-22T08:02:40.180 INFO:tasks.workunit.client.0.smithi023.stdout:./272/.snap/snap-subdir-test 2023-06-22T08:02:40.188 INFO:tasks.workunit.client.0.smithi023.stdout:./100/.snap/snap-subdir-test 2023-06-22T08:02:40.199 INFO:tasks.workunit.client.0.smithi023.stdout:./134/.snap/snap-subdir-test 2023-06-22T08:02:40.210 INFO:tasks.workunit.client.0.smithi023.stdout:./200/.snap/snap-subdir-test 2023-06-22T08:02:40.220 INFO:tasks.workunit.client.0.smithi023.stdout:./115/.snap/snap-subdir-test 2023-06-22T08:02:40.229 INFO:tasks.workunit.client.0.smithi023.stdout:./284/.snap/snap-subdir-test 2023-06-22T08:02:40.239 INFO:tasks.workunit.client.0.smithi023.stdout:./280/.snap/snap-subdir-test 2023-06-22T08:02:40.249 INFO:tasks.workunit.client.0.smithi023.stdout:./116/.snap/snap-subdir-test 2023-06-22T08:02:40.257 INFO:tasks.workunit.client.0.smithi023.stdout:./245/.snap/snap-subdir-test 2023-06-22T08:02:40.267 INFO:tasks.workunit.client.0.smithi023.stdout:./195/.snap/snap-subdir-test 2023-06-22T08:02:40.277 INFO:tasks.workunit.client.0.smithi023.stdout:./138/.snap/snap-subdir-test 2023-06-22T08:02:40.286 INFO:tasks.workunit.client.0.smithi023.stdout:./135/.snap/snap-subdir-test 2023-06-22T08:02:40.295 INFO:tasks.workunit.client.0.smithi023.stdout:./117/.snap/snap-subdir-test 2023-06-22T08:02:40.302 INFO:tasks.workunit.client.0.smithi023.stdout:./241/.snap/snap-subdir-test 2023-06-22T08:02:40.311 INFO:tasks.workunit.client.0.smithi023.stdout:./143/.snap/snap-subdir-test 2023-06-22T08:02:40.320 INFO:tasks.workunit.client.0.smithi023.stdout:./180/.snap/snap-subdir-test 2023-06-22T08:02:40.329 INFO:tasks.workunit.client.0.smithi023.stdout:./126/.snap/snap-subdir-test 2023-06-22T08:02:40.338 INFO:tasks.workunit.client.0.smithi023.stdout:./161/.snap/snap-subdir-test 2023-06-22T08:02:40.347 INFO:tasks.workunit.client.0.smithi023.stdout:./204/.snap/snap-subdir-test 2023-06-22T08:02:40.356 INFO:tasks.workunit.client.0.smithi023.stdout:./176/.snap/snap-subdir-test 2023-06-22T08:02:40.365 INFO:tasks.workunit.client.0.smithi023.stdout:./255/.snap/snap-subdir-test 2023-06-22T08:02:40.374 INFO:tasks.workunit.client.0.smithi023.stdout:./215/.snap/snap-subdir-test 2023-06-22T08:02:40.383 INFO:tasks.workunit.client.0.smithi023.stdout:./111/.snap/snap-subdir-test 2023-06-22T08:02:40.392 INFO:tasks.workunit.client.0.smithi023.stdout:./238/.snap/snap-subdir-test 2023-06-22T08:02:40.402 INFO:tasks.workunit.client.0.smithi023.stdout:./191/.snap/snap-subdir-test 2023-06-22T08:02:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:./184/.snap/snap-subdir-test 2023-06-22T08:02:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:./192/.snap/snap-subdir-test 2023-06-22T08:02:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:./103/.snap/snap-subdir-test 2023-06-22T08:02:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:./128/.snap/snap-subdir-test 2023-06-22T08:02:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:./229/.snap/snap-subdir-test 2023-06-22T08:02:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:./144/.snap/snap-subdir-test 2023-06-22T08:02:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:./146/.snap/snap-subdir-test 2023-06-22T08:02:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:./227/.snap/snap-subdir-test 2023-06-22T08:02:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:./278/.snap/snap-subdir-test 2023-06-22T08:02:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:./141/.snap/snap-subdir-test 2023-06-22T08:02:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:./234/.snap/snap-subdir-test 2023-06-22T08:02:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:./225/.snap/snap-subdir-test 2023-06-22T08:02:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:./182/.snap/snap-subdir-test 2023-06-22T08:02:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:./107/.snap/snap-subdir-test 2023-06-22T08:02:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:./178/.snap/snap-subdir-test 2023-06-22T08:02:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:./266/.snap/snap-subdir-test 2023-06-22T08:02:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:./240/.snap/snap-subdir-test 2023-06-22T08:02:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:./156/.snap/snap-subdir-test 2023-06-22T08:02:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:./131/.snap/snap-subdir-test 2023-06-22T08:02:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:./123/.snap/snap-subdir-test 2023-06-22T08:02:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:./160/.snap/snap-subdir-test 2023-06-22T08:02:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:./165/.snap/snap-subdir-test 2023-06-22T08:02:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:./101/.snap/snap-subdir-test 2023-06-22T08:02:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:./148/.snap/snap-subdir-test 2023-06-22T08:02:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:./168/.snap/snap-subdir-test 2023-06-22T08:02:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:./179/.snap/snap-subdir-test 2023-06-22T08:02:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:./257/.snap/snap-subdir-test 2023-06-22T08:02:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:./252/.snap/snap-subdir-test 2023-06-22T08:02:40.665 INFO:tasks.workunit.client.0.smithi023.stdout:./132/.snap/snap-subdir-test 2023-06-22T08:02:40.675 INFO:tasks.workunit.client.0.smithi023.stdout:./236/.snap/snap-subdir-test 2023-06-22T08:02:40.683 INFO:tasks.workunit.client.0.smithi023.stdout:./211/.snap/snap-subdir-test 2023-06-22T08:02:40.692 INFO:tasks.workunit.client.0.smithi023.stdout:./199/.snap/snap-subdir-test 2023-06-22T08:02:40.700 INFO:tasks.workunit.client.0.smithi023.stdout:./205/.snap/snap-subdir-test 2023-06-22T08:02:40.710 INFO:tasks.workunit.client.0.smithi023.stdout:./163/.snap/snap-subdir-test 2023-06-22T08:02:40.720 INFO:tasks.workunit.client.0.smithi023.stdout:./261/.snap/snap-subdir-test 2023-06-22T08:02:40.729 INFO:tasks.workunit.client.0.smithi023.stdout:./201/.snap/snap-subdir-test 2023-06-22T08:02:40.736 INFO:tasks.workunit.client.0.smithi023.stdout:./282/.snap/snap-subdir-test 2023-06-22T08:02:40.746 INFO:tasks.workunit.client.0.smithi023.stdout:./296/.snap/snap-subdir-test 2023-06-22T08:02:40.755 INFO:tasks.workunit.client.0.smithi023.stdout:./217/.snap/snap-subdir-test 2023-06-22T08:02:40.765 INFO:tasks.workunit.client.0.smithi023.stdout:./153/.snap/snap-subdir-test 2023-06-22T08:02:40.775 INFO:tasks.workunit.client.0.smithi023.stdout:./206/.snap/snap-subdir-test 2023-06-22T08:02:40.788 INFO:tasks.workunit.client.0.smithi023.stdout:./122/.snap/snap-subdir-test 2023-06-22T08:02:40.797 INFO:tasks.workunit.client.0.smithi023.stdout:./162/.snap/snap-subdir-test 2023-06-22T08:02:40.806 INFO:tasks.workunit.client.0.smithi023.stdout:./197/.snap/snap-subdir-test 2023-06-22T08:02:40.816 INFO:tasks.workunit.client.0.smithi023.stdout:./216/.snap/snap-subdir-test 2023-06-22T08:02:41.613 INFO:tasks.workunit.client.0.smithi023.stdout:./254/.snap/snap-subdir-test 2023-06-22T08:02:41.621 INFO:tasks.workunit.client.0.smithi023.stdout:./113/.snap/snap-subdir-test 2023-06-22T08:02:41.631 INFO:tasks.workunit.client.0.smithi023.stdout:./152/.snap/snap-subdir-test 2023-06-22T08:02:41.642 INFO:tasks.workunit.client.0.smithi023.stdout:./121/.snap/snap-subdir-test 2023-06-22T08:02:41.652 INFO:tasks.workunit.client.0.smithi023.stdout:./118/.snap/snap-subdir-test 2023-06-22T08:02:41.662 INFO:tasks.workunit.client.0.smithi023.stdout:./196/.snap/snap-subdir-test 2023-06-22T08:02:41.673 INFO:tasks.workunit.client.0.smithi023.stdout:./124/.snap/snap-subdir-test 2023-06-22T08:02:41.684 INFO:tasks.workunit.client.0.smithi023.stdout:./239/.snap/snap-subdir-test 2023-06-22T08:02:41.693 INFO:tasks.workunit.client.0.smithi023.stdout:./188/.snap/snap-subdir-test 2023-06-22T08:02:41.700 INFO:tasks.workunit.client.0.smithi023.stdout:./287/.snap/snap-subdir-test 2023-06-22T08:02:41.710 INFO:tasks.workunit.client.0.smithi023.stdout:./275/.snap/snap-subdir-test 2023-06-22T08:02:41.718 INFO:tasks.workunit.client.0.smithi023.stdout:./259/.snap/snap-subdir-test 2023-06-22T08:02:41.727 INFO:tasks.workunit.client.0.smithi023.stdout:./224/.snap/snap-subdir-test 2023-06-22T08:02:41.737 INFO:tasks.workunit.client.0.smithi023.stdout:./264/.snap/snap-subdir-test 2023-06-22T08:02:41.746 INFO:tasks.workunit.client.0.smithi023.stdout:./145/.snap/snap-subdir-test 2023-06-22T08:02:41.755 INFO:tasks.workunit.client.0.smithi023.stdout:./279/.snap/snap-subdir-test 2023-06-22T08:02:41.764 INFO:tasks.workunit.client.0.smithi023.stdout:./263/.snap/snap-subdir-test 2023-06-22T08:02:41.768 INFO:tasks.workunit.client.0.smithi023.stdout:./130/.snap/snap-subdir-test 2023-06-22T08:02:41.772 INFO:tasks.workunit.client.0.smithi023.stdout:./193/.snap/snap-subdir-test 2023-06-22T08:02:41.781 INFO:tasks.workunit.client.0.smithi023.stdout:./223/.snap/snap-subdir-test 2023-06-22T08:02:41.788 INFO:tasks.workunit.client.0.smithi023.stdout:./207/.snap/snap-subdir-test 2023-06-22T08:02:41.796 INFO:tasks.workunit.client.0.smithi023.stdout:./276/.snap/snap-subdir-test 2023-06-22T08:02:41.806 INFO:tasks.workunit.client.0.smithi023.stdout:./271/.snap/snap-subdir-test 2023-06-22T08:02:41.816 INFO:tasks.workunit.client.0.smithi023.stdout:./251/.snap/snap-subdir-test 2023-06-22T08:02:41.825 INFO:tasks.workunit.client.0.smithi023.stdout:./175/.snap/snap-subdir-test 2023-06-22T08:02:41.833 INFO:tasks.workunit.client.0.smithi023.stdout:./222/.snap/snap-subdir-test 2023-06-22T08:02:41.842 INFO:tasks.workunit.client.0.smithi023.stdout:./220/.snap/snap-subdir-test 2023-06-22T08:02:41.851 INFO:tasks.workunit.client.0.smithi023.stdout:./203/.snap/snap-subdir-test 2023-06-22T08:02:41.860 INFO:tasks.workunit.client.0.smithi023.stdout:./181/.snap/snap-subdir-test 2023-06-22T08:02:41.869 INFO:tasks.workunit.client.0.smithi023.stdout:./286/.snap/snap-subdir-test 2023-06-22T08:02:41.873 INFO:tasks.workunit.client.0.smithi023.stdout:./127/.snap/snap-subdir-test 2023-06-22T08:02:41.882 INFO:tasks.workunit.client.0.smithi023.stdout:./108/.snap/snap-subdir-test 2023-06-22T08:02:41.891 INFO:tasks.workunit.client.0.smithi023.stdout:./183/.snap/snap-subdir-test 2023-06-22T08:02:41.898 INFO:tasks.workunit.client.0.smithi023.stdout:./294/.snap/snap-subdir-test 2023-06-22T08:02:41.907 INFO:tasks.workunit.client.0.smithi023.stdout:./235/.snap/snap-subdir-test 2023-06-22T08:02:41.911 INFO:tasks.workunit.client.0.smithi023.stdout:./177/.snap/snap-subdir-test 2023-06-22T08:02:41.918 INFO:tasks.workunit.client.0.smithi023.stdout:./169/.snap/snap-subdir-test 2023-06-22T08:02:41.928 INFO:tasks.workunit.client.0.smithi023.stdout:./212/.snap/snap-subdir-test 2023-06-22T08:02:41.935 INFO:tasks.workunit.client.0.smithi023.stdout:./154/.snap/snap-subdir-test 2023-06-22T08:02:41.944 INFO:tasks.workunit.client.0.smithi023.stdout:./171/.snap/snap-subdir-test 2023-06-22T08:02:41.948 INFO:tasks.workunit.client.0.smithi023.stdout:./139/.snap/snap-subdir-test 2023-06-22T08:02:41.952 INFO:tasks.workunit.client.0.smithi023.stdout:./102/.snap/snap-subdir-test 2023-06-22T08:02:41.961 INFO:tasks.workunit.client.0.smithi023.stdout:./129/.snap/snap-subdir-test 2023-06-22T08:02:43.603 INFO:tasks.workunit.client.0.smithi023.stdout:Delete all the files and directories ... 2023-06-22T08:02:44.052 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/19' 2023-06-22T08:02:44.053 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/15' 2023-06-22T08:02:44.053 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/31' 2023-06-22T08:02:44.053 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/36' 2023-06-22T08:02:44.054 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/20' 2023-06-22T08:02:44.054 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/12' 2023-06-22T08:02:44.054 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/39' 2023-06-22T08:02:44.054 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/10' 2023-06-22T08:02:44.055 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/34' 2023-06-22T08:02:44.055 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/38' 2023-06-22T08:02:44.055 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/11' 2023-06-22T08:02:44.055 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/16' 2023-06-22T08:02:44.055 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/32' 2023-06-22T08:02:44.056 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/30' 2023-06-22T08:02:44.056 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/14' 2023-06-22T08:02:44.056 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/18' 2023-06-22T08:02:44.056 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/13' 2023-06-22T08:02:44.056 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/33' 2023-06-22T08:02:44.056 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/17' 2023-06-22T08:02:44.056 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/35' 2023-06-22T08:02:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './100/37' 2023-06-22T08:02:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './100' 2023-06-22T08:02:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/19' 2023-06-22T08:02:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/15' 2023-06-22T08:02:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/31' 2023-06-22T08:02:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/36' 2023-06-22T08:02:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/20' 2023-06-22T08:02:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/12' 2023-06-22T08:02:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/39' 2023-06-22T08:02:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/10' 2023-06-22T08:02:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/34' 2023-06-22T08:02:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/38' 2023-06-22T08:02:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/11' 2023-06-22T08:02:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/16' 2023-06-22T08:02:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/32' 2023-06-22T08:02:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/30' 2023-06-22T08:02:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/14' 2023-06-22T08:02:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/18' 2023-06-22T08:02:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/13' 2023-06-22T08:02:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/33' 2023-06-22T08:02:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/17' 2023-06-22T08:02:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/35' 2023-06-22T08:02:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './101/37' 2023-06-22T08:02:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './101' 2023-06-22T08:02:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/19' 2023-06-22T08:02:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/15' 2023-06-22T08:02:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/31' 2023-06-22T08:02:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/36' 2023-06-22T08:02:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/20' 2023-06-22T08:02:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/12' 2023-06-22T08:02:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/39' 2023-06-22T08:02:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/10' 2023-06-22T08:02:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/34' 2023-06-22T08:02:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/38' 2023-06-22T08:02:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/11' 2023-06-22T08:02:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/16' 2023-06-22T08:02:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/32' 2023-06-22T08:02:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/30' 2023-06-22T08:02:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/14' 2023-06-22T08:02:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/18' 2023-06-22T08:02:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/13' 2023-06-22T08:02:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/33' 2023-06-22T08:02:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/17' 2023-06-22T08:02:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/35' 2023-06-22T08:02:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './102/37' 2023-06-22T08:02:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './102' 2023-06-22T08:02:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/19' 2023-06-22T08:02:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/15' 2023-06-22T08:02:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/31' 2023-06-22T08:02:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/36' 2023-06-22T08:02:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/20' 2023-06-22T08:02:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/12' 2023-06-22T08:02:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/39' 2023-06-22T08:02:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/10' 2023-06-22T08:02:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/34' 2023-06-22T08:02:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/38' 2023-06-22T08:02:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/11' 2023-06-22T08:02:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/16' 2023-06-22T08:02:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/32' 2023-06-22T08:02:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/30' 2023-06-22T08:02:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/14' 2023-06-22T08:02:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/18' 2023-06-22T08:02:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/13' 2023-06-22T08:02:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/33' 2023-06-22T08:02:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/17' 2023-06-22T08:02:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/35' 2023-06-22T08:02:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './103/37' 2023-06-22T08:02:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './103' 2023-06-22T08:02:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/19' 2023-06-22T08:02:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/15' 2023-06-22T08:02:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/31' 2023-06-22T08:02:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/36' 2023-06-22T08:02:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/20' 2023-06-22T08:02:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/12' 2023-06-22T08:02:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/39' 2023-06-22T08:02:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/10' 2023-06-22T08:02:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/34' 2023-06-22T08:02:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/38' 2023-06-22T08:02:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/11' 2023-06-22T08:02:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/16' 2023-06-22T08:02:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/32' 2023-06-22T08:02:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/30' 2023-06-22T08:02:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/14' 2023-06-22T08:02:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/18' 2023-06-22T08:02:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/13' 2023-06-22T08:02:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/33' 2023-06-22T08:02:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/17' 2023-06-22T08:02:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/35' 2023-06-22T08:02:44.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './104/37' 2023-06-22T08:02:44.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './104' 2023-06-22T08:02:44.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/19' 2023-06-22T08:02:44.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/15' 2023-06-22T08:02:44.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/31' 2023-06-22T08:02:44.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/36' 2023-06-22T08:02:44.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/20' 2023-06-22T08:02:44.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/12' 2023-06-22T08:02:44.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/39' 2023-06-22T08:02:44.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/10' 2023-06-22T08:02:44.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/34' 2023-06-22T08:02:44.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/38' 2023-06-22T08:02:44.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/11' 2023-06-22T08:02:44.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/16' 2023-06-22T08:02:44.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/32' 2023-06-22T08:02:44.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/30' 2023-06-22T08:02:44.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/14' 2023-06-22T08:02:44.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/18' 2023-06-22T08:02:44.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/13' 2023-06-22T08:02:44.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/33' 2023-06-22T08:02:44.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/17' 2023-06-22T08:02:44.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/35' 2023-06-22T08:02:44.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './105/37' 2023-06-22T08:02:44.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './105' 2023-06-22T08:02:44.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/19' 2023-06-22T08:02:44.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/15' 2023-06-22T08:02:44.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/31' 2023-06-22T08:02:44.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/36' 2023-06-22T08:02:44.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/20' 2023-06-22T08:02:44.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/12' 2023-06-22T08:02:44.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/39' 2023-06-22T08:02:44.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/10' 2023-06-22T08:02:44.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/34' 2023-06-22T08:02:44.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/38' 2023-06-22T08:02:44.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/11' 2023-06-22T08:02:44.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/16' 2023-06-22T08:02:44.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/32' 2023-06-22T08:02:44.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/30' 2023-06-22T08:02:44.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/14' 2023-06-22T08:02:44.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/18' 2023-06-22T08:02:44.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/13' 2023-06-22T08:02:44.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/33' 2023-06-22T08:02:44.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/17' 2023-06-22T08:02:44.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/35' 2023-06-22T08:02:44.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './106/37' 2023-06-22T08:02:44.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './106' 2023-06-22T08:02:44.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/19' 2023-06-22T08:02:44.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/15' 2023-06-22T08:02:44.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/31' 2023-06-22T08:02:44.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/36' 2023-06-22T08:02:44.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/20' 2023-06-22T08:02:44.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/12' 2023-06-22T08:02:44.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/39' 2023-06-22T08:02:44.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/10' 2023-06-22T08:02:44.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/34' 2023-06-22T08:02:44.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/38' 2023-06-22T08:02:44.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/11' 2023-06-22T08:02:44.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/16' 2023-06-22T08:02:44.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/32' 2023-06-22T08:02:44.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/30' 2023-06-22T08:02:44.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/14' 2023-06-22T08:02:44.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/18' 2023-06-22T08:02:44.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/13' 2023-06-22T08:02:44.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/33' 2023-06-22T08:02:44.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/17' 2023-06-22T08:02:44.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/35' 2023-06-22T08:02:44.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './107/37' 2023-06-22T08:02:44.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './107' 2023-06-22T08:02:44.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/19' 2023-06-22T08:02:44.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/15' 2023-06-22T08:02:44.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/31' 2023-06-22T08:02:44.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/36' 2023-06-22T08:02:44.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/20' 2023-06-22T08:02:44.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/12' 2023-06-22T08:02:44.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/39' 2023-06-22T08:02:44.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/10' 2023-06-22T08:02:44.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/34' 2023-06-22T08:02:44.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/38' 2023-06-22T08:02:44.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/11' 2023-06-22T08:02:44.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/16' 2023-06-22T08:02:44.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/32' 2023-06-22T08:02:44.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/30' 2023-06-22T08:02:44.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/14' 2023-06-22T08:02:44.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/18' 2023-06-22T08:02:44.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/13' 2023-06-22T08:02:44.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/33' 2023-06-22T08:02:44.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/17' 2023-06-22T08:02:44.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/35' 2023-06-22T08:02:44.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed './108/37' 2023-06-22T08:02:44.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './108' 2023-06-22T08:02:44.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/19' 2023-06-22T08:02:44.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/15' 2023-06-22T08:02:44.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/31' 2023-06-22T08:02:44.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/36' 2023-06-22T08:02:44.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/20' 2023-06-22T08:02:44.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/12' 2023-06-22T08:02:44.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/39' 2023-06-22T08:02:44.088 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/10' 2023-06-22T08:02:44.088 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/34' 2023-06-22T08:02:44.088 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/38' 2023-06-22T08:02:44.088 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/11' 2023-06-22T08:02:44.088 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/16' 2023-06-22T08:02:44.088 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/32' 2023-06-22T08:02:44.088 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/30' 2023-06-22T08:02:44.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/14' 2023-06-22T08:02:44.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/18' 2023-06-22T08:02:44.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/13' 2023-06-22T08:02:44.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/33' 2023-06-22T08:02:44.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/17' 2023-06-22T08:02:44.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/35' 2023-06-22T08:02:44.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './109/37' 2023-06-22T08:02:44.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './109' 2023-06-22T08:02:44.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/19' 2023-06-22T08:02:44.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/15' 2023-06-22T08:02:44.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/31' 2023-06-22T08:02:44.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/36' 2023-06-22T08:02:44.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/20' 2023-06-22T08:02:44.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/12' 2023-06-22T08:02:44.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/39' 2023-06-22T08:02:44.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/10' 2023-06-22T08:02:44.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/34' 2023-06-22T08:02:44.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/38' 2023-06-22T08:02:44.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/11' 2023-06-22T08:02:44.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/16' 2023-06-22T08:02:44.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/32' 2023-06-22T08:02:44.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/30' 2023-06-22T08:02:44.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/14' 2023-06-22T08:02:44.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/18' 2023-06-22T08:02:44.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/13' 2023-06-22T08:02:44.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/33' 2023-06-22T08:02:44.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/17' 2023-06-22T08:02:44.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/35' 2023-06-22T08:02:44.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed './110/37' 2023-06-22T08:02:44.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './110' 2023-06-22T08:02:44.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/19' 2023-06-22T08:02:44.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/15' 2023-06-22T08:02:44.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/31' 2023-06-22T08:02:44.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/36' 2023-06-22T08:02:44.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/20' 2023-06-22T08:02:44.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/12' 2023-06-22T08:02:44.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/39' 2023-06-22T08:02:44.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/10' 2023-06-22T08:02:44.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/34' 2023-06-22T08:02:44.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/38' 2023-06-22T08:02:44.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/11' 2023-06-22T08:02:44.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/16' 2023-06-22T08:02:44.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/32' 2023-06-22T08:02:44.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/30' 2023-06-22T08:02:44.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/14' 2023-06-22T08:02:44.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/18' 2023-06-22T08:02:44.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/13' 2023-06-22T08:02:44.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/33' 2023-06-22T08:02:44.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/17' 2023-06-22T08:02:44.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/35' 2023-06-22T08:02:44.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed './111/37' 2023-06-22T08:02:44.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './111' 2023-06-22T08:02:44.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/19' 2023-06-22T08:02:44.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/15' 2023-06-22T08:02:44.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/31' 2023-06-22T08:02:44.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/36' 2023-06-22T08:02:44.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/20' 2023-06-22T08:02:44.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/12' 2023-06-22T08:02:44.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/39' 2023-06-22T08:02:44.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/10' 2023-06-22T08:02:44.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/34' 2023-06-22T08:02:44.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/38' 2023-06-22T08:02:44.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/11' 2023-06-22T08:02:44.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/16' 2023-06-22T08:02:44.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/32' 2023-06-22T08:02:44.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/30' 2023-06-22T08:02:44.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/14' 2023-06-22T08:02:44.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/18' 2023-06-22T08:02:44.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/13' 2023-06-22T08:02:44.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/33' 2023-06-22T08:02:44.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/17' 2023-06-22T08:02:44.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/35' 2023-06-22T08:02:44.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './112/37' 2023-06-22T08:02:44.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './112' 2023-06-22T08:02:44.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/19' 2023-06-22T08:02:44.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/15' 2023-06-22T08:02:44.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/31' 2023-06-22T08:02:44.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/36' 2023-06-22T08:02:44.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/20' 2023-06-22T08:02:44.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/12' 2023-06-22T08:02:44.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/39' 2023-06-22T08:02:44.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/10' 2023-06-22T08:02:44.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/34' 2023-06-22T08:02:44.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/38' 2023-06-22T08:02:44.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/11' 2023-06-22T08:02:44.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/16' 2023-06-22T08:02:44.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/32' 2023-06-22T08:02:44.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/30' 2023-06-22T08:02:44.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/14' 2023-06-22T08:02:44.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/18' 2023-06-22T08:02:44.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/13' 2023-06-22T08:02:44.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/33' 2023-06-22T08:02:44.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/17' 2023-06-22T08:02:44.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/35' 2023-06-22T08:02:44.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './113/37' 2023-06-22T08:02:44.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './113' 2023-06-22T08:02:44.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/19' 2023-06-22T08:02:44.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/15' 2023-06-22T08:02:44.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/31' 2023-06-22T08:02:44.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/36' 2023-06-22T08:02:44.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/20' 2023-06-22T08:02:44.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/12' 2023-06-22T08:02:44.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/39' 2023-06-22T08:02:44.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/10' 2023-06-22T08:02:44.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/34' 2023-06-22T08:02:44.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/38' 2023-06-22T08:02:44.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/11' 2023-06-22T08:02:44.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/16' 2023-06-22T08:02:44.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/32' 2023-06-22T08:02:44.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/30' 2023-06-22T08:02:44.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/14' 2023-06-22T08:02:44.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/18' 2023-06-22T08:02:44.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/13' 2023-06-22T08:02:44.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/33' 2023-06-22T08:02:44.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/17' 2023-06-22T08:02:44.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/35' 2023-06-22T08:02:44.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './114/37' 2023-06-22T08:02:44.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './114' 2023-06-22T08:02:44.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/19' 2023-06-22T08:02:44.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/15' 2023-06-22T08:02:44.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/31' 2023-06-22T08:02:44.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/36' 2023-06-22T08:02:44.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/20' 2023-06-22T08:02:44.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/12' 2023-06-22T08:02:44.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/39' 2023-06-22T08:02:44.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/10' 2023-06-22T08:02:44.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/34' 2023-06-22T08:02:44.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/38' 2023-06-22T08:02:44.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/11' 2023-06-22T08:02:44.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/16' 2023-06-22T08:02:44.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/32' 2023-06-22T08:02:44.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/30' 2023-06-22T08:02:44.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/14' 2023-06-22T08:02:44.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/18' 2023-06-22T08:02:44.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/13' 2023-06-22T08:02:44.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/33' 2023-06-22T08:02:44.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/17' 2023-06-22T08:02:44.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/35' 2023-06-22T08:02:44.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed './115/37' 2023-06-22T08:02:44.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './115' 2023-06-22T08:02:44.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/19' 2023-06-22T08:02:44.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/15' 2023-06-22T08:02:44.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/31' 2023-06-22T08:02:44.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/36' 2023-06-22T08:02:44.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/20' 2023-06-22T08:02:44.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/12' 2023-06-22T08:02:44.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/39' 2023-06-22T08:02:44.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/10' 2023-06-22T08:02:44.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/34' 2023-06-22T08:02:44.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/38' 2023-06-22T08:02:44.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/11' 2023-06-22T08:02:44.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/16' 2023-06-22T08:02:44.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/32' 2023-06-22T08:02:44.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/30' 2023-06-22T08:02:44.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/14' 2023-06-22T08:02:44.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/18' 2023-06-22T08:02:44.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/13' 2023-06-22T08:02:44.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/33' 2023-06-22T08:02:44.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/17' 2023-06-22T08:02:44.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/35' 2023-06-22T08:02:44.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './116/37' 2023-06-22T08:02:44.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './116' 2023-06-22T08:02:44.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/19' 2023-06-22T08:02:44.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/15' 2023-06-22T08:02:44.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/31' 2023-06-22T08:02:44.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/36' 2023-06-22T08:02:44.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/20' 2023-06-22T08:02:44.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/12' 2023-06-22T08:02:44.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/39' 2023-06-22T08:02:44.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/10' 2023-06-22T08:02:44.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/34' 2023-06-22T08:02:44.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/38' 2023-06-22T08:02:44.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/11' 2023-06-22T08:02:44.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/16' 2023-06-22T08:02:44.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/32' 2023-06-22T08:02:44.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/30' 2023-06-22T08:02:44.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/14' 2023-06-22T08:02:44.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/18' 2023-06-22T08:02:44.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/13' 2023-06-22T08:02:44.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/33' 2023-06-22T08:02:44.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/17' 2023-06-22T08:02:44.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/35' 2023-06-22T08:02:44.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './117/37' 2023-06-22T08:02:44.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './117' 2023-06-22T08:02:44.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/19' 2023-06-22T08:02:44.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/15' 2023-06-22T08:02:44.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/31' 2023-06-22T08:02:44.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/36' 2023-06-22T08:02:44.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/20' 2023-06-22T08:02:44.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/12' 2023-06-22T08:02:44.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/39' 2023-06-22T08:02:44.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/10' 2023-06-22T08:02:44.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/34' 2023-06-22T08:02:44.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/38' 2023-06-22T08:02:44.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/11' 2023-06-22T08:02:44.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/16' 2023-06-22T08:02:44.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/32' 2023-06-22T08:02:44.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/30' 2023-06-22T08:02:44.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/14' 2023-06-22T08:02:44.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/18' 2023-06-22T08:02:44.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/13' 2023-06-22T08:02:44.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/33' 2023-06-22T08:02:44.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/17' 2023-06-22T08:02:44.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/35' 2023-06-22T08:02:44.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './118/37' 2023-06-22T08:02:44.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './118' 2023-06-22T08:02:44.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/19' 2023-06-22T08:02:44.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/15' 2023-06-22T08:02:44.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/31' 2023-06-22T08:02:44.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/36' 2023-06-22T08:02:44.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/20' 2023-06-22T08:02:44.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/12' 2023-06-22T08:02:45.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/39' 2023-06-22T08:02:45.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/10' 2023-06-22T08:02:45.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/34' 2023-06-22T08:02:45.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/38' 2023-06-22T08:02:45.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/11' 2023-06-22T08:02:45.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/16' 2023-06-22T08:02:45.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/32' 2023-06-22T08:02:45.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/30' 2023-06-22T08:02:45.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/14' 2023-06-22T08:02:45.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/18' 2023-06-22T08:02:45.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/13' 2023-06-22T08:02:45.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/33' 2023-06-22T08:02:45.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/17' 2023-06-22T08:02:45.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/35' 2023-06-22T08:02:45.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './119/37' 2023-06-22T08:02:45.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './119' 2023-06-22T08:02:45.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/19' 2023-06-22T08:02:45.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/15' 2023-06-22T08:02:45.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/31' 2023-06-22T08:02:45.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/36' 2023-06-22T08:02:45.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/20' 2023-06-22T08:02:45.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/12' 2023-06-22T08:02:45.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/39' 2023-06-22T08:02:45.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/10' 2023-06-22T08:02:45.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/34' 2023-06-22T08:02:45.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/38' 2023-06-22T08:02:45.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/11' 2023-06-22T08:02:45.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/16' 2023-06-22T08:02:45.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/32' 2023-06-22T08:02:45.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/30' 2023-06-22T08:02:45.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/14' 2023-06-22T08:02:45.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/18' 2023-06-22T08:02:45.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/13' 2023-06-22T08:02:45.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/33' 2023-06-22T08:02:45.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/17' 2023-06-22T08:02:45.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/35' 2023-06-22T08:02:45.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './120/37' 2023-06-22T08:02:45.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './120' 2023-06-22T08:02:45.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/19' 2023-06-22T08:02:45.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/15' 2023-06-22T08:02:45.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/31' 2023-06-22T08:02:45.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/36' 2023-06-22T08:02:45.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/20' 2023-06-22T08:02:45.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/12' 2023-06-22T08:02:45.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/39' 2023-06-22T08:02:45.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/10' 2023-06-22T08:02:45.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/34' 2023-06-22T08:02:45.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/38' 2023-06-22T08:02:45.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/11' 2023-06-22T08:02:45.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/16' 2023-06-22T08:02:45.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/32' 2023-06-22T08:02:45.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/30' 2023-06-22T08:02:45.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/14' 2023-06-22T08:02:45.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/18' 2023-06-22T08:02:45.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/13' 2023-06-22T08:02:45.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/33' 2023-06-22T08:02:45.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/17' 2023-06-22T08:02:45.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/35' 2023-06-22T08:02:45.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './121/37' 2023-06-22T08:02:45.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './121' 2023-06-22T08:02:45.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/19' 2023-06-22T08:02:45.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/15' 2023-06-22T08:02:45.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/31' 2023-06-22T08:02:45.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/36' 2023-06-22T08:02:45.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/20' 2023-06-22T08:02:45.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/12' 2023-06-22T08:02:45.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/39' 2023-06-22T08:02:45.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/10' 2023-06-22T08:02:45.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/34' 2023-06-22T08:02:45.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/38' 2023-06-22T08:02:45.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/11' 2023-06-22T08:02:45.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/16' 2023-06-22T08:02:45.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/32' 2023-06-22T08:02:45.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/30' 2023-06-22T08:02:45.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/14' 2023-06-22T08:02:45.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/18' 2023-06-22T08:02:45.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/13' 2023-06-22T08:02:45.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/33' 2023-06-22T08:02:45.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/17' 2023-06-22T08:02:45.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/35' 2023-06-22T08:02:45.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './122/37' 2023-06-22T08:02:45.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './122' 2023-06-22T08:02:45.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/19' 2023-06-22T08:02:45.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/15' 2023-06-22T08:02:45.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/31' 2023-06-22T08:02:45.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/36' 2023-06-22T08:02:45.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/20' 2023-06-22T08:02:45.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/12' 2023-06-22T08:02:45.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/39' 2023-06-22T08:02:45.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/10' 2023-06-22T08:02:45.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/34' 2023-06-22T08:02:45.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/38' 2023-06-22T08:02:45.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/11' 2023-06-22T08:02:45.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/16' 2023-06-22T08:02:45.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/32' 2023-06-22T08:02:45.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/30' 2023-06-22T08:02:45.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/14' 2023-06-22T08:02:45.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/18' 2023-06-22T08:02:45.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/13' 2023-06-22T08:02:45.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/33' 2023-06-22T08:02:45.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/17' 2023-06-22T08:02:45.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/35' 2023-06-22T08:02:45.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './123/37' 2023-06-22T08:02:45.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './123' 2023-06-22T08:02:45.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/19' 2023-06-22T08:02:45.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/15' 2023-06-22T08:02:45.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/31' 2023-06-22T08:02:45.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/36' 2023-06-22T08:02:45.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/20' 2023-06-22T08:02:45.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/12' 2023-06-22T08:02:45.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/39' 2023-06-22T08:02:45.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/10' 2023-06-22T08:02:45.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/34' 2023-06-22T08:02:45.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/38' 2023-06-22T08:02:45.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/11' 2023-06-22T08:02:45.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/16' 2023-06-22T08:02:45.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/32' 2023-06-22T08:02:45.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/30' 2023-06-22T08:02:45.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/14' 2023-06-22T08:02:45.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/18' 2023-06-22T08:02:45.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/13' 2023-06-22T08:02:45.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/33' 2023-06-22T08:02:45.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/17' 2023-06-22T08:02:45.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/35' 2023-06-22T08:02:45.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './124/37' 2023-06-22T08:02:45.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './124' 2023-06-22T08:02:45.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/19' 2023-06-22T08:02:45.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/15' 2023-06-22T08:02:45.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/31' 2023-06-22T08:02:45.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/36' 2023-06-22T08:02:45.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/20' 2023-06-22T08:02:45.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/12' 2023-06-22T08:02:45.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/39' 2023-06-22T08:02:45.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/10' 2023-06-22T08:02:45.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/34' 2023-06-22T08:02:45.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/38' 2023-06-22T08:02:45.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/11' 2023-06-22T08:02:45.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/16' 2023-06-22T08:02:45.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/32' 2023-06-22T08:02:45.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/30' 2023-06-22T08:02:45.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/14' 2023-06-22T08:02:45.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/18' 2023-06-22T08:02:45.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/13' 2023-06-22T08:02:45.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/33' 2023-06-22T08:02:45.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/17' 2023-06-22T08:02:45.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/35' 2023-06-22T08:02:45.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './125/37' 2023-06-22T08:02:45.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './125' 2023-06-22T08:02:45.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/19' 2023-06-22T08:02:45.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/15' 2023-06-22T08:02:45.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/31' 2023-06-22T08:02:45.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/36' 2023-06-22T08:02:45.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/20' 2023-06-22T08:02:45.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/12' 2023-06-22T08:02:45.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/39' 2023-06-22T08:02:45.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/10' 2023-06-22T08:02:45.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/34' 2023-06-22T08:02:45.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/38' 2023-06-22T08:02:45.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/11' 2023-06-22T08:02:45.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/16' 2023-06-22T08:02:45.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/32' 2023-06-22T08:02:45.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/30' 2023-06-22T08:02:45.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/14' 2023-06-22T08:02:45.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/18' 2023-06-22T08:02:45.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/13' 2023-06-22T08:02:45.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/33' 2023-06-22T08:02:45.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/17' 2023-06-22T08:02:45.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/35' 2023-06-22T08:02:45.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed './126/37' 2023-06-22T08:02:45.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './126' 2023-06-22T08:02:45.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/19' 2023-06-22T08:02:45.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/15' 2023-06-22T08:02:45.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/31' 2023-06-22T08:02:45.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/36' 2023-06-22T08:02:45.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/20' 2023-06-22T08:02:45.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/12' 2023-06-22T08:02:45.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/39' 2023-06-22T08:02:45.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/10' 2023-06-22T08:02:45.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/34' 2023-06-22T08:02:45.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/38' 2023-06-22T08:02:45.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/11' 2023-06-22T08:02:45.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/16' 2023-06-22T08:02:45.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/32' 2023-06-22T08:02:45.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/30' 2023-06-22T08:02:45.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/14' 2023-06-22T08:02:45.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/18' 2023-06-22T08:02:45.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/13' 2023-06-22T08:02:45.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/33' 2023-06-22T08:02:45.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/17' 2023-06-22T08:02:45.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/35' 2023-06-22T08:02:45.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './127/37' 2023-06-22T08:02:45.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './127' 2023-06-22T08:02:45.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/19' 2023-06-22T08:02:45.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/15' 2023-06-22T08:02:45.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/31' 2023-06-22T08:02:45.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/36' 2023-06-22T08:02:45.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/20' 2023-06-22T08:02:45.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/12' 2023-06-22T08:02:45.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/39' 2023-06-22T08:02:45.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/10' 2023-06-22T08:02:45.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/34' 2023-06-22T08:02:45.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/38' 2023-06-22T08:02:45.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/11' 2023-06-22T08:02:45.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/16' 2023-06-22T08:02:45.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/32' 2023-06-22T08:02:45.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/30' 2023-06-22T08:02:45.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/14' 2023-06-22T08:02:45.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/18' 2023-06-22T08:02:45.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/13' 2023-06-22T08:02:45.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/33' 2023-06-22T08:02:45.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/17' 2023-06-22T08:02:45.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/35' 2023-06-22T08:02:45.555 INFO:tasks.workunit.client.0.smithi023.stdout:removed './128/37' 2023-06-22T08:02:45.555 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './128' 2023-06-22T08:02:45.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/19' 2023-06-22T08:02:45.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/15' 2023-06-22T08:02:45.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/31' 2023-06-22T08:02:45.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/36' 2023-06-22T08:02:45.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/20' 2023-06-22T08:02:45.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/12' 2023-06-22T08:02:45.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/39' 2023-06-22T08:02:45.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/10' 2023-06-22T08:02:45.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/34' 2023-06-22T08:02:45.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/38' 2023-06-22T08:02:45.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/11' 2023-06-22T08:02:45.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/16' 2023-06-22T08:02:45.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/32' 2023-06-22T08:02:45.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/30' 2023-06-22T08:02:45.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/14' 2023-06-22T08:02:45.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/18' 2023-06-22T08:02:45.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/13' 2023-06-22T08:02:45.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/33' 2023-06-22T08:02:45.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/17' 2023-06-22T08:02:45.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/35' 2023-06-22T08:02:45.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './129/37' 2023-06-22T08:02:45.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './129' 2023-06-22T08:02:45.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/19' 2023-06-22T08:02:45.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/15' 2023-06-22T08:02:45.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/31' 2023-06-22T08:02:45.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/36' 2023-06-22T08:02:45.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/20' 2023-06-22T08:02:45.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/12' 2023-06-22T08:02:45.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/39' 2023-06-22T08:02:45.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/10' 2023-06-22T08:02:45.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/34' 2023-06-22T08:02:45.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/38' 2023-06-22T08:02:45.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/11' 2023-06-22T08:02:45.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/16' 2023-06-22T08:02:45.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/32' 2023-06-22T08:02:45.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/30' 2023-06-22T08:02:45.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/14' 2023-06-22T08:02:45.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/18' 2023-06-22T08:02:45.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/13' 2023-06-22T08:02:45.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/33' 2023-06-22T08:02:45.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/17' 2023-06-22T08:02:45.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/35' 2023-06-22T08:02:45.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './130/37' 2023-06-22T08:02:45.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './130' 2023-06-22T08:02:45.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/19' 2023-06-22T08:02:45.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/15' 2023-06-22T08:02:45.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/31' 2023-06-22T08:02:45.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/36' 2023-06-22T08:02:45.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/20' 2023-06-22T08:02:45.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/12' 2023-06-22T08:02:45.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/39' 2023-06-22T08:02:45.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/10' 2023-06-22T08:02:45.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/34' 2023-06-22T08:02:45.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/38' 2023-06-22T08:02:45.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/11' 2023-06-22T08:02:45.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/16' 2023-06-22T08:02:45.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/32' 2023-06-22T08:02:45.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/30' 2023-06-22T08:02:45.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/14' 2023-06-22T08:02:45.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/18' 2023-06-22T08:02:45.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/13' 2023-06-22T08:02:45.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/33' 2023-06-22T08:02:45.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/17' 2023-06-22T08:02:45.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/35' 2023-06-22T08:02:45.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './131/37' 2023-06-22T08:02:45.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './131' 2023-06-22T08:02:45.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/19' 2023-06-22T08:02:45.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/15' 2023-06-22T08:02:45.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/31' 2023-06-22T08:02:45.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/36' 2023-06-22T08:02:45.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/20' 2023-06-22T08:02:45.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/12' 2023-06-22T08:02:45.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/39' 2023-06-22T08:02:45.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/10' 2023-06-22T08:02:45.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/34' 2023-06-22T08:02:45.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/38' 2023-06-22T08:02:45.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/11' 2023-06-22T08:02:45.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/16' 2023-06-22T08:02:45.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/32' 2023-06-22T08:02:45.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/30' 2023-06-22T08:02:45.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/14' 2023-06-22T08:02:45.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/18' 2023-06-22T08:02:45.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/13' 2023-06-22T08:02:45.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/33' 2023-06-22T08:02:45.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/17' 2023-06-22T08:02:45.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/35' 2023-06-22T08:02:45.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './132/37' 2023-06-22T08:02:45.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './132' 2023-06-22T08:02:45.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/19' 2023-06-22T08:02:45.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/15' 2023-06-22T08:02:45.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/31' 2023-06-22T08:02:45.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/36' 2023-06-22T08:02:45.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/20' 2023-06-22T08:02:45.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/12' 2023-06-22T08:02:45.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/39' 2023-06-22T08:02:45.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/10' 2023-06-22T08:02:45.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/34' 2023-06-22T08:02:45.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/38' 2023-06-22T08:02:45.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/11' 2023-06-22T08:02:45.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/16' 2023-06-22T08:02:45.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/32' 2023-06-22T08:02:45.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/30' 2023-06-22T08:02:45.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/14' 2023-06-22T08:02:45.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/18' 2023-06-22T08:02:45.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/13' 2023-06-22T08:02:45.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/33' 2023-06-22T08:02:45.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/17' 2023-06-22T08:02:45.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/35' 2023-06-22T08:02:45.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './133/37' 2023-06-22T08:02:45.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './133' 2023-06-22T08:02:45.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/19' 2023-06-22T08:02:45.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/15' 2023-06-22T08:02:45.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/31' 2023-06-22T08:02:45.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/36' 2023-06-22T08:02:45.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/20' 2023-06-22T08:02:45.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/12' 2023-06-22T08:02:45.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/39' 2023-06-22T08:02:45.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/10' 2023-06-22T08:02:45.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/34' 2023-06-22T08:02:45.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/38' 2023-06-22T08:02:45.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/11' 2023-06-22T08:02:45.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/16' 2023-06-22T08:02:45.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/32' 2023-06-22T08:02:45.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/30' 2023-06-22T08:02:45.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/14' 2023-06-22T08:02:45.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/18' 2023-06-22T08:02:45.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/13' 2023-06-22T08:02:45.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/33' 2023-06-22T08:02:45.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/17' 2023-06-22T08:02:45.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/35' 2023-06-22T08:02:45.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './134/37' 2023-06-22T08:02:45.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './134' 2023-06-22T08:02:45.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/19' 2023-06-22T08:02:45.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/15' 2023-06-22T08:02:45.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/31' 2023-06-22T08:02:45.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/36' 2023-06-22T08:02:45.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/20' 2023-06-22T08:02:45.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/12' 2023-06-22T08:02:45.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/39' 2023-06-22T08:02:45.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/10' 2023-06-22T08:02:45.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/34' 2023-06-22T08:02:45.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/38' 2023-06-22T08:02:45.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/11' 2023-06-22T08:02:45.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/16' 2023-06-22T08:02:45.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/32' 2023-06-22T08:02:45.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/30' 2023-06-22T08:02:45.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/14' 2023-06-22T08:02:45.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/18' 2023-06-22T08:02:45.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/13' 2023-06-22T08:02:45.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/33' 2023-06-22T08:02:45.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/17' 2023-06-22T08:02:45.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/35' 2023-06-22T08:02:45.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './135/37' 2023-06-22T08:02:45.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './135' 2023-06-22T08:02:45.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/19' 2023-06-22T08:02:45.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/15' 2023-06-22T08:02:45.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/31' 2023-06-22T08:02:45.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/36' 2023-06-22T08:02:45.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/20' 2023-06-22T08:02:45.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/12' 2023-06-22T08:02:45.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/39' 2023-06-22T08:02:45.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/10' 2023-06-22T08:02:45.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/34' 2023-06-22T08:02:45.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/38' 2023-06-22T08:02:45.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/11' 2023-06-22T08:02:45.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/16' 2023-06-22T08:02:45.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/32' 2023-06-22T08:02:45.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/30' 2023-06-22T08:02:45.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/14' 2023-06-22T08:02:45.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/18' 2023-06-22T08:02:45.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/13' 2023-06-22T08:02:45.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/33' 2023-06-22T08:02:45.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/17' 2023-06-22T08:02:45.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/35' 2023-06-22T08:02:45.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './136/37' 2023-06-22T08:02:45.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './136' 2023-06-22T08:02:45.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/19' 2023-06-22T08:02:45.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/15' 2023-06-22T08:02:45.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/31' 2023-06-22T08:02:45.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/36' 2023-06-22T08:02:45.579 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/20' 2023-06-22T08:02:45.579 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/12' 2023-06-22T08:02:45.579 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/39' 2023-06-22T08:02:45.579 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/10' 2023-06-22T08:02:45.579 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/34' 2023-06-22T08:02:45.579 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/38' 2023-06-22T08:02:45.579 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/11' 2023-06-22T08:02:45.580 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/16' 2023-06-22T08:02:45.580 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/32' 2023-06-22T08:02:45.580 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/30' 2023-06-22T08:02:45.580 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/14' 2023-06-22T08:02:45.580 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/18' 2023-06-22T08:02:45.580 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/13' 2023-06-22T08:02:45.581 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/33' 2023-06-22T08:02:45.581 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/17' 2023-06-22T08:02:45.581 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/35' 2023-06-22T08:02:45.581 INFO:tasks.workunit.client.0.smithi023.stdout:removed './137/37' 2023-06-22T08:02:45.581 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './137' 2023-06-22T08:02:45.581 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/19' 2023-06-22T08:02:45.581 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/15' 2023-06-22T08:02:45.581 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/31' 2023-06-22T08:02:45.582 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/36' 2023-06-22T08:02:45.582 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/20' 2023-06-22T08:02:45.582 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/12' 2023-06-22T08:02:45.582 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/39' 2023-06-22T08:02:45.582 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/10' 2023-06-22T08:02:45.582 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/34' 2023-06-22T08:02:45.582 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/38' 2023-06-22T08:02:45.582 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/11' 2023-06-22T08:02:45.583 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/16' 2023-06-22T08:02:47.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/32' 2023-06-22T08:02:47.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/30' 2023-06-22T08:02:47.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/14' 2023-06-22T08:02:47.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/18' 2023-06-22T08:02:47.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/13' 2023-06-22T08:02:47.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/33' 2023-06-22T08:02:47.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/17' 2023-06-22T08:02:47.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/35' 2023-06-22T08:02:47.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './138/37' 2023-06-22T08:02:47.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './138' 2023-06-22T08:02:47.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/19' 2023-06-22T08:02:47.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/15' 2023-06-22T08:02:47.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/31' 2023-06-22T08:02:47.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/36' 2023-06-22T08:02:47.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/20' 2023-06-22T08:02:47.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/12' 2023-06-22T08:02:47.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/39' 2023-06-22T08:02:47.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/10' 2023-06-22T08:02:47.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/34' 2023-06-22T08:02:47.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/38' 2023-06-22T08:02:47.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/11' 2023-06-22T08:02:47.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/16' 2023-06-22T08:02:47.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/32' 2023-06-22T08:02:47.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/30' 2023-06-22T08:02:47.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/14' 2023-06-22T08:02:47.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/18' 2023-06-22T08:02:47.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/13' 2023-06-22T08:02:47.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/33' 2023-06-22T08:02:47.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/17' 2023-06-22T08:02:47.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/35' 2023-06-22T08:02:47.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './139/37' 2023-06-22T08:02:47.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './139' 2023-06-22T08:02:47.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/19' 2023-06-22T08:02:47.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/15' 2023-06-22T08:02:47.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/31' 2023-06-22T08:02:47.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/36' 2023-06-22T08:02:47.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/20' 2023-06-22T08:02:47.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/12' 2023-06-22T08:02:47.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/39' 2023-06-22T08:02:47.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/10' 2023-06-22T08:02:47.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/34' 2023-06-22T08:02:47.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/38' 2023-06-22T08:02:47.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/11' 2023-06-22T08:02:47.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/16' 2023-06-22T08:02:47.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/32' 2023-06-22T08:02:47.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/30' 2023-06-22T08:02:47.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/14' 2023-06-22T08:02:47.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/18' 2023-06-22T08:02:47.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/13' 2023-06-22T08:02:47.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/33' 2023-06-22T08:02:47.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/17' 2023-06-22T08:02:47.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/35' 2023-06-22T08:02:47.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './140/37' 2023-06-22T08:02:47.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './140' 2023-06-22T08:02:47.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/19' 2023-06-22T08:02:47.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/15' 2023-06-22T08:02:47.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/31' 2023-06-22T08:02:47.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/36' 2023-06-22T08:02:47.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/20' 2023-06-22T08:02:47.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/12' 2023-06-22T08:02:47.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/39' 2023-06-22T08:02:47.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/10' 2023-06-22T08:02:47.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/34' 2023-06-22T08:02:47.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/38' 2023-06-22T08:02:47.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/11' 2023-06-22T08:02:47.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/16' 2023-06-22T08:02:47.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/32' 2023-06-22T08:02:47.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/30' 2023-06-22T08:02:47.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/14' 2023-06-22T08:02:47.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/18' 2023-06-22T08:02:47.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/13' 2023-06-22T08:02:47.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/33' 2023-06-22T08:02:47.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/17' 2023-06-22T08:02:47.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/35' 2023-06-22T08:02:47.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './141/37' 2023-06-22T08:02:47.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './141' 2023-06-22T08:02:47.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/19' 2023-06-22T08:02:47.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/15' 2023-06-22T08:02:47.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/31' 2023-06-22T08:02:47.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/36' 2023-06-22T08:02:47.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/20' 2023-06-22T08:02:47.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/12' 2023-06-22T08:02:47.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/39' 2023-06-22T08:02:47.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/10' 2023-06-22T08:02:47.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/34' 2023-06-22T08:02:47.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/38' 2023-06-22T08:02:47.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/11' 2023-06-22T08:02:47.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/16' 2023-06-22T08:02:47.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/32' 2023-06-22T08:02:47.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/30' 2023-06-22T08:02:47.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/14' 2023-06-22T08:02:47.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/18' 2023-06-22T08:02:47.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/13' 2023-06-22T08:02:47.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/33' 2023-06-22T08:02:47.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/17' 2023-06-22T08:02:47.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/35' 2023-06-22T08:02:47.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './142/37' 2023-06-22T08:02:47.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './142' 2023-06-22T08:02:47.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/19' 2023-06-22T08:02:47.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/15' 2023-06-22T08:02:47.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/31' 2023-06-22T08:02:47.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/36' 2023-06-22T08:02:47.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/20' 2023-06-22T08:02:47.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/12' 2023-06-22T08:02:47.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/39' 2023-06-22T08:02:47.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/10' 2023-06-22T08:02:47.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/34' 2023-06-22T08:02:47.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/38' 2023-06-22T08:02:47.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/11' 2023-06-22T08:02:47.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/16' 2023-06-22T08:02:47.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/32' 2023-06-22T08:02:47.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/30' 2023-06-22T08:02:47.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/14' 2023-06-22T08:02:47.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/18' 2023-06-22T08:02:47.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/13' 2023-06-22T08:02:47.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/33' 2023-06-22T08:02:47.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/17' 2023-06-22T08:02:47.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/35' 2023-06-22T08:02:47.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed './143/37' 2023-06-22T08:02:47.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './143' 2023-06-22T08:02:47.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/19' 2023-06-22T08:02:47.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/15' 2023-06-22T08:02:47.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/31' 2023-06-22T08:02:47.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/36' 2023-06-22T08:02:47.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/20' 2023-06-22T08:02:47.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/12' 2023-06-22T08:02:47.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/39' 2023-06-22T08:02:47.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/10' 2023-06-22T08:02:47.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/34' 2023-06-22T08:02:47.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/38' 2023-06-22T08:02:47.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/11' 2023-06-22T08:02:47.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/16' 2023-06-22T08:02:47.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/32' 2023-06-22T08:02:47.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/30' 2023-06-22T08:02:47.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/14' 2023-06-22T08:02:47.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/18' 2023-06-22T08:02:47.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/13' 2023-06-22T08:02:47.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/33' 2023-06-22T08:02:47.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/17' 2023-06-22T08:02:47.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/35' 2023-06-22T08:02:47.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed './144/37' 2023-06-22T08:02:47.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './144' 2023-06-22T08:02:47.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/19' 2023-06-22T08:02:47.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/15' 2023-06-22T08:02:47.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/31' 2023-06-22T08:02:47.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/36' 2023-06-22T08:02:47.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/20' 2023-06-22T08:02:47.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/12' 2023-06-22T08:02:47.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/39' 2023-06-22T08:02:47.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/10' 2023-06-22T08:02:47.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/34' 2023-06-22T08:02:47.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/38' 2023-06-22T08:02:47.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/11' 2023-06-22T08:02:47.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/16' 2023-06-22T08:02:47.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/32' 2023-06-22T08:02:47.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/30' 2023-06-22T08:02:47.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/14' 2023-06-22T08:02:47.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/18' 2023-06-22T08:02:47.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/13' 2023-06-22T08:02:47.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/33' 2023-06-22T08:02:47.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/17' 2023-06-22T08:02:47.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/35' 2023-06-22T08:02:47.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './145/37' 2023-06-22T08:02:47.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './145' 2023-06-22T08:02:47.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/19' 2023-06-22T08:02:47.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/15' 2023-06-22T08:02:47.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/31' 2023-06-22T08:02:47.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/36' 2023-06-22T08:02:47.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/20' 2023-06-22T08:02:47.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/12' 2023-06-22T08:02:47.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/39' 2023-06-22T08:02:47.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/10' 2023-06-22T08:02:47.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/34' 2023-06-22T08:02:47.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/38' 2023-06-22T08:02:47.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/11' 2023-06-22T08:02:47.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/16' 2023-06-22T08:02:47.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/32' 2023-06-22T08:02:47.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/30' 2023-06-22T08:02:47.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/14' 2023-06-22T08:02:47.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/18' 2023-06-22T08:02:47.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/13' 2023-06-22T08:02:47.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/33' 2023-06-22T08:02:47.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/17' 2023-06-22T08:02:47.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/35' 2023-06-22T08:02:47.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './146/37' 2023-06-22T08:02:47.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './146' 2023-06-22T08:02:47.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/19' 2023-06-22T08:02:47.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/15' 2023-06-22T08:02:47.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/31' 2023-06-22T08:02:47.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/36' 2023-06-22T08:02:47.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/20' 2023-06-22T08:02:47.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/12' 2023-06-22T08:02:47.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/39' 2023-06-22T08:02:47.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/10' 2023-06-22T08:02:47.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/34' 2023-06-22T08:02:47.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/38' 2023-06-22T08:02:47.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/11' 2023-06-22T08:02:47.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/16' 2023-06-22T08:02:47.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/32' 2023-06-22T08:02:47.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/30' 2023-06-22T08:02:47.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/14' 2023-06-22T08:02:47.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/18' 2023-06-22T08:02:47.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/13' 2023-06-22T08:02:47.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/33' 2023-06-22T08:02:47.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/17' 2023-06-22T08:02:47.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/35' 2023-06-22T08:02:47.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './147/37' 2023-06-22T08:02:47.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './147' 2023-06-22T08:02:47.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/19' 2023-06-22T08:02:47.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/15' 2023-06-22T08:02:47.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/31' 2023-06-22T08:02:47.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/36' 2023-06-22T08:02:49.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/20' 2023-06-22T08:02:49.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/12' 2023-06-22T08:02:49.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/39' 2023-06-22T08:02:49.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/10' 2023-06-22T08:02:49.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/34' 2023-06-22T08:02:49.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/38' 2023-06-22T08:02:49.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/11' 2023-06-22T08:02:49.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/16' 2023-06-22T08:02:49.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/32' 2023-06-22T08:02:49.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/30' 2023-06-22T08:02:49.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/14' 2023-06-22T08:02:49.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/18' 2023-06-22T08:02:49.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/13' 2023-06-22T08:02:49.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/33' 2023-06-22T08:02:49.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/17' 2023-06-22T08:02:49.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/35' 2023-06-22T08:02:49.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed './148/37' 2023-06-22T08:02:49.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './148' 2023-06-22T08:02:49.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/19' 2023-06-22T08:02:49.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/15' 2023-06-22T08:02:49.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/31' 2023-06-22T08:02:49.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/36' 2023-06-22T08:02:49.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/20' 2023-06-22T08:02:49.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/12' 2023-06-22T08:02:49.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/39' 2023-06-22T08:02:49.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/10' 2023-06-22T08:02:49.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/34' 2023-06-22T08:02:49.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/38' 2023-06-22T08:02:49.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/11' 2023-06-22T08:02:49.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/16' 2023-06-22T08:02:49.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/32' 2023-06-22T08:02:49.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/30' 2023-06-22T08:02:49.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/14' 2023-06-22T08:02:49.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/18' 2023-06-22T08:02:49.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/13' 2023-06-22T08:02:49.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/33' 2023-06-22T08:02:49.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/17' 2023-06-22T08:02:49.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/35' 2023-06-22T08:02:49.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed './149/37' 2023-06-22T08:02:49.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './149' 2023-06-22T08:02:49.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/19' 2023-06-22T08:02:49.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/15' 2023-06-22T08:02:49.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/31' 2023-06-22T08:02:49.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/36' 2023-06-22T08:02:49.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/20' 2023-06-22T08:02:49.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/12' 2023-06-22T08:02:49.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/39' 2023-06-22T08:02:49.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/10' 2023-06-22T08:02:49.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/34' 2023-06-22T08:02:49.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/38' 2023-06-22T08:02:49.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/11' 2023-06-22T08:02:49.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/16' 2023-06-22T08:02:49.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/32' 2023-06-22T08:02:49.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/30' 2023-06-22T08:02:49.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/14' 2023-06-22T08:02:49.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/18' 2023-06-22T08:02:49.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/13' 2023-06-22T08:02:49.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/33' 2023-06-22T08:02:49.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/17' 2023-06-22T08:02:49.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/35' 2023-06-22T08:02:49.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './150/37' 2023-06-22T08:02:49.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './150' 2023-06-22T08:02:49.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/19' 2023-06-22T08:02:49.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/15' 2023-06-22T08:02:49.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/31' 2023-06-22T08:02:49.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/36' 2023-06-22T08:02:49.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/20' 2023-06-22T08:02:49.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/12' 2023-06-22T08:02:49.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/39' 2023-06-22T08:02:49.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/10' 2023-06-22T08:02:49.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/34' 2023-06-22T08:02:49.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/38' 2023-06-22T08:02:49.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/11' 2023-06-22T08:02:49.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/16' 2023-06-22T08:02:49.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/32' 2023-06-22T08:02:49.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/30' 2023-06-22T08:02:49.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/14' 2023-06-22T08:02:49.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/18' 2023-06-22T08:02:49.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/13' 2023-06-22T08:02:49.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/33' 2023-06-22T08:02:49.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/17' 2023-06-22T08:02:49.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/35' 2023-06-22T08:02:49.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed './151/37' 2023-06-22T08:02:49.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './151' 2023-06-22T08:02:49.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/19' 2023-06-22T08:02:49.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/15' 2023-06-22T08:02:49.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/31' 2023-06-22T08:02:49.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/36' 2023-06-22T08:02:49.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/20' 2023-06-22T08:02:49.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/12' 2023-06-22T08:02:49.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/39' 2023-06-22T08:02:49.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/10' 2023-06-22T08:02:49.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/34' 2023-06-22T08:02:49.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/38' 2023-06-22T08:02:49.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/11' 2023-06-22T08:02:49.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/16' 2023-06-22T08:02:49.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/32' 2023-06-22T08:02:49.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/30' 2023-06-22T08:02:49.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/14' 2023-06-22T08:02:49.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/18' 2023-06-22T08:02:49.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/13' 2023-06-22T08:02:49.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/33' 2023-06-22T08:02:49.244 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/17' 2023-06-22T08:02:49.244 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/35' 2023-06-22T08:02:49.244 INFO:tasks.workunit.client.0.smithi023.stdout:removed './152/37' 2023-06-22T08:02:49.244 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './152' 2023-06-22T08:02:49.244 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/19' 2023-06-22T08:02:49.244 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/15' 2023-06-22T08:02:49.245 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/31' 2023-06-22T08:02:49.245 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/36' 2023-06-22T08:02:49.245 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/20' 2023-06-22T08:02:49.245 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/12' 2023-06-22T08:02:49.245 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/39' 2023-06-22T08:02:49.245 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/10' 2023-06-22T08:02:49.246 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/34' 2023-06-22T08:02:49.246 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/38' 2023-06-22T08:02:49.246 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/11' 2023-06-22T08:02:49.246 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/16' 2023-06-22T08:02:49.247 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/32' 2023-06-22T08:02:49.247 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/30' 2023-06-22T08:02:49.247 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/14' 2023-06-22T08:02:49.247 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/18' 2023-06-22T08:02:49.247 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/13' 2023-06-22T08:02:49.247 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/33' 2023-06-22T08:02:49.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/17' 2023-06-22T08:02:49.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/35' 2023-06-22T08:02:49.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed './153/37' 2023-06-22T08:02:49.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './153' 2023-06-22T08:02:49.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/19' 2023-06-22T08:02:49.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/15' 2023-06-22T08:02:49.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/31' 2023-06-22T08:02:49.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/36' 2023-06-22T08:02:49.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/20' 2023-06-22T08:02:49.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/12' 2023-06-22T08:02:49.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/39' 2023-06-22T08:02:49.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/10' 2023-06-22T08:02:49.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/34' 2023-06-22T08:02:49.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/38' 2023-06-22T08:02:49.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/11' 2023-06-22T08:02:49.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/16' 2023-06-22T08:02:49.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/32' 2023-06-22T08:02:49.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/30' 2023-06-22T08:02:49.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/14' 2023-06-22T08:02:49.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/18' 2023-06-22T08:02:49.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/13' 2023-06-22T08:02:49.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/33' 2023-06-22T08:02:49.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/17' 2023-06-22T08:02:49.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/35' 2023-06-22T08:02:49.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './154/37' 2023-06-22T08:02:49.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './154' 2023-06-22T08:02:49.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/19' 2023-06-22T08:02:49.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/15' 2023-06-22T08:02:49.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/31' 2023-06-22T08:02:49.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/36' 2023-06-22T08:02:49.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/20' 2023-06-22T08:02:49.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/12' 2023-06-22T08:02:49.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/39' 2023-06-22T08:02:49.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/10' 2023-06-22T08:02:49.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/34' 2023-06-22T08:02:49.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/38' 2023-06-22T08:02:49.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/11' 2023-06-22T08:02:49.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/16' 2023-06-22T08:02:49.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/32' 2023-06-22T08:02:49.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/30' 2023-06-22T08:02:49.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/14' 2023-06-22T08:02:49.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/18' 2023-06-22T08:02:49.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/13' 2023-06-22T08:02:49.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/33' 2023-06-22T08:02:49.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/17' 2023-06-22T08:02:49.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/35' 2023-06-22T08:02:49.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './155/37' 2023-06-22T08:02:49.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './155' 2023-06-22T08:02:49.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/19' 2023-06-22T08:02:49.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/15' 2023-06-22T08:02:49.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/31' 2023-06-22T08:02:49.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/36' 2023-06-22T08:02:49.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/20' 2023-06-22T08:02:49.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/12' 2023-06-22T08:02:49.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/39' 2023-06-22T08:02:49.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/10' 2023-06-22T08:02:49.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/34' 2023-06-22T08:02:49.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/38' 2023-06-22T08:02:49.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/11' 2023-06-22T08:02:49.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/16' 2023-06-22T08:02:49.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/32' 2023-06-22T08:02:49.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/30' 2023-06-22T08:02:49.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/14' 2023-06-22T08:02:49.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/18' 2023-06-22T08:02:49.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/13' 2023-06-22T08:02:49.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/33' 2023-06-22T08:02:49.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/17' 2023-06-22T08:02:49.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/35' 2023-06-22T08:02:49.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './156/37' 2023-06-22T08:02:49.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './156' 2023-06-22T08:02:49.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/19' 2023-06-22T08:02:49.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/15' 2023-06-22T08:02:49.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/31' 2023-06-22T08:02:49.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/36' 2023-06-22T08:02:49.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/20' 2023-06-22T08:02:49.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/12' 2023-06-22T08:02:49.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/39' 2023-06-22T08:02:49.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/10' 2023-06-22T08:02:49.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/34' 2023-06-22T08:02:49.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/38' 2023-06-22T08:02:49.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/11' 2023-06-22T08:02:49.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/16' 2023-06-22T08:02:49.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/32' 2023-06-22T08:02:49.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/30' 2023-06-22T08:02:49.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/14' 2023-06-22T08:02:49.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/18' 2023-06-22T08:02:49.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/13' 2023-06-22T08:02:49.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/33' 2023-06-22T08:02:50.091 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:02:50.339 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/17' 2023-06-22T08:02:50.339 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/35' 2023-06-22T08:02:50.340 INFO:tasks.workunit.client.0.smithi023.stdout:removed './157/37' 2023-06-22T08:02:50.340 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './157' 2023-06-22T08:02:50.340 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/19' 2023-06-22T08:02:50.340 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/15' 2023-06-22T08:02:50.340 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/31' 2023-06-22T08:02:50.340 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/36' 2023-06-22T08:02:50.341 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/20' 2023-06-22T08:02:50.341 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/12' 2023-06-22T08:02:50.341 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/39' 2023-06-22T08:02:50.341 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/10' 2023-06-22T08:02:50.341 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/34' 2023-06-22T08:02:50.341 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/38' 2023-06-22T08:02:50.341 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/11' 2023-06-22T08:02:50.342 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/16' 2023-06-22T08:02:50.342 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/32' 2023-06-22T08:02:50.342 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/30' 2023-06-22T08:02:50.342 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/14' 2023-06-22T08:02:50.342 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/18' 2023-06-22T08:02:50.342 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/13' 2023-06-22T08:02:50.342 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/33' 2023-06-22T08:02:50.342 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/17' 2023-06-22T08:02:50.343 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/35' 2023-06-22T08:02:50.343 INFO:tasks.workunit.client.0.smithi023.stdout:removed './158/37' 2023-06-22T08:02:50.343 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './158' 2023-06-22T08:02:50.343 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/19' 2023-06-22T08:02:50.343 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/15' 2023-06-22T08:02:50.343 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/31' 2023-06-22T08:02:50.343 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/36' 2023-06-22T08:02:50.343 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/20' 2023-06-22T08:02:50.344 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/12' 2023-06-22T08:02:50.344 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/39' 2023-06-22T08:02:50.344 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/10' 2023-06-22T08:02:50.344 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/34' 2023-06-22T08:02:50.344 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/38' 2023-06-22T08:02:50.344 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/11' 2023-06-22T08:02:50.344 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/16' 2023-06-22T08:02:50.344 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/32' 2023-06-22T08:02:50.345 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/30' 2023-06-22T08:02:50.345 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/14' 2023-06-22T08:02:50.345 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/18' 2023-06-22T08:02:50.345 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/13' 2023-06-22T08:02:50.345 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/33' 2023-06-22T08:02:50.345 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/17' 2023-06-22T08:02:50.346 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/35' 2023-06-22T08:02:50.346 INFO:tasks.workunit.client.0.smithi023.stdout:removed './159/37' 2023-06-22T08:02:50.346 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './159' 2023-06-22T08:02:50.346 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/19' 2023-06-22T08:02:50.346 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/15' 2023-06-22T08:02:50.346 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/31' 2023-06-22T08:02:50.346 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/36' 2023-06-22T08:02:50.346 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/20' 2023-06-22T08:02:50.347 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/12' 2023-06-22T08:02:50.347 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/39' 2023-06-22T08:02:50.347 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/10' 2023-06-22T08:02:50.347 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/34' 2023-06-22T08:02:50.347 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/38' 2023-06-22T08:02:50.347 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/11' 2023-06-22T08:02:50.347 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/16' 2023-06-22T08:02:50.347 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/32' 2023-06-22T08:02:50.348 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/30' 2023-06-22T08:02:50.348 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/14' 2023-06-22T08:02:50.348 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/18' 2023-06-22T08:02:50.348 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/13' 2023-06-22T08:02:50.348 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/33' 2023-06-22T08:02:50.348 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/17' 2023-06-22T08:02:50.348 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/35' 2023-06-22T08:02:50.349 INFO:tasks.workunit.client.0.smithi023.stdout:removed './160/37' 2023-06-22T08:02:50.349 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './160' 2023-06-22T08:02:50.349 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/19' 2023-06-22T08:02:50.349 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/15' 2023-06-22T08:02:50.349 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/31' 2023-06-22T08:02:50.349 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/36' 2023-06-22T08:02:50.349 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/20' 2023-06-22T08:02:50.349 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/12' 2023-06-22T08:02:50.350 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/39' 2023-06-22T08:02:50.350 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/10' 2023-06-22T08:02:50.350 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/34' 2023-06-22T08:02:50.350 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/38' 2023-06-22T08:02:50.350 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/11' 2023-06-22T08:02:50.350 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/16' 2023-06-22T08:02:50.350 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/32' 2023-06-22T08:02:50.350 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/30' 2023-06-22T08:02:50.350 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/14' 2023-06-22T08:02:50.351 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/18' 2023-06-22T08:02:50.351 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/13' 2023-06-22T08:02:50.351 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/33' 2023-06-22T08:02:50.351 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/17' 2023-06-22T08:02:50.351 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/35' 2023-06-22T08:02:50.351 INFO:tasks.workunit.client.0.smithi023.stdout:removed './161/37' 2023-06-22T08:02:50.351 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './161' 2023-06-22T08:02:50.351 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/19' 2023-06-22T08:02:50.352 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/15' 2023-06-22T08:02:50.352 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/31' 2023-06-22T08:02:50.352 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/36' 2023-06-22T08:02:50.352 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/20' 2023-06-22T08:02:50.352 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/12' 2023-06-22T08:02:50.352 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/39' 2023-06-22T08:02:50.352 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/10' 2023-06-22T08:02:50.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/34' 2023-06-22T08:02:50.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/38' 2023-06-22T08:02:50.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/11' 2023-06-22T08:02:50.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/16' 2023-06-22T08:02:50.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/32' 2023-06-22T08:02:50.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/30' 2023-06-22T08:02:50.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/14' 2023-06-22T08:02:50.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/18' 2023-06-22T08:02:50.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/13' 2023-06-22T08:02:50.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/33' 2023-06-22T08:02:50.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/17' 2023-06-22T08:02:50.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/35' 2023-06-22T08:02:50.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './162/37' 2023-06-22T08:02:50.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './162' 2023-06-22T08:02:50.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/19' 2023-06-22T08:02:50.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/15' 2023-06-22T08:02:50.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/31' 2023-06-22T08:02:50.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/36' 2023-06-22T08:02:50.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/20' 2023-06-22T08:02:50.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/12' 2023-06-22T08:02:50.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/39' 2023-06-22T08:02:50.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/10' 2023-06-22T08:02:50.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/34' 2023-06-22T08:02:50.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/38' 2023-06-22T08:02:50.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/11' 2023-06-22T08:02:50.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/16' 2023-06-22T08:02:50.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/32' 2023-06-22T08:02:50.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/30' 2023-06-22T08:02:50.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/14' 2023-06-22T08:02:50.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/18' 2023-06-22T08:02:50.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/13' 2023-06-22T08:02:50.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/33' 2023-06-22T08:02:50.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/17' 2023-06-22T08:02:50.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/35' 2023-06-22T08:02:50.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './163/37' 2023-06-22T08:02:50.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './163' 2023-06-22T08:02:50.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/19' 2023-06-22T08:02:50.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/15' 2023-06-22T08:02:50.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/31' 2023-06-22T08:02:50.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/36' 2023-06-22T08:02:50.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/20' 2023-06-22T08:02:50.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/12' 2023-06-22T08:02:50.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/39' 2023-06-22T08:02:50.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/10' 2023-06-22T08:02:50.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/34' 2023-06-22T08:02:50.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/38' 2023-06-22T08:02:50.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/11' 2023-06-22T08:02:50.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/16' 2023-06-22T08:02:50.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/32' 2023-06-22T08:02:50.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/30' 2023-06-22T08:02:50.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/14' 2023-06-22T08:02:50.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/18' 2023-06-22T08:02:50.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/13' 2023-06-22T08:02:50.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/33' 2023-06-22T08:02:50.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/17' 2023-06-22T08:02:50.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/35' 2023-06-22T08:02:50.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './164/37' 2023-06-22T08:02:50.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './164' 2023-06-22T08:02:50.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/19' 2023-06-22T08:02:50.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/15' 2023-06-22T08:02:50.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/31' 2023-06-22T08:02:50.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/36' 2023-06-22T08:02:50.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/20' 2023-06-22T08:02:50.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/12' 2023-06-22T08:02:50.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/39' 2023-06-22T08:02:50.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/10' 2023-06-22T08:02:50.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/34' 2023-06-22T08:02:50.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/38' 2023-06-22T08:02:50.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/11' 2023-06-22T08:02:50.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/16' 2023-06-22T08:02:50.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/32' 2023-06-22T08:02:50.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/30' 2023-06-22T08:02:50.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/14' 2023-06-22T08:02:50.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/18' 2023-06-22T08:02:50.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/13' 2023-06-22T08:02:50.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/33' 2023-06-22T08:02:50.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/17' 2023-06-22T08:02:50.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/35' 2023-06-22T08:02:50.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './165/37' 2023-06-22T08:02:50.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './165' 2023-06-22T08:02:50.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/19' 2023-06-22T08:02:50.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/15' 2023-06-22T08:02:50.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/31' 2023-06-22T08:02:50.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/36' 2023-06-22T08:02:50.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/20' 2023-06-22T08:02:50.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/12' 2023-06-22T08:02:50.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/39' 2023-06-22T08:02:50.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/10' 2023-06-22T08:02:50.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/34' 2023-06-22T08:02:50.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/38' 2023-06-22T08:02:50.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/11' 2023-06-22T08:02:50.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/16' 2023-06-22T08:02:50.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/32' 2023-06-22T08:02:50.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/30' 2023-06-22T08:02:50.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/14' 2023-06-22T08:02:50.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/18' 2023-06-22T08:02:50.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/13' 2023-06-22T08:02:50.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/33' 2023-06-22T08:02:50.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/17' 2023-06-22T08:02:50.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/35' 2023-06-22T08:02:50.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './166/37' 2023-06-22T08:02:50.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './166' 2023-06-22T08:02:50.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/19' 2023-06-22T08:02:50.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/15' 2023-06-22T08:02:50.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/31' 2023-06-22T08:02:50.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/36' 2023-06-22T08:02:50.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/20' 2023-06-22T08:02:50.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/12' 2023-06-22T08:02:50.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/39' 2023-06-22T08:02:50.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/10' 2023-06-22T08:02:50.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/34' 2023-06-22T08:02:50.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/38' 2023-06-22T08:02:50.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.455+0000 7f3daedff700 1 Processor -- start 2023-06-22T08:02:50.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.456+0000 7f3daedff700 1 -- start start 2023-06-22T08:02:50.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.457+0000 7f3daedff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3da8152dd0 0x7f3da80aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:50.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.457+0000 7f3daedff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3da80c8bd0 0x7f3da806afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:50.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.457+0000 7f3daedff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da80c7400 con 0x7f3da806b500 2023-06-22T08:02:50.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.457+0000 7f3daedff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da80c81c0 con 0x7f3da80c8bd0 2023-06-22T08:02:50.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.457+0000 7f3daedff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da80c84a0 con 0x7f3da8152dd0 2023-06-22T08:02:50.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.457+0000 7f3dacb9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3da8152dd0 0x7f3da80aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:50.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.457+0000 7f3da7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3da80c8bd0 0x7f3da806afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:50.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.457+0000 7f3dacb9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3da8152dd0 0x7f3da80aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:60766/0 (socket says 172.21.15.23:60766) 2023-06-22T08:02:50.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.457+0000 7f3dacb9b700 1 -- 172.21.15.23:0/3661552833 learned_addr learned my addr 172.21.15.23:0/3661552833 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:02:50.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.457+0000 7f3dad39c700 1 --1- 172.21.15.23:0/3661552833 >> v1:172.21.15.67:6789/0 conn(0x7f3da806b500 0x7f3da80cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:56508/0 (socket says 172.21.15.23:56508) 2023-06-22T08:02:50.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.458+0000 7f3dacb9b700 1 -- 172.21.15.23:0/3661552833 >> v2:172.21.15.23:3300/0 conn(0x7f3da80c8bd0 msgr2=0x7f3da806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:02:50.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.458+0000 7f3dacb9b700 1 --2- 172.21.15.23:0/3661552833 >> v2:172.21.15.23:3300/0 conn(0x7f3da80c8bd0 0x7f3da806afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.458+0000 7f3dacb9b700 1 -- 172.21.15.23:0/3661552833 >> v1:172.21.15.67:6789/0 conn(0x7f3da806b500 legacy=0x7f3da80cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:50.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.458+0000 7f3dacb9b700 1 -- 172.21.15.23:0/3661552833 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da80cd370 con 0x7f3da8152dd0 2023-06-22T08:02:50.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.458+0000 7f3da7fff700 1 --2- 172.21.15.23:0/3661552833 >> v2:172.21.15.23:3300/0 conn(0x7f3da80c8bd0 0x7f3da806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:02:50.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.458+0000 7f3da77fe700 1 -- 172.21.15.23:0/3661552833 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046282099 0 0) 0x7f3da80c7400 con 0x7f3da806b500 2023-06-22T08:02:50.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.458+0000 7f3dacb9b700 1 --2- 172.21.15.23:0/3661552833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da8152dd0 0x7f3da80aedc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3da8070880 tx=0x7f3d9c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2cdf4da73eea0e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:50.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.458+0000 7f3da77fe700 1 -- 172.21.15.23:0/3661552833 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c005e30 con 0x7f3da8152dd0 2023-06-22T08:02:50.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.458+0000 7f3da77fe700 1 -- 172.21.15.23:0/3661552833 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9c004030 con 0x7f3da8152dd0 2023-06-22T08:02:50.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.458+0000 7f3da77fe700 1 -- 172.21.15.23:0/3661552833 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c0043e0 con 0x7f3da8152dd0 2023-06-22T08:02:50.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.459+0000 7f3daedff700 1 -- 172.21.15.23:0/3661552833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da8152dd0 msgr2=0x7f3da80aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:50.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.459+0000 7f3daedff700 1 --2- 172.21.15.23:0/3661552833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da8152dd0 0x7f3da80aedc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3da8070880 tx=0x7f3d9c009bd0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.459+0000 7f3daedff700 1 -- 172.21.15.23:0/3661552833 shutdown_connections 2023-06-22T08:02:50.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.459+0000 7f3daedff700 1 --2- 172.21.15.23:0/3661552833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da8152dd0 0x7f3da80aedc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.459+0000 7f3daedff700 1 --2- 172.21.15.23:0/3661552833 >> v2:172.21.15.23:3300/0 conn(0x7f3da80c8bd0 0x7f3da806afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.459+0000 7f3daedff700 1 -- 172.21.15.23:0/3661552833 >> 172.21.15.23:0/3661552833 conn(0x7f3da806fb70 msgr2=0x7f3da806ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:02:50.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.459+0000 7f3daedff700 1 -- 172.21.15.23:0/3661552833 shutdown_connections 2023-06-22T08:02:50.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.459+0000 7f3daedff700 1 -- 172.21.15.23:0/3661552833 wait complete. 2023-06-22T08:02:50.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.461+0000 7f3daedff700 1 Processor -- start 2023-06-22T08:02:50.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.461+0000 7f3daedff700 1 -- start start 2023-06-22T08:02:50.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.462+0000 7f3daedff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da806b500 0x7f3da80cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.462+0000 7f3daedff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3da80c8bd0 0x7f3da80add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.462+0000 7f3daedff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3da8152dd0 0x7f3da80ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.462+0000 7f3daedff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da81eeb10 con 0x7f3da80c8bd0 2023-06-22T08:02:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.462+0000 7f3daedff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da81ef000 con 0x7f3da806b500 2023-06-22T08:02:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.462+0000 7f3daedff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da80aeee0 con 0x7f3da8152dd0 2023-06-22T08:02:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3dacb9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da806b500 0x7f3da80cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3dacb9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da806b500 0x7f3da80cbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:60776/0 (socket says 172.21.15.23:60776) 2023-06-22T08:02:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3dacb9b700 1 -- 172.21.15.23:0/2342218336 learned_addr learned my addr 172.21.15.23:0/2342218336 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:02:50.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3da7fff700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3da80c8bd0 0x7f3da80add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:50.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3dad39c700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3da8152dd0 0x7f3da80ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:50.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3da7fff700 1 -- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3da8152dd0 msgr2=0x7f3da80ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:50.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3da7fff700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3da8152dd0 0x7f3da80ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3da7fff700 1 -- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da806b500 msgr2=0x7f3da80cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:50.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3da7fff700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da806b500 0x7f3da80cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3da7fff700 1 -- 172.21.15.23:0/2342218336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d940094a0 con 0x7f3da80c8bd0 2023-06-22T08:02:50.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3dacb9b700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da806b500 0x7f3da80cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:02:50.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.463+0000 7f3da7fff700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3da80c8bd0 0x7f3da80add10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400e9b0 tx=0x7f3d9400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:50.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.464+0000 7f3da57fa700 1 -- 172.21.15.23:0/2342218336 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9400ee50 con 0x7f3da80c8bd0 2023-06-22T08:02:50.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.464+0000 7f3da57fa700 1 -- 172.21.15.23:0/2342218336 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9400c850 con 0x7f3da80c8bd0 2023-06-22T08:02:50.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.464+0000 7f3da57fa700 1 -- 172.21.15.23:0/2342218336 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9400cc00 con 0x7f3da80c8bd0 2023-06-22T08:02:50.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.464+0000 7f3daedff700 1 -- 172.21.15.23:0/2342218336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c009500 con 0x7f3da80c8bd0 2023-06-22T08:02:50.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.464+0000 7f3daedff700 1 -- 172.21.15.23:0/2342218336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da80af180 con 0x7f3da80c8bd0 2023-06-22T08:02:50.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.465+0000 7f3daedff700 1 -- 172.21.15.23:0/2342218336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da80ada30 con 0x7f3da80c8bd0 2023-06-22T08:02:50.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.466+0000 7f3da57fa700 1 -- 172.21.15.23:0/2342218336 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3d94013470 con 0x7f3da80c8bd0 2023-06-22T08:02:50.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.467+0000 7f3da57fa700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d8c06ace0 0x7f3d8c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:50.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.467+0000 7f3da57fa700 1 -- 172.21.15.23:0/2342218336 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6925+0+0 (secure 0 0 0) 0x7f3d94089ba0 con 0x7f3da80c8bd0 2023-06-22T08:02:50.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.468+0000 7f3dacb9b700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d8c06ace0 0x7f3d8c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:50.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.468+0000 7f3dacb9b700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d8c06ace0 0x7f3d8c06d190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c000c00 tx=0x7f3d9c008070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:50.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.472+0000 7f3da57fa700 1 -- 172.21.15.23:0/2342218336 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3d940548e0 con 0x7f3da80c8bd0 2023-06-22T08:02:50.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.921+0000 7f3daedff700 1 -- 172.21.15.23:0/2342218336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3da806c510 con 0x7f3da80c8bd0 2023-06-22T08:02:50.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.922+0000 7f3da57fa700 1 -- 172.21.15.23:0/2342218336 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6252 (secure 0 0 0) 0x7f3d94057f00 con 0x7f3da80c8bd0 2023-06-22T08:02:50.927 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:02:50.927 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:02:47.290571+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:02:50.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.928+0000 7f3daedff700 1 -- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d8c06ace0 msgr2=0x7f3d8c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:50.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.928+0000 7f3daedff700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d8c06ace0 0x7f3d8c06d190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c000c00 tx=0x7f3d9c008070 comp rx=0 tx=0).stop 2023-06-22T08:02:50.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.928+0000 7f3daedff700 1 -- 172.21.15.23:0/2342218336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3da80c8bd0 msgr2=0x7f3da80add10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:50.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.928+0000 7f3daedff700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3da80c8bd0 0x7f3da80add10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400e9b0 tx=0x7f3d9400ecc0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.929+0000 7f3daedff700 1 -- 172.21.15.23:0/2342218336 shutdown_connections 2023-06-22T08:02:50.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.929+0000 7f3daedff700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3da8152dd0 0x7f3da80ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.929+0000 7f3daedff700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d8c06ace0 0x7f3d8c06d190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.929+0000 7f3daedff700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da806b500 0x7f3da80cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.929+0000 7f3daedff700 1 --2- 172.21.15.23:0/2342218336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3da80c8bd0 0x7f3da80add10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:50.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.929+0000 7f3daedff700 1 -- 172.21.15.23:0/2342218336 >> 172.21.15.23:0/2342218336 conn(0x7f3da806fb70 msgr2=0x7f3da81523d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:02:50.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.929+0000 7f3daedff700 1 -- 172.21.15.23:0/2342218336 shutdown_connections 2023-06-22T08:02:50.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:50.929+0000 7f3daedff700 1 -- 172.21.15.23:0/2342218336 wait complete. 2023-06-22T08:02:50.937 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 25 2023-06-22T08:02:50.966 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=4) 2023-06-22T08:02:50.966 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-06-22T08:02:50.966 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:02:50.998 INFO:tasks.ceph.mds.e:Stopped 2023-06-22T08:02:50.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:02:51.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.237+0000 7fe9df3ff700 1 Processor -- start 2023-06-22T08:02:51.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.237+0000 7fe9df3ff700 1 -- start start 2023-06-22T08:02:51.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.238+0000 7fe9df3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9d80c69e0 0x7fe9d8155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:51.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.238+0000 7fe9df3ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9d8071070 0x7fe9d8155b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:51.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.238+0000 7fe9df3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d8156080 con 0x7fe9d814f850 2023-06-22T08:02:51.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.238+0000 7fe9df3ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d80c51a0 con 0x7fe9d8071070 2023-06-22T08:02:51.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.238+0000 7fe9df3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80c44e0 con 0x7fe9d80c69e0 2023-06-22T08:02:51.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9dd99c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9d8071070 0x7fe9d8155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:51.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9dd19b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe9d814f850 0x7fe9d80c3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:56516/0 (socket says 172.21.15.23:56516) 2023-06-22T08:02:51.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9dd19b700 1 -- 172.21.15.23:0/2857369313 learned_addr learned my addr 172.21.15.23:0/2857369313 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:02:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9dc99a700 1 --2- 172.21.15.23:0/2857369313 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9d80c69e0 0x7fe9d8155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9dd99c700 1 -- 172.21.15.23:0/2857369313 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9d80c69e0 msgr2=0x7fe9d8155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:02:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9dd99c700 1 --2- 172.21.15.23:0/2857369313 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9d80c69e0 0x7fe9d8155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9dd99c700 1 -- 172.21.15.23:0/2857369313 >> v1:172.21.15.67:6789/0 conn(0x7fe9d814f850 legacy=0x7fe9d80c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9dd99c700 1 -- 172.21.15.23:0/2857369313 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d80c5440 con 0x7fe9d8071070 2023-06-22T08:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9cffff700 1 -- 172.21.15.23:0/2857369313 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256595285 0 0) 0x7fe9d8156080 con 0x7fe9d814f850 2023-06-22T08:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9dd99c700 1 --2- 172.21.15.23:0/2857369313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8071070 0x7fe9d8155b40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009d90 tx=0x7fe9d000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=832afc7be9a28049 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.239+0000 7fe9cffff700 1 -- 172.21.15.23:0/2857369313 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000e070 con 0x7fe9d8071070 2023-06-22T08:02:51.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.240+0000 7fe9cffff700 1 -- 172.21.15.23:0/2857369313 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d000bb20 con 0x7fe9d8071070 2023-06-22T08:02:51.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.240+0000 7fe9cffff700 1 -- 172.21.15.23:0/2857369313 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0012430 con 0x7fe9d8071070 2023-06-22T08:02:51.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.240+0000 7fe9df3ff700 1 -- 172.21.15.23:0/2857369313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8071070 msgr2=0x7fe9d8155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.240+0000 7fe9df3ff700 1 --2- 172.21.15.23:0/2857369313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8071070 0x7fe9d8155b40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009d90 tx=0x7fe9d000b330 comp rx=0 tx=0).stop 2023-06-22T08:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.240+0000 7fe9df3ff700 1 -- 172.21.15.23:0/2857369313 shutdown_connections 2023-06-22T08:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.240+0000 7fe9df3ff700 1 --2- 172.21.15.23:0/2857369313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8071070 0x7fe9d8155b40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.240+0000 7fe9df3ff700 1 --2- 172.21.15.23:0/2857369313 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9d80c69e0 0x7fe9d8155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:51.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.240+0000 7fe9df3ff700 1 -- 172.21.15.23:0/2857369313 >> 172.21.15.23:0/2857369313 conn(0x7fe9d80ca3f0 msgr2=0x7fe9d80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:02:51.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.240+0000 7fe9df3ff700 1 -- 172.21.15.23:0/2857369313 shutdown_connections 2023-06-22T08:02:51.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.240+0000 7fe9df3ff700 1 -- 172.21.15.23:0/2857369313 wait complete. 2023-06-22T08:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.242+0000 7fe9df3ff700 1 Processor -- start 2023-06-22T08:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.242+0000 7fe9df3ff700 1 -- start start 2023-06-22T08:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9df3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9d8071070 0x7fe9d81dc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9df3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9d80c69e0 0x7fe9d81dde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9df3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d814f850 0x7fe9d81e04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9df3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80c8d50 con 0x7fe9d80c69e0 2023-06-22T08:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9df3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d81dd2f0 con 0x7fe9d814f850 2023-06-22T08:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9df3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d81dfd40 con 0x7fe9d8071070 2023-06-22T08:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9dc99a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9d80c69e0 0x7fe9d81dde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9dd99c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d814f850 0x7fe9d81e04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9dd19b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9d8071070 0x7fe9d81dc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:51.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9dc99a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9d80c69e0 0x7fe9d81dde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59704/0 (socket says 172.21.15.23:59704) 2023-06-22T08:02:51.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.243+0000 7fe9dc99a700 1 -- 172.21.15.23:0/145829608 learned_addr learned my addr 172.21.15.23:0/145829608 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:02:51.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9dd99c700 1 -- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9d8071070 msgr2=0x7fe9d81dc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:51.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9dd99c700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9d8071070 0x7fe9d81dc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:51.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9dd99c700 1 -- 172.21.15.23:0/145829608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9d80c69e0 msgr2=0x7fe9d81dde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:51.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9dd99c700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9d80c69e0 0x7fe9d81dde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:51.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9dd99c700 1 -- 172.21.15.23:0/145829608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d0009a70 con 0x7fe9d814f850 2023-06-22T08:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9dd99c700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d814f850 0x7fe9d81e04b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0016000 tx=0x7fe9d0004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9cdffb700 1 -- 172.21.15.23:0/145829608 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0012430 con 0x7fe9d814f850 2023-06-22T08:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9cdffb700 1 -- 172.21.15.23:0/145829608 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d0020070 con 0x7fe9d814f850 2023-06-22T08:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9df3ff700 1 -- 172.21.15.23:0/145829608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d81e0020 con 0x7fe9d814f850 2023-06-22T08:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9cdffb700 1 -- 172.21.15.23:0/145829608 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0004280 con 0x7fe9d814f850 2023-06-22T08:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.244+0000 7fe9df3ff700 1 -- 172.21.15.23:0/145829608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d81dd010 con 0x7fe9d814f850 2023-06-22T08:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.245+0000 7fe9df3ff700 1 -- 172.21.15.23:0/145829608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d8066900 con 0x7fe9d814f850 2023-06-22T08:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.246+0000 7fe9cdffb700 1 -- 172.21.15.23:0/145829608 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe9d00043e0 con 0x7fe9d814f850 2023-06-22T08:02:51.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.247+0000 7fe9cdffb700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe9bc06ace0 0x7fe9bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:51.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.247+0000 7fe9cdffb700 1 -- 172.21.15.23:0/145829608 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6925+0+0 (secure 0 0 0) 0x7fe9d0022070 con 0x7fe9d814f850 2023-06-22T08:02:51.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.248+0000 7fe9dd19b700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe9bc06ace0 0x7fe9bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:51.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.248+0000 7fe9dd19b700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe9bc06ace0 0x7fe9bc06d190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4000f60 tx=0x7fe9d4007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:51.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.251+0000 7fe9cdffb700 1 -- 172.21.15.23:0/145829608 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe9d00565d0 con 0x7fe9d814f850 2023-06-22T08:02:51.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.504+0000 7fe9df3ff700 1 -- 172.21.15.23:0/145829608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9d80c2950 con 0x7fe9d814f850 2023-06-22T08:02:51.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.505+0000 7fe9cdffb700 1 -- 172.21.15.23:0/145829608 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+6252 (secure 0 0 0) 0x7fe9d0059bf0 con 0x7fe9d814f850 2023-06-22T08:02:51.509 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:02:51.509 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:02:47.290571+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:02:51.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 -- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe9bc06ace0 msgr2=0x7fe9bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:51.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe9bc06ace0 0x7fe9bc06d190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4000f60 tx=0x7fe9d4007e70 comp rx=0 tx=0).stop 2023-06-22T08:02:51.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 -- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d814f850 msgr2=0x7fe9d81e04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:51.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d814f850 0x7fe9d81e04b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0016000 tx=0x7fe9d0004ad0 comp rx=0 tx=0).stop 2023-06-22T08:02:51.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 -- 172.21.15.23:0/145829608 shutdown_connections 2023-06-22T08:02:51.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe9bc06ace0 0x7fe9bc06d190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:51.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d814f850 0x7fe9d81e04b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:51.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe9d8071070 0x7fe9d81dc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:51.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 --2- 172.21.15.23:0/145829608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9d80c69e0 0x7fe9d81dde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:51.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 -- 172.21.15.23:0/145829608 >> 172.21.15.23:0/145829608 conn(0x7fe9d80ca3f0 msgr2=0x7fe9d806a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:02:51.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 -- 172.21.15.23:0/145829608 shutdown_connections 2023-06-22T08:02:51.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:51.508+0000 7fe9df3ff700 1 -- 172.21.15.23:0/145829608 wait complete. 2023-06-22T08:02:51.513 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 25 2023-06-22T08:02:51.530 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:02:53.531 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:02:53.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.731+0000 7f3b8bfff700 1 Processor -- start 2023-06-22T08:02:53.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.731+0000 7f3b8bfff700 1 -- start start 2023-06-22T08:02:53.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b8bfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b840cb6b0 0x7f3b84069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:53.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b8bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b840c5740 0x7f3b841593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:53.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b8bfff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b84158f60 con 0x7f3b840cba80 2023-06-22T08:02:53.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b8bfff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8406bd20 con 0x7f3b840cb6b0 2023-06-22T08:02:53.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b8bfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b840cc450 con 0x7f3b840c5740 2023-06-22T08:02:53.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b8a59c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f3b840cba80 0x7f3b84152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:46608/0 (socket says 172.21.15.23:46608) 2023-06-22T08:02:53.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b8a59c700 1 -- 172.21.15.23:0/1190097806 learned_addr learned my addr 172.21.15.23:0/1190097806 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:02:53.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b89d9b700 1 --2- 172.21.15.23:0/1190097806 >> v2:172.21.15.67:3300/0 conn(0x7f3b840cb6b0 0x7f3b84069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:53.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b8959a700 1 --2- 172.21.15.23:0/1190097806 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b840c5740 0x7f3b841593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:53.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b88d99700 1 -- 172.21.15.23:0/1190097806 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409609114 0 0) 0x7f3b84158f60 con 0x7f3b840cba80 2023-06-22T08:02:53.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.733+0000 7f3b88d99700 1 -- 172.21.15.23:0/1190097806 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b700035b0 con 0x7f3b840cba80 2023-06-22T08:02:53.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.734+0000 7f3b8959a700 1 -- 172.21.15.23:0/1190097806 >> v2:172.21.15.67:3300/0 conn(0x7f3b840cb6b0 msgr2=0x7f3b84069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:02:53.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.734+0000 7f3b8959a700 1 --2- 172.21.15.23:0/1190097806 >> v2:172.21.15.67:3300/0 conn(0x7f3b840cb6b0 0x7f3b84069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:53.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.734+0000 7f3b8959a700 1 -- 172.21.15.23:0/1190097806 >> v1:172.21.15.23:6789/0 conn(0x7f3b840cba80 legacy=0x7f3b84152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:53.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.734+0000 7f3b8959a700 1 -- 172.21.15.23:0/1190097806 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b84158980 con 0x7f3b840c5740 2023-06-22T08:02:53.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.734+0000 7f3b89d9b700 1 --2- 172.21.15.23:0/1190097806 >> v2:172.21.15.67:3300/0 conn(0x7f3b840cb6b0 0x7f3b84069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:02:53.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.734+0000 7f3b8959a700 1 --2- 172.21.15.23:0/1190097806 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b840c5740 0x7f3b841593f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009840 tx=0x7f3b74009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7373283d4ce2d90 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:53.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.734+0000 7f3b88d99700 1 -- 172.21.15.23:0/1190097806 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74011830 con 0x7f3b840c5740 2023-06-22T08:02:53.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.734+0000 7f3b88d99700 1 -- 172.21.15.23:0/1190097806 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b74011990 con 0x7f3b840c5740 2023-06-22T08:02:53.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.734+0000 7f3b88d99700 1 -- 172.21.15.23:0/1190097806 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b74011d00 con 0x7f3b840c5740 2023-06-22T08:02:53.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.735+0000 7f3b8bfff700 1 -- 172.21.15.23:0/1190097806 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b840c5740 msgr2=0x7f3b841593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:53.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.735+0000 7f3b8bfff700 1 --2- 172.21.15.23:0/1190097806 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b840c5740 0x7f3b841593f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009840 tx=0x7f3b74009b50 comp rx=0 tx=0).stop 2023-06-22T08:02:53.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.735+0000 7f3b8bfff700 1 -- 172.21.15.23:0/1190097806 shutdown_connections 2023-06-22T08:02:53.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.735+0000 7f3b8bfff700 1 --2- 172.21.15.23:0/1190097806 >> v2:172.21.15.67:3300/0 conn(0x7f3b840cb6b0 0x7f3b84069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:53.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.735+0000 7f3b8bfff700 1 --2- 172.21.15.23:0/1190097806 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b840c5740 0x7f3b841593f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:53.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.735+0000 7f3b8bfff700 1 -- 172.21.15.23:0/1190097806 >> 172.21.15.23:0/1190097806 conn(0x7f3b84070d80 msgr2=0x7f3b84071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:02:53.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.735+0000 7f3b8bfff700 1 -- 172.21.15.23:0/1190097806 shutdown_connections 2023-06-22T08:02:53.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.735+0000 7f3b8bfff700 1 -- 172.21.15.23:0/1190097806 wait complete. 2023-06-22T08:02:53.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.737+0000 7f3b8bfff700 1 Processor -- start 2023-06-22T08:02:53.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.737+0000 7f3b8bfff700 1 -- start start 2023-06-22T08:02:53.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b8bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b840cb6b0 0x7f3b8415afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:53.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b8bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840cba80 0x7f3b841434f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:53.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b8bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b84156e90 0x7f3b84143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:53.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b8bfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b841443e0 con 0x7f3b840cb6b0 2023-06-22T08:02:53.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b8bfff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b841f1630 con 0x7f3b840cba80 2023-06-22T08:02:53.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b8bfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8414cb60 con 0x7f3b84156e90 2023-06-22T08:02:53.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b89d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b840cb6b0 0x7f3b8415afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:53.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b89d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b840cb6b0 0x7f3b8415afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59720/0 (socket says 172.21.15.23:59720) 2023-06-22T08:02:53.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b89d9b700 1 -- 172.21.15.23:0/3366733904 learned_addr learned my addr 172.21.15.23:0/3366733904 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:02:53.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b8a59c700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b84156e90 0x7f3b84143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:53.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b8959a700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840cba80 0x7f3b841434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:53.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b89d9b700 1 -- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b84156e90 msgr2=0x7f3b84143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:53.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b89d9b700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b84156e90 0x7f3b84143a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:53.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b89d9b700 1 -- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840cba80 msgr2=0x7f3b841434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:53.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b89d9b700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840cba80 0x7f3b841434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:53.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b89d9b700 1 -- 172.21.15.23:0/3366733904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b800094a0 con 0x7f3b840cb6b0 2023-06-22T08:02:53.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.738+0000 7f3b8959a700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840cba80 0x7f3b841434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:02:53.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.739+0000 7f3b89d9b700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b840cb6b0 0x7f3b8415afd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000ebe0 tx=0x7f3b8000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:53.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.739+0000 7f3b7a7fc700 1 -- 172.21.15.23:0/3366733904 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80009800 con 0x7f3b840cb6b0 2023-06-22T08:02:53.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.739+0000 7f3b7a7fc700 1 -- 172.21.15.23:0/3366733904 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8000c8b0 con 0x7f3b840cb6b0 2023-06-22T08:02:53.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.739+0000 7f3b7a7fc700 1 -- 172.21.15.23:0/3366733904 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8000cc20 con 0x7f3b840cb6b0 2023-06-22T08:02:53.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.739+0000 7f3b8bfff700 1 -- 172.21.15.23:0/3366733904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b740094a0 con 0x7f3b840cb6b0 2023-06-22T08:02:53.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.739+0000 7f3b8bfff700 1 -- 172.21.15.23:0/3366733904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b840c4dd0 con 0x7f3b840cb6b0 2023-06-22T08:02:53.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.740+0000 7f3b8bfff700 1 -- 172.21.15.23:0/3366733904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b840c4950 con 0x7f3b840cb6b0 2023-06-22T08:02:53.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.741+0000 7f3b7a7fc700 1 -- 172.21.15.23:0/3366733904 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3b8000cd80 con 0x7f3b840cb6b0 2023-06-22T08:02:53.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.742+0000 7f3b7a7fc700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3b6806ad10 0x7f3b6806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:53.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.742+0000 7f3b7a7fc700 1 -- 172.21.15.23:0/3366733904 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6925+0+0 (secure 0 0 0) 0x7f3b800897c0 con 0x7f3b840cb6b0 2023-06-22T08:02:53.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.742+0000 7f3b8959a700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3b6806ad10 0x7f3b6806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:53.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.743+0000 7f3b8959a700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3b6806ad10 0x7f3b6806d1c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009fd0 tx=0x7f3b74008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:53.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:53.746+0000 7f3b7a7fc700 1 -- 172.21.15.23:0/3366733904 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3b840c4950 con 0x7f3b840cb6b0 2023-06-22T08:02:54.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.002+0000 7f3b8bfff700 1 -- 172.21.15.23:0/3366733904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b841f17a0 con 0x7f3b840cb6b0 2023-06-22T08:02:54.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.002+0000 7f3b7a7fc700 1 -- 172.21.15.23:0/3366733904 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6254 (secure 0 0 0) 0x7f3b841f17a0 con 0x7f3b840cb6b0 2023-06-22T08:02:54.006 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:02:54.006 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:02:51.659992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:02:54.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 -- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3b6806ad10 msgr2=0x7f3b6806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:54.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3b6806ad10 0x7f3b6806d1c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009fd0 tx=0x7f3b74008030 comp rx=0 tx=0).stop 2023-06-22T08:02:54.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 -- 172.21.15.23:0/3366733904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b840cb6b0 msgr2=0x7f3b8415afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:54.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b840cb6b0 0x7f3b8415afd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000ebe0 tx=0x7f3b8000eef0 comp rx=0 tx=0).stop 2023-06-22T08:02:54.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 -- 172.21.15.23:0/3366733904 shutdown_connections 2023-06-22T08:02:54.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3b84156e90 0x7f3b84143a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:54.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b840cb6b0 0x7f3b8415afd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:54.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3b6806ad10 0x7f3b6806d1c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:54.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 --2- 172.21.15.23:0/3366733904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b840cba80 0x7f3b841434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:54.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 -- 172.21.15.23:0/3366733904 >> 172.21.15.23:0/3366733904 conn(0x7f3b84070d80 msgr2=0x7f3b84150290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:02:54.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 -- 172.21.15.23:0/3366733904 shutdown_connections 2023-06-22T08:02:54.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:54.005+0000 7f3b8bfff700 1 -- 172.21.15.23:0/3366733904 wait complete. 2023-06-22T08:02:54.010 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 26 2023-06-22T08:02:54.027 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:02:56.028 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:02:56.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.236+0000 7ffa925ff700 1 Processor -- start 2023-06-22T08:02:56.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.236+0000 7ffa925ff700 1 -- start start 2023-06-22T08:02:56.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.237+0000 7ffa925ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c97d0 0x7ffa8c159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:56.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.237+0000 7ffa925ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa8c0c45f0 0x7ffa8c0c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:56.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.237+0000 7ffa925ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa8c0c4fb0 con 0x7ffa8c152d70 2023-06-22T08:02:56.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.237+0000 7ffa925ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa8c0c95e0 con 0x7ffa8c0c45f0 2023-06-22T08:02:56.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.237+0000 7ffa925ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c0c9c00 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.237+0000 7ffa8b7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c97d0 0x7ffa8c159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:56.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.237+0000 7ffa90b9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa8c0c45f0 0x7ffa8c0c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:56.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.237+0000 7ffa8bfff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffa8c152d70 0x7ffa8c0cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:60240/0 (socket says 172.21.15.23:60240) 2023-06-22T08:02:56.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.237+0000 7ffa8bfff700 1 -- 172.21.15.23:0/1565987164 learned_addr learned my addr 172.21.15.23:0/1565987164 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:02:56.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.238+0000 7ffa8affd700 1 -- 172.21.15.23:0/1565987164 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607980496 0 0) 0x7ffa8c0c4fb0 con 0x7ffa8c152d70 2023-06-22T08:02:56.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.238+0000 7ffa8b7fe700 1 -- 172.21.15.23:0/1565987164 >> v2:172.21.15.67:3300/0 conn(0x7ffa8c0c45f0 msgr2=0x7ffa8c0c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:02:56.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.238+0000 7ffa8b7fe700 1 --2- 172.21.15.23:0/1565987164 >> v2:172.21.15.67:3300/0 conn(0x7ffa8c0c45f0 0x7ffa8c0c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:56.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.238+0000 7ffa8b7fe700 1 -- 172.21.15.23:0/1565987164 >> v1:172.21.15.67:6789/0 conn(0x7ffa8c152d70 legacy=0x7ffa8c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:56.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.238+0000 7ffa8b7fe700 1 -- 172.21.15.23:0/1565987164 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c15bea0 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.238+0000 7ffa8b7fe700 1 --2- 172.21.15.23:0/1565987164 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c97d0 0x7ffa8c159330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009840 tx=0x7ffa80009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=317a331fd57cdd4d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.238+0000 7ffa8affd700 1 -- 172.21.15.23:0/1565987164 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa80011830 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.238+0000 7ffa8affd700 1 -- 172.21.15.23:0/1565987164 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa80011990 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.238+0000 7ffa8affd700 1 -- 172.21.15.23:0/1565987164 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa80011d20 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.239+0000 7ffa925ff700 1 -- 172.21.15.23:0/1565987164 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c97d0 msgr2=0x7ffa8c159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.239+0000 7ffa925ff700 1 --2- 172.21.15.23:0/1565987164 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c97d0 0x7ffa8c159330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009840 tx=0x7ffa80009b50 comp rx=0 tx=0).stop 2023-06-22T08:02:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.239+0000 7ffa925ff700 1 -- 172.21.15.23:0/1565987164 shutdown_connections 2023-06-22T08:02:56.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.239+0000 7ffa925ff700 1 --2- 172.21.15.23:0/1565987164 >> v2:172.21.15.67:3300/0 conn(0x7ffa8c0c45f0 0x7ffa8c0c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:56.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.239+0000 7ffa925ff700 1 --2- 172.21.15.23:0/1565987164 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c97d0 0x7ffa8c159330 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:56.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.239+0000 7ffa925ff700 1 -- 172.21.15.23:0/1565987164 >> 172.21.15.23:0/1565987164 conn(0x7ffa8c06ae30 msgr2=0x7ffa8c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:02:56.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.239+0000 7ffa925ff700 1 -- 172.21.15.23:0/1565987164 shutdown_connections 2023-06-22T08:02:56.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.239+0000 7ffa925ff700 1 -- 172.21.15.23:0/1565987164 wait complete. 2023-06-22T08:02:56.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.241+0000 7ffa925ff700 1 Processor -- start 2023-06-22T08:02:56.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.241+0000 7ffa925ff700 1 -- start start 2023-06-22T08:02:56.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa925ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c45f0 0x7ffa8c1570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:56.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa925ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa8c0c97d0 0x7ffa8c0b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:56.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa925ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa8c152d70 0x7ffa8c14b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:56.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa925ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c157e80 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa925ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c158160 con 0x7ffa8c152d70 2023-06-22T08:02:56.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa925ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c158440 con 0x7ffa8c0c45f0 2023-06-22T08:02:56.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa8b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa8c0c97d0 0x7ffa8c0b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:56.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa8bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c45f0 0x7ffa8c1570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:56.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa8b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa8c0c97d0 0x7ffa8c0b3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48472/0 (socket says 172.21.15.23:48472) 2023-06-22T08:02:56.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa90b9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa8c152d70 0x7ffa8c14b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:56.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.242+0000 7ffa8b7fe700 1 -- 172.21.15.23:0/2830961300 learned_addr learned my addr 172.21.15.23:0/2830961300 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:02:56.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa8b7fe700 1 -- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c45f0 msgr2=0x7ffa8c1570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:56.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa8b7fe700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c45f0 0x7ffa8c1570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:56.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa8b7fe700 1 -- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa8c152d70 msgr2=0x7ffa8c14b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:56.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa8b7fe700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa8c152d70 0x7ffa8c14b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:56.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa8b7fe700 1 -- 172.21.15.23:0/2830961300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa800094a0 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa90b9c700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa8c152d70 0x7ffa8c14b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:02:56.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa8b7fe700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa8c0c97d0 0x7ffa8c0b3620 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009b90 tx=0x7ffa80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:56.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa88ff9700 1 -- 172.21.15.23:0/2830961300 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa800036a0 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa88ff9700 1 -- 172.21.15.23:0/2830961300 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8001f070 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa925ff700 1 -- 172.21.15.23:0/2830961300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8c158720 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.243+0000 7ffa88ff9700 1 -- 172.21.15.23:0/2830961300 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa80003a40 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.244+0000 7ffa925ff700 1 -- 172.21.15.23:0/2830961300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8c0c8c70 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.245+0000 7ffa925ff700 1 -- 172.21.15.23:0/2830961300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa8c0cd7f0 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.246+0000 7ffa88ff9700 1 -- 172.21.15.23:0/2830961300 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffa80003ba0 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.246+0000 7ffa88ff9700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa7806ace0 0x7ffa7806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:56.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.247+0000 7ffa88ff9700 1 -- 172.21.15.23:0/2830961300 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6897+0+0 (secure 0 0 0) 0x7ffa80021070 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.247+0000 7ffa8bfff700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa7806ace0 0x7ffa7806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:56.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.248+0000 7ffa8bfff700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa7806ace0 0x7ffa7806d190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c000f60 tx=0x7ffa7c007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:56.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.251+0000 7ffa88ff9700 1 -- 172.21.15.23:0/2830961300 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffa8c0cd7f0 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.515+0000 7ffa925ff700 1 -- 172.21.15.23:0/2830961300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa8c0cd7f0 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.516+0000 7ffa88ff9700 1 -- 172.21.15.23:0/2830961300 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6254 (secure 0 0 0) 0x7ffa8c0cd7f0 con 0x7ffa8c0c97d0 2023-06-22T08:02:56.520 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:02:56.520 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:02:51.659992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:02:56.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 -- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa7806ace0 msgr2=0x7ffa7806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:56.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa7806ace0 0x7ffa7806d190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c000f60 tx=0x7ffa7c007e70 comp rx=0 tx=0).stop 2023-06-22T08:02:56.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 -- 172.21.15.23:0/2830961300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa8c0c97d0 msgr2=0x7ffa8c0b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:56.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa8c0c97d0 0x7ffa8c0b3620 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009b90 tx=0x7ffa80004750 comp rx=0 tx=0).stop 2023-06-22T08:02:56.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 -- 172.21.15.23:0/2830961300 shutdown_connections 2023-06-22T08:02:56.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa7806ace0 0x7ffa7806d190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:56.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa8c152d70 0x7ffa8c14b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:56.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa8c0c45f0 0x7ffa8c1570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:56.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 --2- 172.21.15.23:0/2830961300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa8c0c97d0 0x7ffa8c0b3620 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:56.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 -- 172.21.15.23:0/2830961300 >> 172.21.15.23:0/2830961300 conn(0x7ffa8c06ae30 msgr2=0x7ffa8c150fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:02:56.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 -- 172.21.15.23:0/2830961300 shutdown_connections 2023-06-22T08:02:56.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:56.518+0000 7ffa925ff700 1 -- 172.21.15.23:0/2830961300 wait complete. 2023-06-22T08:02:56.524 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 26 2023-06-22T08:02:56.539 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:02:58.540 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:02:58.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.749+0000 7f3f657ff700 1 Processor -- start 2023-06-22T08:02:58.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.749+0000 7f3f657ff700 1 -- start start 2023-06-22T08:02:58.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.750+0000 7f3f657ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f60152ce0 0x7f3f600cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:58.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.750+0000 7f3f657ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f600c9fd0 0x7f3f6015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:58.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.750+0000 7f3f657ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6015c370 con 0x7f3f60158880 2023-06-22T08:02:58.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.750+0000 7f3f657ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f601530b0 con 0x7f3f600c9fd0 2023-06-22T08:02:58.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.750+0000 7f3f657ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f600cf6f0 con 0x7f3f60152ce0 2023-06-22T08:02:58.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5effd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f60152ce0 0x7f3f600cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:58.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5e7fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f600c9fd0 0x7f3f6015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:58.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5f7fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3f60158880 0x7f3f60158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:60254/0 (socket says 172.21.15.23:60254) 2023-06-22T08:02:58.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5f7fe700 1 -- 172.21.15.23:0/2618995167 learned_addr learned my addr 172.21.15.23:0/2618995167 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:02:58.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5dffb700 1 -- 172.21.15.23:0/2618995167 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470955287 0 0) 0x7f3f6015c370 con 0x7f3f60158880 2023-06-22T08:02:58.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5dffb700 1 -- 172.21.15.23:0/2618995167 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f480035b0 con 0x7f3f60158880 2023-06-22T08:02:58.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5effd700 1 -- 172.21.15.23:0/2618995167 >> v2:172.21.15.67:3300/0 conn(0x7f3f600c9fd0 msgr2=0x7f3f6015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:02:58.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5effd700 1 --2- 172.21.15.23:0/2618995167 >> v2:172.21.15.67:3300/0 conn(0x7f3f600c9fd0 0x7f3f6015c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:58.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5effd700 1 -- 172.21.15.23:0/2618995167 >> v1:172.21.15.67:6789/0 conn(0x7f3f60158880 legacy=0x7f3f60158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:58.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5effd700 1 -- 172.21.15.23:0/2618995167 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6015bda0 con 0x7f3f60152ce0 2023-06-22T08:02:58.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.751+0000 7f3f5e7fc700 1 --2- 172.21.15.23:0/2618995167 >> v2:172.21.15.67:3300/0 conn(0x7f3f600c9fd0 0x7f3f6015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:02:58.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.752+0000 7f3f5effd700 1 --2- 172.21.15.23:0/2618995167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f60152ce0 0x7f3f600cf1b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009840 tx=0x7f3f50009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed84febfb6f3c49 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:58.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.752+0000 7f3f5dffb700 1 -- 172.21.15.23:0/2618995167 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50011830 con 0x7f3f60152ce0 2023-06-22T08:02:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.752+0000 7f3f5dffb700 1 -- 172.21.15.23:0/2618995167 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f50011990 con 0x7f3f60152ce0 2023-06-22T08:02:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.752+0000 7f3f5dffb700 1 -- 172.21.15.23:0/2618995167 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50011d20 con 0x7f3f60152ce0 2023-06-22T08:02:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.752+0000 7f3f657ff700 1 -- 172.21.15.23:0/2618995167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f60152ce0 msgr2=0x7f3f600cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.752+0000 7f3f657ff700 1 --2- 172.21.15.23:0/2618995167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f60152ce0 0x7f3f600cf1b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009840 tx=0x7f3f50009b50 comp rx=0 tx=0).stop 2023-06-22T08:02:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.752+0000 7f3f657ff700 1 -- 172.21.15.23:0/2618995167 shutdown_connections 2023-06-22T08:02:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.752+0000 7f3f657ff700 1 --2- 172.21.15.23:0/2618995167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f60152ce0 0x7f3f600cf1b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.752+0000 7f3f657ff700 1 --2- 172.21.15.23:0/2618995167 >> v2:172.21.15.67:3300/0 conn(0x7f3f600c9fd0 0x7f3f6015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.752+0000 7f3f657ff700 1 -- 172.21.15.23:0/2618995167 >> 172.21.15.23:0/2618995167 conn(0x7f3f6006ae30 msgr2=0x7f3f6006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:02:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.753+0000 7f3f657ff700 1 -- 172.21.15.23:0/2618995167 shutdown_connections 2023-06-22T08:02:58.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.753+0000 7f3f657ff700 1 -- 172.21.15.23:0/2618995167 wait complete. 2023-06-22T08:02:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.754+0000 7f3f657ff700 1 Processor -- start 2023-06-22T08:02:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.754+0000 7f3f657ff700 1 -- start start 2023-06-22T08:02:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.755+0000 7f3f657ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f600c9fd0 0x7f3f600ce140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.755+0000 7f3f657ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f60152ce0 0x7f3f600b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.755+0000 7f3f657ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f60158880 0x7f3f600b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.755+0000 7f3f657ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f600cdd20 con 0x7f3f60152ce0 2023-06-22T08:02:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.755+0000 7f3f657ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f601f2200 con 0x7f3f60158880 2023-06-22T08:02:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.755+0000 7f3f657ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f600cd6f0 con 0x7f3f600c9fd0 2023-06-22T08:02:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5effd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f600c9fd0 0x7f3f600ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f60152ce0 0x7f3f600b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5effd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f600c9fd0 0x7f3f600ce140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49530/0 (socket says 172.21.15.23:49530) 2023-06-22T08:02:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5effd700 1 -- 172.21.15.23:0/405964858 learned_addr learned my addr 172.21.15.23:0/405964858 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:02:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5f7fe700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f60158880 0x7f3f600b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:58.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5effd700 1 -- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f60158880 msgr2=0x7f3f600b3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:58.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5effd700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f60158880 0x7f3f600b3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:58.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5effd700 1 -- 172.21.15.23:0/405964858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f60152ce0 msgr2=0x7f3f600b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:58.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5effd700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f60152ce0 0x7f3f600b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:58.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5effd700 1 -- 172.21.15.23:0/405964858 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f540094a0 con 0x7f3f600c9fd0 2023-06-22T08:02:58.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5e7fc700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f60152ce0 0x7f3f600b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:02:58.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.756+0000 7f3f5effd700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f600c9fd0 0x7f3f600ce140 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f50000c00 tx=0x7f3f50004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:58.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.757+0000 7f3f477fe700 1 -- 172.21.15.23:0/405964858 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f50004590 con 0x7f3f600c9fd0 2023-06-22T08:02:58.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.757+0000 7f3f477fe700 1 -- 172.21.15.23:0/405964858 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5001f070 con 0x7f3f600c9fd0 2023-06-22T08:02:58.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.757+0000 7f3f657ff700 1 -- 172.21.15.23:0/405964858 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f500094a0 con 0x7f3f600c9fd0 2023-06-22T08:02:58.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.757+0000 7f3f477fe700 1 -- 172.21.15.23:0/405964858 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f500038e0 con 0x7f3f600c9fd0 2023-06-22T08:02:58.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.757+0000 7f3f657ff700 1 -- 172.21.15.23:0/405964858 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f601523a0 con 0x7f3f600c9fd0 2023-06-22T08:02:58.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.758+0000 7f3f657ff700 1 -- 172.21.15.23:0/405964858 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6004ea20 con 0x7f3f600c9fd0 2023-06-22T08:02:58.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.759+0000 7f3f477fe700 1 -- 172.21.15.23:0/405964858 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3f50003a40 con 0x7f3f600c9fd0 2023-06-22T08:02:58.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.760+0000 7f3f477fe700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f4c06ad10 0x7f3f4c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:02:58.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.760+0000 7f3f477fe700 1 -- 172.21.15.23:0/405964858 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6897+0+0 (secure 0 0 0) 0x7f3f50021070 con 0x7f3f600c9fd0 2023-06-22T08:02:58.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.760+0000 7f3f5e7fc700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f4c06ad10 0x7f3f4c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:02:58.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.760+0000 7f3f5e7fc700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f4c06ad10 0x7f3f4c06d1c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3f600c9100 tx=0x7f3f54009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:02:58.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:58.763+0000 7f3f477fe700 1 -- 172.21.15.23:0/405964858 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3f50055f40 con 0x7f3f600c9fd0 2023-06-22T08:02:59.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.016+0000 7f3f657ff700 1 -- 172.21.15.23:0/405964858 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f600c9db0 con 0x7f3f600c9fd0 2023-06-22T08:02:59.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.017+0000 7f3f477fe700 1 -- 172.21.15.23:0/405964858 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6254 (secure 0 0 0) 0x7f3f50015090 con 0x7f3f600c9fd0 2023-06-22T08:02:59.021 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:02:59.021 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:02:51.659992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:02:59.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.019+0000 7f3f657ff700 1 -- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f4c06ad10 msgr2=0x7f3f4c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:59.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.019+0000 7f3f657ff700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f4c06ad10 0x7f3f4c06d1c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3f600c9100 tx=0x7f3f54009210 comp rx=0 tx=0).stop 2023-06-22T08:02:59.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.019+0000 7f3f657ff700 1 -- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f600c9fd0 msgr2=0x7f3f600ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:02:59.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.019+0000 7f3f657ff700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f600c9fd0 0x7f3f600ce140 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f50000c00 tx=0x7f3f50004750 comp rx=0 tx=0).stop 2023-06-22T08:02:59.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.019+0000 7f3f657ff700 1 -- 172.21.15.23:0/405964858 shutdown_connections 2023-06-22T08:02:59.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.019+0000 7f3f657ff700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f4c06ad10 0x7f3f4c06d1c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:59.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.019+0000 7f3f657ff700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f60158880 0x7f3f600b3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:59.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.019+0000 7f3f657ff700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f600c9fd0 0x7f3f600ce140 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:59.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.019+0000 7f3f657ff700 1 --2- 172.21.15.23:0/405964858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f60152ce0 0x7f3f600b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:02:59.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.019+0000 7f3f657ff700 1 -- 172.21.15.23:0/405964858 >> 172.21.15.23:0/405964858 conn(0x7f3f6006ae30 msgr2=0x7f3f600c3850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:02:59.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.020+0000 7f3f657ff700 1 -- 172.21.15.23:0/405964858 shutdown_connections 2023-06-22T08:02:59.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:02:59.020+0000 7f3f657ff700 1 -- 172.21.15.23:0/405964858 wait complete. 2023-06-22T08:02:59.024 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 26 2023-06-22T08:02:59.040 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:03:01.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:01.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.249+0000 7f4b095ff700 1 Processor -- start 2023-06-22T08:03:01.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.249+0000 7f4b095ff700 1 -- start start 2023-06-22T08:03:01.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b095ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 0x7f4b040c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:01.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b095ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4b040c45f0 0x7f4b040c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:01.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b095ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b040c4fb0 con 0x7f4b04159330 2023-06-22T08:03:01.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b095ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b040c95e0 con 0x7f4b040c45f0 2023-06-22T08:03:01.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b095ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0415bea0 con 0x7f4b04152d70 2023-06-22T08:03:01.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b037fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4b040c45f0 0x7f4b040c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:01.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b02ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 0x7f4b040c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:01.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b037fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4b040c45f0 0x7f4b040c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48486/0 (socket says 172.21.15.23:48486) 2023-06-22T08:03:01.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b027fc700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4b04159330 0x7f4b040cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:60266/0 (socket says 172.21.15.23:60266) 2023-06-22T08:03:01.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b02ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 0x7f4b040c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49546/0 (socket says 172.21.15.23:49546) 2023-06-22T08:03:01.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.251+0000 7f4b037fe700 1 -- 172.21.15.23:0/4195570094 learned_addr learned my addr 172.21.15.23:0/4195570094 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:01.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.252+0000 7f4b037fe700 1 -- 172.21.15.23:0/4195570094 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 msgr2=0x7f4b040c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:01.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.252+0000 7f4b037fe700 1 --2- 172.21.15.23:0/4195570094 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 0x7f4b040c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:01.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.252+0000 7f4b01ffb700 1 -- 172.21.15.23:0/4195570094 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285561091 0 0) 0x7f4b040c4fb0 con 0x7f4b04159330 2023-06-22T08:03:01.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.252+0000 7f4b037fe700 1 -- 172.21.15.23:0/4195570094 >> v1:172.21.15.67:6789/0 conn(0x7f4b04159330 legacy=0x7f4b040cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:01.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.252+0000 7f4b037fe700 1 -- 172.21.15.23:0/4195570094 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0415c160 con 0x7f4b040c45f0 2023-06-22T08:03:01.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.252+0000 7f4b02ffd700 1 --2- 172.21.15.23:0/4195570094 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 0x7f4b040c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:03:01.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.252+0000 7f4b037fe700 1 --2- 172.21.15.23:0/4195570094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 0x7f4b040c4a70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009e50 tx=0x7f4afc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0520266cf0b7ba server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:01.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.252+0000 7f4b01ffb700 1 -- 172.21.15.23:0/4195570094 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc00e070 con 0x7f4b040c45f0 2023-06-22T08:03:01.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.252+0000 7f4b01ffb700 1 -- 172.21.15.23:0/4195570094 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc00b910 con 0x7f4b040c45f0 2023-06-22T08:03:01.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.252+0000 7f4b01ffb700 1 -- 172.21.15.23:0/4195570094 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc00bcc0 con 0x7f4b040c45f0 2023-06-22T08:03:01.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.253+0000 7f4b095ff700 1 -- 172.21.15.23:0/4195570094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 msgr2=0x7f4b040c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:01.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.253+0000 7f4b095ff700 1 --2- 172.21.15.23:0/4195570094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 0x7f4b040c4a70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009e50 tx=0x7f4afc00b330 comp rx=0 tx=0).stop 2023-06-22T08:03:01.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.253+0000 7f4b095ff700 1 -- 172.21.15.23:0/4195570094 shutdown_connections 2023-06-22T08:03:01.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.253+0000 7f4b095ff700 1 --2- 172.21.15.23:0/4195570094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 0x7f4b040c4a70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:01.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.253+0000 7f4b095ff700 1 --2- 172.21.15.23:0/4195570094 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 0x7f4b040c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:01.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.253+0000 7f4b095ff700 1 -- 172.21.15.23:0/4195570094 >> 172.21.15.23:0/4195570094 conn(0x7f4b0406ae30 msgr2=0x7f4b0406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:01.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.253+0000 7f4b095ff700 1 -- 172.21.15.23:0/4195570094 shutdown_connections 2023-06-22T08:03:01.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.253+0000 7f4b095ff700 1 -- 172.21.15.23:0/4195570094 wait complete. 2023-06-22T08:03:01.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.255+0000 7f4b095ff700 1 Processor -- start 2023-06-22T08:03:01.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.255+0000 7f4b095ff700 1 -- start start 2023-06-22T08:03:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b095ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 0x7f4b041f14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b095ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 0x7f4b04154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b095ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b04159330 0x7f4b04155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b095ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b04154770 con 0x7f4b040c45f0 2023-06-22T08:03:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b095ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040b4370 con 0x7f4b04159330 2023-06-22T08:03:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b095ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b041f1d80 con 0x7f4b04152d70 2023-06-22T08:03:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b02ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 0x7f4b041f14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b02ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 0x7f4b041f14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48492/0 (socket says 172.21.15.23:48492) 2023-06-22T08:03:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b02ffd700 1 -- 172.21.15.23:0/2616550315 learned_addr learned my addr 172.21.15.23:0/2616550315 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b027fc700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 0x7f4b04154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b037fe700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b04159330 0x7f4b04155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b02ffd700 1 -- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 msgr2=0x7f4b04154c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b02ffd700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 0x7f4b04154c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b02ffd700 1 -- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b04159330 msgr2=0x7f4b04155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b02ffd700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b04159330 0x7f4b04155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b02ffd700 1 -- 172.21.15.23:0/2616550315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af40094a0 con 0x7f4b040c45f0 2023-06-22T08:03:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.256+0000 7f4b02ffd700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 0x7f4b041f14c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4af400ebe0 tx=0x7f4af4009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.257+0000 7f4aeb7fe700 1 -- 172.21.15.23:0/2616550315 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af400c910 con 0x7f4b040c45f0 2023-06-22T08:03:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.257+0000 7f4aeb7fe700 1 -- 172.21.15.23:0/2616550315 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af400ca70 con 0x7f4b040c45f0 2023-06-22T08:03:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.257+0000 7f4aeb7fe700 1 -- 172.21.15.23:0/2616550315 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af400cde0 con 0x7f4b040c45f0 2023-06-22T08:03:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.257+0000 7f4b095ff700 1 -- 172.21.15.23:0/2616550315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4afc009b30 con 0x7f4b040c45f0 2023-06-22T08:03:01.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.257+0000 7f4b095ff700 1 -- 172.21.15.23:0/2616550315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b040b45c0 con 0x7f4b040c45f0 2023-06-22T08:03:01.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.258+0000 7f4b095ff700 1 -- 172.21.15.23:0/2616550315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0404ea20 con 0x7f4b040c45f0 2023-06-22T08:03:01.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.259+0000 7f4aeb7fe700 1 -- 172.21.15.23:0/2616550315 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4af4013470 con 0x7f4b040c45f0 2023-06-22T08:03:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.260+0000 7f4aeb7fe700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4af006ace0 0x7f4af006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.260+0000 7f4aeb7fe700 1 -- 172.21.15.23:0/2616550315 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6869+0+0 (secure 0 0 0) 0x7f4af4089a20 con 0x7f4b040c45f0 2023-06-22T08:03:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.260+0000 7f4b027fc700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4af006ace0 0x7f4af006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.261+0000 7f4b027fc700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4af006ace0 0x7f4af006d190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4b040cf870 tx=0x7f4af8007c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:01.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.265+0000 7f4aeb7fe700 1 -- 172.21.15.23:0/2616550315 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4af4054700 con 0x7f4b040c45f0 2023-06-22T08:03:01.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.519+0000 7f4b095ff700 1 -- 172.21.15.23:0/2616550315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b0406ac10 con 0x7f4b040c45f0 2023-06-22T08:03:01.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.520+0000 7f4aeb7fe700 1 -- 172.21.15.23:0/2616550315 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6254 (secure 0 0 0) 0x7f4af4057d20 con 0x7f4b040c45f0 2023-06-22T08:03:01.523 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:01.523 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:02:51.659992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:01.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 -- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4af006ace0 msgr2=0x7f4af006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:01.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4af006ace0 0x7f4af006d190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4b040cf870 tx=0x7f4af8007c90 comp rx=0 tx=0).stop 2023-06-22T08:03:01.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 -- 172.21.15.23:0/2616550315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 msgr2=0x7f4b041f14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:01.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 0x7f4b041f14c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4af400ebe0 tx=0x7f4af4009ec0 comp rx=0 tx=0).stop 2023-06-22T08:03:01.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 -- 172.21.15.23:0/2616550315 shutdown_connections 2023-06-22T08:03:01.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4af006ace0 0x7f4af006d190 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:01.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b04159330 0x7f4b04155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:01.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b040c45f0 0x7f4b041f14c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:01.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 --2- 172.21.15.23:0/2616550315 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4b04152d70 0x7f4b04154c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:01.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 -- 172.21.15.23:0/2616550315 >> 172.21.15.23:0/2616550315 conn(0x7f4b0406ae30 msgr2=0x7f4b0406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:01.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 -- 172.21.15.23:0/2616550315 shutdown_connections 2023-06-22T08:03:01.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:01.522+0000 7f4b095ff700 1 -- 172.21.15.23:0/2616550315 wait complete. 2023-06-22T08:03:01.528 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 26 2023-06-22T08:03:01.544 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:03:03.545 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:03.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.753+0000 7f0a22bff700 1 Processor -- start 2023-06-22T08:03:03.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.753+0000 7f0a22bff700 1 -- start start 2023-06-22T08:03:03.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.754+0000 7f0a22bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0a1c152ce0 0x7f0a1c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.754+0000 7f0a22bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a1c0c9fd0 0x7f0a1c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.754+0000 7f0a22bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1c15c370 con 0x7f0a1c158880 2023-06-22T08:03:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.754+0000 7f0a22bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a1c1530b0 con 0x7f0a1c152ce0 2023-06-22T08:03:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.754+0000 7f0a22bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a1c0cf6f0 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.755+0000 7f0a2099b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0a1c152ce0 0x7f0a1c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:03.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.755+0000 7f0a2119c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f0a1c158880 0x7f0a1c158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:37724/0 (socket says 172.21.15.23:37724) 2023-06-22T08:03:03.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.755+0000 7f0a2119c700 1 -- 172.21.15.23:0/1442169244 learned_addr learned my addr 172.21.15.23:0/1442169244 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:03:03.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.755+0000 7f0a1bfff700 1 --2- 172.21.15.23:0/1442169244 >> v2:172.21.15.67:3300/0 conn(0x7f0a1c0c9fd0 0x7f0a1c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:03.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.755+0000 7f0a1b7fe700 1 -- 172.21.15.23:0/1442169244 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180674899 0 0) 0x7f0a1c15c370 con 0x7f0a1c158880 2023-06-22T08:03:03.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.755+0000 7f0a2099b700 1 -- 172.21.15.23:0/1442169244 >> v2:172.21.15.67:3300/0 conn(0x7f0a1c0c9fd0 msgr2=0x7f0a1c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:03.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.755+0000 7f0a2099b700 1 --2- 172.21.15.23:0/1442169244 >> v2:172.21.15.67:3300/0 conn(0x7f0a1c0c9fd0 0x7f0a1c15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:03.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.755+0000 7f0a2099b700 1 -- 172.21.15.23:0/1442169244 >> v1:172.21.15.23:6789/0 conn(0x7f0a1c158880 legacy=0x7f0a1c158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:03.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.755+0000 7f0a2099b700 1 -- 172.21.15.23:0/1442169244 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c15bda0 con 0x7f0a1c152ce0 2023-06-22T08:03:03.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.755+0000 7f0a2099b700 1 --2- 172.21.15.23:0/1442169244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c152ce0 0x7f0a1c0cf1b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c0cc380 tx=0x7f0a10009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523874d9a81e6e4e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:03.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a1b7fe700 1 -- 172.21.15.23:0/1442169244 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10011830 con 0x7f0a1c152ce0 2023-06-22T08:03:03.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a1b7fe700 1 -- 172.21.15.23:0/1442169244 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a10011990 con 0x7f0a1c152ce0 2023-06-22T08:03:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a1b7fe700 1 -- 172.21.15.23:0/1442169244 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10011d20 con 0x7f0a1c152ce0 2023-06-22T08:03:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a22bff700 1 -- 172.21.15.23:0/1442169244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c152ce0 msgr2=0x7f0a1c0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a22bff700 1 --2- 172.21.15.23:0/1442169244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c152ce0 0x7f0a1c0cf1b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c0cc380 tx=0x7f0a10009b20 comp rx=0 tx=0).stop 2023-06-22T08:03:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a22bff700 1 -- 172.21.15.23:0/1442169244 shutdown_connections 2023-06-22T08:03:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a22bff700 1 --2- 172.21.15.23:0/1442169244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c152ce0 0x7f0a1c0cf1b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a22bff700 1 --2- 172.21.15.23:0/1442169244 >> v2:172.21.15.67:3300/0 conn(0x7f0a1c0c9fd0 0x7f0a1c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a22bff700 1 -- 172.21.15.23:0/1442169244 >> 172.21.15.23:0/1442169244 conn(0x7f0a1c06ae30 msgr2=0x7f0a1c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a22bff700 1 -- 172.21.15.23:0/1442169244 shutdown_connections 2023-06-22T08:03:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.756+0000 7f0a22bff700 1 -- 172.21.15.23:0/1442169244 wait complete. 2023-06-22T08:03:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.758+0000 7f0a22bff700 1 Processor -- start 2023-06-22T08:03:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.758+0000 7f0a22bff700 1 -- start start 2023-06-22T08:03:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.759+0000 7f0a22bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c0c9fd0 0x7f0a1c0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.759+0000 7f0a22bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a1c152ce0 0x7f0a1c1eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.759+0000 7f0a22bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a1c158880 0x7f0a1c15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.759+0000 7f0a22bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c0ccd80 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.759+0000 7f0a22bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c0cd6e0 con 0x7f0a1c158880 2023-06-22T08:03:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.759+0000 7f0a22bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c0cc730 con 0x7f0a1c152ce0 2023-06-22T08:03:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.759+0000 7f0a2099b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c0c9fd0 0x7f0a1c0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.759+0000 7f0a2099b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c0c9fd0 0x7f0a1c0cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48520/0 (socket says 172.21.15.23:48520) 2023-06-22T08:03:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.759+0000 7f0a2099b700 1 -- 172.21.15.23:0/2461525205 learned_addr learned my addr 172.21.15.23:0/2461525205 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.759+0000 7f0a2119c700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a1c158880 0x7f0a1c15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a1bfff700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a1c152ce0 0x7f0a1c1eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a2099b700 1 -- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a1c152ce0 msgr2=0x7f0a1c1eeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a2099b700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a1c152ce0 0x7f0a1c1eeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a2099b700 1 -- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a1c158880 msgr2=0x7f0a1c15b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a2099b700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a1c158880 0x7f0a1c15b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a2099b700 1 -- 172.21.15.23:0/2461525205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a100094a0 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a2099b700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c0c9fd0 0x7f0a1c0cd1a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009e70 tx=0x7f0a10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a197fa700 1 -- 172.21.15.23:0/2461525205 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a100045d0 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a197fa700 1 -- 172.21.15.23:0/2461525205 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1001f070 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a197fa700 1 -- 172.21.15.23:0/2461525205 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a100038a0 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a22bff700 1 -- 172.21.15.23:0/2461525205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1c1ef080 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.760+0000 7f0a22bff700 1 -- 172.21.15.23:0/2461525205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1c0cdee0 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.762+0000 7f0a22bff700 1 -- 172.21.15.23:0/2461525205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1c0ccec0 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.762+0000 7f0a197fa700 1 -- 172.21.15.23:0/2461525205 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0a10003a00 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.764+0000 7f0a197fa700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a0006ad10 0x7f0a0006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.764+0000 7f0a197fa700 1 -- 172.21.15.23:0/2461525205 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6869+0+0 (secure 0 0 0) 0x7f0a10021070 con 0x7f0a1c0c9fd0 2023-06-22T08:03:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.764+0000 7f0a1bfff700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a0006ad10 0x7f0a0006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.765+0000 7f0a1bfff700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a0006ad10 0x7f0a0006d1c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c0cab70 tx=0x7f0a08008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:03.768+0000 7f0a197fa700 1 -- 172.21.15.23:0/2461525205 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0a1c0ccec0 con 0x7f0a1c0c9fd0 2023-06-22T08:03:04.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.020+0000 7f0a22bff700 1 -- 172.21.15.23:0/2461525205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a1c0ccec0 con 0x7f0a1c0c9fd0 2023-06-22T08:03:04.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.021+0000 7f0a197fa700 1 -- 172.21.15.23:0/2461525205 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6254 (secure 0 0 0) 0x7f0a1c0ccec0 con 0x7f0a1c0c9fd0 2023-06-22T08:03:04.024 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:04.024 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:02:51.659992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:04.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.023+0000 7f0a22bff700 1 -- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a0006ad10 msgr2=0x7f0a0006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:04.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.023+0000 7f0a22bff700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a0006ad10 0x7f0a0006d1c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c0cab70 tx=0x7f0a08008040 comp rx=0 tx=0).stop 2023-06-22T08:03:04.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.023+0000 7f0a22bff700 1 -- 172.21.15.23:0/2461525205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c0c9fd0 msgr2=0x7f0a1c0cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:04.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.023+0000 7f0a22bff700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c0c9fd0 0x7f0a1c0cd1a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009e70 tx=0x7f0a10004750 comp rx=0 tx=0).stop 2023-06-22T08:03:04.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.023+0000 7f0a22bff700 1 -- 172.21.15.23:0/2461525205 shutdown_connections 2023-06-22T08:03:04.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.023+0000 7f0a22bff700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a0006ad10 0x7f0a0006d1c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:04.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.023+0000 7f0a22bff700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a1c158880 0x7f0a1c15b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:04.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.023+0000 7f0a22bff700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a1c0c9fd0 0x7f0a1c0cd1a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:04.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.023+0000 7f0a22bff700 1 --2- 172.21.15.23:0/2461525205 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a1c152ce0 0x7f0a1c1eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:04.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.023+0000 7f0a22bff700 1 -- 172.21.15.23:0/2461525205 >> 172.21.15.23:0/2461525205 conn(0x7f0a1c06ae30 msgr2=0x7f0a1c0c4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:04.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.024+0000 7f0a22bff700 1 -- 172.21.15.23:0/2461525205 shutdown_connections 2023-06-22T08:03:04.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:04.024+0000 7f0a22bff700 1 -- 172.21.15.23:0/2461525205 wait complete. 2023-06-22T08:03:04.029 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 26 2023-06-22T08:03:04.044 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:03:06.045 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:06.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.242+0000 7fa80d5ff700 1 Processor -- start 2023-06-22T08:03:06.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.242+0000 7fa80d5ff700 1 -- start start 2023-06-22T08:03:06.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.244+0000 7fa80d5ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa80815cb00 0x7fa8080c9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:06.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.244+0000 7fa80d5ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8080c9540 con 0x7fa808071850 2023-06-22T08:03:06.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.244+0000 7fa80d5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa808158c10 con 0x7fa808071c20 2023-06-22T08:03:06.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.244+0000 7fa80d5ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa808158ef0 con 0x7fa80815cb00 2023-06-22T08:03:06.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.244+0000 7fa806ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa808071850 0x7fa80815c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:42790/0 (socket says 172.21.15.23:42790) 2023-06-22T08:03:06.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.244+0000 7fa806ffd700 1 -- 172.21.15.23:0/550951281 learned_addr learned my addr 172.21.15.23:0/550951281 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:03:06.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.244+0000 7fa8077fe700 1 --2- 172.21.15.23:0/550951281 >> v2:172.21.15.23:3300/0 conn(0x7fa80815cb00 0x7fa8080c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:06.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.244+0000 7fa8077fe700 1 -- 172.21.15.23:0/550951281 >> v1:172.21.15.67:6789/0 conn(0x7fa808071c20 legacy=0x7fa808152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:06.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.244+0000 7fa8077fe700 1 -- 172.21.15.23:0/550951281 >> v1:172.21.15.23:6789/0 conn(0x7fa808071850 legacy=0x7fa80815c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:06.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.245+0000 7fa8077fe700 1 -- 172.21.15.23:0/550951281 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8081591d0 con 0x7fa80815cb00 2023-06-22T08:03:06.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.245+0000 7fa805ffb700 1 -- 172.21.15.23:0/550951281 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084843227 0 0) 0x7fa8080c9540 con 0x7fa808071850 2023-06-22T08:03:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.245+0000 7fa8077fe700 1 --2- 172.21.15.23:0/550951281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 0x7fa8080c9830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa800009d90 tx=0x7fa80000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea747737deafe0d1 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.245+0000 7fa805ffb700 1 -- 172.21.15.23:0/550951281 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80000e070 con 0x7fa80815cb00 2023-06-22T08:03:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.245+0000 7fa805ffb700 1 -- 172.21.15.23:0/550951281 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80000bbd0 con 0x7fa80815cb00 2023-06-22T08:03:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.245+0000 7fa805ffb700 1 -- 172.21.15.23:0/550951281 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800012430 con 0x7fa80815cb00 2023-06-22T08:03:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.246+0000 7fa80d5ff700 1 -- 172.21.15.23:0/550951281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 msgr2=0x7fa8080c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.246+0000 7fa80d5ff700 1 --2- 172.21.15.23:0/550951281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 0x7fa8080c9830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa800009d90 tx=0x7fa80000b330 comp rx=0 tx=0).stop 2023-06-22T08:03:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.246+0000 7fa80d5ff700 1 -- 172.21.15.23:0/550951281 shutdown_connections 2023-06-22T08:03:06.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.246+0000 7fa80d5ff700 1 --2- 172.21.15.23:0/550951281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 0x7fa8080c9830 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:06.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.246+0000 7fa80d5ff700 1 -- 172.21.15.23:0/550951281 >> 172.21.15.23:0/550951281 conn(0x7fa80806b890 msgr2=0x7fa80806bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:06.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.246+0000 7fa80d5ff700 1 -- 172.21.15.23:0/550951281 shutdown_connections 2023-06-22T08:03:06.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.246+0000 7fa80d5ff700 1 -- 172.21.15.23:0/550951281 wait complete. 2023-06-22T08:03:06.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.247+0000 7fa80d5ff700 1 Processor -- start 2023-06-22T08:03:06.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.248+0000 7fa80d5ff700 1 -- start start 2023-06-22T08:03:06.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.248+0000 7fa80d5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808071850 0x7fa8081540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:06.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.248+0000 7fa80d5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa808071c20 0x7fa8080b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:06.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.248+0000 7fa80d5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 0x7fa8081f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.248+0000 7fa80d5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa808153420 con 0x7fa80815cb00 2023-06-22T08:03:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.248+0000 7fa80d5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8081ef040 con 0x7fa808071850 2023-06-22T08:03:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.248+0000 7fa80d5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8081555f0 con 0x7fa808071c20 2023-06-22T08:03:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa8077fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 0x7fa8081f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa8077fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 0x7fa8081f1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35714/0 (socket says 172.21.15.23:35714) 2023-06-22T08:03:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa8067fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa808071c20 0x7fa8080b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa8077fe700 1 -- 172.21.15.23:0/3997608001 learned_addr learned my addr 172.21.15.23:0/3997608001 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa806ffd700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808071850 0x7fa8081540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa8077fe700 1 -- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa808071c20 msgr2=0x7fa8080b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa8077fe700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa808071c20 0x7fa8080b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa8077fe700 1 -- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808071850 msgr2=0x7fa8081540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa8077fe700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808071850 0x7fa8081540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa8077fe700 1 -- 172.21.15.23:0/3997608001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa800009a70 con 0x7fa80815cb00 2023-06-22T08:03:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.249+0000 7fa8067fc700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa808071c20 0x7fa8080b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:03:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.250+0000 7fa8077fe700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 0x7fa8081f1a90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa800013000 tx=0x7fa800004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.250+0000 7fa7ef7fe700 1 -- 172.21.15.23:0/3997608001 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80001b810 con 0x7fa80815cb00 2023-06-22T08:03:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.250+0000 7fa7ef7fe700 1 -- 172.21.15.23:0/3997608001 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa800020070 con 0x7fa80815cb00 2023-06-22T08:03:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.250+0000 7fa80d5ff700 1 -- 172.21.15.23:0/3997608001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa808154620 con 0x7fa80815cb00 2023-06-22T08:03:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.250+0000 7fa7ef7fe700 1 -- 172.21.15.23:0/3997608001 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80001bbd0 con 0x7fa80815cb00 2023-06-22T08:03:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.250+0000 7fa80d5ff700 1 -- 172.21.15.23:0/3997608001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa808153600 con 0x7fa80815cb00 2023-06-22T08:03:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.251+0000 7fa80d5ff700 1 -- 172.21.15.23:0/3997608001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80804ea20 con 0x7fa80815cb00 2023-06-22T08:03:06.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.252+0000 7fa7ef7fe700 1 -- 172.21.15.23:0/3997608001 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa80001bd30 con 0x7fa80815cb00 2023-06-22T08:03:06.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.253+0000 7fa7ef7fe700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa7f406ace0 0x7fa7f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:06.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.253+0000 7fa7ef7fe700 1 -- 172.21.15.23:0/3997608001 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6841+0+0 (secure 0 0 0) 0x7fa800022070 con 0x7fa80815cb00 2023-06-22T08:03:06.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.253+0000 7fa806ffd700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa7f406ace0 0x7fa7f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:06.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.254+0000 7fa806ffd700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa7f406ace0 0x7fa7f406d190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8007b10 tx=0x7fa7f8007a60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:06.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.257+0000 7fa7ef7fe700 1 -- 172.21.15.23:0/3997608001 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa800056d20 con 0x7fa80815cb00 2023-06-22T08:03:06.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:03:06.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:03:06.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.515+0000 7fa80d5ff700 1 -- 172.21.15.23:0/3997608001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8080c3c00 con 0x7fa80815cb00 2023-06-22T08:03:06.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.516+0000 7fa7ef7fe700 1 -- 172.21.15.23:0/3997608001 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6254 (secure 0 0 0) 0x7fa80005a340 con 0x7fa80815cb00 2023-06-22T08:03:06.519 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:06.520 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:02:51.659992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:06.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.518+0000 7fa80d5ff700 1 -- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa7f406ace0 msgr2=0x7fa7f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:06.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.518+0000 7fa80d5ff700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa7f406ace0 0x7fa7f406d190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8007b10 tx=0x7fa7f8007a60 comp rx=0 tx=0).stop 2023-06-22T08:03:06.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.519+0000 7fa80d5ff700 1 -- 172.21.15.23:0/3997608001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 msgr2=0x7fa8081f1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:06.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.519+0000 7fa80d5ff700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 0x7fa8081f1a90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa800013000 tx=0x7fa800004ad0 comp rx=0 tx=0).stop 2023-06-22T08:03:06.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.519+0000 7fa80d5ff700 1 -- 172.21.15.23:0/3997608001 shutdown_connections 2023-06-22T08:03:06.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.519+0000 7fa80d5ff700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa80815cb00 0x7fa8081f1a90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:06.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.519+0000 7fa80d5ff700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa7f406ace0 0x7fa7f406d190 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:06.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.519+0000 7fa80d5ff700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa808071850 0x7fa8081540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:06.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.519+0000 7fa80d5ff700 1 --2- 172.21.15.23:0/3997608001 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa808071c20 0x7fa8080b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:06.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.519+0000 7fa80d5ff700 1 -- 172.21.15.23:0/3997608001 >> 172.21.15.23:0/3997608001 conn(0x7fa80806b890 msgr2=0x7fa80806bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:06.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.519+0000 7fa80d5ff700 1 -- 172.21.15.23:0/3997608001 shutdown_connections 2023-06-22T08:03:06.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:06.519+0000 7fa80d5ff700 1 -- 172.21.15.23:0/3997608001 wait complete. 2023-06-22T08:03:06.524 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 26 2023-06-22T08:03:06.539 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:03:08.541 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:08.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.747+0000 7fea54dff700 1 Processor -- start 2023-06-22T08:03:08.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.748+0000 7fea54dff700 1 -- start start 2023-06-22T08:03:08.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea54dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fea50152ce0 0x7fea500cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:08.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea54dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fea500c9fd0 0x7fea5015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:08.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea54dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea50158880 0x7fea50158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:08.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea54dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea501530b0 con 0x7fea500c9fd0 2023-06-22T08:03:08.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea54dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea500cf6f0 con 0x7fea50152ce0 2023-06-22T08:03:08.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea54dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5015c370 con 0x7fea50158880 2023-06-22T08:03:08.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea47fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fea500c9fd0 0x7fea5015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:08.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea47fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fea500c9fd0 0x7fea5015c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35720/0 (socket says 172.21.15.23:35720) 2023-06-22T08:03:08.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea47fff700 1 -- 172.21.15.23:0/1161765121 learned_addr learned my addr 172.21.15.23:0/1161765121 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:08.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea4ed9d700 1 --2- 172.21.15.23:0/1161765121 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea50158880 0x7fea50158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:08.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.749+0000 7fea4e59c700 1 --2- 172.21.15.23:0/1161765121 >> v2:172.21.15.67:3300/0 conn(0x7fea50152ce0 0x7fea500cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:08.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.750+0000 7fea47fff700 1 -- 172.21.15.23:0/1161765121 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea50158880 msgr2=0x7fea50158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:08.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.750+0000 7fea47fff700 1 --2- 172.21.15.23:0/1161765121 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea50158880 0x7fea50158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:08.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.750+0000 7fea47fff700 1 -- 172.21.15.23:0/1161765121 >> v2:172.21.15.67:3300/0 conn(0x7fea50152ce0 msgr2=0x7fea500cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:08.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.750+0000 7fea47fff700 1 --2- 172.21.15.23:0/1161765121 >> v2:172.21.15.67:3300/0 conn(0x7fea50152ce0 0x7fea500cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:08.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.750+0000 7fea47fff700 1 -- 172.21.15.23:0/1161765121 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5015bda0 con 0x7fea500c9fd0 2023-06-22T08:03:08.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.750+0000 7fea47fff700 1 --2- 172.21.15.23:0/1161765121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea500c9fd0 0x7fea5015c800 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fea5008ba10 tx=0x7fea38009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6177de70837b8a3e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:08.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.750+0000 7fea4dd9b700 1 -- 172.21.15.23:0/1161765121 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38011830 con 0x7fea500c9fd0 2023-06-22T08:03:08.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.750+0000 7fea4dd9b700 1 -- 172.21.15.23:0/1161765121 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea38011990 con 0x7fea500c9fd0 2023-06-22T08:03:08.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.750+0000 7fea4dd9b700 1 -- 172.21.15.23:0/1161765121 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38011d20 con 0x7fea500c9fd0 2023-06-22T08:03:08.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.751+0000 7fea54dff700 1 -- 172.21.15.23:0/1161765121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea500c9fd0 msgr2=0x7fea5015c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:08.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.751+0000 7fea54dff700 1 --2- 172.21.15.23:0/1161765121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea500c9fd0 0x7fea5015c800 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fea5008ba10 tx=0x7fea38009bd0 comp rx=0 tx=0).stop 2023-06-22T08:03:08.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.751+0000 7fea54dff700 1 -- 172.21.15.23:0/1161765121 shutdown_connections 2023-06-22T08:03:08.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.751+0000 7fea54dff700 1 --2- 172.21.15.23:0/1161765121 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea50158880 0x7fea50158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:08.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.751+0000 7fea54dff700 1 --2- 172.21.15.23:0/1161765121 >> v2:172.21.15.67:3300/0 conn(0x7fea50152ce0 0x7fea500cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:08.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.751+0000 7fea54dff700 1 --2- 172.21.15.23:0/1161765121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea500c9fd0 0x7fea5015c800 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:08.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.751+0000 7fea54dff700 1 -- 172.21.15.23:0/1161765121 >> 172.21.15.23:0/1161765121 conn(0x7fea5006ae30 msgr2=0x7fea5006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:08.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.751+0000 7fea54dff700 1 -- 172.21.15.23:0/1161765121 shutdown_connections 2023-06-22T08:03:08.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.751+0000 7fea54dff700 1 -- 172.21.15.23:0/1161765121 wait complete. 2023-06-22T08:03:08.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.753+0000 7fea54dff700 1 Processor -- start 2023-06-22T08:03:08.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.753+0000 7fea54dff700 1 -- start start 2023-06-22T08:03:08.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea54dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea500c9fd0 0x7fea501491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:08.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea54dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea50152ce0 0x7fea501ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:08.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea54dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea50158880 0x7fea501f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:08.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea54dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea501ee1c0 con 0x7fea50152ce0 2023-06-22T08:03:08.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea54dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea500cde30 con 0x7fea50158880 2023-06-22T08:03:08.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea54dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea500b2500 con 0x7fea500c9fd0 2023-06-22T08:03:08.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea4e59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea500c9fd0 0x7fea501491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:08.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea47fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea50152ce0 0x7fea501ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:08.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea4e59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea500c9fd0 0x7fea501491b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35962/0 (socket says 172.21.15.23:35962) 2023-06-22T08:03:08.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea4e59c700 1 -- 172.21.15.23:0/1804704630 learned_addr learned my addr 172.21.15.23:0/1804704630 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:08.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea4ed9d700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea50158880 0x7fea501f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:08.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea4e59c700 1 -- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea50158880 msgr2=0x7fea501f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:08.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea4e59c700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea50158880 0x7fea501f1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:08.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea4e59c700 1 -- 172.21.15.23:0/1804704630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea50152ce0 msgr2=0x7fea501ee9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:08.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea4e59c700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea50152ce0 0x7fea501ee9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:08.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.754+0000 7fea4e59c700 1 -- 172.21.15.23:0/1804704630 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38009550 con 0x7fea500c9fd0 2023-06-22T08:03:08.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.755+0000 7fea47fff700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea50152ce0 0x7fea501ee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:03:08.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.755+0000 7fea4e59c700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea500c9fd0 0x7fea501491b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fea4000e9b0 tx=0x7fea4000ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:08.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.755+0000 7fea46ffd700 1 -- 172.21.15.23:0/1804704630 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea40009710 con 0x7fea500c9fd0 2023-06-22T08:03:08.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.755+0000 7fea46ffd700 1 -- 172.21.15.23:0/1804704630 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4000ee50 con 0x7fea500c9fd0 2023-06-22T08:03:08.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.755+0000 7fea46ffd700 1 -- 172.21.15.23:0/1804704630 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4000ca50 con 0x7fea500c9fd0 2023-06-22T08:03:08.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.755+0000 7fea54dff700 1 -- 172.21.15.23:0/1804704630 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea500b27e0 con 0x7fea500c9fd0 2023-06-22T08:03:08.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.755+0000 7fea54dff700 1 -- 172.21.15.23:0/1804704630 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea501ee480 con 0x7fea500c9fd0 2023-06-22T08:03:08.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.756+0000 7fea54dff700 1 -- 172.21.15.23:0/1804704630 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea501ee7f0 con 0x7fea500c9fd0 2023-06-22T08:03:08.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.757+0000 7fea46ffd700 1 -- 172.21.15.23:0/1804704630 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fea4000cbb0 con 0x7fea500c9fd0 2023-06-22T08:03:08.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.758+0000 7fea46ffd700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fea3006ace0 0x7fea3006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:08.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.758+0000 7fea46ffd700 1 -- 172.21.15.23:0/1804704630 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6841+0+0 (secure 0 0 0) 0x7fea40089790 con 0x7fea500c9fd0 2023-06-22T08:03:08.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.758+0000 7fea47fff700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fea3006ace0 0x7fea3006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:08.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.759+0000 7fea47fff700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fea3006ace0 0x7fea3006d190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fea38009c10 tx=0x7fea38008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:08.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:08.763+0000 7fea46ffd700 1 -- 172.21.15.23:0/1804704630 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fea40057fb0 con 0x7fea500c9fd0 2023-06-22T08:03:09.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.015+0000 7fea54dff700 1 -- 172.21.15.23:0/1804704630 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea5004ea20 con 0x7fea500c9fd0 2023-06-22T08:03:09.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.016+0000 7fea46ffd700 1 -- 172.21.15.23:0/1804704630 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6254 (secure 0 0 0) 0x7fea4000ce60 con 0x7fea500c9fd0 2023-06-22T08:03:09.019 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:09.020 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:02:51.659992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4268":{"gid":4268,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.23:6837/1762110734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1762110734},{"type":"v1","addr":"172.21.15.23:6837","nonce":1762110734}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:09.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 -- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fea3006ace0 msgr2=0x7fea3006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:09.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fea3006ace0 0x7fea3006d190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fea38009c10 tx=0x7fea38008030 comp rx=0 tx=0).stop 2023-06-22T08:03:09.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 -- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea500c9fd0 msgr2=0x7fea501491b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:09.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea500c9fd0 0x7fea501491b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fea4000e9b0 tx=0x7fea4000ecc0 comp rx=0 tx=0).stop 2023-06-22T08:03:09.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 -- 172.21.15.23:0/1804704630 shutdown_connections 2023-06-22T08:03:09.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fea3006ace0 0x7fea3006d190 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:09.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea50158880 0x7fea501f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:09.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea500c9fd0 0x7fea501491b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:09.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 --2- 172.21.15.23:0/1804704630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea50152ce0 0x7fea501ee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:09.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 -- 172.21.15.23:0/1804704630 >> 172.21.15.23:0/1804704630 conn(0x7fea5006ae30 msgr2=0x7fea500cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:09.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 -- 172.21.15.23:0/1804704630 shutdown_connections 2023-06-22T08:03:09.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:09.019+0000 7fea54dff700 1 -- 172.21.15.23:0/1804704630 wait complete. 2023-06-22T08:03:09.025 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 26 2023-06-22T08:03:09.040 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:03:11.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.391+0000 7fac697ff700 1 Processor -- start 2023-06-22T08:03:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.391+0000 7fac697ff700 1 -- start start 2023-06-22T08:03:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.393+0000 7fac697ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fac640c8bd0 0x7fac640aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.393+0000 7fac697ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 0x7fac640ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.393+0000 7fac697ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac640c7400 con 0x7fac64152dd0 2023-06-22T08:03:11.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.393+0000 7fac697ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac640c81c0 con 0x7fac640c8bd0 2023-06-22T08:03:11.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.393+0000 7fac697ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac640c84a0 con 0x7fac640cc640 2023-06-22T08:03:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.393+0000 7fac627fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fac640c8bd0 0x7fac640aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.394+0000 7fac627fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fac640c8bd0 0x7fac640aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35752/0 (socket says 172.21.15.23:35752) 2023-06-22T08:03:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.394+0000 7fac627fc700 1 -- 172.21.15.23:0/1278961130 learned_addr learned my addr 172.21.15.23:0/1278961130 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.394+0000 7fac637fe700 1 --2- 172.21.15.23:0/1278961130 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 0x7fac640ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.394+0000 7fac61ffb700 1 -- 172.21.15.23:0/1278961130 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652736251 0 0) 0x7fac640c7400 con 0x7fac64152dd0 2023-06-22T08:03:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.394+0000 7fac627fc700 1 -- 172.21.15.23:0/1278961130 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 msgr2=0x7fac640ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.394+0000 7fac627fc700 1 --2- 172.21.15.23:0/1278961130 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 0x7fac640ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.394+0000 7fac627fc700 1 -- 172.21.15.23:0/1278961130 >> v1:172.21.15.67:6789/0 conn(0x7fac64152dd0 legacy=0x7fac6406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.394+0000 7fac627fc700 1 -- 172.21.15.23:0/1278961130 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac640cd370 con 0x7fac640c8bd0 2023-06-22T08:03:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.394+0000 7fac627fc700 1 --2- 172.21.15.23:0/1278961130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac640c8bd0 0x7fac640aedc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fac580098f0 tx=0x7fac58009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170d7a6a0f89f9f0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.395+0000 7fac61ffb700 1 -- 172.21.15.23:0/1278961130 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58011830 con 0x7fac640c8bd0 2023-06-22T08:03:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.395+0000 7fac61ffb700 1 -- 172.21.15.23:0/1278961130 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac58011990 con 0x7fac640c8bd0 2023-06-22T08:03:11.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.395+0000 7fac61ffb700 1 -- 172.21.15.23:0/1278961130 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58011d20 con 0x7fac640c8bd0 2023-06-22T08:03:11.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.395+0000 7fac697ff700 1 -- 172.21.15.23:0/1278961130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac640c8bd0 msgr2=0x7fac640aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:11.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.395+0000 7fac697ff700 1 --2- 172.21.15.23:0/1278961130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac640c8bd0 0x7fac640aedc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fac580098f0 tx=0x7fac58009c00 comp rx=0 tx=0).stop 2023-06-22T08:03:11.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.396+0000 7fac697ff700 1 -- 172.21.15.23:0/1278961130 shutdown_connections 2023-06-22T08:03:11.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.396+0000 7fac697ff700 1 --2- 172.21.15.23:0/1278961130 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 0x7fac640ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.396+0000 7fac697ff700 1 --2- 172.21.15.23:0/1278961130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac640c8bd0 0x7fac640aedc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.396+0000 7fac697ff700 1 -- 172.21.15.23:0/1278961130 >> 172.21.15.23:0/1278961130 conn(0x7fac6406fb70 msgr2=0x7fac6406ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:11.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.396+0000 7fac697ff700 1 -- 172.21.15.23:0/1278961130 shutdown_connections 2023-06-22T08:03:11.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.396+0000 7fac697ff700 1 -- 172.21.15.23:0/1278961130 wait complete. 2023-06-22T08:03:11.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.398+0000 7fac697ff700 1 Processor -- start 2023-06-22T08:03:11.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.399+0000 7fac697ff700 1 -- start start 2023-06-22T08:03:11.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.400+0000 7fac697ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640c8bd0 0x7fac64159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.400+0000 7fac697ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 0x7fac6414a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.400+0000 7fac697ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac64152dd0 0x7fac6414ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.400+0000 7fac697ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6414e690 con 0x7fac64152dd0 2023-06-22T08:03:11.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.400+0000 7fac697ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6414b220 con 0x7fac640c8bd0 2023-06-22T08:03:11.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.400+0000 7fac697ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6415c930 con 0x7fac640cc640 2023-06-22T08:03:11.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.400+0000 7fac627fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 0x7fac6414a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.400+0000 7fac62ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640c8bd0 0x7fac64159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.401+0000 7fac627fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 0x7fac6414a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35978/0 (socket says 172.21.15.23:35978) 2023-06-22T08:03:11.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.401+0000 7fac627fc700 1 -- 172.21.15.23:0/3048001977 learned_addr learned my addr 172.21.15.23:0/3048001977 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:11.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.401+0000 7fac637fe700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac64152dd0 0x7fac6414ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.401+0000 7fac627fc700 1 -- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640c8bd0 msgr2=0x7fac64159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:11.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.401+0000 7fac627fc700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640c8bd0 0x7fac64159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.401+0000 7fac627fc700 1 -- 172.21.15.23:0/3048001977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac64152dd0 msgr2=0x7fac6414ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:11.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.401+0000 7fac627fc700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac64152dd0 0x7fac6414ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.401+0000 7fac627fc700 1 -- 172.21.15.23:0/3048001977 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58009550 con 0x7fac640cc640 2023-06-22T08:03:11.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.401+0000 7fac627fc700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 0x7fac6414a7a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fac58009c40 tx=0x7fac58004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:11.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.401+0000 7fac4b7fe700 1 -- 172.21.15.23:0/3048001977 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58004590 con 0x7fac640cc640 2023-06-22T08:03:11.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.402+0000 7fac4b7fe700 1 -- 172.21.15.23:0/3048001977 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5801e070 con 0x7fac640cc640 2023-06-22T08:03:11.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.402+0000 7fac697ff700 1 -- 172.21.15.23:0/3048001977 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6415c650 con 0x7fac640cc640 2023-06-22T08:03:11.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.402+0000 7fac4b7fe700 1 -- 172.21.15.23:0/3048001977 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac580038c0 con 0x7fac640cc640 2023-06-22T08:03:11.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.402+0000 7fac697ff700 1 -- 172.21.15.23:0/3048001977 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6414e040 con 0x7fac640cc640 2023-06-22T08:03:11.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.403+0000 7fac697ff700 1 -- 172.21.15.23:0/3048001977 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6414e800 con 0x7fac640cc640 2023-06-22T08:03:11.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.404+0000 7fac4b7fe700 1 -- 172.21.15.23:0/3048001977 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fac58003a20 con 0x7fac640cc640 2023-06-22T08:03:11.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.406+0000 7fac4b7fe700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac5006ace0 0x7fac5006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.406+0000 7fac4b7fe700 1 -- 172.21.15.23:0/3048001977 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7fac5808ac10 con 0x7fac640cc640 2023-06-22T08:03:11.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.406+0000 7fac62ffd700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac5006ace0 0x7fac5006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.406+0000 7fac62ffd700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac5006ace0 0x7fac5006d190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fac54007fd0 tx=0x7fac54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:11.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.408+0000 7fac4b7fe700 1 -- 172.21.15.23:0/3048001977 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fac580559e0 con 0x7fac640cc640 2023-06-22T08:03:11.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.699+0000 7fac697ff700 1 -- 172.21.15.23:0/3048001977 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac6404ea20 con 0x7fac640cc640 2023-06-22T08:03:11.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.700+0000 7fac4b7fe700 1 -- 172.21.15.23:0/3048001977 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5499 (secure 0 0 0) 0x7fac58059000 con 0x7fac640cc640 2023-06-22T08:03:11.704 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:11.704 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:09.287753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:11.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 -- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac5006ace0 msgr2=0x7fac5006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac5006ace0 0x7fac5006d190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fac54007fd0 tx=0x7fac54009040 comp rx=0 tx=0).stop 2023-06-22T08:03:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 -- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 msgr2=0x7fac6414a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 0x7fac6414a7a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fac58009c40 tx=0x7fac58004750 comp rx=0 tx=0).stop 2023-06-22T08:03:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 -- 172.21.15.23:0/3048001977 shutdown_connections 2023-06-22T08:03:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac64152dd0 0x7fac6414ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac5006ace0 0x7fac5006d190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac640c8bd0 0x7fac64159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 --2- 172.21.15.23:0/3048001977 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac640cc640 0x7fac6414a7a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 -- 172.21.15.23:0/3048001977 >> 172.21.15.23:0/3048001977 conn(0x7fac6406fb70 msgr2=0x7fac641523d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 -- 172.21.15.23:0/3048001977 shutdown_connections 2023-06-22T08:03:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.703+0000 7fac697ff700 1 -- 172.21.15.23:0/3048001977 wait complete. 2023-06-22T08:03:11.709 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2023-06-22T08:03:11.730 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-06-22T08:03:11.730 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:03:11.731 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:11.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.932+0000 7f66941ff700 1 Processor -- start 2023-06-22T08:03:11.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.932+0000 7f66941ff700 1 -- start start 2023-06-22T08:03:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.933+0000 7f66941ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f668c152e10 0x7f668c069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.933+0000 7f66941ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f668c0c5740 0x7f668c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.933+0000 7f66941ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668c158f60 con 0x7f668c0cb6b0 2023-06-22T08:03:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.933+0000 7f66941ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668c06bd20 con 0x7f668c152e10 2023-06-22T08:03:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.933+0000 7f66941ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668c0cc450 con 0x7f668c0c5740 2023-06-22T08:03:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.934+0000 7f669179a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f668c152e10 0x7f668c069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.934+0000 7f6691f9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f668c0cb6b0 0x7f668c152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:42798/0 (socket says 172.21.15.23:42798) 2023-06-22T08:03:11.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.934+0000 7f6691f9b700 1 -- 172.21.15.23:0/2400885002 learned_addr learned my addr 172.21.15.23:0/2400885002 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:03:11.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.934+0000 7f669279c700 1 --2- 172.21.15.23:0/2400885002 >> v2:172.21.15.67:3300/0 conn(0x7f668c0c5740 0x7f668c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.934+0000 7f669179a700 1 -- 172.21.15.23:0/2400885002 >> v2:172.21.15.67:3300/0 conn(0x7f668c0c5740 msgr2=0x7f668c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:11.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.934+0000 7f669179a700 1 --2- 172.21.15.23:0/2400885002 >> v2:172.21.15.67:3300/0 conn(0x7f668c0c5740 0x7f668c1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.934+0000 7f669179a700 1 -- 172.21.15.23:0/2400885002 >> v1:172.21.15.23:6789/0 conn(0x7f668c0cb6b0 legacy=0x7f668c152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.934+0000 7f669179a700 1 -- 172.21.15.23:0/2400885002 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c158980 con 0x7f668c152e10 2023-06-22T08:03:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.934+0000 7f6690f99700 1 -- 172.21.15.23:0/2400885002 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630235978 0 0) 0x7f668c158f60 con 0x7f668c0cb6b0 2023-06-22T08:03:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.934+0000 7f669179a700 1 --2- 172.21.15.23:0/2400885002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c152e10 0x7f668c069eb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f668c159930 tx=0x7f667c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c635effde5697394 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f6690f99700 1 -- 172.21.15.23:0/2400885002 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c011830 con 0x7f668c152e10 2023-06-22T08:03:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f6690f99700 1 -- 172.21.15.23:0/2400885002 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c011990 con 0x7f668c152e10 2023-06-22T08:03:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f6690f99700 1 -- 172.21.15.23:0/2400885002 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c011d20 con 0x7f668c152e10 2023-06-22T08:03:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f66941ff700 1 -- 172.21.15.23:0/2400885002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c152e10 msgr2=0x7f668c069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:11.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f66941ff700 1 --2- 172.21.15.23:0/2400885002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c152e10 0x7f668c069eb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f668c159930 tx=0x7f667c009b20 comp rx=0 tx=0).stop 2023-06-22T08:03:11.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f66941ff700 1 -- 172.21.15.23:0/2400885002 shutdown_connections 2023-06-22T08:03:11.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f66941ff700 1 --2- 172.21.15.23:0/2400885002 >> v2:172.21.15.67:3300/0 conn(0x7f668c0c5740 0x7f668c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f66941ff700 1 --2- 172.21.15.23:0/2400885002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c152e10 0x7f668c069eb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f66941ff700 1 -- 172.21.15.23:0/2400885002 >> 172.21.15.23:0/2400885002 conn(0x7f668c070d80 msgr2=0x7f668c071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:11.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f66941ff700 1 -- 172.21.15.23:0/2400885002 shutdown_connections 2023-06-22T08:03:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.935+0000 7f66941ff700 1 -- 172.21.15.23:0/2400885002 wait complete. 2023-06-22T08:03:11.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.937+0000 7f66941ff700 1 Processor -- start 2023-06-22T08:03:11.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.937+0000 7f66941ff700 1 -- start start 2023-06-22T08:03:11.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.938+0000 7f66941ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c0cb6b0 0x7f668c0ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.938+0000 7f66941ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f668c152e10 0x7f668c1576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.938+0000 7f66941ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c1565d0 0x7f668c0c4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.938+0000 7f66941ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c1f2180 con 0x7f668c0cb6b0 2023-06-22T08:03:11.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.938+0000 7f66941ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c156f50 con 0x7f668c1565d0 2023-06-22T08:03:11.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.938+0000 7f66941ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668c157230 con 0x7f668c152e10 2023-06-22T08:03:11.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f6691f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c0cb6b0 0x7f668c0ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f6691f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c0cb6b0 0x7f668c0ca6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35784/0 (socket says 172.21.15.23:35784) 2023-06-22T08:03:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f6691f9b700 1 -- 172.21.15.23:0/2625550079 learned_addr learned my addr 172.21.15.23:0/2625550079 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f669179a700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f668c152e10 0x7f668c1576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f669279c700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c1565d0 0x7f668c0c4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f6691f9b700 1 -- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f668c152e10 msgr2=0x7f668c1576c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f6691f9b700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f668c152e10 0x7f668c1576c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f6691f9b700 1 -- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c1565d0 msgr2=0x7f668c0c4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f6691f9b700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c1565d0 0x7f668c0c4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:11.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f6691f9b700 1 -- 172.21.15.23:0/2625550079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c0094a0 con 0x7f668c0cb6b0 2023-06-22T08:03:11.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f669179a700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f668c152e10 0x7f668c1576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:03:11.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.939+0000 7f6691f9b700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c0cb6b0 0x7f668c0ca6b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f668800c750 tx=0x7f668800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:11.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.940+0000 7f66827fc700 1 -- 172.21.15.23:0/2625550079 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6688004c80 con 0x7f668c0cb6b0 2023-06-22T08:03:11.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.940+0000 7f66827fc700 1 -- 172.21.15.23:0/2625550079 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6688004e00 con 0x7f668c0cb6b0 2023-06-22T08:03:11.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.940+0000 7f66941ff700 1 -- 172.21.15.23:0/2625550079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668c1f1c00 con 0x7f668c0cb6b0 2023-06-22T08:03:11.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.940+0000 7f66827fc700 1 -- 172.21.15.23:0/2625550079 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6688007310 con 0x7f668c0cb6b0 2023-06-22T08:03:11.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.940+0000 7f66941ff700 1 -- 172.21.15.23:0/2625550079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668c1f1ea0 con 0x7f668c0cb6b0 2023-06-22T08:03:11.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.941+0000 7f66941ff700 1 -- 172.21.15.23:0/2625550079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668c157510 con 0x7f668c0cb6b0 2023-06-22T08:03:11.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.942+0000 7f66827fc700 1 -- 172.21.15.23:0/2625550079 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6688007470 con 0x7f668c0cb6b0 2023-06-22T08:03:11.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.943+0000 7f66827fc700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667006ace0 0x7f667006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:11.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.943+0000 7f66827fc700 1 -- 172.21.15.23:0/2625550079 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f6688089630 con 0x7f668c0cb6b0 2023-06-22T08:03:11.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.943+0000 7f669179a700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667006ace0 0x7f667006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:11.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.943+0000 7f669179a700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667006ace0 0x7f667006d190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f668c159930 tx=0x7f667c01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:11.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:11.947+0000 7f66827fc700 1 -- 172.21.15.23:0/2625550079 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f66880542f0 con 0x7f668c0cb6b0 2023-06-22T08:03:12.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.197+0000 7f66941ff700 1 -- 172.21.15.23:0/2625550079 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f668c04ea20 con 0x7f668c0cb6b0 2023-06-22T08:03:12.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.198+0000 7f66827fc700 1 -- 172.21.15.23:0/2625550079 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5499 (secure 0 0 0) 0x7f6688057910 con 0x7f668c0cb6b0 2023-06-22T08:03:12.201 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:12.202 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:09.287753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:12.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 -- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667006ace0 msgr2=0x7f667006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:12.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667006ace0 0x7f667006d190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f668c159930 tx=0x7f667c01c040 comp rx=0 tx=0).stop 2023-06-22T08:03:12.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 -- 172.21.15.23:0/2625550079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c0cb6b0 msgr2=0x7f668c0ca6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:12.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c0cb6b0 0x7f668c0ca6b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f668800c750 tx=0x7f668800cb10 comp rx=0 tx=0).stop 2023-06-22T08:03:12.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 -- 172.21.15.23:0/2625550079 shutdown_connections 2023-06-22T08:03:12.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667006ace0 0x7f667006d190 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:12.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668c1565d0 0x7f668c0c4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:12.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f668c0cb6b0 0x7f668c0ca6b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:12.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 --2- 172.21.15.23:0/2625550079 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f668c152e10 0x7f668c1576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:12.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 -- 172.21.15.23:0/2625550079 >> 172.21.15.23:0/2625550079 conn(0x7f668c070d80 msgr2=0x7f668c072e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:12.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 -- 172.21.15.23:0/2625550079 shutdown_connections 2023-06-22T08:03:12.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:12.201+0000 7f66941ff700 1 -- 172.21.15.23:0/2625550079 wait complete. 2023-06-22T08:03:12.206 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2023-06-22T08:03:12.223 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4257": { "addr": "172.21.15.67:6837/3798178226", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 3798178226, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 3798178226, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4257, "incarnation": 27, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_4269": { "addr": "172.21.15.23:6834/3365097525", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 3365097525, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 3365097525, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4269, "incarnation": 16, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 13 }, "gid_4272": { "addr": "172.21.15.23:6835/3894174037", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 3894174037, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 3894174037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4272, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 12 }, "gid_4279": { "addr": "172.21.15.67:6836/1962314516", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 1962314516, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 1962314516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4285": { "addr": "172.21.15.67:6839/567663545", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 567663545, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 567663545, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:03:09.287753+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4279, "mds_2": 4272, "mds_3": 4269, "mds_4": 4257 } } } ], "standbys": [] } 2023-06-22T08:03:14.224 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.435+0000 7f4f087ff700 1 Processor -- start 2023-06-22T08:03:14.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.435+0000 7f4f087ff700 1 -- start start 2023-06-22T08:03:14.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.436+0000 7f4f087ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4f00152ce0 0x7f4f000cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:14.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.436+0000 7f4f087ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f000c9fd0 0x7f4f0015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:14.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.436+0000 7f4f087ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0015c370 con 0x7f4f00158880 2023-06-22T08:03:14.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.436+0000 7f4f087ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f001530b0 con 0x7f4f00152ce0 2023-06-22T08:03:14.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.436+0000 7f4f087ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f000cf6f0 con 0x7f4f000c9fd0 2023-06-22T08:03:14.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.436+0000 7f4f0659b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4f00152ce0 0x7f4f000cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:14.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.436+0000 7f4f0659b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4f00152ce0 0x7f4f000cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35792/0 (socket says 172.21.15.23:35792) 2023-06-22T08:03:14.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f0659b700 1 -- 172.21.15.23:0/1726495268 learned_addr learned my addr 172.21.15.23:0/1726495268 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:14.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f05d9a700 1 --2- 172.21.15.23:0/1726495268 >> v2:172.21.15.67:3300/0 conn(0x7f4f000c9fd0 0x7f4f0015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:14.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f06d9c700 1 --1- 172.21.15.23:0/1726495268 >> v1:172.21.15.67:6789/0 conn(0x7f4f00158880 0x7f4f00158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:60912/0 (socket says 172.21.15.23:60912) 2023-06-22T08:03:14.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f0659b700 1 -- 172.21.15.23:0/1726495268 >> v2:172.21.15.67:3300/0 conn(0x7f4f000c9fd0 msgr2=0x7f4f0015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:14.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f0659b700 1 --2- 172.21.15.23:0/1726495268 >> v2:172.21.15.67:3300/0 conn(0x7f4f000c9fd0 0x7f4f0015c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:14.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f0659b700 1 -- 172.21.15.23:0/1726495268 >> v1:172.21.15.67:6789/0 conn(0x7f4f00158880 legacy=0x7f4f00158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:14.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f0659b700 1 -- 172.21.15.23:0/1726495268 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0015bda0 con 0x7f4f00152ce0 2023-06-22T08:03:14.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f0659b700 1 --2- 172.21.15.23:0/1726495268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f00152ce0 0x7f4f000cf1b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4f000cc380 tx=0x7f4efc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44cc9f35989fb322 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:14.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f05599700 1 -- 172.21.15.23:0/1726495268 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc011830 con 0x7f4f00152ce0 2023-06-22T08:03:14.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f05599700 1 -- 172.21.15.23:0/1726495268 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc011990 con 0x7f4f00152ce0 2023-06-22T08:03:14.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.437+0000 7f4f05599700 1 -- 172.21.15.23:0/1726495268 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc011d20 con 0x7f4f00152ce0 2023-06-22T08:03:14.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.438+0000 7f4f087ff700 1 -- 172.21.15.23:0/1726495268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f00152ce0 msgr2=0x7f4f000cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:14.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.438+0000 7f4f087ff700 1 --2- 172.21.15.23:0/1726495268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f00152ce0 0x7f4f000cf1b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4f000cc380 tx=0x7f4efc009bd0 comp rx=0 tx=0).stop 2023-06-22T08:03:14.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.438+0000 7f4f087ff700 1 -- 172.21.15.23:0/1726495268 shutdown_connections 2023-06-22T08:03:14.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.438+0000 7f4f087ff700 1 --2- 172.21.15.23:0/1726495268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f00152ce0 0x7f4f000cf1b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:14.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.438+0000 7f4f087ff700 1 --2- 172.21.15.23:0/1726495268 >> v2:172.21.15.67:3300/0 conn(0x7f4f000c9fd0 0x7f4f0015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:14.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.438+0000 7f4f087ff700 1 -- 172.21.15.23:0/1726495268 >> 172.21.15.23:0/1726495268 conn(0x7f4f0006ae30 msgr2=0x7f4f0006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:14.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.438+0000 7f4f087ff700 1 -- 172.21.15.23:0/1726495268 shutdown_connections 2023-06-22T08:03:14.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.438+0000 7f4f087ff700 1 -- 172.21.15.23:0/1726495268 wait complete. 2023-06-22T08:03:14.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.440+0000 7f4f087ff700 1 Processor -- start 2023-06-22T08:03:14.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.440+0000 7f4f087ff700 1 -- start start 2023-06-22T08:03:14.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f087ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f000c9fd0 0x7f4f001eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:14.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f087ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f00152ce0 0x7f4f000ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:14.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f087ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f00158880 0x7f4f0015aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:14.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f087ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000ce1c0 con 0x7f4f000c9fd0 2023-06-22T08:03:14.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f087ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f001ef010 con 0x7f4f00152ce0 2023-06-22T08:03:14.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f087ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000cc9c0 con 0x7f4f00158880 2023-06-22T08:03:14.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f0659b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f000c9fd0 0x7f4f001eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:14.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f0659b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f000c9fd0 0x7f4f001eead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35802/0 (socket says 172.21.15.23:35802) 2023-06-22T08:03:14.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f0659b700 1 -- 172.21.15.23:0/3724014952 learned_addr learned my addr 172.21.15.23:0/3724014952 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:14.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f05d9a700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f00152ce0 0x7f4f000ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:14.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.441+0000 7f4f06d9c700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f00158880 0x7f4f0015aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:14.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4f0659b700 1 -- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f00158880 msgr2=0x7f4f0015aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:14.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4f0659b700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f00158880 0x7f4f0015aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:14.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4f0659b700 1 -- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f00152ce0 msgr2=0x7f4f000ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:14.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4f0659b700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f00152ce0 0x7f4f000ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:14.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4f0659b700 1 -- 172.21.15.23:0/3724014952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4efc009550 con 0x7f4f000c9fd0 2023-06-22T08:03:14.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4f05d9a700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f00152ce0 0x7f4f000ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:03:14.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4f0659b700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f000c9fd0 0x7f4f001eead0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4f0008bc30 tx=0x7f4efc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:14.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4ef6ffd700 1 -- 172.21.15.23:0/3724014952 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc004300 con 0x7f4f000c9fd0 2023-06-22T08:03:14.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4ef6ffd700 1 -- 172.21.15.23:0/3724014952 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc01f070 con 0x7f4f000c9fd0 2023-06-22T08:03:14.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4f087ff700 1 -- 172.21.15.23:0/3724014952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f000ccc10 con 0x7f4f000c9fd0 2023-06-22T08:03:14.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4ef6ffd700 1 -- 172.21.15.23:0/3724014952 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc0036a0 con 0x7f4f000c9fd0 2023-06-22T08:03:14.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.442+0000 7f4f087ff700 1 -- 172.21.15.23:0/3724014952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f000cd1d0 con 0x7f4f000c9fd0 2023-06-22T08:03:14.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.443+0000 7f4f087ff700 1 -- 172.21.15.23:0/3724014952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0015c100 con 0x7f4f000c9fd0 2023-06-22T08:03:14.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.444+0000 7f4ef6ffd700 1 -- 172.21.15.23:0/3724014952 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4efc021030 con 0x7f4f000c9fd0 2023-06-22T08:03:14.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.446+0000 7f4ef6ffd700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ee406ad10 0x7f4ee406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:14.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.446+0000 7f4ef6ffd700 1 -- 172.21.15.23:0/3724014952 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f4efc08ad00 con 0x7f4f000c9fd0 2023-06-22T08:03:14.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.446+0000 7f4f05d9a700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ee406ad10 0x7f4ee406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:14.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.447+0000 7f4f05d9a700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ee406ad10 0x7f4ee406d1c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4f000ca900 tx=0x7f4ef0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:14.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.451+0000 7f4ef6ffd700 1 -- 172.21.15.23:0/3724014952 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4efc055af0 con 0x7f4f000c9fd0 2023-06-22T08:03:14.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.709+0000 7f4f087ff700 1 -- 172.21.15.23:0/3724014952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f0004ea20 con 0x7f4f000c9fd0 2023-06-22T08:03:14.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.709+0000 7f4ef6ffd700 1 -- 172.21.15.23:0/3724014952 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5499 (secure 0 0 0) 0x7f4efc059110 con 0x7f4f000c9fd0 2023-06-22T08:03:14.713 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:14.713 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:09.287753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:14.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 -- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ee406ad10 msgr2=0x7f4ee406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:14.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ee406ad10 0x7f4ee406d1c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4f000ca900 tx=0x7f4ef0009210 comp rx=0 tx=0).stop 2023-06-22T08:03:14.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 -- 172.21.15.23:0/3724014952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f000c9fd0 msgr2=0x7f4f001eead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:14.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f000c9fd0 0x7f4f001eead0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4f0008bc30 tx=0x7f4efc004750 comp rx=0 tx=0).stop 2023-06-22T08:03:14.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 -- 172.21.15.23:0/3724014952 shutdown_connections 2023-06-22T08:03:14.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f00158880 0x7f4f0015aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:14.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f000c9fd0 0x7f4f001eead0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:14.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ee406ad10 0x7f4ee406d1c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:14.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 --2- 172.21.15.23:0/3724014952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f00152ce0 0x7f4f000ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:14.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 -- 172.21.15.23:0/3724014952 >> 172.21.15.23:0/3724014952 conn(0x7f4f0006ae30 msgr2=0x7f4f000cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:14.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 -- 172.21.15.23:0/3724014952 shutdown_connections 2023-06-22T08:03:14.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:14.712+0000 7f4f087ff700 1 -- 172.21.15.23:0/3724014952 wait complete. 2023-06-22T08:03:14.717 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2023-06-22T08:03:14.734 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:03:16.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:16.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.942+0000 7fc63759e700 1 Processor -- start 2023-06-22T08:03:16.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.942+0000 7fc63759e700 1 -- start start 2023-06-22T08:03:16.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.943+0000 7fc63759e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc63806bc40 0x7fc6380c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:16.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc63759e700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc63806ae60 0x7fc63806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:16.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc63759e700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6380c8f80 con 0x7fc6380c9890 2023-06-22T08:03:16.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc63759e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63806c0c0 con 0x7fc63806bc40 2023-06-22T08:03:16.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc63759e700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6380ca1a0 con 0x7fc63806ae60 2023-06-22T08:03:16.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc635d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc63806bc40 0x7fc6380c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:16.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc635d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc63806bc40 0x7fc6380c9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57670/0 (socket says 172.21.15.23:57670) 2023-06-22T08:03:16.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc635d9b700 1 -- 172.21.15.23:0/1782192928 learned_addr learned my addr 172.21.15.23:0/1782192928 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:16.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc636d9d700 1 --2- 172.21.15.23:0/1782192928 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc63806ae60 0x7fc63806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:16.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc635d9b700 1 -- 172.21.15.23:0/1782192928 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc63806ae60 msgr2=0x7fc63806b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:16.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc635d9b700 1 --2- 172.21.15.23:0/1782192928 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc63806ae60 0x7fc63806b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:16.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc635d9b700 1 -- 172.21.15.23:0/1782192928 >> v1:172.21.15.67:6789/0 conn(0x7fc6380c9890 legacy=0x7fc638158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:16.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.944+0000 7fc635d9b700 1 -- 172.21.15.23:0/1782192928 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63806b820 con 0x7fc63806bc40 2023-06-22T08:03:16.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.945+0000 7fc635d9b700 1 --2- 172.21.15.23:0/1782192928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc63806bc40 0x7fc6380c9c60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0098f0 tx=0x7fc62c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de166e5b2cf82d3f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:16.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.945+0000 7fc63559a700 1 -- 172.21.15.23:0/1782192928 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c011830 con 0x7fc63806bc40 2023-06-22T08:03:16.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.945+0000 7fc63559a700 1 -- 172.21.15.23:0/1782192928 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c011990 con 0x7fc63806bc40 2023-06-22T08:03:16.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.945+0000 7fc63559a700 1 -- 172.21.15.23:0/1782192928 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c011d20 con 0x7fc63806bc40 2023-06-22T08:03:16.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.945+0000 7fc63759e700 1 -- 172.21.15.23:0/1782192928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc63806bc40 msgr2=0x7fc6380c9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:16.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.945+0000 7fc63759e700 1 --2- 172.21.15.23:0/1782192928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc63806bc40 0x7fc6380c9c60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0098f0 tx=0x7fc62c009c00 comp rx=0 tx=0).stop 2023-06-22T08:03:16.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.945+0000 7fc63759e700 1 -- 172.21.15.23:0/1782192928 shutdown_connections 2023-06-22T08:03:16.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.945+0000 7fc63759e700 1 --2- 172.21.15.23:0/1782192928 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc63806ae60 0x7fc63806b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:16.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.945+0000 7fc63759e700 1 --2- 172.21.15.23:0/1782192928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc63806bc40 0x7fc6380c9c60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:16.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.945+0000 7fc63759e700 1 -- 172.21.15.23:0/1782192928 >> 172.21.15.23:0/1782192928 conn(0x7fc638065b90 msgr2=0x7fc638065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:16.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.946+0000 7fc63759e700 1 -- 172.21.15.23:0/1782192928 shutdown_connections 2023-06-22T08:03:16.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.946+0000 7fc63759e700 1 -- 172.21.15.23:0/1782192928 wait complete. 2023-06-22T08:03:16.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.947+0000 7fc63759e700 1 Processor -- start 2023-06-22T08:03:16.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.947+0000 7fc63759e700 1 -- start start 2023-06-22T08:03:16.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.948+0000 7fc63759e700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc63806ae60 0x7fc6380cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:16.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.948+0000 7fc63759e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc63806bc40 0x7fc6381eec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:16.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.948+0000 7fc63759e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6380c9890 0x7fc6381f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:16.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.948+0000 7fc63759e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6381f25d0 con 0x7fc6380c9890 2023-06-22T08:03:16.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.948+0000 7fc63759e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6380ced50 con 0x7fc63806bc40 2023-06-22T08:03:16.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.948+0000 7fc63759e700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6380cf030 con 0x7fc63806ae60 2023-06-22T08:03:16.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.948+0000 7fc636d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6380c9890 0x7fc6381f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:16.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc636d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6380c9890 0x7fc6381f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57680/0 (socket says 172.21.15.23:57680) 2023-06-22T08:03:16.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc636d9d700 1 -- 172.21.15.23:0/2761475854 learned_addr learned my addr 172.21.15.23:0/2761475854 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:16.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc63659c700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc63806ae60 0x7fc6380cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:16.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc635d9b700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc63806bc40 0x7fc6381eec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:16.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc636d9d700 1 -- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc63806ae60 msgr2=0x7fc6380cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:16.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc636d9d700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc63806ae60 0x7fc6380cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:16.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc636d9d700 1 -- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc63806bc40 msgr2=0x7fc6381eec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:16.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc636d9d700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc63806bc40 0x7fc6381eec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:16.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc636d9d700 1 -- 172.21.15.23:0/2761475854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62c009550 con 0x7fc6380c9890 2023-06-22T08:03:16.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc636d9d700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6380c9890 0x7fc6381f2090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc63000efc0 tx=0x7fc63000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:16.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.949+0000 7fc626ffd700 1 -- 172.21.15.23:0/2761475854 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630009ea0 con 0x7fc6380c9890 2023-06-22T08:03:16.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.950+0000 7fc626ffd700 1 -- 172.21.15.23:0/2761475854 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63000f040 con 0x7fc6380c9890 2023-06-22T08:03:16.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.950+0000 7fc626ffd700 1 -- 172.21.15.23:0/2761475854 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63000ced0 con 0x7fc6380c9890 2023-06-22T08:03:16.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.950+0000 7fc63759e700 1 -- 172.21.15.23:0/2761475854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6380cf310 con 0x7fc6380c9890 2023-06-22T08:03:16.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.950+0000 7fc63759e700 1 -- 172.21.15.23:0/2761475854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6381ee740 con 0x7fc6380c9890 2023-06-22T08:03:16.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.951+0000 7fc63759e700 1 -- 172.21.15.23:0/2761475854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6381ee930 con 0x7fc6380c9890 2023-06-22T08:03:16.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.952+0000 7fc626ffd700 1 -- 172.21.15.23:0/2761475854 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc630014480 con 0x7fc6380c9890 2023-06-22T08:03:16.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.953+0000 7fc626ffd700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc61806ad10 0x7fc61806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:16.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.953+0000 7fc626ffd700 1 -- 172.21.15.23:0/2761475854 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7fc63008a100 con 0x7fc6380c9890 2023-06-22T08:03:16.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.953+0000 7fc63659c700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc61806ad10 0x7fc61806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:16.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.954+0000 7fc63659c700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc61806ad10 0x7fc61806d1c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc628007fd0 tx=0x7fc628009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:16.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:16.957+0000 7fc626ffd700 1 -- 172.21.15.23:0/2761475854 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc630054d30 con 0x7fc6380c9890 2023-06-22T08:03:17.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.219+0000 7fc63759e700 1 -- 172.21.15.23:0/2761475854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc638061b00 con 0x7fc6380c9890 2023-06-22T08:03:17.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.220+0000 7fc626ffd700 1 -- 172.21.15.23:0/2761475854 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5501 (secure 0 0 0) 0x7fc630018020 con 0x7fc6380c9890 2023-06-22T08:03:17.223 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:17.223 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:15.291604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:resolve","state_seq":53,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:17.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 -- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc61806ad10 msgr2=0x7fc61806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:17.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc61806ad10 0x7fc61806d1c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc628007fd0 tx=0x7fc628009040 comp rx=0 tx=0).stop 2023-06-22T08:03:17.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 -- 172.21.15.23:0/2761475854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6380c9890 msgr2=0x7fc6381f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:17.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6380c9890 0x7fc6381f2090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc63000efc0 tx=0x7fc63000c3a0 comp rx=0 tx=0).stop 2023-06-22T08:03:17.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 -- 172.21.15.23:0/2761475854 shutdown_connections 2023-06-22T08:03:17.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6380c9890 0x7fc6381f2090 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:17.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc63806ae60 0x7fc6380cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:17.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc61806ad10 0x7fc61806d1c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:17.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 --2- 172.21.15.23:0/2761475854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc63806bc40 0x7fc6381eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:17.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 -- 172.21.15.23:0/2761475854 >> 172.21.15.23:0/2761475854 conn(0x7fc638065b90 msgr2=0x7fc638065f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:17.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 -- 172.21.15.23:0/2761475854 shutdown_connections 2023-06-22T08:03:17.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:17.222+0000 7fc63759e700 1 -- 172.21.15.23:0/2761475854 wait complete. 2023-06-22T08:03:17.228 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2023-06-22T08:03:17.244 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:03:19.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:19.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.449+0000 7f2363bff700 1 Processor -- start 2023-06-22T08:03:19.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.450+0000 7f2363bff700 1 -- start start 2023-06-22T08:03:19.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.451+0000 7f2363bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f235c152d70 0x7f235c0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:19.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.451+0000 7f2363bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f235c0c45f0 0x7f235c0c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:19.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.451+0000 7f2363bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235c0c4fb0 con 0x7f235c159330 2023-06-22T08:03:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.451+0000 7f2363bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f235c0c95e0 con 0x7f235c0c45f0 2023-06-22T08:03:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.451+0000 7f2363bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235c15bea0 con 0x7f235c152d70 2023-06-22T08:03:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.451+0000 7f236219c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f235c0c45f0 0x7f235c0c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.451+0000 7f236219c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f235c0c45f0 0x7f235c0c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57682/0 (socket says 172.21.15.23:57682) 2023-06-22T08:03:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.451+0000 7f236219c700 1 -- 172.21.15.23:0/656128230 learned_addr learned my addr 172.21.15.23:0/656128230 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:19.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.451+0000 7f236199b700 1 --2- 172.21.15.23:0/656128230 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f235c152d70 0x7f235c0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:19.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.452+0000 7f236219c700 1 -- 172.21.15.23:0/656128230 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f235c152d70 msgr2=0x7f235c0c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:19.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.452+0000 7f236219c700 1 --2- 172.21.15.23:0/656128230 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f235c152d70 0x7f235c0c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:19.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.452+0000 7f236219c700 1 -- 172.21.15.23:0/656128230 >> v1:172.21.15.67:6789/0 conn(0x7f235c159330 legacy=0x7f235c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:19.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.452+0000 7f236219c700 1 -- 172.21.15.23:0/656128230 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c15c160 con 0x7f235c0c45f0 2023-06-22T08:03:19.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.452+0000 7f236219c700 1 --2- 172.21.15.23:0/656128230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f235c0c45f0 0x7f235c0c4a70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2354009e50 tx=0x7f235400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552cdc8713d231fd server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:19.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.452+0000 7f2360999700 1 -- 172.21.15.23:0/656128230 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235400e070 con 0x7f235c0c45f0 2023-06-22T08:03:19.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.452+0000 7f2360999700 1 -- 172.21.15.23:0/656128230 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235400b910 con 0x7f235c0c45f0 2023-06-22T08:03:19.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.452+0000 7f2360999700 1 -- 172.21.15.23:0/656128230 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235400bcc0 con 0x7f235c0c45f0 2023-06-22T08:03:19.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.453+0000 7f2363bff700 1 -- 172.21.15.23:0/656128230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f235c0c45f0 msgr2=0x7f235c0c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:19.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.453+0000 7f2363bff700 1 --2- 172.21.15.23:0/656128230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f235c0c45f0 0x7f235c0c4a70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2354009e50 tx=0x7f235400b330 comp rx=0 tx=0).stop 2023-06-22T08:03:19.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.453+0000 7f2363bff700 1 -- 172.21.15.23:0/656128230 shutdown_connections 2023-06-22T08:03:19.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.453+0000 7f2363bff700 1 --2- 172.21.15.23:0/656128230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f235c0c45f0 0x7f235c0c4a70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:19.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.453+0000 7f2363bff700 1 --2- 172.21.15.23:0/656128230 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f235c152d70 0x7f235c0c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:19.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.453+0000 7f2363bff700 1 -- 172.21.15.23:0/656128230 >> 172.21.15.23:0/656128230 conn(0x7f235c06ae30 msgr2=0x7f235c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:19.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.453+0000 7f2363bff700 1 -- 172.21.15.23:0/656128230 shutdown_connections 2023-06-22T08:03:19.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.453+0000 7f2363bff700 1 -- 172.21.15.23:0/656128230 wait complete. 2023-06-22T08:03:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.454+0000 7f2363bff700 1 Processor -- start 2023-06-22T08:03:19.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.455+0000 7f2363bff700 1 -- start start 2023-06-22T08:03:19.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f2363bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f235c0c45f0 0x7f235c14dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:19.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f2363bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f235c152d70 0x7f235c157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:19.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f2363bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f235c159330 0x7f235c157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:19.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f2363bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235c1ef0c0 con 0x7f235c152d70 2023-06-22T08:03:19.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f2363bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235c1eeb20 con 0x7f235c0c45f0 2023-06-22T08:03:19.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f2363bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235c0c8f10 con 0x7f235c159330 2023-06-22T08:03:19.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f236199b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f235c0c45f0 0x7f235c14dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f236199b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f235c0c45f0 0x7f235c14dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:44712/0 (socket says 172.21.15.23:44712) 2023-06-22T08:03:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f236199b700 1 -- 172.21.15.23:0/4026105661 learned_addr learned my addr 172.21.15.23:0/4026105661 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f236119a700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f235c152d70 0x7f235c157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f236219c700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f235c159330 0x7f235c157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f236199b700 1 -- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f235c159330 msgr2=0x7f235c157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f236199b700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f235c159330 0x7f235c157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:19.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f236199b700 1 -- 172.21.15.23:0/4026105661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f235c152d70 msgr2=0x7f235c157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f236199b700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f235c152d70 0x7f235c157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.456+0000 7f236199b700 1 -- 172.21.15.23:0/4026105661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2354009b30 con 0x7f235c0c45f0 2023-06-22T08:03:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.457+0000 7f236199b700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f235c0c45f0 0x7f235c14dfd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f235800e9b0 tx=0x7f235800ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.457+0000 7f23527fc700 1 -- 172.21.15.23:0/4026105661 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358009710 con 0x7f235c0c45f0 2023-06-22T08:03:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.457+0000 7f23527fc700 1 -- 172.21.15.23:0/4026105661 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235800ee50 con 0x7f235c0c45f0 2023-06-22T08:03:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.457+0000 7f23527fc700 1 -- 172.21.15.23:0/4026105661 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235800ca50 con 0x7f235c0c45f0 2023-06-22T08:03:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.457+0000 7f2363bff700 1 -- 172.21.15.23:0/4026105661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235c14ea00 con 0x7f235c0c45f0 2023-06-22T08:03:19.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.457+0000 7f2363bff700 1 -- 172.21.15.23:0/4026105661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235c14d3f0 con 0x7f235c0c45f0 2023-06-22T08:03:19.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.458+0000 7f2363bff700 1 -- 172.21.15.23:0/4026105661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f235c14d250 con 0x7f235c0c45f0 2023-06-22T08:03:19.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.459+0000 7f23527fc700 1 -- 172.21.15.23:0/4026105661 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f235800cbb0 con 0x7f235c0c45f0 2023-06-22T08:03:19.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.460+0000 7f23527fc700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f234006ace0 0x7f234006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:19.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.460+0000 7f23527fc700 1 -- 172.21.15.23:0/4026105661 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f23580898f0 con 0x7f235c0c45f0 2023-06-22T08:03:19.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.460+0000 7f236119a700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f234006ace0 0x7f234006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:19.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.461+0000 7f236119a700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f234006ace0 0x7f234006d190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f235c0b4020 tx=0x7f234c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:19.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.465+0000 7f23527fc700 1 -- 172.21.15.23:0/4026105661 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f235c14d250 con 0x7f235c0c45f0 2023-06-22T08:03:19.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.717+0000 7f2363bff700 1 -- 172.21.15.23:0/4026105661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f235c14d250 con 0x7f235c0c45f0 2023-06-22T08:03:19.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.718+0000 7f23527fc700 1 -- 172.21.15.23:0/4026105661 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5501 (secure 0 0 0) 0x7f235c14d250 con 0x7f235c0c45f0 2023-06-22T08:03:19.722 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:19.722 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:15.291604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:resolve","state_seq":53,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.720+0000 7f2363bff700 1 -- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f234006ace0 msgr2=0x7f234006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:19.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.720+0000 7f2363bff700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f234006ace0 0x7f234006d190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f235c0b4020 tx=0x7f234c009040 comp rx=0 tx=0).stop 2023-06-22T08:03:19.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.720+0000 7f2363bff700 1 -- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f235c0c45f0 msgr2=0x7f235c14dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:19.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.720+0000 7f2363bff700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f235c0c45f0 0x7f235c14dfd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f235800e9b0 tx=0x7f235800ecc0 comp rx=0 tx=0).stop 2023-06-22T08:03:19.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.721+0000 7f2363bff700 1 -- 172.21.15.23:0/4026105661 shutdown_connections 2023-06-22T08:03:19.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.721+0000 7f2363bff700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f235c159330 0x7f235c157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:19.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.721+0000 7f2363bff700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f234006ace0 0x7f234006d190 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:19.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.721+0000 7f2363bff700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f235c0c45f0 0x7f235c14dfd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:19.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.721+0000 7f2363bff700 1 --2- 172.21.15.23:0/4026105661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f235c152d70 0x7f235c157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:19.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.721+0000 7f2363bff700 1 -- 172.21.15.23:0/4026105661 >> 172.21.15.23:0/4026105661 conn(0x7f235c06ae30 msgr2=0x7f235c150420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:19.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.721+0000 7f2363bff700 1 -- 172.21.15.23:0/4026105661 shutdown_connections 2023-06-22T08:03:19.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:19.721+0000 7f2363bff700 1 -- 172.21.15.23:0/4026105661 wait complete. 2023-06-22T08:03:19.728 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2023-06-22T08:03:19.742 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:03:21.743 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:21.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.949+0000 7f6a4abff700 1 Processor -- start 2023-06-22T08:03:21.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.949+0000 7f6a4abff700 1 -- start start 2023-06-22T08:03:21.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4abff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6a44152ce0 0x7f6a440cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:21.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4abff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a440c9fd0 0x7f6a4415c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:21.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4abff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a44158880 0x7f6a44158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:21.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4abff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a441530b0 con 0x7f6a44152ce0 2023-06-22T08:03:21.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4abff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a440cf6f0 con 0x7f6a44158880 2023-06-22T08:03:21.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4abff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4415c370 con 0x7f6a440c9fd0 2023-06-22T08:03:21.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4899b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6a44152ce0 0x7f6a440cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:21.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4899b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6a44152ce0 0x7f6a440cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57714/0 (socket says 172.21.15.23:57714) 2023-06-22T08:03:21.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4899b700 1 -- 172.21.15.23:0/141759615 learned_addr learned my addr 172.21.15.23:0/141759615 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:21.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4919c700 1 --2- 172.21.15.23:0/141759615 >> v2:172.21.15.67:3300/0 conn(0x7f6a44158880 0x7f6a44158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:21.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a43fff700 1 --2- 172.21.15.23:0/141759615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a440c9fd0 0x7f6a4415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:21.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4899b700 1 -- 172.21.15.23:0/141759615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a440c9fd0 msgr2=0x7f6a4415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:21.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4899b700 1 --2- 172.21.15.23:0/141759615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a440c9fd0 0x7f6a4415c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:21.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4899b700 1 -- 172.21.15.23:0/141759615 >> v2:172.21.15.67:3300/0 conn(0x7f6a44158880 msgr2=0x7f6a44158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:21.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.951+0000 7f6a4899b700 1 --2- 172.21.15.23:0/141759615 >> v2:172.21.15.67:3300/0 conn(0x7f6a44158880 0x7f6a44158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:21.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.952+0000 7f6a4899b700 1 -- 172.21.15.23:0/141759615 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4415bda0 con 0x7f6a44152ce0 2023-06-22T08:03:21.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.952+0000 7f6a4899b700 1 --2- 172.21.15.23:0/141759615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 0x7f6a440cf1b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6a4408b9e0 tx=0x7f6a38009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06033557619cfa server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:21.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.952+0000 7f6a437fe700 1 -- 172.21.15.23:0/141759615 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a38011830 con 0x7f6a44152ce0 2023-06-22T08:03:21.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.952+0000 7f6a437fe700 1 -- 172.21.15.23:0/141759615 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a38011990 con 0x7f6a44152ce0 2023-06-22T08:03:21.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.952+0000 7f6a437fe700 1 -- 172.21.15.23:0/141759615 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a38011d20 con 0x7f6a44152ce0 2023-06-22T08:03:21.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.952+0000 7f6a4abff700 1 -- 172.21.15.23:0/141759615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 msgr2=0x7f6a440cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:21.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.952+0000 7f6a4abff700 1 --2- 172.21.15.23:0/141759615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 0x7f6a440cf1b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6a4408b9e0 tx=0x7f6a38009bd0 comp rx=0 tx=0).stop 2023-06-22T08:03:21.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.953+0000 7f6a4abff700 1 -- 172.21.15.23:0/141759615 shutdown_connections 2023-06-22T08:03:21.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.953+0000 7f6a4abff700 1 --2- 172.21.15.23:0/141759615 >> v2:172.21.15.67:3300/0 conn(0x7f6a44158880 0x7f6a44158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:21.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.953+0000 7f6a4abff700 1 --2- 172.21.15.23:0/141759615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 0x7f6a440cf1b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.953+0000 7f6a4abff700 1 --2- 172.21.15.23:0/141759615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a440c9fd0 0x7f6a4415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.953+0000 7f6a4abff700 1 -- 172.21.15.23:0/141759615 >> 172.21.15.23:0/141759615 conn(0x7f6a4406ae30 msgr2=0x7f6a4406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.953+0000 7f6a4abff700 1 -- 172.21.15.23:0/141759615 shutdown_connections 2023-06-22T08:03:21.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.953+0000 7f6a4abff700 1 -- 172.21.15.23:0/141759615 wait complete. 2023-06-22T08:03:21.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.955+0000 7f6a4abff700 1 Processor -- start 2023-06-22T08:03:21.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.955+0000 7f6a4abff700 1 -- start start 2023-06-22T08:03:21.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a4abff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a440c9fd0 0x7f6a441497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:21.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a4abff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 0x7f6a44149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:21.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a4abff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a44158880 0x7f6a441f1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:21.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a4abff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a440b2830 con 0x7f6a44152ce0 2023-06-22T08:03:21.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a4abff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a44149350 con 0x7f6a440c9fd0 2023-06-22T08:03:21.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a4abff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4415b210 con 0x7f6a44158880 2023-06-22T08:03:21.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 0x7f6a44149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:21.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 0x7f6a44149d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57724/0 (socket says 172.21.15.23:57724) 2023-06-22T08:03:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a43fff700 1 -- 172.21.15.23:0/2169528950 learned_addr learned my addr 172.21.15.23:0/2169528950 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a4899b700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a440c9fd0 0x7f6a441497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a4919c700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a44158880 0x7f6a441f1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a43fff700 1 -- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a44158880 msgr2=0x7f6a441f1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a43fff700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a44158880 0x7f6a441f1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a43fff700 1 -- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a440c9fd0 msgr2=0x7f6a441497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:21.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a43fff700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a440c9fd0 0x7f6a441497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:21.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.957+0000 7f6a43fff700 1 -- 172.21.15.23:0/2169528950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a38009550 con 0x7f6a44152ce0 2023-06-22T08:03:21.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.958+0000 7f6a43fff700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 0x7f6a44149d20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000e9e0 tx=0x7f6a3000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:21.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.958+0000 7f6a417fa700 1 -- 172.21.15.23:0/2169528950 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30009710 con 0x7f6a44152ce0 2023-06-22T08:03:21.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.958+0000 7f6a417fa700 1 -- 172.21.15.23:0/2169528950 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3000ee80 con 0x7f6a44152ce0 2023-06-22T08:03:21.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.958+0000 7f6a4abff700 1 -- 172.21.15.23:0/2169528950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4415ac50 con 0x7f6a44152ce0 2023-06-22T08:03:21.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.958+0000 7f6a417fa700 1 -- 172.21.15.23:0/2169528950 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3000ca50 con 0x7f6a44152ce0 2023-06-22T08:03:21.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.958+0000 7f6a4abff700 1 -- 172.21.15.23:0/2169528950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a44148fe0 con 0x7f6a44152ce0 2023-06-22T08:03:21.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.959+0000 7f6a4abff700 1 -- 172.21.15.23:0/2169528950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a44149500 con 0x7f6a44152ce0 2023-06-22T08:03:21.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.960+0000 7f6a417fa700 1 -- 172.21.15.23:0/2169528950 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6a3000cbb0 con 0x7f6a44152ce0 2023-06-22T08:03:21.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.961+0000 7f6a417fa700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a2806ad10 0x7f6a2806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:21.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.961+0000 7f6a417fa700 1 -- 172.21.15.23:0/2169528950 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f6a30089b10 con 0x7f6a44152ce0 2023-06-22T08:03:21.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.961+0000 7f6a4899b700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a2806ad10 0x7f6a2806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:21.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.962+0000 7f6a4899b700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a2806ad10 0x7f6a2806d1c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6a38009c10 tx=0x7f6a38008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:21.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:21.965+0000 7f6a417fa700 1 -- 172.21.15.23:0/2169528950 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6a44149500 con 0x7f6a44152ce0 2023-06-22T08:03:22.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.221+0000 7f6a4abff700 1 -- 172.21.15.23:0/2169528950 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a44149500 con 0x7f6a44152ce0 2023-06-22T08:03:22.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.221+0000 7f6a417fa700 1 -- 172.21.15.23:0/2169528950 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5503 (secure 0 0 0) 0x7f6a44149500 con 0x7f6a44152ce0 2023-06-22T08:03:22.225 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:22.225 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:20.673359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:reconnect","state_seq":55,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 -- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a2806ad10 msgr2=0x7f6a2806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a2806ad10 0x7f6a2806d1c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6a38009c10 tx=0x7f6a38008030 comp rx=0 tx=0).stop 2023-06-22T08:03:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 -- 172.21.15.23:0/2169528950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 msgr2=0x7f6a44149d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:22.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 0x7f6a44149d20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000e9e0 tx=0x7f6a3000ecf0 comp rx=0 tx=0).stop 2023-06-22T08:03:22.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 -- 172.21.15.23:0/2169528950 shutdown_connections 2023-06-22T08:03:22.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a44158880 0x7f6a441f1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:22.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a2806ad10 0x7f6a2806d1c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:22.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a440c9fd0 0x7f6a441497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:22.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 --2- 172.21.15.23:0/2169528950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a44152ce0 0x7f6a44149d20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:22.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 -- 172.21.15.23:0/2169528950 >> 172.21.15.23:0/2169528950 conn(0x7f6a4406ae30 msgr2=0x7f6a440c47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 -- 172.21.15.23:0/2169528950 shutdown_connections 2023-06-22T08:03:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:22.224+0000 7f6a4abff700 1 -- 172.21.15.23:0/2169528950 wait complete. 2023-06-22T08:03:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2023-06-22T08:03:22.246 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:03:24.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.454+0000 7fb6847ff700 1 Processor -- start 2023-06-22T08:03:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.455+0000 7fb6847ff700 1 -- start start 2023-06-22T08:03:24.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.456+0000 7fb6847ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb67c152e10 0x7fb67c069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:24.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.456+0000 7fb6847ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c158f60 con 0x7fb67c0cb6b0 2023-06-22T08:03:24.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.456+0000 7fb6847ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c0cc450 con 0x7fb67c0c5740 2023-06-22T08:03:24.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.456+0000 7fb6847ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67c06bd20 con 0x7fb67c152e10 2023-06-22T08:03:24.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.456+0000 7fb681d9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb67c152e10 0x7fb67c069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:24.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.456+0000 7fb68259b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb67c0cb6b0 0x7fb67c152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60410/0 (socket says 172.21.15.23:60410) 2023-06-22T08:03:24.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.456+0000 7fb681d9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb67c152e10 0x7fb67c069eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57736/0 (socket says 172.21.15.23:57736) 2023-06-22T08:03:24.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.456+0000 7fb68259b700 1 -- 172.21.15.23:0/3912020824 learned_addr learned my addr 172.21.15.23:0/3912020824 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:03:24.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.457+0000 7fb681d9a700 1 -- 172.21.15.23:0/3912020824 >> v1:172.21.15.67:6789/0 conn(0x7fb67c0c5740 legacy=0x7fb67c163110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:24.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.457+0000 7fb681d9a700 1 -- 172.21.15.23:0/3912020824 >> v1:172.21.15.23:6789/0 conn(0x7fb67c0cb6b0 legacy=0x7fb67c152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:24.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.457+0000 7fb681d9a700 1 -- 172.21.15.23:0/3912020824 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c158980 con 0x7fb67c152e10 2023-06-22T08:03:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.457+0000 7fb681599700 1 -- 172.21.15.23:0/3912020824 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953732361 0 0) 0x7fb67c158f60 con 0x7fb67c0cb6b0 2023-06-22T08:03:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.457+0000 7fb681d9a700 1 --2- 172.21.15.23:0/3912020824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c152e10 0x7fb67c069eb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009840 tx=0x7fb66c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae851a880f100384 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.457+0000 7fb681599700 1 -- 172.21.15.23:0/3912020824 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c011830 con 0x7fb67c152e10 2023-06-22T08:03:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.457+0000 7fb681599700 1 -- 172.21.15.23:0/3912020824 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c011990 con 0x7fb67c152e10 2023-06-22T08:03:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.457+0000 7fb681599700 1 -- 172.21.15.23:0/3912020824 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c011d20 con 0x7fb67c152e10 2023-06-22T08:03:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.458+0000 7fb6847ff700 1 -- 172.21.15.23:0/3912020824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c152e10 msgr2=0x7fb67c069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.458+0000 7fb6847ff700 1 --2- 172.21.15.23:0/3912020824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c152e10 0x7fb67c069eb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009840 tx=0x7fb66c009b50 comp rx=0 tx=0).stop 2023-06-22T08:03:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.458+0000 7fb6847ff700 1 -- 172.21.15.23:0/3912020824 shutdown_connections 2023-06-22T08:03:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.458+0000 7fb6847ff700 1 --2- 172.21.15.23:0/3912020824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c152e10 0x7fb67c069eb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.458+0000 7fb6847ff700 1 -- 172.21.15.23:0/3912020824 >> 172.21.15.23:0/3912020824 conn(0x7fb67c070d80 msgr2=0x7fb67c071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.458+0000 7fb6847ff700 1 -- 172.21.15.23:0/3912020824 shutdown_connections 2023-06-22T08:03:24.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.458+0000 7fb6847ff700 1 -- 172.21.15.23:0/3912020824 wait complete. 2023-06-22T08:03:24.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.460+0000 7fb6847ff700 1 Processor -- start 2023-06-22T08:03:24.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.460+0000 7fb6847ff700 1 -- start start 2023-06-22T08:03:24.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb6847ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb67c0cb6b0 0x7fb67c155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:24.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb6847ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb67c152e10 0x7fb67c153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:24.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb6847ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c157eb0 0x7fb67c0c39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb6847ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c156ee0 con 0x7fb67c157eb0 2023-06-22T08:03:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb6847ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c0c48e0 con 0x7fb67c0cb6b0 2023-06-22T08:03:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb6847ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c1570f0 con 0x7fb67c152e10 2023-06-22T08:03:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb682d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c157eb0 0x7fb67c0c39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb681d9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb67c152e10 0x7fb67c153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb682d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c157eb0 0x7fb67c0c39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57746/0 (socket says 172.21.15.23:57746) 2023-06-22T08:03:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb682d9c700 1 -- 172.21.15.23:0/4048830043 learned_addr learned my addr 172.21.15.23:0/4048830043 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb68259b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb67c0cb6b0 0x7fb67c155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb682d9c700 1 -- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb67c152e10 msgr2=0x7fb67c153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb682d9c700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb67c152e10 0x7fb67c153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb682d9c700 1 -- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb67c0cb6b0 msgr2=0x7fb67c155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb682d9c700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb67c0cb6b0 0x7fb67c155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.461+0000 7fb682d9c700 1 -- 172.21.15.23:0/4048830043 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c0094a0 con 0x7fb67c157eb0 2023-06-22T08:03:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.462+0000 7fb682d9c700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c157eb0 0x7fb67c0c39d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb67400cc50 tx=0x7fb67400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.462+0000 7fb672ffd700 1 -- 172.21.15.23:0/4048830043 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674003c00 con 0x7fb67c157eb0 2023-06-22T08:03:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.462+0000 7fb672ffd700 1 -- 172.21.15.23:0/4048830043 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb674003d60 con 0x7fb67c157eb0 2023-06-22T08:03:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.462+0000 7fb672ffd700 1 -- 172.21.15.23:0/4048830043 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67400aab0 con 0x7fb67c157eb0 2023-06-22T08:03:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.462+0000 7fb6847ff700 1 -- 172.21.15.23:0/4048830043 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c1573d0 con 0x7fb67c157eb0 2023-06-22T08:03:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.462+0000 7fb6847ff700 1 -- 172.21.15.23:0/4048830043 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c157990 con 0x7fb67c157eb0 2023-06-22T08:03:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.463+0000 7fb6847ff700 1 -- 172.21.15.23:0/4048830043 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c04ea20 con 0x7fb67c157eb0 2023-06-22T08:03:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.464+0000 7fb672ffd700 1 -- 172.21.15.23:0/4048830043 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb67400ac10 con 0x7fb67c157eb0 2023-06-22T08:03:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.465+0000 7fb672ffd700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb66006ace0 0x7fb66006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.465+0000 7fb672ffd700 1 -- 172.21.15.23:0/4048830043 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7fb674089800 con 0x7fb67c157eb0 2023-06-22T08:03:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.465+0000 7fb68259b700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb66006ace0 0x7fb66006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.466+0000 7fb68259b700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb66006ace0 0x7fb66006d190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb678007b10 tx=0x7fb678007a60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.470+0000 7fb672ffd700 1 -- 172.21.15.23:0/4048830043 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb6740544c0 con 0x7fb67c157eb0 2023-06-22T08:03:24.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.726+0000 7fb6847ff700 1 -- 172.21.15.23:0/4048830043 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb67c0c4e40 con 0x7fb67c157eb0 2023-06-22T08:03:24.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.727+0000 7fb672ffd700 1 -- 172.21.15.23:0/4048830043 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5503 (secure 0 0 0) 0x7fb674057ae0 con 0x7fb67c157eb0 2023-06-22T08:03:24.731 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:24.731 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:20.673359+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:reconnect","state_seq":55,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:24.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.729+0000 7fb6847ff700 1 -- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb66006ace0 msgr2=0x7fb66006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:24.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.729+0000 7fb6847ff700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb66006ace0 0x7fb66006d190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb678007b10 tx=0x7fb678007a60 comp rx=0 tx=0).stop 2023-06-22T08:03:24.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.729+0000 7fb6847ff700 1 -- 172.21.15.23:0/4048830043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c157eb0 msgr2=0x7fb67c0c39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:24.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.729+0000 7fb6847ff700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c157eb0 0x7fb67c0c39d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb67400cc50 tx=0x7fb67400cf60 comp rx=0 tx=0).stop 2023-06-22T08:03:24.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.729+0000 7fb6847ff700 1 -- 172.21.15.23:0/4048830043 shutdown_connections 2023-06-22T08:03:24.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.729+0000 7fb6847ff700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb67c157eb0 0x7fb67c0c39d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:24.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.729+0000 7fb6847ff700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb66006ace0 0x7fb66006d190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:24.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.729+0000 7fb6847ff700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb67c0cb6b0 0x7fb67c155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:24.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.729+0000 7fb6847ff700 1 --2- 172.21.15.23:0/4048830043 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb67c152e10 0x7fb67c153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:24.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.729+0000 7fb6847ff700 1 -- 172.21.15.23:0/4048830043 >> 172.21.15.23:0/4048830043 conn(0x7fb67c070d80 msgr2=0x7fb67c0c93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:24.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.730+0000 7fb6847ff700 1 -- 172.21.15.23:0/4048830043 shutdown_connections 2023-06-22T08:03:24.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:24.730+0000 7fb6847ff700 1 -- 172.21.15.23:0/4048830043 wait complete. 2023-06-22T08:03:24.735 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2023-06-22T08:03:24.751 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 29, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 29, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4257": { "addr": "172.21.15.67:6837/3798178226", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 3798178226, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 3798178226, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4257, "incarnation": 27, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:reconnect", "state_seq": 55 }, "gid_4269": { "addr": "172.21.15.23:6834/3365097525", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 3365097525, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 3365097525, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4269, "incarnation": 16, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 13 }, "gid_4272": { "addr": "172.21.15.23:6835/3894174037", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 3894174037, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 3894174037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4272, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 12 }, "gid_4279": { "addr": "172.21.15.67:6836/1962314516", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 1962314516, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 1962314516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4285": { "addr": "172.21.15.67:6839/567663545", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 567663545, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 567663545, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:03:20.673359+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4279, "mds_2": 4272, "mds_3": 4269, "mds_4": 4257 } } } ], "standbys": [] } 2023-06-22T08:03:26.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:26.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.960+0000 7f5d19fff700 1 Processor -- start 2023-06-22T08:03:26.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.960+0000 7f5d19fff700 1 -- start start 2023-06-22T08:03:26.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.961+0000 7f5d19fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5d14152ce0 0x7f5d140cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:26.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.961+0000 7f5d19fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d1415c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:26.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.961+0000 7f5d19fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1415c370 con 0x7f5d14158880 2023-06-22T08:03:26.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.961+0000 7f5d19fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d141530b0 con 0x7f5d14152ce0 2023-06-22T08:03:26.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.961+0000 7f5d19fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140cf6f0 con 0x7f5d140c9fd0 2023-06-22T08:03:26.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.961+0000 7f5d13fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5d14158880 0x7f5d14158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:37886/0 (socket says 172.21.15.23:37886) 2023-06-22T08:03:26.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.961+0000 7f5d13fff700 1 -- 172.21.15.23:0/3002646187 learned_addr learned my addr 172.21.15.23:0/3002646187 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:03:26.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.961+0000 7f5d12ffd700 1 --2- 172.21.15.23:0/3002646187 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d1415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:26.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.961+0000 7f5d137fe700 1 --2- 172.21.15.23:0/3002646187 >> v2:172.21.15.67:3300/0 conn(0x7f5d14152ce0 0x7f5d140cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.962+0000 7f5d127fc700 1 -- 172.21.15.23:0/3002646187 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335181422 0 0) 0x7f5d1415c370 con 0x7f5d14158880 2023-06-22T08:03:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.962+0000 7f5d12ffd700 1 -- 172.21.15.23:0/3002646187 >> v2:172.21.15.67:3300/0 conn(0x7f5d14152ce0 msgr2=0x7f5d140cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.962+0000 7f5d12ffd700 1 --2- 172.21.15.23:0/3002646187 >> v2:172.21.15.67:3300/0 conn(0x7f5d14152ce0 0x7f5d140cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.962+0000 7f5d12ffd700 1 -- 172.21.15.23:0/3002646187 >> v1:172.21.15.23:6789/0 conn(0x7f5d14158880 legacy=0x7f5d14158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.962+0000 7f5d12ffd700 1 -- 172.21.15.23:0/3002646187 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1415bda0 con 0x7f5d140c9fd0 2023-06-22T08:03:26.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.962+0000 7f5d137fe700 1 --2- 172.21.15.23:0/3002646187 >> v2:172.21.15.67:3300/0 conn(0x7f5d14152ce0 0x7f5d140cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:03:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.962+0000 7f5d12ffd700 1 --2- 172.21.15.23:0/3002646187 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d1415c800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009840 tx=0x7f5d08009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c43e42b69639c50 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.962+0000 7f5d127fc700 1 -- 172.21.15.23:0/3002646187 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08011830 con 0x7f5d140c9fd0 2023-06-22T08:03:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.963+0000 7f5d127fc700 1 -- 172.21.15.23:0/3002646187 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d08011990 con 0x7f5d140c9fd0 2023-06-22T08:03:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.963+0000 7f5d127fc700 1 -- 172.21.15.23:0/3002646187 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08011d20 con 0x7f5d140c9fd0 2023-06-22T08:03:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.963+0000 7f5d19fff700 1 -- 172.21.15.23:0/3002646187 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 msgr2=0x7f5d1415c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.963+0000 7f5d19fff700 1 --2- 172.21.15.23:0/3002646187 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d1415c800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5d08009840 tx=0x7f5d08009b50 comp rx=0 tx=0).stop 2023-06-22T08:03:26.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.963+0000 7f5d19fff700 1 -- 172.21.15.23:0/3002646187 shutdown_connections 2023-06-22T08:03:26.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.963+0000 7f5d19fff700 1 --2- 172.21.15.23:0/3002646187 >> v2:172.21.15.67:3300/0 conn(0x7f5d14152ce0 0x7f5d140cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:26.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.963+0000 7f5d19fff700 1 --2- 172.21.15.23:0/3002646187 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d1415c800 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:26.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.963+0000 7f5d19fff700 1 -- 172.21.15.23:0/3002646187 >> 172.21.15.23:0/3002646187 conn(0x7f5d1406ae30 msgr2=0x7f5d1406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:26.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.963+0000 7f5d19fff700 1 -- 172.21.15.23:0/3002646187 shutdown_connections 2023-06-22T08:03:26.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.963+0000 7f5d19fff700 1 -- 172.21.15.23:0/3002646187 wait complete. 2023-06-22T08:03:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.965+0000 7f5d19fff700 1 Processor -- start 2023-06-22T08:03:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.965+0000 7f5d19fff700 1 -- start start 2023-06-22T08:03:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d19fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d141485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d19fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d14152ce0 0x7f5d141eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d19fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5d14158880 0x7f5d141490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:26.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d19fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d141ee660 con 0x7f5d14158880 2023-06-22T08:03:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d19fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d14148b10 con 0x7f5d14152ce0 2023-06-22T08:03:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d19fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d140b4360 con 0x7f5d140c9fd0 2023-06-22T08:03:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d137fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d141485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d137fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d141485d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:55716/0 (socket says 172.21.15.23:55716) 2023-06-22T08:03:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d137fe700 1 -- 172.21.15.23:0/2120314723 learned_addr learned my addr 172.21.15.23:0/2120314723 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d12ffd700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d14152ce0 0x7f5d141eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.966+0000 7f5d13fff700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5d14158880 0x7f5d141490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:26.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5d137fe700 1 -- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d14152ce0 msgr2=0x7f5d141eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5d137fe700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d14152ce0 0x7f5d141eeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5d137fe700 1 -- 172.21.15.23:0/2120314723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5d14158880 msgr2=0x7f5d141490d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5d137fe700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5d14158880 0x7f5d141490d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5d137fe700 1 -- 172.21.15.23:0/2120314723 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d040094a0 con 0x7f5d140c9fd0 2023-06-22T08:03:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5d13fff700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5d14158880 0x7f5d141490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:03:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5d137fe700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d141485d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400ebe0 tx=0x7f5d0400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:26.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5cfbfff700 1 -- 172.21.15.23:0/2120314723 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04009800 con 0x7f5d140c9fd0 2023-06-22T08:03:26.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5cfbfff700 1 -- 172.21.15.23:0/2120314723 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0400c910 con 0x7f5d140c9fd0 2023-06-22T08:03:26.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5cfbfff700 1 -- 172.21.15.23:0/2120314723 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0400cc80 con 0x7f5d140c9fd0 2023-06-22T08:03:26.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5d19fff700 1 -- 172.21.15.23:0/2120314723 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d080094a0 con 0x7f5d140c9fd0 2023-06-22T08:03:26.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.967+0000 7f5d19fff700 1 -- 172.21.15.23:0/2120314723 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d140b3da0 con 0x7f5d140c9fd0 2023-06-22T08:03:26.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.969+0000 7f5d19fff700 1 -- 172.21.15.23:0/2120314723 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d141ee940 con 0x7f5d140c9fd0 2023-06-22T08:03:26.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.970+0000 7f5cfbfff700 1 -- 172.21.15.23:0/2120314723 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5d04005a20 con 0x7f5d140c9fd0 2023-06-22T08:03:26.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.970+0000 7f5cfbfff700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5d0006ace0 0x7f5d0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:26.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.971+0000 7f5cfbfff700 1 -- 172.21.15.23:0/2120314723 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f5d04089af0 con 0x7f5d140c9fd0 2023-06-22T08:03:26.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.971+0000 7f5d12ffd700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5d0006ace0 0x7f5d0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:26.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.971+0000 7f5d12ffd700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5d0006ace0 0x7f5d0006d190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5d080090b0 tx=0x7f5d0801c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:26.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:26.974+0000 7f5cfbfff700 1 -- 172.21.15.23:0/2120314723 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5d040582c0 con 0x7f5d140c9fd0 2023-06-22T08:03:27.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.230+0000 7f5d19fff700 1 -- 172.21.15.23:0/2120314723 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d140b4710 con 0x7f5d140c9fd0 2023-06-22T08:03:27.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.231+0000 7f5cfbfff700 1 -- 172.21.15.23:0/2120314723 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5500 (secure 0 0 0) 0x7f5d04021030 con 0x7f5d140c9fd0 2023-06-22T08:03:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:27.235 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:26.298279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:rejoin","state_seq":57,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.233+0000 7f5d19fff700 1 -- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5d0006ace0 msgr2=0x7f5d0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.233+0000 7f5d19fff700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5d0006ace0 0x7f5d0006d190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5d080090b0 tx=0x7f5d0801c040 comp rx=0 tx=0).stop 2023-06-22T08:03:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.233+0000 7f5d19fff700 1 -- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 msgr2=0x7f5d141485d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.233+0000 7f5d19fff700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d141485d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400ebe0 tx=0x7f5d0400eef0 comp rx=0 tx=0).stop 2023-06-22T08:03:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.234+0000 7f5d19fff700 1 -- 172.21.15.23:0/2120314723 shutdown_connections 2023-06-22T08:03:27.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.234+0000 7f5d19fff700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5d14158880 0x7f5d141490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.234+0000 7f5d19fff700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5d140c9fd0 0x7f5d141485d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.234+0000 7f5d19fff700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5d0006ace0 0x7f5d0006d190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.234+0000 7f5d19fff700 1 --2- 172.21.15.23:0/2120314723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d14152ce0 0x7f5d141eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.234+0000 7f5d19fff700 1 -- 172.21.15.23:0/2120314723 >> 172.21.15.23:0/2120314723 conn(0x7f5d1406ae30 msgr2=0x7f5d140c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.234+0000 7f5d19fff700 1 -- 172.21.15.23:0/2120314723 shutdown_connections 2023-06-22T08:03:27.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:27.234+0000 7f5d19fff700 1 -- 172.21.15.23:0/2120314723 wait complete. 2023-06-22T08:03:27.239 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2023-06-22T08:03:27.255 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:03:29.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:29.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.467+0000 7ff3cedff700 1 Processor -- start 2023-06-22T08:03:29.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.467+0000 7ff3cedff700 1 -- start start 2023-06-22T08:03:29.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.468+0000 7ff3cedff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff3c80c3a40 0x7ff3c80c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:29.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cedff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff3c815a320 0x7ff3c8160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:29.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cedff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c80c6a60 con 0x7ff3c806d050 2023-06-22T08:03:29.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cedff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c8157570 con 0x7ff3c815a320 2023-06-22T08:03:29.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cedff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c80c4350 con 0x7ff3c80c3a40 2023-06-22T08:03:29.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cd39c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff3c815a320 0x7ff3c8160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:29.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3ccb9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff3c80c3a40 0x7ff3c80c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:29.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cd39c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff3c815a320 0x7ff3c8160ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39620/0 (socket says 172.21.15.23:39620) 2023-06-22T08:03:29.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cd39c700 1 -- 172.21.15.23:0/2385590131 learned_addr learned my addr 172.21.15.23:0/2385590131 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:29.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cd39c700 1 -- 172.21.15.23:0/2385590131 >> v2:172.21.15.67:3300/0 conn(0x7ff3c80c3a40 msgr2=0x7ff3c80c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:29.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cd39c700 1 --2- 172.21.15.23:0/2385590131 >> v2:172.21.15.67:3300/0 conn(0x7ff3c80c3a40 0x7ff3c80c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cd39c700 1 -- 172.21.15.23:0/2385590131 >> v1:172.21.15.67:6789/0 conn(0x7ff3c806d050 legacy=0x7ff3c8159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.469+0000 7ff3cd39c700 1 -- 172.21.15.23:0/2385590131 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c814d690 con 0x7ff3c815a320 2023-06-22T08:03:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.470+0000 7ff3cd39c700 1 --2- 172.21.15.23:0/2385590131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c815a320 0x7ff3c8160ec0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff3c806af80 tx=0x7ff3c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3559b114d18ba918 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.470+0000 7ff3c77fe700 1 -- 172.21.15.23:0/2385590131 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000e070 con 0x7ff3c815a320 2023-06-22T08:03:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.470+0000 7ff3c77fe700 1 -- 172.21.15.23:0/2385590131 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c000b910 con 0x7ff3c815a320 2023-06-22T08:03:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.470+0000 7ff3c77fe700 1 -- 172.21.15.23:0/2385590131 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000bcc0 con 0x7ff3c815a320 2023-06-22T08:03:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.470+0000 7ff3cedff700 1 -- 172.21.15.23:0/2385590131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c815a320 msgr2=0x7ff3c8160ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:29.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.470+0000 7ff3cedff700 1 --2- 172.21.15.23:0/2385590131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c815a320 0x7ff3c8160ec0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff3c806af80 tx=0x7ff3c000b330 comp rx=0 tx=0).stop 2023-06-22T08:03:29.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.471+0000 7ff3cedff700 1 -- 172.21.15.23:0/2385590131 shutdown_connections 2023-06-22T08:03:29.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.471+0000 7ff3cedff700 1 --2- 172.21.15.23:0/2385590131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c815a320 0x7ff3c8160ec0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:29.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.471+0000 7ff3cedff700 1 --2- 172.21.15.23:0/2385590131 >> v2:172.21.15.67:3300/0 conn(0x7ff3c80c3a40 0x7ff3c80c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:29.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.471+0000 7ff3cedff700 1 -- 172.21.15.23:0/2385590131 >> 172.21.15.23:0/2385590131 conn(0x7ff3c80ca3f0 msgr2=0x7ff3c80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:29.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.471+0000 7ff3cedff700 1 -- 172.21.15.23:0/2385590131 shutdown_connections 2023-06-22T08:03:29.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.471+0000 7ff3cedff700 1 -- 172.21.15.23:0/2385590131 wait complete. 2023-06-22T08:03:29.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.473+0000 7ff3cedff700 1 Processor -- start 2023-06-22T08:03:29.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.473+0000 7ff3cedff700 1 -- start start 2023-06-22T08:03:29.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3cedff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c806d050 0x7ff3c81f1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:29.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3cedff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3c80c3a40 0x7ff3c8145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:29.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3cedff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c815a320 0x7ff3c813fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:29.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3cedff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c8144a60 con 0x7ff3c815a320 2023-06-22T08:03:29.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3cedff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c81451d0 con 0x7ff3c806d050 2023-06-22T08:03:29.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3cedff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c80c7250 con 0x7ff3c80c3a40 2023-06-22T08:03:29.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3ccb9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c806d050 0x7ff3c81f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:29.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3ccb9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c806d050 0x7ff3c81f1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:46420/0 (socket says 172.21.15.23:46420) 2023-06-22T08:03:29.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3ccb9b700 1 -- 172.21.15.23:0/1060860040 learned_addr learned my addr 172.21.15.23:0/1060860040 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:29.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3cd39c700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c815a320 0x7ff3c813fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:29.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3c7fff700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3c80c3a40 0x7ff3c8145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:29.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3ccb9b700 1 -- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3c80c3a40 msgr2=0x7ff3c8145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3ccb9b700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3c80c3a40 0x7ff3c8145940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3ccb9b700 1 -- 172.21.15.23:0/1060860040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c815a320 msgr2=0x7ff3c813fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3ccb9b700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c815a320 0x7ff3c813fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3ccb9b700 1 -- 172.21.15.23:0/1060860040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0009b30 con 0x7ff3c806d050 2023-06-22T08:03:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.474+0000 7ff3cd39c700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c815a320 0x7ff3c813fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:03:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.475+0000 7ff3ccb9b700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c806d050 0x7ff3c81f1800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00e9b0 tx=0x7ff3bc00ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.475+0000 7ff3c57fa700 1 -- 172.21.15.23:0/1060860040 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc009710 con 0x7ff3c806d050 2023-06-22T08:03:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.475+0000 7ff3c57fa700 1 -- 172.21.15.23:0/1060860040 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc00ee50 con 0x7ff3c806d050 2023-06-22T08:03:29.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.475+0000 7ff3c57fa700 1 -- 172.21.15.23:0/1060860040 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc00ca50 con 0x7ff3c806d050 2023-06-22T08:03:29.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.475+0000 7ff3cedff700 1 -- 172.21.15.23:0/1060860040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c81412e0 con 0x7ff3c806d050 2023-06-22T08:03:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.475+0000 7ff3cedff700 1 -- 172.21.15.23:0/1060860040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c81f23f0 con 0x7ff3c806d050 2023-06-22T08:03:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.476+0000 7ff3cedff700 1 -- 172.21.15.23:0/1060860040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c804ea20 con 0x7ff3c806d050 2023-06-22T08:03:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.477+0000 7ff3c57fa700 1 -- 172.21.15.23:0/1060860040 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff3bc00cbb0 con 0x7ff3c806d050 2023-06-22T08:03:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.478+0000 7ff3c57fa700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff3ac06ace0 0x7ff3ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.478+0000 7ff3c57fa700 1 -- 172.21.15.23:0/1060860040 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7ff3bc0897d0 con 0x7ff3c806d050 2023-06-22T08:03:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.479+0000 7ff3c7fff700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff3ac06ace0 0x7ff3ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:29.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.479+0000 7ff3c7fff700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff3ac06ace0 0x7ff3ac06d190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80c7920 tx=0x7ff3b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:29.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.483+0000 7ff3c57fa700 1 -- 172.21.15.23:0/1060860040 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff3bc057fb0 con 0x7ff3c806d050 2023-06-22T08:03:29.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.738+0000 7ff3cedff700 1 -- 172.21.15.23:0/1060860040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3c80c5f30 con 0x7ff3c806d050 2023-06-22T08:03:29.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.739+0000 7ff3c57fa700 1 -- 172.21.15.23:0/1060860040 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5500 (secure 0 0 0) 0x7ff3bc00ce60 con 0x7ff3c806d050 2023-06-22T08:03:29.743 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:29.743 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:26.298279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:rejoin","state_seq":57,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:29.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 -- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff3ac06ace0 msgr2=0x7ff3ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:29.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff3ac06ace0 0x7ff3ac06d190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80c7920 tx=0x7ff3b4009040 comp rx=0 tx=0).stop 2023-06-22T08:03:29.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 -- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c806d050 msgr2=0x7ff3c81f1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:29.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c806d050 0x7ff3c81f1800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc00e9b0 tx=0x7ff3bc00ecc0 comp rx=0 tx=0).stop 2023-06-22T08:03:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 -- 172.21.15.23:0/1060860040 shutdown_connections 2023-06-22T08:03:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c815a320 0x7ff3c813fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff3ac06ace0 0x7ff3ac06d190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c806d050 0x7ff3c81f1800 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 --2- 172.21.15.23:0/1060860040 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3c80c3a40 0x7ff3c8145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 -- 172.21.15.23:0/1060860040 >> 172.21.15.23:0/1060860040 conn(0x7ff3c80ca3f0 msgr2=0x7ff3c80710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 -- 172.21.15.23:0/1060860040 shutdown_connections 2023-06-22T08:03:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:29.742+0000 7ff3cedff700 1 -- 172.21.15.23:0/1060860040 wait complete. 2023-06-22T08:03:29.747 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2023-06-22T08:03:29.763 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:03:31.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/11' 2023-06-22T08:03:31.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/16' 2023-06-22T08:03:31.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/32' 2023-06-22T08:03:31.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/30' 2023-06-22T08:03:31.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/14' 2023-06-22T08:03:31.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/18' 2023-06-22T08:03:31.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/13' 2023-06-22T08:03:31.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/33' 2023-06-22T08:03:31.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/17' 2023-06-22T08:03:31.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/35' 2023-06-22T08:03:31.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './167/37' 2023-06-22T08:03:31.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './167' 2023-06-22T08:03:31.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/19' 2023-06-22T08:03:31.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/15' 2023-06-22T08:03:31.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/31' 2023-06-22T08:03:31.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/36' 2023-06-22T08:03:31.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/20' 2023-06-22T08:03:31.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/12' 2023-06-22T08:03:31.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/39' 2023-06-22T08:03:31.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/10' 2023-06-22T08:03:31.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/34' 2023-06-22T08:03:31.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/38' 2023-06-22T08:03:31.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/11' 2023-06-22T08:03:31.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/16' 2023-06-22T08:03:31.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/32' 2023-06-22T08:03:31.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/30' 2023-06-22T08:03:31.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/14' 2023-06-22T08:03:31.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/18' 2023-06-22T08:03:31.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/13' 2023-06-22T08:03:31.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/33' 2023-06-22T08:03:31.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/17' 2023-06-22T08:03:31.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/35' 2023-06-22T08:03:31.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './168/37' 2023-06-22T08:03:31.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './168' 2023-06-22T08:03:31.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/19' 2023-06-22T08:03:31.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/15' 2023-06-22T08:03:31.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/31' 2023-06-22T08:03:31.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/36' 2023-06-22T08:03:31.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/20' 2023-06-22T08:03:31.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/12' 2023-06-22T08:03:31.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/39' 2023-06-22T08:03:31.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/10' 2023-06-22T08:03:31.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/34' 2023-06-22T08:03:31.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/38' 2023-06-22T08:03:31.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/11' 2023-06-22T08:03:31.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/16' 2023-06-22T08:03:31.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/32' 2023-06-22T08:03:31.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/30' 2023-06-22T08:03:31.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/14' 2023-06-22T08:03:31.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/18' 2023-06-22T08:03:31.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/13' 2023-06-22T08:03:31.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/33' 2023-06-22T08:03:31.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/17' 2023-06-22T08:03:31.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/35' 2023-06-22T08:03:31.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './169/37' 2023-06-22T08:03:31.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './169' 2023-06-22T08:03:31.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/19' 2023-06-22T08:03:31.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/15' 2023-06-22T08:03:31.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/31' 2023-06-22T08:03:31.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/36' 2023-06-22T08:03:31.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/20' 2023-06-22T08:03:31.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/12' 2023-06-22T08:03:31.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/39' 2023-06-22T08:03:31.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/10' 2023-06-22T08:03:31.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/34' 2023-06-22T08:03:31.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/38' 2023-06-22T08:03:31.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/11' 2023-06-22T08:03:31.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/16' 2023-06-22T08:03:31.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/32' 2023-06-22T08:03:31.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/30' 2023-06-22T08:03:31.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/14' 2023-06-22T08:03:31.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/18' 2023-06-22T08:03:31.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/13' 2023-06-22T08:03:31.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/33' 2023-06-22T08:03:31.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/17' 2023-06-22T08:03:31.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/35' 2023-06-22T08:03:31.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed './170/37' 2023-06-22T08:03:31.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './170' 2023-06-22T08:03:31.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/19' 2023-06-22T08:03:31.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/15' 2023-06-22T08:03:31.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/31' 2023-06-22T08:03:31.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/36' 2023-06-22T08:03:31.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/20' 2023-06-22T08:03:31.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/12' 2023-06-22T08:03:31.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/39' 2023-06-22T08:03:31.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/10' 2023-06-22T08:03:31.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/34' 2023-06-22T08:03:31.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/38' 2023-06-22T08:03:31.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/11' 2023-06-22T08:03:31.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/16' 2023-06-22T08:03:31.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/32' 2023-06-22T08:03:31.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/30' 2023-06-22T08:03:31.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/14' 2023-06-22T08:03:31.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/18' 2023-06-22T08:03:31.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/13' 2023-06-22T08:03:31.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/33' 2023-06-22T08:03:31.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/17' 2023-06-22T08:03:31.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/35' 2023-06-22T08:03:31.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './171/37' 2023-06-22T08:03:31.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './171' 2023-06-22T08:03:31.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/19' 2023-06-22T08:03:31.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/15' 2023-06-22T08:03:31.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/31' 2023-06-22T08:03:31.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/36' 2023-06-22T08:03:31.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/20' 2023-06-22T08:03:31.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/12' 2023-06-22T08:03:31.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/39' 2023-06-22T08:03:31.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/10' 2023-06-22T08:03:31.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/34' 2023-06-22T08:03:31.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/38' 2023-06-22T08:03:31.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/11' 2023-06-22T08:03:31.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/16' 2023-06-22T08:03:31.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/32' 2023-06-22T08:03:31.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/30' 2023-06-22T08:03:31.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/14' 2023-06-22T08:03:31.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/18' 2023-06-22T08:03:31.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/13' 2023-06-22T08:03:31.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/33' 2023-06-22T08:03:31.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/17' 2023-06-22T08:03:31.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/35' 2023-06-22T08:03:31.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './172/37' 2023-06-22T08:03:31.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './172' 2023-06-22T08:03:31.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/19' 2023-06-22T08:03:31.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/15' 2023-06-22T08:03:31.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/31' 2023-06-22T08:03:31.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/36' 2023-06-22T08:03:31.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/20' 2023-06-22T08:03:31.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/12' 2023-06-22T08:03:31.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/39' 2023-06-22T08:03:31.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/10' 2023-06-22T08:03:31.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/34' 2023-06-22T08:03:31.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/38' 2023-06-22T08:03:31.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/11' 2023-06-22T08:03:31.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/16' 2023-06-22T08:03:31.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/32' 2023-06-22T08:03:31.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/30' 2023-06-22T08:03:31.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/14' 2023-06-22T08:03:31.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/18' 2023-06-22T08:03:31.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/13' 2023-06-22T08:03:31.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/33' 2023-06-22T08:03:31.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/17' 2023-06-22T08:03:31.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/35' 2023-06-22T08:03:31.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './173/37' 2023-06-22T08:03:31.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './173' 2023-06-22T08:03:31.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/19' 2023-06-22T08:03:31.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/15' 2023-06-22T08:03:31.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/31' 2023-06-22T08:03:31.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/36' 2023-06-22T08:03:31.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/20' 2023-06-22T08:03:31.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/12' 2023-06-22T08:03:31.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/39' 2023-06-22T08:03:31.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/10' 2023-06-22T08:03:31.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/34' 2023-06-22T08:03:31.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/38' 2023-06-22T08:03:31.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/11' 2023-06-22T08:03:31.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/16' 2023-06-22T08:03:31.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/32' 2023-06-22T08:03:31.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/30' 2023-06-22T08:03:31.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/14' 2023-06-22T08:03:31.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/18' 2023-06-22T08:03:31.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/13' 2023-06-22T08:03:31.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/33' 2023-06-22T08:03:31.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/17' 2023-06-22T08:03:31.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/35' 2023-06-22T08:03:31.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './174/37' 2023-06-22T08:03:31.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './174' 2023-06-22T08:03:31.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/19' 2023-06-22T08:03:31.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/15' 2023-06-22T08:03:31.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/31' 2023-06-22T08:03:31.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/36' 2023-06-22T08:03:31.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/20' 2023-06-22T08:03:31.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/12' 2023-06-22T08:03:31.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/39' 2023-06-22T08:03:31.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/10' 2023-06-22T08:03:31.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/34' 2023-06-22T08:03:31.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/38' 2023-06-22T08:03:31.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/11' 2023-06-22T08:03:31.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/16' 2023-06-22T08:03:31.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/32' 2023-06-22T08:03:31.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/30' 2023-06-22T08:03:31.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/14' 2023-06-22T08:03:31.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/18' 2023-06-22T08:03:31.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/13' 2023-06-22T08:03:31.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/33' 2023-06-22T08:03:31.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/17' 2023-06-22T08:03:31.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/35' 2023-06-22T08:03:31.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './175/37' 2023-06-22T08:03:31.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './175' 2023-06-22T08:03:31.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/19' 2023-06-22T08:03:31.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/15' 2023-06-22T08:03:31.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/31' 2023-06-22T08:03:31.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/36' 2023-06-22T08:03:31.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/20' 2023-06-22T08:03:31.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/12' 2023-06-22T08:03:31.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/39' 2023-06-22T08:03:31.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/10' 2023-06-22T08:03:31.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/34' 2023-06-22T08:03:31.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/38' 2023-06-22T08:03:31.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/11' 2023-06-22T08:03:31.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/16' 2023-06-22T08:03:31.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/32' 2023-06-22T08:03:31.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/30' 2023-06-22T08:03:31.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/14' 2023-06-22T08:03:31.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/18' 2023-06-22T08:03:31.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/13' 2023-06-22T08:03:31.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/33' 2023-06-22T08:03:31.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/17' 2023-06-22T08:03:31.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/35' 2023-06-22T08:03:31.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './176/37' 2023-06-22T08:03:31.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './176' 2023-06-22T08:03:31.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/19' 2023-06-22T08:03:31.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/15' 2023-06-22T08:03:31.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:32.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:31.999+0000 7fd99cdff700 1 Processor -- start 2023-06-22T08:03:32.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:31.999+0000 7fd99cdff700 1 -- start start 2023-06-22T08:03:32.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd99cdff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd9980c3a40 0x7fd9980c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:32.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd99cdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 0x7fd998159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:32.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd99cdff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd998156fc0 0x7fd99815a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:32.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd99cdff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd998157570 con 0x7fd9980c3a40 2023-06-22T08:03:32.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd99cdff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9980c4350 con 0x7fd998156fc0 2023-06-22T08:03:32.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd99cdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9980c6a60 con 0x7fd99806d050 2023-06-22T08:03:32.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd99659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd9980c3a40 0x7fd9980c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:32.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd99659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd9980c3a40 0x7fd9980c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39630/0 (socket says 172.21.15.23:39630) 2023-06-22T08:03:32.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd99659c700 1 -- 172.21.15.23:0/2663201413 learned_addr learned my addr 172.21.15.23:0/2663201413 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:32.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd98ffff700 1 --2- 172.21.15.23:0/2663201413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 0x7fd998159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:32.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.001+0000 7fd996d9d700 1 --2- 172.21.15.23:0/2663201413 >> v2:172.21.15.67:3300/0 conn(0x7fd998156fc0 0x7fd99815a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:32.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.002+0000 7fd99659c700 1 -- 172.21.15.23:0/2663201413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 msgr2=0x7fd998159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:32.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.002+0000 7fd99659c700 1 --2- 172.21.15.23:0/2663201413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 0x7fd998159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.002+0000 7fd99659c700 1 -- 172.21.15.23:0/2663201413 >> v2:172.21.15.67:3300/0 conn(0x7fd998156fc0 msgr2=0x7fd99815a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:32.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.002+0000 7fd99659c700 1 --2- 172.21.15.23:0/2663201413 >> v2:172.21.15.67:3300/0 conn(0x7fd998156fc0 0x7fd99815a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.002+0000 7fd99659c700 1 -- 172.21.15.23:0/2663201413 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99814d690 con 0x7fd9980c3a40 2023-06-22T08:03:32.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.002+0000 7fd99659c700 1 --2- 172.21.15.23:0/2663201413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9980c3a40 0x7fd9980c3e10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd99806e5f0 tx=0x7fd988009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10aa21bd60720c06 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:32.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.002+0000 7fd995d9b700 1 -- 172.21.15.23:0/2663201413 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd988011830 con 0x7fd9980c3a40 2023-06-22T08:03:32.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.002+0000 7fd995d9b700 1 -- 172.21.15.23:0/2663201413 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd988011990 con 0x7fd9980c3a40 2023-06-22T08:03:32.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.002+0000 7fd995d9b700 1 -- 172.21.15.23:0/2663201413 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd988011d20 con 0x7fd9980c3a40 2023-06-22T08:03:32.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.003+0000 7fd99cdff700 1 -- 172.21.15.23:0/2663201413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9980c3a40 msgr2=0x7fd9980c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:32.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.003+0000 7fd99cdff700 1 --2- 172.21.15.23:0/2663201413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9980c3a40 0x7fd9980c3e10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd99806e5f0 tx=0x7fd988009bd0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.003+0000 7fd99cdff700 1 -- 172.21.15.23:0/2663201413 shutdown_connections 2023-06-22T08:03:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.003+0000 7fd99cdff700 1 --2- 172.21.15.23:0/2663201413 >> v2:172.21.15.67:3300/0 conn(0x7fd998156fc0 0x7fd99815a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.003+0000 7fd99cdff700 1 --2- 172.21.15.23:0/2663201413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9980c3a40 0x7fd9980c3e10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.003+0000 7fd99cdff700 1 --2- 172.21.15.23:0/2663201413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 0x7fd998159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.003+0000 7fd99cdff700 1 -- 172.21.15.23:0/2663201413 >> 172.21.15.23:0/2663201413 conn(0x7fd9980ca3f0 msgr2=0x7fd9980ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.003+0000 7fd99cdff700 1 -- 172.21.15.23:0/2663201413 shutdown_connections 2023-06-22T08:03:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.003+0000 7fd99cdff700 1 -- 172.21.15.23:0/2663201413 wait complete. 2023-06-22T08:03:32.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.006+0000 7fd99cdff700 1 Processor -- start 2023-06-22T08:03:32.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.006+0000 7fd99cdff700 1 -- start start 2023-06-22T08:03:32.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd99cdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 0x7fd99813dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:32.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd99cdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9980c3a40 0x7fd99813ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:32.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd99cdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd998156fc0 0x7fd9981f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:32.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd99cdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9981f2820 con 0x7fd998156fc0 2023-06-22T08:03:32.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd99cdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99813eb20 con 0x7fd9980c3a40 2023-06-22T08:03:32.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd99cdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9981594e0 con 0x7fd99806d050 2023-06-22T08:03:32.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd996d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd998156fc0 0x7fd9981f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:32.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd996d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd998156fc0 0x7fd9981f1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39646/0 (socket says 172.21.15.23:39646) 2023-06-22T08:03:32.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd996d9d700 1 -- 172.21.15.23:0/3019475654 learned_addr learned my addr 172.21.15.23:0/3019475654 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:32.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd99659c700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 0x7fd99813dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:32.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.007+0000 7fd98ffff700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9980c3a40 0x7fd99813ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:32.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.008+0000 7fd99659c700 1 -- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9980c3a40 msgr2=0x7fd99813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:32.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.008+0000 7fd99659c700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9980c3a40 0x7fd99813ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.008+0000 7fd99659c700 1 -- 172.21.15.23:0/3019475654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd998156fc0 msgr2=0x7fd9981f1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:32.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.008+0000 7fd99659c700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd998156fc0 0x7fd9981f1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.008+0000 7fd99659c700 1 -- 172.21.15.23:0/3019475654 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd988009550 con 0x7fd99806d050 2023-06-22T08:03:32.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.008+0000 7fd99659c700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 0x7fd99813dbf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd99806af80 tx=0x7fd988004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:32.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.008+0000 7fd98effd700 1 -- 172.21.15.23:0/3019475654 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd988004450 con 0x7fd99806d050 2023-06-22T08:03:32.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.008+0000 7fd98effd700 1 -- 172.21.15.23:0/3019475654 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9880045d0 con 0x7fd99806d050 2023-06-22T08:03:32.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.008+0000 7fd99cdff700 1 -- 172.21.15.23:0/3019475654 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd998157700 con 0x7fd99806d050 2023-06-22T08:03:32.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.008+0000 7fd98effd700 1 -- 172.21.15.23:0/3019475654 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9880038e0 con 0x7fd99806d050 2023-06-22T08:03:32.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.009+0000 7fd99cdff700 1 -- 172.21.15.23:0/3019475654 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9980c94f0 con 0x7fd99806d050 2023-06-22T08:03:32.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.010+0000 7fd99cdff700 1 -- 172.21.15.23:0/3019475654 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99804ea20 con 0x7fd99806d050 2023-06-22T08:03:32.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.011+0000 7fd98effd700 1 -- 172.21.15.23:0/3019475654 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd988003a40 con 0x7fd99806d050 2023-06-22T08:03:32.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.012+0000 7fd98effd700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd97806ace0 0x7fd97806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:32.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.012+0000 7fd98effd700 1 -- 172.21.15.23:0/3019475654 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7fd988021030 con 0x7fd99806d050 2023-06-22T08:03:32.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.012+0000 7fd98ffff700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd97806ace0 0x7fd97806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:32.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.013+0000 7fd98ffff700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd97806ace0 0x7fd97806d190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd9981453c0 tx=0x7fd980009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:32.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.016+0000 7fd98effd700 1 -- 172.21.15.23:0/3019475654 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd9880558e0 con 0x7fd99806d050 2023-06-22T08:03:32.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/31' 2023-06-22T08:03:32.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/36' 2023-06-22T08:03:32.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/20' 2023-06-22T08:03:32.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/12' 2023-06-22T08:03:32.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/39' 2023-06-22T08:03:32.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/10' 2023-06-22T08:03:32.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/34' 2023-06-22T08:03:32.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/38' 2023-06-22T08:03:32.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/11' 2023-06-22T08:03:32.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/16' 2023-06-22T08:03:32.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/32' 2023-06-22T08:03:32.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/30' 2023-06-22T08:03:32.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/14' 2023-06-22T08:03:32.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/18' 2023-06-22T08:03:32.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/13' 2023-06-22T08:03:32.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/33' 2023-06-22T08:03:32.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/17' 2023-06-22T08:03:32.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/35' 2023-06-22T08:03:32.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed './177/37' 2023-06-22T08:03:32.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './177' 2023-06-22T08:03:32.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/19' 2023-06-22T08:03:32.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/15' 2023-06-22T08:03:32.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/31' 2023-06-22T08:03:32.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/36' 2023-06-22T08:03:32.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/20' 2023-06-22T08:03:32.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/12' 2023-06-22T08:03:32.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/39' 2023-06-22T08:03:32.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/10' 2023-06-22T08:03:32.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/34' 2023-06-22T08:03:32.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/38' 2023-06-22T08:03:32.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/11' 2023-06-22T08:03:32.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/16' 2023-06-22T08:03:32.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/32' 2023-06-22T08:03:32.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/30' 2023-06-22T08:03:32.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/14' 2023-06-22T08:03:32.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/18' 2023-06-22T08:03:32.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/13' 2023-06-22T08:03:32.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/33' 2023-06-22T08:03:32.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/17' 2023-06-22T08:03:32.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/35' 2023-06-22T08:03:32.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed './178/37' 2023-06-22T08:03:32.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './178' 2023-06-22T08:03:32.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/19' 2023-06-22T08:03:32.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/15' 2023-06-22T08:03:32.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/31' 2023-06-22T08:03:32.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/36' 2023-06-22T08:03:32.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/20' 2023-06-22T08:03:32.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/12' 2023-06-22T08:03:32.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/39' 2023-06-22T08:03:32.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/10' 2023-06-22T08:03:32.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/34' 2023-06-22T08:03:32.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/38' 2023-06-22T08:03:32.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/11' 2023-06-22T08:03:32.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/16' 2023-06-22T08:03:32.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/32' 2023-06-22T08:03:32.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/30' 2023-06-22T08:03:32.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/14' 2023-06-22T08:03:32.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/18' 2023-06-22T08:03:32.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/13' 2023-06-22T08:03:32.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/33' 2023-06-22T08:03:32.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/17' 2023-06-22T08:03:32.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/35' 2023-06-22T08:03:32.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed './179/37' 2023-06-22T08:03:32.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './179' 2023-06-22T08:03:32.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/19' 2023-06-22T08:03:32.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/15' 2023-06-22T08:03:32.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/31' 2023-06-22T08:03:32.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/36' 2023-06-22T08:03:32.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/20' 2023-06-22T08:03:32.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/12' 2023-06-22T08:03:32.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/39' 2023-06-22T08:03:32.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/10' 2023-06-22T08:03:32.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/34' 2023-06-22T08:03:32.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/38' 2023-06-22T08:03:32.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/11' 2023-06-22T08:03:32.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/16' 2023-06-22T08:03:32.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/32' 2023-06-22T08:03:32.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/30' 2023-06-22T08:03:32.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/14' 2023-06-22T08:03:32.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/18' 2023-06-22T08:03:32.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/13' 2023-06-22T08:03:32.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/33' 2023-06-22T08:03:32.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/17' 2023-06-22T08:03:32.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/35' 2023-06-22T08:03:32.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed './180/37' 2023-06-22T08:03:32.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './180' 2023-06-22T08:03:32.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/19' 2023-06-22T08:03:32.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/15' 2023-06-22T08:03:32.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/31' 2023-06-22T08:03:32.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/36' 2023-06-22T08:03:32.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/20' 2023-06-22T08:03:32.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/12' 2023-06-22T08:03:32.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/39' 2023-06-22T08:03:32.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/10' 2023-06-22T08:03:32.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/34' 2023-06-22T08:03:32.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/38' 2023-06-22T08:03:32.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/11' 2023-06-22T08:03:32.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/16' 2023-06-22T08:03:32.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/32' 2023-06-22T08:03:32.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/30' 2023-06-22T08:03:32.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/14' 2023-06-22T08:03:32.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/18' 2023-06-22T08:03:32.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/13' 2023-06-22T08:03:32.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/33' 2023-06-22T08:03:32.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/17' 2023-06-22T08:03:32.167 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/35' 2023-06-22T08:03:32.167 INFO:tasks.workunit.client.0.smithi023.stdout:removed './181/37' 2023-06-22T08:03:32.167 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './181' 2023-06-22T08:03:32.167 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/19' 2023-06-22T08:03:32.167 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/15' 2023-06-22T08:03:32.167 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/31' 2023-06-22T08:03:32.167 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/36' 2023-06-22T08:03:32.167 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/20' 2023-06-22T08:03:32.168 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/12' 2023-06-22T08:03:32.168 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/39' 2023-06-22T08:03:32.168 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/10' 2023-06-22T08:03:32.168 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/34' 2023-06-22T08:03:32.168 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/38' 2023-06-22T08:03:32.168 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/11' 2023-06-22T08:03:32.168 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/16' 2023-06-22T08:03:32.169 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/32' 2023-06-22T08:03:32.169 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/30' 2023-06-22T08:03:32.169 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/14' 2023-06-22T08:03:32.169 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/18' 2023-06-22T08:03:32.169 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/13' 2023-06-22T08:03:32.169 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/33' 2023-06-22T08:03:32.169 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/17' 2023-06-22T08:03:32.169 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/35' 2023-06-22T08:03:32.170 INFO:tasks.workunit.client.0.smithi023.stdout:removed './182/37' 2023-06-22T08:03:32.170 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './182' 2023-06-22T08:03:32.170 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/19' 2023-06-22T08:03:32.170 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/15' 2023-06-22T08:03:32.170 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/31' 2023-06-22T08:03:32.170 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/36' 2023-06-22T08:03:32.170 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/20' 2023-06-22T08:03:32.170 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/12' 2023-06-22T08:03:32.171 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/39' 2023-06-22T08:03:32.171 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/10' 2023-06-22T08:03:32.171 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/34' 2023-06-22T08:03:32.171 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/38' 2023-06-22T08:03:32.171 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/11' 2023-06-22T08:03:32.171 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/16' 2023-06-22T08:03:32.171 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/32' 2023-06-22T08:03:32.171 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/30' 2023-06-22T08:03:32.172 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/14' 2023-06-22T08:03:32.172 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/18' 2023-06-22T08:03:32.172 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/13' 2023-06-22T08:03:32.172 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/33' 2023-06-22T08:03:32.172 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/17' 2023-06-22T08:03:32.172 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/35' 2023-06-22T08:03:32.172 INFO:tasks.workunit.client.0.smithi023.stdout:removed './183/37' 2023-06-22T08:03:32.172 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './183' 2023-06-22T08:03:32.173 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/19' 2023-06-22T08:03:32.173 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/15' 2023-06-22T08:03:32.173 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/31' 2023-06-22T08:03:32.173 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/36' 2023-06-22T08:03:32.173 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/20' 2023-06-22T08:03:32.173 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/12' 2023-06-22T08:03:32.173 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/39' 2023-06-22T08:03:32.173 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/10' 2023-06-22T08:03:32.174 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/34' 2023-06-22T08:03:32.174 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/38' 2023-06-22T08:03:32.174 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/11' 2023-06-22T08:03:32.174 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/16' 2023-06-22T08:03:32.174 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/32' 2023-06-22T08:03:32.174 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/30' 2023-06-22T08:03:32.174 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/14' 2023-06-22T08:03:32.174 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/18' 2023-06-22T08:03:32.175 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/13' 2023-06-22T08:03:32.175 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/33' 2023-06-22T08:03:32.175 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/17' 2023-06-22T08:03:32.175 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/35' 2023-06-22T08:03:32.175 INFO:tasks.workunit.client.0.smithi023.stdout:removed './184/37' 2023-06-22T08:03:32.175 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './184' 2023-06-22T08:03:32.175 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/19' 2023-06-22T08:03:32.175 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/15' 2023-06-22T08:03:32.176 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/31' 2023-06-22T08:03:32.176 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/36' 2023-06-22T08:03:32.176 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/20' 2023-06-22T08:03:32.176 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/12' 2023-06-22T08:03:32.176 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/39' 2023-06-22T08:03:32.176 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/10' 2023-06-22T08:03:32.176 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/34' 2023-06-22T08:03:32.177 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/38' 2023-06-22T08:03:32.177 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/11' 2023-06-22T08:03:32.177 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/16' 2023-06-22T08:03:32.177 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/32' 2023-06-22T08:03:32.177 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/30' 2023-06-22T08:03:32.177 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/14' 2023-06-22T08:03:32.177 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/18' 2023-06-22T08:03:32.178 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/13' 2023-06-22T08:03:32.178 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/33' 2023-06-22T08:03:32.178 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/17' 2023-06-22T08:03:32.178 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/35' 2023-06-22T08:03:32.178 INFO:tasks.workunit.client.0.smithi023.stdout:removed './185/37' 2023-06-22T08:03:32.178 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './185' 2023-06-22T08:03:32.178 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/19' 2023-06-22T08:03:32.179 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/15' 2023-06-22T08:03:32.179 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/31' 2023-06-22T08:03:32.179 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/36' 2023-06-22T08:03:32.179 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/20' 2023-06-22T08:03:32.179 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/12' 2023-06-22T08:03:32.179 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/39' 2023-06-22T08:03:32.179 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/10' 2023-06-22T08:03:32.180 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/34' 2023-06-22T08:03:32.180 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/38' 2023-06-22T08:03:32.180 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/11' 2023-06-22T08:03:32.180 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/16' 2023-06-22T08:03:32.180 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/32' 2023-06-22T08:03:32.180 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/30' 2023-06-22T08:03:32.180 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/14' 2023-06-22T08:03:32.180 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/18' 2023-06-22T08:03:32.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.319+0000 7fd99cdff700 1 -- 172.21.15.23:0/3019475654 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9980c3fa0 con 0x7fd99806d050 2023-06-22T08:03:32.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.320+0000 7fd98effd700 1 -- 172.21.15.23:0/3019475654 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5500 (secure 0 0 0) 0x7fd988015020 con 0x7fd99806d050 2023-06-22T08:03:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:32.323 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:31.679109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:active","state_seq":59,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:32.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 -- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd97806ace0 msgr2=0x7fd97806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:32.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd97806ace0 0x7fd97806d190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd9981453c0 tx=0x7fd980009210 comp rx=0 tx=0).stop 2023-06-22T08:03:32.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 -- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 msgr2=0x7fd99813dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:32.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 0x7fd99813dbf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd99806af80 tx=0x7fd988004750 comp rx=0 tx=0).stop 2023-06-22T08:03:32.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 -- 172.21.15.23:0/3019475654 shutdown_connections 2023-06-22T08:03:32.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd998156fc0 0x7fd9981f1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd99806d050 0x7fd99813dbf0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd97806ace0 0x7fd97806d190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 --2- 172.21.15.23:0/3019475654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9980c3a40 0x7fd99813ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:32.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 -- 172.21.15.23:0/3019475654 >> 172.21.15.23:0/3019475654 conn(0x7fd9980ca3f0 msgr2=0x7fd998069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:32.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 -- 172.21.15.23:0/3019475654 shutdown_connections 2023-06-22T08:03:32.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:32.322+0000 7fd99cdff700 1 -- 172.21.15.23:0/3019475654 wait complete. 2023-06-22T08:03:32.328 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2023-06-22T08:03:32.345 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=4, replacing gid=4268 2023-06-22T08:03:32.346 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.e 2023-06-22T08:03:32.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/13' 2023-06-22T08:03:32.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/33' 2023-06-22T08:03:32.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/17' 2023-06-22T08:03:32.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/35' 2023-06-22T08:03:32.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './186/37' 2023-06-22T08:03:32.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './186' 2023-06-22T08:03:32.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/19' 2023-06-22T08:03:32.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/15' 2023-06-22T08:03:32.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/31' 2023-06-22T08:03:32.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/36' 2023-06-22T08:03:32.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/20' 2023-06-22T08:03:32.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/12' 2023-06-22T08:03:32.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/39' 2023-06-22T08:03:32.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/10' 2023-06-22T08:03:32.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/34' 2023-06-22T08:03:32.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/38' 2023-06-22T08:03:32.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/11' 2023-06-22T08:03:32.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/16' 2023-06-22T08:03:32.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/32' 2023-06-22T08:03:32.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/30' 2023-06-22T08:03:32.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/14' 2023-06-22T08:03:32.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/18' 2023-06-22T08:03:32.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/13' 2023-06-22T08:03:32.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/33' 2023-06-22T08:03:32.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/17' 2023-06-22T08:03:32.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/35' 2023-06-22T08:03:32.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './187/37' 2023-06-22T08:03:32.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './187' 2023-06-22T08:03:32.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/19' 2023-06-22T08:03:32.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/15' 2023-06-22T08:03:32.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/31' 2023-06-22T08:03:32.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/36' 2023-06-22T08:03:32.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/20' 2023-06-22T08:03:32.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/12' 2023-06-22T08:03:32.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/39' 2023-06-22T08:03:32.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/10' 2023-06-22T08:03:32.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/34' 2023-06-22T08:03:32.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/38' 2023-06-22T08:03:32.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/11' 2023-06-22T08:03:32.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/16' 2023-06-22T08:03:32.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/32' 2023-06-22T08:03:32.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/30' 2023-06-22T08:03:32.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/14' 2023-06-22T08:03:32.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/18' 2023-06-22T08:03:32.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/13' 2023-06-22T08:03:32.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/33' 2023-06-22T08:03:32.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/17' 2023-06-22T08:03:32.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/35' 2023-06-22T08:03:32.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './188/37' 2023-06-22T08:03:32.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './188' 2023-06-22T08:03:32.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/19' 2023-06-22T08:03:32.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/15' 2023-06-22T08:03:32.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/31' 2023-06-22T08:03:32.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/36' 2023-06-22T08:03:32.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/20' 2023-06-22T08:03:32.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/12' 2023-06-22T08:03:32.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/39' 2023-06-22T08:03:32.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/10' 2023-06-22T08:03:32.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/34' 2023-06-22T08:03:32.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/38' 2023-06-22T08:03:32.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/11' 2023-06-22T08:03:32.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/16' 2023-06-22T08:03:32.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/32' 2023-06-22T08:03:32.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/30' 2023-06-22T08:03:32.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/14' 2023-06-22T08:03:32.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/18' 2023-06-22T08:03:32.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/13' 2023-06-22T08:03:32.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/33' 2023-06-22T08:03:32.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/17' 2023-06-22T08:03:32.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/35' 2023-06-22T08:03:32.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './189/37' 2023-06-22T08:03:32.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './189' 2023-06-22T08:03:32.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/19' 2023-06-22T08:03:32.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/15' 2023-06-22T08:03:32.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/31' 2023-06-22T08:03:32.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/36' 2023-06-22T08:03:32.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/20' 2023-06-22T08:03:32.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/12' 2023-06-22T08:03:32.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/39' 2023-06-22T08:03:32.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/10' 2023-06-22T08:03:32.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/34' 2023-06-22T08:03:32.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/38' 2023-06-22T08:03:32.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/11' 2023-06-22T08:03:32.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/16' 2023-06-22T08:03:32.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/32' 2023-06-22T08:03:32.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/30' 2023-06-22T08:03:32.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/14' 2023-06-22T08:03:32.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/18' 2023-06-22T08:03:32.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/13' 2023-06-22T08:03:32.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/33' 2023-06-22T08:03:32.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/17' 2023-06-22T08:03:32.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/35' 2023-06-22T08:03:32.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './190/37' 2023-06-22T08:03:32.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './190' 2023-06-22T08:03:32.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/19' 2023-06-22T08:03:32.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/15' 2023-06-22T08:03:32.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/31' 2023-06-22T08:03:32.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/36' 2023-06-22T08:03:32.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/20' 2023-06-22T08:03:32.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/12' 2023-06-22T08:03:32.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/39' 2023-06-22T08:03:32.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/10' 2023-06-22T08:03:32.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/34' 2023-06-22T08:03:32.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/38' 2023-06-22T08:03:32.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/11' 2023-06-22T08:03:32.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/16' 2023-06-22T08:03:32.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/32' 2023-06-22T08:03:32.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/30' 2023-06-22T08:03:32.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/14' 2023-06-22T08:03:32.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/18' 2023-06-22T08:03:32.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/13' 2023-06-22T08:03:32.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/33' 2023-06-22T08:03:32.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/17' 2023-06-22T08:03:32.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/35' 2023-06-22T08:03:32.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed './191/37' 2023-06-22T08:03:32.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './191' 2023-06-22T08:03:32.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/19' 2023-06-22T08:03:32.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/15' 2023-06-22T08:03:32.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/31' 2023-06-22T08:03:32.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/36' 2023-06-22T08:03:32.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/20' 2023-06-22T08:03:32.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/12' 2023-06-22T08:03:32.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/39' 2023-06-22T08:03:32.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/10' 2023-06-22T08:03:32.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/34' 2023-06-22T08:03:32.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/38' 2023-06-22T08:03:32.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/11' 2023-06-22T08:03:32.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/16' 2023-06-22T08:03:32.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/32' 2023-06-22T08:03:32.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/30' 2023-06-22T08:03:32.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/14' 2023-06-22T08:03:32.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/18' 2023-06-22T08:03:32.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/13' 2023-06-22T08:03:32.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/33' 2023-06-22T08:03:32.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/17' 2023-06-22T08:03:32.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/35' 2023-06-22T08:03:32.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './192/37' 2023-06-22T08:03:32.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './192' 2023-06-22T08:03:32.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/19' 2023-06-22T08:03:32.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/15' 2023-06-22T08:03:32.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/31' 2023-06-22T08:03:32.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/36' 2023-06-22T08:03:32.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/20' 2023-06-22T08:03:32.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/12' 2023-06-22T08:03:32.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/39' 2023-06-22T08:03:32.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/10' 2023-06-22T08:03:32.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/34' 2023-06-22T08:03:32.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/38' 2023-06-22T08:03:32.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/11' 2023-06-22T08:03:32.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/16' 2023-06-22T08:03:32.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/32' 2023-06-22T08:03:32.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/30' 2023-06-22T08:03:32.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/14' 2023-06-22T08:03:32.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/18' 2023-06-22T08:03:32.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/13' 2023-06-22T08:03:32.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/33' 2023-06-22T08:03:32.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/17' 2023-06-22T08:03:32.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/35' 2023-06-22T08:03:32.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './193/37' 2023-06-22T08:03:32.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './193' 2023-06-22T08:03:32.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/19' 2023-06-22T08:03:32.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/15' 2023-06-22T08:03:32.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/31' 2023-06-22T08:03:32.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/36' 2023-06-22T08:03:32.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/20' 2023-06-22T08:03:32.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/12' 2023-06-22T08:03:32.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/39' 2023-06-22T08:03:32.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/10' 2023-06-22T08:03:32.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/34' 2023-06-22T08:03:32.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/38' 2023-06-22T08:03:32.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/11' 2023-06-22T08:03:32.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/16' 2023-06-22T08:03:32.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/32' 2023-06-22T08:03:32.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/30' 2023-06-22T08:03:32.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/14' 2023-06-22T08:03:32.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/18' 2023-06-22T08:03:32.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/13' 2023-06-22T08:03:32.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/33' 2023-06-22T08:03:32.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/17' 2023-06-22T08:03:32.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/35' 2023-06-22T08:03:32.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed './194/37' 2023-06-22T08:03:32.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './194' 2023-06-22T08:03:32.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/19' 2023-06-22T08:03:32.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/15' 2023-06-22T08:03:32.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/31' 2023-06-22T08:03:32.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/36' 2023-06-22T08:03:32.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/20' 2023-06-22T08:03:32.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/12' 2023-06-22T08:03:32.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/39' 2023-06-22T08:03:32.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/10' 2023-06-22T08:03:32.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/34' 2023-06-22T08:03:32.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/38' 2023-06-22T08:03:32.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/11' 2023-06-22T08:03:32.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/16' 2023-06-22T08:03:32.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/32' 2023-06-22T08:03:32.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/30' 2023-06-22T08:03:32.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/14' 2023-06-22T08:03:32.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/18' 2023-06-22T08:03:32.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/13' 2023-06-22T08:03:32.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/33' 2023-06-22T08:03:32.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/17' 2023-06-22T08:03:32.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/35' 2023-06-22T08:03:32.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed './195/37' 2023-06-22T08:03:32.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './195' 2023-06-22T08:03:32.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/19' 2023-06-22T08:03:32.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/15' 2023-06-22T08:03:32.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/31' 2023-06-22T08:03:32.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/36' 2023-06-22T08:03:32.804 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/20' 2023-06-22T08:03:32.804 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/12' 2023-06-22T08:03:32.804 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/39' 2023-06-22T08:03:32.804 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/10' 2023-06-22T08:03:33.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/34' 2023-06-22T08:03:33.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/38' 2023-06-22T08:03:33.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/11' 2023-06-22T08:03:33.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/16' 2023-06-22T08:03:33.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/32' 2023-06-22T08:03:33.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/30' 2023-06-22T08:03:33.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/14' 2023-06-22T08:03:33.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/18' 2023-06-22T08:03:33.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/13' 2023-06-22T08:03:33.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/33' 2023-06-22T08:03:33.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/17' 2023-06-22T08:03:33.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/35' 2023-06-22T08:03:33.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './196/37' 2023-06-22T08:03:33.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './196' 2023-06-22T08:03:33.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/19' 2023-06-22T08:03:33.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/15' 2023-06-22T08:03:33.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/31' 2023-06-22T08:03:33.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/36' 2023-06-22T08:03:33.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/20' 2023-06-22T08:03:33.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/12' 2023-06-22T08:03:33.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/39' 2023-06-22T08:03:33.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/10' 2023-06-22T08:03:33.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/34' 2023-06-22T08:03:33.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/38' 2023-06-22T08:03:33.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/11' 2023-06-22T08:03:33.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/16' 2023-06-22T08:03:33.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/32' 2023-06-22T08:03:33.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/30' 2023-06-22T08:03:33.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/14' 2023-06-22T08:03:33.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/18' 2023-06-22T08:03:33.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/13' 2023-06-22T08:03:33.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/33' 2023-06-22T08:03:33.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/17' 2023-06-22T08:03:33.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/35' 2023-06-22T08:03:33.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './197/37' 2023-06-22T08:03:33.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './197' 2023-06-22T08:03:33.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/19' 2023-06-22T08:03:33.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/15' 2023-06-22T08:03:33.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/31' 2023-06-22T08:03:33.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/36' 2023-06-22T08:03:33.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/20' 2023-06-22T08:03:33.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/12' 2023-06-22T08:03:33.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/39' 2023-06-22T08:03:33.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/10' 2023-06-22T08:03:33.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/34' 2023-06-22T08:03:33.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/38' 2023-06-22T08:03:33.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/11' 2023-06-22T08:03:33.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/16' 2023-06-22T08:03:33.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/32' 2023-06-22T08:03:33.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/30' 2023-06-22T08:03:33.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/14' 2023-06-22T08:03:33.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/18' 2023-06-22T08:03:33.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/13' 2023-06-22T08:03:33.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/33' 2023-06-22T08:03:33.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/17' 2023-06-22T08:03:33.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/35' 2023-06-22T08:03:33.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './198/37' 2023-06-22T08:03:33.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './198' 2023-06-22T08:03:33.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/19' 2023-06-22T08:03:33.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/15' 2023-06-22T08:03:33.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/31' 2023-06-22T08:03:33.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/36' 2023-06-22T08:03:33.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/20' 2023-06-22T08:03:33.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/12' 2023-06-22T08:03:33.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/39' 2023-06-22T08:03:33.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/10' 2023-06-22T08:03:33.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/34' 2023-06-22T08:03:33.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/38' 2023-06-22T08:03:33.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/11' 2023-06-22T08:03:33.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/16' 2023-06-22T08:03:33.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/32' 2023-06-22T08:03:33.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/30' 2023-06-22T08:03:33.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/14' 2023-06-22T08:03:33.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/18' 2023-06-22T08:03:33.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/13' 2023-06-22T08:03:33.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/33' 2023-06-22T08:03:33.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/17' 2023-06-22T08:03:33.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/35' 2023-06-22T08:03:33.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './199/37' 2023-06-22T08:03:33.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './199' 2023-06-22T08:03:33.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/21' 2023-06-22T08:03:33.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/27' 2023-06-22T08:03:33.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/31' 2023-06-22T08:03:33.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/36' 2023-06-22T08:03:33.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/20' 2023-06-22T08:03:33.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/39' 2023-06-22T08:03:33.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/34' 2023-06-22T08:03:33.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/38' 2023-06-22T08:03:33.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/28' 2023-06-22T08:03:33.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/24' 2023-06-22T08:03:33.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/29' 2023-06-22T08:03:33.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/32' 2023-06-22T08:03:33.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/22' 2023-06-22T08:03:33.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/25' 2023-06-22T08:03:33.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/30' 2023-06-22T08:03:33.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/33' 2023-06-22T08:03:33.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/23' 2023-06-22T08:03:33.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/35' 2023-06-22T08:03:33.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/26' 2023-06-22T08:03:33.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './200/37' 2023-06-22T08:03:33.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './200' 2023-06-22T08:03:33.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/21' 2023-06-22T08:03:33.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/27' 2023-06-22T08:03:33.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/31' 2023-06-22T08:03:33.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/36' 2023-06-22T08:03:33.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/20' 2023-06-22T08:03:33.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/39' 2023-06-22T08:03:33.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/34' 2023-06-22T08:03:33.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/38' 2023-06-22T08:03:33.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/28' 2023-06-22T08:03:33.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/24' 2023-06-22T08:03:33.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/29' 2023-06-22T08:03:33.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/32' 2023-06-22T08:03:33.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/22' 2023-06-22T08:03:33.478 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/25' 2023-06-22T08:03:33.478 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/30' 2023-06-22T08:03:33.478 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/33' 2023-06-22T08:03:33.478 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/23' 2023-06-22T08:03:33.478 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/35' 2023-06-22T08:03:33.478 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/26' 2023-06-22T08:03:33.478 INFO:tasks.workunit.client.0.smithi023.stdout:removed './201/37' 2023-06-22T08:03:33.478 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './201' 2023-06-22T08:03:33.479 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/21' 2023-06-22T08:03:33.479 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/27' 2023-06-22T08:03:33.479 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/31' 2023-06-22T08:03:33.479 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/36' 2023-06-22T08:03:33.479 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/20' 2023-06-22T08:03:33.479 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/39' 2023-06-22T08:03:33.479 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/34' 2023-06-22T08:03:33.479 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/38' 2023-06-22T08:03:33.480 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/28' 2023-06-22T08:03:33.480 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/24' 2023-06-22T08:03:33.480 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/29' 2023-06-22T08:03:33.480 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/32' 2023-06-22T08:03:33.480 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/22' 2023-06-22T08:03:33.480 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/25' 2023-06-22T08:03:33.480 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/30' 2023-06-22T08:03:33.481 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/33' 2023-06-22T08:03:33.481 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/23' 2023-06-22T08:03:33.481 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/35' 2023-06-22T08:03:33.481 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/26' 2023-06-22T08:03:33.481 INFO:tasks.workunit.client.0.smithi023.stdout:removed './202/37' 2023-06-22T08:03:33.481 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './202' 2023-06-22T08:03:33.481 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/21' 2023-06-22T08:03:33.481 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/27' 2023-06-22T08:03:33.482 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/31' 2023-06-22T08:03:33.482 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/36' 2023-06-22T08:03:33.482 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/20' 2023-06-22T08:03:33.482 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/39' 2023-06-22T08:03:33.482 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/34' 2023-06-22T08:03:33.482 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/38' 2023-06-22T08:03:33.482 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/28' 2023-06-22T08:03:33.482 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/24' 2023-06-22T08:03:33.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/29' 2023-06-22T08:03:33.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/32' 2023-06-22T08:03:33.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/22' 2023-06-22T08:03:33.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/25' 2023-06-22T08:03:33.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/30' 2023-06-22T08:03:33.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/33' 2023-06-22T08:03:33.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/23' 2023-06-22T08:03:33.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/35' 2023-06-22T08:03:33.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/26' 2023-06-22T08:03:33.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed './203/37' 2023-06-22T08:03:33.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './203' 2023-06-22T08:03:33.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/21' 2023-06-22T08:03:33.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/27' 2023-06-22T08:03:33.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/31' 2023-06-22T08:03:33.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/36' 2023-06-22T08:03:33.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/20' 2023-06-22T08:03:33.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/39' 2023-06-22T08:03:33.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/34' 2023-06-22T08:03:33.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/38' 2023-06-22T08:03:33.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/28' 2023-06-22T08:03:33.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/24' 2023-06-22T08:03:33.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/29' 2023-06-22T08:03:33.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/32' 2023-06-22T08:03:33.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/22' 2023-06-22T08:03:33.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/25' 2023-06-22T08:03:33.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/30' 2023-06-22T08:03:33.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/33' 2023-06-22T08:03:33.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/23' 2023-06-22T08:03:33.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/35' 2023-06-22T08:03:33.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/26' 2023-06-22T08:03:33.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed './204/37' 2023-06-22T08:03:33.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './204' 2023-06-22T08:03:33.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/21' 2023-06-22T08:03:33.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/27' 2023-06-22T08:03:33.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/31' 2023-06-22T08:03:33.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/36' 2023-06-22T08:03:33.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/20' 2023-06-22T08:03:33.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/39' 2023-06-22T08:03:33.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/34' 2023-06-22T08:03:33.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/38' 2023-06-22T08:03:33.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/28' 2023-06-22T08:03:33.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/24' 2023-06-22T08:03:33.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/29' 2023-06-22T08:03:33.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/32' 2023-06-22T08:03:33.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/22' 2023-06-22T08:03:33.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/25' 2023-06-22T08:03:33.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/30' 2023-06-22T08:03:33.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/33' 2023-06-22T08:03:33.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/23' 2023-06-22T08:03:33.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/35' 2023-06-22T08:03:33.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/26' 2023-06-22T08:03:33.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './205/37' 2023-06-22T08:03:33.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './205' 2023-06-22T08:03:33.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/21' 2023-06-22T08:03:33.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/27' 2023-06-22T08:03:33.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/31' 2023-06-22T08:03:33.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/36' 2023-06-22T08:03:33.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/20' 2023-06-22T08:03:33.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/39' 2023-06-22T08:03:34.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/34' 2023-06-22T08:03:34.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/38' 2023-06-22T08:03:34.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/28' 2023-06-22T08:03:34.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/24' 2023-06-22T08:03:34.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/29' 2023-06-22T08:03:34.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/32' 2023-06-22T08:03:34.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/22' 2023-06-22T08:03:34.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/25' 2023-06-22T08:03:34.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/30' 2023-06-22T08:03:34.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/33' 2023-06-22T08:03:34.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/23' 2023-06-22T08:03:34.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/35' 2023-06-22T08:03:34.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/26' 2023-06-22T08:03:34.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './206/37' 2023-06-22T08:03:34.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './206' 2023-06-22T08:03:34.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/21' 2023-06-22T08:03:34.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/27' 2023-06-22T08:03:34.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/31' 2023-06-22T08:03:34.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/36' 2023-06-22T08:03:34.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/20' 2023-06-22T08:03:34.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/39' 2023-06-22T08:03:34.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/34' 2023-06-22T08:03:34.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/38' 2023-06-22T08:03:34.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/28' 2023-06-22T08:03:34.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/24' 2023-06-22T08:03:34.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/29' 2023-06-22T08:03:34.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/32' 2023-06-22T08:03:34.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/22' 2023-06-22T08:03:34.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/25' 2023-06-22T08:03:34.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/30' 2023-06-22T08:03:34.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/33' 2023-06-22T08:03:34.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/23' 2023-06-22T08:03:34.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/35' 2023-06-22T08:03:34.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/26' 2023-06-22T08:03:34.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './207/37' 2023-06-22T08:03:34.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './207' 2023-06-22T08:03:34.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/21' 2023-06-22T08:03:34.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/27' 2023-06-22T08:03:34.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/31' 2023-06-22T08:03:34.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/36' 2023-06-22T08:03:34.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/20' 2023-06-22T08:03:34.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/39' 2023-06-22T08:03:34.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/34' 2023-06-22T08:03:34.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/38' 2023-06-22T08:03:34.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/28' 2023-06-22T08:03:34.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/24' 2023-06-22T08:03:34.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/29' 2023-06-22T08:03:34.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/32' 2023-06-22T08:03:34.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/22' 2023-06-22T08:03:34.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/25' 2023-06-22T08:03:34.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/30' 2023-06-22T08:03:34.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/33' 2023-06-22T08:03:34.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/23' 2023-06-22T08:03:34.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/35' 2023-06-22T08:03:34.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/26' 2023-06-22T08:03:34.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './208/37' 2023-06-22T08:03:34.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './208' 2023-06-22T08:03:34.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/21' 2023-06-22T08:03:34.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/27' 2023-06-22T08:03:34.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/31' 2023-06-22T08:03:34.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/36' 2023-06-22T08:03:34.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/20' 2023-06-22T08:03:34.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/39' 2023-06-22T08:03:34.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/34' 2023-06-22T08:03:34.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/38' 2023-06-22T08:03:34.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/28' 2023-06-22T08:03:34.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/24' 2023-06-22T08:03:34.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/29' 2023-06-22T08:03:34.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/32' 2023-06-22T08:03:34.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/22' 2023-06-22T08:03:34.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/25' 2023-06-22T08:03:34.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/30' 2023-06-22T08:03:34.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/33' 2023-06-22T08:03:34.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/23' 2023-06-22T08:03:34.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/35' 2023-06-22T08:03:34.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/26' 2023-06-22T08:03:34.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './209/37' 2023-06-22T08:03:34.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './209' 2023-06-22T08:03:34.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/21' 2023-06-22T08:03:34.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/27' 2023-06-22T08:03:34.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/31' 2023-06-22T08:03:34.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/36' 2023-06-22T08:03:34.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/20' 2023-06-22T08:03:34.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/39' 2023-06-22T08:03:34.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/34' 2023-06-22T08:03:34.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/38' 2023-06-22T08:03:34.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/28' 2023-06-22T08:03:34.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/24' 2023-06-22T08:03:34.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/29' 2023-06-22T08:03:34.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/32' 2023-06-22T08:03:34.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/22' 2023-06-22T08:03:34.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/25' 2023-06-22T08:03:34.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/30' 2023-06-22T08:03:34.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/33' 2023-06-22T08:03:34.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/23' 2023-06-22T08:03:34.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/35' 2023-06-22T08:03:34.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/26' 2023-06-22T08:03:34.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './210/37' 2023-06-22T08:03:34.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './210' 2023-06-22T08:03:34.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/21' 2023-06-22T08:03:34.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/27' 2023-06-22T08:03:34.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/31' 2023-06-22T08:03:34.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/36' 2023-06-22T08:03:34.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/20' 2023-06-22T08:03:34.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/39' 2023-06-22T08:03:34.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/34' 2023-06-22T08:03:34.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/38' 2023-06-22T08:03:34.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/28' 2023-06-22T08:03:34.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/24' 2023-06-22T08:03:34.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/29' 2023-06-22T08:03:34.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/32' 2023-06-22T08:03:34.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/22' 2023-06-22T08:03:34.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/25' 2023-06-22T08:03:34.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/30' 2023-06-22T08:03:34.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/33' 2023-06-22T08:03:34.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/23' 2023-06-22T08:03:34.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/35' 2023-06-22T08:03:34.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/26' 2023-06-22T08:03:34.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './211/37' 2023-06-22T08:03:34.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './211' 2023-06-22T08:03:34.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/21' 2023-06-22T08:03:34.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/27' 2023-06-22T08:03:34.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/31' 2023-06-22T08:03:34.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/36' 2023-06-22T08:03:34.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/20' 2023-06-22T08:03:34.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/39' 2023-06-22T08:03:34.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/34' 2023-06-22T08:03:34.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/38' 2023-06-22T08:03:34.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/28' 2023-06-22T08:03:34.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/24' 2023-06-22T08:03:34.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/29' 2023-06-22T08:03:34.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/32' 2023-06-22T08:03:34.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/22' 2023-06-22T08:03:34.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/25' 2023-06-22T08:03:34.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/30' 2023-06-22T08:03:34.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/33' 2023-06-22T08:03:34.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/23' 2023-06-22T08:03:34.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/35' 2023-06-22T08:03:34.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/26' 2023-06-22T08:03:34.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed './212/37' 2023-06-22T08:03:34.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './212' 2023-06-22T08:03:34.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/21' 2023-06-22T08:03:34.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/27' 2023-06-22T08:03:34.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/31' 2023-06-22T08:03:34.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/36' 2023-06-22T08:03:34.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/20' 2023-06-22T08:03:34.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/39' 2023-06-22T08:03:34.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/34' 2023-06-22T08:03:34.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/38' 2023-06-22T08:03:34.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/28' 2023-06-22T08:03:34.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/24' 2023-06-22T08:03:34.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/29' 2023-06-22T08:03:34.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/32' 2023-06-22T08:03:34.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/22' 2023-06-22T08:03:34.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/25' 2023-06-22T08:03:34.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/30' 2023-06-22T08:03:34.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/33' 2023-06-22T08:03:34.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/23' 2023-06-22T08:03:34.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/35' 2023-06-22T08:03:34.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/26' 2023-06-22T08:03:34.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed './213/37' 2023-06-22T08:03:34.274 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './213' 2023-06-22T08:03:34.274 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/21' 2023-06-22T08:03:34.274 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/27' 2023-06-22T08:03:34.274 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/31' 2023-06-22T08:03:34.275 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/36' 2023-06-22T08:03:34.275 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/20' 2023-06-22T08:03:34.275 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/39' 2023-06-22T08:03:34.275 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/34' 2023-06-22T08:03:34.276 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/38' 2023-06-22T08:03:34.276 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/28' 2023-06-22T08:03:34.276 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/24' 2023-06-22T08:03:34.276 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/29' 2023-06-22T08:03:34.276 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/32' 2023-06-22T08:03:34.276 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/22' 2023-06-22T08:03:34.277 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/25' 2023-06-22T08:03:34.277 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/30' 2023-06-22T08:03:34.277 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/33' 2023-06-22T08:03:34.277 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/23' 2023-06-22T08:03:34.277 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/35' 2023-06-22T08:03:34.278 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/26' 2023-06-22T08:03:34.278 INFO:tasks.workunit.client.0.smithi023.stdout:removed './214/37' 2023-06-22T08:03:34.278 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './214' 2023-06-22T08:03:34.278 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/21' 2023-06-22T08:03:34.278 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/27' 2023-06-22T08:03:34.278 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/31' 2023-06-22T08:03:34.278 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/36' 2023-06-22T08:03:34.278 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/20' 2023-06-22T08:03:34.279 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/39' 2023-06-22T08:03:34.279 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/34' 2023-06-22T08:03:34.279 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/38' 2023-06-22T08:03:34.279 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/28' 2023-06-22T08:03:34.279 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/24' 2023-06-22T08:03:34.279 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/29' 2023-06-22T08:03:34.279 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/32' 2023-06-22T08:03:34.280 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/22' 2023-06-22T08:03:34.280 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/25' 2023-06-22T08:03:34.281 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/30' 2023-06-22T08:03:34.281 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/33' 2023-06-22T08:03:34.282 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/23' 2023-06-22T08:03:34.282 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/35' 2023-06-22T08:03:34.282 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/26' 2023-06-22T08:03:34.282 INFO:tasks.workunit.client.0.smithi023.stdout:removed './215/37' 2023-06-22T08:03:34.283 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './215' 2023-06-22T08:03:34.283 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/21' 2023-06-22T08:03:34.283 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/27' 2023-06-22T08:03:34.283 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/31' 2023-06-22T08:03:34.284 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/36' 2023-06-22T08:03:34.284 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/20' 2023-06-22T08:03:34.284 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/39' 2023-06-22T08:03:34.285 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/34' 2023-06-22T08:03:34.286 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/38' 2023-06-22T08:03:34.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/28' 2023-06-22T08:03:34.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/24' 2023-06-22T08:03:34.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/29' 2023-06-22T08:03:34.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/32' 2023-06-22T08:03:34.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/22' 2023-06-22T08:03:34.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/25' 2023-06-22T08:03:34.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/30' 2023-06-22T08:03:34.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/33' 2023-06-22T08:03:34.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/23' 2023-06-22T08:03:34.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/35' 2023-06-22T08:03:34.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/26' 2023-06-22T08:03:34.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed './216/37' 2023-06-22T08:03:34.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './216' 2023-06-22T08:03:34.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/21' 2023-06-22T08:03:34.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/27' 2023-06-22T08:03:34.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/31' 2023-06-22T08:03:34.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/36' 2023-06-22T08:03:34.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/20' 2023-06-22T08:03:34.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/39' 2023-06-22T08:03:34.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/34' 2023-06-22T08:03:34.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/38' 2023-06-22T08:03:34.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/28' 2023-06-22T08:03:34.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/24' 2023-06-22T08:03:34.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/29' 2023-06-22T08:03:34.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/32' 2023-06-22T08:03:34.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/22' 2023-06-22T08:03:34.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/25' 2023-06-22T08:03:34.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/30' 2023-06-22T08:03:34.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/33' 2023-06-22T08:03:34.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/23' 2023-06-22T08:03:34.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/35' 2023-06-22T08:03:34.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/26' 2023-06-22T08:03:34.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed './217/37' 2023-06-22T08:03:34.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './217' 2023-06-22T08:03:34.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/21' 2023-06-22T08:03:34.987 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/27' 2023-06-22T08:03:34.987 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/31' 2023-06-22T08:03:34.987 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/36' 2023-06-22T08:03:34.987 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/20' 2023-06-22T08:03:34.987 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/39' 2023-06-22T08:03:34.987 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/34' 2023-06-22T08:03:34.988 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/38' 2023-06-22T08:03:34.988 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/28' 2023-06-22T08:03:34.988 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/24' 2023-06-22T08:03:34.988 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/29' 2023-06-22T08:03:34.988 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/32' 2023-06-22T08:03:34.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/22' 2023-06-22T08:03:34.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/25' 2023-06-22T08:03:34.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/30' 2023-06-22T08:03:34.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/33' 2023-06-22T08:03:34.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/23' 2023-06-22T08:03:34.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/35' 2023-06-22T08:03:34.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/26' 2023-06-22T08:03:34.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed './218/37' 2023-06-22T08:03:34.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './218' 2023-06-22T08:03:34.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/21' 2023-06-22T08:03:34.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/27' 2023-06-22T08:03:34.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/31' 2023-06-22T08:03:34.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/36' 2023-06-22T08:03:34.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/20' 2023-06-22T08:03:34.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/39' 2023-06-22T08:03:34.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/34' 2023-06-22T08:03:34.991 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/38' 2023-06-22T08:03:34.991 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/28' 2023-06-22T08:03:34.991 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/24' 2023-06-22T08:03:34.991 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/29' 2023-06-22T08:03:34.991 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/32' 2023-06-22T08:03:34.991 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/22' 2023-06-22T08:03:34.991 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/25' 2023-06-22T08:03:34.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/30' 2023-06-22T08:03:34.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/33' 2023-06-22T08:03:34.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/23' 2023-06-22T08:03:34.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/35' 2023-06-22T08:03:34.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/26' 2023-06-22T08:03:34.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed './219/37' 2023-06-22T08:03:34.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './219' 2023-06-22T08:03:34.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/21' 2023-06-22T08:03:34.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/27' 2023-06-22T08:03:34.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/31' 2023-06-22T08:03:34.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/36' 2023-06-22T08:03:34.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/20' 2023-06-22T08:03:34.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/39' 2023-06-22T08:03:34.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/34' 2023-06-22T08:03:34.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/38' 2023-06-22T08:03:34.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/28' 2023-06-22T08:03:34.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/24' 2023-06-22T08:03:34.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/29' 2023-06-22T08:03:34.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/32' 2023-06-22T08:03:34.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/22' 2023-06-22T08:03:34.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/25' 2023-06-22T08:03:34.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/30' 2023-06-22T08:03:34.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/33' 2023-06-22T08:03:34.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/23' 2023-06-22T08:03:34.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/35' 2023-06-22T08:03:34.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/26' 2023-06-22T08:03:34.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed './220/37' 2023-06-22T08:03:34.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './220' 2023-06-22T08:03:34.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/21' 2023-06-22T08:03:34.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/27' 2023-06-22T08:03:34.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/31' 2023-06-22T08:03:34.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/36' 2023-06-22T08:03:34.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/20' 2023-06-22T08:03:34.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/39' 2023-06-22T08:03:34.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/34' 2023-06-22T08:03:34.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/38' 2023-06-22T08:03:34.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/28' 2023-06-22T08:03:34.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/24' 2023-06-22T08:03:34.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/29' 2023-06-22T08:03:34.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/32' 2023-06-22T08:03:34.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/22' 2023-06-22T08:03:34.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/25' 2023-06-22T08:03:34.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/30' 2023-06-22T08:03:34.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/33' 2023-06-22T08:03:34.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/23' 2023-06-22T08:03:34.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/35' 2023-06-22T08:03:34.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/26' 2023-06-22T08:03:34.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed './221/37' 2023-06-22T08:03:34.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './221' 2023-06-22T08:03:34.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/21' 2023-06-22T08:03:34.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/27' 2023-06-22T08:03:34.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/31' 2023-06-22T08:03:34.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/36' 2023-06-22T08:03:34.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/20' 2023-06-22T08:03:34.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/39' 2023-06-22T08:03:34.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/34' 2023-06-22T08:03:34.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/38' 2023-06-22T08:03:34.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/28' 2023-06-22T08:03:34.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/24' 2023-06-22T08:03:34.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/29' 2023-06-22T08:03:34.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/32' 2023-06-22T08:03:34.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/22' 2023-06-22T08:03:35.000 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/25' 2023-06-22T08:03:35.000 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/30' 2023-06-22T08:03:35.000 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/33' 2023-06-22T08:03:35.000 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/23' 2023-06-22T08:03:35.001 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/35' 2023-06-22T08:03:35.001 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/26' 2023-06-22T08:03:35.001 INFO:tasks.workunit.client.0.smithi023.stdout:removed './222/37' 2023-06-22T08:03:35.002 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './222' 2023-06-22T08:03:35.002 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/21' 2023-06-22T08:03:35.002 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/27' 2023-06-22T08:03:35.002 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/31' 2023-06-22T08:03:35.003 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/36' 2023-06-22T08:03:35.003 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/20' 2023-06-22T08:03:35.003 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/39' 2023-06-22T08:03:35.003 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/34' 2023-06-22T08:03:35.004 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/38' 2023-06-22T08:03:35.004 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/28' 2023-06-22T08:03:35.004 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/24' 2023-06-22T08:03:35.004 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/29' 2023-06-22T08:03:35.005 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/32' 2023-06-22T08:03:35.005 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/22' 2023-06-22T08:03:35.005 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/25' 2023-06-22T08:03:35.006 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/30' 2023-06-22T08:03:35.006 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/33' 2023-06-22T08:03:35.006 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/23' 2023-06-22T08:03:35.006 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/35' 2023-06-22T08:03:35.006 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/26' 2023-06-22T08:03:35.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed './223/37' 2023-06-22T08:03:35.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './223' 2023-06-22T08:03:35.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/21' 2023-06-22T08:03:35.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/27' 2023-06-22T08:03:35.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/31' 2023-06-22T08:03:35.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/36' 2023-06-22T08:03:35.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/20' 2023-06-22T08:03:35.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/39' 2023-06-22T08:03:35.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/34' 2023-06-22T08:03:35.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/38' 2023-06-22T08:03:35.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/28' 2023-06-22T08:03:35.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/24' 2023-06-22T08:03:35.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/29' 2023-06-22T08:03:35.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/32' 2023-06-22T08:03:35.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/22' 2023-06-22T08:03:35.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/25' 2023-06-22T08:03:35.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/30' 2023-06-22T08:03:35.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/33' 2023-06-22T08:03:35.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/23' 2023-06-22T08:03:35.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/35' 2023-06-22T08:03:35.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/26' 2023-06-22T08:03:35.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed './224/37' 2023-06-22T08:03:35.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './224' 2023-06-22T08:03:35.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/21' 2023-06-22T08:03:35.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/27' 2023-06-22T08:03:35.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/31' 2023-06-22T08:03:35.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/36' 2023-06-22T08:03:35.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/20' 2023-06-22T08:03:35.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/39' 2023-06-22T08:03:35.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/34' 2023-06-22T08:03:35.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/38' 2023-06-22T08:03:35.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/28' 2023-06-22T08:03:35.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/24' 2023-06-22T08:03:35.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/29' 2023-06-22T08:03:35.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/32' 2023-06-22T08:03:35.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/22' 2023-06-22T08:03:35.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/25' 2023-06-22T08:03:35.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/30' 2023-06-22T08:03:35.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/33' 2023-06-22T08:03:35.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/23' 2023-06-22T08:03:35.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/35' 2023-06-22T08:03:35.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/26' 2023-06-22T08:03:35.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed './225/37' 2023-06-22T08:03:35.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './225' 2023-06-22T08:03:35.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/21' 2023-06-22T08:03:35.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/27' 2023-06-22T08:03:35.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/31' 2023-06-22T08:03:35.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/36' 2023-06-22T08:03:35.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/20' 2023-06-22T08:03:35.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/39' 2023-06-22T08:03:35.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/34' 2023-06-22T08:03:35.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/38' 2023-06-22T08:03:35.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/28' 2023-06-22T08:03:35.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/24' 2023-06-22T08:03:35.347 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-06-22T08:03:35.347 INFO:tasks.ceph.mds.e:Restarting daemon 2023-06-22T08:03:35.348 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-06-22T08:03:35.352 INFO:tasks.ceph.mds.e:Started 2023-06-22T08:03:35.352 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:35.599 INFO:tasks.ceph.mds.e.smithi023.stdout:starting mds.e at 2023-06-22T08:03:35.607 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:03:35.603+0000 7fa5334b2700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-06-22T08:03:35.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.688+0000 7fb0a9fff700 1 Processor -- start 2023-06-22T08:03:35.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.689+0000 7fb0a9fff700 1 -- start start 2023-06-22T08:03:35.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.691+0000 7fb0a9fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb0a415fa70 0x7fb0a40cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:35.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.691+0000 7fb0a9fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0a40c5e70 0x7fb0a40cb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:35.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.691+0000 7fb0a9fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 0x7fb0a415e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:35.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.691+0000 7fb0a9fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a415e1f0 con 0x7fb0a415fa70 2023-06-22T08:03:35.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.691+0000 7fb0a9fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a415d010 con 0x7fb0a40c5e70 2023-06-22T08:03:35.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.691+0000 7fb0a9fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a415d2f0 con 0x7fb0a40c62d0 2023-06-22T08:03:35.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.692+0000 7fb0a3fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 0x7fb0a415e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:35.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.692+0000 7fb0a3fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 0x7fb0a415e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:55744/0 (socket says 172.21.15.23:55744) 2023-06-22T08:03:35.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.692+0000 7fb0a3fff700 1 -- 172.21.15.23:0/2320515310 learned_addr learned my addr 172.21.15.23:0/2320515310 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:35.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.693+0000 7fb0a37fe700 1 --2- 172.21.15.23:0/2320515310 >> v2:172.21.15.23:3300/0 conn(0x7fb0a415fa70 0x7fb0a40cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:35.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.694+0000 7fb0a37fe700 1 -- 172.21.15.23:0/2320515310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 msgr2=0x7fb0a415e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:35.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.694+0000 7fb0a37fe700 1 --2- 172.21.15.23:0/2320515310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 0x7fb0a415e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:35.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.694+0000 7fb0a37fe700 1 -- 172.21.15.23:0/2320515310 >> v2:172.21.15.67:3300/0 conn(0x7fb0a40c5e70 msgr2=0x7fb0a40cb980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:03:35.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.694+0000 7fb0a37fe700 1 --2- 172.21.15.23:0/2320515310 >> v2:172.21.15.67:3300/0 conn(0x7fb0a40c5e70 0x7fb0a40cb980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:35.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.694+0000 7fb0a37fe700 1 -- 172.21.15.23:0/2320515310 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a415d5d0 con 0x7fb0a415fa70 2023-06-22T08:03:35.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.695+0000 7fb0a37fe700 1 --2- 172.21.15.23:0/2320515310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a415fa70 0x7fb0a40cb440 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4070710 tx=0x7fb09c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5227ed38319920 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:35.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.696+0000 7fb0a27fc700 1 -- 172.21.15.23:0/2320515310 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c00e070 con 0x7fb0a415fa70 2023-06-22T08:03:35.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.696+0000 7fb0a27fc700 1 -- 172.21.15.23:0/2320515310 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c00bbd0 con 0x7fb0a415fa70 2023-06-22T08:03:35.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.696+0000 7fb0a27fc700 1 -- 172.21.15.23:0/2320515310 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c012430 con 0x7fb0a415fa70 2023-06-22T08:03:35.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.697+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2320515310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a415fa70 msgr2=0x7fb0a40cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:35.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.697+0000 7fb0a9fff700 1 --2- 172.21.15.23:0/2320515310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a415fa70 0x7fb0a40cb440 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4070710 tx=0x7fb09c00b330 comp rx=0 tx=0).stop 2023-06-22T08:03:35.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.698+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2320515310 shutdown_connections 2023-06-22T08:03:35.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.698+0000 7fb0a9fff700 1 --2- 172.21.15.23:0/2320515310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 0x7fb0a415e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:35.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.698+0000 7fb0a9fff700 1 --2- 172.21.15.23:0/2320515310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a415fa70 0x7fb0a40cb440 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:35.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.698+0000 7fb0a9fff700 1 --2- 172.21.15.23:0/2320515310 >> v2:172.21.15.67:3300/0 conn(0x7fb0a40c5e70 0x7fb0a40cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:35.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.698+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2320515310 >> 172.21.15.23:0/2320515310 conn(0x7fb0a4070d80 msgr2=0x7fb0a4071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:35.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.699+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2320515310 shutdown_connections 2023-06-22T08:03:35.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.700+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2320515310 wait complete. 2023-06-22T08:03:35.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.703+0000 7fb0a9fff700 1 Processor -- start 2023-06-22T08:03:35.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.704+0000 7fb0a9fff700 1 -- start start 2023-06-22T08:03:35.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.706+0000 7fb0a9fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a40c5e70 0x7fb0a40ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:35.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.706+0000 7fb0a9fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 0x7fb0a40c8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:35.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.706+0000 7fb0a9fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a415fa70 0x7fb0a40c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:35.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.706+0000 7fb0a9fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a4139770 con 0x7fb0a40c5e70 2023-06-22T08:03:35.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.706+0000 7fb0a9fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a4147630 con 0x7fb0a415fa70 2023-06-22T08:03:35.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.706+0000 7fb0a9fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a4158bf0 con 0x7fb0a40c62d0 2023-06-22T08:03:35.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.706+0000 7fb0a37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a40c5e70 0x7fb0a40ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:35.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.706+0000 7fb0a37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a40c5e70 0x7fb0a40ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39668/0 (socket says 172.21.15.23:39668) 2023-06-22T08:03:35.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.706+0000 7fb0a37fe700 1 -- 172.21.15.23:0/2537939458 learned_addr learned my addr 172.21.15.23:0/2537939458 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:35.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.707+0000 7fb0a2ffd700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 0x7fb0a40c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:35.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.708+0000 7fb0a3fff700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a415fa70 0x7fb0a40c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:35.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.708+0000 7fb0a37fe700 1 -- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 msgr2=0x7fb0a40c8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:35.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.708+0000 7fb0a37fe700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 0x7fb0a40c8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:35.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.708+0000 7fb0a37fe700 1 -- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a415fa70 msgr2=0x7fb0a40c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:35.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.708+0000 7fb0a37fe700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a415fa70 0x7fb0a40c8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:35.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.708+0000 7fb0a37fe700 1 -- 172.21.15.23:0/2537939458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c009a70 con 0x7fb0a40c5e70 2023-06-22T08:03:35.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.709+0000 7fb0a37fe700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a40c5e70 0x7fb0a40ca6c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb09c000f80 tx=0x7fb09c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:35.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.710+0000 7fb08bfff700 1 -- 172.21.15.23:0/2537939458 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c004930 con 0x7fb0a40c5e70 2023-06-22T08:03:35.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.710+0000 7fb08bfff700 1 -- 172.21.15.23:0/2537939458 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c020070 con 0x7fb0a40c5e70 2023-06-22T08:03:35.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.710+0000 7fb08bfff700 1 -- 172.21.15.23:0/2537939458 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c0276a0 con 0x7fb0a40c5e70 2023-06-22T08:03:35.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.710+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2537939458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a4158ed0 con 0x7fb0a40c5e70 2023-06-22T08:03:35.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.710+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2537939458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a4159490 con 0x7fb0a40c5e70 2023-06-22T08:03:35.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.712+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2537939458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a4159650 con 0x7fb0a40c5e70 2023-06-22T08:03:35.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.715+0000 7fb08bfff700 1 -- 172.21.15.23:0/2537939458 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb09c012430 con 0x7fb0a40c5e70 2023-06-22T08:03:35.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.717+0000 7fb08bfff700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09006adb0 0x7fb09006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:35.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.717+0000 7fb08bfff700 1 -- 172.21.15.23:0/2537939458 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7fb09c022070 con 0x7fb0a40c5e70 2023-06-22T08:03:35.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.719+0000 7fb0a2ffd700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09006adb0 0x7fb09006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:35.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.723+0000 7fb0a2ffd700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09006adb0 0x7fb09006d260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb0a413bcb0 tx=0x7fb098009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:35.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:35.726+0000 7fb08bfff700 1 -- 172.21.15.23:0/2537939458 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb09c056800 con 0x7fb0a40c5e70 2023-06-22T08:03:35.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/29' 2023-06-22T08:03:35.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/32' 2023-06-22T08:03:35.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/22' 2023-06-22T08:03:35.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/25' 2023-06-22T08:03:35.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/30' 2023-06-22T08:03:35.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/33' 2023-06-22T08:03:35.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/23' 2023-06-22T08:03:35.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/35' 2023-06-22T08:03:35.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/26' 2023-06-22T08:03:35.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed './226/37' 2023-06-22T08:03:35.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './226' 2023-06-22T08:03:35.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/21' 2023-06-22T08:03:35.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/27' 2023-06-22T08:03:35.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/31' 2023-06-22T08:03:35.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/36' 2023-06-22T08:03:35.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/20' 2023-06-22T08:03:35.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/39' 2023-06-22T08:03:35.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/34' 2023-06-22T08:03:35.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/38' 2023-06-22T08:03:35.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/28' 2023-06-22T08:03:35.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/24' 2023-06-22T08:03:35.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/29' 2023-06-22T08:03:35.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/32' 2023-06-22T08:03:35.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/22' 2023-06-22T08:03:35.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/25' 2023-06-22T08:03:35.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/30' 2023-06-22T08:03:35.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/33' 2023-06-22T08:03:35.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/23' 2023-06-22T08:03:35.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/35' 2023-06-22T08:03:35.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/26' 2023-06-22T08:03:35.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed './227/37' 2023-06-22T08:03:35.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './227' 2023-06-22T08:03:35.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/21' 2023-06-22T08:03:35.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/27' 2023-06-22T08:03:35.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/31' 2023-06-22T08:03:35.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/36' 2023-06-22T08:03:35.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/20' 2023-06-22T08:03:35.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/39' 2023-06-22T08:03:35.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/34' 2023-06-22T08:03:35.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/38' 2023-06-22T08:03:35.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/28' 2023-06-22T08:03:35.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/24' 2023-06-22T08:03:35.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/29' 2023-06-22T08:03:35.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/32' 2023-06-22T08:03:35.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/22' 2023-06-22T08:03:35.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/25' 2023-06-22T08:03:35.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/30' 2023-06-22T08:03:35.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/33' 2023-06-22T08:03:35.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/23' 2023-06-22T08:03:35.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/35' 2023-06-22T08:03:35.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/26' 2023-06-22T08:03:35.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed './228/37' 2023-06-22T08:03:35.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './228' 2023-06-22T08:03:35.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/21' 2023-06-22T08:03:35.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/27' 2023-06-22T08:03:35.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/31' 2023-06-22T08:03:35.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/36' 2023-06-22T08:03:35.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/20' 2023-06-22T08:03:35.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/39' 2023-06-22T08:03:35.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/34' 2023-06-22T08:03:35.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/38' 2023-06-22T08:03:35.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/28' 2023-06-22T08:03:35.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/24' 2023-06-22T08:03:35.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/29' 2023-06-22T08:03:35.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/32' 2023-06-22T08:03:35.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/22' 2023-06-22T08:03:35.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/25' 2023-06-22T08:03:35.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/30' 2023-06-22T08:03:35.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/33' 2023-06-22T08:03:35.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/23' 2023-06-22T08:03:35.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/35' 2023-06-22T08:03:35.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/26' 2023-06-22T08:03:35.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed './229/37' 2023-06-22T08:03:35.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './229' 2023-06-22T08:03:35.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/21' 2023-06-22T08:03:35.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/27' 2023-06-22T08:03:35.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/31' 2023-06-22T08:03:35.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/36' 2023-06-22T08:03:35.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/20' 2023-06-22T08:03:35.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/39' 2023-06-22T08:03:35.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/34' 2023-06-22T08:03:35.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/38' 2023-06-22T08:03:35.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/28' 2023-06-22T08:03:35.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/24' 2023-06-22T08:03:35.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/29' 2023-06-22T08:03:35.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/32' 2023-06-22T08:03:35.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/22' 2023-06-22T08:03:35.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/25' 2023-06-22T08:03:35.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/30' 2023-06-22T08:03:35.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/33' 2023-06-22T08:03:35.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/23' 2023-06-22T08:03:35.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/35' 2023-06-22T08:03:35.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/26' 2023-06-22T08:03:35.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed './230/37' 2023-06-22T08:03:35.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './230' 2023-06-22T08:03:35.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/21' 2023-06-22T08:03:35.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/27' 2023-06-22T08:03:35.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/31' 2023-06-22T08:03:35.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/36' 2023-06-22T08:03:35.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/20' 2023-06-22T08:03:35.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/39' 2023-06-22T08:03:35.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/34' 2023-06-22T08:03:35.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/38' 2023-06-22T08:03:35.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/28' 2023-06-22T08:03:35.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/24' 2023-06-22T08:03:35.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/29' 2023-06-22T08:03:35.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/32' 2023-06-22T08:03:35.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/22' 2023-06-22T08:03:35.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/25' 2023-06-22T08:03:35.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/30' 2023-06-22T08:03:35.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/33' 2023-06-22T08:03:35.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/23' 2023-06-22T08:03:35.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/35' 2023-06-22T08:03:35.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/26' 2023-06-22T08:03:35.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './231/37' 2023-06-22T08:03:35.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './231' 2023-06-22T08:03:35.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/21' 2023-06-22T08:03:35.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/27' 2023-06-22T08:03:35.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/31' 2023-06-22T08:03:35.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/36' 2023-06-22T08:03:35.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/20' 2023-06-22T08:03:35.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/39' 2023-06-22T08:03:35.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/34' 2023-06-22T08:03:35.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/38' 2023-06-22T08:03:35.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/28' 2023-06-22T08:03:35.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/24' 2023-06-22T08:03:35.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/29' 2023-06-22T08:03:35.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/32' 2023-06-22T08:03:35.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/22' 2023-06-22T08:03:35.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/25' 2023-06-22T08:03:35.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/30' 2023-06-22T08:03:35.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/33' 2023-06-22T08:03:35.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/23' 2023-06-22T08:03:35.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/35' 2023-06-22T08:03:35.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/26' 2023-06-22T08:03:35.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './232/37' 2023-06-22T08:03:35.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './232' 2023-06-22T08:03:35.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/21' 2023-06-22T08:03:35.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/27' 2023-06-22T08:03:35.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/31' 2023-06-22T08:03:35.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/36' 2023-06-22T08:03:35.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/20' 2023-06-22T08:03:35.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/39' 2023-06-22T08:03:35.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/34' 2023-06-22T08:03:35.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/38' 2023-06-22T08:03:35.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/28' 2023-06-22T08:03:35.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/24' 2023-06-22T08:03:35.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/29' 2023-06-22T08:03:35.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/32' 2023-06-22T08:03:35.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/22' 2023-06-22T08:03:35.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/25' 2023-06-22T08:03:35.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/30' 2023-06-22T08:03:35.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/33' 2023-06-22T08:03:35.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/23' 2023-06-22T08:03:35.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/35' 2023-06-22T08:03:35.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/26' 2023-06-22T08:03:35.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './233/37' 2023-06-22T08:03:35.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './233' 2023-06-22T08:03:35.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/21' 2023-06-22T08:03:35.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/27' 2023-06-22T08:03:35.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/31' 2023-06-22T08:03:35.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/36' 2023-06-22T08:03:35.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/20' 2023-06-22T08:03:35.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/39' 2023-06-22T08:03:35.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/34' 2023-06-22T08:03:35.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/38' 2023-06-22T08:03:35.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/28' 2023-06-22T08:03:35.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/24' 2023-06-22T08:03:35.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/29' 2023-06-22T08:03:35.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/32' 2023-06-22T08:03:35.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/22' 2023-06-22T08:03:35.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/25' 2023-06-22T08:03:35.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/30' 2023-06-22T08:03:35.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/33' 2023-06-22T08:03:35.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/23' 2023-06-22T08:03:35.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/35' 2023-06-22T08:03:35.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/26' 2023-06-22T08:03:35.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed './234/37' 2023-06-22T08:03:35.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './234' 2023-06-22T08:03:35.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/21' 2023-06-22T08:03:35.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/27' 2023-06-22T08:03:35.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/31' 2023-06-22T08:03:35.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/36' 2023-06-22T08:03:35.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/20' 2023-06-22T08:03:35.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/39' 2023-06-22T08:03:35.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/34' 2023-06-22T08:03:35.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/38' 2023-06-22T08:03:35.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/28' 2023-06-22T08:03:35.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/24' 2023-06-22T08:03:35.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/29' 2023-06-22T08:03:35.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/32' 2023-06-22T08:03:35.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/22' 2023-06-22T08:03:35.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/25' 2023-06-22T08:03:35.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/30' 2023-06-22T08:03:35.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/33' 2023-06-22T08:03:35.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/23' 2023-06-22T08:03:35.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/35' 2023-06-22T08:03:35.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/26' 2023-06-22T08:03:35.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './235/37' 2023-06-22T08:03:35.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './235' 2023-06-22T08:03:35.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/21' 2023-06-22T08:03:35.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/27' 2023-06-22T08:03:35.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/31' 2023-06-22T08:03:35.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/36' 2023-06-22T08:03:35.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/20' 2023-06-22T08:03:35.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/39' 2023-06-22T08:03:35.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/34' 2023-06-22T08:03:35.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/38' 2023-06-22T08:03:35.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/28' 2023-06-22T08:03:35.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/24' 2023-06-22T08:03:35.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/29' 2023-06-22T08:03:35.784 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/32' 2023-06-22T08:03:36.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.098+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2537939458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0a40c7f50 con 0x7fb0a40c5e70 2023-06-22T08:03:36.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.099+0000 7fb08bfff700 1 -- 172.21.15.23:0/2537939458 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+5500 (secure 0 0 0) 0x7fb09c059e20 con 0x7fb0a40c5e70 2023-06-22T08:03:36.102 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:36.102 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:31.679109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:active","state_seq":59,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:36.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09006adb0 msgr2=0x7fb09006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:36.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09006adb0 0x7fb09006d260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb0a413bcb0 tx=0x7fb098009190 comp rx=0 tx=0).stop 2023-06-22T08:03:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2537939458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a40c5e70 msgr2=0x7fb0a40ca6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a40c5e70 0x7fb0a40ca6c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb09c000f80 tx=0x7fb09c004ad0 comp rx=0 tx=0).stop 2023-06-22T08:03:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2537939458 shutdown_connections 2023-06-22T08:03:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb09006adb0 0x7fb09006d260 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0a415fa70 0x7fb0a40c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:36.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a40c5e70 0x7fb0a40ca6c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 --2- 172.21.15.23:0/2537939458 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0a40c62d0 0x7fb0a40c8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2537939458 >> 172.21.15.23:0/2537939458 conn(0x7fb0a4070d80 msgr2=0x7fb0a415e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2537939458 shutdown_connections 2023-06-22T08:03:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:36.101+0000 7fb0a9fff700 1 -- 172.21.15.23:0/2537939458 wait complete. 2023-06-22T08:03:36.106 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2023-06-22T08:03:36.125 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-06-22T08:03:36.493 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:03:36.497 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:03:36.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/22' 2023-06-22T08:03:36.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/25' 2023-06-22T08:03:36.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/30' 2023-06-22T08:03:36.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/33' 2023-06-22T08:03:36.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/23' 2023-06-22T08:03:36.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/35' 2023-06-22T08:03:36.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/26' 2023-06-22T08:03:36.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './236/37' 2023-06-22T08:03:36.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './236' 2023-06-22T08:03:36.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/21' 2023-06-22T08:03:36.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/27' 2023-06-22T08:03:36.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/31' 2023-06-22T08:03:36.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/36' 2023-06-22T08:03:36.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/20' 2023-06-22T08:03:36.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/39' 2023-06-22T08:03:36.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/34' 2023-06-22T08:03:36.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/38' 2023-06-22T08:03:36.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/28' 2023-06-22T08:03:36.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/24' 2023-06-22T08:03:36.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/29' 2023-06-22T08:03:36.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/32' 2023-06-22T08:03:36.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/22' 2023-06-22T08:03:36.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/25' 2023-06-22T08:03:36.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/30' 2023-06-22T08:03:36.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/33' 2023-06-22T08:03:36.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/23' 2023-06-22T08:03:36.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/35' 2023-06-22T08:03:36.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/26' 2023-06-22T08:03:36.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed './237/37' 2023-06-22T08:03:36.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './237' 2023-06-22T08:03:36.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/21' 2023-06-22T08:03:36.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/27' 2023-06-22T08:03:36.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/31' 2023-06-22T08:03:36.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/36' 2023-06-22T08:03:36.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/20' 2023-06-22T08:03:36.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/39' 2023-06-22T08:03:36.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/34' 2023-06-22T08:03:36.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/38' 2023-06-22T08:03:36.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/28' 2023-06-22T08:03:36.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/24' 2023-06-22T08:03:36.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/29' 2023-06-22T08:03:36.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/32' 2023-06-22T08:03:36.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/22' 2023-06-22T08:03:36.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/25' 2023-06-22T08:03:36.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/30' 2023-06-22T08:03:36.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/33' 2023-06-22T08:03:36.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/23' 2023-06-22T08:03:36.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/35' 2023-06-22T08:03:36.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/26' 2023-06-22T08:03:36.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed './238/37' 2023-06-22T08:03:36.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './238' 2023-06-22T08:03:36.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/21' 2023-06-22T08:03:36.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/27' 2023-06-22T08:03:36.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/31' 2023-06-22T08:03:36.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/36' 2023-06-22T08:03:36.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/20' 2023-06-22T08:03:36.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/39' 2023-06-22T08:03:36.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/34' 2023-06-22T08:03:36.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/38' 2023-06-22T08:03:36.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/28' 2023-06-22T08:03:36.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/24' 2023-06-22T08:03:36.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/29' 2023-06-22T08:03:36.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/32' 2023-06-22T08:03:36.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/22' 2023-06-22T08:03:36.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/25' 2023-06-22T08:03:36.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/30' 2023-06-22T08:03:36.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/33' 2023-06-22T08:03:36.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/23' 2023-06-22T08:03:36.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/35' 2023-06-22T08:03:36.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/26' 2023-06-22T08:03:36.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './239/37' 2023-06-22T08:03:36.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './239' 2023-06-22T08:03:36.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/21' 2023-06-22T08:03:36.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/27' 2023-06-22T08:03:36.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/31' 2023-06-22T08:03:36.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/36' 2023-06-22T08:03:36.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/20' 2023-06-22T08:03:36.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/39' 2023-06-22T08:03:36.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/34' 2023-06-22T08:03:36.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/38' 2023-06-22T08:03:36.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/28' 2023-06-22T08:03:36.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/24' 2023-06-22T08:03:36.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/29' 2023-06-22T08:03:36.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/32' 2023-06-22T08:03:36.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/22' 2023-06-22T08:03:36.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/25' 2023-06-22T08:03:36.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/30' 2023-06-22T08:03:36.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/33' 2023-06-22T08:03:36.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/23' 2023-06-22T08:03:36.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/35' 2023-06-22T08:03:36.522 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/26' 2023-06-22T08:03:36.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './240/37' 2023-06-22T08:03:36.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './240' 2023-06-22T08:03:36.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/21' 2023-06-22T08:03:36.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/27' 2023-06-22T08:03:36.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/31' 2023-06-22T08:03:36.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/36' 2023-06-22T08:03:36.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/20' 2023-06-22T08:03:36.523 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/39' 2023-06-22T08:03:36.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/34' 2023-06-22T08:03:36.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/38' 2023-06-22T08:03:36.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/28' 2023-06-22T08:03:36.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/24' 2023-06-22T08:03:36.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/29' 2023-06-22T08:03:36.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/32' 2023-06-22T08:03:36.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/22' 2023-06-22T08:03:36.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/25' 2023-06-22T08:03:36.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/30' 2023-06-22T08:03:36.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/33' 2023-06-22T08:03:36.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/23' 2023-06-22T08:03:36.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/35' 2023-06-22T08:03:36.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/26' 2023-06-22T08:03:36.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed './241/37' 2023-06-22T08:03:36.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './241' 2023-06-22T08:03:36.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/21' 2023-06-22T08:03:36.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/27' 2023-06-22T08:03:36.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/31' 2023-06-22T08:03:36.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/36' 2023-06-22T08:03:36.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/20' 2023-06-22T08:03:36.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/39' 2023-06-22T08:03:36.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/34' 2023-06-22T08:03:36.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/38' 2023-06-22T08:03:36.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/28' 2023-06-22T08:03:36.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/24' 2023-06-22T08:03:36.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/29' 2023-06-22T08:03:36.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/32' 2023-06-22T08:03:36.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/22' 2023-06-22T08:03:36.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/25' 2023-06-22T08:03:36.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/30' 2023-06-22T08:03:36.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/33' 2023-06-22T08:03:36.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/23' 2023-06-22T08:03:36.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/35' 2023-06-22T08:03:36.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/26' 2023-06-22T08:03:36.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed './242/37' 2023-06-22T08:03:36.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './242' 2023-06-22T08:03:36.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/21' 2023-06-22T08:03:36.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/27' 2023-06-22T08:03:36.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/31' 2023-06-22T08:03:36.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/36' 2023-06-22T08:03:36.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/20' 2023-06-22T08:03:36.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/39' 2023-06-22T08:03:36.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/34' 2023-06-22T08:03:36.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/38' 2023-06-22T08:03:36.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/28' 2023-06-22T08:03:36.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/24' 2023-06-22T08:03:36.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/29' 2023-06-22T08:03:36.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/32' 2023-06-22T08:03:36.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/22' 2023-06-22T08:03:36.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/25' 2023-06-22T08:03:36.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/30' 2023-06-22T08:03:36.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/33' 2023-06-22T08:03:36.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/23' 2023-06-22T08:03:36.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/35' 2023-06-22T08:03:36.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/26' 2023-06-22T08:03:36.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './243/37' 2023-06-22T08:03:36.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './243' 2023-06-22T08:03:36.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/21' 2023-06-22T08:03:36.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/27' 2023-06-22T08:03:36.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/31' 2023-06-22T08:03:36.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/36' 2023-06-22T08:03:36.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/20' 2023-06-22T08:03:36.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/39' 2023-06-22T08:03:36.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/34' 2023-06-22T08:03:36.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/38' 2023-06-22T08:03:36.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/28' 2023-06-22T08:03:36.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/24' 2023-06-22T08:03:36.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/29' 2023-06-22T08:03:36.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/32' 2023-06-22T08:03:36.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/22' 2023-06-22T08:03:36.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/25' 2023-06-22T08:03:36.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/30' 2023-06-22T08:03:36.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/33' 2023-06-22T08:03:36.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/23' 2023-06-22T08:03:36.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/35' 2023-06-22T08:03:36.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/26' 2023-06-22T08:03:36.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './244/37' 2023-06-22T08:03:36.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './244' 2023-06-22T08:03:36.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/21' 2023-06-22T08:03:36.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/27' 2023-06-22T08:03:36.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/31' 2023-06-22T08:03:36.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/36' 2023-06-22T08:03:36.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/20' 2023-06-22T08:03:36.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/39' 2023-06-22T08:03:36.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/34' 2023-06-22T08:03:36.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/38' 2023-06-22T08:03:36.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/28' 2023-06-22T08:03:36.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/24' 2023-06-22T08:03:36.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/29' 2023-06-22T08:03:36.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/32' 2023-06-22T08:03:36.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/22' 2023-06-22T08:03:36.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/25' 2023-06-22T08:03:36.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/30' 2023-06-22T08:03:36.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/33' 2023-06-22T08:03:36.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/23' 2023-06-22T08:03:36.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/35' 2023-06-22T08:03:36.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/26' 2023-06-22T08:03:36.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed './245/37' 2023-06-22T08:03:36.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './245' 2023-06-22T08:03:36.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/21' 2023-06-22T08:03:36.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/27' 2023-06-22T08:03:36.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/31' 2023-06-22T08:03:36.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/36' 2023-06-22T08:03:36.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/20' 2023-06-22T08:03:36.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/39' 2023-06-22T08:03:36.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/34' 2023-06-22T08:03:36.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/38' 2023-06-22T08:03:36.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/28' 2023-06-22T08:03:36.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/24' 2023-06-22T08:03:36.538 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/29' 2023-06-22T08:03:36.538 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/32' 2023-06-22T08:03:36.538 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/22' 2023-06-22T08:03:37.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/25' 2023-06-22T08:03:37.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/30' 2023-06-22T08:03:37.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/33' 2023-06-22T08:03:37.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/23' 2023-06-22T08:03:37.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/35' 2023-06-22T08:03:37.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/26' 2023-06-22T08:03:37.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './246/37' 2023-06-22T08:03:37.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './246' 2023-06-22T08:03:37.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/21' 2023-06-22T08:03:37.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/27' 2023-06-22T08:03:37.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/31' 2023-06-22T08:03:37.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/36' 2023-06-22T08:03:37.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/20' 2023-06-22T08:03:37.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/39' 2023-06-22T08:03:37.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/34' 2023-06-22T08:03:37.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/38' 2023-06-22T08:03:37.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/28' 2023-06-22T08:03:37.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/24' 2023-06-22T08:03:37.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/29' 2023-06-22T08:03:37.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/32' 2023-06-22T08:03:37.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/22' 2023-06-22T08:03:37.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/25' 2023-06-22T08:03:37.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/30' 2023-06-22T08:03:37.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/33' 2023-06-22T08:03:37.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/23' 2023-06-22T08:03:37.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/35' 2023-06-22T08:03:37.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/26' 2023-06-22T08:03:37.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed './247/37' 2023-06-22T08:03:37.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './247' 2023-06-22T08:03:37.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/21' 2023-06-22T08:03:37.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/27' 2023-06-22T08:03:37.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/31' 2023-06-22T08:03:37.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/36' 2023-06-22T08:03:37.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/20' 2023-06-22T08:03:37.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/39' 2023-06-22T08:03:37.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/34' 2023-06-22T08:03:37.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/38' 2023-06-22T08:03:37.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/28' 2023-06-22T08:03:37.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/24' 2023-06-22T08:03:37.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/29' 2023-06-22T08:03:37.241 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/32' 2023-06-22T08:03:37.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/22' 2023-06-22T08:03:37.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/25' 2023-06-22T08:03:37.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/30' 2023-06-22T08:03:37.242 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/33' 2023-06-22T08:03:37.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/23' 2023-06-22T08:03:37.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/35' 2023-06-22T08:03:37.243 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/26' 2023-06-22T08:03:37.244 INFO:tasks.workunit.client.0.smithi023.stdout:removed './248/37' 2023-06-22T08:03:37.244 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './248' 2023-06-22T08:03:37.244 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/21' 2023-06-22T08:03:37.245 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/27' 2023-06-22T08:03:37.245 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/31' 2023-06-22T08:03:37.245 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/36' 2023-06-22T08:03:37.245 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/20' 2023-06-22T08:03:37.246 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/39' 2023-06-22T08:03:37.246 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/34' 2023-06-22T08:03:37.246 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/38' 2023-06-22T08:03:37.246 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/28' 2023-06-22T08:03:37.247 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/24' 2023-06-22T08:03:37.247 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/29' 2023-06-22T08:03:37.247 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/32' 2023-06-22T08:03:37.247 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/22' 2023-06-22T08:03:37.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/25' 2023-06-22T08:03:37.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/30' 2023-06-22T08:03:37.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/33' 2023-06-22T08:03:37.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/23' 2023-06-22T08:03:37.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/35' 2023-06-22T08:03:37.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/26' 2023-06-22T08:03:37.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed './249/37' 2023-06-22T08:03:37.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './249' 2023-06-22T08:03:37.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/21' 2023-06-22T08:03:37.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/27' 2023-06-22T08:03:37.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/31' 2023-06-22T08:03:37.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/36' 2023-06-22T08:03:37.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/20' 2023-06-22T08:03:37.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/39' 2023-06-22T08:03:37.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/34' 2023-06-22T08:03:37.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/38' 2023-06-22T08:03:37.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/28' 2023-06-22T08:03:37.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/24' 2023-06-22T08:03:37.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/29' 2023-06-22T08:03:37.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/32' 2023-06-22T08:03:37.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/22' 2023-06-22T08:03:37.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/25' 2023-06-22T08:03:37.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/30' 2023-06-22T08:03:37.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/33' 2023-06-22T08:03:37.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/23' 2023-06-22T08:03:37.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/35' 2023-06-22T08:03:37.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/26' 2023-06-22T08:03:37.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed './250/37' 2023-06-22T08:03:37.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './250' 2023-06-22T08:03:37.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/21' 2023-06-22T08:03:37.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/27' 2023-06-22T08:03:37.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/31' 2023-06-22T08:03:37.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/36' 2023-06-22T08:03:37.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/20' 2023-06-22T08:03:37.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/39' 2023-06-22T08:03:37.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/34' 2023-06-22T08:03:37.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/38' 2023-06-22T08:03:37.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/28' 2023-06-22T08:03:37.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/24' 2023-06-22T08:03:37.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/29' 2023-06-22T08:03:37.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/32' 2023-06-22T08:03:37.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/22' 2023-06-22T08:03:37.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/25' 2023-06-22T08:03:37.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/30' 2023-06-22T08:03:37.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/33' 2023-06-22T08:03:37.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/23' 2023-06-22T08:03:37.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/35' 2023-06-22T08:03:37.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/26' 2023-06-22T08:03:37.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed './251/37' 2023-06-22T08:03:37.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './251' 2023-06-22T08:03:37.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/21' 2023-06-22T08:03:37.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/27' 2023-06-22T08:03:37.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/31' 2023-06-22T08:03:37.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/36' 2023-06-22T08:03:37.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/20' 2023-06-22T08:03:37.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/39' 2023-06-22T08:03:37.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/34' 2023-06-22T08:03:37.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/38' 2023-06-22T08:03:37.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/28' 2023-06-22T08:03:37.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/24' 2023-06-22T08:03:37.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/29' 2023-06-22T08:03:37.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/32' 2023-06-22T08:03:37.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/22' 2023-06-22T08:03:37.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/25' 2023-06-22T08:03:37.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/30' 2023-06-22T08:03:37.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/33' 2023-06-22T08:03:37.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/23' 2023-06-22T08:03:37.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/35' 2023-06-22T08:03:37.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/26' 2023-06-22T08:03:37.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed './252/37' 2023-06-22T08:03:37.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './252' 2023-06-22T08:03:37.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/21' 2023-06-22T08:03:37.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/27' 2023-06-22T08:03:37.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/31' 2023-06-22T08:03:37.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/36' 2023-06-22T08:03:37.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/20' 2023-06-22T08:03:37.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/39' 2023-06-22T08:03:37.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/34' 2023-06-22T08:03:37.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/38' 2023-06-22T08:03:37.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/28' 2023-06-22T08:03:37.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/24' 2023-06-22T08:03:37.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/29' 2023-06-22T08:03:37.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/32' 2023-06-22T08:03:37.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/22' 2023-06-22T08:03:37.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/25' 2023-06-22T08:03:37.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/30' 2023-06-22T08:03:37.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/33' 2023-06-22T08:03:37.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/23' 2023-06-22T08:03:37.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/35' 2023-06-22T08:03:37.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/26' 2023-06-22T08:03:37.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './253/37' 2023-06-22T08:03:37.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './253' 2023-06-22T08:03:37.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/21' 2023-06-22T08:03:37.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/27' 2023-06-22T08:03:37.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/31' 2023-06-22T08:03:37.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/36' 2023-06-22T08:03:37.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/20' 2023-06-22T08:03:37.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/39' 2023-06-22T08:03:37.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/34' 2023-06-22T08:03:37.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/38' 2023-06-22T08:03:37.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/28' 2023-06-22T08:03:37.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/24' 2023-06-22T08:03:37.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/29' 2023-06-22T08:03:37.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/32' 2023-06-22T08:03:37.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/22' 2023-06-22T08:03:37.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/25' 2023-06-22T08:03:37.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/30' 2023-06-22T08:03:37.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/33' 2023-06-22T08:03:37.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/23' 2023-06-22T08:03:37.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/35' 2023-06-22T08:03:37.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/26' 2023-06-22T08:03:37.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './254/37' 2023-06-22T08:03:37.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './254' 2023-06-22T08:03:37.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/21' 2023-06-22T08:03:37.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/27' 2023-06-22T08:03:37.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/31' 2023-06-22T08:03:37.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/36' 2023-06-22T08:03:37.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/20' 2023-06-22T08:03:37.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/39' 2023-06-22T08:03:37.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/34' 2023-06-22T08:03:37.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/38' 2023-06-22T08:03:37.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/28' 2023-06-22T08:03:37.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/24' 2023-06-22T08:03:37.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/29' 2023-06-22T08:03:37.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/32' 2023-06-22T08:03:37.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/22' 2023-06-22T08:03:37.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/25' 2023-06-22T08:03:37.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/30' 2023-06-22T08:03:37.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/33' 2023-06-22T08:03:37.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/23' 2023-06-22T08:03:37.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/35' 2023-06-22T08:03:37.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/26' 2023-06-22T08:03:37.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './255/37' 2023-06-22T08:03:37.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './255' 2023-06-22T08:03:37.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/21' 2023-06-22T08:03:37.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/27' 2023-06-22T08:03:37.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/31' 2023-06-22T08:03:37.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/36' 2023-06-22T08:03:37.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/20' 2023-06-22T08:03:37.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/39' 2023-06-22T08:03:37.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/34' 2023-06-22T08:03:37.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/38' 2023-06-22T08:03:37.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/28' 2023-06-22T08:03:37.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/24' 2023-06-22T08:03:37.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/29' 2023-06-22T08:03:37.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/32' 2023-06-22T08:03:37.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/22' 2023-06-22T08:03:37.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/25' 2023-06-22T08:03:37.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/30' 2023-06-22T08:03:37.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/33' 2023-06-22T08:03:37.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/23' 2023-06-22T08:03:37.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/35' 2023-06-22T08:03:37.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/26' 2023-06-22T08:03:37.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './256/37' 2023-06-22T08:03:37.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './256' 2023-06-22T08:03:37.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/21' 2023-06-22T08:03:37.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/27' 2023-06-22T08:03:37.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/31' 2023-06-22T08:03:37.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/36' 2023-06-22T08:03:37.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/20' 2023-06-22T08:03:37.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/39' 2023-06-22T08:03:37.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/34' 2023-06-22T08:03:37.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/38' 2023-06-22T08:03:37.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/28' 2023-06-22T08:03:37.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/24' 2023-06-22T08:03:37.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/29' 2023-06-22T08:03:37.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/32' 2023-06-22T08:03:37.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/22' 2023-06-22T08:03:37.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/25' 2023-06-22T08:03:37.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/30' 2023-06-22T08:03:37.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/33' 2023-06-22T08:03:37.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/23' 2023-06-22T08:03:37.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/35' 2023-06-22T08:03:37.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/26' 2023-06-22T08:03:37.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './257/37' 2023-06-22T08:03:37.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './257' 2023-06-22T08:03:37.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/21' 2023-06-22T08:03:37.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/27' 2023-06-22T08:03:37.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/31' 2023-06-22T08:03:37.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/36' 2023-06-22T08:03:37.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/20' 2023-06-22T08:03:37.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/39' 2023-06-22T08:03:37.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/34' 2023-06-22T08:03:37.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/38' 2023-06-22T08:03:37.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/28' 2023-06-22T08:03:37.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/24' 2023-06-22T08:03:37.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/29' 2023-06-22T08:03:37.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/32' 2023-06-22T08:03:37.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/22' 2023-06-22T08:03:37.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/25' 2023-06-22T08:03:37.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/30' 2023-06-22T08:03:37.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/33' 2023-06-22T08:03:37.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/23' 2023-06-22T08:03:37.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/35' 2023-06-22T08:03:37.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/26' 2023-06-22T08:03:37.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed './258/37' 2023-06-22T08:03:37.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './258' 2023-06-22T08:03:37.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/21' 2023-06-22T08:03:37.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/27' 2023-06-22T08:03:37.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/31' 2023-06-22T08:03:37.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/36' 2023-06-22T08:03:37.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/20' 2023-06-22T08:03:37.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/39' 2023-06-22T08:03:37.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/34' 2023-06-22T08:03:37.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/38' 2023-06-22T08:03:37.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/28' 2023-06-22T08:03:37.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/24' 2023-06-22T08:03:37.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/29' 2023-06-22T08:03:37.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/32' 2023-06-22T08:03:37.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/22' 2023-06-22T08:03:37.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/25' 2023-06-22T08:03:37.943 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/30' 2023-06-22T08:03:37.943 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/33' 2023-06-22T08:03:37.943 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/23' 2023-06-22T08:03:37.943 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/35' 2023-06-22T08:03:37.943 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/26' 2023-06-22T08:03:37.943 INFO:tasks.workunit.client.0.smithi023.stdout:removed './259/37' 2023-06-22T08:03:37.943 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './259' 2023-06-22T08:03:37.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/21' 2023-06-22T08:03:37.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/27' 2023-06-22T08:03:37.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/31' 2023-06-22T08:03:37.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/36' 2023-06-22T08:03:37.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/20' 2023-06-22T08:03:37.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/39' 2023-06-22T08:03:37.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/34' 2023-06-22T08:03:37.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/38' 2023-06-22T08:03:37.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/28' 2023-06-22T08:03:37.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/24' 2023-06-22T08:03:37.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/29' 2023-06-22T08:03:37.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/32' 2023-06-22T08:03:37.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/22' 2023-06-22T08:03:37.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/25' 2023-06-22T08:03:37.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/30' 2023-06-22T08:03:37.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/33' 2023-06-22T08:03:37.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/23' 2023-06-22T08:03:37.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/35' 2023-06-22T08:03:37.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/26' 2023-06-22T08:03:37.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed './260/37' 2023-06-22T08:03:37.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './260' 2023-06-22T08:03:37.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/21' 2023-06-22T08:03:37.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/27' 2023-06-22T08:03:37.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/31' 2023-06-22T08:03:37.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/36' 2023-06-22T08:03:37.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/20' 2023-06-22T08:03:37.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/39' 2023-06-22T08:03:37.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/34' 2023-06-22T08:03:37.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/38' 2023-06-22T08:03:37.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/28' 2023-06-22T08:03:37.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/24' 2023-06-22T08:03:37.948 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/29' 2023-06-22T08:03:37.948 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/32' 2023-06-22T08:03:37.948 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/22' 2023-06-22T08:03:37.948 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/25' 2023-06-22T08:03:37.948 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/30' 2023-06-22T08:03:37.948 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/33' 2023-06-22T08:03:37.948 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/23' 2023-06-22T08:03:37.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/35' 2023-06-22T08:03:37.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/26' 2023-06-22T08:03:37.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed './261/37' 2023-06-22T08:03:37.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './261' 2023-06-22T08:03:37.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/21' 2023-06-22T08:03:37.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/27' 2023-06-22T08:03:37.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/31' 2023-06-22T08:03:37.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/36' 2023-06-22T08:03:37.950 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/20' 2023-06-22T08:03:37.950 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/39' 2023-06-22T08:03:37.950 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/34' 2023-06-22T08:03:37.950 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/38' 2023-06-22T08:03:37.950 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/28' 2023-06-22T08:03:37.950 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/24' 2023-06-22T08:03:37.950 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/29' 2023-06-22T08:03:37.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/32' 2023-06-22T08:03:37.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/22' 2023-06-22T08:03:37.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/25' 2023-06-22T08:03:37.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/30' 2023-06-22T08:03:37.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/33' 2023-06-22T08:03:37.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/23' 2023-06-22T08:03:37.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/35' 2023-06-22T08:03:37.952 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/26' 2023-06-22T08:03:37.952 INFO:tasks.workunit.client.0.smithi023.stdout:removed './262/37' 2023-06-22T08:03:37.952 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './262' 2023-06-22T08:03:37.952 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/21' 2023-06-22T08:03:37.952 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/27' 2023-06-22T08:03:37.952 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/31' 2023-06-22T08:03:37.952 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/36' 2023-06-22T08:03:37.953 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/20' 2023-06-22T08:03:37.953 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/39' 2023-06-22T08:03:37.953 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/34' 2023-06-22T08:03:37.953 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/38' 2023-06-22T08:03:37.953 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/28' 2023-06-22T08:03:37.954 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/24' 2023-06-22T08:03:37.954 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/29' 2023-06-22T08:03:37.954 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/32' 2023-06-22T08:03:37.954 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/22' 2023-06-22T08:03:37.954 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/25' 2023-06-22T08:03:37.954 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/30' 2023-06-22T08:03:37.954 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/33' 2023-06-22T08:03:37.954 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/23' 2023-06-22T08:03:37.955 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/35' 2023-06-22T08:03:37.955 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/26' 2023-06-22T08:03:37.955 INFO:tasks.workunit.client.0.smithi023.stdout:removed './263/37' 2023-06-22T08:03:37.955 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './263' 2023-06-22T08:03:37.955 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/21' 2023-06-22T08:03:37.955 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/27' 2023-06-22T08:03:37.955 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/31' 2023-06-22T08:03:37.956 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/36' 2023-06-22T08:03:37.956 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/20' 2023-06-22T08:03:37.956 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/39' 2023-06-22T08:03:37.956 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/34' 2023-06-22T08:03:37.956 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/38' 2023-06-22T08:03:37.956 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/28' 2023-06-22T08:03:37.956 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/24' 2023-06-22T08:03:37.957 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/29' 2023-06-22T08:03:37.957 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/32' 2023-06-22T08:03:37.957 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/22' 2023-06-22T08:03:37.957 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/25' 2023-06-22T08:03:37.957 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/30' 2023-06-22T08:03:37.957 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/33' 2023-06-22T08:03:37.957 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/23' 2023-06-22T08:03:37.957 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/35' 2023-06-22T08:03:37.958 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/26' 2023-06-22T08:03:37.958 INFO:tasks.workunit.client.0.smithi023.stdout:removed './264/37' 2023-06-22T08:03:37.958 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './264' 2023-06-22T08:03:37.958 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/21' 2023-06-22T08:03:37.958 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/27' 2023-06-22T08:03:37.958 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/31' 2023-06-22T08:03:37.959 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/36' 2023-06-22T08:03:37.959 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/20' 2023-06-22T08:03:37.959 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/39' 2023-06-22T08:03:37.959 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/34' 2023-06-22T08:03:37.959 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/38' 2023-06-22T08:03:37.959 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/28' 2023-06-22T08:03:37.959 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/24' 2023-06-22T08:03:37.960 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/29' 2023-06-22T08:03:37.960 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/32' 2023-06-22T08:03:37.960 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/22' 2023-06-22T08:03:37.960 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/25' 2023-06-22T08:03:37.960 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/30' 2023-06-22T08:03:37.960 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/33' 2023-06-22T08:03:37.960 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/23' 2023-06-22T08:03:37.961 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/35' 2023-06-22T08:03:37.961 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/26' 2023-06-22T08:03:37.961 INFO:tasks.workunit.client.0.smithi023.stdout:removed './265/37' 2023-06-22T08:03:37.961 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './265' 2023-06-22T08:03:37.961 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/21' 2023-06-22T08:03:37.961 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/27' 2023-06-22T08:03:37.962 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/31' 2023-06-22T08:03:37.962 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/36' 2023-06-22T08:03:37.962 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/20' 2023-06-22T08:03:37.962 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/39' 2023-06-22T08:03:37.962 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/34' 2023-06-22T08:03:37.962 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/38' 2023-06-22T08:03:37.962 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/28' 2023-06-22T08:03:37.962 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/24' 2023-06-22T08:03:37.963 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/29' 2023-06-22T08:03:37.963 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/32' 2023-06-22T08:03:37.963 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/22' 2023-06-22T08:03:37.963 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/25' 2023-06-22T08:03:37.963 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/30' 2023-06-22T08:03:37.963 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/33' 2023-06-22T08:03:37.964 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/23' 2023-06-22T08:03:38.127 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:38.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.360+0000 7efd0b5ff700 1 Processor -- start 2023-06-22T08:03:38.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.360+0000 7efd0b5ff700 1 -- start start 2023-06-22T08:03:38.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd0b5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd040c3a40 0x7efd040c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:38.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd0b5ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efd0406d050 0x7efd04159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:38.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd0b5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 0x7efd0415a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:38.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd0b5ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd04157570 con 0x7efd0406d050 2023-06-22T08:03:38.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd0b5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd040c4350 con 0x7efd040c3a40 2023-06-22T08:03:38.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd0b5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd040c6a60 con 0x7efd04156fc0 2023-06-22T08:03:38.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd09b9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 0x7efd0415a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd0939b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd040c3a40 0x7efd040c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd09b9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 0x7efd0415a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:57546/0 (socket says 172.21.15.23:57546) 2023-06-22T08:03:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd0939b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd040c3a40 0x7efd040c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:41616/0 (socket says 172.21.15.23:41616) 2023-06-22T08:03:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd09b9c700 1 -- 172.21.15.23:0/1678392524 learned_addr learned my addr 172.21.15.23:0/1678392524 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.362+0000 7efd08b9a700 1 --2- 172.21.15.23:0/1678392524 >> v2:172.21.15.23:3300/0 conn(0x7efd0406d050 0x7efd04159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.363+0000 7efd09b9c700 1 -- 172.21.15.23:0/1678392524 >> v2:172.21.15.67:3300/0 conn(0x7efd040c3a40 msgr2=0x7efd040c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.363+0000 7efd09b9c700 1 --2- 172.21.15.23:0/1678392524 >> v2:172.21.15.67:3300/0 conn(0x7efd040c3a40 0x7efd040c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.363+0000 7efd09b9c700 1 -- 172.21.15.23:0/1678392524 >> v2:172.21.15.23:3300/0 conn(0x7efd0406d050 msgr2=0x7efd04159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.363+0000 7efd09b9c700 1 --2- 172.21.15.23:0/1678392524 >> v2:172.21.15.23:3300/0 conn(0x7efd0406d050 0x7efd04159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.363+0000 7efd09b9c700 1 -- 172.21.15.23:0/1678392524 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0414d690 con 0x7efd04156fc0 2023-06-22T08:03:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.363+0000 7efd0939b700 1 --2- 172.21.15.23:0/1678392524 >> v2:172.21.15.67:3300/0 conn(0x7efd040c3a40 0x7efd040c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:03:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.363+0000 7efd09b9c700 1 --2- 172.21.15.23:0/1678392524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 0x7efd0415a150 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd0406af80 tx=0x7efcfc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8405b701164ebc41 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.363+0000 7efcfbfff700 1 -- 172.21.15.23:0/1678392524 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00e070 con 0x7efd04156fc0 2023-06-22T08:03:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.363+0000 7efcfbfff700 1 -- 172.21.15.23:0/1678392524 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc00b910 con 0x7efd04156fc0 2023-06-22T08:03:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.363+0000 7efcfbfff700 1 -- 172.21.15.23:0/1678392524 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00bcc0 con 0x7efd04156fc0 2023-06-22T08:03:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.364+0000 7efd0b5ff700 1 -- 172.21.15.23:0/1678392524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 msgr2=0x7efd0415a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.364+0000 7efd0b5ff700 1 --2- 172.21.15.23:0/1678392524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 0x7efd0415a150 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd0406af80 tx=0x7efcfc00b330 comp rx=0 tx=0).stop 2023-06-22T08:03:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.364+0000 7efd0b5ff700 1 -- 172.21.15.23:0/1678392524 shutdown_connections 2023-06-22T08:03:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.364+0000 7efd0b5ff700 1 --2- 172.21.15.23:0/1678392524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 0x7efd0415a150 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.364+0000 7efd0b5ff700 1 --2- 172.21.15.23:0/1678392524 >> v2:172.21.15.67:3300/0 conn(0x7efd040c3a40 0x7efd040c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.364+0000 7efd0b5ff700 1 --2- 172.21.15.23:0/1678392524 >> v2:172.21.15.23:3300/0 conn(0x7efd0406d050 0x7efd04159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.364+0000 7efd0b5ff700 1 -- 172.21.15.23:0/1678392524 >> 172.21.15.23:0/1678392524 conn(0x7efd040ca3f0 msgr2=0x7efd040ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:38.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.364+0000 7efd0b5ff700 1 -- 172.21.15.23:0/1678392524 shutdown_connections 2023-06-22T08:03:38.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.364+0000 7efd0b5ff700 1 -- 172.21.15.23:0/1678392524 wait complete. 2023-06-22T08:03:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.367+0000 7efd0b5ff700 1 Processor -- start 2023-06-22T08:03:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.367+0000 7efd0b5ff700 1 -- start start 2023-06-22T08:03:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd0b5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd0406d050 0x7efd04144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd0b5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd040c3a40 0x7efd04155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd0b5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 0x7efd04155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd0b5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd04150280 con 0x7efd0406d050 2023-06-22T08:03:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd0b5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd04144890 con 0x7efd040c3a40 2023-06-22T08:03:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd0b5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd040ca070 con 0x7efd04156fc0 2023-06-22T08:03:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd0939b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd0406d050 0x7efd04144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd0939b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd0406d050 0x7efd04144350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46404/0 (socket says 172.21.15.23:46404) 2023-06-22T08:03:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd0939b700 1 -- 172.21.15.23:0/3503301565 learned_addr learned my addr 172.21.15.23:0/3503301565 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd09b9c700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 0x7efd04155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.369+0000 7efd08b9a700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd040c3a40 0x7efd04155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efd0939b700 1 -- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 msgr2=0x7efd04155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efd0939b700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 0x7efd04155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efd0939b700 1 -- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd040c3a40 msgr2=0x7efd04155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efd0939b700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd040c3a40 0x7efd04155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efd0939b700 1 -- 172.21.15.23:0/3503301565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd000094a0 con 0x7efd0406d050 2023-06-22T08:03:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efd0939b700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd0406d050 0x7efd04144350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efd0000ebe0 tx=0x7efd0000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efcf9ffb700 1 -- 172.21.15.23:0/3503301565 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0000c850 con 0x7efd0406d050 2023-06-22T08:03:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efcf9ffb700 1 -- 172.21.15.23:0/3503301565 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0000c9b0 con 0x7efd0406d050 2023-06-22T08:03:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efcf9ffb700 1 -- 172.21.15.23:0/3503301565 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0000cd40 con 0x7efd0406d050 2023-06-22T08:03:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efd0b5ff700 1 -- 172.21.15.23:0/3503301565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcfc009b30 con 0x7efd0406d050 2023-06-22T08:03:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.370+0000 7efd0b5ff700 1 -- 172.21.15.23:0/3503301565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd040c97d0 con 0x7efd0406d050 2023-06-22T08:03:38.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.372+0000 7efd0b5ff700 1 -- 172.21.15.23:0/3503301565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0404ea20 con 0x7efd0406d050 2023-06-22T08:03:38.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.373+0000 7efcf9ffb700 1 -- 172.21.15.23:0/3503301565 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efd00013470 con 0x7efd0406d050 2023-06-22T08:03:38.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.374+0000 7efcf9ffb700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efce806ace0 0x7efce806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:38.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.374+0000 7efcf9ffb700 1 -- 172.21.15.23:0/3503301565 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7efd00089cc0 con 0x7efd0406d050 2023-06-22T08:03:38.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.375+0000 7efd08b9a700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efce806ace0 0x7efce806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:38.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.375+0000 7efd08b9a700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efce806ace0 0x7efce806d190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efd0414cda0 tx=0x7efcf4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:38.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.382+0000 7efcf9ffb700 1 -- 172.21.15.23:0/3503301565 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efd00090080 con 0x7efd0406d050 2023-06-22T08:03:38.548 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/35' 2023-06-22T08:03:38.549 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/26' 2023-06-22T08:03:38.549 INFO:tasks.workunit.client.0.smithi023.stdout:removed './266/37' 2023-06-22T08:03:38.549 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './266' 2023-06-22T08:03:38.549 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/21' 2023-06-22T08:03:38.549 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/27' 2023-06-22T08:03:38.549 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/31' 2023-06-22T08:03:38.550 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/36' 2023-06-22T08:03:38.550 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/20' 2023-06-22T08:03:38.550 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/39' 2023-06-22T08:03:38.550 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/34' 2023-06-22T08:03:38.550 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/38' 2023-06-22T08:03:38.550 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/28' 2023-06-22T08:03:38.550 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/24' 2023-06-22T08:03:38.551 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/29' 2023-06-22T08:03:38.551 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/32' 2023-06-22T08:03:38.551 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/22' 2023-06-22T08:03:38.551 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/25' 2023-06-22T08:03:38.551 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/30' 2023-06-22T08:03:38.551 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/33' 2023-06-22T08:03:38.551 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/23' 2023-06-22T08:03:38.552 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/35' 2023-06-22T08:03:38.552 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/26' 2023-06-22T08:03:38.552 INFO:tasks.workunit.client.0.smithi023.stdout:removed './267/37' 2023-06-22T08:03:38.552 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './267' 2023-06-22T08:03:38.552 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/21' 2023-06-22T08:03:38.552 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/27' 2023-06-22T08:03:38.552 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/31' 2023-06-22T08:03:38.552 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/36' 2023-06-22T08:03:38.553 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/20' 2023-06-22T08:03:38.553 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/39' 2023-06-22T08:03:38.553 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/34' 2023-06-22T08:03:38.553 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/38' 2023-06-22T08:03:38.553 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/28' 2023-06-22T08:03:38.553 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/24' 2023-06-22T08:03:38.553 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/29' 2023-06-22T08:03:38.554 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/32' 2023-06-22T08:03:38.554 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/22' 2023-06-22T08:03:38.554 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/25' 2023-06-22T08:03:38.554 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/30' 2023-06-22T08:03:38.554 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/33' 2023-06-22T08:03:38.554 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/23' 2023-06-22T08:03:38.555 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/35' 2023-06-22T08:03:38.555 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/26' 2023-06-22T08:03:38.555 INFO:tasks.workunit.client.0.smithi023.stdout:removed './268/37' 2023-06-22T08:03:38.555 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './268' 2023-06-22T08:03:38.555 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/21' 2023-06-22T08:03:38.555 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/27' 2023-06-22T08:03:38.555 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/31' 2023-06-22T08:03:38.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/36' 2023-06-22T08:03:38.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/20' 2023-06-22T08:03:38.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/39' 2023-06-22T08:03:38.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/34' 2023-06-22T08:03:38.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/38' 2023-06-22T08:03:38.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/28' 2023-06-22T08:03:38.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/24' 2023-06-22T08:03:38.556 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/29' 2023-06-22T08:03:38.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/32' 2023-06-22T08:03:38.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/22' 2023-06-22T08:03:38.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/25' 2023-06-22T08:03:38.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/30' 2023-06-22T08:03:38.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/33' 2023-06-22T08:03:38.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/23' 2023-06-22T08:03:38.557 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/35' 2023-06-22T08:03:38.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/26' 2023-06-22T08:03:38.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './269/37' 2023-06-22T08:03:38.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './269' 2023-06-22T08:03:38.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/21' 2023-06-22T08:03:38.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/27' 2023-06-22T08:03:38.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/31' 2023-06-22T08:03:38.558 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/36' 2023-06-22T08:03:38.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/20' 2023-06-22T08:03:38.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/39' 2023-06-22T08:03:38.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/34' 2023-06-22T08:03:38.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/38' 2023-06-22T08:03:38.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/28' 2023-06-22T08:03:38.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/24' 2023-06-22T08:03:38.559 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/29' 2023-06-22T08:03:38.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/32' 2023-06-22T08:03:38.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/22' 2023-06-22T08:03:38.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/25' 2023-06-22T08:03:38.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/30' 2023-06-22T08:03:38.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/33' 2023-06-22T08:03:38.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/23' 2023-06-22T08:03:38.560 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/35' 2023-06-22T08:03:38.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/26' 2023-06-22T08:03:38.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './270/37' 2023-06-22T08:03:38.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './270' 2023-06-22T08:03:38.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/21' 2023-06-22T08:03:38.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/27' 2023-06-22T08:03:38.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/31' 2023-06-22T08:03:38.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/36' 2023-06-22T08:03:38.561 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/20' 2023-06-22T08:03:38.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/39' 2023-06-22T08:03:38.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/34' 2023-06-22T08:03:38.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/38' 2023-06-22T08:03:38.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/28' 2023-06-22T08:03:38.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/24' 2023-06-22T08:03:38.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/29' 2023-06-22T08:03:38.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/32' 2023-06-22T08:03:38.562 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/22' 2023-06-22T08:03:38.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/25' 2023-06-22T08:03:38.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/30' 2023-06-22T08:03:38.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/33' 2023-06-22T08:03:38.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/23' 2023-06-22T08:03:38.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/35' 2023-06-22T08:03:38.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/26' 2023-06-22T08:03:38.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './271/37' 2023-06-22T08:03:38.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './271' 2023-06-22T08:03:38.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/21' 2023-06-22T08:03:38.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/27' 2023-06-22T08:03:38.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/31' 2023-06-22T08:03:38.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/36' 2023-06-22T08:03:38.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/20' 2023-06-22T08:03:38.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/39' 2023-06-22T08:03:38.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/34' 2023-06-22T08:03:38.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/38' 2023-06-22T08:03:38.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/28' 2023-06-22T08:03:38.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/24' 2023-06-22T08:03:38.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/29' 2023-06-22T08:03:38.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/32' 2023-06-22T08:03:38.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/22' 2023-06-22T08:03:38.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/25' 2023-06-22T08:03:38.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/30' 2023-06-22T08:03:38.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/33' 2023-06-22T08:03:38.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/23' 2023-06-22T08:03:38.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/35' 2023-06-22T08:03:38.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/26' 2023-06-22T08:03:38.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './272/37' 2023-06-22T08:03:38.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './272' 2023-06-22T08:03:38.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/21' 2023-06-22T08:03:38.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/27' 2023-06-22T08:03:38.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/31' 2023-06-22T08:03:38.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/36' 2023-06-22T08:03:38.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/20' 2023-06-22T08:03:38.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/39' 2023-06-22T08:03:38.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/34' 2023-06-22T08:03:38.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/38' 2023-06-22T08:03:38.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/28' 2023-06-22T08:03:38.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/24' 2023-06-22T08:03:38.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/29' 2023-06-22T08:03:38.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/32' 2023-06-22T08:03:38.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/22' 2023-06-22T08:03:38.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/25' 2023-06-22T08:03:38.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/30' 2023-06-22T08:03:38.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/33' 2023-06-22T08:03:38.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/23' 2023-06-22T08:03:38.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/35' 2023-06-22T08:03:38.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/26' 2023-06-22T08:03:38.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './273/37' 2023-06-22T08:03:38.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './273' 2023-06-22T08:03:38.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/21' 2023-06-22T08:03:38.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/27' 2023-06-22T08:03:38.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/31' 2023-06-22T08:03:38.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/36' 2023-06-22T08:03:38.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/20' 2023-06-22T08:03:38.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/39' 2023-06-22T08:03:38.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/34' 2023-06-22T08:03:38.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/38' 2023-06-22T08:03:38.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/28' 2023-06-22T08:03:38.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/24' 2023-06-22T08:03:38.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/29' 2023-06-22T08:03:38.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/32' 2023-06-22T08:03:38.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/22' 2023-06-22T08:03:38.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/25' 2023-06-22T08:03:38.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/30' 2023-06-22T08:03:38.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/33' 2023-06-22T08:03:38.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/23' 2023-06-22T08:03:38.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/35' 2023-06-22T08:03:38.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/26' 2023-06-22T08:03:38.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './274/37' 2023-06-22T08:03:38.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './274' 2023-06-22T08:03:38.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/21' 2023-06-22T08:03:38.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/27' 2023-06-22T08:03:38.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/31' 2023-06-22T08:03:38.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/36' 2023-06-22T08:03:38.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/20' 2023-06-22T08:03:38.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/39' 2023-06-22T08:03:38.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/34' 2023-06-22T08:03:38.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/38' 2023-06-22T08:03:38.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/28' 2023-06-22T08:03:38.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/24' 2023-06-22T08:03:38.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/29' 2023-06-22T08:03:38.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/32' 2023-06-22T08:03:38.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/22' 2023-06-22T08:03:38.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/25' 2023-06-22T08:03:38.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/30' 2023-06-22T08:03:38.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/33' 2023-06-22T08:03:38.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/23' 2023-06-22T08:03:38.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/35' 2023-06-22T08:03:38.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/26' 2023-06-22T08:03:38.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './275/37' 2023-06-22T08:03:38.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './275' 2023-06-22T08:03:38.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/21' 2023-06-22T08:03:38.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/27' 2023-06-22T08:03:38.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/31' 2023-06-22T08:03:38.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/36' 2023-06-22T08:03:38.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/20' 2023-06-22T08:03:38.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/39' 2023-06-22T08:03:38.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/34' 2023-06-22T08:03:38.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/38' 2023-06-22T08:03:38.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/28' 2023-06-22T08:03:38.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/24' 2023-06-22T08:03:38.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/29' 2023-06-22T08:03:38.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/32' 2023-06-22T08:03:38.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/22' 2023-06-22T08:03:38.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/25' 2023-06-22T08:03:38.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/30' 2023-06-22T08:03:38.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/33' 2023-06-22T08:03:38.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/23' 2023-06-22T08:03:38.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/35' 2023-06-22T08:03:38.578 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/26' 2023-06-22T08:03:38.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.827+0000 7efd0b5ff700 1 -- 172.21.15.23:0/3503301565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd040c5f30 con 0x7efd0406d050 2023-06-22T08:03:38.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.828+0000 7efcf9ffb700 1 -- 172.21.15.23:0/3503301565 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+6256 (secure 0 0 0) 0x7efd00054980 con 0x7efd0406d050 2023-06-22T08:03:38.832 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:38.833 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5211,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:31.679109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:active","state_seq":59,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:38.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.833+0000 7efd0b5ff700 1 -- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efce806ace0 msgr2=0x7efce806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:38.837 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.834+0000 7efd0b5ff700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efce806ace0 0x7efce806d190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efd0414cda0 tx=0x7efcf4009210 comp rx=0 tx=0).stop 2023-06-22T08:03:38.837 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.834+0000 7efd0b5ff700 1 -- 172.21.15.23:0/3503301565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd0406d050 msgr2=0x7efd04144350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:38.837 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.834+0000 7efd0b5ff700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd0406d050 0x7efd04144350 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efd0000ebe0 tx=0x7efd0000eef0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.834+0000 7efd0b5ff700 1 -- 172.21.15.23:0/3503301565 shutdown_connections 2023-06-22T08:03:38.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.834+0000 7efd0b5ff700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd04156fc0 0x7efd04155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.834+0000 7efd0b5ff700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd0406d050 0x7efd04144350 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.834+0000 7efd0b5ff700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efce806ace0 0x7efce806d190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.834+0000 7efd0b5ff700 1 --2- 172.21.15.23:0/3503301565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd040c3a40 0x7efd04155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:38.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.834+0000 7efd0b5ff700 1 -- 172.21.15.23:0/3503301565 >> 172.21.15.23:0/3503301565 conn(0x7efd040ca3f0 msgr2=0x7efd04159c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:38.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.835+0000 7efd0b5ff700 1 -- 172.21.15.23:0/3503301565 shutdown_connections 2023-06-22T08:03:38.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:38.835+0000 7efd0b5ff700 1 -- 172.21.15.23:0/3503301565 wait complete. 2023-06-22T08:03:38.842 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 32 2023-06-22T08:03:38.874 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-06-22T08:03:38.875 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before thrashing 2023-06-22T08:03:39.441 INFO:tasks.workunit.client.0.smithi023.stdout:removed './276/37' 2023-06-22T08:03:39.441 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './276' 2023-06-22T08:03:39.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/21' 2023-06-22T08:03:39.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/27' 2023-06-22T08:03:39.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/31' 2023-06-22T08:03:39.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/36' 2023-06-22T08:03:39.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/20' 2023-06-22T08:03:39.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/39' 2023-06-22T08:03:39.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/34' 2023-06-22T08:03:39.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/38' 2023-06-22T08:03:39.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/28' 2023-06-22T08:03:39.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/24' 2023-06-22T08:03:39.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/29' 2023-06-22T08:03:39.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/32' 2023-06-22T08:03:39.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/22' 2023-06-22T08:03:39.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/25' 2023-06-22T08:03:39.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/30' 2023-06-22T08:03:39.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/33' 2023-06-22T08:03:39.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/23' 2023-06-22T08:03:39.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/35' 2023-06-22T08:03:39.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/26' 2023-06-22T08:03:39.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './277/37' 2023-06-22T08:03:39.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './277' 2023-06-22T08:03:39.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/21' 2023-06-22T08:03:39.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/27' 2023-06-22T08:03:39.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/31' 2023-06-22T08:03:39.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/36' 2023-06-22T08:03:39.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/20' 2023-06-22T08:03:39.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/39' 2023-06-22T08:03:39.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/34' 2023-06-22T08:03:39.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/38' 2023-06-22T08:03:39.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/28' 2023-06-22T08:03:39.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/24' 2023-06-22T08:03:39.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/29' 2023-06-22T08:03:39.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/32' 2023-06-22T08:03:39.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/22' 2023-06-22T08:03:39.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/25' 2023-06-22T08:03:39.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/30' 2023-06-22T08:03:39.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/33' 2023-06-22T08:03:39.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/23' 2023-06-22T08:03:39.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/35' 2023-06-22T08:03:39.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/26' 2023-06-22T08:03:39.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './278/37' 2023-06-22T08:03:39.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './278' 2023-06-22T08:03:39.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/21' 2023-06-22T08:03:39.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/27' 2023-06-22T08:03:39.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/31' 2023-06-22T08:03:39.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/36' 2023-06-22T08:03:39.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/20' 2023-06-22T08:03:39.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/39' 2023-06-22T08:03:39.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/34' 2023-06-22T08:03:39.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/38' 2023-06-22T08:03:39.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/28' 2023-06-22T08:03:39.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/24' 2023-06-22T08:03:39.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/29' 2023-06-22T08:03:39.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/32' 2023-06-22T08:03:39.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/22' 2023-06-22T08:03:39.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/25' 2023-06-22T08:03:39.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/30' 2023-06-22T08:03:39.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/33' 2023-06-22T08:03:39.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/23' 2023-06-22T08:03:39.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/35' 2023-06-22T08:03:39.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/26' 2023-06-22T08:03:39.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed './279/37' 2023-06-22T08:03:39.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './279' 2023-06-22T08:03:39.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/21' 2023-06-22T08:03:39.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/27' 2023-06-22T08:03:39.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/31' 2023-06-22T08:03:39.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/36' 2023-06-22T08:03:39.453 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/20' 2023-06-22T08:03:39.453 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/39' 2023-06-22T08:03:39.453 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/34' 2023-06-22T08:03:39.454 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/38' 2023-06-22T08:03:39.454 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/28' 2023-06-22T08:03:39.454 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/24' 2023-06-22T08:03:39.454 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/29' 2023-06-22T08:03:39.455 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/32' 2023-06-22T08:03:39.455 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/22' 2023-06-22T08:03:39.455 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/25' 2023-06-22T08:03:39.456 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/30' 2023-06-22T08:03:39.456 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/33' 2023-06-22T08:03:39.456 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/23' 2023-06-22T08:03:39.456 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/35' 2023-06-22T08:03:39.457 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/26' 2023-06-22T08:03:39.457 INFO:tasks.workunit.client.0.smithi023.stdout:removed './280/37' 2023-06-22T08:03:39.457 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './280' 2023-06-22T08:03:39.457 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/21' 2023-06-22T08:03:39.458 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/27' 2023-06-22T08:03:39.458 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/31' 2023-06-22T08:03:39.458 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/36' 2023-06-22T08:03:39.459 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/20' 2023-06-22T08:03:39.459 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/39' 2023-06-22T08:03:39.459 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/34' 2023-06-22T08:03:39.459 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/38' 2023-06-22T08:03:39.459 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/28' 2023-06-22T08:03:39.460 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/24' 2023-06-22T08:03:39.460 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/29' 2023-06-22T08:03:39.460 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/32' 2023-06-22T08:03:39.460 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/22' 2023-06-22T08:03:39.460 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/25' 2023-06-22T08:03:39.461 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/30' 2023-06-22T08:03:39.461 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/33' 2023-06-22T08:03:39.461 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/23' 2023-06-22T08:03:39.462 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/35' 2023-06-22T08:03:39.462 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/26' 2023-06-22T08:03:39.462 INFO:tasks.workunit.client.0.smithi023.stdout:removed './281/37' 2023-06-22T08:03:39.462 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './281' 2023-06-22T08:03:39.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/21' 2023-06-22T08:03:39.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/27' 2023-06-22T08:03:39.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/31' 2023-06-22T08:03:39.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/36' 2023-06-22T08:03:39.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/20' 2023-06-22T08:03:39.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/39' 2023-06-22T08:03:39.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/34' 2023-06-22T08:03:39.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/38' 2023-06-22T08:03:39.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/28' 2023-06-22T08:03:39.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/24' 2023-06-22T08:03:39.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/29' 2023-06-22T08:03:39.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/32' 2023-06-22T08:03:39.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/22' 2023-06-22T08:03:39.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/25' 2023-06-22T08:03:39.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/30' 2023-06-22T08:03:39.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/33' 2023-06-22T08:03:39.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/23' 2023-06-22T08:03:39.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/35' 2023-06-22T08:03:39.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/26' 2023-06-22T08:03:39.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './282/37' 2023-06-22T08:03:39.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './282' 2023-06-22T08:03:39.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/21' 2023-06-22T08:03:39.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/27' 2023-06-22T08:03:39.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/31' 2023-06-22T08:03:39.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/36' 2023-06-22T08:03:39.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/20' 2023-06-22T08:03:39.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/39' 2023-06-22T08:03:39.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/34' 2023-06-22T08:03:39.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/38' 2023-06-22T08:03:39.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/28' 2023-06-22T08:03:39.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/24' 2023-06-22T08:03:39.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/29' 2023-06-22T08:03:39.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/32' 2023-06-22T08:03:39.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/22' 2023-06-22T08:03:39.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/25' 2023-06-22T08:03:39.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/30' 2023-06-22T08:03:39.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/33' 2023-06-22T08:03:39.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/23' 2023-06-22T08:03:39.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/35' 2023-06-22T08:03:39.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/26' 2023-06-22T08:03:39.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './283/37' 2023-06-22T08:03:39.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './283' 2023-06-22T08:03:39.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/21' 2023-06-22T08:03:39.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/27' 2023-06-22T08:03:39.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/31' 2023-06-22T08:03:39.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/36' 2023-06-22T08:03:39.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/20' 2023-06-22T08:03:39.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/39' 2023-06-22T08:03:39.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/34' 2023-06-22T08:03:39.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/38' 2023-06-22T08:03:39.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/28' 2023-06-22T08:03:39.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/24' 2023-06-22T08:03:39.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/29' 2023-06-22T08:03:39.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/32' 2023-06-22T08:03:39.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/22' 2023-06-22T08:03:39.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/25' 2023-06-22T08:03:39.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/30' 2023-06-22T08:03:39.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/33' 2023-06-22T08:03:39.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/23' 2023-06-22T08:03:39.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/35' 2023-06-22T08:03:39.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/26' 2023-06-22T08:03:39.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './284/37' 2023-06-22T08:03:39.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './284' 2023-06-22T08:03:39.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/21' 2023-06-22T08:03:39.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/27' 2023-06-22T08:03:39.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/31' 2023-06-22T08:03:39.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/36' 2023-06-22T08:03:39.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/20' 2023-06-22T08:03:39.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/39' 2023-06-22T08:03:39.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/34' 2023-06-22T08:03:39.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/38' 2023-06-22T08:03:39.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/28' 2023-06-22T08:03:39.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/24' 2023-06-22T08:03:39.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/29' 2023-06-22T08:03:39.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/32' 2023-06-22T08:03:39.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/22' 2023-06-22T08:03:39.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/25' 2023-06-22T08:03:39.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/30' 2023-06-22T08:03:39.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/33' 2023-06-22T08:03:39.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/23' 2023-06-22T08:03:39.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/35' 2023-06-22T08:03:39.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/26' 2023-06-22T08:03:39.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed './285/37' 2023-06-22T08:03:39.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './285' 2023-06-22T08:03:39.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/21' 2023-06-22T08:03:39.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/27' 2023-06-22T08:03:39.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/31' 2023-06-22T08:03:39.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/36' 2023-06-22T08:03:39.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/20' 2023-06-22T08:03:39.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/39' 2023-06-22T08:03:39.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/34' 2023-06-22T08:03:39.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/38' 2023-06-22T08:03:39.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/28' 2023-06-22T08:03:39.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/24' 2023-06-22T08:03:39.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/29' 2023-06-22T08:03:39.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/32' 2023-06-22T08:03:39.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/22' 2023-06-22T08:03:39.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/25' 2023-06-22T08:03:39.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/30' 2023-06-22T08:03:39.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/33' 2023-06-22T08:03:39.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/23' 2023-06-22T08:03:39.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/35' 2023-06-22T08:03:39.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/26' 2023-06-22T08:03:39.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed './286/37' 2023-06-22T08:03:39.477 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './286' 2023-06-22T08:03:40.187 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/21' 2023-06-22T08:03:40.187 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/27' 2023-06-22T08:03:40.187 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/31' 2023-06-22T08:03:40.188 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/36' 2023-06-22T08:03:40.188 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/20' 2023-06-22T08:03:40.188 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/39' 2023-06-22T08:03:40.188 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/34' 2023-06-22T08:03:40.188 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/38' 2023-06-22T08:03:40.188 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/28' 2023-06-22T08:03:40.188 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/24' 2023-06-22T08:03:40.189 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/29' 2023-06-22T08:03:40.189 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/32' 2023-06-22T08:03:40.189 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/22' 2023-06-22T08:03:40.189 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/25' 2023-06-22T08:03:40.189 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/30' 2023-06-22T08:03:40.189 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/33' 2023-06-22T08:03:40.189 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/23' 2023-06-22T08:03:40.190 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/35' 2023-06-22T08:03:40.190 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/26' 2023-06-22T08:03:40.190 INFO:tasks.workunit.client.0.smithi023.stdout:removed './287/37' 2023-06-22T08:03:40.190 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './287' 2023-06-22T08:03:40.190 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/21' 2023-06-22T08:03:40.190 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/27' 2023-06-22T08:03:40.190 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/31' 2023-06-22T08:03:40.190 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/36' 2023-06-22T08:03:40.191 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/20' 2023-06-22T08:03:40.191 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/39' 2023-06-22T08:03:40.191 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/34' 2023-06-22T08:03:40.191 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/38' 2023-06-22T08:03:40.191 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/28' 2023-06-22T08:03:40.191 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/24' 2023-06-22T08:03:40.191 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/29' 2023-06-22T08:03:40.191 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/32' 2023-06-22T08:03:40.192 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/22' 2023-06-22T08:03:40.192 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/25' 2023-06-22T08:03:40.192 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/30' 2023-06-22T08:03:40.192 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/33' 2023-06-22T08:03:40.192 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/23' 2023-06-22T08:03:40.192 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/35' 2023-06-22T08:03:40.192 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/26' 2023-06-22T08:03:40.193 INFO:tasks.workunit.client.0.smithi023.stdout:removed './288/37' 2023-06-22T08:03:40.193 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './288' 2023-06-22T08:03:40.193 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/21' 2023-06-22T08:03:40.193 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/27' 2023-06-22T08:03:40.193 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/31' 2023-06-22T08:03:40.193 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/36' 2023-06-22T08:03:40.193 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/20' 2023-06-22T08:03:40.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/39' 2023-06-22T08:03:40.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/34' 2023-06-22T08:03:40.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/38' 2023-06-22T08:03:40.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/28' 2023-06-22T08:03:40.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/24' 2023-06-22T08:03:40.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/29' 2023-06-22T08:03:40.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/32' 2023-06-22T08:03:40.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/22' 2023-06-22T08:03:40.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/25' 2023-06-22T08:03:40.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/30' 2023-06-22T08:03:40.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/33' 2023-06-22T08:03:40.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/23' 2023-06-22T08:03:40.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/35' 2023-06-22T08:03:40.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/26' 2023-06-22T08:03:40.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed './289/37' 2023-06-22T08:03:40.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './289' 2023-06-22T08:03:40.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/21' 2023-06-22T08:03:40.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/27' 2023-06-22T08:03:40.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/31' 2023-06-22T08:03:40.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/36' 2023-06-22T08:03:40.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/20' 2023-06-22T08:03:40.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/39' 2023-06-22T08:03:40.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/34' 2023-06-22T08:03:40.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/38' 2023-06-22T08:03:40.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/28' 2023-06-22T08:03:40.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/24' 2023-06-22T08:03:40.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/29' 2023-06-22T08:03:40.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/32' 2023-06-22T08:03:40.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/22' 2023-06-22T08:03:40.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/25' 2023-06-22T08:03:40.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/30' 2023-06-22T08:03:40.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/33' 2023-06-22T08:03:40.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/23' 2023-06-22T08:03:40.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/35' 2023-06-22T08:03:40.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/26' 2023-06-22T08:03:40.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed './290/37' 2023-06-22T08:03:40.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './290' 2023-06-22T08:03:40.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/21' 2023-06-22T08:03:40.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/27' 2023-06-22T08:03:40.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/31' 2023-06-22T08:03:40.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/36' 2023-06-22T08:03:40.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/20' 2023-06-22T08:03:40.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/39' 2023-06-22T08:03:40.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/34' 2023-06-22T08:03:40.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/38' 2023-06-22T08:03:40.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/28' 2023-06-22T08:03:40.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/24' 2023-06-22T08:03:40.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/29' 2023-06-22T08:03:40.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/32' 2023-06-22T08:03:40.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/22' 2023-06-22T08:03:40.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/25' 2023-06-22T08:03:40.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/30' 2023-06-22T08:03:40.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/33' 2023-06-22T08:03:40.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/23' 2023-06-22T08:03:40.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/35' 2023-06-22T08:03:40.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/26' 2023-06-22T08:03:40.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed './291/37' 2023-06-22T08:03:40.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './291' 2023-06-22T08:03:40.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/21' 2023-06-22T08:03:40.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/27' 2023-06-22T08:03:40.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/31' 2023-06-22T08:03:40.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/36' 2023-06-22T08:03:40.203 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/20' 2023-06-22T08:03:40.203 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/39' 2023-06-22T08:03:40.203 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/34' 2023-06-22T08:03:40.203 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/38' 2023-06-22T08:03:40.203 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/28' 2023-06-22T08:03:40.203 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/24' 2023-06-22T08:03:40.203 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/29' 2023-06-22T08:03:40.204 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/32' 2023-06-22T08:03:40.204 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/22' 2023-06-22T08:03:40.204 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/25' 2023-06-22T08:03:40.204 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/30' 2023-06-22T08:03:40.204 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/33' 2023-06-22T08:03:40.204 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/23' 2023-06-22T08:03:40.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/35' 2023-06-22T08:03:40.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/26' 2023-06-22T08:03:40.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed './292/37' 2023-06-22T08:03:40.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './292' 2023-06-22T08:03:40.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/21' 2023-06-22T08:03:40.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/27' 2023-06-22T08:03:40.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/31' 2023-06-22T08:03:40.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/36' 2023-06-22T08:03:40.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/20' 2023-06-22T08:03:40.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/39' 2023-06-22T08:03:40.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/34' 2023-06-22T08:03:40.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/38' 2023-06-22T08:03:40.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/28' 2023-06-22T08:03:40.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/24' 2023-06-22T08:03:40.207 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/29' 2023-06-22T08:03:40.207 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/32' 2023-06-22T08:03:40.207 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/22' 2023-06-22T08:03:40.207 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/25' 2023-06-22T08:03:40.207 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/30' 2023-06-22T08:03:40.207 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/33' 2023-06-22T08:03:40.208 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/23' 2023-06-22T08:03:40.208 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/35' 2023-06-22T08:03:40.208 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/26' 2023-06-22T08:03:40.208 INFO:tasks.workunit.client.0.smithi023.stdout:removed './293/37' 2023-06-22T08:03:40.208 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './293' 2023-06-22T08:03:40.208 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/21' 2023-06-22T08:03:40.208 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/27' 2023-06-22T08:03:40.209 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/31' 2023-06-22T08:03:40.209 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/36' 2023-06-22T08:03:40.209 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/20' 2023-06-22T08:03:40.209 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/39' 2023-06-22T08:03:40.209 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/34' 2023-06-22T08:03:40.209 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/38' 2023-06-22T08:03:40.209 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/28' 2023-06-22T08:03:40.209 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/24' 2023-06-22T08:03:40.210 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/29' 2023-06-22T08:03:40.210 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/32' 2023-06-22T08:03:40.210 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/22' 2023-06-22T08:03:40.210 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/25' 2023-06-22T08:03:40.210 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/30' 2023-06-22T08:03:40.210 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/33' 2023-06-22T08:03:40.210 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/23' 2023-06-22T08:03:40.211 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/35' 2023-06-22T08:03:40.211 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/26' 2023-06-22T08:03:40.211 INFO:tasks.workunit.client.0.smithi023.stdout:removed './294/37' 2023-06-22T08:03:40.211 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './294' 2023-06-22T08:03:40.211 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/21' 2023-06-22T08:03:40.211 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/27' 2023-06-22T08:03:40.211 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/31' 2023-06-22T08:03:40.211 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/36' 2023-06-22T08:03:40.212 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/20' 2023-06-22T08:03:40.212 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/39' 2023-06-22T08:03:40.212 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/34' 2023-06-22T08:03:40.212 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/38' 2023-06-22T08:03:40.212 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/28' 2023-06-22T08:03:40.212 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/24' 2023-06-22T08:03:40.212 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/29' 2023-06-22T08:03:40.213 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/32' 2023-06-22T08:03:40.213 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/22' 2023-06-22T08:03:40.213 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/25' 2023-06-22T08:03:40.213 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/30' 2023-06-22T08:03:40.213 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/33' 2023-06-22T08:03:40.213 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/23' 2023-06-22T08:03:40.213 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/35' 2023-06-22T08:03:40.214 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/26' 2023-06-22T08:03:40.214 INFO:tasks.workunit.client.0.smithi023.stdout:removed './295/37' 2023-06-22T08:03:40.214 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './295' 2023-06-22T08:03:40.214 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/21' 2023-06-22T08:03:40.214 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/27' 2023-06-22T08:03:40.214 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/31' 2023-06-22T08:03:40.214 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/36' 2023-06-22T08:03:40.215 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/20' 2023-06-22T08:03:40.215 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/39' 2023-06-22T08:03:40.215 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/34' 2023-06-22T08:03:40.215 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/38' 2023-06-22T08:03:40.215 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/28' 2023-06-22T08:03:40.215 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/24' 2023-06-22T08:03:40.216 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/29' 2023-06-22T08:03:40.216 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/32' 2023-06-22T08:03:40.216 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/22' 2023-06-22T08:03:40.216 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/25' 2023-06-22T08:03:40.216 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/30' 2023-06-22T08:03:40.216 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/33' 2023-06-22T08:03:40.216 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/23' 2023-06-22T08:03:40.217 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/35' 2023-06-22T08:03:40.217 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/26' 2023-06-22T08:03:40.217 INFO:tasks.workunit.client.0.smithi023.stdout:removed './296/37' 2023-06-22T08:03:40.217 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './296' 2023-06-22T08:03:40.217 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/21' 2023-06-22T08:03:40.217 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/27' 2023-06-22T08:03:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/31' 2023-06-22T08:03:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/36' 2023-06-22T08:03:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/20' 2023-06-22T08:03:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/39' 2023-06-22T08:03:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/34' 2023-06-22T08:03:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/38' 2023-06-22T08:03:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/28' 2023-06-22T08:03:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/24' 2023-06-22T08:03:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/29' 2023-06-22T08:03:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/32' 2023-06-22T08:03:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/22' 2023-06-22T08:03:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/25' 2023-06-22T08:03:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/30' 2023-06-22T08:03:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/33' 2023-06-22T08:03:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/23' 2023-06-22T08:03:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/35' 2023-06-22T08:03:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/26' 2023-06-22T08:03:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed './297/37' 2023-06-22T08:03:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './297' 2023-06-22T08:03:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/21' 2023-06-22T08:03:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/27' 2023-06-22T08:03:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/31' 2023-06-22T08:03:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/36' 2023-06-22T08:03:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/20' 2023-06-22T08:03:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/39' 2023-06-22T08:03:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/34' 2023-06-22T08:03:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/38' 2023-06-22T08:03:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/28' 2023-06-22T08:03:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/24' 2023-06-22T08:03:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/29' 2023-06-22T08:03:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/32' 2023-06-22T08:03:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/22' 2023-06-22T08:03:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/25' 2023-06-22T08:03:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/30' 2023-06-22T08:03:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/33' 2023-06-22T08:03:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/23' 2023-06-22T08:03:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/35' 2023-06-22T08:03:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/26' 2023-06-22T08:03:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed './298/37' 2023-06-22T08:03:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './298' 2023-06-22T08:03:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/21' 2023-06-22T08:03:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/27' 2023-06-22T08:03:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/31' 2023-06-22T08:03:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/36' 2023-06-22T08:03:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/20' 2023-06-22T08:03:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/39' 2023-06-22T08:03:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/34' 2023-06-22T08:03:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/38' 2023-06-22T08:03:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/28' 2023-06-22T08:03:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/24' 2023-06-22T08:03:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/29' 2023-06-22T08:03:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/32' 2023-06-22T08:03:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/22' 2023-06-22T08:03:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/25' 2023-06-22T08:03:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/30' 2023-06-22T08:03:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/33' 2023-06-22T08:03:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/23' 2023-06-22T08:03:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/35' 2023-06-22T08:03:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/26' 2023-06-22T08:03:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './299/37' 2023-06-22T08:03:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './299' 2023-06-22T08:03:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/19' 2023-06-22T08:03:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/15' 2023-06-22T08:03:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/20' 2023-06-22T08:03:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/12' 2023-06-22T08:03:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/10' 2023-06-22T08:03:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/11' 2023-06-22T08:03:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/16' 2023-06-22T08:03:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/14' 2023-06-22T08:03:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/18' 2023-06-22T08:03:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/13' 2023-06-22T08:03:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/166/17' 2023-06-22T08:03:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/166' 2023-06-22T08:03:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/19' 2023-06-22T08:03:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/15' 2023-06-22T08:03:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/20' 2023-06-22T08:03:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/12' 2023-06-22T08:03:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/10' 2023-06-22T08:03:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/11' 2023-06-22T08:03:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/16' 2023-06-22T08:03:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/14' 2023-06-22T08:03:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/18' 2023-06-22T08:03:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/13' 2023-06-22T08:03:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/147/17' 2023-06-22T08:03:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/147' 2023-06-22T08:03:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/19' 2023-06-22T08:03:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/15' 2023-06-22T08:03:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/20' 2023-06-22T08:03:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/12' 2023-06-22T08:03:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/10' 2023-06-22T08:03:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/11' 2023-06-22T08:03:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/16' 2023-06-22T08:03:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/14' 2023-06-22T08:03:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/18' 2023-06-22T08:03:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/13' 2023-06-22T08:03:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/151/17' 2023-06-22T08:03:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/151' 2023-06-22T08:03:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/19' 2023-06-22T08:03:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/15' 2023-06-22T08:03:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/20' 2023-06-22T08:03:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/12' 2023-06-22T08:03:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/10' 2023-06-22T08:03:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/11' 2023-06-22T08:03:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/16' 2023-06-22T08:03:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/14' 2023-06-22T08:03:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/18' 2023-06-22T08:03:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/13' 2023-06-22T08:03:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/170/17' 2023-06-22T08:03:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/170' 2023-06-22T08:03:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/19' 2023-06-22T08:03:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/15' 2023-06-22T08:03:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/20' 2023-06-22T08:03:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/12' 2023-06-22T08:03:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/10' 2023-06-22T08:03:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/11' 2023-06-22T08:03:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/16' 2023-06-22T08:03:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/14' 2023-06-22T08:03:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/18' 2023-06-22T08:03:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/13' 2023-06-22T08:03:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/167/17' 2023-06-22T08:03:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/167' 2023-06-22T08:03:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/19' 2023-06-22T08:03:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/15' 2023-06-22T08:03:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/20' 2023-06-22T08:03:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/12' 2023-06-22T08:03:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/10' 2023-06-22T08:03:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/11' 2023-06-22T08:03:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/16' 2023-06-22T08:03:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/14' 2023-06-22T08:03:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/18' 2023-06-22T08:03:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/13' 2023-06-22T08:03:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/140/17' 2023-06-22T08:03:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/140' 2023-06-22T08:03:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/19' 2023-06-22T08:03:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/15' 2023-06-22T08:03:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/20' 2023-06-22T08:03:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/12' 2023-06-22T08:03:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/10' 2023-06-22T08:03:41.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/11' 2023-06-22T08:03:41.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/16' 2023-06-22T08:03:41.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/14' 2023-06-22T08:03:41.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/18' 2023-06-22T08:03:41.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/13' 2023-06-22T08:03:41.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/142/17' 2023-06-22T08:03:41.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/142' 2023-06-22T08:03:41.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/19' 2023-06-22T08:03:41.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/15' 2023-06-22T08:03:41.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/20' 2023-06-22T08:03:41.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/12' 2023-06-22T08:03:41.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/10' 2023-06-22T08:03:41.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/11' 2023-06-22T08:03:41.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/16' 2023-06-22T08:03:41.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/14' 2023-06-22T08:03:41.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/18' 2023-06-22T08:03:41.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/13' 2023-06-22T08:03:41.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/150/17' 2023-06-22T08:03:41.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/150' 2023-06-22T08:03:41.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/19' 2023-06-22T08:03:41.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/15' 2023-06-22T08:03:41.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/20' 2023-06-22T08:03:41.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/12' 2023-06-22T08:03:41.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/10' 2023-06-22T08:03:41.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/11' 2023-06-22T08:03:41.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/16' 2023-06-22T08:03:41.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/14' 2023-06-22T08:03:41.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/18' 2023-06-22T08:03:41.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/13' 2023-06-22T08:03:41.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/105/17' 2023-06-22T08:03:41.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/105' 2023-06-22T08:03:41.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/19' 2023-06-22T08:03:41.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/15' 2023-06-22T08:03:41.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/20' 2023-06-22T08:03:41.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/12' 2023-06-22T08:03:41.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/10' 2023-06-22T08:03:41.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/11' 2023-06-22T08:03:41.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/16' 2023-06-22T08:03:41.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/14' 2023-06-22T08:03:41.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/18' 2023-06-22T08:03:41.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/13' 2023-06-22T08:03:41.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/149/17' 2023-06-22T08:03:41.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/149' 2023-06-22T08:03:41.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/19' 2023-06-22T08:03:41.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/15' 2023-06-22T08:03:41.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/20' 2023-06-22T08:03:41.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/12' 2023-06-22T08:03:41.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/10' 2023-06-22T08:03:41.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/11' 2023-06-22T08:03:41.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/16' 2023-06-22T08:03:41.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/14' 2023-06-22T08:03:41.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/18' 2023-06-22T08:03:41.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/13' 2023-06-22T08:03:41.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/109/17' 2023-06-22T08:03:41.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/109' 2023-06-22T08:03:41.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/19' 2023-06-22T08:03:41.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/15' 2023-06-22T08:03:41.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/20' 2023-06-22T08:03:41.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/12' 2023-06-22T08:03:41.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/10' 2023-06-22T08:03:41.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/11' 2023-06-22T08:03:41.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/16' 2023-06-22T08:03:41.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/14' 2023-06-22T08:03:41.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/18' 2023-06-22T08:03:41.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/13' 2023-06-22T08:03:41.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/158/17' 2023-06-22T08:03:41.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/158' 2023-06-22T08:03:41.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/19' 2023-06-22T08:03:41.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/15' 2023-06-22T08:03:41.072 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/20' 2023-06-22T08:03:41.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/12' 2023-06-22T08:03:41.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/10' 2023-06-22T08:03:41.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/11' 2023-06-22T08:03:41.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/16' 2023-06-22T08:03:41.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/14' 2023-06-22T08:03:41.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/18' 2023-06-22T08:03:41.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/13' 2023-06-22T08:03:41.073 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/106/17' 2023-06-22T08:03:41.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/106' 2023-06-22T08:03:41.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/19' 2023-06-22T08:03:41.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/15' 2023-06-22T08:03:41.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/20' 2023-06-22T08:03:41.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/12' 2023-06-22T08:03:41.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/10' 2023-06-22T08:03:41.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/11' 2023-06-22T08:03:41.074 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/16' 2023-06-22T08:03:41.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/14' 2023-06-22T08:03:41.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/18' 2023-06-22T08:03:41.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/13' 2023-06-22T08:03:41.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/189/17' 2023-06-22T08:03:41.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/189' 2023-06-22T08:03:41.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/19' 2023-06-22T08:03:41.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/15' 2023-06-22T08:03:41.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/20' 2023-06-22T08:03:41.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/12' 2023-06-22T08:03:41.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/10' 2023-06-22T08:03:41.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/11' 2023-06-22T08:03:41.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/16' 2023-06-22T08:03:41.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/14' 2023-06-22T08:03:41.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/18' 2023-06-22T08:03:41.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/13' 2023-06-22T08:03:41.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/133/17' 2023-06-22T08:03:41.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/133' 2023-06-22T08:03:41.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/19' 2023-06-22T08:03:41.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/15' 2023-06-22T08:03:41.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/20' 2023-06-22T08:03:41.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/12' 2023-06-22T08:03:41.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/10' 2023-06-22T08:03:41.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/11' 2023-06-22T08:03:41.352 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/16' 2023-06-22T08:03:41.352 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/14' 2023-06-22T08:03:41.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/18' 2023-06-22T08:03:41.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/13' 2023-06-22T08:03:41.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/173/17' 2023-06-22T08:03:41.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/173' 2023-06-22T08:03:41.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/19' 2023-06-22T08:03:41.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/15' 2023-06-22T08:03:41.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/20' 2023-06-22T08:03:41.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/12' 2023-06-22T08:03:41.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/10' 2023-06-22T08:03:41.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/11' 2023-06-22T08:03:41.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/16' 2023-06-22T08:03:41.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/14' 2023-06-22T08:03:41.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/18' 2023-06-22T08:03:41.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/13' 2023-06-22T08:03:41.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/174/17' 2023-06-22T08:03:41.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/174' 2023-06-22T08:03:41.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/19' 2023-06-22T08:03:41.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/15' 2023-06-22T08:03:41.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/20' 2023-06-22T08:03:41.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/12' 2023-06-22T08:03:41.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/10' 2023-06-22T08:03:41.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/11' 2023-06-22T08:03:41.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/16' 2023-06-22T08:03:41.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/14' 2023-06-22T08:03:41.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/18' 2023-06-22T08:03:41.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/13' 2023-06-22T08:03:41.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/155/17' 2023-06-22T08:03:41.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/155' 2023-06-22T08:03:41.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/19' 2023-06-22T08:03:41.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/15' 2023-06-22T08:03:41.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/20' 2023-06-22T08:03:41.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/12' 2023-06-22T08:03:41.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/10' 2023-06-22T08:03:41.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/11' 2023-06-22T08:03:41.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/16' 2023-06-22T08:03:41.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/14' 2023-06-22T08:03:41.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/18' 2023-06-22T08:03:41.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/13' 2023-06-22T08:03:41.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/114/17' 2023-06-22T08:03:41.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/114' 2023-06-22T08:03:41.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/19' 2023-06-22T08:03:41.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/15' 2023-06-22T08:03:41.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/20' 2023-06-22T08:03:41.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/12' 2023-06-22T08:03:41.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/10' 2023-06-22T08:03:41.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/11' 2023-06-22T08:03:41.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/16' 2023-06-22T08:03:41.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/14' 2023-06-22T08:03:41.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/18' 2023-06-22T08:03:41.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/13' 2023-06-22T08:03:41.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/159/17' 2023-06-22T08:03:41.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/159' 2023-06-22T08:03:41.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/19' 2023-06-22T08:03:41.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/15' 2023-06-22T08:03:41.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/20' 2023-06-22T08:03:41.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/12' 2023-06-22T08:03:41.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/10' 2023-06-22T08:03:41.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/11' 2023-06-22T08:03:41.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/16' 2023-06-22T08:03:41.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/14' 2023-06-22T08:03:41.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/18' 2023-06-22T08:03:41.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/13' 2023-06-22T08:03:41.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/187/17' 2023-06-22T08:03:41.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/187' 2023-06-22T08:03:41.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/19' 2023-06-22T08:03:41.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/15' 2023-06-22T08:03:41.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/20' 2023-06-22T08:03:41.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/12' 2023-06-22T08:03:41.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/10' 2023-06-22T08:03:41.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/11' 2023-06-22T08:03:41.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/16' 2023-06-22T08:03:41.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/14' 2023-06-22T08:03:41.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/18' 2023-06-22T08:03:41.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/13' 2023-06-22T08:03:41.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/185/17' 2023-06-22T08:03:41.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/185' 2023-06-22T08:03:41.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/19' 2023-06-22T08:03:41.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/15' 2023-06-22T08:03:41.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/20' 2023-06-22T08:03:41.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/12' 2023-06-22T08:03:41.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/10' 2023-06-22T08:03:41.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/11' 2023-06-22T08:03:41.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/16' 2023-06-22T08:03:41.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/14' 2023-06-22T08:03:41.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/18' 2023-06-22T08:03:41.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/13' 2023-06-22T08:03:41.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/120/17' 2023-06-22T08:03:41.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/120' 2023-06-22T08:03:41.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/19' 2023-06-22T08:03:41.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/15' 2023-06-22T08:03:41.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/20' 2023-06-22T08:03:41.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/12' 2023-06-22T08:03:41.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/10' 2023-06-22T08:03:41.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/11' 2023-06-22T08:03:41.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/16' 2023-06-22T08:03:41.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/14' 2023-06-22T08:03:41.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/18' 2023-06-22T08:03:41.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/13' 2023-06-22T08:03:41.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/110/17' 2023-06-22T08:03:41.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/110' 2023-06-22T08:03:41.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/19' 2023-06-22T08:03:41.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/15' 2023-06-22T08:03:41.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/20' 2023-06-22T08:03:41.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/12' 2023-06-22T08:03:41.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/10' 2023-06-22T08:03:41.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/11' 2023-06-22T08:03:41.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/16' 2023-06-22T08:03:41.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/14' 2023-06-22T08:03:41.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/18' 2023-06-22T08:03:41.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/13' 2023-06-22T08:03:41.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/137/17' 2023-06-22T08:03:41.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/137' 2023-06-22T08:03:41.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/19' 2023-06-22T08:03:41.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/15' 2023-06-22T08:03:41.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/20' 2023-06-22T08:03:41.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/12' 2023-06-22T08:03:41.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/10' 2023-06-22T08:03:41.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/11' 2023-06-22T08:03:41.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/16' 2023-06-22T08:03:41.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/14' 2023-06-22T08:03:41.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/18' 2023-06-22T08:03:41.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/13' 2023-06-22T08:03:41.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/198/17' 2023-06-22T08:03:41.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/198' 2023-06-22T08:03:41.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/19' 2023-06-22T08:03:41.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/15' 2023-06-22T08:03:41.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/20' 2023-06-22T08:03:41.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/12' 2023-06-22T08:03:41.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/10' 2023-06-22T08:03:41.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/11' 2023-06-22T08:03:41.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/16' 2023-06-22T08:03:41.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/14' 2023-06-22T08:03:41.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/18' 2023-06-22T08:03:41.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/13' 2023-06-22T08:03:41.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/136/17' 2023-06-22T08:03:41.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/136' 2023-06-22T08:03:41.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/19' 2023-06-22T08:03:41.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/15' 2023-06-22T08:03:41.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/20' 2023-06-22T08:03:41.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/12' 2023-06-22T08:03:41.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/10' 2023-06-22T08:03:41.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/11' 2023-06-22T08:03:41.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/16' 2023-06-22T08:03:41.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/14' 2023-06-22T08:03:41.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/18' 2023-06-22T08:03:41.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/13' 2023-06-22T08:03:41.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/125/17' 2023-06-22T08:03:41.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/125' 2023-06-22T08:03:41.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/19' 2023-06-22T08:03:41.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/15' 2023-06-22T08:03:41.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/20' 2023-06-22T08:03:41.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/12' 2023-06-22T08:03:41.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/10' 2023-06-22T08:03:41.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/11' 2023-06-22T08:03:41.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/16' 2023-06-22T08:03:41.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/14' 2023-06-22T08:03:41.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/18' 2023-06-22T08:03:41.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/13' 2023-06-22T08:03:41.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/194/17' 2023-06-22T08:03:41.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/194' 2023-06-22T08:03:41.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/19' 2023-06-22T08:03:41.690 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/15' 2023-06-22T08:03:41.690 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/20' 2023-06-22T08:03:41.690 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/12' 2023-06-22T08:03:41.690 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/10' 2023-06-22T08:03:41.690 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/11' 2023-06-22T08:03:41.690 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/16' 2023-06-22T08:03:41.690 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/14' 2023-06-22T08:03:41.690 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/18' 2023-06-22T08:03:41.691 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/13' 2023-06-22T08:03:41.691 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/119/17' 2023-06-22T08:03:41.691 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/119' 2023-06-22T08:03:41.691 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/19' 2023-06-22T08:03:41.691 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/15' 2023-06-22T08:03:41.691 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/20' 2023-06-22T08:03:41.691 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/12' 2023-06-22T08:03:41.691 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/10' 2023-06-22T08:03:41.692 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/11' 2023-06-22T08:03:41.692 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/16' 2023-06-22T08:03:41.692 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/14' 2023-06-22T08:03:41.692 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/18' 2023-06-22T08:03:41.692 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/13' 2023-06-22T08:03:41.692 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/157/17' 2023-06-22T08:03:41.692 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/157' 2023-06-22T08:03:41.693 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/19' 2023-06-22T08:03:41.693 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/15' 2023-06-22T08:03:41.693 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/20' 2023-06-22T08:03:41.693 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/12' 2023-06-22T08:03:41.693 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/10' 2023-06-22T08:03:41.693 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/11' 2023-06-22T08:03:41.693 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/16' 2023-06-22T08:03:41.693 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/14' 2023-06-22T08:03:41.694 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/18' 2023-06-22T08:03:41.694 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/13' 2023-06-22T08:03:41.694 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/172/17' 2023-06-22T08:03:41.694 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/172' 2023-06-22T08:03:41.694 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/19' 2023-06-22T08:03:41.694 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/15' 2023-06-22T08:03:41.694 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/20' 2023-06-22T08:03:41.694 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/12' 2023-06-22T08:03:41.695 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/10' 2023-06-22T08:03:41.695 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/11' 2023-06-22T08:03:41.695 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/16' 2023-06-22T08:03:41.695 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/14' 2023-06-22T08:03:41.695 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/18' 2023-06-22T08:03:41.695 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/13' 2023-06-22T08:03:41.695 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/186/17' 2023-06-22T08:03:41.695 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/186' 2023-06-22T08:03:41.696 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/19' 2023-06-22T08:03:41.696 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/15' 2023-06-22T08:03:41.696 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/20' 2023-06-22T08:03:41.696 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/12' 2023-06-22T08:03:41.696 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/10' 2023-06-22T08:03:41.697 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/11' 2023-06-22T08:03:41.700 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/16' 2023-06-22T08:03:41.700 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/14' 2023-06-22T08:03:42.283 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/18' 2023-06-22T08:03:42.283 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/13' 2023-06-22T08:03:42.283 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/190/17' 2023-06-22T08:03:42.283 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/190' 2023-06-22T08:03:42.283 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/19' 2023-06-22T08:03:42.284 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/15' 2023-06-22T08:03:42.284 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/20' 2023-06-22T08:03:42.284 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/12' 2023-06-22T08:03:42.284 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/10' 2023-06-22T08:03:42.284 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/11' 2023-06-22T08:03:42.284 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/16' 2023-06-22T08:03:42.284 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/14' 2023-06-22T08:03:42.285 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/18' 2023-06-22T08:03:42.285 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/13' 2023-06-22T08:03:42.285 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/104/17' 2023-06-22T08:03:42.285 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/104' 2023-06-22T08:03:42.285 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/19' 2023-06-22T08:03:42.285 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/15' 2023-06-22T08:03:42.285 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/20' 2023-06-22T08:03:42.285 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/12' 2023-06-22T08:03:42.286 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/10' 2023-06-22T08:03:42.286 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/11' 2023-06-22T08:03:42.286 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/16' 2023-06-22T08:03:42.286 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/14' 2023-06-22T08:03:42.286 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/18' 2023-06-22T08:03:42.286 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/13' 2023-06-22T08:03:42.286 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/164/17' 2023-06-22T08:03:42.286 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/164' 2023-06-22T08:03:42.287 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/19' 2023-06-22T08:03:42.287 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/15' 2023-06-22T08:03:42.287 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/20' 2023-06-22T08:03:42.287 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/12' 2023-06-22T08:03:42.287 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/10' 2023-06-22T08:03:42.287 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/11' 2023-06-22T08:03:42.287 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/16' 2023-06-22T08:03:42.287 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/14' 2023-06-22T08:03:42.288 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/18' 2023-06-22T08:03:42.288 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/13' 2023-06-22T08:03:42.288 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/112/17' 2023-06-22T08:03:42.288 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/112' 2023-06-22T08:03:42.288 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/19' 2023-06-22T08:03:42.288 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/15' 2023-06-22T08:03:42.288 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/20' 2023-06-22T08:03:42.289 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/12' 2023-06-22T08:03:42.289 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/10' 2023-06-22T08:03:42.289 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/11' 2023-06-22T08:03:42.289 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/16' 2023-06-22T08:03:42.289 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/14' 2023-06-22T08:03:42.289 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/18' 2023-06-22T08:03:42.289 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/13' 2023-06-22T08:03:42.290 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/100/17' 2023-06-22T08:03:42.290 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/100' 2023-06-22T08:03:42.290 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/19' 2023-06-22T08:03:42.290 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/15' 2023-06-22T08:03:42.290 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/20' 2023-06-22T08:03:42.290 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/12' 2023-06-22T08:03:42.290 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/10' 2023-06-22T08:03:42.290 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/11' 2023-06-22T08:03:42.291 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/16' 2023-06-22T08:03:42.291 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/14' 2023-06-22T08:03:42.291 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/18' 2023-06-22T08:03:42.291 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/13' 2023-06-22T08:03:42.291 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/134/17' 2023-06-22T08:03:42.291 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/134' 2023-06-22T08:03:42.291 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/19' 2023-06-22T08:03:42.291 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/15' 2023-06-22T08:03:42.292 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/20' 2023-06-22T08:03:42.292 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/12' 2023-06-22T08:03:42.292 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/10' 2023-06-22T08:03:42.292 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/11' 2023-06-22T08:03:42.293 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/16' 2023-06-22T08:03:42.293 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/14' 2023-06-22T08:03:42.293 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/18' 2023-06-22T08:03:42.293 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/13' 2023-06-22T08:03:42.293 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/115/17' 2023-06-22T08:03:42.293 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/115' 2023-06-22T08:03:42.293 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/19' 2023-06-22T08:03:42.294 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/15' 2023-06-22T08:03:42.294 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/20' 2023-06-22T08:03:42.294 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/12' 2023-06-22T08:03:42.294 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/10' 2023-06-22T08:03:42.294 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/11' 2023-06-22T08:03:42.294 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/16' 2023-06-22T08:03:42.294 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/14' 2023-06-22T08:03:42.294 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/18' 2023-06-22T08:03:42.295 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/13' 2023-06-22T08:03:42.295 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/116/17' 2023-06-22T08:03:42.295 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/116' 2023-06-22T08:03:42.295 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/19' 2023-06-22T08:03:42.295 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/15' 2023-06-22T08:03:42.295 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/20' 2023-06-22T08:03:42.295 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/12' 2023-06-22T08:03:42.295 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/10' 2023-06-22T08:03:42.296 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/11' 2023-06-22T08:03:42.296 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/16' 2023-06-22T08:03:42.296 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/14' 2023-06-22T08:03:42.296 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/18' 2023-06-22T08:03:42.296 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/13' 2023-06-22T08:03:42.296 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/195/17' 2023-06-22T08:03:42.296 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/195' 2023-06-22T08:03:42.297 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/19' 2023-06-22T08:03:42.297 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/15' 2023-06-22T08:03:42.297 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/20' 2023-06-22T08:03:42.297 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/12' 2023-06-22T08:03:42.297 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/10' 2023-06-22T08:03:42.297 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/11' 2023-06-22T08:03:42.297 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/16' 2023-06-22T08:03:42.297 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/14' 2023-06-22T08:03:42.298 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/18' 2023-06-22T08:03:42.563 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/13' 2023-06-22T08:03:42.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/138/17' 2023-06-22T08:03:42.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/138' 2023-06-22T08:03:42.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/19' 2023-06-22T08:03:42.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/15' 2023-06-22T08:03:42.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/20' 2023-06-22T08:03:42.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/12' 2023-06-22T08:03:42.564 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/10' 2023-06-22T08:03:42.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/11' 2023-06-22T08:03:42.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/16' 2023-06-22T08:03:42.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/14' 2023-06-22T08:03:42.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/18' 2023-06-22T08:03:42.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/13' 2023-06-22T08:03:42.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/135/17' 2023-06-22T08:03:42.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/135' 2023-06-22T08:03:42.565 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/19' 2023-06-22T08:03:42.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/15' 2023-06-22T08:03:42.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/20' 2023-06-22T08:03:42.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/12' 2023-06-22T08:03:42.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/10' 2023-06-22T08:03:42.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/11' 2023-06-22T08:03:42.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/16' 2023-06-22T08:03:42.566 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/14' 2023-06-22T08:03:42.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/18' 2023-06-22T08:03:42.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/13' 2023-06-22T08:03:42.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/117/17' 2023-06-22T08:03:42.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/117' 2023-06-22T08:03:42.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/19' 2023-06-22T08:03:42.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/15' 2023-06-22T08:03:42.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/20' 2023-06-22T08:03:42.567 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/12' 2023-06-22T08:03:42.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/10' 2023-06-22T08:03:42.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/11' 2023-06-22T08:03:42.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/16' 2023-06-22T08:03:42.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/14' 2023-06-22T08:03:42.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/18' 2023-06-22T08:03:42.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/13' 2023-06-22T08:03:42.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/143/17' 2023-06-22T08:03:42.568 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/143' 2023-06-22T08:03:42.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/19' 2023-06-22T08:03:42.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/15' 2023-06-22T08:03:42.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/20' 2023-06-22T08:03:42.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/12' 2023-06-22T08:03:42.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/10' 2023-06-22T08:03:42.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/11' 2023-06-22T08:03:42.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/16' 2023-06-22T08:03:42.569 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/14' 2023-06-22T08:03:42.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/18' 2023-06-22T08:03:42.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/13' 2023-06-22T08:03:42.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/180/17' 2023-06-22T08:03:42.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/180' 2023-06-22T08:03:42.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/19' 2023-06-22T08:03:42.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/15' 2023-06-22T08:03:42.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/20' 2023-06-22T08:03:42.570 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/12' 2023-06-22T08:03:42.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/10' 2023-06-22T08:03:42.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/11' 2023-06-22T08:03:42.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/16' 2023-06-22T08:03:42.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/14' 2023-06-22T08:03:42.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/18' 2023-06-22T08:03:42.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/13' 2023-06-22T08:03:42.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/126/17' 2023-06-22T08:03:42.571 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/126' 2023-06-22T08:03:42.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/19' 2023-06-22T08:03:42.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/15' 2023-06-22T08:03:42.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/20' 2023-06-22T08:03:42.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/12' 2023-06-22T08:03:42.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/10' 2023-06-22T08:03:42.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/11' 2023-06-22T08:03:42.572 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/16' 2023-06-22T08:03:42.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/14' 2023-06-22T08:03:42.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/18' 2023-06-22T08:03:42.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/13' 2023-06-22T08:03:42.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/161/17' 2023-06-22T08:03:42.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/161' 2023-06-22T08:03:42.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/19' 2023-06-22T08:03:42.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/15' 2023-06-22T08:03:42.573 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/20' 2023-06-22T08:03:42.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/12' 2023-06-22T08:03:42.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/10' 2023-06-22T08:03:42.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/11' 2023-06-22T08:03:42.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/16' 2023-06-22T08:03:42.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/14' 2023-06-22T08:03:42.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/18' 2023-06-22T08:03:42.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/13' 2023-06-22T08:03:42.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/176/17' 2023-06-22T08:03:42.574 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/176' 2023-06-22T08:03:42.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/19' 2023-06-22T08:03:42.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/15' 2023-06-22T08:03:42.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/20' 2023-06-22T08:03:42.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/12' 2023-06-22T08:03:42.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/10' 2023-06-22T08:03:42.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/11' 2023-06-22T08:03:42.575 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/16' 2023-06-22T08:03:42.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/14' 2023-06-22T08:03:42.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/18' 2023-06-22T08:03:42.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/13' 2023-06-22T08:03:42.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/111/17' 2023-06-22T08:03:42.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/111' 2023-06-22T08:03:42.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/19' 2023-06-22T08:03:42.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/15' 2023-06-22T08:03:42.576 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/20' 2023-06-22T08:03:42.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/12' 2023-06-22T08:03:42.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/10' 2023-06-22T08:03:42.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/11' 2023-06-22T08:03:42.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/16' 2023-06-22T08:03:42.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/14' 2023-06-22T08:03:42.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/18' 2023-06-22T08:03:42.577 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/13' 2023-06-22T08:03:42.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/191/17' 2023-06-22T08:03:42.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/191' 2023-06-22T08:03:42.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/19' 2023-06-22T08:03:42.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/15' 2023-06-22T08:03:42.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/20' 2023-06-22T08:03:42.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/12' 2023-06-22T08:03:42.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/10' 2023-06-22T08:03:42.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/11' 2023-06-22T08:03:42.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/16' 2023-06-22T08:03:42.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/14' 2023-06-22T08:03:42.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/18' 2023-06-22T08:03:42.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/13' 2023-06-22T08:03:42.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/184/17' 2023-06-22T08:03:42.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/184' 2023-06-22T08:03:42.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/19' 2023-06-22T08:03:42.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/15' 2023-06-22T08:03:42.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/20' 2023-06-22T08:03:42.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/12' 2023-06-22T08:03:42.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/10' 2023-06-22T08:03:42.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/11' 2023-06-22T08:03:42.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/16' 2023-06-22T08:03:42.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/14' 2023-06-22T08:03:42.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/18' 2023-06-22T08:03:42.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/13' 2023-06-22T08:03:42.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/192/17' 2023-06-22T08:03:42.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/192' 2023-06-22T08:03:42.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/19' 2023-06-22T08:03:42.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/15' 2023-06-22T08:03:42.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/20' 2023-06-22T08:03:42.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/12' 2023-06-22T08:03:42.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/10' 2023-06-22T08:03:42.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/11' 2023-06-22T08:03:42.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/16' 2023-06-22T08:03:42.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/14' 2023-06-22T08:03:42.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/18' 2023-06-22T08:03:42.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/13' 2023-06-22T08:03:42.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/103/17' 2023-06-22T08:03:42.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/103' 2023-06-22T08:03:42.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/19' 2023-06-22T08:03:42.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/15' 2023-06-22T08:03:42.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/20' 2023-06-22T08:03:42.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/12' 2023-06-22T08:03:42.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/10' 2023-06-22T08:03:42.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/11' 2023-06-22T08:03:42.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/16' 2023-06-22T08:03:42.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/14' 2023-06-22T08:03:42.938 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/18' 2023-06-22T08:03:42.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/13' 2023-06-22T08:03:42.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/128/17' 2023-06-22T08:03:42.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/128' 2023-06-22T08:03:42.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/19' 2023-06-22T08:03:42.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/15' 2023-06-22T08:03:42.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/20' 2023-06-22T08:03:42.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/12' 2023-06-22T08:03:42.939 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/10' 2023-06-22T08:03:42.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/11' 2023-06-22T08:03:42.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/16' 2023-06-22T08:03:42.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/14' 2023-06-22T08:03:42.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/18' 2023-06-22T08:03:42.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/13' 2023-06-22T08:03:42.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/144/17' 2023-06-22T08:03:42.940 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/144' 2023-06-22T08:03:42.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/19' 2023-06-22T08:03:42.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/15' 2023-06-22T08:03:42.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/20' 2023-06-22T08:03:42.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/12' 2023-06-22T08:03:42.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/10' 2023-06-22T08:03:42.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/11' 2023-06-22T08:03:42.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/16' 2023-06-22T08:03:42.941 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/14' 2023-06-22T08:03:42.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/18' 2023-06-22T08:03:42.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/13' 2023-06-22T08:03:42.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/146/17' 2023-06-22T08:03:42.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/146' 2023-06-22T08:03:42.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/19' 2023-06-22T08:03:42.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/15' 2023-06-22T08:03:42.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/20' 2023-06-22T08:03:42.942 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/12' 2023-06-22T08:03:42.943 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/10' 2023-06-22T08:03:42.943 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/11' 2023-06-22T08:03:42.943 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/16' 2023-06-22T08:03:42.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/14' 2023-06-22T08:03:42.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/18' 2023-06-22T08:03:42.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/13' 2023-06-22T08:03:42.944 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/141/17' 2023-06-22T08:03:42.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/141' 2023-06-22T08:03:42.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/19' 2023-06-22T08:03:42.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/15' 2023-06-22T08:03:42.945 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/20' 2023-06-22T08:03:42.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/12' 2023-06-22T08:03:42.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/10' 2023-06-22T08:03:42.946 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/11' 2023-06-22T08:03:42.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/16' 2023-06-22T08:03:42.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/14' 2023-06-22T08:03:42.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/18' 2023-06-22T08:03:42.947 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/13' 2023-06-22T08:03:42.948 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/182/17' 2023-06-22T08:03:42.948 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/182' 2023-06-22T08:03:42.948 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/19' 2023-06-22T08:03:42.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/15' 2023-06-22T08:03:42.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/20' 2023-06-22T08:03:42.949 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/12' 2023-06-22T08:03:42.950 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/10' 2023-06-22T08:03:42.950 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/11' 2023-06-22T08:03:42.950 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/16' 2023-06-22T08:03:42.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/14' 2023-06-22T08:03:42.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/18' 2023-06-22T08:03:42.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/13' 2023-06-22T08:03:42.951 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/107/17' 2023-06-22T08:03:43.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/107' 2023-06-22T08:03:43.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/19' 2023-06-22T08:03:43.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/15' 2023-06-22T08:03:43.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/20' 2023-06-22T08:03:43.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/12' 2023-06-22T08:03:43.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/10' 2023-06-22T08:03:43.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/11' 2023-06-22T08:03:43.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/16' 2023-06-22T08:03:43.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/14' 2023-06-22T08:03:43.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/18' 2023-06-22T08:03:43.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/13' 2023-06-22T08:03:43.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/178/17' 2023-06-22T08:03:43.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/178' 2023-06-22T08:03:43.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/19' 2023-06-22T08:03:43.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/15' 2023-06-22T08:03:43.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/20' 2023-06-22T08:03:43.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/12' 2023-06-22T08:03:43.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/10' 2023-06-22T08:03:43.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/11' 2023-06-22T08:03:43.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/16' 2023-06-22T08:03:43.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/14' 2023-06-22T08:03:43.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/18' 2023-06-22T08:03:43.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/13' 2023-06-22T08:03:43.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/156/17' 2023-06-22T08:03:43.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/156' 2023-06-22T08:03:43.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/19' 2023-06-22T08:03:43.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/15' 2023-06-22T08:03:43.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/20' 2023-06-22T08:03:43.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/12' 2023-06-22T08:03:43.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/10' 2023-06-22T08:03:43.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/11' 2023-06-22T08:03:43.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/16' 2023-06-22T08:03:43.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/14' 2023-06-22T08:03:43.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/18' 2023-06-22T08:03:43.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/13' 2023-06-22T08:03:43.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/131/17' 2023-06-22T08:03:43.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/131' 2023-06-22T08:03:43.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/19' 2023-06-22T08:03:43.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/15' 2023-06-22T08:03:43.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/20' 2023-06-22T08:03:43.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/12' 2023-06-22T08:03:43.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/10' 2023-06-22T08:03:43.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/11' 2023-06-22T08:03:43.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/16' 2023-06-22T08:03:43.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/14' 2023-06-22T08:03:43.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/18' 2023-06-22T08:03:43.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/13' 2023-06-22T08:03:43.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/123/17' 2023-06-22T08:03:43.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/123' 2023-06-22T08:03:43.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/19' 2023-06-22T08:03:43.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/15' 2023-06-22T08:03:43.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/20' 2023-06-22T08:03:43.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/12' 2023-06-22T08:03:43.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/10' 2023-06-22T08:03:43.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/11' 2023-06-22T08:03:43.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/16' 2023-06-22T08:03:43.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/14' 2023-06-22T08:03:43.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/18' 2023-06-22T08:03:43.368 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/13' 2023-06-22T08:03:43.368 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/160/17' 2023-06-22T08:03:43.368 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/160' 2023-06-22T08:03:43.369 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/19' 2023-06-22T08:03:43.369 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/15' 2023-06-22T08:03:43.369 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/20' 2023-06-22T08:03:43.370 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/12' 2023-06-22T08:03:43.370 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/10' 2023-06-22T08:03:43.370 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/11' 2023-06-22T08:03:43.371 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/16' 2023-06-22T08:03:43.371 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/14' 2023-06-22T08:03:43.371 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/18' 2023-06-22T08:03:43.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/13' 2023-06-22T08:03:43.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/165/17' 2023-06-22T08:03:43.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/165' 2023-06-22T08:03:43.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/19' 2023-06-22T08:03:43.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/15' 2023-06-22T08:03:43.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/20' 2023-06-22T08:03:43.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/12' 2023-06-22T08:03:43.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/10' 2023-06-22T08:03:43.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/11' 2023-06-22T08:03:43.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/16' 2023-06-22T08:03:43.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/14' 2023-06-22T08:03:43.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/18' 2023-06-22T08:03:43.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/13' 2023-06-22T08:03:43.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/101/17' 2023-06-22T08:03:43.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/101' 2023-06-22T08:03:43.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/19' 2023-06-22T08:03:43.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/15' 2023-06-22T08:03:43.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/20' 2023-06-22T08:03:43.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/12' 2023-06-22T08:03:43.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/10' 2023-06-22T08:03:43.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/11' 2023-06-22T08:03:43.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/16' 2023-06-22T08:03:43.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/14' 2023-06-22T08:03:43.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/18' 2023-06-22T08:03:43.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/13' 2023-06-22T08:03:43.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/148/17' 2023-06-22T08:03:43.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/148' 2023-06-22T08:03:43.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/19' 2023-06-22T08:03:43.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/15' 2023-06-22T08:03:43.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/20' 2023-06-22T08:03:43.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/12' 2023-06-22T08:03:43.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/10' 2023-06-22T08:03:43.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/11' 2023-06-22T08:03:43.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/16' 2023-06-22T08:03:43.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/14' 2023-06-22T08:03:43.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/18' 2023-06-22T08:03:43.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/13' 2023-06-22T08:03:43.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/168/17' 2023-06-22T08:03:43.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/168' 2023-06-22T08:03:43.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/19' 2023-06-22T08:03:43.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/15' 2023-06-22T08:03:43.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/20' 2023-06-22T08:03:43.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/12' 2023-06-22T08:03:43.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/10' 2023-06-22T08:03:43.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/11' 2023-06-22T08:03:43.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/16' 2023-06-22T08:03:43.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/14' 2023-06-22T08:03:43.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/18' 2023-06-22T08:03:43.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/13' 2023-06-22T08:03:43.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/179/17' 2023-06-22T08:03:43.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/179' 2023-06-22T08:03:43.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/19' 2023-06-22T08:03:43.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/15' 2023-06-22T08:03:43.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/20' 2023-06-22T08:03:43.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/12' 2023-06-22T08:03:43.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/10' 2023-06-22T08:03:43.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/11' 2023-06-22T08:03:43.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/16' 2023-06-22T08:03:43.632 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/14' 2023-06-22T08:03:43.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/18' 2023-06-22T08:03:43.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/13' 2023-06-22T08:03:43.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/132/17' 2023-06-22T08:03:43.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/132' 2023-06-22T08:03:43.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/19' 2023-06-22T08:03:43.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/15' 2023-06-22T08:03:43.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/20' 2023-06-22T08:03:43.633 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/12' 2023-06-22T08:03:43.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/10' 2023-06-22T08:03:43.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/11' 2023-06-22T08:03:43.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/16' 2023-06-22T08:03:43.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/14' 2023-06-22T08:03:43.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/18' 2023-06-22T08:03:43.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/13' 2023-06-22T08:03:43.634 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/199/17' 2023-06-22T08:03:43.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/199' 2023-06-22T08:03:43.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/19' 2023-06-22T08:03:43.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/15' 2023-06-22T08:03:43.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/20' 2023-06-22T08:03:43.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/12' 2023-06-22T08:03:43.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/10' 2023-06-22T08:03:43.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/11' 2023-06-22T08:03:43.635 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/16' 2023-06-22T08:03:43.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/14' 2023-06-22T08:03:43.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/18' 2023-06-22T08:03:43.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/13' 2023-06-22T08:03:43.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/163/17' 2023-06-22T08:03:43.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/163' 2023-06-22T08:03:43.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/19' 2023-06-22T08:03:43.636 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/15' 2023-06-22T08:03:43.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/20' 2023-06-22T08:03:43.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/12' 2023-06-22T08:03:43.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/10' 2023-06-22T08:03:43.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/11' 2023-06-22T08:03:43.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/16' 2023-06-22T08:03:43.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/14' 2023-06-22T08:03:43.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/18' 2023-06-22T08:03:43.637 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/13' 2023-06-22T08:03:43.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/153/17' 2023-06-22T08:03:43.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/153' 2023-06-22T08:03:43.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/19' 2023-06-22T08:03:43.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/15' 2023-06-22T08:03:43.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/20' 2023-06-22T08:03:43.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/12' 2023-06-22T08:03:43.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/10' 2023-06-22T08:03:43.638 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/11' 2023-06-22T08:03:43.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/16' 2023-06-22T08:03:43.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/14' 2023-06-22T08:03:43.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/18' 2023-06-22T08:03:43.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/13' 2023-06-22T08:03:43.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/122/17' 2023-06-22T08:03:43.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/122' 2023-06-22T08:03:43.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/19' 2023-06-22T08:03:43.639 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/15' 2023-06-22T08:03:43.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/20' 2023-06-22T08:03:43.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/12' 2023-06-22T08:03:43.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/10' 2023-06-22T08:03:43.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/11' 2023-06-22T08:03:43.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/16' 2023-06-22T08:03:43.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/14' 2023-06-22T08:03:43.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/18' 2023-06-22T08:03:43.640 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/13' 2023-06-22T08:03:43.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/162/17' 2023-06-22T08:03:43.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/162' 2023-06-22T08:03:43.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/19' 2023-06-22T08:03:43.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/15' 2023-06-22T08:03:43.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/20' 2023-06-22T08:03:43.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/12' 2023-06-22T08:03:43.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/10' 2023-06-22T08:03:43.641 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/11' 2023-06-22T08:03:43.642 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/16' 2023-06-22T08:03:43.642 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/14' 2023-06-22T08:03:43.642 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/18' 2023-06-22T08:03:43.642 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/13' 2023-06-22T08:03:43.642 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/197/17' 2023-06-22T08:03:43.642 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/197' 2023-06-22T08:03:43.642 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/19' 2023-06-22T08:03:43.642 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/15' 2023-06-22T08:03:43.643 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/20' 2023-06-22T08:03:43.643 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/12' 2023-06-22T08:03:43.643 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/10' 2023-06-22T08:03:43.643 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/11' 2023-06-22T08:03:43.643 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/16' 2023-06-22T08:03:43.643 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/14' 2023-06-22T08:03:43.643 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/18' 2023-06-22T08:03:43.643 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/13' 2023-06-22T08:03:43.644 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/113/17' 2023-06-22T08:03:43.644 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/113' 2023-06-22T08:03:43.644 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/19' 2023-06-22T08:03:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/15' 2023-06-22T08:03:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/20' 2023-06-22T08:03:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/12' 2023-06-22T08:03:44.057 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/10' 2023-06-22T08:03:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/11' 2023-06-22T08:03:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/16' 2023-06-22T08:03:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/14' 2023-06-22T08:03:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/18' 2023-06-22T08:03:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/13' 2023-06-22T08:03:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/152/17' 2023-06-22T08:03:44.058 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/152' 2023-06-22T08:03:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/19' 2023-06-22T08:03:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/15' 2023-06-22T08:03:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/20' 2023-06-22T08:03:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/12' 2023-06-22T08:03:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/10' 2023-06-22T08:03:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/11' 2023-06-22T08:03:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/16' 2023-06-22T08:03:44.059 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/14' 2023-06-22T08:03:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/18' 2023-06-22T08:03:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/13' 2023-06-22T08:03:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/121/17' 2023-06-22T08:03:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/121' 2023-06-22T08:03:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/19' 2023-06-22T08:03:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/15' 2023-06-22T08:03:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/20' 2023-06-22T08:03:44.060 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/12' 2023-06-22T08:03:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/10' 2023-06-22T08:03:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/11' 2023-06-22T08:03:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/16' 2023-06-22T08:03:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/14' 2023-06-22T08:03:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/18' 2023-06-22T08:03:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/13' 2023-06-22T08:03:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/118/17' 2023-06-22T08:03:44.061 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/118' 2023-06-22T08:03:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/19' 2023-06-22T08:03:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/15' 2023-06-22T08:03:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/20' 2023-06-22T08:03:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/12' 2023-06-22T08:03:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/10' 2023-06-22T08:03:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/11' 2023-06-22T08:03:44.062 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/16' 2023-06-22T08:03:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/14' 2023-06-22T08:03:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/18' 2023-06-22T08:03:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/13' 2023-06-22T08:03:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/196/17' 2023-06-22T08:03:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/196' 2023-06-22T08:03:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/19' 2023-06-22T08:03:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/15' 2023-06-22T08:03:44.063 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/20' 2023-06-22T08:03:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/12' 2023-06-22T08:03:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/10' 2023-06-22T08:03:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/11' 2023-06-22T08:03:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/16' 2023-06-22T08:03:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/14' 2023-06-22T08:03:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/18' 2023-06-22T08:03:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/13' 2023-06-22T08:03:44.064 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/124/17' 2023-06-22T08:03:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/124' 2023-06-22T08:03:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/19' 2023-06-22T08:03:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/15' 2023-06-22T08:03:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/20' 2023-06-22T08:03:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/12' 2023-06-22T08:03:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/10' 2023-06-22T08:03:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/11' 2023-06-22T08:03:44.065 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/16' 2023-06-22T08:03:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/14' 2023-06-22T08:03:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/18' 2023-06-22T08:03:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/13' 2023-06-22T08:03:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/188/17' 2023-06-22T08:03:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/188' 2023-06-22T08:03:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/19' 2023-06-22T08:03:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/15' 2023-06-22T08:03:44.066 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/20' 2023-06-22T08:03:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/12' 2023-06-22T08:03:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/10' 2023-06-22T08:03:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/11' 2023-06-22T08:03:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/16' 2023-06-22T08:03:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/14' 2023-06-22T08:03:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/18' 2023-06-22T08:03:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/13' 2023-06-22T08:03:44.067 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/145/17' 2023-06-22T08:03:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/145' 2023-06-22T08:03:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/19' 2023-06-22T08:03:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/15' 2023-06-22T08:03:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/20' 2023-06-22T08:03:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/12' 2023-06-22T08:03:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/10' 2023-06-22T08:03:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/11' 2023-06-22T08:03:44.068 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/16' 2023-06-22T08:03:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/14' 2023-06-22T08:03:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/18' 2023-06-22T08:03:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/13' 2023-06-22T08:03:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/130/17' 2023-06-22T08:03:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/130' 2023-06-22T08:03:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/19' 2023-06-22T08:03:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/15' 2023-06-22T08:03:44.069 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/20' 2023-06-22T08:03:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/12' 2023-06-22T08:03:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/10' 2023-06-22T08:03:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/11' 2023-06-22T08:03:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/16' 2023-06-22T08:03:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/14' 2023-06-22T08:03:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/18' 2023-06-22T08:03:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/13' 2023-06-22T08:03:44.070 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/193/17' 2023-06-22T08:03:44.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/193' 2023-06-22T08:03:44.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/19' 2023-06-22T08:03:44.071 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/15' 2023-06-22T08:03:44.438 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/20' 2023-06-22T08:03:44.439 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/12' 2023-06-22T08:03:44.439 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/10' 2023-06-22T08:03:44.439 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/11' 2023-06-22T08:03:44.439 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/16' 2023-06-22T08:03:44.439 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/14' 2023-06-22T08:03:44.439 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/18' 2023-06-22T08:03:44.439 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/13' 2023-06-22T08:03:44.440 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/175/17' 2023-06-22T08:03:44.440 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/175' 2023-06-22T08:03:44.440 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/19' 2023-06-22T08:03:44.440 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/15' 2023-06-22T08:03:44.440 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/20' 2023-06-22T08:03:44.440 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/12' 2023-06-22T08:03:44.440 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/10' 2023-06-22T08:03:44.441 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/11' 2023-06-22T08:03:44.441 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/16' 2023-06-22T08:03:44.441 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/14' 2023-06-22T08:03:44.441 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/18' 2023-06-22T08:03:44.441 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/13' 2023-06-22T08:03:44.441 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/181/17' 2023-06-22T08:03:44.441 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/181' 2023-06-22T08:03:44.441 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/19' 2023-06-22T08:03:44.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/15' 2023-06-22T08:03:44.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/20' 2023-06-22T08:03:44.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/12' 2023-06-22T08:03:44.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/10' 2023-06-22T08:03:44.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/11' 2023-06-22T08:03:44.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/16' 2023-06-22T08:03:44.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/14' 2023-06-22T08:03:44.442 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/18' 2023-06-22T08:03:44.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/13' 2023-06-22T08:03:44.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/127/17' 2023-06-22T08:03:44.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/127' 2023-06-22T08:03:44.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/19' 2023-06-22T08:03:44.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/15' 2023-06-22T08:03:44.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/20' 2023-06-22T08:03:44.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/12' 2023-06-22T08:03:44.443 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/10' 2023-06-22T08:03:44.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/11' 2023-06-22T08:03:44.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/16' 2023-06-22T08:03:44.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/14' 2023-06-22T08:03:44.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/18' 2023-06-22T08:03:44.444 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/13' 2023-06-22T08:03:44.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/108/17' 2023-06-22T08:03:44.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/108' 2023-06-22T08:03:44.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/19' 2023-06-22T08:03:44.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/15' 2023-06-22T08:03:44.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/20' 2023-06-22T08:03:44.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/12' 2023-06-22T08:03:44.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/10' 2023-06-22T08:03:44.445 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/11' 2023-06-22T08:03:44.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/16' 2023-06-22T08:03:44.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/14' 2023-06-22T08:03:44.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/18' 2023-06-22T08:03:44.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/13' 2023-06-22T08:03:44.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/183/17' 2023-06-22T08:03:44.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/183' 2023-06-22T08:03:44.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/19' 2023-06-22T08:03:44.446 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/15' 2023-06-22T08:03:44.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/20' 2023-06-22T08:03:44.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/12' 2023-06-22T08:03:44.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/10' 2023-06-22T08:03:44.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/11' 2023-06-22T08:03:44.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/16' 2023-06-22T08:03:44.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/14' 2023-06-22T08:03:44.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/18' 2023-06-22T08:03:44.447 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/13' 2023-06-22T08:03:44.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/177/17' 2023-06-22T08:03:44.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/177' 2023-06-22T08:03:44.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/19' 2023-06-22T08:03:44.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/15' 2023-06-22T08:03:44.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/20' 2023-06-22T08:03:44.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/12' 2023-06-22T08:03:44.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/10' 2023-06-22T08:03:44.448 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/11' 2023-06-22T08:03:44.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/16' 2023-06-22T08:03:44.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/14' 2023-06-22T08:03:44.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/18' 2023-06-22T08:03:44.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/13' 2023-06-22T08:03:44.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/169/17' 2023-06-22T08:03:44.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/169' 2023-06-22T08:03:44.449 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/19' 2023-06-22T08:03:44.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/15' 2023-06-22T08:03:44.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/20' 2023-06-22T08:03:44.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/12' 2023-06-22T08:03:44.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/10' 2023-06-22T08:03:44.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/11' 2023-06-22T08:03:44.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/16' 2023-06-22T08:03:44.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/14' 2023-06-22T08:03:44.450 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/18' 2023-06-22T08:03:44.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/13' 2023-06-22T08:03:44.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/154/17' 2023-06-22T08:03:44.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/154' 2023-06-22T08:03:44.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/19' 2023-06-22T08:03:44.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/15' 2023-06-22T08:03:44.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/20' 2023-06-22T08:03:44.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/12' 2023-06-22T08:03:44.451 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/10' 2023-06-22T08:03:44.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/11' 2023-06-22T08:03:44.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/16' 2023-06-22T08:03:44.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/14' 2023-06-22T08:03:44.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/18' 2023-06-22T08:03:44.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/13' 2023-06-22T08:03:44.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/171/17' 2023-06-22T08:03:44.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/171' 2023-06-22T08:03:44.452 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/19' 2023-06-22T08:03:44.453 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/15' 2023-06-22T08:03:44.453 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/20' 2023-06-22T08:03:44.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/12' 2023-06-22T08:03:44.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/10' 2023-06-22T08:03:44.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/11' 2023-06-22T08:03:44.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/16' 2023-06-22T08:03:44.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/14' 2023-06-22T08:03:44.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/18' 2023-06-22T08:03:44.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/13' 2023-06-22T08:03:44.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/139/17' 2023-06-22T08:03:44.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/139' 2023-06-22T08:03:44.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/19' 2023-06-22T08:03:44.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/15' 2023-06-22T08:03:44.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/20' 2023-06-22T08:03:44.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/12' 2023-06-22T08:03:44.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/10' 2023-06-22T08:03:44.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/11' 2023-06-22T08:03:44.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/16' 2023-06-22T08:03:44.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/14' 2023-06-22T08:03:44.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/18' 2023-06-22T08:03:44.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/13' 2023-06-22T08:03:44.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/102/17' 2023-06-22T08:03:44.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/102' 2023-06-22T08:03:44.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/19' 2023-06-22T08:03:44.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/15' 2023-06-22T08:03:44.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/20' 2023-06-22T08:03:44.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/12' 2023-06-22T08:03:44.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/10' 2023-06-22T08:03:44.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/11' 2023-06-22T08:03:44.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/16' 2023-06-22T08:03:44.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/14' 2023-06-22T08:03:44.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/18' 2023-06-22T08:03:44.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/13' 2023-06-22T08:03:44.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed './copyofsnap1/test1/129/17' 2023-06-22T08:03:44.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1/129' 2023-06-22T08:03:44.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1/test1' 2023-06-22T08:03:44.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory './copyofsnap1' 2023-06-22T08:03:44.535 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:03:44.536 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:03:44.537 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:03:44.621 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-06-22T08:03:44.621 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-06-22T08:03:44.676 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:03:44.677 INFO:tasks.workunit.client.0.smithi023.stderr:+ touch foo 2023-06-22T08:03:44.680 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod +x foo 2023-06-22T08:03:44.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:03:44.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ find .snap/s/foo -executable 2023-06-22T08:03:44.693 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep foo 2023-06-22T08:03:44.697 INFO:tasks.workunit.client.0.smithi023.stdout:.snap/s/foo 2023-06-22T08:03:44.698 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:03:44.704 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm foo 2023-06-22T08:03:44.709 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:03:44.710 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:03:44.710 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:03:44.710 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:03:44.785 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-06-22T08:03:44.786 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-06-22T08:03:44.871 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:03:44.871 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir foo 2023-06-22T08:03:44.876 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:44.888 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo x 2023-06-22T08:03:44.889 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:03:45.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.173+0000 7fb93d1ff700 1 Processor -- start 2023-06-22T08:03:45.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.173+0000 7fb93d1ff700 1 -- start start 2023-06-22T08:03:45.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.174+0000 7fb93d1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c9fd0 0x7fb9380cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.174+0000 7fb93d1ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb93815c370 con 0x7fb938152ce0 2023-06-22T08:03:45.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.174+0000 7fb93d1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb93815bda0 con 0x7fb9380c40b0 2023-06-22T08:03:45.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.174+0000 7fb93d1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9381530b0 con 0x7fb9380c9fd0 2023-06-22T08:03:45.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.174+0000 7fb936d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb938152ce0 0x7fb93815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:45884/0 (socket says 172.21.15.23:45884) 2023-06-22T08:03:45.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.174+0000 7fb936d9d700 1 -- 172.21.15.23:0/1779416354 learned_addr learned my addr 172.21.15.23:0/1779416354 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:03:45.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.174+0000 7fb93659c700 1 --2- 172.21.15.23:0/1779416354 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c9fd0 0x7fb9380cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.175+0000 7fb935d9b700 1 -- 172.21.15.23:0/1779416354 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522145919 0 0) 0x7fb93815c370 con 0x7fb938152ce0 2023-06-22T08:03:45.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.175+0000 7fb935d9b700 1 -- 172.21.15.23:0/1779416354 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9200035b0 con 0x7fb938152ce0 2023-06-22T08:03:45.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.175+0000 7fb935d9b700 1 -- 172.21.15.23:0/1779416354 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705920950 0 0) 0x7fb93815bda0 con 0x7fb9380c40b0 2023-06-22T08:03:45.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.175+0000 7fb935d9b700 1 -- 172.21.15.23:0/1779416354 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb93815c370 con 0x7fb9380c40b0 2023-06-22T08:03:45.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.175+0000 7fb93659c700 1 -- 172.21.15.23:0/1779416354 >> v1:172.21.15.67:6789/0 conn(0x7fb9380c40b0 legacy=0x7fb9380c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.175+0000 7fb93659c700 1 -- 172.21.15.23:0/1779416354 >> v1:172.21.15.23:6789/0 conn(0x7fb938152ce0 legacy=0x7fb93815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.175+0000 7fb93659c700 1 -- 172.21.15.23:0/1779416354 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9380c9460 con 0x7fb9380c9fd0 2023-06-22T08:03:45.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.175+0000 7fb93659c700 1 --2- 172.21.15.23:0/1779416354 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c9fd0 0x7fb9380cf1b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb92c009870 tx=0x7fb92c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab2647f18fe41d0e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:45.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.175+0000 7fb935d9b700 1 -- 172.21.15.23:0/1779416354 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c011830 con 0x7fb9380c9fd0 2023-06-22T08:03:45.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.175+0000 7fb935d9b700 1 -- 172.21.15.23:0/1779416354 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92c011990 con 0x7fb9380c9fd0 2023-06-22T08:03:45.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.176+0000 7fb935d9b700 1 -- 172.21.15.23:0/1779416354 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c011d20 con 0x7fb9380c9fd0 2023-06-22T08:03:45.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.176+0000 7fb93d1ff700 1 -- 172.21.15.23:0/1779416354 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c9fd0 msgr2=0x7fb9380cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.176+0000 7fb93d1ff700 1 --2- 172.21.15.23:0/1779416354 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c9fd0 0x7fb9380cf1b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb92c009870 tx=0x7fb92c009b80 comp rx=0 tx=0).stop 2023-06-22T08:03:45.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.176+0000 7fb93d1ff700 1 -- 172.21.15.23:0/1779416354 shutdown_connections 2023-06-22T08:03:45.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.176+0000 7fb93d1ff700 1 --2- 172.21.15.23:0/1779416354 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c9fd0 0x7fb9380cf1b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.176+0000 7fb93d1ff700 1 -- 172.21.15.23:0/1779416354 >> 172.21.15.23:0/1779416354 conn(0x7fb93806ae30 msgr2=0x7fb93806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:45.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.176+0000 7fb93d1ff700 1 -- 172.21.15.23:0/1779416354 shutdown_connections 2023-06-22T08:03:45.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.176+0000 7fb93d1ff700 1 -- 172.21.15.23:0/1779416354 wait complete. 2023-06-22T08:03:45.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.179+0000 7fb93d1ff700 1 Processor -- start 2023-06-22T08:03:45.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.179+0000 7fb93d1ff700 1 -- start start 2023-06-22T08:03:45.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.180+0000 7fb93d1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c40b0 0x7fb9380cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.180+0000 7fb93d1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9380c9fd0 0x7fb9381e3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.180+0000 7fb93d1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb938152ce0 0x7fb9381f1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb93d1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9381f2420 con 0x7fb9380c9fd0 2023-06-22T08:03:45.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb93d1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9380ce300 con 0x7fb938152ce0 2023-06-22T08:03:45.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb93d1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9381e2640 con 0x7fb9380c40b0 2023-06-22T08:03:45.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb936d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c40b0 0x7fb9380cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb936d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c40b0 0x7fb9380cd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:57578/0 (socket says 172.21.15.23:57578) 2023-06-22T08:03:45.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb93759e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb938152ce0 0x7fb9381f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb936d9d700 1 -- 172.21.15.23:0/464855774 learned_addr learned my addr 172.21.15.23:0/464855774 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:45.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb93659c700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9380c9fd0 0x7fb9381e3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb936d9d700 1 -- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb938152ce0 msgr2=0x7fb9381f1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb936d9d700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb938152ce0 0x7fb9381f1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb936d9d700 1 -- 172.21.15.23:0/464855774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9380c9fd0 msgr2=0x7fb9381e3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb936d9d700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9380c9fd0 0x7fb9381e3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb936d9d700 1 -- 172.21.15.23:0/464855774 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c0094a0 con 0x7fb9380c40b0 2023-06-22T08:03:45.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.181+0000 7fb936d9d700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c40b0 0x7fb9380cd050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb92800c700 tx=0x7fb92800ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:45.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.182+0000 7fb91f7fe700 1 -- 172.21.15.23:0/464855774 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb928005d70 con 0x7fb9380c40b0 2023-06-22T08:03:45.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.182+0000 7fb91f7fe700 1 -- 172.21.15.23:0/464855774 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9280070d0 con 0x7fb9380c40b0 2023-06-22T08:03:45.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.182+0000 7fb93d1ff700 1 -- 172.21.15.23:0/464855774 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9381f2140 con 0x7fb9380c40b0 2023-06-22T08:03:45.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.182+0000 7fb91f7fe700 1 -- 172.21.15.23:0/464855774 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb928007440 con 0x7fb9380c40b0 2023-06-22T08:03:45.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.182+0000 7fb93d1ff700 1 -- 172.21.15.23:0/464855774 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb938156130 con 0x7fb9380c40b0 2023-06-22T08:03:45.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.183+0000 7fb93d1ff700 1 -- 172.21.15.23:0/464855774 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb938155cb0 con 0x7fb9380c40b0 2023-06-22T08:03:45.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.184+0000 7fb91f7fe700 1 -- 172.21.15.23:0/464855774 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb9280075a0 con 0x7fb9380c40b0 2023-06-22T08:03:45.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.185+0000 7fb91f7fe700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb92406ace0 0x7fb92406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.185+0000 7fb91f7fe700 1 -- 172.21.15.23:0/464855774 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7fb928088ae0 con 0x7fb9380c40b0 2023-06-22T08:03:45.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.185+0000 7fb93659c700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb92406ace0 0x7fb92406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.186+0000 7fb93659c700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb92406ace0 0x7fb92406d190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0097c0 tx=0x7fb92c01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:45.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.189+0000 7fb91f7fe700 1 -- 172.21.15.23:0/464855774 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb938155cb0 con 0x7fb9380c40b0 2023-06-22T08:03:45.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.445+0000 7fb93d1ff700 1 -- 172.21.15.23:0/464855774 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9381e3040 con 0x7fb9380c40b0 2023-06-22T08:03:45.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.446+0000 7fb91f7fe700 1 -- 172.21.15.23:0/464855774 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6261 (secure 0 0 0) 0x7fb9381e3040 con 0x7fb9380c40b0 2023-06-22T08:03:45.449 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:45.450 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5211,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:41.679281+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:active","state_seq":59,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:45.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 -- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb92406ace0 msgr2=0x7fb92406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb92406ace0 0x7fb92406d190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb92c0097c0 tx=0x7fb92c01c040 comp rx=0 tx=0).stop 2023-06-22T08:03:45.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 -- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c40b0 msgr2=0x7fb9380cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c40b0 0x7fb9380cd050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb92800c700 tx=0x7fb92800ca10 comp rx=0 tx=0).stop 2023-06-22T08:03:45.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 -- 172.21.15.23:0/464855774 shutdown_connections 2023-06-22T08:03:45.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb92406ace0 0x7fb92406d190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb938152ce0 0x7fb9381f1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9380c40b0 0x7fb9380cd050 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 --2- 172.21.15.23:0/464855774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9380c9fd0 0x7fb9381e3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 -- 172.21.15.23:0/464855774 >> 172.21.15.23:0/464855774 conn(0x7fb93806ae30 msgr2=0x7fb938151dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:45.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 -- 172.21.15.23:0/464855774 shutdown_connections 2023-06-22T08:03:45.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.448+0000 7fb93d1ff700 1 -- 172.21.15.23:0/464855774 wait complete. 2023-06-22T08:03:45.456 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 33 2023-06-22T08:03:45.471 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=4) 2023-06-22T08:03:45.471 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-22T08:03:45.472 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:03:45.481 INFO:tasks.ceph.mds.d:Stopped 2023-06-22T08:03:45.482 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:45.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.677+0000 7f7cf83ff700 1 Processor -- start 2023-06-22T08:03:45.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.677+0000 7f7cf83ff700 1 -- start start 2023-06-22T08:03:45.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.678+0000 7f7cf83ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7cf00cb6b0 0x7f7cf0069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.678+0000 7f7cf83ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7cf0152e10 0x7f7cf01593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.678+0000 7f7cf83ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf0158f60 con 0x7f7cf00c5740 2023-06-22T08:03:45.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.678+0000 7f7cf83ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf006bd20 con 0x7f7cf0152e10 2023-06-22T08:03:45.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf83ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf00cc450 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf699c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7cf0152e10 0x7f7cf01593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf619b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7cf00cb6b0 0x7f7cf0069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf699c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7cf0152e10 0x7f7cf01593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46428/0 (socket says 172.21.15.23:46428) 2023-06-22T08:03:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf699c700 1 -- 172.21.15.23:0/2415644285 learned_addr learned my addr 172.21.15.23:0/2415644285 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf599a700 1 --1- 172.21.15.23:0/2415644285 >> v1:172.21.15.67:6789/0 conn(0x7f7cf00c5740 0x7f7cf0152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:56090/0 (socket says 172.21.15.23:56090) 2023-06-22T08:03:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf699c700 1 -- 172.21.15.23:0/2415644285 >> v2:172.21.15.67:3300/0 conn(0x7f7cf00cb6b0 msgr2=0x7f7cf0069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf699c700 1 --2- 172.21.15.23:0/2415644285 >> v2:172.21.15.67:3300/0 conn(0x7f7cf00cb6b0 0x7f7cf0069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf699c700 1 -- 172.21.15.23:0/2415644285 >> v1:172.21.15.67:6789/0 conn(0x7f7cf00c5740 legacy=0x7f7cf0152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf699c700 1 -- 172.21.15.23:0/2415644285 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf0158980 con 0x7f7cf0152e10 2023-06-22T08:03:45.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.679+0000 7f7cf619b700 1 --2- 172.21.15.23:0/2415644285 >> v2:172.21.15.67:3300/0 conn(0x7f7cf00cb6b0 0x7f7cf0069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:03:45.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.680+0000 7f7cf699c700 1 --2- 172.21.15.23:0/2415644285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0152e10 0x7f7cf01593f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00cc230 tx=0x7f7ce800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d928156e4943ea server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:45.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.680+0000 7f7cf5199700 1 -- 172.21.15.23:0/2415644285 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800e070 con 0x7f7cf0152e10 2023-06-22T08:03:45.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.680+0000 7f7cf5199700 1 -- 172.21.15.23:0/2415644285 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce800b910 con 0x7f7cf0152e10 2023-06-22T08:03:45.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.680+0000 7f7cf5199700 1 -- 172.21.15.23:0/2415644285 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800bcc0 con 0x7f7cf0152e10 2023-06-22T08:03:45.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.680+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2415644285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0152e10 msgr2=0x7f7cf01593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.680+0000 7f7cf83ff700 1 --2- 172.21.15.23:0/2415644285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0152e10 0x7f7cf01593f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00cc230 tx=0x7f7ce800b330 comp rx=0 tx=0).stop 2023-06-22T08:03:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.680+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2415644285 shutdown_connections 2023-06-22T08:03:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.680+0000 7f7cf83ff700 1 --2- 172.21.15.23:0/2415644285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0152e10 0x7f7cf01593f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.680+0000 7f7cf83ff700 1 --2- 172.21.15.23:0/2415644285 >> v2:172.21.15.67:3300/0 conn(0x7f7cf00cb6b0 0x7f7cf0069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.680+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2415644285 >> 172.21.15.23:0/2415644285 conn(0x7f7cf0070d80 msgr2=0x7f7cf0071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.681+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2415644285 shutdown_connections 2023-06-22T08:03:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.681+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2415644285 wait complete. 2023-06-22T08:03:45.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.682+0000 7f7cf83ff700 1 Processor -- start 2023-06-22T08:03:45.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.682+0000 7f7cf83ff700 1 -- start start 2023-06-22T08:03:45.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.683+0000 7f7cf83ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf00cb6b0 0x7f7cf01f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.683+0000 7f7cf83ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0152e10 0x7f7cf00ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.683+0000 7f7cf83ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7cf00c6bf0 0x7f7cf0143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.683+0000 7f7cf83ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01e4250 con 0x7f7cf0152e10 2023-06-22T08:03:45.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.683+0000 7f7cf83ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01e49f0 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.683+0000 7f7cf83ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01f1a60 con 0x7f7cf00c6bf0 2023-06-22T08:03:45.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf619b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf00cb6b0 0x7f7cf01f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf599a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0152e10 0x7f7cf00ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf699c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7cf00c6bf0 0x7f7cf0143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf619b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf00cb6b0 0x7f7cf01f1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:41648/0 (socket says 172.21.15.23:41648) 2023-06-22T08:03:45.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf619b700 1 -- 172.21.15.23:0/2070941170 learned_addr learned my addr 172.21.15.23:0/2070941170 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:45.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf619b700 1 -- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7cf00c6bf0 msgr2=0x7f7cf0143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf619b700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7cf00c6bf0 0x7f7cf0143610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf619b700 1 -- 172.21.15.23:0/2070941170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0152e10 msgr2=0x7f7cf00ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf619b700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0152e10 0x7f7cf00ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf619b700 1 -- 172.21.15.23:0/2070941170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec0094a0 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf599a700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0152e10 0x7f7cf00ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:03:45.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7cf619b700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf00cb6b0 0x7f7cf01f1ef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00ec10 tx=0x7f7cec00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:45.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.684+0000 7f7ce6ffd700 1 -- 172.21.15.23:0/2070941170 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec00c850 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.685+0000 7f7ce6ffd700 1 -- 172.21.15.23:0/2070941170 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec00c9b0 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.685+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2070941170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce8009b30 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.685+0000 7f7ce6ffd700 1 -- 172.21.15.23:0/2070941170 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec00cd20 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.685+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2070941170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf01e4430 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.686+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2070941170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf004ea20 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.687+0000 7f7ce6ffd700 1 -- 172.21.15.23:0/2070941170 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7cec013470 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.688+0000 7f7ce6ffd700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7cd406ace0 0x7f7cd406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:45.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.688+0000 7f7ce6ffd700 1 -- 172.21.15.23:0/2070941170 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6899+0+0 (secure 0 0 0) 0x7f7cec089990 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.688+0000 7f7cf599a700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7cd406ace0 0x7f7cd406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:45.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.688+0000 7f7cf599a700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7cd406ace0 0x7f7cd406d190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00c98b0 tx=0x7f7ce0007ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:45.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.692+0000 7f7ce6ffd700 1 -- 172.21.15.23:0/2070941170 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7cec0545c0 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.945+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2070941170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cf0156960 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.946+0000 7f7ce6ffd700 1 -- 172.21.15.23:0/2070941170 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6261 (secure 0 0 0) 0x7f7cec021090 con 0x7f7cf00cb6b0 2023-06-22T08:03:45.949 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:45.950 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5211,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:41.679281+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:active","state_seq":59,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:45.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7cd406ace0 msgr2=0x7f7cd406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7cd406ace0 0x7f7cd406d190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00c98b0 tx=0x7f7ce0007ed0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf00cb6b0 msgr2=0x7f7cf01f1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:45.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf00cb6b0 0x7f7cf01f1ef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00ec10 tx=0x7f7cec00ef20 comp rx=0 tx=0).stop 2023-06-22T08:03:45.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2070941170 shutdown_connections 2023-06-22T08:03:45.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7cf00c6bf0 0x7f7cf0143610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7cd406ace0 0x7f7cd406d190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf00cb6b0 0x7f7cf01f1ef0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 --2- 172.21.15.23:0/2070941170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0152e10 0x7f7cf00ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:45.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2070941170 >> 172.21.15.23:0/2070941170 conn(0x7f7cf0070d80 msgr2=0x7f7cf0157ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:45.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2070941170 shutdown_connections 2023-06-22T08:03:45.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:45.949+0000 7f7cf83ff700 1 -- 172.21.15.23:0/2070941170 wait complete. 2023-06-22T08:03:45.954 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 33 2023-06-22T08:03:45.970 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:03:47.971 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:48.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.180+0000 7f8a1ffff700 1 Processor -- start 2023-06-22T08:03:48.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.180+0000 7f8a1ffff700 1 -- start start 2023-06-22T08:03:48.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.181+0000 7f8a1ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c9fd0 0x7f8a180cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.181+0000 7f8a1ffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a180c40b0 0x7f8a180c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.181+0000 7f8a1ffff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1815c370 con 0x7f8a18152ce0 2023-06-22T08:03:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.181+0000 7f8a1ffff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a181530b0 con 0x7f8a180c40b0 2023-06-22T08:03:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.181+0000 7f8a1ffff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180cf6f0 con 0x7f8a180c9fd0 2023-06-22T08:03:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.181+0000 7f8a1d59a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c9fd0 0x7f8a180cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.181+0000 7f8a1e59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a180c40b0 0x7f8a180c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.181+0000 7f8a1dd9b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a18152ce0 0x7f8a1815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58750/0 (socket says 172.21.15.23:58750) 2023-06-22T08:03:48.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.181+0000 7f8a1dd9b700 1 -- 172.21.15.23:0/1596532536 learned_addr learned my addr 172.21.15.23:0/1596532536 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:03:48.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.182+0000 7f8a1cd99700 1 -- 172.21.15.23:0/1596532536 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937494639 0 0) 0x7f8a1815c370 con 0x7f8a18152ce0 2023-06-22T08:03:48.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.182+0000 7f8a1cd99700 1 -- 172.21.15.23:0/1596532536 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a040035b0 con 0x7f8a18152ce0 2023-06-22T08:03:48.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.182+0000 7f8a1d59a700 1 -- 172.21.15.23:0/1596532536 >> v2:172.21.15.67:3300/0 conn(0x7f8a180c40b0 msgr2=0x7f8a180c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:48.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.182+0000 7f8a1d59a700 1 --2- 172.21.15.23:0/1596532536 >> v2:172.21.15.67:3300/0 conn(0x7f8a180c40b0 0x7f8a180c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:48.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.182+0000 7f8a1d59a700 1 -- 172.21.15.23:0/1596532536 >> v1:172.21.15.67:6789/0 conn(0x7f8a18152ce0 legacy=0x7f8a1815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:48.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.182+0000 7f8a1d59a700 1 -- 172.21.15.23:0/1596532536 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1815bda0 con 0x7f8a180c9fd0 2023-06-22T08:03:48.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.182+0000 7f8a1d59a700 1 --2- 172.21.15.23:0/1596532536 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c9fd0 0x7f8a180cf1b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009840 tx=0x7f8a08009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=852f02f70e1d948e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:48.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.182+0000 7f8a1cd99700 1 -- 172.21.15.23:0/1596532536 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a08011830 con 0x7f8a180c9fd0 2023-06-22T08:03:48.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.182+0000 7f8a1cd99700 1 -- 172.21.15.23:0/1596532536 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a08011990 con 0x7f8a180c9fd0 2023-06-22T08:03:48.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.182+0000 7f8a1cd99700 1 -- 172.21.15.23:0/1596532536 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a08011d00 con 0x7f8a180c9fd0 2023-06-22T08:03:48.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.183+0000 7f8a1ffff700 1 -- 172.21.15.23:0/1596532536 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c9fd0 msgr2=0x7f8a180cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:48.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.183+0000 7f8a1ffff700 1 --2- 172.21.15.23:0/1596532536 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c9fd0 0x7f8a180cf1b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009840 tx=0x7f8a08009b50 comp rx=0 tx=0).stop 2023-06-22T08:03:48.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.183+0000 7f8a1ffff700 1 -- 172.21.15.23:0/1596532536 shutdown_connections 2023-06-22T08:03:48.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.183+0000 7f8a1ffff700 1 --2- 172.21.15.23:0/1596532536 >> v2:172.21.15.67:3300/0 conn(0x7f8a180c40b0 0x7f8a180c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:48.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.183+0000 7f8a1ffff700 1 --2- 172.21.15.23:0/1596532536 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c9fd0 0x7f8a180cf1b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:48.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.183+0000 7f8a1ffff700 1 -- 172.21.15.23:0/1596532536 >> 172.21.15.23:0/1596532536 conn(0x7f8a1806ae30 msgr2=0x7f8a1806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:48.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.183+0000 7f8a1ffff700 1 -- 172.21.15.23:0/1596532536 shutdown_connections 2023-06-22T08:03:48.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.183+0000 7f8a1ffff700 1 -- 172.21.15.23:0/1596532536 wait complete. 2023-06-22T08:03:48.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.185+0000 7f8a1ffff700 1 Processor -- start 2023-06-22T08:03:48.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.185+0000 7f8a1ffff700 1 -- start start 2023-06-22T08:03:48.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c40b0 0x7f8a1814a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:48.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a180c9fd0 0x7f8a180cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:48.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a18152ce0 0x7f8a18150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:48.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1ffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180cddc0 con 0x7f8a18152ce0 2023-06-22T08:03:48.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1ffff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a181e4320 con 0x7f8a180c9fd0 2023-06-22T08:03:48.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1ffff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a181e3e90 con 0x7f8a180c40b0 2023-06-22T08:03:48.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a18152ce0 0x7f8a18150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:48.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1d59a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a180c9fd0 0x7f8a180cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:48.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1dd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c40b0 0x7f8a1814a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:48.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a18152ce0 0x7f8a18150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46652/0 (socket says 172.21.15.23:46652) 2023-06-22T08:03:48.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.186+0000 7f8a1e59c700 1 -- 172.21.15.23:0/3705486377 learned_addr learned my addr 172.21.15.23:0/3705486377 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:48.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a1d59a700 1 -- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c40b0 msgr2=0x7f8a1814a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:48.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a1d59a700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c40b0 0x7f8a1814a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:48.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a1d59a700 1 -- 172.21.15.23:0/3705486377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a18152ce0 msgr2=0x7f8a18150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:48.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a1d59a700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a18152ce0 0x7f8a18150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:48.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a1d59a700 1 -- 172.21.15.23:0/3705486377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a080094a0 con 0x7f8a180c9fd0 2023-06-22T08:03:48.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a1dd9b700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c40b0 0x7f8a1814a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:03:48.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a1d59a700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a180c9fd0 0x7f8a180cc690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8a08016000 tx=0x7f8a08004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:48.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a0e7fc700 1 -- 172.21.15.23:0/3705486377 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a08004550 con 0x7f8a180c9fd0 2023-06-22T08:03:48.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a0e7fc700 1 -- 172.21.15.23:0/3705486377 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0801f070 con 0x7f8a180c9fd0 2023-06-22T08:03:48.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a0e7fc700 1 -- 172.21.15.23:0/3705486377 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a080038c0 con 0x7f8a180c9fd0 2023-06-22T08:03:48.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.187+0000 7f8a1ffff700 1 -- 172.21.15.23:0/3705486377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a181ee560 con 0x7f8a180c9fd0 2023-06-22T08:03:48.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.188+0000 7f8a1ffff700 1 -- 172.21.15.23:0/3705486377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a180cf2b0 con 0x7f8a180c9fd0 2023-06-22T08:03:48.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.189+0000 7f8a1ffff700 1 -- 172.21.15.23:0/3705486377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a180cf4f0 con 0x7f8a180c9fd0 2023-06-22T08:03:48.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.190+0000 7f8a0e7fc700 1 -- 172.21.15.23:0/3705486377 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8a08003a20 con 0x7f8a180c9fd0 2023-06-22T08:03:48.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.191+0000 7f8a0e7fc700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89fc06ace0 0x7f89fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:48.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.191+0000 7f8a0e7fc700 1 -- 172.21.15.23:0/3705486377 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 7011+0+0 (secure 0 0 0) 0x7f8a08021070 con 0x7f8a180c9fd0 2023-06-22T08:03:48.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.191+0000 7f8a1dd9b700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89fc06ace0 0x7f89fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:48.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.192+0000 7f8a1dd9b700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89fc06ace0 0x7f89fc06d190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8a14007fd0 tx=0x7f8a14007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:48.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.196+0000 7f8a0e7fc700 1 -- 172.21.15.23:0/3705486377 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8a180cf4f0 con 0x7f8a180c9fd0 2023-06-22T08:03:48.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.447+0000 7f8a1ffff700 1 -- 172.21.15.23:0/3705486377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a180cf4f0 con 0x7f8a180c9fd0 2023-06-22T08:03:48.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.448+0000 7f8a0e7fc700 1 -- 172.21.15.23:0/3705486377 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6269 (secure 0 0 0) 0x7f8a180cf4f0 con 0x7f8a180c9fd0 2023-06-22T08:03:48.451 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:48.452 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5211,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:47.314059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:active","state_seq":59,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:48.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 -- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89fc06ace0 msgr2=0x7f89fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:48.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89fc06ace0 0x7f89fc06d190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8a14007fd0 tx=0x7f8a14007dc0 comp rx=0 tx=0).stop 2023-06-22T08:03:48.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 -- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a180c9fd0 msgr2=0x7f8a180cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:48.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a180c9fd0 0x7f8a180cc690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8a08016000 tx=0x7f8a08004750 comp rx=0 tx=0).stop 2023-06-22T08:03:48.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 -- 172.21.15.23:0/3705486377 shutdown_connections 2023-06-22T08:03:48.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a18152ce0 0x7f8a18150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:48.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a180c40b0 0x7f8a1814a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:48.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89fc06ace0 0x7f89fc06d190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:48.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 --2- 172.21.15.23:0/3705486377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a180c9fd0 0x7f8a180cc690 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:48.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 -- 172.21.15.23:0/3705486377 >> 172.21.15.23:0/3705486377 conn(0x7f8a1806ae30 msgr2=0x7f8a18150500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:48.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 -- 172.21.15.23:0/3705486377 shutdown_connections 2023-06-22T08:03:48.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:48.450+0000 7f8a1ffff700 1 -- 172.21.15.23:0/3705486377 wait complete. 2023-06-22T08:03:48.458 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 34 2023-06-22T08:03:48.472 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:03:49.346 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir foo/.snap/ss 2023-06-22T08:03:50.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:50.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.682+0000 7f185a7ff700 1 Processor -- start 2023-06-22T08:03:50.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.682+0000 7f185a7ff700 1 -- start start 2023-06-22T08:03:50.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.683+0000 7f185a7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18540c4d00 0x7f1854159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:50.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.683+0000 7f185a7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18540c9c30 con 0x7f1854152d70 2023-06-22T08:03:50.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.683+0000 7f185a7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185415bea0 con 0x7f18540c97d0 2023-06-22T08:03:50.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.683+0000 7f185a7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18540c95e0 con 0x7f18540c4d00 2023-06-22T08:03:50.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.683+0000 7f1853fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1854152d70 0x7f18540cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41514/0 (socket says 172.21.15.23:41514) 2023-06-22T08:03:50.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.683+0000 7f1853fff700 1 -- 172.21.15.23:0/1411126838 learned_addr learned my addr 172.21.15.23:0/1411126838 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:03:50.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.684+0000 7f1858d9c700 1 --2- 172.21.15.23:0/1411126838 >> v2:172.21.15.67:3300/0 conn(0x7f18540c4d00 0x7f1854159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:50.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.684+0000 7f1852ffd700 1 -- 172.21.15.23:0/1411126838 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207126350 0 0) 0x7f18540c9c30 con 0x7f1854152d70 2023-06-22T08:03:50.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.684+0000 7f1852ffd700 1 -- 172.21.15.23:0/1411126838 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f183c0035b0 con 0x7f1854152d70 2023-06-22T08:03:50.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.684+0000 7f1852ffd700 1 -- 172.21.15.23:0/1411126838 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105633292 0 0) 0x7f185415bea0 con 0x7f18540c97d0 2023-06-22T08:03:50.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.684+0000 7f1852ffd700 1 -- 172.21.15.23:0/1411126838 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18540c9c30 con 0x7f18540c97d0 2023-06-22T08:03:50.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.684+0000 7f1858d9c700 1 -- 172.21.15.23:0/1411126838 >> v1:172.21.15.67:6789/0 conn(0x7f18540c97d0 legacy=0x7f18540c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:50.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.684+0000 7f1858d9c700 1 -- 172.21.15.23:0/1411126838 >> v1:172.21.15.23:6789/0 conn(0x7f1854152d70 legacy=0x7f18540cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:50.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.684+0000 7f1852ffd700 1 -- 172.21.15.23:0/1411126838 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 942783690 0 0) 0x7f183c0035b0 con 0x7f1854152d70 2023-06-22T08:03:50.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.684+0000 7f1858d9c700 1 -- 172.21.15.23:0/1411126838 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185415c160 con 0x7f18540c4d00 2023-06-22T08:03:50.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.684+0000 7f1858d9c700 1 --2- 172.21.15.23:0/1411126838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18540c4d00 0x7f1854159330 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f184c009d90 tx=0x7f184c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=596e483e48da3fe6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:50.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.685+0000 7f1852ffd700 1 -- 172.21.15.23:0/1411126838 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c00e070 con 0x7f18540c4d00 2023-06-22T08:03:50.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.685+0000 7f1852ffd700 1 -- 172.21.15.23:0/1411126838 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c003a20 con 0x7f18540c4d00 2023-06-22T08:03:50.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.685+0000 7f1852ffd700 1 -- 172.21.15.23:0/1411126838 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844001670 con 0x7f18540c4d00 2023-06-22T08:03:50.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.685+0000 7f185a7ff700 1 -- 172.21.15.23:0/1411126838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18540c4d00 msgr2=0x7f1854159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:50.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.685+0000 7f185a7ff700 1 --2- 172.21.15.23:0/1411126838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18540c4d00 0x7f1854159330 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f184c009d90 tx=0x7f184c00b330 comp rx=0 tx=0).stop 2023-06-22T08:03:50.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.685+0000 7f185a7ff700 1 -- 172.21.15.23:0/1411126838 shutdown_connections 2023-06-22T08:03:50.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.685+0000 7f185a7ff700 1 --2- 172.21.15.23:0/1411126838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18540c4d00 0x7f1854159330 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:50.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.685+0000 7f185a7ff700 1 -- 172.21.15.23:0/1411126838 >> 172.21.15.23:0/1411126838 conn(0x7f185406ae30 msgr2=0x7f185406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:50.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.685+0000 7f185a7ff700 1 -- 172.21.15.23:0/1411126838 shutdown_connections 2023-06-22T08:03:50.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.685+0000 7f185a7ff700 1 -- 172.21.15.23:0/1411126838 wait complete. 2023-06-22T08:03:50.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.687+0000 7f185a7ff700 1 Processor -- start 2023-06-22T08:03:50.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.687+0000 7f185a7ff700 1 -- start start 2023-06-22T08:03:50.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f185a7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18540c4d00 0x7f1854151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:50.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f185a7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18540c97d0 0x7f18541ee900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:50.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f185a7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1854152d70 0x7f18541eee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:50.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f185a7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185414da40 con 0x7f1854152d70 2023-06-22T08:03:50.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f185a7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185414dd20 con 0x7f18540c97d0 2023-06-22T08:03:50.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f185a7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185414e000 con 0x7f18540c4d00 2023-06-22T08:03:50.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f1858d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1854152d70 0x7f18541eee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:50.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f1858d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1854152d70 0x7f18541eee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46668/0 (socket says 172.21.15.23:46668) 2023-06-22T08:03:50.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f1858d9c700 1 -- 172.21.15.23:0/659324404 learned_addr learned my addr 172.21.15.23:0/659324404 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:50.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f1853fff700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18540c4d00 0x7f1854151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:50.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.688+0000 7f18537fe700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18540c97d0 0x7f18541ee900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:50.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f1858d9c700 1 -- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18540c4d00 msgr2=0x7f1854151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:50.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f1858d9c700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18540c4d00 0x7f1854151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:50.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f1858d9c700 1 -- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18540c97d0 msgr2=0x7f18541ee900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f1858d9c700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18540c97d0 0x7f18541ee900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f1858d9c700 1 -- 172.21.15.23:0/659324404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c009a70 con 0x7f1854152d70 2023-06-22T08:03:50.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f1858d9c700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1854152d70 0x7f18541eee40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f184c00b370 tx=0x7f184c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:50.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f1850ff9700 1 -- 172.21.15.23:0/659324404 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c004440 con 0x7f1854152d70 2023-06-22T08:03:50.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f1850ff9700 1 -- 172.21.15.23:0/659324404 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c0045a0 con 0x7f1854152d70 2023-06-22T08:03:50.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f1850ff9700 1 -- 172.21.15.23:0/659324404 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c004950 con 0x7f1854152d70 2023-06-22T08:03:50.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f185a7ff700 1 -- 172.21.15.23:0/659324404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185414e2e0 con 0x7f1854152d70 2023-06-22T08:03:50.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.689+0000 7f185a7ff700 1 -- 172.21.15.23:0/659324404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185414d730 con 0x7f1854152d70 2023-06-22T08:03:50.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.690+0000 7f185a7ff700 1 -- 172.21.15.23:0/659324404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18541ee620 con 0x7f1854152d70 2023-06-22T08:03:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.691+0000 7f1850ff9700 1 -- 172.21.15.23:0/659324404 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f184c01d470 con 0x7f1854152d70 2023-06-22T08:03:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.692+0000 7f1850ff9700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f184006ace0 0x7f184006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:50.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.692+0000 7f1850ff9700 1 -- 172.21.15.23:0/659324404 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 7011+0+0 (secure 0 0 0) 0x7f184c08ae10 con 0x7f1854152d70 2023-06-22T08:03:50.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.693+0000 7f1853fff700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f184006ace0 0x7f184006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:50.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.693+0000 7f1853fff700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f184006ace0 0x7f184006d190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1844000fd0 tx=0x7f1844001690 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:50.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.697+0000 7f1850ff9700 1 -- 172.21.15.23:0/659324404 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f18541ee620 con 0x7f1854152d70 2023-06-22T08:03:50.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.954+0000 7f185a7ff700 1 -- 172.21.15.23:0/659324404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18541ee620 con 0x7f1854152d70 2023-06-22T08:03:50.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.954+0000 7f1850ff9700 1 -- 172.21.15.23:0/659324404 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6269 (secure 0 0 0) 0x7f18541ee620 con 0x7f1854152d70 2023-06-22T08:03:50.958 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:50.958 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5211,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:47.314059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:active","state_seq":59,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:50.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 -- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f184006ace0 msgr2=0x7f184006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:50.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f184006ace0 0x7f184006d190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1844000fd0 tx=0x7f1844001690 comp rx=0 tx=0).stop 2023-06-22T08:03:50.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 -- 172.21.15.23:0/659324404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1854152d70 msgr2=0x7f18541eee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:50.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1854152d70 0x7f18541eee40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f184c00b370 tx=0x7f184c004ad0 comp rx=0 tx=0).stop 2023-06-22T08:03:50.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 -- 172.21.15.23:0/659324404 shutdown_connections 2023-06-22T08:03:50.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1854152d70 0x7f18541eee40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:50.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f18540c4d00 0x7f1854151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:50.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f184006ace0 0x7f184006d190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:50.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 --2- 172.21.15.23:0/659324404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18540c97d0 0x7f18541ee900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:50.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 -- 172.21.15.23:0/659324404 >> 172.21.15.23:0/659324404 conn(0x7f185406ae30 msgr2=0x7f18540caa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:50.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 -- 172.21.15.23:0/659324404 shutdown_connections 2023-06-22T08:03:50.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:50.957+0000 7f185a7ff700 1 -- 172.21.15.23:0/659324404 wait complete. 2023-06-22T08:03:50.962 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 34 2023-06-22T08:03:50.979 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:03:52.980 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:53.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.188+0000 7f6b6a7ff700 1 Processor -- start 2023-06-22T08:03:53.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.188+0000 7f6b6a7ff700 1 -- start start 2023-06-22T08:03:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.189+0000 7f6b6a7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b640c97d0 0x7f6b64159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.189+0000 7f6b6a7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6b640c45f0 0x7f6b640c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.189+0000 7f6b6a7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b640c9c30 con 0x7f6b64152d70 2023-06-22T08:03:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.189+0000 7f6b6a7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b640c95e0 con 0x7f6b640c45f0 2023-06-22T08:03:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.189+0000 7f6b6a7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b6415bea0 con 0x7f6b640c97d0 2023-06-22T08:03:53.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.189+0000 7f6b68d9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6b640c45f0 0x7f6b640c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:53.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.189+0000 7f6b63fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f6b64152d70 0x7f6b640cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41520/0 (socket says 172.21.15.23:41520) 2023-06-22T08:03:53.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.189+0000 7f6b63fff700 1 -- 172.21.15.23:0/2116873076 learned_addr learned my addr 172.21.15.23:0/2116873076 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:03:53.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.189+0000 7f6b637fe700 1 --2- 172.21.15.23:0/2116873076 >> v2:172.21.15.67:3300/0 conn(0x7f6b640c97d0 0x7f6b64159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:53.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.190+0000 7f6b68d9c700 1 -- 172.21.15.23:0/2116873076 >> v2:172.21.15.67:3300/0 conn(0x7f6b640c97d0 msgr2=0x7f6b64159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:53.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.190+0000 7f6b68d9c700 1 --2- 172.21.15.23:0/2116873076 >> v2:172.21.15.67:3300/0 conn(0x7f6b640c97d0 0x7f6b64159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:53.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.190+0000 7f6b68d9c700 1 -- 172.21.15.23:0/2116873076 >> v1:172.21.15.23:6789/0 conn(0x7f6b64152d70 legacy=0x7f6b640cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:53.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.190+0000 7f6b68d9c700 1 -- 172.21.15.23:0/2116873076 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6415c160 con 0x7f6b640c45f0 2023-06-22T08:03:53.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.190+0000 7f6b68d9c700 1 --2- 172.21.15.23:0/2116873076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c45f0 0x7f6b640c4a10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6b6408ba40 tx=0x7f6b5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e578da34ac9bae31 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:53.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.190+0000 7f6b62ffd700 1 -- 172.21.15.23:0/2116873076 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c00e070 con 0x7f6b640c45f0 2023-06-22T08:03:53.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.190+0000 7f6b62ffd700 1 -- 172.21.15.23:0/2116873076 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c00bb20 con 0x7f6b640c45f0 2023-06-22T08:03:53.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.190+0000 7f6b62ffd700 1 -- 172.21.15.23:0/2116873076 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c012430 con 0x7f6b640c45f0 2023-06-22T08:03:53.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.191+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/2116873076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c45f0 msgr2=0x7f6b640c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:53.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.191+0000 7f6b6a7ff700 1 --2- 172.21.15.23:0/2116873076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c45f0 0x7f6b640c4a10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6b6408ba40 tx=0x7f6b5c00b330 comp rx=0 tx=0).stop 2023-06-22T08:03:53.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.191+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/2116873076 shutdown_connections 2023-06-22T08:03:53.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.191+0000 7f6b6a7ff700 1 --2- 172.21.15.23:0/2116873076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c45f0 0x7f6b640c4a10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.191+0000 7f6b6a7ff700 1 --2- 172.21.15.23:0/2116873076 >> v2:172.21.15.67:3300/0 conn(0x7f6b640c97d0 0x7f6b64159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.191+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/2116873076 >> 172.21.15.23:0/2116873076 conn(0x7f6b6406ae30 msgr2=0x7f6b6406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.191+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/2116873076 shutdown_connections 2023-06-22T08:03:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.191+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/2116873076 wait complete. 2023-06-22T08:03:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.193+0000 7f6b6a7ff700 1 Processor -- start 2023-06-22T08:03:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.193+0000 7f6b6a7ff700 1 -- start start 2023-06-22T08:03:53.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b6a7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640c45f0 0x7f6b641ef300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:53.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b6a7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c97d0 0x7f6b641f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:53.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b6a7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6b64152d70 0x7f6b640b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:53.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b6a7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b641f1400 con 0x7f6b640c97d0 2023-06-22T08:03:53.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b6a7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b64157620 con 0x7f6b640c45f0 2023-06-22T08:03:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b6a7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b64158080 con 0x7f6b64152d70 2023-06-22T08:03:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b63fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640c45f0 0x7f6b641ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b637fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c97d0 0x7f6b641f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b68d9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6b64152d70 0x7f6b640b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b637fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c97d0 0x7f6b641f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46694/0 (socket says 172.21.15.23:46694) 2023-06-22T08:03:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b637fe700 1 -- 172.21.15.23:0/221919098 learned_addr learned my addr 172.21.15.23:0/221919098 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.194+0000 7f6b63fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640c45f0 0x7f6b641ef300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:55992/0 (socket says 172.21.15.23:55992) 2023-06-22T08:03:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.195+0000 7f6b637fe700 1 -- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6b64152d70 msgr2=0x7f6b640b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:53.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.195+0000 7f6b637fe700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6b64152d70 0x7f6b640b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:53.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.195+0000 7f6b637fe700 1 -- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640c45f0 msgr2=0x7f6b641ef300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:53.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.195+0000 7f6b637fe700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640c45f0 0x7f6b641ef300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:53.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.195+0000 7f6b637fe700 1 -- 172.21.15.23:0/221919098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c009a70 con 0x7f6b640c97d0 2023-06-22T08:03:53.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.195+0000 7f6b637fe700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c97d0 0x7f6b641f1e50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800e9b0 tx=0x7f6b5800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:53.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.195+0000 7f6b60ff9700 1 -- 172.21.15.23:0/221919098 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58009710 con 0x7f6b640c97d0 2023-06-22T08:03:53.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.195+0000 7f6b60ff9700 1 -- 172.21.15.23:0/221919098 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5800ee50 con 0x7f6b640c97d0 2023-06-22T08:03:53.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.195+0000 7f6b60ff9700 1 -- 172.21.15.23:0/221919098 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5800ca50 con 0x7f6b640c97d0 2023-06-22T08:03:53.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.195+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/221919098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b64158360 con 0x7f6b640c97d0 2023-06-22T08:03:53.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.196+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/221919098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b641f1650 con 0x7f6b640c97d0 2023-06-22T08:03:53.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.197+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/221919098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6404ea20 con 0x7f6b640c97d0 2023-06-22T08:03:53.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.198+0000 7f6b60ff9700 1 -- 172.21.15.23:0/221919098 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6b5800cbb0 con 0x7f6b640c97d0 2023-06-22T08:03:53.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.198+0000 7f6b60ff9700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6b5006ace0 0x7f6b5006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:53.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.199+0000 7f6b60ff9700 1 -- 172.21.15.23:0/221919098 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 7011+0+0 (secure 0 0 0) 0x7f6b580897a0 con 0x7f6b640c97d0 2023-06-22T08:03:53.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.199+0000 7f6b63fff700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6b5006ace0 0x7f6b5006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:53.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.199+0000 7f6b63fff700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6b5006ace0 0x7f6b5006d190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b54005910 tx=0x7f6b54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:53.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.203+0000 7f6b60ff9700 1 -- 172.21.15.23:0/221919098 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6b58057fb0 con 0x7f6b640c97d0 2023-06-22T08:03:53.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.455+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/221919098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b640c9e00 con 0x7f6b640c97d0 2023-06-22T08:03:53.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.456+0000 7f6b60ff9700 1 -- 172.21.15.23:0/221919098 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6269 (secure 0 0 0) 0x7f6b5800ce60 con 0x7f6b640c97d0 2023-06-22T08:03:53.460 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:53.460 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5211,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:47.314059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:active","state_seq":59,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.458+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6b5006ace0 msgr2=0x7f6b5006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.458+0000 7f6b6a7ff700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6b5006ace0 0x7f6b5006d190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b54005910 tx=0x7f6b54009040 comp rx=0 tx=0).stop 2023-06-22T08:03:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.458+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/221919098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c97d0 msgr2=0x7f6b641f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.458+0000 7f6b6a7ff700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c97d0 0x7f6b641f1e50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800e9b0 tx=0x7f6b5800ecc0 comp rx=0 tx=0).stop 2023-06-22T08:03:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.458+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/221919098 shutdown_connections 2023-06-22T08:03:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.458+0000 7f6b6a7ff700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6b64152d70 0x7f6b640b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.458+0000 7f6b6a7ff700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6b5006ace0 0x7f6b5006d190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.458+0000 7f6b6a7ff700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640c45f0 0x7f6b641ef300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.458+0000 7f6b6a7ff700 1 --2- 172.21.15.23:0/221919098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b640c97d0 0x7f6b641f1e50 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.458+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/221919098 >> 172.21.15.23:0/221919098 conn(0x7f6b6406ae30 msgr2=0x7f6b640ca8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.459+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/221919098 shutdown_connections 2023-06-22T08:03:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:53.459+0000 7f6b6a7ff700 1 -- 172.21.15.23:0/221919098 wait complete. 2023-06-22T08:03:53.464 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 34 2023-06-22T08:03:53.480 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:03:55.481 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.688+0000 7f004d1ff700 1 Processor -- start 2023-06-22T08:03:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.688+0000 7f004d1ff700 1 -- start start 2023-06-22T08:03:55.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.689+0000 7f004d1ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0048071850 0x7f00480c9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:55.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.689+0000 7f004d1ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0048071c20 0x7f00480c4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:55.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.689+0000 7f004d1ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00480c9540 con 0x7f004815c3f0 2023-06-22T08:03:55.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.689+0000 7f004d1ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0048158c10 con 0x7f0048071c20 2023-06-22T08:03:55.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.689+0000 7f004d1ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0048158ef0 con 0x7f0048071850 2023-06-22T08:03:55.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.689+0000 7f004659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0048071c20 0x7f00480c4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:55.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.689+0000 7f0046d9d700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0048071850 0x7f00480c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.690+0000 7f004659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0048071c20 0x7f00480c4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46700/0 (socket says 172.21.15.23:46700) 2023-06-22T08:03:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.690+0000 7f004659c700 1 -- 172.21.15.23:0/388499320 learned_addr learned my addr 172.21.15.23:0/388499320 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.690+0000 7f004659c700 1 -- 172.21.15.23:0/388499320 >> v2:172.21.15.67:3300/0 conn(0x7f0048071850 msgr2=0x7f00480c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.690+0000 7f004659c700 1 --2- 172.21.15.23:0/388499320 >> v2:172.21.15.67:3300/0 conn(0x7f0048071850 0x7f00480c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.690+0000 7f004659c700 1 -- 172.21.15.23:0/388499320 >> v1:172.21.15.23:6789/0 conn(0x7f004815c3f0 legacy=0x7f004815c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.690+0000 7f004659c700 1 -- 172.21.15.23:0/388499320 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00481591d0 con 0x7f0048071c20 2023-06-22T08:03:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.690+0000 7f004659c700 1 --2- 172.21.15.23:0/388499320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0048071c20 0x7f00480c4a40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f00480cc430 tx=0x7f003c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be084be0ad3ce228 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.690+0000 7f0045d9b700 1 -- 172.21.15.23:0/388499320 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c011830 con 0x7f0048071c20 2023-06-22T08:03:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.690+0000 7f0045d9b700 1 -- 172.21.15.23:0/388499320 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003c011990 con 0x7f0048071c20 2023-06-22T08:03:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.691+0000 7f0045d9b700 1 -- 172.21.15.23:0/388499320 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c011d20 con 0x7f0048071c20 2023-06-22T08:03:55.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.691+0000 7f004d1ff700 1 -- 172.21.15.23:0/388499320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0048071c20 msgr2=0x7f00480c4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.691+0000 7f004d1ff700 1 --2- 172.21.15.23:0/388499320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0048071c20 0x7f00480c4a40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f00480cc430 tx=0x7f003c009bd0 comp rx=0 tx=0).stop 2023-06-22T08:03:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.691+0000 7f004d1ff700 1 -- 172.21.15.23:0/388499320 shutdown_connections 2023-06-22T08:03:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.691+0000 7f004d1ff700 1 --2- 172.21.15.23:0/388499320 >> v2:172.21.15.67:3300/0 conn(0x7f0048071850 0x7f00480c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.691+0000 7f004d1ff700 1 --2- 172.21.15.23:0/388499320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0048071c20 0x7f00480c4a40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.691+0000 7f004d1ff700 1 -- 172.21.15.23:0/388499320 >> 172.21.15.23:0/388499320 conn(0x7f004806b890 msgr2=0x7f004806bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.691+0000 7f004d1ff700 1 -- 172.21.15.23:0/388499320 shutdown_connections 2023-06-22T08:03:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.691+0000 7f004d1ff700 1 -- 172.21.15.23:0/388499320 wait complete. 2023-06-22T08:03:55.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.693+0000 7f004d1ff700 1 Processor -- start 2023-06-22T08:03:55.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.693+0000 7f004d1ff700 1 -- start start 2023-06-22T08:03:55.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f004d1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0048071850 0x7f0048159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:55.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f004d1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048071c20 0x7f00480b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:55.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f004d1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004815c3f0 0x7f00481f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:55.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f004d1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00481f1f30 con 0x7f0048071850 2023-06-22T08:03:55.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f004d1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00480cac50 con 0x7f0048071c20 2023-06-22T08:03:55.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f004d1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00481f1900 con 0x7f004815c3f0 2023-06-22T08:03:55.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f0046d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0048071850 0x7f0048159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:55.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f004659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048071c20 0x7f00480b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:55.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f004659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048071c20 0x7f00480b1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:56022/0 (socket says 172.21.15.23:56022) 2023-06-22T08:03:55.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f004659c700 1 -- 172.21.15.23:0/703342534 learned_addr learned my addr 172.21.15.23:0/703342534 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:55.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.694+0000 7f004759e700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004815c3f0 0x7f00481f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:55.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.695+0000 7f004659c700 1 -- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004815c3f0 msgr2=0x7f00481f2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:55.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.695+0000 7f004659c700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004815c3f0 0x7f00481f2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:55.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.695+0000 7f004659c700 1 -- 172.21.15.23:0/703342534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0048071850 msgr2=0x7f0048159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:55.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.695+0000 7f004659c700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0048071850 0x7f0048159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.695+0000 7f004659c700 1 -- 172.21.15.23:0/703342534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c009550 con 0x7f0048071c20 2023-06-22T08:03:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.695+0000 7f004659c700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048071c20 0x7f00480b1150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f003c000c00 tx=0x7f003c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.695+0000 7f002f7fe700 1 -- 172.21.15.23:0/703342534 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c004380 con 0x7f0048071c20 2023-06-22T08:03:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.695+0000 7f002f7fe700 1 -- 172.21.15.23:0/703342534 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003c01f070 con 0x7f0048071c20 2023-06-22T08:03:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.695+0000 7f002f7fe700 1 -- 172.21.15.23:0/703342534 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c0036a0 con 0x7f0048071c20 2023-06-22T08:03:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.695+0000 7f004d1ff700 1 -- 172.21.15.23:0/703342534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00481ef1b0 con 0x7f0048071c20 2023-06-22T08:03:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.696+0000 7f004d1ff700 1 -- 172.21.15.23:0/703342534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f004815ba90 con 0x7f0048071c20 2023-06-22T08:03:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.696+0000 7f004d1ff700 1 -- 172.21.15.23:0/703342534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00481f2070 con 0x7f0048071c20 2023-06-22T08:03:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.698+0000 7f002f7fe700 1 -- 172.21.15.23:0/703342534 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f003c021030 con 0x7f0048071c20 2023-06-22T08:03:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.698+0000 7f002f7fe700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f003406ad10 0x7f003406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.699+0000 7f002f7fe700 1 -- 172.21.15.23:0/703342534 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6955+0+0 (secure 0 0 0) 0x7f003c08a250 con 0x7f0048071c20 2023-06-22T08:03:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.699+0000 7f0046d9d700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f003406ad10 0x7f003406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.699+0000 7f0046d9d700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f003406ad10 0x7f003406d1c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0038009be0 tx=0x7f0038009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:55.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.702+0000 7f002f7fe700 1 -- 172.21.15.23:0/703342534 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f003c054ee0 con 0x7f0048071c20 2023-06-22T08:03:55.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.952+0000 7f004d1ff700 1 -- 172.21.15.23:0/703342534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00481f2070 con 0x7f0048071c20 2023-06-22T08:03:55.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.954+0000 7f002f7fe700 1 -- 172.21.15.23:0/703342534 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6269 (secure 0 0 0) 0x7f00481f2070 con 0x7f0048071c20 2023-06-22T08:03:55.957 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:55.957 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5211,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:47.314059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:active","state_seq":59,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:55.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.955+0000 7f004d1ff700 1 -- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f003406ad10 msgr2=0x7f003406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:55.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.955+0000 7f004d1ff700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f003406ad10 0x7f003406d1c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0038009be0 tx=0x7f0038009210 comp rx=0 tx=0).stop 2023-06-22T08:03:55.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.955+0000 7f004d1ff700 1 -- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048071c20 msgr2=0x7f00480b1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:55.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.955+0000 7f004d1ff700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048071c20 0x7f00480b1150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f003c000c00 tx=0x7f003c004750 comp rx=0 tx=0).stop 2023-06-22T08:03:55.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.955+0000 7f004d1ff700 1 -- 172.21.15.23:0/703342534 shutdown_connections 2023-06-22T08:03:55.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.955+0000 7f004d1ff700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004815c3f0 0x7f00481f2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:55.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.955+0000 7f004d1ff700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0048071850 0x7f0048159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:55.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.955+0000 7f004d1ff700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f003406ad10 0x7f003406d1c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:55.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.955+0000 7f004d1ff700 1 --2- 172.21.15.23:0/703342534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0048071c20 0x7f00480b1150 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:55.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.955+0000 7f004d1ff700 1 -- 172.21.15.23:0/703342534 >> 172.21.15.23:0/703342534 conn(0x7f004806b890 msgr2=0x7f0048151940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:55.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.956+0000 7f004d1ff700 1 -- 172.21.15.23:0/703342534 shutdown_connections 2023-06-22T08:03:55.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:55.956+0000 7f004d1ff700 1 -- 172.21.15.23:0/703342534 wait complete. 2023-06-22T08:03:55.961 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 34 2023-06-22T08:03:55.977 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:03:57.978 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.185+0000 7f3cfbdff700 1 Processor -- start 2023-06-22T08:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.185+0000 7f3cfbdff700 1 -- start start 2023-06-22T08:03:58.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.186+0000 7f3cfbdff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3cf414f850 0x7f3cf40c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:58.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.186+0000 7f3cfbdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf40c69e0 0x7f3cf40c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:58.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.186+0000 7f3cfbdff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf40c51a0 con 0x7f3cf4071070 2023-06-22T08:03:58.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.186+0000 7f3cfbdff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf40c44e0 con 0x7f3cf414f850 2023-06-22T08:03:58.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.186+0000 7f3cfbdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf40c5440 con 0x7f3cf40c69e0 2023-06-22T08:03:58.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.187+0000 7f3cf9b9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3cf414f850 0x7f3cf40c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:58.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.187+0000 7f3cf9b9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3cf414f850 0x7f3cf40c3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47424/0 (socket says 172.21.15.23:47424) 2023-06-22T08:03:58.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.187+0000 7f3cf9b9b700 1 -- 172.21.15.23:0/225181 learned_addr learned my addr 172.21.15.23:0/225181 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:58.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.187+0000 7f3cf939a700 1 --2- 172.21.15.23:0/225181 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf40c69e0 0x7f3cf40c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.187+0000 7f3cf9b9b700 1 -- 172.21.15.23:0/225181 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf40c69e0 msgr2=0x7f3cf40c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.187+0000 7f3cf9b9b700 1 --2- 172.21.15.23:0/225181 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf40c69e0 0x7f3cf40c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.187+0000 7f3cf9b9b700 1 -- 172.21.15.23:0/225181 >> v1:172.21.15.67:6789/0 conn(0x7f3cf4071070 legacy=0x7f3cf4155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.187+0000 7f3cf9b9b700 1 -- 172.21.15.23:0/225181 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf40c5720 con 0x7f3cf414f850 2023-06-22T08:03:58.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.187+0000 7f3cf9b9b700 1 --2- 172.21.15.23:0/225181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf414f850 0x7f3cf40c3080 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3cf00098f0 tx=0x7f3cf0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e56b74085134c4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:58.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.187+0000 7f3cf8b99700 1 -- 172.21.15.23:0/225181 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0011830 con 0x7f3cf414f850 2023-06-22T08:03:58.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.188+0000 7f3cf8b99700 1 -- 172.21.15.23:0/225181 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf0011990 con 0x7f3cf414f850 2023-06-22T08:03:58.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.188+0000 7f3cf8b99700 1 -- 172.21.15.23:0/225181 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0011d20 con 0x7f3cf414f850 2023-06-22T08:03:58.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.188+0000 7f3cfbdff700 1 -- 172.21.15.23:0/225181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf414f850 msgr2=0x7f3cf40c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:58.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.188+0000 7f3cfbdff700 1 --2- 172.21.15.23:0/225181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf414f850 0x7f3cf40c3080 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3cf00098f0 tx=0x7f3cf0009c00 comp rx=0 tx=0).stop 2023-06-22T08:03:58.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.188+0000 7f3cfbdff700 1 -- 172.21.15.23:0/225181 shutdown_connections 2023-06-22T08:03:58.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.188+0000 7f3cfbdff700 1 --2- 172.21.15.23:0/225181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf414f850 0x7f3cf40c3080 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:58.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.188+0000 7f3cfbdff700 1 --2- 172.21.15.23:0/225181 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf40c69e0 0x7f3cf40c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:58.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.188+0000 7f3cfbdff700 1 -- 172.21.15.23:0/225181 >> 172.21.15.23:0/225181 conn(0x7f3cf40ca3f0 msgr2=0x7f3cf40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:58.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.188+0000 7f3cfbdff700 1 -- 172.21.15.23:0/225181 shutdown_connections 2023-06-22T08:03:58.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.188+0000 7f3cfbdff700 1 -- 172.21.15.23:0/225181 wait complete. 2023-06-22T08:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.190+0000 7f3cfbdff700 1 Processor -- start 2023-06-22T08:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.190+0000 7f3cfbdff700 1 -- start start 2023-06-22T08:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cfbdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf4071070 0x7f3cf41eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cfbdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf40c69e0 0x7f3cf41ef050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cfbdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf414f850 0x7f3cf41427c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:58.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cfbdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4151130 con 0x7f3cf40c69e0 2023-06-22T08:03:58.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cfbdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4152cc0 con 0x7f3cf4071070 2023-06-22T08:03:58.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cfbdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf41ee610 con 0x7f3cf414f850 2023-06-22T08:03:58.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cf939a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf40c69e0 0x7f3cf41ef050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:58.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cf939a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf40c69e0 0x7f3cf41ef050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47438/0 (socket says 172.21.15.23:47438) 2023-06-22T08:03:58.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cf939a700 1 -- 172.21.15.23:0/298903984 learned_addr learned my addr 172.21.15.23:0/298903984 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:03:58.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cf9b9b700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf4071070 0x7f3cf41eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:58.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.191+0000 7f3cfa39c700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf414f850 0x7f3cf41427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:58.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cf939a700 1 -- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf414f850 msgr2=0x7f3cf41427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:58.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cf939a700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf414f850 0x7f3cf41427c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:58.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cf939a700 1 -- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf4071070 msgr2=0x7f3cf41eeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:58.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cf939a700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf4071070 0x7f3cf41eeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:58.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cf939a700 1 -- 172.21.15.23:0/298903984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0009550 con 0x7f3cf40c69e0 2023-06-22T08:03:58.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cfa39c700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf414f850 0x7f3cf41427c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:03:58.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cf939a700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf40c69e0 0x7f3cf41ef050 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400e980 tx=0x7f3ce400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:58.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cea7fc700 1 -- 172.21.15.23:0/298903984 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce4009710 con 0x7f3cf40c69e0 2023-06-22T08:03:58.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cea7fc700 1 -- 172.21.15.23:0/298903984 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce400eed0 con 0x7f3cf40c69e0 2023-06-22T08:03:58.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cea7fc700 1 -- 172.21.15.23:0/298903984 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce400ca90 con 0x7f3cf40c69e0 2023-06-22T08:03:58.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cfbdff700 1 -- 172.21.15.23:0/298903984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf4152f60 con 0x7f3cf40c69e0 2023-06-22T08:03:58.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.192+0000 7f3cfbdff700 1 -- 172.21.15.23:0/298903984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf41502a0 con 0x7f3cf40c69e0 2023-06-22T08:03:58.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.193+0000 7f3cfbdff700 1 -- 172.21.15.23:0/298903984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf4066900 con 0x7f3cf40c69e0 2023-06-22T08:03:58.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.194+0000 7f3cea7fc700 1 -- 172.21.15.23:0/298903984 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3ce400cbf0 con 0x7f3cf40c69e0 2023-06-22T08:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.195+0000 7f3cea7fc700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cd806ace0 0x7f3cd806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.195+0000 7f3cea7fc700 1 -- 172.21.15.23:0/298903984 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6955+0+0 (secure 0 0 0) 0x7f3ce4089a00 con 0x7f3cf40c69e0 2023-06-22T08:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.196+0000 7f3cf9b9b700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cd806ace0 0x7f3cd806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.196+0000 7f3cf9b9b700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cd806ace0 0x7f3cd806d190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0014040 tx=0x7f3cf0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.200+0000 7f3cea7fc700 1 -- 172.21.15.23:0/298903984 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3ce4054600 con 0x7f3cf40c69e0 2023-06-22T08:03:58.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.447+0000 7f3cfbdff700 1 -- 172.21.15.23:0/298903984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cf40c2950 con 0x7f3cf40c69e0 2023-06-22T08:03:58.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.448+0000 7f3cea7fc700 1 -- 172.21.15.23:0/298903984 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+6269 (secure 0 0 0) 0x7f3ce4017090 con 0x7f3cf40c69e0 2023-06-22T08:03:58.451 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:03:58.451 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5211,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:47.314059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"d","rank":4,"incarnation":27,"state":"up:active","state_seq":59,"addr":"172.21.15.67:6837/3798178226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":3798178226},{"type":"v1","addr":"172.21.15.67:6837","nonce":3798178226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:03:58.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 -- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cd806ace0 msgr2=0x7f3cd806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:58.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cd806ace0 0x7f3cd806d190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0014040 tx=0x7f3cf0008030 comp rx=0 tx=0).stop 2023-06-22T08:03:58.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 -- 172.21.15.23:0/298903984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf40c69e0 msgr2=0x7f3cf41ef050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:03:58.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf40c69e0 0x7f3cf41ef050 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3ce400e980 tx=0x7f3ce400ec90 comp rx=0 tx=0).stop 2023-06-22T08:03:58.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 -- 172.21.15.23:0/298903984 shutdown_connections 2023-06-22T08:03:58.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3cf414f850 0x7f3cf41427c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:58.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3cd806ace0 0x7f3cd806d190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:58.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3cf4071070 0x7f3cf41eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:58.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 --2- 172.21.15.23:0/298903984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3cf40c69e0 0x7f3cf41ef050 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:03:58.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 -- 172.21.15.23:0/298903984 >> 172.21.15.23:0/298903984 conn(0x7f3cf40ca3f0 msgr2=0x7f3cf406fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:03:58.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 -- 172.21.15.23:0/298903984 shutdown_connections 2023-06-22T08:03:58.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:03:58.450+0000 7f3cfbdff700 1 -- 172.21.15.23:0/298903984 wait complete. 2023-06-22T08:03:58.455 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 34 2023-06-22T08:03:58.472 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:03:59.318 INFO:tasks.workunit.client.0.smithi023.stderr:+ ln foo/x foo/xx 2023-06-22T08:03:59.333 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat foo/.snap/ss/x 2023-06-22T08:04:00.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:00.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.681+0000 7f08e99ff700 1 Processor -- start 2023-06-22T08:04:00.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.682+0000 7f08e99ff700 1 -- start start 2023-06-22T08:04:00.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08e99ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08e414f850 0x7f08e40c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:00.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08e99ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08e40c69e0 0x7f08e40c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:00.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08e99ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 0x7f08e4155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08e99ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e40c51a0 con 0x7f08e414f850 2023-06-22T08:04:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08e99ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e40c44e0 con 0x7f08e40c69e0 2023-06-22T08:04:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08e99ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e40c5440 con 0x7f08e4071070 2023-06-22T08:04:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08e2ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08e414f850 0x7f08e40c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08dbfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08e40c69e0 0x7f08e40c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08e37fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 0x7f08e4155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08dbfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08e40c69e0 0x7f08e40c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:34368/0 (socket says 172.21.15.23:34368) 2023-06-22T08:04:00.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.683+0000 7f08dbfff700 1 -- 172.21.15.23:0/698036721 learned_addr learned my addr 172.21.15.23:0/698036721 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:00.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.684+0000 7f08dbfff700 1 -- 172.21.15.23:0/698036721 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 msgr2=0x7f08e4155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:00.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.684+0000 7f08dbfff700 1 --2- 172.21.15.23:0/698036721 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 0x7f08e4155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.684+0000 7f08dbfff700 1 -- 172.21.15.23:0/698036721 >> v2:172.21.15.23:3300/0 conn(0x7f08e414f850 msgr2=0x7f08e40c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:00.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.684+0000 7f08dbfff700 1 --2- 172.21.15.23:0/698036721 >> v2:172.21.15.23:3300/0 conn(0x7f08e414f850 0x7f08e40c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.684+0000 7f08dbfff700 1 -- 172.21.15.23:0/698036721 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e40c5720 con 0x7f08e40c69e0 2023-06-22T08:04:00.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.684+0000 7f08e37fe700 1 --2- 172.21.15.23:0/698036721 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 0x7f08e4155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:04:00.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.684+0000 7f08dbfff700 1 --2- 172.21.15.23:0/698036721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40c69e0 0x7f08e40c35c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f08e406f9a0 tx=0x7f08cc00be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72343593ba83f62c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:00.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.684+0000 7f08e27fc700 1 -- 172.21.15.23:0/698036721 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc00dc70 con 0x7f08e40c69e0 2023-06-22T08:04:00.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.684+0000 7f08e27fc700 1 -- 172.21.15.23:0/698036721 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08cc00ddd0 con 0x7f08e40c69e0 2023-06-22T08:04:00.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.684+0000 7f08e27fc700 1 -- 172.21.15.23:0/698036721 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc010690 con 0x7f08e40c69e0 2023-06-22T08:04:00.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.685+0000 7f08e99ff700 1 -- 172.21.15.23:0/698036721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40c69e0 msgr2=0x7f08e40c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:00.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.685+0000 7f08e99ff700 1 --2- 172.21.15.23:0/698036721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40c69e0 0x7f08e40c35c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f08e406f9a0 tx=0x7f08cc00be70 comp rx=0 tx=0).stop 2023-06-22T08:04:00.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.685+0000 7f08e99ff700 1 -- 172.21.15.23:0/698036721 shutdown_connections 2023-06-22T08:04:00.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.685+0000 7f08e99ff700 1 --2- 172.21.15.23:0/698036721 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 0x7f08e4155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.685+0000 7f08e99ff700 1 --2- 172.21.15.23:0/698036721 >> v2:172.21.15.23:3300/0 conn(0x7f08e414f850 0x7f08e40c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.685+0000 7f08e99ff700 1 --2- 172.21.15.23:0/698036721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40c69e0 0x7f08e40c35c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.685+0000 7f08e99ff700 1 -- 172.21.15.23:0/698036721 >> 172.21.15.23:0/698036721 conn(0x7f08e40ca3f0 msgr2=0x7f08e40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:00.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.685+0000 7f08e99ff700 1 -- 172.21.15.23:0/698036721 shutdown_connections 2023-06-22T08:04:00.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.685+0000 7f08e99ff700 1 -- 172.21.15.23:0/698036721 wait complete. 2023-06-22T08:04:00.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.687+0000 7f08e99ff700 1 Processor -- start 2023-06-22T08:04:00.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.687+0000 7f08e99ff700 1 -- start start 2023-06-22T08:04:00.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08e99ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 0x7f08e41efb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:00.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08e99ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40c69e0 0x7f08e41f1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:00.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08e99ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e414f850 0x7f08e41f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:00.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08e99ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e40ca260 con 0x7f08e414f850 2023-06-22T08:04:00.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08e99ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e40c8730 con 0x7f08e40c69e0 2023-06-22T08:04:00.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08e99ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e40c9cb0 con 0x7f08e4071070 2023-06-22T08:04:00.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08e37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e414f850 0x7f08e41f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:00.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08e37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e414f850 0x7f08e41f23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47456/0 (socket says 172.21.15.23:47456) 2023-06-22T08:04:00.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08e37fe700 1 -- 172.21.15.23:0/4073121524 learned_addr learned my addr 172.21.15.23:0/4073121524 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:00.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08dbfff700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40c69e0 0x7f08e41f1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:00.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.688+0000 7f08e2ffd700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 0x7f08e41efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:00.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e37fe700 1 -- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 msgr2=0x7f08e41efb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:00.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e37fe700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 0x7f08e41efb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e37fe700 1 -- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40c69e0 msgr2=0x7f08e41f1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:00.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e37fe700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40c69e0 0x7f08e41f1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e37fe700 1 -- 172.21.15.23:0/4073121524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc009a70 con 0x7f08e414f850 2023-06-22T08:04:00.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08dbfff700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40c69e0 0x7f08e41f1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:04:00.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e37fe700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e414f850 0x7f08e41f23d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00ef00 tx=0x7f08dc00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:00.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e0ff9700 1 -- 172.21.15.23:0/4073121524 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc010040 con 0x7f08e414f850 2023-06-22T08:04:00.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e0ff9700 1 -- 172.21.15.23:0/4073121524 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08dc00ba80 con 0x7f08e414f850 2023-06-22T08:04:00.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e0ff9700 1 -- 172.21.15.23:0/4073121524 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc00be10 con 0x7f08e414f850 2023-06-22T08:04:00.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e99ff700 1 -- 172.21.15.23:0/4073121524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc00b7f0 con 0x7f08e414f850 2023-06-22T08:04:00.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.689+0000 7f08e99ff700 1 -- 172.21.15.23:0/4073121524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e4149ac0 con 0x7f08e414f850 2023-06-22T08:04:00.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.690+0000 7f08e99ff700 1 -- 172.21.15.23:0/4073121524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e40c2950 con 0x7f08e414f850 2023-06-22T08:04:00.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.691+0000 7f08e0ff9700 1 -- 172.21.15.23:0/4073121524 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f08dc01d410 con 0x7f08e414f850 2023-06-22T08:04:00.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.692+0000 7f08e0ff9700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f08c406ad10 0x7f08c406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:00.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.692+0000 7f08e0ff9700 1 -- 172.21.15.23:0/4073121524 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6985+0+0 (secure 0 0 0) 0x7f08dc089ac0 con 0x7f08e414f850 2023-06-22T08:04:00.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.693+0000 7f08e2ffd700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f08c406ad10 0x7f08c406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:00.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.693+0000 7f08e2ffd700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f08c406ad10 0x7f08c406d1c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009b40 tx=0x7f08d4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:00.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.697+0000 7f08e0ff9700 1 -- 172.21.15.23:0/4073121524 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f08dc054750 con 0x7f08e414f850 2023-06-22T08:04:00.719 INFO:tasks.workunit.client.0.smithi023.stdout:x 2023-06-22T08:04:00.720 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir foo/.snap/ss 2023-06-22T08:04:00.728 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo a 2023-06-22T08:04:00.730 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo b 2023-06-22T08:04:00.730 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir foo/.snap/s 2023-06-22T08:04:00.741 INFO:tasks.workunit.client.0.smithi023.stderr:++ cat foo/.snap/s/a 2023-06-22T08:04:00.750 INFO:tasks.workunit.client.0.smithi023.stderr:+ r=a 2023-06-22T08:04:00.751 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' -z a ']' 2023-06-22T08:04:00.751 INFO:tasks.workunit.client.0.smithi023.stderr:+ ln foo/b foo/b2 2023-06-22T08:04:00.754 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat foo/.snap/s/b 2023-06-22T08:04:00.763 INFO:tasks.workunit.client.0.smithi023.stdout:b 2023-06-22T08:04:00.763 INFO:tasks.workunit.client.0.smithi023.stdout:this used to hang: 2023-06-22T08:04:00.763 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'this used to hang:' 2023-06-22T08:04:00.764 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo more 2023-06-22T08:04:00.765 INFO:tasks.workunit.client.0.smithi023.stdout:oh, it didn't hang! good job. 2023-06-22T08:04:00.765 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2023-06-22T08:04:00.765 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat foo/b 2023-06-22T08:04:00.765 INFO:tasks.workunit.client.0.smithi023.stdout:b 2023-06-22T08:04:00.766 INFO:tasks.workunit.client.0.smithi023.stdout:more 2023-06-22T08:04:00.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir foo/.snap/s 2023-06-22T08:04:00.772 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -r foo 2023-06-22T08:04:00.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.962+0000 7f08e99ff700 1 -- 172.21.15.23:0/4073121524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08e414fc20 con 0x7f08e414f850 2023-06-22T08:04:00.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.963+0000 7f08e0ff9700 1 -- 172.21.15.23:0/4073121524 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5512 (secure 0 0 0) 0x7f08dc057d70 con 0x7f08e414f850 2023-06-22T08:04:00.967 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:00.967 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:59.300212+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:00.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.965+0000 7f08e99ff700 1 -- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f08c406ad10 msgr2=0x7f08c406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:00.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.965+0000 7f08e99ff700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f08c406ad10 0x7f08c406d1c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009b40 tx=0x7f08d4009210 comp rx=0 tx=0).stop 2023-06-22T08:04:00.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.965+0000 7f08e99ff700 1 -- 172.21.15.23:0/4073121524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e414f850 msgr2=0x7f08e41f23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:00.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.965+0000 7f08e99ff700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e414f850 0x7f08e41f23d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00ef00 tx=0x7f08dc00b390 comp rx=0 tx=0).stop 2023-06-22T08:04:00.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.966+0000 7f08e99ff700 1 -- 172.21.15.23:0/4073121524 shutdown_connections 2023-06-22T08:04:00.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.966+0000 7f08e99ff700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08e414f850 0x7f08e41f23d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.966+0000 7f08e99ff700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f08e4071070 0x7f08e41efb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.966+0000 7f08e99ff700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f08c406ad10 0x7f08c406d1c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.966+0000 7f08e99ff700 1 --2- 172.21.15.23:0/4073121524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08e40c69e0 0x7f08e41f1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:00.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.966+0000 7f08e99ff700 1 -- 172.21.15.23:0/4073121524 >> 172.21.15.23:0/4073121524 conn(0x7f08e40ca3f0 msgr2=0x7f08e414d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:00.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.966+0000 7f08e99ff700 1 -- 172.21.15.23:0/4073121524 shutdown_connections 2023-06-22T08:04:00.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:00.966+0000 7f08e99ff700 1 -- 172.21.15.23:0/4073121524 wait complete. 2023-06-22T08:04:00.971 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 35 2023-06-22T08:04:00.987 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-06-22T08:04:00.987 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:04:00.988 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:01.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.183+0000 7fdab05ff700 1 Processor -- start 2023-06-22T08:04:01.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.183+0000 7fdab05ff700 1 -- start start 2023-06-22T08:04:01.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdab05ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdaa8152ce0 0x7fdaa80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:01.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdab05ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdaa80c9fd0 0x7fdaa815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:01.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdab05ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa815c370 con 0x7fdaa8158880 2023-06-22T08:04:01.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdab05ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa81530b0 con 0x7fdaa8152ce0 2023-06-22T08:04:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdab05ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa80cf6f0 con 0x7fdaa80c9fd0 2023-06-22T08:04:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdaae39b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdaa8152ce0 0x7fdaa80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdaae39b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdaa8152ce0 0x7fdaa80cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47460/0 (socket says 172.21.15.23:47460) 2023-06-22T08:04:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdaae39b700 1 -- 172.21.15.23:0/2328004108 learned_addr learned my addr 172.21.15.23:0/2328004108 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:01.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdaadb9a700 1 --2- 172.21.15.23:0/2328004108 >> v2:172.21.15.67:3300/0 conn(0x7fdaa80c9fd0 0x7fdaa815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:01.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdaae39b700 1 -- 172.21.15.23:0/2328004108 >> v2:172.21.15.67:3300/0 conn(0x7fdaa80c9fd0 msgr2=0x7fdaa815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:01.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdaae39b700 1 --2- 172.21.15.23:0/2328004108 >> v2:172.21.15.67:3300/0 conn(0x7fdaa80c9fd0 0x7fdaa815c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:01.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdaae39b700 1 -- 172.21.15.23:0/2328004108 >> v1:172.21.15.23:6789/0 conn(0x7fdaa8158880 legacy=0x7fdaa8158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:01.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.185+0000 7fdaae39b700 1 -- 172.21.15.23:0/2328004108 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa815bda0 con 0x7fdaa8152ce0 2023-06-22T08:04:01.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.186+0000 7fdaae39b700 1 --2- 172.21.15.23:0/2328004108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdaa8152ce0 0x7fdaa80cf1b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80cc380 tx=0x7fdaa4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17888577a9ff99e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:01.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.186+0000 7fdaad399700 1 -- 172.21.15.23:0/2328004108 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4011830 con 0x7fdaa8152ce0 2023-06-22T08:04:01.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.186+0000 7fdaad399700 1 -- 172.21.15.23:0/2328004108 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa4011990 con 0x7fdaa8152ce0 2023-06-22T08:04:01.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.186+0000 7fdaad399700 1 -- 172.21.15.23:0/2328004108 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4011d20 con 0x7fdaa8152ce0 2023-06-22T08:04:01.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.186+0000 7fdab05ff700 1 -- 172.21.15.23:0/2328004108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdaa8152ce0 msgr2=0x7fdaa80cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:01.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.186+0000 7fdab05ff700 1 --2- 172.21.15.23:0/2328004108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdaa8152ce0 0x7fdaa80cf1b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80cc380 tx=0x7fdaa4009bd0 comp rx=0 tx=0).stop 2023-06-22T08:04:01.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.187+0000 7fdab05ff700 1 -- 172.21.15.23:0/2328004108 shutdown_connections 2023-06-22T08:04:01.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.187+0000 7fdab05ff700 1 --2- 172.21.15.23:0/2328004108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdaa8152ce0 0x7fdaa80cf1b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:01.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.187+0000 7fdab05ff700 1 --2- 172.21.15.23:0/2328004108 >> v2:172.21.15.67:3300/0 conn(0x7fdaa80c9fd0 0x7fdaa815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:01.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.187+0000 7fdab05ff700 1 -- 172.21.15.23:0/2328004108 >> 172.21.15.23:0/2328004108 conn(0x7fdaa806ae30 msgr2=0x7fdaa806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:01.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.187+0000 7fdab05ff700 1 -- 172.21.15.23:0/2328004108 shutdown_connections 2023-06-22T08:04:01.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.187+0000 7fdab05ff700 1 -- 172.21.15.23:0/2328004108 wait complete. 2023-06-22T08:04:01.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.188+0000 7fdab05ff700 1 Processor -- start 2023-06-22T08:04:01.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.189+0000 7fdab05ff700 1 -- start start 2023-06-22T08:04:01.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.189+0000 7fdab05ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa80c9fd0 0x7fdaa81eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:01.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.189+0000 7fdab05ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdaa8152ce0 0x7fdaa80ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:01.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.189+0000 7fdab05ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdaa8158880 0x7fdaa815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:01.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.189+0000 7fdab05ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa80ce1c0 con 0x7fdaa8158880 2023-06-22T08:04:01.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.189+0000 7fdab05ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa81ef010 con 0x7fdaa80c9fd0 2023-06-22T08:04:01.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.189+0000 7fdab05ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa80cc9c0 con 0x7fdaa8152ce0 2023-06-22T08:04:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaadb9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdaa8152ce0 0x7fdaa80ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaaeb9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdaa8158880 0x7fdaa815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaae39b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa80c9fd0 0x7fdaa81eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaadb9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdaa8152ce0 0x7fdaa80ce5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:53210/0 (socket says 172.21.15.23:53210) 2023-06-22T08:04:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaadb9a700 1 -- 172.21.15.23:0/3356665118 learned_addr learned my addr 172.21.15.23:0/3356665118 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:01.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaaeb9c700 1 -- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdaa8152ce0 msgr2=0x7fdaa80ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaaeb9c700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdaa8152ce0 0x7fdaa80ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaaeb9c700 1 -- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa80c9fd0 msgr2=0x7fdaa81eead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaaeb9c700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa80c9fd0 0x7fdaa81eead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaaeb9c700 1 -- 172.21.15.23:0/3356665118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa4009550 con 0x7fdaa8158880 2023-06-22T08:04:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.190+0000 7fdaaeb9c700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdaa8158880 0x7fdaa815aeb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000cb10 tx=0x7fdaa000ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.191+0000 7fda9effd700 1 -- 172.21.15.23:0/3356665118 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0003c00 con 0x7fdaa8158880 2023-06-22T08:04:01.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.191+0000 7fda9effd700 1 -- 172.21.15.23:0/3356665118 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa0003d60 con 0x7fdaa8158880 2023-06-22T08:04:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.191+0000 7fda9effd700 1 -- 172.21.15.23:0/3356665118 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa000a9e0 con 0x7fdaa8158880 2023-06-22T08:04:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.191+0000 7fdab05ff700 1 -- 172.21.15.23:0/3356665118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa80ccc10 con 0x7fdaa8158880 2023-06-22T08:04:01.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.191+0000 7fdab05ff700 1 -- 172.21.15.23:0/3356665118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa80cd1d0 con 0x7fdaa8158880 2023-06-22T08:04:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.192+0000 7fdab05ff700 1 -- 172.21.15.23:0/3356665118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa80ce300 con 0x7fdaa8158880 2023-06-22T08:04:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.193+0000 7fda9effd700 1 -- 172.21.15.23:0/3356665118 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdaa000ab40 con 0x7fdaa8158880 2023-06-22T08:04:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.194+0000 7fda9effd700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fda8c06ad10 0x7fda8c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.195+0000 7fda9effd700 1 -- 172.21.15.23:0/3356665118 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6985+0+0 (secure 0 0 0) 0x7fdaa0089690 con 0x7fdaa8158880 2023-06-22T08:04:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.195+0000 7fdaae39b700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fda8c06ad10 0x7fda8c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:01.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.195+0000 7fdaae39b700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fda8c06ad10 0x7fda8c06d1c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4000c00 tx=0x7fdaa4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:01.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.199+0000 7fda9effd700 1 -- 172.21.15.23:0/3356665118 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdaa80ce300 con 0x7fdaa8158880 2023-06-22T08:04:01.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.449+0000 7fdab05ff700 1 -- 172.21.15.23:0/3356665118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaa80ce300 con 0x7fdaa8158880 2023-06-22T08:04:01.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.449+0000 7fda9effd700 1 -- 172.21.15.23:0/3356665118 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5512 (secure 0 0 0) 0x7fdaa80ce300 con 0x7fdaa8158880 2023-06-22T08:04:01.453 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:01.453 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:59.300212+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:01.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 -- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fda8c06ad10 msgr2=0x7fda8c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:01.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fda8c06ad10 0x7fda8c06d1c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4000c00 tx=0x7fdaa4008030 comp rx=0 tx=0).stop 2023-06-22T08:04:01.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 -- 172.21.15.23:0/3356665118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdaa8158880 msgr2=0x7fdaa815aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:01.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdaa8158880 0x7fdaa815aeb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000cb10 tx=0x7fdaa000ce20 comp rx=0 tx=0).stop 2023-06-22T08:04:01.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 -- 172.21.15.23:0/3356665118 shutdown_connections 2023-06-22T08:04:01.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdaa8158880 0x7fdaa815aeb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:01.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fda8c06ad10 0x7fda8c06d1c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:01.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdaa80c9fd0 0x7fdaa81eead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:01.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 --2- 172.21.15.23:0/3356665118 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdaa8152ce0 0x7fdaa80ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:01.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 -- 172.21.15.23:0/3356665118 >> 172.21.15.23:0/3356665118 conn(0x7fdaa806ae30 msgr2=0x7fdaa80cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:01.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 -- 172.21.15.23:0/3356665118 shutdown_connections 2023-06-22T08:04:01.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:01.452+0000 7fdab05ff700 1 -- 172.21.15.23:0/3356665118 wait complete. 2023-06-22T08:04:01.457 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 35 2023-06-22T08:04:01.475 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4269": { "addr": "172.21.15.23:6834/3365097525", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 3365097525, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 3365097525, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4269, "incarnation": 16, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 13 }, "gid_4272": { "addr": "172.21.15.23:6835/3894174037", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 3894174037, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 3894174037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4272, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 12 }, "gid_4279": { "addr": "172.21.15.67:6836/1962314516", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 1962314516, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 1962314516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4285": { "addr": "172.21.15.67:6839/567663545", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 567663545, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 567663545, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:03:59.300212+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4279, "mds_2": 4272, "mds_3": 4269, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:04:03.476 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:03.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.683+0000 7fdf527ff700 1 Processor -- start 2023-06-22T08:04:03.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.683+0000 7fdf527ff700 1 -- start start 2023-06-22T08:04:03.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.684+0000 7fdf527ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf4c0cb6b0 0x7fdf4c069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:03.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.684+0000 7fdf527ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c152e10 0x7fdf4c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:03.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.684+0000 7fdf527ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c158f60 con 0x7fdf4c0c5740 2023-06-22T08:04:03.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.684+0000 7fdf527ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c06bd20 con 0x7fdf4c0cb6b0 2023-06-22T08:04:03.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.684+0000 7fdf527ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0cc450 con 0x7fdf4c152e10 2023-06-22T08:04:03.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.685+0000 7fdf4b7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fdf4c0c5740 0x7fdf4c152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34022/0 (socket says 172.21.15.23:34022) 2023-06-22T08:04:03.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.685+0000 7fdf4b7fe700 1 -- 172.21.15.23:0/1787821661 learned_addr learned my addr 172.21.15.23:0/1787821661 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:04:03.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.685+0000 7fdf4bfff700 1 --2- 172.21.15.23:0/1787821661 >> v2:172.21.15.67:3300/0 conn(0x7fdf4c0cb6b0 0x7fdf4c069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:03.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.685+0000 7fdf50d9c700 1 --2- 172.21.15.23:0/1787821661 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c152e10 0x7fdf4c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:03.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.685+0000 7fdf4bfff700 1 -- 172.21.15.23:0/1787821661 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c152e10 msgr2=0x7fdf4c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:03.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.685+0000 7fdf4bfff700 1 --2- 172.21.15.23:0/1787821661 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c152e10 0x7fdf4c1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:03.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.685+0000 7fdf4bfff700 1 -- 172.21.15.23:0/1787821661 >> v1:172.21.15.23:6789/0 conn(0x7fdf4c0c5740 legacy=0x7fdf4c152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:03.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.685+0000 7fdf4bfff700 1 -- 172.21.15.23:0/1787821661 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c158980 con 0x7fdf4c0cb6b0 2023-06-22T08:04:03.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.685+0000 7fdf4bfff700 1 --2- 172.21.15.23:0/1787821661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0cb6b0 0x7fdf4c069eb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009840 tx=0x7fdf3c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=749eaf94e9b9bff0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:03.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.687+0000 7fdf4affd700 1 -- 172.21.15.23:0/1787821661 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c011830 con 0x7fdf4c0cb6b0 2023-06-22T08:04:03.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.687+0000 7fdf4affd700 1 -- 172.21.15.23:0/1787821661 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c011990 con 0x7fdf4c0cb6b0 2023-06-22T08:04:03.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.687+0000 7fdf4affd700 1 -- 172.21.15.23:0/1787821661 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c011d20 con 0x7fdf4c0cb6b0 2023-06-22T08:04:03.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.687+0000 7fdf527ff700 1 -- 172.21.15.23:0/1787821661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0cb6b0 msgr2=0x7fdf4c069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:03.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.687+0000 7fdf527ff700 1 --2- 172.21.15.23:0/1787821661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0cb6b0 0x7fdf4c069eb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009840 tx=0x7fdf3c009b50 comp rx=0 tx=0).stop 2023-06-22T08:04:03.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.687+0000 7fdf527ff700 1 -- 172.21.15.23:0/1787821661 shutdown_connections 2023-06-22T08:04:03.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.687+0000 7fdf527ff700 1 --2- 172.21.15.23:0/1787821661 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c152e10 0x7fdf4c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:03.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.687+0000 7fdf527ff700 1 --2- 172.21.15.23:0/1787821661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c0cb6b0 0x7fdf4c069eb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:03.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.687+0000 7fdf527ff700 1 -- 172.21.15.23:0/1787821661 >> 172.21.15.23:0/1787821661 conn(0x7fdf4c070d80 msgr2=0x7fdf4c071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:03.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.688+0000 7fdf527ff700 1 -- 172.21.15.23:0/1787821661 shutdown_connections 2023-06-22T08:04:03.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.688+0000 7fdf527ff700 1 -- 172.21.15.23:0/1787821661 wait complete. 2023-06-22T08:04:03.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.689+0000 7fdf527ff700 1 Processor -- start 2023-06-22T08:04:03.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.689+0000 7fdf527ff700 1 -- start start 2023-06-22T08:04:03.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.690+0000 7fdf527ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c0cb6b0 0x7fdf4c1f1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:03.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.690+0000 7fdf527ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4c152e10 0x7fdf4c0c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:03.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.690+0000 7fdf527ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c158390 0x7fdf4c148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:03.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.690+0000 7fdf527ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c1f1ea0 con 0x7fdf4c152e10 2023-06-22T08:04:03.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf527ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c154450 con 0x7fdf4c158390 2023-06-22T08:04:03.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf527ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0c8770 con 0x7fdf4c0cb6b0 2023-06-22T08:04:03.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4c152e10 0x7fdf4c0c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:03.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c0cb6b0 0x7fdf4c1f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:03.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c0cb6b0 0x7fdf4c1f1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:53228/0 (socket says 172.21.15.23:53228) 2023-06-22T08:04:03.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4bfff700 1 -- 172.21.15.23:0/1842249759 learned_addr learned my addr 172.21.15.23:0/1842249759 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:03.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf50d9c700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c158390 0x7fdf4c148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:03.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4b7fe700 1 -- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c0cb6b0 msgr2=0x7fdf4c1f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:03.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4b7fe700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c0cb6b0 0x7fdf4c1f1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:03.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4b7fe700 1 -- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c158390 msgr2=0x7fdf4c148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:03.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4b7fe700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c158390 0x7fdf4c148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:03.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4b7fe700 1 -- 172.21.15.23:0/1842249759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c0094a0 con 0x7fdf4c152e10 2023-06-22T08:04:03.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4bfff700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c0cb6b0 0x7fdf4c1f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:04:03.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.691+0000 7fdf4b7fe700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4c152e10 0x7fdf4c0c4830 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0c9380 tx=0x7fdf4000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:03.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.692+0000 7fdf48ff9700 1 -- 172.21.15.23:0/1842249759 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf40005d70 con 0x7fdf4c152e10 2023-06-22T08:04:03.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.692+0000 7fdf48ff9700 1 -- 172.21.15.23:0/1842249759 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4000ce50 con 0x7fdf4c152e10 2023-06-22T08:04:03.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.692+0000 7fdf48ff9700 1 -- 172.21.15.23:0/1842249759 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf400072d0 con 0x7fdf4c152e10 2023-06-22T08:04:03.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.692+0000 7fdf527ff700 1 -- 172.21.15.23:0/1842249759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c1578d0 con 0x7fdf4c152e10 2023-06-22T08:04:03.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.692+0000 7fdf527ff700 1 -- 172.21.15.23:0/1842249759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c0c8d10 con 0x7fdf4c152e10 2023-06-22T08:04:03.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.693+0000 7fdf527ff700 1 -- 172.21.15.23:0/1842249759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4c04ea20 con 0x7fdf4c152e10 2023-06-22T08:04:03.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.694+0000 7fdf48ff9700 1 -- 172.21.15.23:0/1842249759 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdf40007430 con 0x7fdf4c152e10 2023-06-22T08:04:03.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.695+0000 7fdf48ff9700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdf3806ace0 0x7fdf3806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:03.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.695+0000 7fdf48ff9700 1 -- 172.21.15.23:0/1842249759 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6985+0+0 (secure 0 0 0) 0x7fdf400888f0 con 0x7fdf4c152e10 2023-06-22T08:04:03.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.695+0000 7fdf4bfff700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdf3806ace0 0x7fdf3806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:03.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.696+0000 7fdf4bfff700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdf3806ace0 0x7fdf3806d190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009b90 tx=0x7fdf3c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:03.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.699+0000 7fdf48ff9700 1 -- 172.21.15.23:0/1842249759 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdf40053740 con 0x7fdf4c152e10 2023-06-22T08:04:03.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.954+0000 7fdf527ff700 1 -- 172.21.15.23:0/1842249759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf4c150d80 con 0x7fdf4c152e10 2023-06-22T08:04:03.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.954+0000 7fdf48ff9700 1 -- 172.21.15.23:0/1842249759 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5512 (secure 0 0 0) 0x7fdf40056d60 con 0x7fdf4c152e10 2023-06-22T08:04:03.958 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:03.958 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:03:59.300212+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:03.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.956+0000 7fdf527ff700 1 -- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdf3806ace0 msgr2=0x7fdf3806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:03.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.956+0000 7fdf527ff700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdf3806ace0 0x7fdf3806d190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009b90 tx=0x7fdf3c008030 comp rx=0 tx=0).stop 2023-06-22T08:04:03.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.956+0000 7fdf527ff700 1 -- 172.21.15.23:0/1842249759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4c152e10 msgr2=0x7fdf4c0c4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:03.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.956+0000 7fdf527ff700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4c152e10 0x7fdf4c0c4830 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0c9380 tx=0x7fdf4000c7b0 comp rx=0 tx=0).stop 2023-06-22T08:04:03.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.957+0000 7fdf527ff700 1 -- 172.21.15.23:0/1842249759 shutdown_connections 2023-06-22T08:04:03.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.957+0000 7fdf527ff700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdf3806ace0 0x7fdf3806d190 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:03.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.957+0000 7fdf527ff700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf4c158390 0x7fdf4c148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:03.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.957+0000 7fdf527ff700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdf4c0cb6b0 0x7fdf4c1f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:03.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.957+0000 7fdf527ff700 1 --2- 172.21.15.23:0/1842249759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4c152e10 0x7fdf4c0c4830 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:03.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.957+0000 7fdf527ff700 1 -- 172.21.15.23:0/1842249759 >> 172.21.15.23:0/1842249759 conn(0x7fdf4c070d80 msgr2=0x7fdf4c0ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:03.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.957+0000 7fdf527ff700 1 -- 172.21.15.23:0/1842249759 shutdown_connections 2023-06-22T08:04:03.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:03.957+0000 7fdf527ff700 1 -- 172.21.15.23:0/1842249759 wait complete. 2023-06-22T08:04:03.964 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 35 2023-06-22T08:04:03.978 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:04:05.979 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:06.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.181+0000 7f01fc3ff700 1 Processor -- start 2023-06-22T08:04:06.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.182+0000 7f01fc3ff700 1 -- start start 2023-06-22T08:04:06.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.183+0000 7f01fc3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01f4152ce0 0x7f01f40cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:06.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.183+0000 7f01fc3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f01f40c40b0 0x7f01f40c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:06.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.183+0000 7f01fc3ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f415c370 con 0x7f01f40c9fd0 2023-06-22T08:04:06.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.183+0000 7f01fc3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f41530b0 con 0x7f01f40c40b0 2023-06-22T08:04:06.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.183+0000 7f01fc3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f40cf6f0 con 0x7f01f4152ce0 2023-06-22T08:04:06.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.183+0000 7f01f999a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f01f40c9fd0 0x7f01f415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41110/0 (socket says 172.21.15.23:41110) 2023-06-22T08:04:06.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.183+0000 7f01f999a700 1 -- 172.21.15.23:0/4151013127 learned_addr learned my addr 172.21.15.23:0/4151013127 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:04:06.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.183+0000 7f01fa99c700 1 --2- 172.21.15.23:0/4151013127 >> v2:172.21.15.23:3300/0 conn(0x7f01f40c40b0 0x7f01f40c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:06.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.183+0000 7f01fa19b700 1 --2- 172.21.15.23:0/4151013127 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01f4152ce0 0x7f01f40cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:06.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.184+0000 7f01f9199700 1 -- 172.21.15.23:0/4151013127 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083300044 0 0) 0x7f01f415c370 con 0x7f01f40c9fd0 2023-06-22T08:04:06.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.184+0000 7f01f9199700 1 -- 172.21.15.23:0/4151013127 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e00035b0 con 0x7f01f40c9fd0 2023-06-22T08:04:06.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.184+0000 7f01fa99c700 1 -- 172.21.15.23:0/4151013127 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01f4152ce0 msgr2=0x7f01f40cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:06.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.184+0000 7f01fa99c700 1 --2- 172.21.15.23:0/4151013127 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01f4152ce0 0x7f01f40cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.184+0000 7f01fa99c700 1 -- 172.21.15.23:0/4151013127 >> v1:172.21.15.23:6789/0 conn(0x7f01f40c9fd0 legacy=0x7f01f415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:06.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.184+0000 7f01fa99c700 1 -- 172.21.15.23:0/4151013127 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f415bda0 con 0x7f01f40c40b0 2023-06-22T08:04:06.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.184+0000 7f01fa99c700 1 --2- 172.21.15.23:0/4151013127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 0x7f01f40c4530 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009d90 tx=0x7f01ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f11f4884b57da4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:06.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.184+0000 7f01f9199700 1 -- 172.21.15.23:0/4151013127 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec00e070 con 0x7f01f40c40b0 2023-06-22T08:04:06.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.184+0000 7f01f9199700 1 -- 172.21.15.23:0/4151013127 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ec00bb20 con 0x7f01f40c40b0 2023-06-22T08:04:06.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.184+0000 7f01f9199700 1 -- 172.21.15.23:0/4151013127 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec012430 con 0x7f01f40c40b0 2023-06-22T08:04:06.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.185+0000 7f01fc3ff700 1 -- 172.21.15.23:0/4151013127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 msgr2=0x7f01f40c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:06.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.185+0000 7f01fc3ff700 1 --2- 172.21.15.23:0/4151013127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 0x7f01f40c4530 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009d90 tx=0x7f01ec00b330 comp rx=0 tx=0).stop 2023-06-22T08:04:06.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.185+0000 7f01fc3ff700 1 -- 172.21.15.23:0/4151013127 shutdown_connections 2023-06-22T08:04:06.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.185+0000 7f01fc3ff700 1 --2- 172.21.15.23:0/4151013127 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01f4152ce0 0x7f01f40cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.185+0000 7f01fc3ff700 1 --2- 172.21.15.23:0/4151013127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 0x7f01f40c4530 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.185+0000 7f01fc3ff700 1 -- 172.21.15.23:0/4151013127 >> 172.21.15.23:0/4151013127 conn(0x7f01f406ae30 msgr2=0x7f01f406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:06.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.185+0000 7f01fc3ff700 1 -- 172.21.15.23:0/4151013127 shutdown_connections 2023-06-22T08:04:06.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.185+0000 7f01fc3ff700 1 -- 172.21.15.23:0/4151013127 wait complete. 2023-06-22T08:04:06.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.187+0000 7f01fc3ff700 1 Processor -- start 2023-06-22T08:04:06.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.187+0000 7f01fc3ff700 1 -- start start 2023-06-22T08:04:06.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fc3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 0x7f01f41598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:06.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fc3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01f40c9fd0 0x7f01f414adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:06.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fc3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01f4152ce0 0x7f01f41e1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:06.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fc3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f41ef0a0 con 0x7f01f40c40b0 2023-06-22T08:04:06.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fc3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f41ee490 con 0x7f01f4152ce0 2023-06-22T08:04:06.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fc3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f41ee770 con 0x7f01f40c9fd0 2023-06-22T08:04:06.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fa19b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 0x7f01f41598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:06.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fa19b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 0x7f01f41598a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58104/0 (socket says 172.21.15.23:58104) 2023-06-22T08:04:06.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fa19b700 1 -- 172.21.15.23:0/2663365262 learned_addr learned my addr 172.21.15.23:0/2663365262 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:06.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01f999a700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01f40c9fd0 0x7f01f414adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:06.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fa99c700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01f4152ce0 0x7f01f41e1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:06.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fa19b700 1 -- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01f40c9fd0 msgr2=0x7f01f414adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:06.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.188+0000 7f01fa19b700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01f40c9fd0 0x7f01f414adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.189+0000 7f01fa19b700 1 -- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01f4152ce0 msgr2=0x7f01f41e1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:06.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.189+0000 7f01fa19b700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01f4152ce0 0x7f01f41e1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.189+0000 7f01fa19b700 1 -- 172.21.15.23:0/2663365262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec009a70 con 0x7f01f40c40b0 2023-06-22T08:04:06.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.189+0000 7f01fa19b700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 0x7f01f41598a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f01f000e9b0 tx=0x7f01f000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:06.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.189+0000 7f01eaffd700 1 -- 172.21.15.23:0/2663365262 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f0009710 con 0x7f01f40c40b0 2023-06-22T08:04:06.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.189+0000 7f01eaffd700 1 -- 172.21.15.23:0/2663365262 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f000ee50 con 0x7f01f40c40b0 2023-06-22T08:04:06.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.189+0000 7f01eaffd700 1 -- 172.21.15.23:0/2663365262 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f000ca50 con 0x7f01f40c40b0 2023-06-22T08:04:06.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.189+0000 7f01fc3ff700 1 -- 172.21.15.23:0/2663365262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f41eea50 con 0x7f01f40c40b0 2023-06-22T08:04:06.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.189+0000 7f01fc3ff700 1 -- 172.21.15.23:0/2663365262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f41e2060 con 0x7f01f40c40b0 2023-06-22T08:04:06.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.190+0000 7f01fc3ff700 1 -- 172.21.15.23:0/2663365262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f40b19e0 con 0x7f01f40c40b0 2023-06-22T08:04:06.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.191+0000 7f01eaffd700 1 -- 172.21.15.23:0/2663365262 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f01f000cbb0 con 0x7f01f40c40b0 2023-06-22T08:04:06.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.192+0000 7f01eaffd700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01d806ace0 0x7f01d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:06.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.192+0000 7f01eaffd700 1 -- 172.21.15.23:0/2663365262 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6985+0+0 (secure 0 0 0) 0x7f01f00898e0 con 0x7f01f40c40b0 2023-06-22T08:04:06.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.193+0000 7f01f999a700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01d806ace0 0x7f01d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:06.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.193+0000 7f01f999a700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01d806ace0 0x7f01d806d190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f01f41e3140 tx=0x7f01e4007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:06.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.197+0000 7f01eaffd700 1 -- 172.21.15.23:0/2663365262 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f01f0057fe0 con 0x7f01f40c40b0 2023-06-22T08:04:06.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.449+0000 7f01fc3ff700 1 -- 172.21.15.23:0/2663365262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01f404ea20 con 0x7f01f40c40b0 2023-06-22T08:04:06.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.450+0000 7f01eaffd700 1 -- 172.21.15.23:0/2663365262 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5513 (secure 0 0 0) 0x7f01f000ce60 con 0x7f01f40c40b0 2023-06-22T08:04:06.453 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:06.453 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:04:05.304565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:resolve","state_seq":9,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:06.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 -- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01d806ace0 msgr2=0x7f01d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:06.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01d806ace0 0x7f01d806d190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f01f41e3140 tx=0x7f01e4007dc0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 -- 172.21.15.23:0/2663365262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 msgr2=0x7f01f41598a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:06.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 0x7f01f41598a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f01f000e9b0 tx=0x7f01f000ecc0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 -- 172.21.15.23:0/2663365262 shutdown_connections 2023-06-22T08:04:06.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01d806ace0 0x7f01d806d190 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01f4152ce0 0x7f01f41e1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f40c40b0 0x7f01f41598a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 --2- 172.21.15.23:0/2663365262 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01f40c9fd0 0x7f01f414adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:06.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 -- 172.21.15.23:0/2663365262 >> 172.21.15.23:0/2663365262 conn(0x7f01f406ae30 msgr2=0x7f01f40ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:06.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 -- 172.21.15.23:0/2663365262 shutdown_connections 2023-06-22T08:04:06.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:06.452+0000 7f01fc3ff700 1 -- 172.21.15.23:0/2663365262 wait complete. 2023-06-22T08:04:06.457 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 36 2023-06-22T08:04:06.473 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:04:06.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:04:06.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:04:08.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:08.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.682+0000 7f6bea9ff700 1 Processor -- start 2023-06-22T08:04:08.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.682+0000 7f6bea9ff700 1 -- start start 2023-06-22T08:04:08.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.683+0000 7f6bea9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be4152a90 0x7f6be40c9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:08.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.683+0000 7f6bea9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6be40ca640 0x7f6be415bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:08.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.683+0000 7f6bea9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be40c4c50 con 0x7f6be40c9680 2023-06-22T08:04:08.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.683+0000 7f6bea9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be4158ac0 con 0x7f6be40ca640 2023-06-22T08:04:08.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.683+0000 7f6bea9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be415cba0 con 0x7f6be4152a90 2023-06-22T08:04:08.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.683+0000 7f6be8f9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6be40ca640 0x7f6be415bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:08.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.683+0000 7f6be3fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be4152a90 0x7f6be40c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:08.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be8f9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6be40ca640 0x7f6be415bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58118/0 (socket says 172.21.15.23:58118) 2023-06-22T08:04:08.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be8f9c700 1 -- 172.21.15.23:0/1256756393 learned_addr learned my addr 172.21.15.23:0/1256756393 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:08.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be37fe700 1 --1- 172.21.15.23:0/1256756393 >> v1:172.21.15.67:6789/0 conn(0x7f6be40c9680 0x7f6be40c9f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:37160/0 (socket says 172.21.15.23:37160) 2023-06-22T08:04:08.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be8f9c700 1 -- 172.21.15.23:0/1256756393 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be4152a90 msgr2=0x7f6be40c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:08.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be8f9c700 1 --2- 172.21.15.23:0/1256756393 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be4152a90 0x7f6be40c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:08.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be8f9c700 1 -- 172.21.15.23:0/1256756393 >> v1:172.21.15.67:6789/0 conn(0x7f6be40c9680 legacy=0x7f6be40c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:08.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be8f9c700 1 -- 172.21.15.23:0/1256756393 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be40c4040 con 0x7f6be40ca640 2023-06-22T08:04:08.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be8f9c700 1 --2- 172.21.15.23:0/1256756393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be40ca640 0x7f6be415bca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009e50 tx=0x7f6bdc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65398bf5b542c58a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:08.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be2ffd700 1 -- 172.21.15.23:0/1256756393 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc00e070 con 0x7f6be40ca640 2023-06-22T08:04:08.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be2ffd700 1 -- 172.21.15.23:0/1256756393 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc00b910 con 0x7f6be40ca640 2023-06-22T08:04:08.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.684+0000 7f6be2ffd700 1 -- 172.21.15.23:0/1256756393 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc00bcc0 con 0x7f6be40ca640 2023-06-22T08:04:08.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.685+0000 7f6bea9ff700 1 -- 172.21.15.23:0/1256756393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be40ca640 msgr2=0x7f6be415bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:08.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.685+0000 7f6bea9ff700 1 --2- 172.21.15.23:0/1256756393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be40ca640 0x7f6be415bca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009e50 tx=0x7f6bdc00b330 comp rx=0 tx=0).stop 2023-06-22T08:04:08.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.685+0000 7f6bea9ff700 1 -- 172.21.15.23:0/1256756393 shutdown_connections 2023-06-22T08:04:08.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.685+0000 7f6bea9ff700 1 --2- 172.21.15.23:0/1256756393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be40ca640 0x7f6be415bca0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:08.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.685+0000 7f6bea9ff700 1 --2- 172.21.15.23:0/1256756393 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be4152a90 0x7f6be40c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:08.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.685+0000 7f6bea9ff700 1 -- 172.21.15.23:0/1256756393 >> 172.21.15.23:0/1256756393 conn(0x7f6be406ace0 msgr2=0x7f6be406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:08.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.685+0000 7f6bea9ff700 1 -- 172.21.15.23:0/1256756393 shutdown_connections 2023-06-22T08:04:08.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.685+0000 7f6bea9ff700 1 -- 172.21.15.23:0/1256756393 wait complete. 2023-06-22T08:04:08.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.687+0000 7f6bea9ff700 1 Processor -- start 2023-06-22T08:04:08.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.688+0000 7f6bea9ff700 1 -- start start 2023-06-22T08:04:08.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6bea9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be40c9680 0x7f6be41ef590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:08.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6bea9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be40ca640 0x7f6be414c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:08.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6bea9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be4152a90 0x7f6be414a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:08.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6bea9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be414b030 con 0x7f6be4152a90 2023-06-22T08:04:08.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6bea9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be414b310 con 0x7f6be40c9680 2023-06-22T08:04:08.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6bea9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be414b5f0 con 0x7f6be40ca640 2023-06-22T08:04:08.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6be8f9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be4152a90 0x7f6be414a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:08.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6be8f9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be4152a90 0x7f6be414a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58124/0 (socket says 172.21.15.23:58124) 2023-06-22T08:04:08.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6be8f9c700 1 -- 172.21.15.23:0/3216560032 learned_addr learned my addr 172.21.15.23:0/3216560032 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:08.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6be3fff700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be40c9680 0x7f6be41ef590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:08.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.689+0000 7f6be37fe700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be40ca640 0x7f6be414c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:08.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6be8f9c700 1 -- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be40ca640 msgr2=0x7f6be414c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:08.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6be8f9c700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be40ca640 0x7f6be414c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:08.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6be8f9c700 1 -- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be40c9680 msgr2=0x7f6be41ef590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:08.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6be8f9c700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be40c9680 0x7f6be41ef590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:08.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6be8f9c700 1 -- 172.21.15.23:0/3216560032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc009b30 con 0x7f6be4152a90 2023-06-22T08:04:08.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6be8f9c700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be4152a90 0x7f6be414a9c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6be408bba0 tx=0x7f6bdc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:08.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6be0ff9700 1 -- 172.21.15.23:0/3216560032 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc004800 con 0x7f6be4152a90 2023-06-22T08:04:08.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6be0ff9700 1 -- 172.21.15.23:0/3216560032 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc004960 con 0x7f6be4152a90 2023-06-22T08:04:08.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6be0ff9700 1 -- 172.21.15.23:0/3216560032 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc01c680 con 0x7f6be4152a90 2023-06-22T08:04:08.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6bea9ff700 1 -- 172.21.15.23:0/3216560032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be41e2cb0 con 0x7f6be4152a90 2023-06-22T08:04:08.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.690+0000 7f6bea9ff700 1 -- 172.21.15.23:0/3216560032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be41e3270 con 0x7f6be4152a90 2023-06-22T08:04:08.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.692+0000 7f6bea9ff700 1 -- 172.21.15.23:0/3216560032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be404ea20 con 0x7f6be4152a90 2023-06-22T08:04:08.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.692+0000 7f6be0ff9700 1 -- 172.21.15.23:0/3216560032 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6bdc01c7e0 con 0x7f6be4152a90 2023-06-22T08:04:08.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.693+0000 7f6be0ff9700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bd006ace0 0x7f6bd006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:08.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.693+0000 7f6be0ff9700 1 -- 172.21.15.23:0/3216560032 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 7097+0+0 (secure 0 0 0) 0x7f6bdc08b4e0 con 0x7f6be4152a90 2023-06-22T08:04:08.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.694+0000 7f6be3fff700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bd006ace0 0x7f6bd006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:08.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.694+0000 7f6be3fff700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bd006ace0 0x7f6bd006d190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009be0 tx=0x7f6bd4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:08.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.698+0000 7f6be0ff9700 1 -- 172.21.15.23:0/3216560032 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6bdc056170 con 0x7f6be4152a90 2023-06-22T08:04:08.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.953+0000 7f6bea9ff700 1 -- 172.21.15.23:0/3216560032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6be40c5740 con 0x7f6be4152a90 2023-06-22T08:04:08.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.954+0000 7f6be0ff9700 1 -- 172.21.15.23:0/3216560032 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5513 (secure 0 0 0) 0x7f6bdc059790 con 0x7f6be4152a90 2023-06-22T08:04:08.957 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:08.958 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:04:05.304565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:resolve","state_seq":9,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:08.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 -- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bd006ace0 msgr2=0x7f6bd006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:08.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bd006ace0 0x7f6bd006d190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009be0 tx=0x7f6bd4009210 comp rx=0 tx=0).stop 2023-06-22T08:04:08.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 -- 172.21.15.23:0/3216560032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be4152a90 msgr2=0x7f6be414a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:08.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be4152a90 0x7f6be414a9c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6be408bba0 tx=0x7f6bdc004ad0 comp rx=0 tx=0).stop 2023-06-22T08:04:08.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 -- 172.21.15.23:0/3216560032 shutdown_connections 2023-06-22T08:04:08.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be4152a90 0x7f6be414a9c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:08.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bd006ace0 0x7f6bd006d190 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:08.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be40c9680 0x7f6be41ef590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:08.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 --2- 172.21.15.23:0/3216560032 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be40ca640 0x7f6be414c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:08.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 -- 172.21.15.23:0/3216560032 >> 172.21.15.23:0/3216560032 conn(0x7f6be406ace0 msgr2=0x7f6be40c9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:08.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 -- 172.21.15.23:0/3216560032 shutdown_connections 2023-06-22T08:04:08.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:08.956+0000 7f6bea9ff700 1 -- 172.21.15.23:0/3216560032 wait complete. 2023-06-22T08:04:08.961 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 36 2023-06-22T08:04:08.978 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:04:10.979 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:11.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.188+0000 7f5cd559d700 1 Processor -- start 2023-06-22T08:04:11.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.188+0000 7f5cd559d700 1 -- start start 2023-06-22T08:04:11.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.189+0000 7f5cd559d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9890 0x7f5cd006bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:11.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.189+0000 7f5cd559d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5cd00c9c60 0x7f5cd0158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:11.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.189+0000 7f5cd559d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd00c8f80 con 0x7f5cd006ae60 2023-06-22T08:04:11.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.189+0000 7f5cd559d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd00c94b0 con 0x7f5cd00c9c60 2023-06-22T08:04:11.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.189+0000 7f5cd559d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd00b4620 con 0x7f5cd00c9890 2023-06-22T08:04:11.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.189+0000 7f5ccf7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5cd00c9c60 0x7f5cd0158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:11.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.189+0000 7f5ccffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9890 0x7f5cd006bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:11.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.189+0000 7f5ccffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9890 0x7f5cd006bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35930/0 (socket says 172.21.15.23:35930) 2023-06-22T08:04:11.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.189+0000 7f5ccffff700 1 -- 172.21.15.23:0/1438749852 learned_addr learned my addr 172.21.15.23:0/1438749852 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:11.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.190+0000 7f5ccf7fe700 1 -- 172.21.15.23:0/1438749852 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9890 msgr2=0x7f5cd006bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:11.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.190+0000 7f5ccf7fe700 1 --2- 172.21.15.23:0/1438749852 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9890 0x7f5cd006bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:11.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.190+0000 7f5ccf7fe700 1 -- 172.21.15.23:0/1438749852 >> v1:172.21.15.23:6789/0 conn(0x7f5cd006ae60 legacy=0x7f5cd015b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:11.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.190+0000 7f5ccf7fe700 1 -- 172.21.15.23:0/1438749852 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd01593f0 con 0x7f5cd00c9c60 2023-06-22T08:04:11.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.190+0000 7f5ccf7fe700 1 --2- 172.21.15.23:0/1438749852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd00c9c60 0x7f5cd0158eb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009840 tx=0x7f5cc4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da00d38bf0813a93 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:11.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.190+0000 7f5cceffd700 1 -- 172.21.15.23:0/1438749852 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4011830 con 0x7f5cd00c9c60 2023-06-22T08:04:11.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.190+0000 7f5cceffd700 1 -- 172.21.15.23:0/1438749852 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc4011990 con 0x7f5cd00c9c60 2023-06-22T08:04:11.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.190+0000 7f5cceffd700 1 -- 172.21.15.23:0/1438749852 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4011d20 con 0x7f5cd00c9c60 2023-06-22T08:04:11.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.191+0000 7f5cd559d700 1 -- 172.21.15.23:0/1438749852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd00c9c60 msgr2=0x7f5cd0158eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:11.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.191+0000 7f5cd559d700 1 --2- 172.21.15.23:0/1438749852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd00c9c60 0x7f5cd0158eb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009840 tx=0x7f5cc4009b50 comp rx=0 tx=0).stop 2023-06-22T08:04:11.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.191+0000 7f5cd559d700 1 -- 172.21.15.23:0/1438749852 shutdown_connections 2023-06-22T08:04:11.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.191+0000 7f5cd559d700 1 --2- 172.21.15.23:0/1438749852 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9890 0x7f5cd006bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:11.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.191+0000 7f5cd559d700 1 --2- 172.21.15.23:0/1438749852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd00c9c60 0x7f5cd0158eb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:11.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.191+0000 7f5cd559d700 1 -- 172.21.15.23:0/1438749852 >> 172.21.15.23:0/1438749852 conn(0x7f5cd0065b90 msgr2=0x7f5cd0065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:11.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.191+0000 7f5cd559d700 1 -- 172.21.15.23:0/1438749852 shutdown_connections 2023-06-22T08:04:11.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.191+0000 7f5cd559d700 1 -- 172.21.15.23:0/1438749852 wait complete. 2023-06-22T08:04:11.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.193+0000 7f5cd559d700 1 Processor -- start 2023-06-22T08:04:11.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.193+0000 7f5cd559d700 1 -- start start 2023-06-22T08:04:11.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5cd559d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd006ae60 0x7f5cd01e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:11.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5cd559d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cd00c9890 0x7f5cd01f1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:11.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5cd559d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9c60 0x7f5cd00cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:11.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5cd559d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd01e22d0 con 0x7f5cd006ae60 2023-06-22T08:04:11.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5cd559d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd01dea60 con 0x7f5cd00c9890 2023-06-22T08:04:11.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5cd559d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd01f2180 con 0x7f5cd00c9c60 2023-06-22T08:04:11.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5ccffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd006ae60 0x7f5cd01e1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:11.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5ccffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd006ae60 0x7f5cd01e1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58146/0 (socket says 172.21.15.23:58146) 2023-06-22T08:04:11.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5ccffff700 1 -- 172.21.15.23:0/4264255639 learned_addr learned my addr 172.21.15.23:0/4264255639 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:11.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5ccf7fe700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cd00c9890 0x7f5cd01f1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:11.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.194+0000 7f5cd4d9c700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9c60 0x7f5cd00cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:11.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5ccffff700 1 -- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9c60 msgr2=0x7f5cd00cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:11.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5ccffff700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9c60 0x7f5cd00cdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:11.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5ccffff700 1 -- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cd00c9890 msgr2=0x7f5cd01f1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:11.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5ccffff700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cd00c9890 0x7f5cd01f1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:11.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5ccffff700 1 -- 172.21.15.23:0/4264255639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc40094a0 con 0x7f5cd006ae60 2023-06-22T08:04:11.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5ccffff700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd006ae60 0x7f5cd01e1d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000e910 tx=0x7f5cc000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:11.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5cccff9700 1 -- 172.21.15.23:0/4264255639 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc00096a0 con 0x7f5cd006ae60 2023-06-22T08:04:11.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5cccff9700 1 -- 172.21.15.23:0/4264255639 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc0013680 con 0x7f5cd006ae60 2023-06-22T08:04:11.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5cd559d700 1 -- 172.21.15.23:0/4264255639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd01de580 con 0x7f5cd006ae60 2023-06-22T08:04:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5cccff9700 1 -- 172.21.15.23:0/4264255639 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0013a30 con 0x7f5cd006ae60 2023-06-22T08:04:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.195+0000 7f5cd559d700 1 -- 172.21.15.23:0/4264255639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd00b0f10 con 0x7f5cd006ae60 2023-06-22T08:04:11.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.197+0000 7f5cd559d700 1 -- 172.21.15.23:0/4264255639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd006bf50 con 0x7f5cd006ae60 2023-06-22T08:04:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.197+0000 7f5cccff9700 1 -- 172.21.15.23:0/4264255639 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5cc0013b90 con 0x7f5cd006ae60 2023-06-22T08:04:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.198+0000 7f5cccff9700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ad10 0x7f5cbc06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.198+0000 7f5cccff9700 1 -- 172.21.15.23:0/4264255639 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 7097+0+0 (secure 0 0 0) 0x7f5cc0089a60 con 0x7f5cd006ae60 2023-06-22T08:04:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.199+0000 7f5ccf7fe700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ad10 0x7f5cbc06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:11.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.199+0000 7f5ccf7fe700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ad10 0x7f5cbc06d1c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009fd0 tx=0x7f5cc401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:11.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.203+0000 7f5cccff9700 1 -- 172.21.15.23:0/4264255639 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5cc0054790 con 0x7f5cd006ae60 2023-06-22T08:04:11.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.459+0000 7f5cd559d700 1 -- 172.21.15.23:0/4264255639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cd0061b00 con 0x7f5cd006ae60 2023-06-22T08:04:11.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.460+0000 7f5cccff9700 1 -- 172.21.15.23:0/4264255639 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5516 (secure 0 0 0) 0x7f5cc0057db0 con 0x7f5cd006ae60 2023-06-22T08:04:11.463 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:11.463 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:04:10.725147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:11.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 -- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ad10 msgr2=0x7f5cbc06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:11.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ad10 0x7f5cbc06d1c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009fd0 tx=0x7f5cc401c040 comp rx=0 tx=0).stop 2023-06-22T08:04:11.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 -- 172.21.15.23:0/4264255639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd006ae60 msgr2=0x7f5cd01e1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:11.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd006ae60 0x7f5cd01e1d90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000e910 tx=0x7f5cc000ec20 comp rx=0 tx=0).stop 2023-06-22T08:04:11.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 -- 172.21.15.23:0/4264255639 shutdown_connections 2023-06-22T08:04:11.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5cd00c9c60 0x7f5cd00cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:11.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5cd006ae60 0x7f5cd01e1d90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:11.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5cbc06ad10 0x7f5cbc06d1c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:11.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 --2- 172.21.15.23:0/4264255639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cd00c9890 0x7f5cd01f1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:11.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 -- 172.21.15.23:0/4264255639 >> 172.21.15.23:0/4264255639 conn(0x7f5cd0065b90 msgr2=0x7f5cd0156010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 -- 172.21.15.23:0/4264255639 shutdown_connections 2023-06-22T08:04:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:11.462+0000 7f5cd559d700 1 -- 172.21.15.23:0/4264255639 wait complete. 2023-06-22T08:04:11.467 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 37 2023-06-22T08:04:11.483 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:04:13.484 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:13.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.690+0000 7f5af71ff700 1 Processor -- start 2023-06-22T08:04:13.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.690+0000 7f5af71ff700 1 -- start start 2023-06-22T08:04:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.692+0000 7f5af71ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5af00c97d0 0x7f5af0159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.692+0000 7f5af71ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af00c9c30 con 0x7f5af0152d70 2023-06-22T08:04:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.692+0000 7f5af71ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af015bea0 con 0x7f5af00c45f0 2023-06-22T08:04:13.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.692+0000 7f5af71ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af00c95e0 con 0x7f5af00c97d0 2023-06-22T08:04:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.692+0000 7f5aeffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5af00c97d0 0x7f5af0159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.692+0000 7f5aeffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5af00c97d0 0x7f5af0159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58152/0 (socket says 172.21.15.23:58152) 2023-06-22T08:04:13.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.692+0000 7f5aeffff700 1 -- 172.21.15.23:0/662272721 learned_addr learned my addr 172.21.15.23:0/662272721 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.693+0000 7f5aef7fe700 1 -- 172.21.15.23:0/662272721 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819218709 0 0) 0x7f5af015bea0 con 0x7f5af00c45f0 2023-06-22T08:04:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.693+0000 7f5aeffff700 1 -- 172.21.15.23:0/662272721 >> v1:172.21.15.67:6789/0 conn(0x7f5af00c45f0 legacy=0x7f5af00c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:13.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.693+0000 7f5aeffff700 1 -- 172.21.15.23:0/662272721 >> v1:172.21.15.23:6789/0 conn(0x7f5af0152d70 legacy=0x7f5af00cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.693+0000 7f5aeffff700 1 -- 172.21.15.23:0/662272721 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af015c160 con 0x7f5af00c97d0 2023-06-22T08:04:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.693+0000 7f5aeffff700 1 --2- 172.21.15.23:0/662272721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af00c97d0 0x7f5af0159330 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5adc0098f0 tx=0x7f5adc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb03a116eaa8b45 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.693+0000 7f5aef7fe700 1 -- 172.21.15.23:0/662272721 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc00b8a0 con 0x7f5af00c97d0 2023-06-22T08:04:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.693+0000 7f5aef7fe700 1 -- 172.21.15.23:0/662272721 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc00ba00 con 0x7f5af00c97d0 2023-06-22T08:04:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.693+0000 7f5aef7fe700 1 -- 172.21.15.23:0/662272721 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc00bd90 con 0x7f5af00c97d0 2023-06-22T08:04:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.694+0000 7f5af71ff700 1 -- 172.21.15.23:0/662272721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af00c97d0 msgr2=0x7f5af0159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.694+0000 7f5af71ff700 1 --2- 172.21.15.23:0/662272721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af00c97d0 0x7f5af0159330 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5adc0098f0 tx=0x7f5adc009c00 comp rx=0 tx=0).stop 2023-06-22T08:04:13.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.694+0000 7f5af71ff700 1 -- 172.21.15.23:0/662272721 shutdown_connections 2023-06-22T08:04:13.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.694+0000 7f5af71ff700 1 --2- 172.21.15.23:0/662272721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af00c97d0 0x7f5af0159330 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:13.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.694+0000 7f5af71ff700 1 -- 172.21.15.23:0/662272721 >> 172.21.15.23:0/662272721 conn(0x7f5af006ae30 msgr2=0x7f5af006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:13.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.694+0000 7f5af71ff700 1 -- 172.21.15.23:0/662272721 shutdown_connections 2023-06-22T08:04:13.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.694+0000 7f5af71ff700 1 -- 172.21.15.23:0/662272721 wait complete. 2023-06-22T08:04:13.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.696+0000 7f5af71ff700 1 Processor -- start 2023-06-22T08:04:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.696+0000 7f5af71ff700 1 -- start start 2023-06-22T08:04:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af71ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af00c45f0 0x7f5af014b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af71ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5af00c97d0 0x7f5af01e2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af71ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af0152d70 0x7f5af015ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af71ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af014bce0 con 0x7f5af0152d70 2023-06-22T08:04:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af71ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af01e1e90 con 0x7f5af00c45f0 2023-06-22T08:04:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af71ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af014ba70 con 0x7f5af00c97d0 2023-06-22T08:04:13.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af579c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af0152d70 0x7f5af015ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af4f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af00c45f0 0x7f5af014b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5aeffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5af00c97d0 0x7f5af01e2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af4f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af00c45f0 0x7f5af014b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50588/0 (socket says 172.21.15.23:50588) 2023-06-22T08:04:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af4f9b700 1 -- 172.21.15.23:0/2049042434 learned_addr learned my addr 172.21.15.23:0/2049042434 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af579c700 1 -- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5af00c97d0 msgr2=0x7f5af01e2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af579c700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5af00c97d0 0x7f5af01e2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:13.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af579c700 1 -- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af00c45f0 msgr2=0x7f5af014b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:13.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af579c700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af00c45f0 0x7f5af014b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:13.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af579c700 1 -- 172.21.15.23:0/2049042434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc009550 con 0x7f5af0152d70 2023-06-22T08:04:13.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5aeffff700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5af00c97d0 0x7f5af01e2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:04:13.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.697+0000 7f5af4f9b700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af00c45f0 0x7f5af014b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:04:13.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.698+0000 7f5af579c700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af0152d70 0x7f5af015ae00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800c820 tx=0x7f5ae800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:13.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.698+0000 7f5aed7fa700 1 -- 172.21.15.23:0/2049042434 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8003e30 con 0x7f5af0152d70 2023-06-22T08:04:13.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.698+0000 7f5aed7fa700 1 -- 172.21.15.23:0/2049042434 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae8005070 con 0x7f5af0152d70 2023-06-22T08:04:13.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.698+0000 7f5af71ff700 1 -- 172.21.15.23:0/2049042434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af00b2ae0 con 0x7f5af0152d70 2023-06-22T08:04:13.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.698+0000 7f5aed7fa700 1 -- 172.21.15.23:0/2049042434 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8005420 con 0x7f5af0152d70 2023-06-22T08:04:13.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.698+0000 7f5af71ff700 1 -- 172.21.15.23:0/2049042434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af00b2520 con 0x7f5af0152d70 2023-06-22T08:04:13.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.699+0000 7f5af71ff700 1 -- 172.21.15.23:0/2049042434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af004ea20 con 0x7f5af0152d70 2023-06-22T08:04:13.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.700+0000 7f5aed7fa700 1 -- 172.21.15.23:0/2049042434 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5ae8005580 con 0x7f5af0152d70 2023-06-22T08:04:13.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.701+0000 7f5aed7fa700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ad406ace0 0x7f5ad406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:13.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.701+0000 7f5aed7fa700 1 -- 172.21.15.23:0/2049042434 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 7097+0+0 (secure 0 0 0) 0x7f5ae8088e20 con 0x7f5af0152d70 2023-06-22T08:04:13.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.702+0000 7f5af4f9b700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ad406ace0 0x7f5ad406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:13.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.702+0000 7f5af4f9b700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ad406ace0 0x7f5ad406d190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4001c70 tx=0x7f5ae4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:13.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.705+0000 7f5aed7fa700 1 -- 172.21.15.23:0/2049042434 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5ae8053ab0 con 0x7f5af0152d70 2023-06-22T08:04:13.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.956+0000 7f5af71ff700 1 -- 172.21.15.23:0/2049042434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5af01e1fd0 con 0x7f5af0152d70 2023-06-22T08:04:13.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.957+0000 7f5aed7fa700 1 -- 172.21.15.23:0/2049042434 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5516 (secure 0 0 0) 0x7f5ae8017020 con 0x7f5af0152d70 2023-06-22T08:04:13.961 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:13.961 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:04:10.725147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:13.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.959+0000 7f5af71ff700 1 -- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ad406ace0 msgr2=0x7f5ad406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:13.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.959+0000 7f5af71ff700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ad406ace0 0x7f5ad406d190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4001c70 tx=0x7f5ae4009040 comp rx=0 tx=0).stop 2023-06-22T08:04:13.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.959+0000 7f5af71ff700 1 -- 172.21.15.23:0/2049042434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af0152d70 msgr2=0x7f5af015ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:13.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.959+0000 7f5af71ff700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af0152d70 0x7f5af015ae00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800c820 tx=0x7f5ae800cb30 comp rx=0 tx=0).stop 2023-06-22T08:04:13.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.960+0000 7f5af71ff700 1 -- 172.21.15.23:0/2049042434 shutdown_connections 2023-06-22T08:04:13.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.960+0000 7f5af71ff700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5af0152d70 0x7f5af015ae00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:13.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.960+0000 7f5af71ff700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ad406ace0 0x7f5ad406d190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:13.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.960+0000 7f5af71ff700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5af00c45f0 0x7f5af014b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:13.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.960+0000 7f5af71ff700 1 --2- 172.21.15.23:0/2049042434 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5af00c97d0 0x7f5af01e2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:13.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.960+0000 7f5af71ff700 1 -- 172.21.15.23:0/2049042434 >> 172.21.15.23:0/2049042434 conn(0x7f5af006ae30 msgr2=0x7f5af014f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:13.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.960+0000 7f5af71ff700 1 -- 172.21.15.23:0/2049042434 shutdown_connections 2023-06-22T08:04:13.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:13.960+0000 7f5af71ff700 1 -- 172.21.15.23:0/2049042434 wait complete. 2023-06-22T08:04:13.966 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 37 2023-06-22T08:04:13.982 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4269": { "addr": "172.21.15.23:6834/3365097525", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 3365097525, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 3365097525, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4269, "incarnation": 16, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 13 }, "gid_4272": { "addr": "172.21.15.23:6835/3894174037", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 3894174037, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 3894174037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4272, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 12 }, "gid_4279": { "addr": "172.21.15.67:6836/1962314516", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 1962314516, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 1962314516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4285": { "addr": "172.21.15.67:6839/567663545", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 567663545, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 567663545, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:04:10.725147+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4279, "mds_2": 4272, "mds_3": 4269, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:04:15.983 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:16.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.189+0000 7fe702fff700 1 Processor -- start 2023-06-22T08:04:16.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.189+0000 7fe702fff700 1 -- start start 2023-06-22T08:04:16.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.190+0000 7fe702fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 0x7fe6fc069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:16.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe702fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6fc158f60 con 0x7fe6fc0c5740 2023-06-22T08:04:16.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe702fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6fc0cc450 con 0x7fe6fc152e10 2023-06-22T08:04:16.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe702fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc06bd20 con 0x7fe6fc0cb6b0 2023-06-22T08:04:16.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe6fbfff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe6fc0c5740 0x7fe6fc152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:39328/0 (socket says 172.21.15.23:39328) 2023-06-22T08:04:16.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe6fbfff700 1 -- 172.21.15.23:0/587061663 learned_addr learned my addr 172.21.15.23:0/587061663 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:04:16.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe700d9b700 1 --2- 172.21.15.23:0/587061663 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 0x7fe6fc069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:16.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe6fb7fe700 1 -- 172.21.15.23:0/587061663 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140211474 0 0) 0x7fe6fc158f60 con 0x7fe6fc0c5740 2023-06-22T08:04:16.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe6fb7fe700 1 -- 172.21.15.23:0/587061663 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ec0035b0 con 0x7fe6fc0c5740 2023-06-22T08:04:16.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe6fb7fe700 1 -- 172.21.15.23:0/587061663 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785930698 0 0) 0x7fe6fc0cc450 con 0x7fe6fc152e10 2023-06-22T08:04:16.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe6fb7fe700 1 -- 172.21.15.23:0/587061663 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6fc158f60 con 0x7fe6fc152e10 2023-06-22T08:04:16.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe700d9b700 1 -- 172.21.15.23:0/587061663 >> v1:172.21.15.67:6789/0 conn(0x7fe6fc152e10 legacy=0x7fe6fc163110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:16.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe700d9b700 1 -- 172.21.15.23:0/587061663 >> v1:172.21.15.23:6789/0 conn(0x7fe6fc0c5740 legacy=0x7fe6fc152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:16.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.191+0000 7fe700d9b700 1 -- 172.21.15.23:0/587061663 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc158980 con 0x7fe6fc0cb6b0 2023-06-22T08:04:16.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.192+0000 7fe700d9b700 1 --2- 172.21.15.23:0/587061663 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 0x7fe6fc069eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0009870 tx=0x7fe6f0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27eeeef8cf5ae997 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:16.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.192+0000 7fe6fb7fe700 1 -- 172.21.15.23:0/587061663 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0011830 con 0x7fe6fc0cb6b0 2023-06-22T08:04:16.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.192+0000 7fe6fb7fe700 1 -- 172.21.15.23:0/587061663 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f0011990 con 0x7fe6fc0cb6b0 2023-06-22T08:04:16.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.192+0000 7fe6fb7fe700 1 -- 172.21.15.23:0/587061663 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0011d20 con 0x7fe6fc0cb6b0 2023-06-22T08:04:16.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.192+0000 7fe702fff700 1 -- 172.21.15.23:0/587061663 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 msgr2=0x7fe6fc069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:16.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.192+0000 7fe702fff700 1 --2- 172.21.15.23:0/587061663 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 0x7fe6fc069eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0009870 tx=0x7fe6f0009b80 comp rx=0 tx=0).stop 2023-06-22T08:04:16.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.193+0000 7fe702fff700 1 -- 172.21.15.23:0/587061663 shutdown_connections 2023-06-22T08:04:16.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.193+0000 7fe702fff700 1 --2- 172.21.15.23:0/587061663 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 0x7fe6fc069eb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:16.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.193+0000 7fe702fff700 1 -- 172.21.15.23:0/587061663 >> 172.21.15.23:0/587061663 conn(0x7fe6fc070d80 msgr2=0x7fe6fc071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:16.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.193+0000 7fe702fff700 1 -- 172.21.15.23:0/587061663 shutdown_connections 2023-06-22T08:04:16.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.193+0000 7fe702fff700 1 -- 172.21.15.23:0/587061663 wait complete. 2023-06-22T08:04:16.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.194+0000 7fe702fff700 1 Processor -- start 2023-06-22T08:04:16.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.195+0000 7fe702fff700 1 -- start start 2023-06-22T08:04:16.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.195+0000 7fe702fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 0x7fe6fc0c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:16.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.195+0000 7fe702fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6fc152e10 0x7fe6fc14ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:16.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.195+0000 7fe702fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6fc0c88d0 0x7fe6fc0c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:16.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.195+0000 7fe702fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc14e2f0 con 0x7fe6fc152e10 2023-06-22T08:04:16.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.195+0000 7fe702fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc0c52f0 con 0x7fe6fc0c88d0 2023-06-22T08:04:16.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.195+0000 7fe702fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc14f4d0 con 0x7fe6fc0cb6b0 2023-06-22T08:04:16.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe6fbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6fc152e10 0x7fe6fc14ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:16.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe700d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 0x7fe6fc0c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:16.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe6fbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6fc152e10 0x7fe6fc14ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46078/0 (socket says 172.21.15.23:46078) 2023-06-22T08:04:16.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe6fbfff700 1 -- 172.21.15.23:0/3700689191 learned_addr learned my addr 172.21.15.23:0/3700689191 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:16.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe70159c700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6fc0c88d0 0x7fe6fc0c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:16.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe6fbfff700 1 -- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 msgr2=0x7fe6fc0c4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:16.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe6fbfff700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 0x7fe6fc0c4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:16.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe6fbfff700 1 -- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6fc0c88d0 msgr2=0x7fe6fc0c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:16.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe6fbfff700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6fc0c88d0 0x7fe6fc0c7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:16.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe6fbfff700 1 -- 172.21.15.23:0/3700689191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f00094a0 con 0x7fe6fc152e10 2023-06-22T08:04:16.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe700d9b700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 0x7fe6fc0c4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:04:16.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.196+0000 7fe6fbfff700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6fc152e10 0x7fe6fc14ab10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800c700 tx=0x7fe6e800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:16.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.197+0000 7fe6f97fa700 1 -- 172.21.15.23:0/3700689191 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8005d70 con 0x7fe6fc152e10 2023-06-22T08:04:16.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.197+0000 7fe6f97fa700 1 -- 172.21.15.23:0/3700689191 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e80070d0 con 0x7fe6fc152e10 2023-06-22T08:04:16.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.197+0000 7fe6f97fa700 1 -- 172.21.15.23:0/3700689191 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8007440 con 0x7fe6fc152e10 2023-06-22T08:04:16.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.197+0000 7fe702fff700 1 -- 172.21.15.23:0/3700689191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6fc0c5500 con 0x7fe6fc152e10 2023-06-22T08:04:16.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.197+0000 7fe702fff700 1 -- 172.21.15.23:0/3700689191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc1f1d30 con 0x7fe6fc152e10 2023-06-22T08:04:16.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.198+0000 7fe702fff700 1 -- 172.21.15.23:0/3700689191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6fc04ea20 con 0x7fe6fc152e10 2023-06-22T08:04:16.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.199+0000 7fe6f97fa700 1 -- 172.21.15.23:0/3700689191 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe6e80075a0 con 0x7fe6fc152e10 2023-06-22T08:04:16.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.200+0000 7fe6f97fa700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe6e006ad10 0x7fe6e006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:16.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.200+0000 7fe6f97fa700 1 -- 172.21.15.23:0/3700689191 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 7041+0+0 (secure 0 0 0) 0x7fe6e8088c30 con 0x7fe6fc152e10 2023-06-22T08:04:16.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.200+0000 7fe700d9b700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe6e006ad10 0x7fe6e006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:16.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.201+0000 7fe700d9b700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe6e006ad10 0x7fe6e006d1c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00090b0 tx=0x7fe6f001c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:16.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.205+0000 7fe6f97fa700 1 -- 172.21.15.23:0/3700689191 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe6e8053880 con 0x7fe6fc152e10 2023-06-22T08:04:16.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.460+0000 7fe702fff700 1 -- 172.21.15.23:0/3700689191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6fc0c6960 con 0x7fe6fc152e10 2023-06-22T08:04:16.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.460+0000 7fe6f97fa700 1 -- 172.21.15.23:0/3700689191 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5513 (secure 0 0 0) 0x7fe6e8017020 con 0x7fe6fc152e10 2023-06-22T08:04:16.464 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:16.464 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:04:16.314372+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 -- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe6e006ad10 msgr2=0x7fe6e006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe6e006ad10 0x7fe6e006d1c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00090b0 tx=0x7fe6f001c040 comp rx=0 tx=0).stop 2023-06-22T08:04:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 -- 172.21.15.23:0/3700689191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6fc152e10 msgr2=0x7fe6fc14ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6fc152e10 0x7fe6fc14ab10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800c700 tx=0x7fe6e800ca10 comp rx=0 tx=0).stop 2023-06-22T08:04:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 -- 172.21.15.23:0/3700689191 shutdown_connections 2023-06-22T08:04:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe6e006ad10 0x7fe6e006d1c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6fc0c88d0 0x7fe6fc0c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe6fc0cb6b0 0x7fe6fc0c4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 --2- 172.21.15.23:0/3700689191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6fc152e10 0x7fe6fc14ab10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 -- 172.21.15.23:0/3700689191 >> 172.21.15.23:0/3700689191 conn(0x7fe6fc070d80 msgr2=0x7fe6fc072e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 -- 172.21.15.23:0/3700689191 shutdown_connections 2023-06-22T08:04:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:16.463+0000 7fe702fff700 1 -- 172.21.15.23:0/3700689191 wait complete. 2023-06-22T08:04:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 38 2023-06-22T08:04:16.487 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:04:18.488 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:18.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.696+0000 7f9434fff700 1 Processor -- start 2023-06-22T08:04:18.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.697+0000 7f9434fff700 1 -- start start 2023-06-22T08:04:18.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.698+0000 7f9434fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 0x7f9430069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:18.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.698+0000 7f9434fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9430158f60 con 0x7f94300cb6b0 2023-06-22T08:04:18.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.698+0000 7f9434fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94300cc450 con 0x7f94300c5740 2023-06-22T08:04:18.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.698+0000 7f9434fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943006bd20 con 0x7f9430152e10 2023-06-22T08:04:18.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.698+0000 7f942e59c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f94300cb6b0 0x7f9430152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:39334/0 (socket says 172.21.15.23:39334) 2023-06-22T08:04:18.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.698+0000 7f942e59c700 1 -- 172.21.15.23:0/3838771083 learned_addr learned my addr 172.21.15.23:0/3838771083 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:04:18.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.698+0000 7f942dd9b700 1 --2- 172.21.15.23:0/3838771083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 0x7f9430069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:18.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942d59a700 1 -- 172.21.15.23:0/3838771083 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896536536 0 0) 0x7f9430158f60 con 0x7f94300cb6b0 2023-06-22T08:04:18.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942d59a700 1 -- 172.21.15.23:0/3838771083 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94180035b0 con 0x7f94300cb6b0 2023-06-22T08:04:18.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942d59a700 1 -- 172.21.15.23:0/3838771083 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696517921 0 0) 0x7f94300cc450 con 0x7f94300c5740 2023-06-22T08:04:18.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942d59a700 1 -- 172.21.15.23:0/3838771083 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9430158f60 con 0x7f94300c5740 2023-06-22T08:04:18.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942dd9b700 1 -- 172.21.15.23:0/3838771083 >> v1:172.21.15.67:6789/0 conn(0x7f94300c5740 legacy=0x7f9430163110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:18.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942dd9b700 1 -- 172.21.15.23:0/3838771083 >> v1:172.21.15.23:6789/0 conn(0x7f94300cb6b0 legacy=0x7f9430152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:18.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942dd9b700 1 -- 172.21.15.23:0/3838771083 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9430158980 con 0x7f9430152e10 2023-06-22T08:04:18.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942dd9b700 1 --2- 172.21.15.23:0/3838771083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 0x7f9430069eb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9424009870 tx=0x7f9424009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78674bc2a9a2731e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:18.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942d59a700 1 -- 172.21.15.23:0/3838771083 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424011830 con 0x7f9430152e10 2023-06-22T08:04:18.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942d59a700 1 -- 172.21.15.23:0/3838771083 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9424011990 con 0x7f9430152e10 2023-06-22T08:04:18.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.699+0000 7f942d59a700 1 -- 172.21.15.23:0/3838771083 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424011d20 con 0x7f9430152e10 2023-06-22T08:04:18.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.700+0000 7f9434fff700 1 -- 172.21.15.23:0/3838771083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 msgr2=0x7f9430069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:18.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.700+0000 7f9434fff700 1 --2- 172.21.15.23:0/3838771083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 0x7f9430069eb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9424009870 tx=0x7f9424009b80 comp rx=0 tx=0).stop 2023-06-22T08:04:18.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.700+0000 7f9434fff700 1 -- 172.21.15.23:0/3838771083 shutdown_connections 2023-06-22T08:04:18.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.700+0000 7f9434fff700 1 --2- 172.21.15.23:0/3838771083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 0x7f9430069eb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:18.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.700+0000 7f9434fff700 1 -- 172.21.15.23:0/3838771083 >> 172.21.15.23:0/3838771083 conn(0x7f9430070d80 msgr2=0x7f9430071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:18.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.700+0000 7f9434fff700 1 -- 172.21.15.23:0/3838771083 shutdown_connections 2023-06-22T08:04:18.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.700+0000 7f9434fff700 1 -- 172.21.15.23:0/3838771083 wait complete. 2023-06-22T08:04:18.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.702+0000 7f9434fff700 1 Processor -- start 2023-06-22T08:04:18.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.703+0000 7f9434fff700 1 -- start start 2023-06-22T08:04:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f9434fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94300cb6b0 0x7f94301f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f9434fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 0x7f94300ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f9434fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943015b6d0 0x7f94301ee930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f9434fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94300c87f0 con 0x7f94300cb6b0 2023-06-22T08:04:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f9434fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94301e16c0 con 0x7f943015b6d0 2023-06-22T08:04:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f9434fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94301e19a0 con 0x7f9430152e10 2023-06-22T08:04:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f942e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94300cb6b0 0x7f94301f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f942e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94300cb6b0 0x7f94301f2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46092/0 (socket says 172.21.15.23:46092) 2023-06-22T08:04:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f942e59c700 1 -- 172.21.15.23:0/2106188290 learned_addr learned my addr 172.21.15.23:0/2106188290 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f942ed9d700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943015b6d0 0x7f94301ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.704+0000 7f942dd9b700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 0x7f94300ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f942e59c700 1 -- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 msgr2=0x7f94300ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f942e59c700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 0x7f94300ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f942e59c700 1 -- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943015b6d0 msgr2=0x7f94301ee930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f942e59c700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943015b6d0 0x7f94301ee930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f942e59c700 1 -- 172.21.15.23:0/2106188290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94240094a0 con 0x7f94300cb6b0 2023-06-22T08:04:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f942e59c700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94300cb6b0 0x7f94301f2280 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f942000c700 tx=0x7f942000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f941effd700 1 -- 172.21.15.23:0/2106188290 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420005d70 con 0x7f94300cb6b0 2023-06-22T08:04:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f941effd700 1 -- 172.21.15.23:0/2106188290 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94200070d0 con 0x7f94300cb6b0 2023-06-22T08:04:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f9434fff700 1 -- 172.21.15.23:0/2106188290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94301e1c80 con 0x7f94300cb6b0 2023-06-22T08:04:18.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f941effd700 1 -- 172.21.15.23:0/2106188290 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420007440 con 0x7f94300cb6b0 2023-06-22T08:04:18.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.705+0000 7f9434fff700 1 -- 172.21.15.23:0/2106188290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94301ee410 con 0x7f94300cb6b0 2023-06-22T08:04:18.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.706+0000 7f9434fff700 1 -- 172.21.15.23:0/2106188290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94301ee600 con 0x7f94300cb6b0 2023-06-22T08:04:18.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.707+0000 7f941effd700 1 -- 172.21.15.23:0/2106188290 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f94200075a0 con 0x7f94300cb6b0 2023-06-22T08:04:18.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.708+0000 7f941effd700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f941006ace0 0x7f941006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:18.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.708+0000 7f941effd700 1 -- 172.21.15.23:0/2106188290 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f9420088c20 con 0x7f94300cb6b0 2023-06-22T08:04:18.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.709+0000 7f942dd9b700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f941006ace0 0x7f941006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:18.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.709+0000 7f942dd9b700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f941006ace0 0x7f941006d190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9424009870 tx=0x7f942401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:18.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.713+0000 7f941effd700 1 -- 172.21.15.23:0/2106188290 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9420053910 con 0x7f94300cb6b0 2023-06-22T08:04:18.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.967+0000 7f9434fff700 1 -- 172.21.15.23:0/2106188290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f943004ea20 con 0x7f94300cb6b0 2023-06-22T08:04:18.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.968+0000 7f941effd700 1 -- 172.21.15.23:0/2106188290 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5513 (secure 0 0 0) 0x7f9420017070 con 0x7f94300cb6b0 2023-06-22T08:04:18.972 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:18.972 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:04:16.314372+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:18.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 -- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f941006ace0 msgr2=0x7f941006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:18.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f941006ace0 0x7f941006d190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9424009870 tx=0x7f942401c040 comp rx=0 tx=0).stop 2023-06-22T08:04:18.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 -- 172.21.15.23:0/2106188290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94300cb6b0 msgr2=0x7f94301f2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:18.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94300cb6b0 0x7f94301f2280 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f942000c700 tx=0x7f942000ca10 comp rx=0 tx=0).stop 2023-06-22T08:04:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 -- 172.21.15.23:0/2106188290 shutdown_connections 2023-06-22T08:04:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f941006ace0 0x7f941006d190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943015b6d0 0x7f94301ee930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94300cb6b0 0x7f94301f2280 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 --2- 172.21.15.23:0/2106188290 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9430152e10 0x7f94300ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 -- 172.21.15.23:0/2106188290 >> 172.21.15.23:0/2106188290 conn(0x7f9430070d80 msgr2=0x7f9430150b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 -- 172.21.15.23:0/2106188290 shutdown_connections 2023-06-22T08:04:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:18.971+0000 7f9434fff700 1 -- 172.21.15.23:0/2106188290 wait complete. 2023-06-22T08:04:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 38 2023-06-22T08:04:18.992 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:04:20.993 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.202+0000 7fe6317ff700 1 Processor -- start 2023-06-22T08:04:21.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.202+0000 7fe6317ff700 1 -- start start 2023-06-22T08:04:21.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.203+0000 7fe6317ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe62c152a10 0x7fe62c0c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.203+0000 7fe6317ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe62c0c9200 0x7fe62c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.203+0000 7fe6317ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe62c159730 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.203+0000 7fe6317ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62c0c5470 con 0x7fe62c152a10 2023-06-22T08:04:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.203+0000 7fe6317ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62c158910 con 0x7fe62c0c9200 2023-06-22T08:04:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.203+0000 7fe62affd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe62c152a10 0x7fe62c0c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.203+0000 7fe62affd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe62c152a10 0x7fe62c0c4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46096/0 (socket says 172.21.15.23:46096) 2023-06-22T08:04:21.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.203+0000 7fe62affd700 1 -- 172.21.15.23:0/1100458870 learned_addr learned my addr 172.21.15.23:0/1100458870 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:21.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.204+0000 7fe62a7fc700 1 --2- 172.21.15.23:0/1100458870 >> v2:172.21.15.67:3300/0 conn(0x7fe62c0c9200 0x7fe62c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:21.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.204+0000 7fe62affd700 1 -- 172.21.15.23:0/1100458870 >> v2:172.21.15.67:3300/0 conn(0x7fe62c0c9200 msgr2=0x7fe62c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:21.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.204+0000 7fe62affd700 1 --2- 172.21.15.23:0/1100458870 >> v2:172.21.15.67:3300/0 conn(0x7fe62c0c9200 0x7fe62c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.204+0000 7fe62affd700 1 -- 172.21.15.23:0/1100458870 >> v1:172.21.15.67:6789/0 conn(0x7fe62c0cb7a0 legacy=0x7fe62c15c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:21.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.204+0000 7fe62affd700 1 -- 172.21.15.23:0/1100458870 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c158bb0 con 0x7fe62c152a10 2023-06-22T08:04:21.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.204+0000 7fe62affd700 1 --2- 172.21.15.23:0/1100458870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c152a10 0x7fe62c0c4a30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe62c08c4c0 tx=0x7fe61c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b2cfc49c04526e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:21.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.204+0000 7fe629ffb700 1 -- 172.21.15.23:0/1100458870 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c011830 con 0x7fe62c152a10 2023-06-22T08:04:21.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.204+0000 7fe629ffb700 1 -- 172.21.15.23:0/1100458870 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c011990 con 0x7fe62c152a10 2023-06-22T08:04:21.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.204+0000 7fe629ffb700 1 -- 172.21.15.23:0/1100458870 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c011d20 con 0x7fe62c152a10 2023-06-22T08:04:21.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.205+0000 7fe6317ff700 1 -- 172.21.15.23:0/1100458870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c152a10 msgr2=0x7fe62c0c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:21.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.205+0000 7fe6317ff700 1 --2- 172.21.15.23:0/1100458870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c152a10 0x7fe62c0c4a30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe62c08c4c0 tx=0x7fe61c009bd0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.205+0000 7fe6317ff700 1 -- 172.21.15.23:0/1100458870 shutdown_connections 2023-06-22T08:04:21.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.205+0000 7fe6317ff700 1 --2- 172.21.15.23:0/1100458870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c152a10 0x7fe62c0c4a30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.205+0000 7fe6317ff700 1 --2- 172.21.15.23:0/1100458870 >> v2:172.21.15.67:3300/0 conn(0x7fe62c0c9200 0x7fe62c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.205+0000 7fe6317ff700 1 -- 172.21.15.23:0/1100458870 >> 172.21.15.23:0/1100458870 conn(0x7fe62c065b90 msgr2=0x7fe62c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:21.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.205+0000 7fe6317ff700 1 -- 172.21.15.23:0/1100458870 shutdown_connections 2023-06-22T08:04:21.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.205+0000 7fe6317ff700 1 -- 172.21.15.23:0/1100458870 wait complete. 2023-06-22T08:04:21.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.207+0000 7fe6317ff700 1 Processor -- start 2023-06-22T08:04:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.207+0000 7fe6317ff700 1 -- start start 2023-06-22T08:04:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe6317ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe62c0c9200 0x7fe62c14aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe6317ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c0cb7a0 0x7fe62c1579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe6317ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe62c152a10 0x7fe62c1ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe6317ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c1599b0 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe6317ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0b41e0 con 0x7fe62c152a10 2023-06-22T08:04:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe6317ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c159c70 con 0x7fe62c0c9200 2023-06-22T08:04:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe62a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c0cb7a0 0x7fe62c1579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe62a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c0cb7a0 0x7fe62c1579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46106/0 (socket says 172.21.15.23:46106) 2023-06-22T08:04:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe62a7fc700 1 -- 172.21.15.23:0/1015070345 learned_addr learned my addr 172.21.15.23:0/1015070345 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe62affd700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe62c0c9200 0x7fe62c14aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.208+0000 7fe62b7fe700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe62c152a10 0x7fe62c1ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.209+0000 7fe62a7fc700 1 -- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe62c0c9200 msgr2=0x7fe62c14aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.209+0000 7fe62a7fc700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe62c0c9200 0x7fe62c14aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.209+0000 7fe62a7fc700 1 -- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe62c152a10 msgr2=0x7fe62c1ee1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.209+0000 7fe62a7fc700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe62c152a10 0x7fe62c1ee1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.209+0000 7fe62a7fc700 1 -- 172.21.15.23:0/1015070345 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c009550 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.209+0000 7fe62a7fc700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c0cb7a0 0x7fe62c1579d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe62000e9b0 tx=0x7fe62000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.209+0000 7fe6137fe700 1 -- 172.21.15.23:0/1015070345 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620009710 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.209+0000 7fe6137fe700 1 -- 172.21.15.23:0/1015070345 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62000ee50 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.209+0000 7fe6317ff700 1 -- 172.21.15.23:0/1015070345 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62c159f50 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.209+0000 7fe6137fe700 1 -- 172.21.15.23:0/1015070345 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62000ca50 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.210+0000 7fe6317ff700 1 -- 172.21.15.23:0/1015070345 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62c1f1a00 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.211+0000 7fe6317ff700 1 -- 172.21.15.23:0/1015070345 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62c1f2030 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.211+0000 7fe6137fe700 1 -- 172.21.15.23:0/1015070345 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe62000cbb0 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.212+0000 7fe6137fe700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe61806ad30 0x7fe61806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:21.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.212+0000 7fe6137fe700 1 -- 172.21.15.23:0/1015070345 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 7041+0+0 (secure 0 0 0) 0x7fe620089910 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.213+0000 7fe62affd700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe61806ad30 0x7fe61806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:21.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.213+0000 7fe62affd700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe61806ad30 0x7fe61806d1e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe61c014040 tx=0x7fe61c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:21.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.217+0000 7fe6137fe700 1 -- 172.21.15.23:0/1015070345 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe620057fc0 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.330 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:04:21.331 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:04:21.331 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:04:21.332 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:04:21.409 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-06-22T08:04:21.409 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-06-22T08:04:21.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.487+0000 7fe6317ff700 1 -- 172.21.15.23:0/1015070345 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe62c0ccf70 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.488+0000 7fe6137fe700 1 -- 172.21.15.23:0/1015070345 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5513 (secure 0 0 0) 0x7fe620017070 con 0x7fe62c0cb7a0 2023-06-22T08:04:21.491 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:21.492 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:04:16.314372+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:21.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 -- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe61806ad30 msgr2=0x7fe61806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:21.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe61806ad30 0x7fe61806d1e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe61c014040 tx=0x7fe61c008030 comp rx=0 tx=0).stop 2023-06-22T08:04:21.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 -- 172.21.15.23:0/1015070345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c0cb7a0 msgr2=0x7fe62c1579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:21.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c0cb7a0 0x7fe62c1579d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe62000e9b0 tx=0x7fe62000ecc0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 -- 172.21.15.23:0/1015070345 shutdown_connections 2023-06-22T08:04:21.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe61806ad30 0x7fe61806d1e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe62c152a10 0x7fe62c1ee1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe62c0c9200 0x7fe62c14aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 --2- 172.21.15.23:0/1015070345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe62c0cb7a0 0x7fe62c1579d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:21.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 -- 172.21.15.23:0/1015070345 >> 172.21.15.23:0/1015070345 conn(0x7fe62c065b90 msgr2=0x7fe62c0cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:21.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 -- 172.21.15.23:0/1015070345 shutdown_connections 2023-06-22T08:04:21.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:21.491+0000 7fe6317ff700 1 -- 172.21.15.23:0/1015070345 wait complete. 2023-06-22T08:04:21.496 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 38 2023-06-22T08:04:21.512 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:04:23.514 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:23.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.721+0000 7fe3e07ff700 1 Processor -- start 2023-06-22T08:04:23.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.721+0000 7fe3e07ff700 1 -- start start 2023-06-22T08:04:23.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.722+0000 7fe3e07ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c9fd0 0x7fe3d80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:23.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.722+0000 7fe3e07ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d815c370 con 0x7fe3d80c40b0 2023-06-22T08:04:23.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.722+0000 7fe3e07ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d815bda0 con 0x7fe3d8152ce0 2023-06-22T08:04:23.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.722+0000 7fe3e07ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d81530b0 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.722+0000 7fe3ded9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe3d80c40b0 0x7fe3d80c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:39348/0 (socket says 172.21.15.23:39348) 2023-06-22T08:04:23.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.722+0000 7fe3ded9c700 1 -- 172.21.15.23:0/597980197 learned_addr learned my addr 172.21.15.23:0/597980197 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:04:23.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.722+0000 7fe3ddd9a700 1 --2- 172.21.15.23:0/597980197 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c9fd0 0x7fe3d80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:23.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.723+0000 7fe3dd599700 1 -- 172.21.15.23:0/597980197 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644326938 0 0) 0x7fe3d815c370 con 0x7fe3d80c40b0 2023-06-22T08:04:23.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.723+0000 7fe3dd599700 1 -- 172.21.15.23:0/597980197 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c40035b0 con 0x7fe3d80c40b0 2023-06-22T08:04:23.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.723+0000 7fe3dd599700 1 -- 172.21.15.23:0/597980197 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221764690 0 0) 0x7fe3d815bda0 con 0x7fe3d8152ce0 2023-06-22T08:04:23.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.723+0000 7fe3ddd9a700 1 -- 172.21.15.23:0/597980197 >> v1:172.21.15.67:6789/0 conn(0x7fe3d8152ce0 legacy=0x7fe3d815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:23.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.723+0000 7fe3ddd9a700 1 -- 172.21.15.23:0/597980197 >> v1:172.21.15.23:6789/0 conn(0x7fe3d80c40b0 legacy=0x7fe3d80c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:23.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.723+0000 7fe3ddd9a700 1 -- 172.21.15.23:0/597980197 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d80c9460 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.723+0000 7fe3ddd9a700 1 --2- 172.21.15.23:0/597980197 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c9fd0 0x7fe3d80cf1b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009870 tx=0x7fe3c8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98c7781b3c2132b5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:23.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.723+0000 7fe3dd599700 1 -- 172.21.15.23:0/597980197 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8011830 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.723+0000 7fe3dd599700 1 -- 172.21.15.23:0/597980197 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c8011990 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.723+0000 7fe3dd599700 1 -- 172.21.15.23:0/597980197 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8011d20 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.724+0000 7fe3e07ff700 1 -- 172.21.15.23:0/597980197 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c9fd0 msgr2=0x7fe3d80cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:23.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.724+0000 7fe3e07ff700 1 --2- 172.21.15.23:0/597980197 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c9fd0 0x7fe3d80cf1b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009870 tx=0x7fe3c8009b80 comp rx=0 tx=0).stop 2023-06-22T08:04:23.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.724+0000 7fe3e07ff700 1 -- 172.21.15.23:0/597980197 shutdown_connections 2023-06-22T08:04:23.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.724+0000 7fe3e07ff700 1 --2- 172.21.15.23:0/597980197 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c9fd0 0x7fe3d80cf1b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:23.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.724+0000 7fe3e07ff700 1 -- 172.21.15.23:0/597980197 >> 172.21.15.23:0/597980197 conn(0x7fe3d806ae30 msgr2=0x7fe3d806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:23.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.724+0000 7fe3e07ff700 1 -- 172.21.15.23:0/597980197 shutdown_connections 2023-06-22T08:04:23.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.724+0000 7fe3e07ff700 1 -- 172.21.15.23:0/597980197 wait complete. 2023-06-22T08:04:23.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.726+0000 7fe3e07ff700 1 Processor -- start 2023-06-22T08:04:23.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.726+0000 7fe3e07ff700 1 -- start start 2023-06-22T08:04:23.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3e07ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c40b0 0x7fe3d8154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:23.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3e07ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3d80c9fd0 0x7fe3d81f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:23.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3e07ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3d8152ce0 0x7fe3d814a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:23.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3e07ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d8155570 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3e07ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d8154160 con 0x7fe3d8152ce0 2023-06-22T08:04:23.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3e07ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d814d100 con 0x7fe3d80c40b0 2023-06-22T08:04:23.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3ddd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3d80c9fd0 0x7fe3d81f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:23.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3ddd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3d80c9fd0 0x7fe3d81f1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46114/0 (socket says 172.21.15.23:46114) 2023-06-22T08:04:23.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3ddd9a700 1 -- 172.21.15.23:0/1808095189 learned_addr learned my addr 172.21.15.23:0/1808095189 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:23.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3ded9c700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3d8152ce0 0x7fe3d814a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:23.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3de59b700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c40b0 0x7fe3d8154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:23.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3ddd9a700 1 -- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c40b0 msgr2=0x7fe3d8154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:23.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3ddd9a700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c40b0 0x7fe3d8154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:23.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3ddd9a700 1 -- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3d8152ce0 msgr2=0x7fe3d814a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:23.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3ddd9a700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3d8152ce0 0x7fe3d814a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:23.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.727+0000 7fe3ddd9a700 1 -- 172.21.15.23:0/1808095189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c80094a0 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.728+0000 7fe3ddd9a700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3d80c9fd0 0x7fe3d81f1470 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80090b0 tx=0x7fe3c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:23.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.728+0000 7fe3ceffd700 1 -- 172.21.15.23:0/1808095189 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c801a7e0 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.728+0000 7fe3ceffd700 1 -- 172.21.15.23:0/1808095189 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c801a940 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.728+0000 7fe3e07ff700 1 -- 172.21.15.23:0/1808095189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d8154ac0 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.728+0000 7fe3ceffd700 1 -- 172.21.15.23:0/1808095189 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c801acd0 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.728+0000 7fe3e07ff700 1 -- 172.21.15.23:0/1808095189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d8153d80 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.729+0000 7fe3e07ff700 1 -- 172.21.15.23:0/1808095189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d8153f40 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.730+0000 7fe3ceffd700 1 -- 172.21.15.23:0/1808095189 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe3c8022070 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.731+0000 7fe3ceffd700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3bc06ace0 0x7fe3bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:23.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.731+0000 7fe3ceffd700 1 -- 172.21.15.23:0/1808095189 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 7041+0+0 (secure 0 0 0) 0x7fe3c808a950 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.731+0000 7fe3de59b700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3bc06ace0 0x7fe3bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:23.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.732+0000 7fe3de59b700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3bc06ace0 0x7fe3bc06d190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4000f60 tx=0x7fe3d4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:23.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.735+0000 7fe3ceffd700 1 -- 172.21.15.23:0/1808095189 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe3c80556a0 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.992+0000 7fe3e07ff700 1 -- 172.21.15.23:0/1808095189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3d81f21b0 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.992+0000 7fe3ceffd700 1 -- 172.21.15.23:0/1808095189 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5513 (secure 0 0 0) 0x7fe3d81f21b0 con 0x7fe3d80c9fd0 2023-06-22T08:04:23.997 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:23.997 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:04:21.690852+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:23.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 -- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3bc06ace0 msgr2=0x7fe3bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:23.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3bc06ace0 0x7fe3bc06d190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4000f60 tx=0x7fe3d4009040 comp rx=0 tx=0).stop 2023-06-22T08:04:23.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 -- 172.21.15.23:0/1808095189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3d80c9fd0 msgr2=0x7fe3d81f1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:23.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3d80c9fd0 0x7fe3d81f1470 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80090b0 tx=0x7fe3c8004750 comp rx=0 tx=0).stop 2023-06-22T08:04:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 -- 172.21.15.23:0/1808095189 shutdown_connections 2023-06-22T08:04:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe3bc06ace0 0x7fe3bc06d190 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3d8152ce0 0x7fe3d814a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe3d80c40b0 0x7fe3d8154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 --2- 172.21.15.23:0/1808095189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3d80c9fd0 0x7fe3d81f1470 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 -- 172.21.15.23:0/1808095189 >> 172.21.15.23:0/1808095189 conn(0x7fe3d806ae30 msgr2=0x7fe3d814f130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 -- 172.21.15.23:0/1808095189 shutdown_connections 2023-06-22T08:04:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:23.995+0000 7fe3e07ff700 1 -- 172.21.15.23:0/1808095189 wait complete. 2023-06-22T08:04:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 39 2023-06-22T08:04:24.017 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=4, replacing gid=4257 2023-06-22T08:04:24.017 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.d 2023-06-22T08:04:25.610 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:04:25.607+0000 7fa5334b2700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-06-22T08:04:25.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:04:25.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:25.738 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/bar 2023-06-22T08:04:26.702 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/foo 2023-06-22T08:04:26.713 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/foo 2023-06-22T08:04:26.722 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:26.727 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/bar/a 2023-06-22T08:04:30.018 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-06-22T08:04:30.019 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-22T08:04:30.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-22T08:04:30.022 INFO:tasks.ceph.mds.d:Started 2023-06-22T08:04:30.022 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:30.114 INFO:tasks.ceph.mds.d.smithi067.stdout:starting mds.d at 2023-06-22T08:04:30.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.228+0000 7fb5aa7ff700 1 Processor -- start 2023-06-22T08:04:30.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.228+0000 7fb5aa7ff700 1 -- start start 2023-06-22T08:04:30.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.230+0000 7fb5aa7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb5a4152e10 0x7fb5a4069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:30.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.230+0000 7fb5aa7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5a40c5740 0x7fb5a41593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:30.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.230+0000 7fb5aa7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a4158f60 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.230+0000 7fb5aa7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a406bd20 con 0x7fb5a4152e10 2023-06-22T08:04:30.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.230+0000 7fb5aa7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a40cc450 con 0x7fb5a40c5740 2023-06-22T08:04:30.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.230+0000 7fb5a37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb5a4152e10 0x7fb5a4069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:30.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.230+0000 7fb5a37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb5a4152e10 0x7fb5a4069eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35112/0 (socket says 172.21.15.23:35112) 2023-06-22T08:04:30.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.230+0000 7fb5a8d9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5a40c5740 0x7fb5a41593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:30.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.230+0000 7fb5a37fe700 1 -- 172.21.15.23:0/4292169185 learned_addr learned my addr 172.21.15.23:0/4292169185 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:30.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.230+0000 7fb5a3fff700 1 --1- 172.21.15.23:0/4292169185 >> v1:172.21.15.23:6789/0 conn(0x7fb5a40cb6b0 0x7fb5a4152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:45354/0 (socket says 172.21.15.23:45354) 2023-06-22T08:04:30.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.231+0000 7fb5a37fe700 1 -- 172.21.15.23:0/4292169185 >> v2:172.21.15.67:3300/0 conn(0x7fb5a40c5740 msgr2=0x7fb5a41593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:30.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.231+0000 7fb5a37fe700 1 --2- 172.21.15.23:0/4292169185 >> v2:172.21.15.67:3300/0 conn(0x7fb5a40c5740 0x7fb5a41593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:30.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.231+0000 7fb5a37fe700 1 -- 172.21.15.23:0/4292169185 >> v1:172.21.15.23:6789/0 conn(0x7fb5a40cb6b0 legacy=0x7fb5a4152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:30.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.231+0000 7fb5a37fe700 1 -- 172.21.15.23:0/4292169185 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a4158980 con 0x7fb5a4152e10 2023-06-22T08:04:30.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.231+0000 7fb5a8d9c700 1 --2- 172.21.15.23:0/4292169185 >> v2:172.21.15.67:3300/0 conn(0x7fb5a40c5740 0x7fb5a41593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:04:30.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.231+0000 7fb5a37fe700 1 --2- 172.21.15.23:0/4292169185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a4152e10 0x7fb5a4069eb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4159930 tx=0x7fb598009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e8a691607d47d2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:30.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.231+0000 7fb5a2ffd700 1 -- 172.21.15.23:0/4292169185 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598011830 con 0x7fb5a4152e10 2023-06-22T08:04:30.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.231+0000 7fb5a2ffd700 1 -- 172.21.15.23:0/4292169185 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb598011990 con 0x7fb5a4152e10 2023-06-22T08:04:30.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.231+0000 7fb5a2ffd700 1 -- 172.21.15.23:0/4292169185 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598011d20 con 0x7fb5a4152e10 2023-06-22T08:04:30.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.232+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/4292169185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a4152e10 msgr2=0x7fb5a4069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:30.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.232+0000 7fb5aa7ff700 1 --2- 172.21.15.23:0/4292169185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a4152e10 0x7fb5a4069eb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4159930 tx=0x7fb598009bd0 comp rx=0 tx=0).stop 2023-06-22T08:04:30.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.232+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/4292169185 shutdown_connections 2023-06-22T08:04:30.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.232+0000 7fb5aa7ff700 1 --2- 172.21.15.23:0/4292169185 >> v2:172.21.15.67:3300/0 conn(0x7fb5a40c5740 0x7fb5a41593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:30.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.232+0000 7fb5aa7ff700 1 --2- 172.21.15.23:0/4292169185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a4152e10 0x7fb5a4069eb0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:30.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.232+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/4292169185 >> 172.21.15.23:0/4292169185 conn(0x7fb5a4070d80 msgr2=0x7fb5a4071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:30.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.232+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/4292169185 shutdown_connections 2023-06-22T08:04:30.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.232+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/4292169185 wait complete. 2023-06-22T08:04:30.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.234+0000 7fb5aa7ff700 1 Processor -- start 2023-06-22T08:04:30.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.235+0000 7fb5aa7ff700 1 -- start start 2023-06-22T08:04:30.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5aa7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a40cb6b0 0x7fb5a40c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:30.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5aa7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5a4152e10 0x7fb5a41511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:30.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5aa7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a414b9c0 0x7fb5a41f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:30.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5aa7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a40c4a10 con 0x7fb5a414b9c0 2023-06-22T08:04:30.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5aa7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a4150a20 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5aa7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a40c3150 con 0x7fb5a4152e10 2023-06-22T08:04:30.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a40cb6b0 0x7fb5a40c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:30.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a8d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a414b9c0 0x7fb5a41f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:30.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a37fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5a4152e10 0x7fb5a41511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:30.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a40cb6b0 0x7fb5a40c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:39166/0 (socket says 172.21.15.23:39166) 2023-06-22T08:04:30.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a3fff700 1 -- 172.21.15.23:0/1365498178 learned_addr learned my addr 172.21.15.23:0/1365498178 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:30.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a3fff700 1 -- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5a4152e10 msgr2=0x7fb5a41511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:30.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a3fff700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5a4152e10 0x7fb5a41511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:30.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a3fff700 1 -- 172.21.15.23:0/1365498178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a414b9c0 msgr2=0x7fb5a41f1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:30.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a3fff700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a414b9c0 0x7fb5a41f1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:30.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a3fff700 1 -- 172.21.15.23:0/1365498178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c009a70 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.236+0000 7fb5a8d9c700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a414b9c0 0x7fb5a41f1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:04:30.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.237+0000 7fb5a3fff700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a40cb6b0 0x7fb5a40c7440 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb59400c550 tx=0x7fb59400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:30.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.237+0000 7fb5a0ff9700 1 -- 172.21.15.23:0/1365498178 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594005cb0 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.237+0000 7fb5a0ff9700 1 -- 172.21.15.23:0/1365498178 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb594005e10 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.237+0000 7fb5a0ff9700 1 -- 172.21.15.23:0/1365498178 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5940072d0 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.237+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/1365498178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb598009550 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.237+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/1365498178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a4150cc0 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.238+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/1365498178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a406b550 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.239+0000 7fb5a0ff9700 1 -- 172.21.15.23:0/1365498178 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb594007430 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.240+0000 7fb5a0ff9700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb59006ace0 0x7fb59006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:30.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.240+0000 7fb5a0ff9700 1 -- 172.21.15.23:0/1365498178 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 7041+0+0 (secure 0 0 0) 0x7fb594088c10 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.240+0000 7fb5a37fe700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb59006ace0 0x7fb59006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:30.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.241+0000 7fb5a37fe700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb59006ace0 0x7fb59006d190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4159930 tx=0x7fb598008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:30.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.245+0000 7fb5a0ff9700 1 -- 172.21.15.23:0/1365498178 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb594053860 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.496+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/1365498178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5a404ea20 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.497+0000 7fb5a0ff9700 1 -- 172.21.15.23:0/1365498178 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5513 (secure 0 0 0) 0x7fb594017020 con 0x7fb5a40cb6b0 2023-06-22T08:04:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:30.502 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:04:21.690852+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb59006ace0 msgr2=0x7fb59006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:30.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb59006ace0 0x7fb59006d190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4159930 tx=0x7fb598008030 comp rx=0 tx=0).stop 2023-06-22T08:04:30.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a40cb6b0 msgr2=0x7fb5a40c7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:30.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a40cb6b0 0x7fb5a40c7440 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb59400c550 tx=0x7fb59400c860 comp rx=0 tx=0).stop 2023-06-22T08:04:30.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/1365498178 shutdown_connections 2023-06-22T08:04:30.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a414b9c0 0x7fb5a41f1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:30.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb59006ace0 0x7fb59006d190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a40cb6b0 0x7fb5a40c7440 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 --2- 172.21.15.23:0/1365498178 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5a4152e10 0x7fb5a41511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/1365498178 >> 172.21.15.23:0/1365498178 conn(0x7fb5a4070d80 msgr2=0x7fb5a4157880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:30.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/1365498178 shutdown_connections 2023-06-22T08:04:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:30.500+0000 7fb5aa7ff700 1 -- 172.21.15.23:0/1365498178 wait complete. 2023-06-22T08:04:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 39 2023-06-22T08:04:30.523 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-06-22T08:04:30.722 INFO:tasks.workunit.client.0.smithi023.stdout: File: .snap/bar/a 2023-06-22T08:04:30.722 INFO:tasks.workunit.client.0.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-06-22T08:04:30.723 INFO:tasks.workunit.client.0.smithi023.stdout:Device: fdh/253d Inode: 1099511632724 Links: 2 2023-06-22T08:04:30.723 INFO:tasks.workunit.client.0.smithi023.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1265/ ubuntu) 2023-06-22T08:04:30.723 INFO:tasks.workunit.client.0.smithi023.stdout:Context: system_u:object_r:cephfs_t:s0 2023-06-22T08:04:30.723 INFO:tasks.workunit.client.0.smithi023.stdout:Access: 2023-06-22 08:04:25.732970139 +0000 2023-06-22T08:04:30.723 INFO:tasks.workunit.client.0.smithi023.stdout:Modify: 2023-06-22 08:04:25.732970139 +0000 2023-06-22T08:04:30.723 INFO:tasks.workunit.client.0.smithi023.stdout:Change: 2023-06-22 08:04:25.732970139 +0000 2023-06-22T08:04:30.723 INFO:tasks.workunit.client.0.smithi023.stdout: Birth: 2023-06-22 08:04:25.732970139 +0000 2023-06-22T08:04:30.724 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/bar 2023-06-22T08:04:30.730 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:04:30.731 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:04:30.731 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:04:30.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:04:30.806 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-06-22T08:04:30.806 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-06-22T08:04:30.862 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:04:30.863 INFO:tasks.workunit.client.0.smithi023.stderr:++ seq 1 20 2023-06-22T08:04:30.864 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:30.864 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:30.868 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:30.868 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:30.882 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:30.889 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:30.891 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:31.126 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:31.265 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:31.265 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:31.265 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:31.266 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:31.266 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:31.266 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:31.266 INFO:tasks.workunit.client.0.smithi023.stdout: 146932 ? 00:00:00 ps 2023-06-22T08:04:31.266 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:31.279 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:31.284 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:31.292 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:31.292 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:31.296 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:31.296 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:31.308 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:31.316 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:31.318 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:31.417 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:31.518 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:31.519 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:31.519 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:31.519 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:31.519 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:31.519 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:31.520 INFO:tasks.workunit.client.0.smithi023.stdout: 146942 ? 00:00:00 ps 2023-06-22T08:04:31.520 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:31.533 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:31.537 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:31.545 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:31.546 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:31.550 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:31.550 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:31.561 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:31.569 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:31.571 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:31.658 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:31.761 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:31.761 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:31.761 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:31.761 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:31.762 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:31.762 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:31.762 INFO:tasks.workunit.client.0.smithi023.stdout: 146952 ? 00:00:00 ps 2023-06-22T08:04:31.763 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:31.776 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:31.780 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:31.789 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:31.789 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:31.793 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:31.793 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:31.805 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:31.813 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:31.815 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:31.892 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:31.994 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:31.995 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:31.995 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:31.995 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:31.995 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:31.995 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:31.995 INFO:tasks.workunit.client.0.smithi023.stdout: 146962 ? 00:00:00 ps 2023-06-22T08:04:31.996 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:32.009 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:32.014 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:32.022 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:32.022 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:32.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:32.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:32.039 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:32.047 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:32.048 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:32.134 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:32.235 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:32.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:32.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:32.235 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:32.236 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:32.236 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:32.236 INFO:tasks.workunit.client.0.smithi023.stdout: 146972 ? 00:00:00 ps 2023-06-22T08:04:32.236 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:32.250 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:32.255 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:32.265 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:32.265 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:32.270 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:32.270 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:32.283 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:32.291 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:32.292 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:32.359 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:32.462 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:32.463 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:32.463 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:32.463 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:32.463 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:32.464 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:32.464 INFO:tasks.workunit.client.0.smithi023.stdout: 146982 ? 00:00:00 ps 2023-06-22T08:04:32.464 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:32.476 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:32.480 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:32.489 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:32.489 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:32.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:32.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:32.505 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:32.513 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:32.514 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:32.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:04:32.577 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:32.681 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:32.681 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:32.681 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:32.681 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:32.681 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:32.682 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:32.682 INFO:tasks.workunit.client.0.smithi023.stdout: 147011 ? 00:00:00 ps 2023-06-22T08:04:32.682 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:32.695 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:32.700 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:32.708 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:32.709 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:32.713 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:32.713 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:32.726 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:32.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.728+0000 7f5f1b9ff700 1 Processor -- start 2023-06-22T08:04:32.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.728+0000 7f5f1b9ff700 1 -- start start 2023-06-22T08:04:32.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.730+0000 7f5f1b9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14152d70 0x7f5f140c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:32.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.730+0000 7f5f1b9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f14159330 0x7f5f140cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:32.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.730+0000 7f5f1b9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5f140cb800 0x7f5f140c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:32.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.730+0000 7f5f1b9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f140c95e0 con 0x7f5f140cb800 2023-06-22T08:04:32.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.730+0000 7f5f1b9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f1415bea0 con 0x7f5f14159330 2023-06-22T08:04:32.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.730+0000 7f5f1b9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1415c160 con 0x7f5f14152d70 2023-06-22T08:04:32.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f1979b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14152d70 0x7f5f140c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:32.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f18f9a700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f14159330 0x7f5f140cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:32.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f1979b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14152d70 0x7f5f140c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:34134/0 (socket says 172.21.15.23:34134) 2023-06-22T08:04:32.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f1979b700 1 -- 172.21.15.23:0/3381623255 learned_addr learned my addr 172.21.15.23:0/3381623255 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:32.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f19f9c700 1 --2- 172.21.15.23:0/3381623255 >> v2:172.21.15.23:3300/0 conn(0x7f5f140cb800 0x7f5f140c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:32.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f1979b700 1 -- 172.21.15.23:0/3381623255 >> v2:172.21.15.67:3300/0 conn(0x7f5f14159330 msgr2=0x7f5f140cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:32.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f1979b700 1 --2- 172.21.15.23:0/3381623255 >> v2:172.21.15.67:3300/0 conn(0x7f5f14159330 0x7f5f140cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:32.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f1979b700 1 -- 172.21.15.23:0/3381623255 >> v2:172.21.15.23:3300/0 conn(0x7f5f140cb800 msgr2=0x7f5f140c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:04:32.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f1979b700 1 --2- 172.21.15.23:0/3381623255 >> v2:172.21.15.23:3300/0 conn(0x7f5f140cb800 0x7f5f140c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:32.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f1979b700 1 -- 172.21.15.23:0/3381623255 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1415c440 con 0x7f5f14152d70 2023-06-22T08:04:32.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f18f9a700 1 --2- 172.21.15.23:0/3381623255 >> v2:172.21.15.67:3300/0 conn(0x7f5f14159330 0x7f5f140cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:04:32.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f19f9c700 1 --2- 172.21.15.23:0/3381623255 >> v2:172.21.15.23:3300/0 conn(0x7f5f140cb800 0x7f5f140c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:04:32.740 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:32.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.731+0000 7f5f1979b700 1 --2- 172.21.15.23:0/3381623255 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14152d70 0x7f5f140c97d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5f140737d0 tx=0x7f5f10009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5145b1bc2819ee4e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:32.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.732+0000 7f5f0bfff700 1 -- 172.21.15.23:0/3381623255 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10011830 con 0x7f5f14152d70 2023-06-22T08:04:32.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.732+0000 7f5f0bfff700 1 -- 172.21.15.23:0/3381623255 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f10011990 con 0x7f5f14152d70 2023-06-22T08:04:32.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.732+0000 7f5f0bfff700 1 -- 172.21.15.23:0/3381623255 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10011d20 con 0x7f5f14152d70 2023-06-22T08:04:32.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.732+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/3381623255 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14152d70 msgr2=0x7f5f140c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:32.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.732+0000 7f5f1b9ff700 1 --2- 172.21.15.23:0/3381623255 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14152d70 0x7f5f140c97d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5f140737d0 tx=0x7f5f10009bd0 comp rx=0 tx=0).stop 2023-06-22T08:04:32.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.732+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/3381623255 shutdown_connections 2023-06-22T08:04:32.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.732+0000 7f5f1b9ff700 1 --2- 172.21.15.23:0/3381623255 >> v2:172.21.15.23:3300/0 conn(0x7f5f140cb800 0x7f5f140c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:32.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.732+0000 7f5f1b9ff700 1 --2- 172.21.15.23:0/3381623255 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14152d70 0x7f5f140c97d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:32.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.732+0000 7f5f1b9ff700 1 --2- 172.21.15.23:0/3381623255 >> v2:172.21.15.67:3300/0 conn(0x7f5f14159330 0x7f5f140cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:32.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.732+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/3381623255 >> 172.21.15.23:0/3381623255 conn(0x7f5f1406ae30 msgr2=0x7f5f1406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:32.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.733+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/3381623255 shutdown_connections 2023-06-22T08:04:32.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.733+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/3381623255 wait complete. 2023-06-22T08:04:32.743 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:32.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.735+0000 7f5f1b9ff700 1 Processor -- start 2023-06-22T08:04:32.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.735+0000 7f5f1b9ff700 1 -- start start 2023-06-22T08:04:32.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f1b9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f140cb800 0x7f5f141f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:32.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f1b9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f14152d70 0x7f5f140cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f1b9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14159330 0x7f5f1414c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f1b9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140c5930 con 0x7f5f140cb800 2023-06-22T08:04:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f1b9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140b3380 con 0x7f5f14152d70 2023-06-22T08:04:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f1b9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f140b3bd0 con 0x7f5f14159330 2023-06-22T08:04:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f18f9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f14152d70 0x7f5f140cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f19f9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14159330 0x7f5f1414c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f1979b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f140cb800 0x7f5f141f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f18f9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f14152d70 0x7f5f140cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:39180/0 (socket says 172.21.15.23:39180) 2023-06-22T08:04:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.736+0000 7f5f18f9a700 1 -- 172.21.15.23:0/1819465738 learned_addr learned my addr 172.21.15.23:0/1819465738 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:04:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f18f9a700 1 -- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14159330 msgr2=0x7f5f1414c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f18f9a700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14159330 0x7f5f1414c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f18f9a700 1 -- 172.21.15.23:0/1819465738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f140cb800 msgr2=0x7f5f141f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f18f9a700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f140cb800 0x7f5f141f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f18f9a700 1 -- 172.21.15.23:0/1819465738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c009a70 con 0x7f5f14152d70 2023-06-22T08:04:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f18f9a700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f14152d70 0x7f5f140cd5e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400ebe0 tx=0x7f5f0400eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f09ffb700 1 -- 172.21.15.23:0/1819465738 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f04009760 con 0x7f5f14152d70 2023-06-22T08:04:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f09ffb700 1 -- 172.21.15.23:0/1819465738 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0400c850 con 0x7f5f14152d70 2023-06-22T08:04:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f09ffb700 1 -- 172.21.15.23:0/1819465738 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0400cbe0 con 0x7f5f14152d70 2023-06-22T08:04:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/1819465738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f040094a0 con 0x7f5f14152d70 2023-06-22T08:04:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.737+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/1819465738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f10009550 con 0x7f5f14152d70 2023-06-22T08:04:32.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.738+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/1819465738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f140cbd50 con 0x7f5f14152d70 2023-06-22T08:04:32.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.740+0000 7f5f09ffb700 1 -- 172.21.15.23:0/1819465738 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5f0400cd40 con 0x7f5f14152d70 2023-06-22T08:04:32.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.740+0000 7f5f09ffb700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ef806ad10 0x7f5ef806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:04:32.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.741+0000 7f5f09ffb700 1 -- 172.21.15.23:0/1819465738 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f5f040897c0 con 0x7f5f14152d70 2023-06-22T08:04:32.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.741+0000 7f5f1979b700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ef806ad10 0x7f5ef806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:04:32.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.741+0000 7f5f1979b700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ef806ad10 0x7f5ef806d1c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009c10 tx=0x7f5f10008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:04:32.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:32.745+0000 7f5f09ffb700 1 -- 172.21.15.23:0/1819465738 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5f04054480 con 0x7f5f14152d70 2023-06-22T08:04:32.836 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:32.942 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:32.942 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:32.942 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:32.942 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:32.943 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:32.943 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:32.943 INFO:tasks.workunit.client.0.smithi023.stdout: 147046 ? 00:00:00 ps 2023-06-22T08:04:32.943 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:32.957 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:32.961 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:32.969 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:32.969 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:32.974 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:32.974 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:32.987 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:32.996 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:32.997 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.005+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/1819465738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f14152510 con 0x7f5f14152d70 2023-06-22T08:04:33.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.006+0000 7f5f09ffb700 1 -- 172.21.15.23:0/1819465738 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6269 (secure 0 0 0) 0x7f5f04057aa0 con 0x7f5f14152d70 2023-06-22T08:04:33.010 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:04:33.010 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5060,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:04:21.690852+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:04:33.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ef806ad10 msgr2=0x7f5ef806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:33.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ef806ad10 0x7f5ef806d1c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5f10009c10 tx=0x7f5f10008030 comp rx=0 tx=0).stop 2023-06-22T08:04:33.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f14152d70 msgr2=0x7f5f140cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:04:33.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f14152d70 0x7f5f140cd5e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400ebe0 tx=0x7f5f0400eef0 comp rx=0 tx=0).stop 2023-06-22T08:04:33.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/1819465738 shutdown_connections 2023-06-22T08:04:33.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5f14159330 0x7f5f1414c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:33.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f140cb800 0x7f5f141f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:33.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ef806ad10 0x7f5ef806d1c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:33.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 --2- 172.21.15.23:0/1819465738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f14152d70 0x7f5f140cd5e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:04:33.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/1819465738 >> 172.21.15.23:0/1819465738 conn(0x7f5f1406ae30 msgr2=0x7f5f1415b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:04:33.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/1819465738 shutdown_connections 2023-06-22T08:04:33.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:04:33.009+0000 7f5f1b9ff700 1 -- 172.21.15.23:0/1819465738 wait complete. 2023-06-22T08:04:33.014 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 40 2023-06-22T08:04:33.031 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-06-22T08:04:33.031 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-06-22T08:04:33.086 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:33.188 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:33.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:33.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:33.189 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:33.189 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:33.189 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:33.189 INFO:tasks.workunit.client.0.smithi023.stdout: 147058 ? 00:00:00 ps 2023-06-22T08:04:33.190 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:33.204 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:33.208 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:33.217 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:33.217 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:33.221 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:33.221 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:33.234 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:33.242 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:33.245 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:33.362 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:33.464 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:33.464 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:33.464 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:33.464 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:33.464 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:33.464 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:33.465 INFO:tasks.workunit.client.0.smithi023.stdout: 147068 ? 00:00:00 ps 2023-06-22T08:04:33.465 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:33.478 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:33.482 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:33.491 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:33.492 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:33.496 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:33.496 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:33.508 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:33.516 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:33.518 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:33.587 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:33.692 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:33.692 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:33.692 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:33.693 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:33.693 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:33.693 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:33.693 INFO:tasks.workunit.client.0.smithi023.stdout: 147078 ? 00:00:00 ps 2023-06-22T08:04:33.694 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:33.707 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:33.711 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:33.720 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:33.720 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:33.724 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:33.724 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:33.736 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:33.745 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:33.747 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:33.854 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:33.958 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:33.958 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:33.958 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:33.958 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:33.958 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:33.959 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:33.959 INFO:tasks.workunit.client.0.smithi023.stdout: 147088 ? 00:00:00 ps 2023-06-22T08:04:33.959 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:33.972 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:33.976 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:33.985 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:33.985 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:33.990 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:33.990 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:34.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:34.011 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:34.013 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:34.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:34.181 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:34.181 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:34.182 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:34.182 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:34.182 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:34.182 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:34.183 INFO:tasks.workunit.client.0.smithi023.stdout: 147098 ? 00:00:00 ps 2023-06-22T08:04:34.183 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:34.196 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:34.200 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:34.209 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:34.210 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:34.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:34.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:34.227 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:34.235 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:34.237 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:34.313 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:34.415 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:34.416 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:34.416 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:34.416 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:34.417 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:34.417 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:34.417 INFO:tasks.workunit.client.0.smithi023.stdout: 147108 ? 00:00:00 ps 2023-06-22T08:04:34.418 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:34.430 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:34.434 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:34.442 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:34.443 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:34.447 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:34.447 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:34.459 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:34.466 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:34.468 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:34.538 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:34.642 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:34.642 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:34.643 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:34.643 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:34.643 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:34.644 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:34.644 INFO:tasks.workunit.client.0.smithi023.stdout: 147118 ? 00:00:00 ps 2023-06-22T08:04:34.645 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:34.656 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:34.661 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:34.670 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:34.670 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:34.674 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:34.675 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:34.687 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:34.696 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:34.698 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:34.764 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:34.865 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:34.865 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:34.865 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:34.865 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:34.866 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:34.866 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:34.866 INFO:tasks.workunit.client.0.smithi023.stdout: 147128 ? 00:00:00 ps 2023-06-22T08:04:34.866 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:34.879 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:34.885 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:34.895 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:34.896 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:34.901 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:34.901 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:34.914 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:34.922 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:34.924 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:34.989 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:35.090 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:35.091 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:35.091 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:35.091 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:35.091 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:35.091 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:35.091 INFO:tasks.workunit.client.0.smithi023.stdout: 147138 ? 00:00:00 ps 2023-06-22T08:04:35.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:35.105 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:35.109 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:35.116 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:35.117 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:35.121 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:35.121 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:35.133 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:35.141 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:35.143 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:35.198 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:35.301 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:35.301 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:35.301 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:35.302 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:35.302 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:35.302 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:35.302 INFO:tasks.workunit.client.0.smithi023.stdout: 147148 ? 00:00:00 ps 2023-06-22T08:04:35.303 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:35.318 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:35.323 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:35.333 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:35.333 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:35.338 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:35.338 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:35.351 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:35.360 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:35.362 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:35.448 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:35.552 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:35.553 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:35.553 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:35.553 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:35.553 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:35.554 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:35.554 INFO:tasks.workunit.client.0.smithi023.stdout: 147158 ? 00:00:00 ps 2023-06-22T08:04:35.554 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:35.567 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:35.571 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:35.579 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:04:35.580 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:04:35.584 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/one 2023-06-22T08:04:35.584 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat 2023-06-22T08:04:35.597 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/two 2023-06-22T08:04:35.604 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a/foo 2023-06-22T08:04:35.606 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:04:35.665 INFO:tasks.workunit.client.0.smithi023.stderr:+ ps 2023-06-22T08:04:35.767 INFO:tasks.workunit.client.0.smithi023.stdout: PID TTY TIME CMD 2023-06-22T08:04:35.768 INFO:tasks.workunit.client.0.smithi023.stdout: 1122 ? 00:00:00 systemd 2023-06-22T08:04:35.768 INFO:tasks.workunit.client.0.smithi023.stdout: 1125 ? 00:00:00 (sd-pam) 2023-06-22T08:04:35.768 INFO:tasks.workunit.client.0.smithi023.stdout: 85746 ? 00:00:01 sshd 2023-06-22T08:04:35.768 INFO:tasks.workunit.client.0.smithi023.stdout: 146903 ? 00:00:00 timeout 2023-06-22T08:04:35.768 INFO:tasks.workunit.client.0.smithi023.stdout: 146924 ? 00:00:00 snaptest-double 2023-06-22T08:04:35.768 INFO:tasks.workunit.client.0.smithi023.stdout: 147168 ? 00:00:00 ps 2023-06-22T08:04:35.769 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/one a/.snap/two 2023-06-22T08:04:35.783 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a/foo 2023-06-22T08:04:36.710 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:04:36.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:04:45.741 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a 2023-06-22T08:04:45.760 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:04:45.762 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:04:45.762 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:04:45.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:04:45.842 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-06-22T08:04:45.842 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-06-22T08:04:45.901 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/foo 2023-06-22T08:04:45.913 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2023-06-22T08:04:45.914 INFO:tasks.workunit.client.0.smithi023.stdout:We want ENOENT, not ESTALE, here. 2023-06-22T08:04:45.914 INFO:tasks.workunit.client.0.smithi023.stderr:++ seq 1 100 2023-06-22T08:04:45.915 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.916 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/1 2023-06-22T08:04:45.916 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.920 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2023-06-22T08:04:45.920 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/2 2023-06-22T08:04:45.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.923 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2023-06-22T08:04:45.923 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.924 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/3 2023-06-22T08:04:45.924 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.926 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2023-06-22T08:04:45.926 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.927 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/4 2023-06-22T08:04:45.927 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.929 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2023-06-22T08:04:45.930 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.930 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/5 2023-06-22T08:04:45.930 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.932 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2023-06-22T08:04:45.933 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.933 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/6 2023-06-22T08:04:45.934 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.935 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2023-06-22T08:04:45.936 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.936 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/7 2023-06-22T08:04:45.937 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.939 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2023-06-22T08:04:45.939 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.939 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/8 2023-06-22T08:04:45.940 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.942 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2023-06-22T08:04:45.943 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.943 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/9 2023-06-22T08:04:45.943 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.945 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2023-06-22T08:04:45.946 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.946 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/10 2023-06-22T08:04:45.946 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.948 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2023-06-22T08:04:45.949 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.949 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/11 2023-06-22T08:04:45.950 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.951 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2023-06-22T08:04:45.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/12 2023-06-22T08:04:45.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.955 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2023-06-22T08:04:45.955 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.956 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/13 2023-06-22T08:04:45.956 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.958 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2023-06-22T08:04:45.958 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.959 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/14 2023-06-22T08:04:45.959 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.961 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2023-06-22T08:04:45.962 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.962 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/15 2023-06-22T08:04:45.963 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.965 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2023-06-22T08:04:45.965 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.965 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/16 2023-06-22T08:04:45.966 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.968 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2023-06-22T08:04:45.968 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.969 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/17 2023-06-22T08:04:45.969 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.971 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2023-06-22T08:04:45.971 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.972 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/18 2023-06-22T08:04:45.972 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.974 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2023-06-22T08:04:45.975 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.975 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/19 2023-06-22T08:04:45.975 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.977 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2023-06-22T08:04:45.978 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.978 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/20 2023-06-22T08:04:45.979 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.981 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2023-06-22T08:04:45.981 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.982 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/21 2023-06-22T08:04:45.982 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.984 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2023-06-22T08:04:45.985 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.985 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/22 2023-06-22T08:04:45.985 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.987 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2023-06-22T08:04:45.988 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.988 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/23 2023-06-22T08:04:45.989 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.991 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2023-06-22T08:04:45.991 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.991 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/24 2023-06-22T08:04:45.992 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.994 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2023-06-22T08:04:45.994 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.995 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/25 2023-06-22T08:04:45.995 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:45.997 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2023-06-22T08:04:45.998 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:45.998 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/26 2023-06-22T08:04:45.999 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.001 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2023-06-22T08:04:46.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/27 2023-06-22T08:04:46.003 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.004 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2023-06-22T08:04:46.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/28 2023-06-22T08:04:46.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.007 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2023-06-22T08:04:46.008 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.008 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/29 2023-06-22T08:04:46.009 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.011 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2023-06-22T08:04:46.011 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.012 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/30 2023-06-22T08:04:46.012 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.014 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2023-06-22T08:04:46.015 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.015 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/31 2023-06-22T08:04:46.015 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.018 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2023-06-22T08:04:46.018 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.018 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/32 2023-06-22T08:04:46.019 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.021 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2023-06-22T08:04:46.021 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.022 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/33 2023-06-22T08:04:46.022 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.024 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2023-06-22T08:04:46.025 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.025 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/34 2023-06-22T08:04:46.025 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.027 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2023-06-22T08:04:46.028 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.028 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/35 2023-06-22T08:04:46.029 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.031 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2023-06-22T08:04:46.031 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.032 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/36 2023-06-22T08:04:46.032 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.034 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2023-06-22T08:04:46.034 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.035 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/37 2023-06-22T08:04:46.035 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.038 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2023-06-22T08:04:46.038 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.038 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/38 2023-06-22T08:04:46.039 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.041 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2023-06-22T08:04:46.041 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.041 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/39 2023-06-22T08:04:46.042 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.044 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2023-06-22T08:04:46.045 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.045 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/40 2023-06-22T08:04:46.046 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.048 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2023-06-22T08:04:46.048 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.049 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/41 2023-06-22T08:04:46.049 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.051 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2023-06-22T08:04:46.051 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.052 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/42 2023-06-22T08:04:46.052 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.054 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2023-06-22T08:04:46.055 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.055 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/43 2023-06-22T08:04:46.055 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.058 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2023-06-22T08:04:46.058 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.058 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/44 2023-06-22T08:04:46.059 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.061 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2023-06-22T08:04:46.062 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.062 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/45 2023-06-22T08:04:46.062 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.064 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2023-06-22T08:04:46.064 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.065 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/46 2023-06-22T08:04:46.065 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.068 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2023-06-22T08:04:46.068 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.068 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/47 2023-06-22T08:04:46.069 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.072 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2023-06-22T08:04:46.072 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.072 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/48 2023-06-22T08:04:46.073 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.075 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2023-06-22T08:04:46.076 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.076 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/49 2023-06-22T08:04:46.077 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.078 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2023-06-22T08:04:46.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/50 2023-06-22T08:04:46.080 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.081 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2023-06-22T08:04:46.082 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.082 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/51 2023-06-22T08:04:46.083 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.085 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2023-06-22T08:04:46.085 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.086 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/52 2023-06-22T08:04:46.086 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.089 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2023-06-22T08:04:46.089 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.089 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/53 2023-06-22T08:04:46.090 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.092 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2023-06-22T08:04:46.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/54 2023-06-22T08:04:46.093 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.095 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2023-06-22T08:04:46.095 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.096 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/55 2023-06-22T08:04:46.097 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.098 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2023-06-22T08:04:46.099 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.099 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/56 2023-06-22T08:04:46.100 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.102 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2023-06-22T08:04:46.102 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.102 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/57 2023-06-22T08:04:46.103 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.105 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2023-06-22T08:04:46.105 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.106 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/58 2023-06-22T08:04:46.106 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.108 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2023-06-22T08:04:46.109 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.109 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/59 2023-06-22T08:04:46.110 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.112 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2023-06-22T08:04:46.112 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.113 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/60 2023-06-22T08:04:46.113 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.115 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2023-06-22T08:04:46.115 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.116 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/61 2023-06-22T08:04:46.116 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.118 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2023-06-22T08:04:46.119 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.119 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/62 2023-06-22T08:04:46.120 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.121 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2023-06-22T08:04:46.122 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.122 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/63 2023-06-22T08:04:46.123 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.125 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2023-06-22T08:04:46.125 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.126 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/64 2023-06-22T08:04:46.126 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.128 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2023-06-22T08:04:46.129 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.129 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/65 2023-06-22T08:04:46.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.132 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2023-06-22T08:04:46.132 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.133 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/66 2023-06-22T08:04:46.133 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.135 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2023-06-22T08:04:46.135 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.136 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/67 2023-06-22T08:04:46.136 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.138 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2023-06-22T08:04:46.138 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.139 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/68 2023-06-22T08:04:46.139 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.142 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2023-06-22T08:04:46.142 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.142 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/69 2023-06-22T08:04:46.143 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.145 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2023-06-22T08:04:46.145 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.146 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/70 2023-06-22T08:04:46.146 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.148 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2023-06-22T08:04:46.149 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.149 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/71 2023-06-22T08:04:46.150 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.152 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2023-06-22T08:04:46.152 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.152 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/72 2023-06-22T08:04:46.153 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.155 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2023-06-22T08:04:46.155 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.155 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/73 2023-06-22T08:04:46.156 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.158 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2023-06-22T08:04:46.158 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.159 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/74 2023-06-22T08:04:46.159 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.161 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2023-06-22T08:04:46.162 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.162 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/75 2023-06-22T08:04:46.162 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.164 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2023-06-22T08:04:46.165 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.165 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/76 2023-06-22T08:04:46.166 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.168 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2023-06-22T08:04:46.168 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.169 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/77 2023-06-22T08:04:46.169 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.171 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2023-06-22T08:04:46.171 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.172 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/78 2023-06-22T08:04:46.172 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.174 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2023-06-22T08:04:46.175 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.175 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/79 2023-06-22T08:04:46.176 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.177 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2023-06-22T08:04:46.178 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.178 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/80 2023-06-22T08:04:46.179 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.181 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2023-06-22T08:04:46.181 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.182 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/81 2023-06-22T08:04:46.182 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.184 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2023-06-22T08:04:46.184 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.185 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/82 2023-06-22T08:04:46.185 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.187 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2023-06-22T08:04:46.187 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.188 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/83 2023-06-22T08:04:46.188 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.191 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2023-06-22T08:04:46.191 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.191 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/84 2023-06-22T08:04:46.192 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.194 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2023-06-22T08:04:46.194 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.194 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/85 2023-06-22T08:04:46.195 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.197 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2023-06-22T08:04:46.197 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.198 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/86 2023-06-22T08:04:46.198 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.200 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2023-06-22T08:04:46.201 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.201 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/87 2023-06-22T08:04:46.201 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.203 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2023-06-22T08:04:46.204 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.204 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/88 2023-06-22T08:04:46.205 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.206 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2023-06-22T08:04:46.207 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.207 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/89 2023-06-22T08:04:46.208 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.210 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2023-06-22T08:04:46.210 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.210 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/90 2023-06-22T08:04:46.211 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.213 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2023-06-22T08:04:46.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/91 2023-06-22T08:04:46.215 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.217 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2023-06-22T08:04:46.217 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.217 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/92 2023-06-22T08:04:46.218 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.220 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2023-06-22T08:04:46.221 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.221 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/93 2023-06-22T08:04:46.222 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.223 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2023-06-22T08:04:46.224 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.224 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/94 2023-06-22T08:04:46.225 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.227 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2023-06-22T08:04:46.228 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.228 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/95 2023-06-22T08:04:46.228 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.230 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2023-06-22T08:04:46.230 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.231 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/96 2023-06-22T08:04:46.231 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.233 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2023-06-22T08:04:46.234 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.234 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/97 2023-06-22T08:04:46.234 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.236 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2023-06-22T08:04:46.237 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.237 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/98 2023-06-22T08:04:46.238 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.240 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2023-06-22T08:04:46.240 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.241 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/99 2023-06-22T08:04:46.241 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.243 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2023-06-22T08:04:46.243 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 100` 2023-06-22T08:04:46.244 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/foo/100 2023-06-22T08:04:46.244 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'No such file' 2023-06-22T08:04:46.246 INFO:tasks.workunit.client.0.smithi023.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2023-06-22T08:04:46.246 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/foo 2023-06-22T08:04:46.254 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:04:46.255 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:04:46.255 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:04:46.255 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:04:46.329 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-06-22T08:04:46.329 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-06-22T08:04:46.386 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:04:46.386 INFO:tasks.workunit.client.0.smithi023.stderr:+ retried=false 2023-06-22T08:04:46.386 INFO:tasks.workunit.client.0.smithi023.stderr:+ trap -- retry EXIT 2023-06-22T08:04:46.387 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -rf ceph 2023-06-22T08:04:46.388 INFO:tasks.workunit.client.0.smithi023.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-06-22T08:04:46.394 INFO:tasks.workunit.client.0.smithi023.stderr:Cloning into 'ceph'... 2023-06-22T08:05:06.975 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:05:06.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:05:37.074 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:05:37.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:05:42.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:05:42.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.233+0000 7ffa591ff700 1 Processor -- start 2023-06-22T08:05:42.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.233+0000 7ffa591ff700 1 -- start start 2023-06-22T08:05:42.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.234+0000 7ffa591ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa5415a320 0x7ffa540c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:42.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.234+0000 7ffa591ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa540c6a60 con 0x7ffa540c3a40 2023-06-22T08:05:42.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.234+0000 7ffa591ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5414d690 con 0x7ffa5406d050 2023-06-22T08:05:42.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.234+0000 7ffa591ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa54157570 con 0x7ffa5415a320 2023-06-22T08:05:42.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa52d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ffa540c3a40 0x7ffa54159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33754/0 (socket says 172.21.15.23:33754) 2023-06-22T08:05:42.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa5359e700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffa5406d050 0x7ffa54160ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:37458/0 (socket says 172.21.15.23:37458) 2023-06-22T08:05:42.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa5259c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa5415a320 0x7ffa540c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:42.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa52d9d700 1 -- 172.21.15.23:0/4221995811 learned_addr learned my addr 172.21.15.23:0/4221995811 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:05:42.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa51d9b700 1 -- 172.21.15.23:0/4221995811 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871628481 0 0) 0x7ffa540c6a60 con 0x7ffa540c3a40 2023-06-22T08:05:42.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa51d9b700 1 -- 172.21.15.23:0/4221995811 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa3c0035b0 con 0x7ffa540c3a40 2023-06-22T08:05:42.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa5259c700 1 -- 172.21.15.23:0/4221995811 >> v1:172.21.15.67:6789/0 conn(0x7ffa5406d050 legacy=0x7ffa54160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa51d9b700 1 -- 172.21.15.23:0/4221995811 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150257498 0 0) 0x7ffa5414d690 con 0x7ffa5406d050 2023-06-22T08:05:42.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa5259c700 1 -- 172.21.15.23:0/4221995811 >> v1:172.21.15.23:6789/0 conn(0x7ffa540c3a40 legacy=0x7ffa54159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa5259c700 1 -- 172.21.15.23:0/4221995811 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa540c6430 con 0x7ffa5415a320 2023-06-22T08:05:42.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.235+0000 7ffa5259c700 1 --2- 172.21.15.23:0/4221995811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5415a320 0x7ffa540c3e10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009840 tx=0x7ffa48009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42c1060991e3e651 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:42.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.236+0000 7ffa51d9b700 1 -- 172.21.15.23:0/4221995811 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa48011830 con 0x7ffa5415a320 2023-06-22T08:05:42.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.236+0000 7ffa51d9b700 1 -- 172.21.15.23:0/4221995811 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa48011990 con 0x7ffa5415a320 2023-06-22T08:05:42.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.236+0000 7ffa51d9b700 1 -- 172.21.15.23:0/4221995811 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa48011d20 con 0x7ffa5415a320 2023-06-22T08:05:42.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.236+0000 7ffa591ff700 1 -- 172.21.15.23:0/4221995811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5415a320 msgr2=0x7ffa540c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.236+0000 7ffa591ff700 1 --2- 172.21.15.23:0/4221995811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5415a320 0x7ffa540c3e10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009840 tx=0x7ffa48009b50 comp rx=0 tx=0).stop 2023-06-22T08:05:42.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.236+0000 7ffa591ff700 1 -- 172.21.15.23:0/4221995811 shutdown_connections 2023-06-22T08:05:42.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.236+0000 7ffa591ff700 1 --2- 172.21.15.23:0/4221995811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa5415a320 0x7ffa540c3e10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:42.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.236+0000 7ffa591ff700 1 -- 172.21.15.23:0/4221995811 >> 172.21.15.23:0/4221995811 conn(0x7ffa540ca3f0 msgr2=0x7ffa540ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:42.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.236+0000 7ffa591ff700 1 -- 172.21.15.23:0/4221995811 shutdown_connections 2023-06-22T08:05:42.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.236+0000 7ffa591ff700 1 -- 172.21.15.23:0/4221995811 wait complete. 2023-06-22T08:05:42.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.238+0000 7ffa591ff700 1 Processor -- start 2023-06-22T08:05:42.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.238+0000 7ffa591ff700 1 -- start start 2023-06-22T08:05:42.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa591ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa5406d050 0x7ffa540c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:42.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa591ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa540c3a40 0x7ffa541eeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:42.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa591ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa5415a320 0x7ffa541eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:42.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa591ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa541ee5b0 con 0x7ffa5406d050 2023-06-22T08:05:42.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa591ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa540c8630 con 0x7ffa540c3a40 2023-06-22T08:05:42.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa591ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa540c8910 con 0x7ffa5415a320 2023-06-22T08:05:42.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa5259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa540c3a40 0x7ffa541eeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:42.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa5259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa540c3a40 0x7ffa541eeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:55922/0 (socket says 172.21.15.23:55922) 2023-06-22T08:05:42.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa5259c700 1 -- 172.21.15.23:0/3838623498 learned_addr learned my addr 172.21.15.23:0/3838623498 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:42.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa5359e700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa5415a320 0x7ffa541eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:42.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.239+0000 7ffa52d9d700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa5406d050 0x7ffa540c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:42.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa5259c700 1 -- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa5415a320 msgr2=0x7ffa541eeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa5259c700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa5415a320 0x7ffa541eeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:42.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa5259c700 1 -- 172.21.15.23:0/3838623498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa5406d050 msgr2=0x7ffa540c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa5259c700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa5406d050 0x7ffa540c9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:42.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa5259c700 1 -- 172.21.15.23:0/3838623498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa480094a0 con 0x7ffa540c3a40 2023-06-22T08:05:42.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa5359e700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa5415a320 0x7ffa541eeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:05:42.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa5259c700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa540c3a40 0x7ffa541eeab0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffa480090b0 tx=0x7ffa48004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:42.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa3b7fe700 1 -- 172.21.15.23:0/3838623498 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4801a7e0 con 0x7ffa540c3a40 2023-06-22T08:05:42.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa3b7fe700 1 -- 172.21.15.23:0/3838623498 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa4801a940 con 0x7ffa540c3a40 2023-06-22T08:05:42.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa591ff700 1 -- 172.21.15.23:0/3838623498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa540c8bf0 con 0x7ffa540c3a40 2023-06-22T08:05:42.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.240+0000 7ffa3b7fe700 1 -- 172.21.15.23:0/3838623498 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa4801acf0 con 0x7ffa540c3a40 2023-06-22T08:05:42.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.241+0000 7ffa591ff700 1 -- 172.21.15.23:0/3838623498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa540c91b0 con 0x7ffa540c3a40 2023-06-22T08:05:42.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.241+0000 7ffa591ff700 1 -- 172.21.15.23:0/3838623498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5404ea20 con 0x7ffa540c3a40 2023-06-22T08:05:42.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.243+0000 7ffa3b7fe700 1 -- 172.21.15.23:0/3838623498 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffa48022070 con 0x7ffa540c3a40 2023-06-22T08:05:42.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.244+0000 7ffa3b7fe700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa4006ace0 0x7ffa4006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:42.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.244+0000 7ffa3b7fe700 1 -- 172.21.15.23:0/3838623498 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7041+0+0 (secure 0 0 0) 0x7ffa4808a9c0 con 0x7ffa540c3a40 2023-06-22T08:05:42.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.244+0000 7ffa52d9d700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa4006ace0 0x7ffa4006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:42.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.245+0000 7ffa52d9d700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa4006ace0 0x7ffa4006d190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffa44007fd0 tx=0x7ffa44009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:42.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.247+0000 7ffa3b7fe700 1 -- 172.21.15.23:0/3838623498 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffa48055800 con 0x7ffa540c3a40 2023-06-22T08:05:42.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.496+0000 7ffa591ff700 1 -- 172.21.15.23:0/3838623498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa54147b40 con 0x7ffa540c3a40 2023-06-22T08:05:42.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.497+0000 7ffa3b7fe700 1 -- 172.21.15.23:0/3838623498 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6257 (secure 0 0 0) 0x7ffa48058e20 con 0x7ffa540c3a40 2023-06-22T08:05:42.500 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:05:42.500 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5060,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:05:41.716490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:05:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 -- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa4006ace0 msgr2=0x7ffa4006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa4006ace0 0x7ffa4006d190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffa44007fd0 tx=0x7ffa44009040 comp rx=0 tx=0).stop 2023-06-22T08:05:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 -- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa540c3a40 msgr2=0x7ffa541eeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa540c3a40 0x7ffa541eeab0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffa480090b0 tx=0x7ffa48004750 comp rx=0 tx=0).stop 2023-06-22T08:05:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 -- 172.21.15.23:0/3838623498 shutdown_connections 2023-06-22T08:05:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffa5415a320 0x7ffa541eeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa5406d050 0x7ffa540c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffa4006ace0 0x7ffa4006d190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 --2- 172.21.15.23:0/3838623498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa540c3a40 0x7ffa541eeab0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 -- 172.21.15.23:0/3838623498 >> 172.21.15.23:0/3838623498 conn(0x7ffa540ca3f0 msgr2=0x7ffa54070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 -- 172.21.15.23:0/3838623498 shutdown_connections 2023-06-22T08:05:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.500+0000 7ffa591ff700 1 -- 172.21.15.23:0/3838623498 wait complete. 2023-06-22T08:05:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 43 2023-06-22T08:05:42.524 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=3) 2023-06-22T08:05:42.525 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-22T08:05:42.525 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:05:42.584 INFO:tasks.ceph.mds.c:Stopped 2023-06-22T08:05:42.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:05:42.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.822+0000 7f6a249ff700 1 Processor -- start 2023-06-22T08:05:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.822+0000 7f6a249ff700 1 -- start start 2023-06-22T08:05:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.823+0000 7f6a249ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 0x7f6a200c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.823+0000 7f6a249ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2015c370 con 0x7f6a200c9fd0 2023-06-22T08:05:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.823+0000 7f6a249ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2015bda0 con 0x7f6a20152ce0 2023-06-22T08:05:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.823+0000 7f6a249ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201530b0 con 0x7f6a200cf1b0 2023-06-22T08:05:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.824+0000 7f6a1e59c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6a20152ce0 0x7f6a2015c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:37470/0 (socket says 172.21.15.23:37470) 2023-06-22T08:05:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.824+0000 7f6a1e59c700 1 -- 172.21.15.23:0/3473269857 learned_addr learned my addr 172.21.15.23:0/3473269857 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:05:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.824+0000 7f6a1ed9d700 1 --2- 172.21.15.23:0/3473269857 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 0x7f6a200c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.824+0000 7f6a1d59a700 1 -- 172.21.15.23:0/3473269857 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842617570 0 0) 0x7f6a2015bda0 con 0x7f6a20152ce0 2023-06-22T08:05:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.824+0000 7f6a1d59a700 1 -- 172.21.15.23:0/3473269857 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a080035b0 con 0x7f6a20152ce0 2023-06-22T08:05:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.824+0000 7f6a1ed9d700 1 -- 172.21.15.23:0/3473269857 >> v1:172.21.15.67:6789/0 conn(0x7f6a20152ce0 legacy=0x7f6a2015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.824+0000 7f6a1ed9d700 1 -- 172.21.15.23:0/3473269857 >> v1:172.21.15.23:6789/0 conn(0x7f6a200c9fd0 legacy=0x7f6a200c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.824+0000 7f6a1d59a700 1 -- 172.21.15.23:0/3473269857 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416340056 0 0) 0x7f6a2015c370 con 0x7f6a200c9fd0 2023-06-22T08:05:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.824+0000 7f6a1ed9d700 1 -- 172.21.15.23:0/3473269857 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a200cf630 con 0x7f6a200cf1b0 2023-06-22T08:05:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.824+0000 7f6a1ed9d700 1 --2- 172.21.15.23:0/3473269857 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 0x7f6a200c47c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009dc0 tx=0x7f6a1800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce73c642ebeefca0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.825+0000 7f6a1d59a700 1 -- 172.21.15.23:0/3473269857 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1800e070 con 0x7f6a200cf1b0 2023-06-22T08:05:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.825+0000 7f6a1d59a700 1 -- 172.21.15.23:0/3473269857 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1800bb20 con 0x7f6a200cf1b0 2023-06-22T08:05:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.825+0000 7f6a1d59a700 1 -- 172.21.15.23:0/3473269857 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1800bed0 con 0x7f6a200cf1b0 2023-06-22T08:05:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.825+0000 7f6a249ff700 1 -- 172.21.15.23:0/3473269857 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 msgr2=0x7f6a200c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.825+0000 7f6a249ff700 1 --2- 172.21.15.23:0/3473269857 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 0x7f6a200c47c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6a18009dc0 tx=0x7f6a1800b330 comp rx=0 tx=0).stop 2023-06-22T08:05:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.825+0000 7f6a249ff700 1 -- 172.21.15.23:0/3473269857 shutdown_connections 2023-06-22T08:05:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.825+0000 7f6a249ff700 1 --2- 172.21.15.23:0/3473269857 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 0x7f6a200c47c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.825+0000 7f6a249ff700 1 -- 172.21.15.23:0/3473269857 >> 172.21.15.23:0/3473269857 conn(0x7f6a2006ae30 msgr2=0x7f6a2006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.825+0000 7f6a249ff700 1 -- 172.21.15.23:0/3473269857 shutdown_connections 2023-06-22T08:05:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.825+0000 7f6a249ff700 1 -- 172.21.15.23:0/3473269857 wait complete. 2023-06-22T08:05:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.827+0000 7f6a249ff700 1 Processor -- start 2023-06-22T08:05:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.827+0000 7f6a249ff700 1 -- start start 2023-06-22T08:05:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a249ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a200c9fd0 0x7f6a200cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a249ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 0x7f6a2015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a249ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20152ce0 0x7f6a201e1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a249ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201e42e0 con 0x7f6a20152ce0 2023-06-22T08:05:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a249ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a201eefb0 con 0x7f6a200c9fd0 2023-06-22T08:05:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a249ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a200ceaa0 con 0x7f6a200cf1b0 2023-06-22T08:05:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a1e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a200c9fd0 0x7f6a200cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a1e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a200c9fd0 0x7f6a200cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:55926/0 (socket says 172.21.15.23:55926) 2023-06-22T08:05:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a1e59c700 1 -- 172.21.15.23:0/1697172386 learned_addr learned my addr 172.21.15.23:0/1697172386 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a1ed9d700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20152ce0 0x7f6a201e1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.828+0000 7f6a1dd9b700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 0x7f6a2015ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.829+0000 7f6a1ed9d700 1 -- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 msgr2=0x7f6a2015ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.829+0000 7f6a1ed9d700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 0x7f6a2015ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.829+0000 7f6a1ed9d700 1 -- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a200c9fd0 msgr2=0x7f6a200cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.829+0000 7f6a1ed9d700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a200c9fd0 0x7f6a200cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.829+0000 7f6a1ed9d700 1 -- 172.21.15.23:0/1697172386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a18009a70 con 0x7f6a20152ce0 2023-06-22T08:05:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.829+0000 7f6a1e59c700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a200c9fd0 0x7f6a200cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:05:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.829+0000 7f6a1ed9d700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20152ce0 0x7f6a201e1480 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6a200cc380 tx=0x7f6a18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.829+0000 7f6a0effd700 1 -- 172.21.15.23:0/1697172386 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1801c810 con 0x7f6a20152ce0 2023-06-22T08:05:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.830+0000 7f6a0effd700 1 -- 172.21.15.23:0/1697172386 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a18020070 con 0x7f6a20152ce0 2023-06-22T08:05:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.830+0000 7f6a0effd700 1 -- 172.21.15.23:0/1697172386 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1801cbb0 con 0x7f6a20152ce0 2023-06-22T08:05:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.830+0000 7f6a249ff700 1 -- 172.21.15.23:0/1697172386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a200ced20 con 0x7f6a20152ce0 2023-06-22T08:05:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.830+0000 7f6a249ff700 1 -- 172.21.15.23:0/1697172386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a200ce110 con 0x7f6a20152ce0 2023-06-22T08:05:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.831+0000 7f6a249ff700 1 -- 172.21.15.23:0/1697172386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a201ef160 con 0x7f6a20152ce0 2023-06-22T08:05:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.832+0000 7f6a0effd700 1 -- 172.21.15.23:0/1697172386 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6a1801cd10 con 0x7f6a20152ce0 2023-06-22T08:05:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.833+0000 7f6a0effd700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a0006ace0 0x7f6a0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.833+0000 7f6a0effd700 1 -- 172.21.15.23:0/1697172386 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f6a18022070 con 0x7f6a20152ce0 2023-06-22T08:05:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.833+0000 7f6a1e59c700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a0006ace0 0x7f6a0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.834+0000 7f6a1e59c700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a0006ace0 0x7f6a0006d190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6a10001220 tx=0x7f6a10009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:42.838+0000 7f6a0effd700 1 -- 172.21.15.23:0/1697172386 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6a201ef160 con 0x7f6a20152ce0 2023-06-22T08:05:43.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.104+0000 7f6a249ff700 1 -- 172.21.15.23:0/1697172386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a201ef160 con 0x7f6a20152ce0 2023-06-22T08:05:43.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.104+0000 7f6a0effd700 1 -- 172.21.15.23:0/1697172386 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6257 (secure 0 0 0) 0x7f6a201ef160 con 0x7f6a20152ce0 2023-06-22T08:05:43.108 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:05:43.108 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5060,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:05:41.716490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:05:43.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 -- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a0006ace0 msgr2=0x7f6a0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:43.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a0006ace0 0x7f6a0006d190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6a10001220 tx=0x7f6a10009040 comp rx=0 tx=0).stop 2023-06-22T08:05:43.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 -- 172.21.15.23:0/1697172386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20152ce0 msgr2=0x7f6a201e1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:43.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20152ce0 0x7f6a201e1480 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6a200cc380 tx=0x7f6a18004ad0 comp rx=0 tx=0).stop 2023-06-22T08:05:43.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 -- 172.21.15.23:0/1697172386 shutdown_connections 2023-06-22T08:05:43.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20152ce0 0x7f6a201e1480 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:43.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6a0006ace0 0x7f6a0006d190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:43.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a200c9fd0 0x7f6a200cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:43.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 --2- 172.21.15.23:0/1697172386 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6a200cf1b0 0x7f6a2015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:43.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 -- 172.21.15.23:0/1697172386 >> 172.21.15.23:0/1697172386 conn(0x7f6a2006ae30 msgr2=0x7f6a2014c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:43.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 -- 172.21.15.23:0/1697172386 shutdown_connections 2023-06-22T08:05:43.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:43.107+0000 7f6a249ff700 1 -- 172.21.15.23:0/1697172386 wait complete. 2023-06-22T08:05:43.113 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 43 2023-06-22T08:05:43.129 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:05:43.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2023-06-22T08:05:45.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:05:45.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.338+0000 7ff87ebff700 1 Processor -- start 2023-06-22T08:05:45.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.338+0000 7ff87ebff700 1 -- start start 2023-06-22T08:05:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.339+0000 7ff87ebff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff878152ce0 0x7ff8780cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.339+0000 7ff87ebff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8780c9fd0 0x7ff87815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.339+0000 7ff87ebff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87815c370 con 0x7ff878158880 2023-06-22T08:05:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.339+0000 7ff87ebff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8781530b0 con 0x7ff878152ce0 2023-06-22T08:05:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.339+0000 7ff87ebff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8780cf6f0 con 0x7ff8780c9fd0 2023-06-22T08:05:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.339+0000 7ff877fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8780c9fd0 0x7ff87815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:45.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.339+0000 7ff87c99b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff878152ce0 0x7ff8780cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:45.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.340+0000 7ff87c99b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff878152ce0 0x7ff8780cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39236/0 (socket says 172.21.15.23:39236) 2023-06-22T08:05:45.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.340+0000 7ff877fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8780c9fd0 0x7ff87815c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:55940/0 (socket says 172.21.15.23:55940) 2023-06-22T08:05:45.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.340+0000 7ff87c99b700 1 -- 172.21.15.23:0/3561916040 learned_addr learned my addr 172.21.15.23:0/3561916040 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:45.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.340+0000 7ff877fff700 1 -- 172.21.15.23:0/3561916040 >> v2:172.21.15.23:3300/0 conn(0x7ff878152ce0 msgr2=0x7ff8780cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:05:45.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.340+0000 7ff877fff700 1 --2- 172.21.15.23:0/3561916040 >> v2:172.21.15.23:3300/0 conn(0x7ff878152ce0 0x7ff8780cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:45.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.340+0000 7ff877fff700 1 -- 172.21.15.23:0/3561916040 >> v1:172.21.15.23:6789/0 conn(0x7ff878158880 legacy=0x7ff878158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:45.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.340+0000 7ff877fff700 1 -- 172.21.15.23:0/3561916040 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87815bda0 con 0x7ff8780c9fd0 2023-06-22T08:05:45.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.340+0000 7ff87c99b700 1 --2- 172.21.15.23:0/3561916040 >> v2:172.21.15.23:3300/0 conn(0x7ff878152ce0 0x7ff8780cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:05:45.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.340+0000 7ff877fff700 1 --2- 172.21.15.23:0/3561916040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780c9fd0 0x7ff87815c800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff8780cc380 tx=0x7ff864009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93b697ee228917db server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:45.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.341+0000 7ff8777fe700 1 -- 172.21.15.23:0/3561916040 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff864011830 con 0x7ff8780c9fd0 2023-06-22T08:05:45.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.341+0000 7ff8777fe700 1 -- 172.21.15.23:0/3561916040 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff864011990 con 0x7ff8780c9fd0 2023-06-22T08:05:45.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.341+0000 7ff8777fe700 1 -- 172.21.15.23:0/3561916040 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff864011d20 con 0x7ff8780c9fd0 2023-06-22T08:05:45.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.342+0000 7ff87ebff700 1 -- 172.21.15.23:0/3561916040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780c9fd0 msgr2=0x7ff87815c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:45.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.342+0000 7ff87ebff700 1 --2- 172.21.15.23:0/3561916040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780c9fd0 0x7ff87815c800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff8780cc380 tx=0x7ff864009b20 comp rx=0 tx=0).stop 2023-06-22T08:05:45.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.342+0000 7ff87ebff700 1 -- 172.21.15.23:0/3561916040 shutdown_connections 2023-06-22T08:05:45.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.342+0000 7ff87ebff700 1 --2- 172.21.15.23:0/3561916040 >> v2:172.21.15.23:3300/0 conn(0x7ff878152ce0 0x7ff8780cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:45.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.342+0000 7ff87ebff700 1 --2- 172.21.15.23:0/3561916040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780c9fd0 0x7ff87815c800 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:45.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.342+0000 7ff87ebff700 1 -- 172.21.15.23:0/3561916040 >> 172.21.15.23:0/3561916040 conn(0x7ff87806ae30 msgr2=0x7ff87806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:45.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.342+0000 7ff87ebff700 1 -- 172.21.15.23:0/3561916040 shutdown_connections 2023-06-22T08:05:45.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.342+0000 7ff87ebff700 1 -- 172.21.15.23:0/3561916040 wait complete. 2023-06-22T08:05:45.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.344+0000 7ff87ebff700 1 Processor -- start 2023-06-22T08:05:45.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.344+0000 7ff87ebff700 1 -- start start 2023-06-22T08:05:45.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff87ebff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780c9fd0 0x7ff8781564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:45.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff87ebff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff878152ce0 0x7ff87815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:45.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff87ebff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff878158880 0x7ff878142290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:45.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff87ebff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87815b890 con 0x7ff878158880 2023-06-22T08:05:45.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff87ebff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff878147c90 con 0x7ff8780c9fd0 2023-06-22T08:05:45.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff87ebff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8780cd140 con 0x7ff878152ce0 2023-06-22T08:05:45.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff877fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff878152ce0 0x7ff87815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:45.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff87c99b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780c9fd0 0x7ff8781564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:45.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff877fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff878152ce0 0x7ff87815aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:59542/0 (socket says 172.21.15.23:59542) 2023-06-22T08:05:45.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff877fff700 1 -- 172.21.15.23:0/28852671 learned_addr learned my addr 172.21.15.23:0/28852671 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:45.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.345+0000 7ff87d19c700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff878158880 0x7ff878142290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:45.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff877fff700 1 -- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780c9fd0 msgr2=0x7ff8781564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:45.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff877fff700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780c9fd0 0x7ff8781564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:45.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff877fff700 1 -- 172.21.15.23:0/28852671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff878158880 msgr2=0x7ff878142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:45.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff877fff700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff878158880 0x7ff878142290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:45.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff877fff700 1 -- 172.21.15.23:0/28852671 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c009550 con 0x7ff878152ce0 2023-06-22T08:05:45.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff87c99b700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780c9fd0 0x7ff8781564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:05:45.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff877fff700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff878152ce0 0x7ff87815aeb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff864000c00 tx=0x7ff864004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:45.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff8757fa700 1 -- 172.21.15.23:0/28852671 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8640036a0 con 0x7ff878152ce0 2023-06-22T08:05:45.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff8757fa700 1 -- 172.21.15.23:0/28852671 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86401f070 con 0x7ff878152ce0 2023-06-22T08:05:45.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff87ebff700 1 -- 172.21.15.23:0/28852671 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8640094a0 con 0x7ff878152ce0 2023-06-22T08:05:45.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.346+0000 7ff8757fa700 1 -- 172.21.15.23:0/28852671 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff864003a40 con 0x7ff878152ce0 2023-06-22T08:05:45.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.347+0000 7ff87ebff700 1 -- 172.21.15.23:0/28852671 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87815bb00 con 0x7ff878152ce0 2023-06-22T08:05:45.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.348+0000 7ff87ebff700 1 -- 172.21.15.23:0/28852671 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87804ea20 con 0x7ff878152ce0 2023-06-22T08:05:45.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.349+0000 7ff8757fa700 1 -- 172.21.15.23:0/28852671 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff864003ba0 con 0x7ff878152ce0 2023-06-22T08:05:45.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.350+0000 7ff8757fa700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff85c06ace0 0x7ff85c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:45.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.350+0000 7ff8757fa700 1 -- 172.21.15.23:0/28852671 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7041+0+0 (secure 0 0 0) 0x7ff864021070 con 0x7ff878152ce0 2023-06-22T08:05:45.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.350+0000 7ff87c99b700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff85c06ace0 0x7ff85c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:45.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.350+0000 7ff87c99b700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff85c06ace0 0x7ff85c06d190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff86c006d30 tx=0x7ff86c0092c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:45.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.354+0000 7ff8757fa700 1 -- 172.21.15.23:0/28852671 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff864055d50 con 0x7ff878152ce0 2023-06-22T08:05:45.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.631+0000 7ff87ebff700 1 -- 172.21.15.23:0/28852671 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8780c9db0 con 0x7ff878152ce0 2023-06-22T08:05:45.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.632+0000 7ff8757fa700 1 -- 172.21.15.23:0/28852671 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6257 (secure 0 0 0) 0x7ff864059370 con 0x7ff878152ce0 2023-06-22T08:05:45.636 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:05:45.637 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5060,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:05:41.716490+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:05:45.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 -- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff85c06ace0 msgr2=0x7ff85c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:45.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff85c06ace0 0x7ff85c06d190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff86c006d30 tx=0x7ff86c0092c0 comp rx=0 tx=0).stop 2023-06-22T08:05:45.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 -- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff878152ce0 msgr2=0x7ff87815aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:45.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff878152ce0 0x7ff87815aeb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff864000c00 tx=0x7ff864004750 comp rx=0 tx=0).stop 2023-06-22T08:05:45.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 -- 172.21.15.23:0/28852671 shutdown_connections 2023-06-22T08:05:45.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff878158880 0x7ff878142290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:45.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff85c06ace0 0x7ff85c06d190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:45.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8780c9fd0 0x7ff8781564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:45.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 --2- 172.21.15.23:0/28852671 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff878152ce0 0x7ff87815aeb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:45.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 -- 172.21.15.23:0/28852671 >> 172.21.15.23:0/28852671 conn(0x7ff87806ae30 msgr2=0x7ff8780b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:45.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 -- 172.21.15.23:0/28852671 shutdown_connections 2023-06-22T08:05:45.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:45.635+0000 7ff87ebff700 1 -- 172.21.15.23:0/28852671 wait complete. 2023-06-22T08:05:45.642 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 43 2023-06-22T08:05:45.659 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:05:47.660 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:05:47.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.863+0000 7f66f11ff700 1 Processor -- start 2023-06-22T08:05:47.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.863+0000 7f66f11ff700 1 -- start start 2023-06-22T08:05:47.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.864+0000 7f66f11ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec071850 0x7f66ec0c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:47.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.864+0000 7f66f11ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec15bf50 con 0x7f66ec0c4a30 2023-06-22T08:05:47.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.864+0000 7f66f11ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec0c9550 con 0x7f66ec071c20 2023-06-22T08:05:47.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.864+0000 7f66f11ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec0c4eb0 con 0x7f66ec071850 2023-06-22T08:05:47.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66ea59c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f66ec071c20 0x7f66ec152750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:34358/0 (socket says 172.21.15.23:34358) 2023-06-22T08:05:47.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66ea59c700 1 -- 172.21.15.23:0/3791440334 learned_addr learned my addr 172.21.15.23:0/3791440334 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:05:47.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66ead9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec071850 0x7f66ec0c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:47.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66e9d9b700 1 -- 172.21.15.23:0/3791440334 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751537406 0 0) 0x7f66ec0c9550 con 0x7f66ec071c20 2023-06-22T08:05:47.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66e9d9b700 1 -- 172.21.15.23:0/3791440334 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d40035b0 con 0x7f66ec071c20 2023-06-22T08:05:47.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66e9d9b700 1 -- 172.21.15.23:0/3791440334 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090313972 0 0) 0x7f66ec15bf50 con 0x7f66ec0c4a30 2023-06-22T08:05:47.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66e9d9b700 1 -- 172.21.15.23:0/3791440334 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66ec0c9550 con 0x7f66ec0c4a30 2023-06-22T08:05:47.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66ead9d700 1 -- 172.21.15.23:0/3791440334 >> v1:172.21.15.67:6789/0 conn(0x7f66ec071c20 legacy=0x7f66ec152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:47.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66ead9d700 1 -- 172.21.15.23:0/3791440334 >> v1:172.21.15.23:6789/0 conn(0x7f66ec0c4a30 legacy=0x7f66ec15c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:47.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66ead9d700 1 -- 172.21.15.23:0/3791440334 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec158c00 con 0x7f66ec071850 2023-06-22T08:05:47.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.865+0000 7f66ead9d700 1 --2- 172.21.15.23:0/3791440334 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec071850 0x7f66ec0c9880 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f66dc009870 tx=0x7f66dc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c5e5b44dbfc05e1 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:47.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.866+0000 7f66e9d9b700 1 -- 172.21.15.23:0/3791440334 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc011830 con 0x7f66ec071850 2023-06-22T08:05:47.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.866+0000 7f66e9d9b700 1 -- 172.21.15.23:0/3791440334 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66dc011990 con 0x7f66ec071850 2023-06-22T08:05:47.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.866+0000 7f66e9d9b700 1 -- 172.21.15.23:0/3791440334 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc011d20 con 0x7f66ec071850 2023-06-22T08:05:47.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.866+0000 7f66f11ff700 1 -- 172.21.15.23:0/3791440334 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec071850 msgr2=0x7f66ec0c9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:47.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.866+0000 7f66f11ff700 1 --2- 172.21.15.23:0/3791440334 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec071850 0x7f66ec0c9880 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f66dc009870 tx=0x7f66dc009b80 comp rx=0 tx=0).stop 2023-06-22T08:05:47.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.866+0000 7f66f11ff700 1 -- 172.21.15.23:0/3791440334 shutdown_connections 2023-06-22T08:05:47.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.866+0000 7f66f11ff700 1 --2- 172.21.15.23:0/3791440334 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec071850 0x7f66ec0c9880 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:47.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.866+0000 7f66f11ff700 1 -- 172.21.15.23:0/3791440334 >> 172.21.15.23:0/3791440334 conn(0x7f66ec06b8b0 msgr2=0x7f66ec06bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:47.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.866+0000 7f66f11ff700 1 -- 172.21.15.23:0/3791440334 shutdown_connections 2023-06-22T08:05:47.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.866+0000 7f66f11ff700 1 -- 172.21.15.23:0/3791440334 wait complete. 2023-06-22T08:05:47.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.868+0000 7f66f11ff700 1 Processor -- start 2023-06-22T08:05:47.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.868+0000 7f66f11ff700 1 -- start start 2023-06-22T08:05:47.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66f11ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ec071850 0x7f66ec1ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:47.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66f11ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66ec071c20 0x7f66ec157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:47.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66f11ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec0c4a30 0x7f66ec14feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66f11ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec158920 con 0x7f66ec071c20 2023-06-22T08:05:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66f11ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec0c5cf0 con 0x7f66ec071850 2023-06-22T08:05:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66f11ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec1ef6f0 con 0x7f66ec0c4a30 2023-06-22T08:05:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66ead9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ec071850 0x7f66ec1ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66ead9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ec071850 0x7f66ec1ef1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:54120/0 (socket says 172.21.15.23:54120) 2023-06-22T08:05:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66ead9d700 1 -- 172.21.15.23:0/1908035841 learned_addr learned my addr 172.21.15.23:0/1908035841 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66ea59c700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66ec071c20 0x7f66ec157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.869+0000 7f66eb59e700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec0c4a30 0x7f66ec14feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.870+0000 7f66ead9d700 1 -- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec0c4a30 msgr2=0x7f66ec14feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.870+0000 7f66ead9d700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec0c4a30 0x7f66ec14feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.870+0000 7f66ead9d700 1 -- 172.21.15.23:0/1908035841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66ec071c20 msgr2=0x7f66ec157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.870+0000 7f66ead9d700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66ec071c20 0x7f66ec157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.870+0000 7f66ead9d700 1 -- 172.21.15.23:0/1908035841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc0094a0 con 0x7f66ec071850 2023-06-22T08:05:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.870+0000 7f66ead9d700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ec071850 0x7f66ec1ef1b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0090b0 tx=0x7f66dc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.870+0000 7f66d37fe700 1 -- 172.21.15.23:0/1908035841 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc01a6e0 con 0x7f66ec071850 2023-06-22T08:05:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.870+0000 7f66d37fe700 1 -- 172.21.15.23:0/1908035841 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66dc01a840 con 0x7f66ec071850 2023-06-22T08:05:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.870+0000 7f66d37fe700 1 -- 172.21.15.23:0/1908035841 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc01abb0 con 0x7f66ec071850 2023-06-22T08:05:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.870+0000 7f66f11ff700 1 -- 172.21.15.23:0/1908035841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec156ba0 con 0x7f66ec071850 2023-06-22T08:05:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.871+0000 7f66f11ff700 1 -- 172.21.15.23:0/1908035841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ec158100 con 0x7f66ec071850 2023-06-22T08:05:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.872+0000 7f66f11ff700 1 -- 172.21.15.23:0/1908035841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ec156e80 con 0x7f66ec071850 2023-06-22T08:05:47.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.873+0000 7f66d37fe700 1 -- 172.21.15.23:0/1908035841 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f66dc022070 con 0x7f66ec071850 2023-06-22T08:05:47.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.873+0000 7f66d37fe700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66d806ace0 0x7f66d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:47.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.874+0000 7f66d37fe700 1 -- 172.21.15.23:0/1908035841 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f66dc08a4d0 con 0x7f66ec071850 2023-06-22T08:05:47.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.874+0000 7f66ea59c700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66d806ace0 0x7f66d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:47.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.874+0000 7f66ea59c700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66d806ace0 0x7f66d806d190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f66ec1507e0 tx=0x7f66e0007e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:47.882+0000 7f66d37fe700 1 -- 172.21.15.23:0/1908035841 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f66dc0551c0 con 0x7f66ec071850 2023-06-22T08:05:48.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.139+0000 7f66f11ff700 1 -- 172.21.15.23:0/1908035841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66ec04ea20 con 0x7f66ec071850 2023-06-22T08:05:48.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.140+0000 7f66d37fe700 1 -- 172.21.15.23:0/1908035841 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f66dc0587e0 con 0x7f66ec071850 2023-06-22T08:05:48.144 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:05:48.144 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5060,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:05:47.344924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:05:48.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 -- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66d806ace0 msgr2=0x7f66d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:48.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66d806ace0 0x7f66d806d190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f66ec1507e0 tx=0x7f66e0007e00 comp rx=0 tx=0).stop 2023-06-22T08:05:48.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 -- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ec071850 msgr2=0x7f66ec1ef1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:48.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ec071850 0x7f66ec1ef1b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0090b0 tx=0x7f66dc004750 comp rx=0 tx=0).stop 2023-06-22T08:05:48.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 -- 172.21.15.23:0/1908035841 shutdown_connections 2023-06-22T08:05:48.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66ec0c4a30 0x7f66ec14feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:48.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f66d806ace0 0x7f66d806d190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:48.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66ec071850 0x7f66ec1ef1b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:48.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 --2- 172.21.15.23:0/1908035841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66ec071c20 0x7f66ec157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:48.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 -- 172.21.15.23:0/1908035841 >> 172.21.15.23:0/1908035841 conn(0x7f66ec06b8b0 msgr2=0x7f66ec14edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:48.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 -- 172.21.15.23:0/1908035841 shutdown_connections 2023-06-22T08:05:48.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:48.142+0000 7f66f11ff700 1 -- 172.21.15.23:0/1908035841 wait complete. 2023-06-22T08:05:48.150 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 44 2023-06-22T08:05:48.165 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:05:50.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2023-06-22T08:05:50.167 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:05:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.374+0000 7f7788fff700 1 Processor -- start 2023-06-22T08:05:50.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.374+0000 7f7788fff700 1 -- start start 2023-06-22T08:05:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.376+0000 7f7788fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f77840c97d0 0x7f7784159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.376+0000 7f7788fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c45f0 0x7f77840c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.376+0000 7f7788fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77840c9c30 con 0x7f7784152d70 2023-06-22T08:05:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.376+0000 7f7788fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77840c95e0 con 0x7f77840c97d0 2023-06-22T08:05:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.376+0000 7f7788fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77840c4f90 con 0x7f77840c45f0 2023-06-22T08:05:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.376+0000 7f778259c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7784152d70 0x7f77840cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34362/0 (socket says 172.21.15.23:34362) 2023-06-22T08:05:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.376+0000 7f778259c700 1 -- 172.21.15.23:0/829411183 learned_addr learned my addr 172.21.15.23:0/829411183 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:05:50.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.376+0000 7f7782d9d700 1 --2- 172.21.15.23:0/829411183 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c45f0 0x7f77840c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:50.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.376+0000 7f777bfff700 1 --2- 172.21.15.23:0/829411183 >> v2:172.21.15.67:3300/0 conn(0x7f77840c97d0 0x7f7784159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:50.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.377+0000 7f7781d9b700 1 -- 172.21.15.23:0/829411183 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84240063 0 0) 0x7f77840c9c30 con 0x7f7784152d70 2023-06-22T08:05:50.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.377+0000 7f7781d9b700 1 -- 172.21.15.23:0/829411183 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77700035b0 con 0x7f7784152d70 2023-06-22T08:05:50.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.377+0000 7f7782d9d700 1 -- 172.21.15.23:0/829411183 >> v2:172.21.15.67:3300/0 conn(0x7f77840c97d0 msgr2=0x7f7784159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:05:50.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.377+0000 7f7782d9d700 1 --2- 172.21.15.23:0/829411183 >> v2:172.21.15.67:3300/0 conn(0x7f77840c97d0 0x7f7784159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.377+0000 7f7782d9d700 1 -- 172.21.15.23:0/829411183 >> v1:172.21.15.23:6789/0 conn(0x7f7784152d70 legacy=0x7f77840cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:50.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.377+0000 7f7782d9d700 1 -- 172.21.15.23:0/829411183 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778415bea0 con 0x7f77840c45f0 2023-06-22T08:05:50.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.377+0000 7f7782d9d700 1 --2- 172.21.15.23:0/829411183 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c45f0 0x7f77840c4a50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f777c009d90 tx=0x7f777c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73ee44f7702c5534 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:50.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.377+0000 7f7781d9b700 1 -- 172.21.15.23:0/829411183 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c00e070 con 0x7f77840c45f0 2023-06-22T08:05:50.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.377+0000 7f7781d9b700 1 -- 172.21.15.23:0/829411183 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c00bb20 con 0x7f77840c45f0 2023-06-22T08:05:50.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.377+0000 7f7781d9b700 1 -- 172.21.15.23:0/829411183 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c012430 con 0x7f77840c45f0 2023-06-22T08:05:50.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.378+0000 7f7788fff700 1 -- 172.21.15.23:0/829411183 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c45f0 msgr2=0x7f77840c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:50.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.378+0000 7f7788fff700 1 --2- 172.21.15.23:0/829411183 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c45f0 0x7f77840c4a50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f777c009d90 tx=0x7f777c00b330 comp rx=0 tx=0).stop 2023-06-22T08:05:50.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.378+0000 7f7788fff700 1 -- 172.21.15.23:0/829411183 shutdown_connections 2023-06-22T08:05:50.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.378+0000 7f7788fff700 1 --2- 172.21.15.23:0/829411183 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c45f0 0x7f77840c4a50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.378+0000 7f7788fff700 1 --2- 172.21.15.23:0/829411183 >> v2:172.21.15.67:3300/0 conn(0x7f77840c97d0 0x7f7784159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.378+0000 7f7788fff700 1 -- 172.21.15.23:0/829411183 >> 172.21.15.23:0/829411183 conn(0x7f778406ae30 msgr2=0x7f778406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:50.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.378+0000 7f7788fff700 1 -- 172.21.15.23:0/829411183 shutdown_connections 2023-06-22T08:05:50.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.378+0000 7f7788fff700 1 -- 172.21.15.23:0/829411183 wait complete. 2023-06-22T08:05:50.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.380+0000 7f7788fff700 1 Processor -- start 2023-06-22T08:05:50.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.380+0000 7f7788fff700 1 -- start start 2023-06-22T08:05:50.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7788fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77840c45f0 0x7f77840cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:50.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7788fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c97d0 0x7f778415af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:50.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7788fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7784152d70 0x7f77840c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:50.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7788fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77840cd560 con 0x7f7784152d70 2023-06-22T08:05:50.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7788fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77840cf550 con 0x7f77840c45f0 2023-06-22T08:05:50.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7788fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77841ee140 con 0x7f77840c97d0 2023-06-22T08:05:50.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7782d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7784152d70 0x7f77840c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:50.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7782d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7784152d70 0x7f77840c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57326/0 (socket says 172.21.15.23:57326) 2023-06-22T08:05:50.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7782d9d700 1 -- 172.21.15.23:0/1967735987 learned_addr learned my addr 172.21.15.23:0/1967735987 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:50.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f778259c700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77840c45f0 0x7f77840cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:50.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f777bfff700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c97d0 0x7f778415af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:50.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7782d9d700 1 -- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c97d0 msgr2=0x7f778415af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:50.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7782d9d700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c97d0 0x7f778415af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7782d9d700 1 -- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77840c45f0 msgr2=0x7f77840cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:50.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7782d9d700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77840c45f0 0x7f77840cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.381+0000 7f7782d9d700 1 -- 172.21.15.23:0/1967735987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c009a70 con 0x7f7784152d70 2023-06-22T08:05:50.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.382+0000 7f7782d9d700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7784152d70 0x7f77840c7eb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f77840c50d0 tx=0x7f777c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:50.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.382+0000 7f777affd700 1 -- 172.21.15.23:0/1967735987 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c0129f0 con 0x7f7784152d70 2023-06-22T08:05:50.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.382+0000 7f777affd700 1 -- 172.21.15.23:0/1967735987 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c020070 con 0x7f7784152d70 2023-06-22T08:05:50.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.382+0000 7f7788fff700 1 -- 172.21.15.23:0/1967735987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77841ee420 con 0x7f7784152d70 2023-06-22T08:05:50.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.382+0000 7f777affd700 1 -- 172.21.15.23:0/1967735987 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c012db0 con 0x7f7784152d70 2023-06-22T08:05:50.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.382+0000 7f7788fff700 1 -- 172.21.15.23:0/1967735987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77841ee9e0 con 0x7f7784152d70 2023-06-22T08:05:50.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.383+0000 7f7788fff700 1 -- 172.21.15.23:0/1967735987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77840cd6a0 con 0x7f7784152d70 2023-06-22T08:05:50.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.384+0000 7f777affd700 1 -- 172.21.15.23:0/1967735987 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f777c004040 con 0x7f7784152d70 2023-06-22T08:05:50.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.385+0000 7f777affd700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f776406ace0 0x7f776406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:50.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.385+0000 7f777affd700 1 -- 172.21.15.23:0/1967735987 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f777c022070 con 0x7f7784152d70 2023-06-22T08:05:50.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.385+0000 7f778259c700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f776406ace0 0x7f776406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:50.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.386+0000 7f778259c700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f776406ace0 0x7f776406d190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7774007fd0 tx=0x7f7774007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:50.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.390+0000 7f777affd700 1 -- 172.21.15.23:0/1967735987 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f77840cd6a0 con 0x7f7784152d70 2023-06-22T08:05:50.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.642+0000 7f7788fff700 1 -- 172.21.15.23:0/1967735987 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77840cd6a0 con 0x7f7784152d70 2023-06-22T08:05:50.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.643+0000 7f777affd700 1 -- 172.21.15.23:0/1967735987 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f777c05d880 con 0x7f7784152d70 2023-06-22T08:05:50.646 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:05:50.647 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5060,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:05:47.344924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:05:50.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 -- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f776406ace0 msgr2=0x7f776406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:50.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f776406ace0 0x7f776406d190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7774007fd0 tx=0x7f7774007dc0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 -- 172.21.15.23:0/1967735987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7784152d70 msgr2=0x7f77840c7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:50.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7784152d70 0x7f77840c7eb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f77840c50d0 tx=0x7f777c004ad0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 -- 172.21.15.23:0/1967735987 shutdown_connections 2023-06-22T08:05:50.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7784152d70 0x7f77840c7eb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f776406ace0 0x7f776406d190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77840c45f0 0x7f77840cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 --2- 172.21.15.23:0/1967735987 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77840c97d0 0x7f778415af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:50.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 -- 172.21.15.23:0/1967735987 >> 172.21.15.23:0/1967735987 conn(0x7f778406ae30 msgr2=0x7f778414f660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:50.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 -- 172.21.15.23:0/1967735987 shutdown_connections 2023-06-22T08:05:50.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:50.645+0000 7f7788fff700 1 -- 172.21.15.23:0/1967735987 wait complete. 2023-06-22T08:05:50.651 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 44 2023-06-22T08:05:50.667 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:05:52.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:05:52.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.868+0000 7fa3d4fff700 1 Processor -- start 2023-06-22T08:05:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.868+0000 7fa3d4fff700 1 -- start start 2023-06-22T08:05:52.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.869+0000 7fa3d4fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3d0071c20 0x7fa3d00c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:52.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.869+0000 7fa3d4fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d015bf50 con 0x7fa3d00c4a30 2023-06-22T08:05:52.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.869+0000 7fa3d4fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d00c9550 con 0x7fa3d0071850 2023-06-22T08:05:52.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.869+0000 7fa3d4fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d00c4eb0 con 0x7fa3d0071c20 2023-06-22T08:05:52.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3ce59c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa3d0071850 0x7fa3d0152750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:34368/0 (socket says 172.21.15.23:34368) 2023-06-22T08:05:52.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3ce59c700 1 -- 172.21.15.23:0/2459869933 learned_addr learned my addr 172.21.15.23:0/2459869933 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:05:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3cdd9b700 1 --2- 172.21.15.23:0/2459869933 >> v2:172.21.15.23:3300/0 conn(0x7fa3d0071c20 0x7fa3d00c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3cd59a700 1 -- 172.21.15.23:0/2459869933 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454107261 0 0) 0x7fa3d00c9550 con 0x7fa3d0071850 2023-06-22T08:05:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3cd59a700 1 -- 172.21.15.23:0/2459869933 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b80035b0 con 0x7fa3d0071850 2023-06-22T08:05:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3cd59a700 1 -- 172.21.15.23:0/2459869933 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093115553 0 0) 0x7fa3d015bf50 con 0x7fa3d00c4a30 2023-06-22T08:05:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3cd59a700 1 -- 172.21.15.23:0/2459869933 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d00c9550 con 0x7fa3d00c4a30 2023-06-22T08:05:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3cdd9b700 1 -- 172.21.15.23:0/2459869933 >> v1:172.21.15.67:6789/0 conn(0x7fa3d0071850 legacy=0x7fa3d0152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:52.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3cd59a700 1 -- 172.21.15.23:0/2459869933 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3777592786 0 0) 0x7fa3b80035b0 con 0x7fa3d0071850 2023-06-22T08:05:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3cdd9b700 1 -- 172.21.15.23:0/2459869933 >> v1:172.21.15.23:6789/0 conn(0x7fa3d00c4a30 legacy=0x7fa3d015c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.870+0000 7fa3cdd9b700 1 -- 172.21.15.23:0/2459869933 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d0158c00 con 0x7fa3d0071c20 2023-06-22T08:05:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.871+0000 7fa3cdd9b700 1 --2- 172.21.15.23:0/2459869933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d0071c20 0x7fa3d00c9880 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa3c40097c0 tx=0x7fa3c4009ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595ad578cf80023b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.871+0000 7fa3cd59a700 1 -- 172.21.15.23:0/2459869933 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0001670 con 0x7fa3d0071c20 2023-06-22T08:05:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.871+0000 7fa3cd59a700 1 -- 172.21.15.23:0/2459869933 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c4004030 con 0x7fa3d0071c20 2023-06-22T08:05:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.871+0000 7fa3cd59a700 1 -- 172.21.15.23:0/2459869933 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40043c0 con 0x7fa3d0071c20 2023-06-22T08:05:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.871+0000 7fa3d4fff700 1 -- 172.21.15.23:0/2459869933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d0071c20 msgr2=0x7fa3d00c9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:52.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.871+0000 7fa3d4fff700 1 --2- 172.21.15.23:0/2459869933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d0071c20 0x7fa3d00c9880 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa3c40097c0 tx=0x7fa3c4009ad0 comp rx=0 tx=0).stop 2023-06-22T08:05:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.872+0000 7fa3d4fff700 1 -- 172.21.15.23:0/2459869933 shutdown_connections 2023-06-22T08:05:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.872+0000 7fa3d4fff700 1 --2- 172.21.15.23:0/2459869933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d0071c20 0x7fa3d00c9880 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.872+0000 7fa3d4fff700 1 -- 172.21.15.23:0/2459869933 >> 172.21.15.23:0/2459869933 conn(0x7fa3d006b8b0 msgr2=0x7fa3d006bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.872+0000 7fa3d4fff700 1 -- 172.21.15.23:0/2459869933 shutdown_connections 2023-06-22T08:05:52.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.872+0000 7fa3d4fff700 1 -- 172.21.15.23:0/2459869933 wait complete. 2023-06-22T08:05:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.873+0000 7fa3d4fff700 1 Processor -- start 2023-06-22T08:05:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.874+0000 7fa3d4fff700 1 -- start start 2023-06-22T08:05:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.874+0000 7fa3d4fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d0071850 0x7fa3d0157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:52.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.874+0000 7fa3d4fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d0071c20 0x7fa3d00cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3d4fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3d00c4a30 0x7fa3d00b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3d4fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d01f2370 con 0x7fa3d0071c20 2023-06-22T08:05:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3d4fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d0150460 con 0x7fa3d00c4a30 2023-06-22T08:05:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3d4fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d0157bd0 con 0x7fa3d0071850 2023-06-22T08:05:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3ce59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d0071850 0x7fa3d0157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3cdd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d0071c20 0x7fa3d00cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3ce59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d0071850 0x7fa3d0157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35542/0 (socket says 172.21.15.23:35542) 2023-06-22T08:05:52.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3ce59c700 1 -- 172.21.15.23:0/3659022846 learned_addr learned my addr 172.21.15.23:0/3659022846 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3ced9d700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3d00c4a30 0x7fa3d00b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3cdd9b700 1 -- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d0071850 msgr2=0x7fa3d0157690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3cdd9b700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d0071850 0x7fa3d0157690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3cdd9b700 1 -- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3d00c4a30 msgr2=0x7fa3d00b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3cdd9b700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3d00c4a30 0x7fa3d00b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.875+0000 7fa3cdd9b700 1 -- 172.21.15.23:0/3659022846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c40094a0 con 0x7fa3d0071c20 2023-06-22T08:05:52.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.876+0000 7fa3cdd9b700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d0071c20 0x7fa3d00cde50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00cb1b0 tx=0x7fa3c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:52.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.876+0000 7fa3beffd700 1 -- 172.21.15.23:0/3659022846 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4003ea0 con 0x7fa3d0071c20 2023-06-22T08:05:52.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.876+0000 7fa3beffd700 1 -- 172.21.15.23:0/3659022846 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c401f070 con 0x7fa3d0071c20 2023-06-22T08:05:52.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.876+0000 7fa3d4fff700 1 -- 172.21.15.23:0/3659022846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d01f1cf0 con 0x7fa3d0071c20 2023-06-22T08:05:52.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.876+0000 7fa3beffd700 1 -- 172.21.15.23:0/3659022846 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c4011850 con 0x7fa3d0071c20 2023-06-22T08:05:52.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.876+0000 7fa3d4fff700 1 -- 172.21.15.23:0/3659022846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d0150180 con 0x7fa3d0071c20 2023-06-22T08:05:52.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.877+0000 7fa3d4fff700 1 -- 172.21.15.23:0/3659022846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d01f1b50 con 0x7fa3d0071c20 2023-06-22T08:05:52.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.878+0000 7fa3beffd700 1 -- 172.21.15.23:0/3659022846 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa3c40119b0 con 0x7fa3d0071c20 2023-06-22T08:05:52.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.880+0000 7fa3beffd700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3b006ace0 0x7fa3b006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:52.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.880+0000 7fa3beffd700 1 -- 172.21.15.23:0/3659022846 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7041+0+0 (secure 0 0 0) 0x7fa3c4021070 con 0x7fa3d0071c20 2023-06-22T08:05:52.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.880+0000 7fa3ce59c700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3b006ace0 0x7fa3b006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:52.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.880+0000 7fa3ce59c700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3b006ace0 0x7fa3b006d190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0001380 tx=0x7fa3c0001d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:52.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:52.884+0000 7fa3beffd700 1 -- 172.21.15.23:0/3659022846 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa3d01f1b50 con 0x7fa3d0071c20 2023-06-22T08:05:53.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.141+0000 7fa3d4fff700 1 -- 172.21.15.23:0/3659022846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3d01f24b0 con 0x7fa3d0071c20 2023-06-22T08:05:53.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.142+0000 7fa3beffd700 1 -- 172.21.15.23:0/3659022846 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6255 (secure 0 0 0) 0x7fa3d01f24b0 con 0x7fa3d0071c20 2023-06-22T08:05:53.145 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:05:53.146 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5060,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:05:47.344924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:05:53.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 -- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3b006ace0 msgr2=0x7fa3b006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:53.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3b006ace0 0x7fa3b006d190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0001380 tx=0x7fa3c0001d40 comp rx=0 tx=0).stop 2023-06-22T08:05:53.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 -- 172.21.15.23:0/3659022846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d0071c20 msgr2=0x7fa3d00cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:53.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d0071c20 0x7fa3d00cde50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00cb1b0 tx=0x7fa3c4004750 comp rx=0 tx=0).stop 2023-06-22T08:05:53.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 -- 172.21.15.23:0/3659022846 shutdown_connections 2023-06-22T08:05:53.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3b006ace0 0x7fa3b006d190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:53.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3d00c4a30 0x7fa3d00b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:53.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d0071850 0x7fa3d0157690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:53.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 --2- 172.21.15.23:0/3659022846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d0071c20 0x7fa3d00cde50 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:53.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 -- 172.21.15.23:0/3659022846 >> 172.21.15.23:0/3659022846 conn(0x7fa3d006b8b0 msgr2=0x7fa3d0152520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:53.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 -- 172.21.15.23:0/3659022846 shutdown_connections 2023-06-22T08:05:53.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:53.144+0000 7fa3d4fff700 1 -- 172.21.15.23:0/3659022846 wait complete. 2023-06-22T08:05:53.150 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 44 2023-06-22T08:05:53.167 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:05:55.171 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:05:55.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.378+0000 7f9688fff700 1 Processor -- start 2023-06-22T08:05:55.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.379+0000 7f9688fff700 1 -- start start 2023-06-22T08:05:55.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f9688fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9684152ce0 0x7f96840cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:55.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f9688fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96840c9fd0 0x7f968415c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:55.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f9688fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968415c370 con 0x7f9684158880 2023-06-22T08:05:55.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f9688fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96841530b0 con 0x7f96840c9fd0 2023-06-22T08:05:55.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f9688fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96840cf6f0 con 0x7f9684152ce0 2023-06-22T08:05:55.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f968259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9684152ce0 0x7f96840cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:55.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f9681d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96840c9fd0 0x7f968415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:55.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f9681d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96840c9fd0 0x7f968415c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:54158/0 (socket says 172.21.15.23:54158) 2023-06-22T08:05:55.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f968259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9684152ce0 0x7f96840cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35558/0 (socket says 172.21.15.23:35558) 2023-06-22T08:05:55.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f9681d9b700 1 -- 172.21.15.23:0/3382018022 learned_addr learned my addr 172.21.15.23:0/3382018022 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:55.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f968259c700 1 -- 172.21.15.23:0/3382018022 >> v2:172.21.15.67:3300/0 conn(0x7f96840c9fd0 msgr2=0x7f968415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:05:55.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f968259c700 1 --2- 172.21.15.23:0/3382018022 >> v2:172.21.15.67:3300/0 conn(0x7f96840c9fd0 0x7f968415c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f968259c700 1 -- 172.21.15.23:0/3382018022 >> v1:172.21.15.23:6789/0 conn(0x7f9684158880 legacy=0x7f9684158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f968259c700 1 -- 172.21.15.23:0/3382018022 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968415bda0 con 0x7f9684152ce0 2023-06-22T08:05:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.380+0000 7f9681d9b700 1 --2- 172.21.15.23:0/3382018022 >> v2:172.21.15.67:3300/0 conn(0x7f96840c9fd0 0x7f968415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:05:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.381+0000 7f968259c700 1 --2- 172.21.15.23:0/3382018022 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9684152ce0 0x7f96840cf1b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9674009840 tx=0x7f9674009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f44cbabde1c101a6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.381+0000 7f968159a700 1 -- 172.21.15.23:0/3382018022 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674011830 con 0x7f9684152ce0 2023-06-22T08:05:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.381+0000 7f968159a700 1 -- 172.21.15.23:0/3382018022 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9674011990 con 0x7f9684152ce0 2023-06-22T08:05:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.381+0000 7f968159a700 1 -- 172.21.15.23:0/3382018022 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674011d20 con 0x7f9684152ce0 2023-06-22T08:05:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.381+0000 7f9688fff700 1 -- 172.21.15.23:0/3382018022 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9684152ce0 msgr2=0x7f96840cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.382+0000 7f9688fff700 1 --2- 172.21.15.23:0/3382018022 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9684152ce0 0x7f96840cf1b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9674009840 tx=0x7f9674009b50 comp rx=0 tx=0).stop 2023-06-22T08:05:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.382+0000 7f9688fff700 1 -- 172.21.15.23:0/3382018022 shutdown_connections 2023-06-22T08:05:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.382+0000 7f9688fff700 1 --2- 172.21.15.23:0/3382018022 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9684152ce0 0x7f96840cf1b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.382+0000 7f9688fff700 1 --2- 172.21.15.23:0/3382018022 >> v2:172.21.15.67:3300/0 conn(0x7f96840c9fd0 0x7f968415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.382+0000 7f9688fff700 1 -- 172.21.15.23:0/3382018022 >> 172.21.15.23:0/3382018022 conn(0x7f968406ae30 msgr2=0x7f968406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.382+0000 7f9688fff700 1 -- 172.21.15.23:0/3382018022 shutdown_connections 2023-06-22T08:05:55.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.382+0000 7f9688fff700 1 -- 172.21.15.23:0/3382018022 wait complete. 2023-06-22T08:05:55.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.384+0000 7f9688fff700 1 Processor -- start 2023-06-22T08:05:55.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.384+0000 7f9688fff700 1 -- start start 2023-06-22T08:05:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.385+0000 7f9688fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96840c9fd0 0x7f96841e4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.385+0000 7f9688fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9684152ce0 0x7f9684147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9688fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9684158880 0x7f96841ee2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9688fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96840b49b0 con 0x7f9684158880 2023-06-22T08:05:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9688fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96840b4150 con 0x7f9684152ce0 2023-06-22T08:05:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9688fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96841e4ac0 con 0x7f96840c9fd0 2023-06-22T08:05:55.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9681d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9684152ce0 0x7f9684147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:55.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f968259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96840c9fd0 0x7f96841e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:55.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9681d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9684152ce0 0x7f9684147ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:54166/0 (socket says 172.21.15.23:54166) 2023-06-22T08:05:55.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9681d9b700 1 -- 172.21.15.23:0/3493238107 learned_addr learned my addr 172.21.15.23:0/3493238107 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:55.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9682d9d700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9684158880 0x7f96841ee2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:55.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9681d9b700 1 -- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96840c9fd0 msgr2=0x7f96841e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:55.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9681d9b700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96840c9fd0 0x7f96841e4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:55.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9681d9b700 1 -- 172.21.15.23:0/3493238107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9684158880 msgr2=0x7f96841ee2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:55.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9681d9b700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9684158880 0x7f96841ee2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:55.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f9681d9b700 1 -- 172.21.15.23:0/3493238107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9678009550 con 0x7f9684152ce0 2023-06-22T08:05:55.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.386+0000 7f968259c700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96840c9fd0 0x7f96841e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:05:55.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.387+0000 7f9681d9b700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9684152ce0 0x7f9684147ae0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f967800e9e0 tx=0x7f967800ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:55.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.387+0000 7f9672ffd700 1 -- 172.21.15.23:0/3493238107 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967800ee80 con 0x7f9684152ce0 2023-06-22T08:05:55.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.387+0000 7f9672ffd700 1 -- 172.21.15.23:0/3493238107 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967800c850 con 0x7f9684152ce0 2023-06-22T08:05:55.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.387+0000 7f9688fff700 1 -- 172.21.15.23:0/3493238107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96740094a0 con 0x7f9684152ce0 2023-06-22T08:05:55.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.387+0000 7f9672ffd700 1 -- 172.21.15.23:0/3493238107 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967800cbc0 con 0x7f9684152ce0 2023-06-22T08:05:55.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.387+0000 7f9688fff700 1 -- 172.21.15.23:0/3493238107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96840b43f0 con 0x7f9684152ce0 2023-06-22T08:05:55.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.388+0000 7f9688fff700 1 -- 172.21.15.23:0/3493238107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968404ea20 con 0x7f9684152ce0 2023-06-22T08:05:55.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.389+0000 7f9672ffd700 1 -- 172.21.15.23:0/3493238107 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f967800cd20 con 0x7f9684152ce0 2023-06-22T08:05:55.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.390+0000 7f9672ffd700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f966406ace0 0x7f966406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:55.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.390+0000 7f9672ffd700 1 -- 172.21.15.23:0/3493238107 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 7041+0+0 (secure 0 0 0) 0x7f96780899e0 con 0x7f9684152ce0 2023-06-22T08:05:55.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.390+0000 7f968259c700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f966406ace0 0x7f966406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:55.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.391+0000 7f968259c700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f966406ace0 0x7f966406d190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9674000c00 tx=0x7f9674008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:55.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.395+0000 7f9672ffd700 1 -- 172.21.15.23:0/3493238107 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9678054580 con 0x7f9684152ce0 2023-06-22T08:05:55.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.681+0000 7f9688fff700 1 -- 172.21.15.23:0/3493238107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9684152500 con 0x7f9684152ce0 2023-06-22T08:05:55.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.682+0000 7f9672ffd700 1 -- 172.21.15.23:0/3493238107 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f9678057ba0 con 0x7f9684152ce0 2023-06-22T08:05:55.686 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:05:55.686 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5060,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:05:47.344924+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":4269,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4269":{"gid":4269,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/3365097525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3365097525},{"type":"v1","addr":"172.21.15.23:6834","nonce":3365097525}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:05:55.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 -- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f966406ace0 msgr2=0x7f966406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:55.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f966406ace0 0x7f966406d190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9674000c00 tx=0x7f9674008030 comp rx=0 tx=0).stop 2023-06-22T08:05:55.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 -- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9684152ce0 msgr2=0x7f9684147ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9684152ce0 0x7f9684147ae0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f967800e9e0 tx=0x7f967800ecf0 comp rx=0 tx=0).stop 2023-06-22T08:05:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 -- 172.21.15.23:0/3493238107 shutdown_connections 2023-06-22T08:05:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9684158880 0x7f96841ee2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96840c9fd0 0x7f96841e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f966406ace0 0x7f966406d190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 --2- 172.21.15.23:0/3493238107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9684152ce0 0x7f9684147ae0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 -- 172.21.15.23:0/3493238107 >> 172.21.15.23:0/3493238107 conn(0x7f968406ae30 msgr2=0x7f96840c4410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 -- 172.21.15.23:0/3493238107 shutdown_connections 2023-06-22T08:05:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:55.684+0000 7f9688fff700 1 -- 172.21.15.23:0/3493238107 wait complete. 2023-06-22T08:05:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 44 2023-06-22T08:05:55.707 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:05:57.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:05:57.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.914+0000 7f3db99ff700 1 Processor -- start 2023-06-22T08:05:57.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.914+0000 7f3db99ff700 1 -- start start 2023-06-22T08:05:57.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.915+0000 7f3db99ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3db40c9fd0 0x7f3db40cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:57.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.915+0000 7f3db99ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 0x7f3db40c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:57.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.915+0000 7f3db99ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db415c370 con 0x7f3db4152ce0 2023-06-22T08:05:57.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db99ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db41530b0 con 0x7f3db40c9fd0 2023-06-22T08:05:57.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db99ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db40cf6f0 con 0x7f3db40c40b0 2023-06-22T08:05:57.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db27fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3db40c9fd0 0x7f3db40cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:57.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db37fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 0x7f3db40c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:57.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db27fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3db40c9fd0 0x7f3db40cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:41798/0 (socket says 172.21.15.23:41798) 2023-06-22T08:05:57.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db27fc700 1 -- 172.21.15.23:0/2457953113 learned_addr learned my addr 172.21.15.23:0/2457953113 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:57.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db2ffd700 1 --1- 172.21.15.23:0/2457953113 >> v1:172.21.15.23:6789/0 conn(0x7f3db4152ce0 0x7f3db415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:35262/0 (socket says 172.21.15.23:35262) 2023-06-22T08:05:57.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db27fc700 1 -- 172.21.15.23:0/2457953113 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 msgr2=0x7f3db40c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:05:57.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db27fc700 1 --2- 172.21.15.23:0/2457953113 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 0x7f3db40c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:57.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db27fc700 1 -- 172.21.15.23:0/2457953113 >> v1:172.21.15.23:6789/0 conn(0x7f3db4152ce0 legacy=0x7f3db415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:57.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db27fc700 1 -- 172.21.15.23:0/2457953113 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db415bda0 con 0x7f3db40c9fd0 2023-06-22T08:05:57.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.916+0000 7f3db37fe700 1 --2- 172.21.15.23:0/2457953113 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 0x7f3db40c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:05:57.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.917+0000 7f3db27fc700 1 --2- 172.21.15.23:0/2457953113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3db40c9fd0 0x7f3db40cf1b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3da80098f0 tx=0x7f3da8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6a5ae8c343974b8 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:57.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.917+0000 7f3db1ffb700 1 -- 172.21.15.23:0/2457953113 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da8011830 con 0x7f3db40c9fd0 2023-06-22T08:05:57.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.917+0000 7f3db1ffb700 1 -- 172.21.15.23:0/2457953113 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da8011990 con 0x7f3db40c9fd0 2023-06-22T08:05:57.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.917+0000 7f3db1ffb700 1 -- 172.21.15.23:0/2457953113 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da8011d20 con 0x7f3db40c9fd0 2023-06-22T08:05:57.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.917+0000 7f3db99ff700 1 -- 172.21.15.23:0/2457953113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3db40c9fd0 msgr2=0x7f3db40cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:57.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.917+0000 7f3db99ff700 1 --2- 172.21.15.23:0/2457953113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3db40c9fd0 0x7f3db40cf1b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3da80098f0 tx=0x7f3da8009c00 comp rx=0 tx=0).stop 2023-06-22T08:05:57.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.918+0000 7f3db99ff700 1 -- 172.21.15.23:0/2457953113 shutdown_connections 2023-06-22T08:05:57.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.918+0000 7f3db99ff700 1 --2- 172.21.15.23:0/2457953113 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 0x7f3db40c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:57.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.918+0000 7f3db99ff700 1 --2- 172.21.15.23:0/2457953113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3db40c9fd0 0x7f3db40cf1b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:57.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.918+0000 7f3db99ff700 1 -- 172.21.15.23:0/2457953113 >> 172.21.15.23:0/2457953113 conn(0x7f3db406ae30 msgr2=0x7f3db406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:57.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.918+0000 7f3db99ff700 1 -- 172.21.15.23:0/2457953113 shutdown_connections 2023-06-22T08:05:57.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.918+0000 7f3db99ff700 1 -- 172.21.15.23:0/2457953113 wait complete. 2023-06-22T08:05:57.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.919+0000 7f3db99ff700 1 Processor -- start 2023-06-22T08:05:57.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.920+0000 7f3db99ff700 1 -- start start 2023-06-22T08:05:57.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.920+0000 7f3db99ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 0x7f3db40b2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.920+0000 7f3db99ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3db40c9fd0 0x7f3db414b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db99ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3db4152ce0 0x7f3db41f2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db99ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db40cdc50 con 0x7f3db40c9fd0 2023-06-22T08:05:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db99ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41e2d60 con 0x7f3db4152ce0 2023-06-22T08:05:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db99ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41e31f0 con 0x7f3db40c40b0 2023-06-22T08:05:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db2ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 0x7f3db40b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3db40c9fd0 0x7f3db414b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:57.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3db40c9fd0 0x7f3db414b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54134/0 (socket says 172.21.15.23:54134) 2023-06-22T08:05:57.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db2ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 0x7f3db40b2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:60118/0 (socket says 172.21.15.23:60118) 2023-06-22T08:05:57.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db27fc700 1 -- 172.21.15.23:0/3482861883 learned_addr learned my addr 172.21.15.23:0/3482861883 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:57.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db37fe700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3db4152ce0 0x7f3db41f2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:57.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db27fc700 1 -- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 msgr2=0x7f3db40b2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:57.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db27fc700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 0x7f3db40b2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:57.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db27fc700 1 -- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3db4152ce0 msgr2=0x7f3db41f2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:57.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db27fc700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3db4152ce0 0x7f3db41f2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:57.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db27fc700 1 -- 172.21.15.23:0/3482861883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac009a70 con 0x7f3db40c9fd0 2023-06-22T08:05:57.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.921+0000 7f3db27fc700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3db40c9fd0 0x7f3db414b730 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3da8000c00 tx=0x7f3da8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:57.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.922+0000 7f3d9bfff700 1 -- 172.21.15.23:0/3482861883 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da80043c0 con 0x7f3db40c9fd0 2023-06-22T08:05:57.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.922+0000 7f3d9bfff700 1 -- 172.21.15.23:0/3482861883 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da801f070 con 0x7f3db40c9fd0 2023-06-22T08:05:57.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.922+0000 7f3d9bfff700 1 -- 172.21.15.23:0/3482861883 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da80038c0 con 0x7f3db40c9fd0 2023-06-22T08:05:57.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.922+0000 7f3db99ff700 1 -- 172.21.15.23:0/3482861883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da8009550 con 0x7f3db40c9fd0 2023-06-22T08:05:57.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.922+0000 7f3db99ff700 1 -- 172.21.15.23:0/3482861883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db41e2430 con 0x7f3db40c9fd0 2023-06-22T08:05:57.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.923+0000 7f3db99ff700 1 -- 172.21.15.23:0/3482861883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db408c230 con 0x7f3db40c9fd0 2023-06-22T08:05:57.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.924+0000 7f3d9bfff700 1 -- 172.21.15.23:0/3482861883 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3da8021030 con 0x7f3db40c9fd0 2023-06-22T08:05:57.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.925+0000 7f3d9bfff700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3da006ace0 0x7f3da006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:57.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.925+0000 7f3d9bfff700 1 -- 172.21.15.23:0/3482861883 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f3da808abd0 con 0x7f3db40c9fd0 2023-06-22T08:05:57.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.925+0000 7f3db2ffd700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3da006ace0 0x7f3da006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:57.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.926+0000 7f3db2ffd700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3da006ace0 0x7f3da006d190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3da4000f60 tx=0x7f3da4007ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:57.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:57.931+0000 7f3d9bfff700 1 -- 172.21.15.23:0/3482861883 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3da8055880 con 0x7f3db40c9fd0 2023-06-22T08:05:58.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.194+0000 7f3db99ff700 1 -- 172.21.15.23:0/3482861883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3db404ea20 con 0x7f3db40c9fd0 2023-06-22T08:05:58.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.195+0000 7f3d9bfff700 1 -- 172.21.15.23:0/3482861883 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5491 (secure 0 0 0) 0x7f3da8058ea0 con 0x7f3db40c9fd0 2023-06-22T08:05:58.199 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:05:58.199 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:05:56.352512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:05:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 -- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3da006ace0 msgr2=0x7f3da006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3da006ace0 0x7f3da006d190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3da4000f60 tx=0x7f3da4007ed0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 -- 172.21.15.23:0/3482861883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3db40c9fd0 msgr2=0x7f3db414b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3db40c9fd0 0x7f3db414b730 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3da8000c00 tx=0x7f3da8004750 comp rx=0 tx=0).stop 2023-06-22T08:05:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 -- 172.21.15.23:0/3482861883 shutdown_connections 2023-06-22T08:05:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3da006ace0 0x7f3da006d190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3db4152ce0 0x7f3db41f2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3db40c40b0 0x7f3db40b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 --2- 172.21.15.23:0/3482861883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3db40c9fd0 0x7f3db414b730 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 -- 172.21.15.23:0/3482861883 >> 172.21.15.23:0/3482861883 conn(0x7f3db406ae30 msgr2=0x7f3db40cb840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 -- 172.21.15.23:0/3482861883 shutdown_connections 2023-06-22T08:05:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.197+0000 7f3db99ff700 1 -- 172.21.15.23:0/3482861883 wait complete. 2023-06-22T08:05:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 46 2023-06-22T08:05:58.219 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-06-22T08:05:58.219 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:05:58.220 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:05:58.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.461+0000 7fd30c9ff700 1 Processor -- start 2023-06-22T08:05:58.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.461+0000 7fd30c9ff700 1 -- start start 2023-06-22T08:05:58.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.463+0000 7fd30c9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3080c9fd0 0x7fd3080cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:58.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.463+0000 7fd30c9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3080c40b0 0x7fd3080c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:58.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.463+0000 7fd30c9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30815c370 con 0x7fd308152ce0 2023-06-22T08:05:58.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.463+0000 7fd30c9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3081530b0 con 0x7fd3080c9fd0 2023-06-22T08:05:58.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.463+0000 7fd30c9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3080cf6f0 con 0x7fd3080c40b0 2023-06-22T08:05:58.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.463+0000 7fd2fffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd3080c9fd0 0x7fd3080cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:58.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.463+0000 7fd30659c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd308152ce0 0x7fd30815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:50464/0 (socket says 172.21.15.23:50464) 2023-06-22T08:05:58.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.463+0000 7fd30659c700 1 -- 172.21.15.23:0/741734909 learned_addr learned my addr 172.21.15.23:0/741734909 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:05:58.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.463+0000 7fd306d9d700 1 --2- 172.21.15.23:0/741734909 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3080c40b0 0x7fd3080c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:58.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.464+0000 7fd306d9d700 1 -- 172.21.15.23:0/741734909 >> v2:172.21.15.67:3300/0 conn(0x7fd3080c9fd0 msgr2=0x7fd3080cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:05:58.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.464+0000 7fd306d9d700 1 --2- 172.21.15.23:0/741734909 >> v2:172.21.15.67:3300/0 conn(0x7fd3080c9fd0 0x7fd3080cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.464+0000 7fd306d9d700 1 -- 172.21.15.23:0/741734909 >> v1:172.21.15.67:6789/0 conn(0x7fd308152ce0 legacy=0x7fd30815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:58.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.464+0000 7fd306d9d700 1 -- 172.21.15.23:0/741734909 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30815bda0 con 0x7fd3080c40b0 2023-06-22T08:05:58.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.464+0000 7fd2fffff700 1 --2- 172.21.15.23:0/741734909 >> v2:172.21.15.67:3300/0 conn(0x7fd3080c9fd0 0x7fd3080cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:05:58.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.464+0000 7fd305d9b700 1 -- 172.21.15.23:0/741734909 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395063296 0 0) 0x7fd30815c370 con 0x7fd308152ce0 2023-06-22T08:05:58.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.464+0000 7fd306d9d700 1 --2- 172.21.15.23:0/741734909 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3080c40b0 0x7fd3080c4530 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd300009d90 tx=0x7fd30000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d29dd503a5b3edf server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:58.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.464+0000 7fd305d9b700 1 -- 172.21.15.23:0/741734909 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30000e070 con 0x7fd3080c40b0 2023-06-22T08:05:58.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.464+0000 7fd305d9b700 1 -- 172.21.15.23:0/741734909 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30000bb20 con 0x7fd3080c40b0 2023-06-22T08:05:58.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.464+0000 7fd305d9b700 1 -- 172.21.15.23:0/741734909 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd300012430 con 0x7fd3080c40b0 2023-06-22T08:05:58.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.465+0000 7fd30c9ff700 1 -- 172.21.15.23:0/741734909 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3080c40b0 msgr2=0x7fd3080c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:58.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.465+0000 7fd30c9ff700 1 --2- 172.21.15.23:0/741734909 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3080c40b0 0x7fd3080c4530 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd300009d90 tx=0x7fd30000b330 comp rx=0 tx=0).stop 2023-06-22T08:05:58.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.465+0000 7fd30c9ff700 1 -- 172.21.15.23:0/741734909 shutdown_connections 2023-06-22T08:05:58.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.465+0000 7fd30c9ff700 1 --2- 172.21.15.23:0/741734909 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3080c40b0 0x7fd3080c4530 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.465+0000 7fd30c9ff700 1 --2- 172.21.15.23:0/741734909 >> v2:172.21.15.67:3300/0 conn(0x7fd3080c9fd0 0x7fd3080cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.465+0000 7fd30c9ff700 1 -- 172.21.15.23:0/741734909 >> 172.21.15.23:0/741734909 conn(0x7fd30806ae30 msgr2=0x7fd30806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:58.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.465+0000 7fd30c9ff700 1 -- 172.21.15.23:0/741734909 shutdown_connections 2023-06-22T08:05:58.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.465+0000 7fd30c9ff700 1 -- 172.21.15.23:0/741734909 wait complete. 2023-06-22T08:05:58.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.467+0000 7fd30c9ff700 1 Processor -- start 2023-06-22T08:05:58.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.467+0000 7fd30c9ff700 1 -- start start 2023-06-22T08:05:58.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd30c9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3080c40b0 0x7fd3080b4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:58.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd30c9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3080c9fd0 0x7fd3081f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:58.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd30c9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd308152ce0 0x7fd3081f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:58.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd30c9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3080c7a80 con 0x7fd3080c9fd0 2023-06-22T08:05:58.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd30c9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3081ee180 con 0x7fd3080c40b0 2023-06-22T08:05:58.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd30c9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3080c7150 con 0x7fd308152ce0 2023-06-22T08:05:58.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd30659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3080c40b0 0x7fd3080b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:58.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd306d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd308152ce0 0x7fd3081f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:58.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd30659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3080c40b0 0x7fd3080b4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:41822/0 (socket says 172.21.15.23:41822) 2023-06-22T08:05:58.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd30659c700 1 -- 172.21.15.23:0/1935646010 learned_addr learned my addr 172.21.15.23:0/1935646010 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:05:58.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.468+0000 7fd2fffff700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3080c9fd0 0x7fd3081f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:58.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd30659c700 1 -- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd308152ce0 msgr2=0x7fd3081f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:58.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd30659c700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd308152ce0 0x7fd3081f1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd30659c700 1 -- 172.21.15.23:0/1935646010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3080c9fd0 msgr2=0x7fd3081f16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:58.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd30659c700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3080c9fd0 0x7fd3081f16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd30659c700 1 -- 172.21.15.23:0/1935646010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f00094a0 con 0x7fd3080c40b0 2023-06-22T08:05:58.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd306d9d700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd308152ce0 0x7fd3081f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:05:58.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd30659c700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3080c40b0 0x7fd3080b4040 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800c550 tx=0x7fd2f800c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:58.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd2feffd700 1 -- 172.21.15.23:0/1935646010 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8004c80 con 0x7fd3080c40b0 2023-06-22T08:05:58.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd2feffd700 1 -- 172.21.15.23:0/1935646010 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f8004e00 con 0x7fd3080c40b0 2023-06-22T08:05:58.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd30c9ff700 1 -- 172.21.15.23:0/1935646010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd300009a70 con 0x7fd3080c40b0 2023-06-22T08:05:58.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd2feffd700 1 -- 172.21.15.23:0/1935646010 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8007310 con 0x7fd3080c40b0 2023-06-22T08:05:58.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.469+0000 7fd30c9ff700 1 -- 172.21.15.23:0/1935646010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3080c7710 con 0x7fd3080c40b0 2023-06-22T08:05:58.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.470+0000 7fd30c9ff700 1 -- 172.21.15.23:0/1935646010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30804ea20 con 0x7fd3080c40b0 2023-06-22T08:05:58.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.471+0000 7fd2feffd700 1 -- 172.21.15.23:0/1935646010 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd2f8007470 con 0x7fd3080c40b0 2023-06-22T08:05:58.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.472+0000 7fd2feffd700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd2e806ace0 0x7fd2e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:05:58.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.473+0000 7fd2feffd700 1 -- 172.21.15.23:0/1935646010 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7fd2f8089440 con 0x7fd3080c40b0 2023-06-22T08:05:58.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.473+0000 7fd2fffff700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd2e806ace0 0x7fd2e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:05:58.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.473+0000 7fd2fffff700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd2e806ace0 0x7fd2e806d190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd30814e9a0 tx=0x7fd2f0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:05:58.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.477+0000 7fd2feffd700 1 -- 172.21.15.23:0/1935646010 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd2f80540f0 con 0x7fd3080c40b0 2023-06-22T08:05:58.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.754+0000 7fd30c9ff700 1 -- 172.21.15.23:0/1935646010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3081597c0 con 0x7fd3080c40b0 2023-06-22T08:05:58.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.755+0000 7fd2feffd700 1 -- 172.21.15.23:0/1935646010 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5491 (secure 0 0 0) 0x7fd2f8057710 con 0x7fd3080c40b0 2023-06-22T08:05:58.758 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:05:58.759 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:05:56.352512+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:05:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.757+0000 7fd30c9ff700 1 -- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd2e806ace0 msgr2=0x7fd2e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.757+0000 7fd30c9ff700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd2e806ace0 0x7fd2e806d190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd30814e9a0 tx=0x7fd2f0009210 comp rx=0 tx=0).stop 2023-06-22T08:05:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.757+0000 7fd30c9ff700 1 -- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3080c40b0 msgr2=0x7fd3080b4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:05:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.757+0000 7fd30c9ff700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3080c40b0 0x7fd3080b4040 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd2f800c550 tx=0x7fd2f800c860 comp rx=0 tx=0).stop 2023-06-22T08:05:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.757+0000 7fd30c9ff700 1 -- 172.21.15.23:0/1935646010 shutdown_connections 2023-06-22T08:05:58.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.757+0000 7fd30c9ff700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd308152ce0 0x7fd3081f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.757+0000 7fd30c9ff700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd2e806ace0 0x7fd2e806d190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.757+0000 7fd30c9ff700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3080c40b0 0x7fd3080b4040 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.757+0000 7fd30c9ff700 1 --2- 172.21.15.23:0/1935646010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3080c9fd0 0x7fd3081f16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:05:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.757+0000 7fd30c9ff700 1 -- 172.21.15.23:0/1935646010 >> 172.21.15.23:0/1935646010 conn(0x7fd30806ae30 msgr2=0x7fd30814d740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:05:58.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.758+0000 7fd30c9ff700 1 -- 172.21.15.23:0/1935646010 shutdown_connections 2023-06-22T08:05:58.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:05:58.758+0000 7fd30c9ff700 1 -- 172.21.15.23:0/1935646010 wait complete. 2023-06-22T08:05:58.763 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 46 2023-06-22T08:05:58.781 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 46, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 46, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4272": { "addr": "172.21.15.23:6835/3894174037", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 3894174037, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 3894174037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4272, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 12 }, "gid_4279": { "addr": "172.21.15.67:6836/1962314516", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 1962314516, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 1962314516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4285": { "addr": "172.21.15.67:6839/567663545", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 567663545, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 567663545, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5060": { "addr": "172.21.15.67:6837/1551460358", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1551460358, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1551460358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5060, "incarnation": 46, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:replay", "state_seq": 1 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:05:56.352512+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4279, "mds_2": 4272, "mds_3": 5060, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:06:00.782 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:06:01.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.253+0000 7fbdce1ff700 1 Processor -- start 2023-06-22T08:06:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.255+0000 7fbdce1ff700 1 -- start start 2023-06-22T08:06:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.258+0000 7fbdce1ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbdc815d9b0 0x7fbdc806d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.258+0000 7fbdce1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdc806a280 0x7fbdc80c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.258+0000 7fbdce1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc80c4d70 con 0x7fbdc8069eb0 2023-06-22T08:06:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.258+0000 7fbdce1ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdc80703c0 con 0x7fbdc815d9b0 2023-06-22T08:06:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.258+0000 7fbdce1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80c5610 con 0x7fbdc806a280 2023-06-22T08:06:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.260+0000 7fbdc6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbdc815d9b0 0x7fbdc806d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.260+0000 7fbdc6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbdc815d9b0 0x7fbdc806d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54158/0 (socket says 172.21.15.23:54158) 2023-06-22T08:06:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.260+0000 7fbdc6ffd700 1 -- 172.21.15.23:0/684196253 learned_addr learned my addr 172.21.15.23:0/684196253 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.261+0000 7fbdc7fff700 1 --2- 172.21.15.23:0/684196253 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdc806a280 0x7fbdc80c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.262+0000 7fbdc67fc700 1 -- 172.21.15.23:0/684196253 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004510012 0 0) 0x7fbdc80c4d70 con 0x7fbdc8069eb0 2023-06-22T08:06:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.262+0000 7fbdc67fc700 1 -- 172.21.15.23:0/684196253 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb00035b0 con 0x7fbdc8069eb0 2023-06-22T08:06:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.263+0000 7fbdc6ffd700 1 -- 172.21.15.23:0/684196253 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdc806a280 msgr2=0x7fbdc80c5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:06:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.263+0000 7fbdc6ffd700 1 --2- 172.21.15.23:0/684196253 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdc806a280 0x7fbdc80c5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.263+0000 7fbdc6ffd700 1 -- 172.21.15.23:0/684196253 >> v1:172.21.15.67:6789/0 conn(0x7fbdc8069eb0 legacy=0x7fbdc806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.263+0000 7fbdc6ffd700 1 -- 172.21.15.23:0/684196253 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc80c5fe0 con 0x7fbdc815d9b0 2023-06-22T08:06:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.264+0000 7fbdc6ffd700 1 --2- 172.21.15.23:0/684196253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc815d9b0 0x7fbdc806d050 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc0098f0 tx=0x7fbdbc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2007a215aa267ede server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:01.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.264+0000 7fbdc67fc700 1 -- 172.21.15.23:0/684196253 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc011830 con 0x7fbdc815d9b0 2023-06-22T08:06:01.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.264+0000 7fbdc67fc700 1 -- 172.21.15.23:0/684196253 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdbc011990 con 0x7fbdc815d9b0 2023-06-22T08:06:01.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.264+0000 7fbdc67fc700 1 -- 172.21.15.23:0/684196253 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc011d20 con 0x7fbdc815d9b0 2023-06-22T08:06:01.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.266+0000 7fbdce1ff700 1 -- 172.21.15.23:0/684196253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc815d9b0 msgr2=0x7fbdc806d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:01.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.266+0000 7fbdce1ff700 1 --2- 172.21.15.23:0/684196253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc815d9b0 0x7fbdc806d050 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc0098f0 tx=0x7fbdbc009c00 comp rx=0 tx=0).stop 2023-06-22T08:06:01.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.266+0000 7fbdce1ff700 1 -- 172.21.15.23:0/684196253 shutdown_connections 2023-06-22T08:06:01.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.266+0000 7fbdce1ff700 1 --2- 172.21.15.23:0/684196253 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdc806a280 0x7fbdc80c5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:01.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.266+0000 7fbdce1ff700 1 --2- 172.21.15.23:0/684196253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc815d9b0 0x7fbdc806d050 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:01.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.266+0000 7fbdce1ff700 1 -- 172.21.15.23:0/684196253 >> 172.21.15.23:0/684196253 conn(0x7fbdc80ca3f0 msgr2=0x7fbdc80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:01.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.267+0000 7fbdce1ff700 1 -- 172.21.15.23:0/684196253 shutdown_connections 2023-06-22T08:06:01.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.267+0000 7fbdce1ff700 1 -- 172.21.15.23:0/684196253 wait complete. 2023-06-22T08:06:01.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.270+0000 7fbdce1ff700 1 Processor -- start 2023-06-22T08:06:01.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.271+0000 7fbdce1ff700 1 -- start start 2023-06-22T08:06:01.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.273+0000 7fbdce1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc8069eb0 0x7fbdc813c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:01.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.273+0000 7fbdce1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc806a280 0x7fbdc80c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:01.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.273+0000 7fbdce1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdc815d9b0 0x7fbdc81fe2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:01.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.273+0000 7fbdce1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc813bfd0 con 0x7fbdc806a280 2023-06-22T08:06:01.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.273+0000 7fbdce1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8156990 con 0x7fbdc8069eb0 2023-06-22T08:06:01.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.273+0000 7fbdce1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc813cf20 con 0x7fbdc815d9b0 2023-06-22T08:06:01.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.274+0000 7fbdc77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc8069eb0 0x7fbdc813c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:01.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.274+0000 7fbdc77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc8069eb0 0x7fbdc813c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:41834/0 (socket says 172.21.15.23:41834) 2023-06-22T08:06:01.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.274+0000 7fbdc77fe700 1 -- 172.21.15.23:0/2989837489 learned_addr learned my addr 172.21.15.23:0/2989837489 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:01.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.274+0000 7fbdc7fff700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdc815d9b0 0x7fbdc81fe2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:01.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.275+0000 7fbdc6ffd700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc806a280 0x7fbdc80c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:01.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.276+0000 7fbdc77fe700 1 -- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdc815d9b0 msgr2=0x7fbdc81fe2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:01.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.276+0000 7fbdc77fe700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdc815d9b0 0x7fbdc81fe2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:01.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.276+0000 7fbdc77fe700 1 -- 172.21.15.23:0/2989837489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc806a280 msgr2=0x7fbdc80c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:01.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.276+0000 7fbdc77fe700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc806a280 0x7fbdc80c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:01.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.276+0000 7fbdc77fe700 1 -- 172.21.15.23:0/2989837489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc009550 con 0x7fbdc8069eb0 2023-06-22T08:06:01.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.280+0000 7fbdc77fe700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc8069eb0 0x7fbdc813c9e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800c500 tx=0x7fbdb800c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:01.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.285+0000 7fbdb7fff700 1 -- 172.21.15.23:0/2989837489 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb80052d0 con 0x7fbdc8069eb0 2023-06-22T08:06:01.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.285+0000 7fbdb7fff700 1 -- 172.21.15.23:0/2989837489 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb8005430 con 0x7fbdc8069eb0 2023-06-22T08:06:01.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.285+0000 7fbdb7fff700 1 -- 172.21.15.23:0/2989837489 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb80038c0 con 0x7fbdc8069eb0 2023-06-22T08:06:01.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.286+0000 7fbdce1ff700 1 -- 172.21.15.23:0/2989837489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc815aec0 con 0x7fbdc8069eb0 2023-06-22T08:06:01.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.286+0000 7fbdce1ff700 1 -- 172.21.15.23:0/2989837489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc815b480 con 0x7fbdc8069eb0 2023-06-22T08:06:01.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.290+0000 7fbdce1ff700 1 -- 172.21.15.23:0/2989837489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda40052f0 con 0x7fbdc8069eb0 2023-06-22T08:06:01.299 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.296+0000 7fbdb7fff700 1 -- 172.21.15.23:0/2989837489 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbdb8003a20 con 0x7fbdc8069eb0 2023-06-22T08:06:01.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.297+0000 7fbdb7fff700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdac06adb0 0x7fbdac06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:01.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.297+0000 7fbdb7fff700 1 -- 172.21.15.23:0/2989837489 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7fbdb8089170 con 0x7fbdc8069eb0 2023-06-22T08:06:01.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.302+0000 7fbdc6ffd700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdac06adb0 0x7fbdac06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:01.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.302+0000 7fbdb7fff700 1 -- 172.21.15.23:0/2989837489 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbdb8053d80 con 0x7fbdc8069eb0 2023-06-22T08:06:01.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.317+0000 7fbdc6ffd700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdac06adb0 0x7fbdac06d260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc0098f0 tx=0x7fbdbc01e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:01.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.815+0000 7fbdce1ff700 1 -- 172.21.15.23:0/2989837489 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbda4005870 con 0x7fbdc8069eb0 2023-06-22T08:06:01.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.816+0000 7fbdb7fff700 1 -- 172.21.15.23:0/2989837489 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5490 (secure 0 0 0) 0x7fbdb80573a0 con 0x7fbdc8069eb0 2023-06-22T08:06:01.820 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:06:01.820 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:01.715183+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:06:01.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 -- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdac06adb0 msgr2=0x7fbdac06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:01.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdac06adb0 0x7fbdac06d260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc0098f0 tx=0x7fbdbc01e040 comp rx=0 tx=0).stop 2023-06-22T08:06:01.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 -- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc8069eb0 msgr2=0x7fbdc813c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:01.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc8069eb0 0x7fbdc813c9e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800c500 tx=0x7fbdb800c810 comp rx=0 tx=0).stop 2023-06-22T08:06:01.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 -- 172.21.15.23:0/2989837489 shutdown_connections 2023-06-22T08:06:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbdc815d9b0 0x7fbdc81fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbdac06adb0 0x7fbdac06d260 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdc8069eb0 0x7fbdc813c9e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 --2- 172.21.15.23:0/2989837489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc806a280 0x7fbdc80c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 -- 172.21.15.23:0/2989837489 >> 172.21.15.23:0/2989837489 conn(0x7fbdc80ca3f0 msgr2=0x7fbdc8151c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 -- 172.21.15.23:0/2989837489 shutdown_connections 2023-06-22T08:06:01.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:01.819+0000 7fbdce1ff700 1 -- 172.21.15.23:0/2989837489 wait complete. 2023-06-22T08:06:01.824 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 47 2023-06-22T08:06:01.840 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:06:03.175 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 7% (833/11797) Updating files: 8% (944/11797) Updating files: 9% (1062/11797) Updating files: 10% (1180/11797) Updating files: 11% (1298/11797) Updating files: 12% (1416/11797) Updating files: 13% (1534/11797) Updating files: 13% (1549/11797) Updating files: 14% (1652/11797) Updating files: 15% (1770/11797) Updating files: 16% (1888/11797) Updating files: 16% (1968/11797) Updating files: 17% (2006/11797) Updating files: 18% (2124/11797) Updating files: 19% (2242/11797) Updating files: 20% (2360/11797) Updating files: 21% (2478/11797) Updating files: 22% (2596/11797) Updating files: 22% (2706/11797) Updating files: 23% (2714/11797) Updating files: 24% (2832/11797) Updating files: 25% (2950/11797) Updating files: 26% (3068/11797) Updating files: 26% (3101/11797) Updating files: 27% (3186/11797) Updating files: 28% (3304/11797) Updating files: 29% (3422/11797) Updating files: 30% (3540/11797) Updating files: 31% (3658/11797) Updating files: 32% (3776/11797) Updating files: 32% (3826/11797) Updating files: 33% (3894/11797) Updating files: 34% (4011/11797) Updating files: 35% (4129/11797) Updating files: 36% (4247/11797) Updating files: 37% (4365/11797) Updating files: 38% (4483/11797) Updating files: 38% (4525/11797) Updating files: 39% (4601/11797) Updating files: 40% (4719/11797) Updating files: 41% (4837/11797) Updating files: 42% (4955/11797) Updating files: 42% (5011/11797) Updating files: 43% (5073/11797) Updating files: 44% (5191/11797) Updating files: 45% (5309/11797) Updating files: 46% (5427/11797) Updating files: 46% (5536/11797) Updating files: 47% (5545/11797) Updating files: 48% (5663/11797) Updating files: 49% (5781/11797) Updating files: 50% (5899/11797) Updating files: 51% (6017/11797) Updating files: 52% (6135/11797) Updating files: 52% (6224/11797) Updating files: 53% (6253/11797) Updating files: 54% (6371/11797) Updating files: 55% (6489/11797) Updating files: 56% (6607/11797) Updating files: 57% (6725/11797) Updating files: 57% (6836/11797) Updating files: 58% (6843/11797) Updating files: 59% (6961/11797) Updating files: 60% (7079/11797) Updating files: 61% (7197/11797) Updating files: 62% (7315/11797) Updating files: 63% (7433/11797) Updating files: 63% (7493/11797) Updating files: 64% (7551/11797) Updating files: 65% (7669/11797) Updating files: 66% (7787/11797) Updating files: 67% (7904/11797) Updating files: 68% (8022/11797) Updating files: 69% (8140/11797) Updating files: 69% (8224/11797) Updating files: 70% (8258/11797) Updating files: 71% (8376/11797) Updating files: 72% (8494/11797) Updating files: 73% (8612/11797) Updating files: 73% (8709/11797) Updating files: 74% (8730/11797) Updating files: 75% (8848/11797) Updating files: 76% (8966/11797) Updating files: 77% (9084/11797) Updating files: 78% (9202/11797) Updating files: 78% (9246/11797) Updating files: 79% (9320/11797) Updating files: 80% (9438/11797) Updating files: 81% (9556/11797) Updating files: 82% (9674/11797) Updating files: 82% (9787/11797) Updating files: 83% (9792/11797) Updating files: 84% (9910/11797) Updating files: 85% (10028/11797) Updating files: 86% (10146/11797) Updating files: 87% (10264/11797) Updating files: 87% (10371/11797) Updating files: 88% (10382/11797) Updating files: 89% (10500/11797) Updating files: 90% (10618/11797) Updating files: 91% (10736/11797) Updating files: 92% (10854/11797) Updating files: 93% (10972/11797) Updating files: 94% (11090/11797) Updating files: 94% (11155/11797) Updating files: 95% (11208/11797) Updating files: 96% (11326/11797) Updating files: 97% (11444/11797) Updating files: 98% (11562/11797) Updating files: 99% (11680/11797) Updating files: 99% (11765/11797) Updating files: 99% (11768/11797) Updating files: 100% (11797/11797) Updating files: 100% (11797/11797), done. 2023-06-22T08:06:03.222 INFO:tasks.workunit.client.0.smithi023.stderr:+ trap - EXIT 2023-06-22T08:06:03.222 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ceph 2023-06-22T08:06:03.223 INFO:tasks.workunit.client.0.smithi023.stderr:++ seq 1 90 2023-06-22T08:06:03.225 INFO:tasks.workunit.client.0.smithi023.stderr:+ versions='1 2023-06-22T08:06:03.225 INFO:tasks.workunit.client.0.smithi023.stderr:2 2023-06-22T08:06:03.225 INFO:tasks.workunit.client.0.smithi023.stderr:3 2023-06-22T08:06:03.225 INFO:tasks.workunit.client.0.smithi023.stderr:4 2023-06-22T08:06:03.226 INFO:tasks.workunit.client.0.smithi023.stderr:5 2023-06-22T08:06:03.226 INFO:tasks.workunit.client.0.smithi023.stderr:6 2023-06-22T08:06:03.226 INFO:tasks.workunit.client.0.smithi023.stderr:7 2023-06-22T08:06:03.226 INFO:tasks.workunit.client.0.smithi023.stderr:8 2023-06-22T08:06:03.227 INFO:tasks.workunit.client.0.smithi023.stderr:9 2023-06-22T08:06:03.227 INFO:tasks.workunit.client.0.smithi023.stderr:10 2023-06-22T08:06:03.227 INFO:tasks.workunit.client.0.smithi023.stderr:11 2023-06-22T08:06:03.227 INFO:tasks.workunit.client.0.smithi023.stderr:12 2023-06-22T08:06:03.228 INFO:tasks.workunit.client.0.smithi023.stderr:13 2023-06-22T08:06:03.228 INFO:tasks.workunit.client.0.smithi023.stderr:14 2023-06-22T08:06:03.228 INFO:tasks.workunit.client.0.smithi023.stderr:15 2023-06-22T08:06:03.228 INFO:tasks.workunit.client.0.smithi023.stderr:16 2023-06-22T08:06:03.228 INFO:tasks.workunit.client.0.smithi023.stderr:17 2023-06-22T08:06:03.229 INFO:tasks.workunit.client.0.smithi023.stderr:18 2023-06-22T08:06:03.229 INFO:tasks.workunit.client.0.smithi023.stderr:19 2023-06-22T08:06:03.229 INFO:tasks.workunit.client.0.smithi023.stderr:20 2023-06-22T08:06:03.229 INFO:tasks.workunit.client.0.smithi023.stderr:21 2023-06-22T08:06:03.230 INFO:tasks.workunit.client.0.smithi023.stderr:22 2023-06-22T08:06:03.230 INFO:tasks.workunit.client.0.smithi023.stderr:23 2023-06-22T08:06:03.230 INFO:tasks.workunit.client.0.smithi023.stderr:24 2023-06-22T08:06:03.230 INFO:tasks.workunit.client.0.smithi023.stderr:25 2023-06-22T08:06:03.231 INFO:tasks.workunit.client.0.smithi023.stderr:26 2023-06-22T08:06:03.231 INFO:tasks.workunit.client.0.smithi023.stderr:27 2023-06-22T08:06:03.232 INFO:tasks.workunit.client.0.smithi023.stderr:28 2023-06-22T08:06:03.232 INFO:tasks.workunit.client.0.smithi023.stderr:29 2023-06-22T08:06:03.232 INFO:tasks.workunit.client.0.smithi023.stderr:30 2023-06-22T08:06:03.232 INFO:tasks.workunit.client.0.smithi023.stderr:31 2023-06-22T08:06:03.233 INFO:tasks.workunit.client.0.smithi023.stderr:32 2023-06-22T08:06:03.233 INFO:tasks.workunit.client.0.smithi023.stderr:33 2023-06-22T08:06:03.233 INFO:tasks.workunit.client.0.smithi023.stderr:34 2023-06-22T08:06:03.233 INFO:tasks.workunit.client.0.smithi023.stderr:35 2023-06-22T08:06:03.233 INFO:tasks.workunit.client.0.smithi023.stderr:36 2023-06-22T08:06:03.233 INFO:tasks.workunit.client.0.smithi023.stderr:37 2023-06-22T08:06:03.233 INFO:tasks.workunit.client.0.smithi023.stderr:38 2023-06-22T08:06:03.234 INFO:tasks.workunit.client.0.smithi023.stderr:39 2023-06-22T08:06:03.234 INFO:tasks.workunit.client.0.smithi023.stderr:40 2023-06-22T08:06:03.234 INFO:tasks.workunit.client.0.smithi023.stderr:41 2023-06-22T08:06:03.234 INFO:tasks.workunit.client.0.smithi023.stderr:42 2023-06-22T08:06:03.234 INFO:tasks.workunit.client.0.smithi023.stderr:43 2023-06-22T08:06:03.234 INFO:tasks.workunit.client.0.smithi023.stderr:44 2023-06-22T08:06:03.234 INFO:tasks.workunit.client.0.smithi023.stderr:45 2023-06-22T08:06:03.235 INFO:tasks.workunit.client.0.smithi023.stderr:46 2023-06-22T08:06:03.235 INFO:tasks.workunit.client.0.smithi023.stderr:47 2023-06-22T08:06:03.235 INFO:tasks.workunit.client.0.smithi023.stderr:48 2023-06-22T08:06:03.235 INFO:tasks.workunit.client.0.smithi023.stderr:49 2023-06-22T08:06:03.235 INFO:tasks.workunit.client.0.smithi023.stderr:50 2023-06-22T08:06:03.235 INFO:tasks.workunit.client.0.smithi023.stderr:51 2023-06-22T08:06:03.235 INFO:tasks.workunit.client.0.smithi023.stderr:52 2023-06-22T08:06:03.236 INFO:tasks.workunit.client.0.smithi023.stderr:53 2023-06-22T08:06:03.236 INFO:tasks.workunit.client.0.smithi023.stderr:54 2023-06-22T08:06:03.236 INFO:tasks.workunit.client.0.smithi023.stderr:55 2023-06-22T08:06:03.236 INFO:tasks.workunit.client.0.smithi023.stderr:56 2023-06-22T08:06:03.236 INFO:tasks.workunit.client.0.smithi023.stderr:57 2023-06-22T08:06:03.236 INFO:tasks.workunit.client.0.smithi023.stderr:58 2023-06-22T08:06:03.236 INFO:tasks.workunit.client.0.smithi023.stderr:59 2023-06-22T08:06:03.237 INFO:tasks.workunit.client.0.smithi023.stderr:60 2023-06-22T08:06:03.237 INFO:tasks.workunit.client.0.smithi023.stderr:61 2023-06-22T08:06:03.237 INFO:tasks.workunit.client.0.smithi023.stderr:62 2023-06-22T08:06:03.237 INFO:tasks.workunit.client.0.smithi023.stderr:63 2023-06-22T08:06:03.237 INFO:tasks.workunit.client.0.smithi023.stderr:64 2023-06-22T08:06:03.237 INFO:tasks.workunit.client.0.smithi023.stderr:65 2023-06-22T08:06:03.237 INFO:tasks.workunit.client.0.smithi023.stderr:66 2023-06-22T08:06:03.237 INFO:tasks.workunit.client.0.smithi023.stderr:67 2023-06-22T08:06:03.238 INFO:tasks.workunit.client.0.smithi023.stderr:68 2023-06-22T08:06:03.238 INFO:tasks.workunit.client.0.smithi023.stderr:69 2023-06-22T08:06:03.238 INFO:tasks.workunit.client.0.smithi023.stderr:70 2023-06-22T08:06:03.238 INFO:tasks.workunit.client.0.smithi023.stderr:71 2023-06-22T08:06:03.238 INFO:tasks.workunit.client.0.smithi023.stderr:72 2023-06-22T08:06:03.238 INFO:tasks.workunit.client.0.smithi023.stderr:73 2023-06-22T08:06:03.238 INFO:tasks.workunit.client.0.smithi023.stderr:74 2023-06-22T08:06:03.238 INFO:tasks.workunit.client.0.smithi023.stderr:75 2023-06-22T08:06:03.239 INFO:tasks.workunit.client.0.smithi023.stderr:76 2023-06-22T08:06:03.239 INFO:tasks.workunit.client.0.smithi023.stderr:77 2023-06-22T08:06:03.239 INFO:tasks.workunit.client.0.smithi023.stderr:78 2023-06-22T08:06:03.240 INFO:tasks.workunit.client.0.smithi023.stderr:79 2023-06-22T08:06:03.240 INFO:tasks.workunit.client.0.smithi023.stderr:80 2023-06-22T08:06:03.241 INFO:tasks.workunit.client.0.smithi023.stderr:81 2023-06-22T08:06:03.241 INFO:tasks.workunit.client.0.smithi023.stderr:82 2023-06-22T08:06:03.241 INFO:tasks.workunit.client.0.smithi023.stderr:83 2023-06-22T08:06:03.241 INFO:tasks.workunit.client.0.smithi023.stderr:84 2023-06-22T08:06:03.241 INFO:tasks.workunit.client.0.smithi023.stderr:85 2023-06-22T08:06:03.241 INFO:tasks.workunit.client.0.smithi023.stderr:86 2023-06-22T08:06:03.242 INFO:tasks.workunit.client.0.smithi023.stderr:87 2023-06-22T08:06:03.242 INFO:tasks.workunit.client.0.smithi023.stderr:88 2023-06-22T08:06:03.242 INFO:tasks.workunit.client.0.smithi023.stderr:89 2023-06-22T08:06:03.242 INFO:tasks.workunit.client.0.smithi023.stderr:90' 2023-06-22T08:06:03.242 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:06:03.242 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 1 -eq 48 ']' 2023-06-22T08:06:03.242 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.1 2023-06-22T08:06:03.243 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.1 2023-06-22T08:06:03.243 INFO:tasks.workunit.client.0.smithi023.stdout:v0.1 2023-06-22T08:06:03.243 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.1 2023-06-22T08:06:03.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:06:04.090 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.087+0000 7f2b583ff700 1 Processor -- start 2023-06-22T08:06:04.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.087+0000 7f2b583ff700 1 -- start start 2023-06-22T08:06:04.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.090+0000 7f2b583ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2b5006d050 0x7f2b5006a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:04.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.090+0000 7f2b583ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b500c4d70 con 0x7f2b5015d9b0 2023-06-22T08:06:04.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.090+0000 7f2b583ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b500c5610 con 0x7f2b50069eb0 2023-06-22T08:06:04.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.090+0000 7f2b583ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b500703c0 con 0x7f2b5006d050 2023-06-22T08:06:04.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.090+0000 7f2b5619b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2b50069eb0 0x7f2b5006fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:50476/0 (socket says 172.21.15.23:50476) 2023-06-22T08:06:04.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.090+0000 7f2b5619b700 1 -- 172.21.15.23:0/1049277553 learned_addr learned my addr 172.21.15.23:0/1049277553 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:06:04.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.091+0000 7f2b5699c700 1 --2- 172.21.15.23:0/1049277553 >> v2:172.21.15.23:3300/0 conn(0x7f2b5006d050 0x7f2b5006a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:04.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.092+0000 7f2b5699c700 1 -- 172.21.15.23:0/1049277553 >> v1:172.21.15.67:6789/0 conn(0x7f2b50069eb0 legacy=0x7f2b5006fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:04.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.092+0000 7f2b5699c700 1 -- 172.21.15.23:0/1049277553 >> v1:172.21.15.23:6789/0 conn(0x7f2b5015d9b0 legacy=0x7f2b50165590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:06:04.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.092+0000 7f2b5699c700 1 -- 172.21.15.23:0/1049277553 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b50071340 con 0x7f2b5006d050 2023-06-22T08:06:04.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.093+0000 7f2b5699c700 1 --2- 172.21.15.23:0/1049277553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b5006d050 0x7f2b5006a280 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2b48009d90 tx=0x7f2b4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55723e43a8e3e841 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:04.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.094+0000 7f2b55199700 1 -- 172.21.15.23:0/1049277553 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4800e070 con 0x7f2b5006d050 2023-06-22T08:06:04.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.094+0000 7f2b55199700 1 -- 172.21.15.23:0/1049277553 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4800bad0 con 0x7f2b5006d050 2023-06-22T08:06:04.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.094+0000 7f2b55199700 1 -- 172.21.15.23:0/1049277553 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4800be60 con 0x7f2b5006d050 2023-06-22T08:06:04.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.095+0000 7f2b583ff700 1 -- 172.21.15.23:0/1049277553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b5006d050 msgr2=0x7f2b5006a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:04.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.095+0000 7f2b583ff700 1 --2- 172.21.15.23:0/1049277553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b5006d050 0x7f2b5006a280 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2b48009d90 tx=0x7f2b4800b330 comp rx=0 tx=0).stop 2023-06-22T08:06:04.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.097+0000 7f2b583ff700 1 -- 172.21.15.23:0/1049277553 shutdown_connections 2023-06-22T08:06:04.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.097+0000 7f2b583ff700 1 --2- 172.21.15.23:0/1049277553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b5006d050 0x7f2b5006a280 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:04.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.097+0000 7f2b583ff700 1 -- 172.21.15.23:0/1049277553 >> 172.21.15.23:0/1049277553 conn(0x7f2b500ca3f0 msgr2=0x7f2b500ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:04.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.097+0000 7f2b583ff700 1 -- 172.21.15.23:0/1049277553 shutdown_connections 2023-06-22T08:06:04.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.097+0000 7f2b583ff700 1 -- 172.21.15.23:0/1049277553 wait complete. 2023-06-22T08:06:04.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.099+0000 7f2b583ff700 1 Processor -- start 2023-06-22T08:06:04.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.100+0000 7f2b583ff700 1 -- start start 2023-06-22T08:06:04.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.103+0000 7f2b583ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b50069eb0 0x7f2b500c3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:04.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.103+0000 7f2b583ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b5006d050 0x7f2b5015fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:04.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.103+0000 7f2b583ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b5015d9b0 0x7f2b50160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:04.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.103+0000 7f2b583ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b50157da0 con 0x7f2b5006d050 2023-06-22T08:06:04.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.103+0000 7f2b583ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b501f2ce0 con 0x7f2b5015d9b0 2023-06-22T08:06:04.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.103+0000 7f2b583ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b501f2fa0 con 0x7f2b50069eb0 2023-06-22T08:06:04.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.104+0000 7f2b5699c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b5015d9b0 0x7f2b50160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:04.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.104+0000 7f2b5699c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b5015d9b0 0x7f2b50160410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:41846/0 (socket says 172.21.15.23:41846) 2023-06-22T08:06:04.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.104+0000 7f2b5699c700 1 -- 172.21.15.23:0/2089392336 learned_addr learned my addr 172.21.15.23:0/2089392336 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:04.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.104+0000 7f2b5599a700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b5006d050 0x7f2b5015fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:04.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.104+0000 7f2b5619b700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b50069eb0 0x7f2b500c3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:04.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.105+0000 7f2b5619b700 1 -- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b5015d9b0 msgr2=0x7f2b50160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:04.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.105+0000 7f2b5619b700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b5015d9b0 0x7f2b50160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:04.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.105+0000 7f2b5619b700 1 -- 172.21.15.23:0/2089392336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b5006d050 msgr2=0x7f2b5015fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:04.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.105+0000 7f2b5619b700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b5006d050 0x7f2b5015fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:04.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.105+0000 7f2b5619b700 1 -- 172.21.15.23:0/2089392336 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b48009a70 con 0x7f2b50069eb0 2023-06-22T08:06:04.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.106+0000 7f2b5619b700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b50069eb0 0x7f2b500c3590 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00c780 tx=0x7f2b4c00ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:04.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.106+0000 7f2b46ffd700 1 -- 172.21.15.23:0/2089392336 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c004c80 con 0x7f2b50069eb0 2023-06-22T08:06:04.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.106+0000 7f2b46ffd700 1 -- 172.21.15.23:0/2089392336 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4c004de0 con 0x7f2b50069eb0 2023-06-22T08:06:04.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.106+0000 7f2b46ffd700 1 -- 172.21.15.23:0/2089392336 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c0072f0 con 0x7f2b50069eb0 2023-06-22T08:06:04.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.106+0000 7f2b583ff700 1 -- 172.21.15.23:0/2089392336 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b501f3280 con 0x7f2b50069eb0 2023-06-22T08:06:04.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.107+0000 7f2b583ff700 1 -- 172.21.15.23:0/2089392336 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b501f3840 con 0x7f2b50069eb0 2023-06-22T08:06:04.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.109+0000 7f2b583ff700 1 -- 172.21.15.23:0/2089392336 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5004ea20 con 0x7f2b50069eb0 2023-06-22T08:06:04.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.112+0000 7f2b46ffd700 1 -- 172.21.15.23:0/2089392336 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2b4c007450 con 0x7f2b50069eb0 2023-06-22T08:06:04.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.113+0000 7f2b46ffd700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b3406adb0 0x7f2b3406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:04.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.113+0000 7f2b46ffd700 1 -- 172.21.15.23:0/2089392336 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f2b4c089950 con 0x7f2b50069eb0 2023-06-22T08:06:04.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.114+0000 7f2b5599a700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b3406adb0 0x7f2b3406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:04.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.115+0000 7f2b5599a700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b3406adb0 0x7f2b3406d260 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2b501f0600 tx=0x7f2b400090e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:04.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.119+0000 7f2b46ffd700 1 -- 172.21.15.23:0/2089392336 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2b4c054560 con 0x7f2b50069eb0 2023-06-22T08:06:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.409+0000 7f2b583ff700 1 -- 172.21.15.23:0/2089392336 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b500c1a80 con 0x7f2b50069eb0 2023-06-22T08:06:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.410+0000 7f2b46ffd700 1 -- 172.21.15.23:0/2089392336 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5492 (secure 0 0 0) 0x7f2b4c057b80 con 0x7f2b50069eb0 2023-06-22T08:06:04.414 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:06:04.414 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:02.733247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:resolve","state_seq":25,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:06:04.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.412+0000 7f2b583ff700 1 -- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b3406adb0 msgr2=0x7f2b3406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:04.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.412+0000 7f2b583ff700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b3406adb0 0x7f2b3406d260 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2b501f0600 tx=0x7f2b400090e0 comp rx=0 tx=0).stop 2023-06-22T08:06:04.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.413+0000 7f2b583ff700 1 -- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b50069eb0 msgr2=0x7f2b500c3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:04.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.413+0000 7f2b583ff700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b50069eb0 0x7f2b500c3590 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00c780 tx=0x7f2b4c00ca90 comp rx=0 tx=0).stop 2023-06-22T08:06:04.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.413+0000 7f2b583ff700 1 -- 172.21.15.23:0/2089392336 shutdown_connections 2023-06-22T08:06:04.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.413+0000 7f2b583ff700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b3406adb0 0x7f2b3406d260 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:04.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.413+0000 7f2b583ff700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b5015d9b0 0x7f2b50160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:04.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.413+0000 7f2b583ff700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b50069eb0 0x7f2b500c3590 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:04.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.413+0000 7f2b583ff700 1 --2- 172.21.15.23:0/2089392336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b5006d050 0x7f2b5015fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:04.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.413+0000 7f2b583ff700 1 -- 172.21.15.23:0/2089392336 >> 172.21.15.23:0/2089392336 conn(0x7f2b500ca3f0 msgr2=0x7f2b501513d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.413+0000 7f2b583ff700 1 -- 172.21.15.23:0/2089392336 shutdown_connections 2023-06-22T08:06:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:04.413+0000 7f2b583ff700 1 -- 172.21.15.23:0/2089392336 wait complete. 2023-06-22T08:06:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 48 2023-06-22T08:06:04.435 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:06:06.436 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:06:07.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.012+0000 7fd35cbff700 1 Processor -- start 2023-06-22T08:06:07.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.012+0000 7fd35cbff700 1 -- start start 2023-06-22T08:06:07.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.015+0000 7fd35cbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd358069eb0 0x7fd35806d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:07.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.015+0000 7fd35cbff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd35815d9b0 0x7fd35806a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:07.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.015+0000 7fd35cbff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd3580c5aa0 0x7fd35806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:07.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.015+0000 7fd35cbff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35806a7c0 con 0x7fd3580c5aa0 2023-06-22T08:06:07.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.015+0000 7fd35cbff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3580c4d70 con 0x7fd35815d9b0 2023-06-22T08:06:07.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.015+0000 7fd35cbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3580c5610 con 0x7fd358069eb0 2023-06-22T08:06:07.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.015+0000 7fd35659c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd358069eb0 0x7fd35806d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:07.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.015+0000 7fd35659c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd358069eb0 0x7fd35806d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:41926/0 (socket says 172.21.15.23:41926) 2023-06-22T08:06:07.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.015+0000 7fd355d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd35815d9b0 0x7fd35806a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.015+0000 7fd35659c700 1 -- 172.21.15.23:0/1993530566 learned_addr learned my addr 172.21.15.23:0/1993530566 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.016+0000 7fd356d9d700 1 --2- 172.21.15.23:0/1993530566 >> v2:172.21.15.23:3300/0 conn(0x7fd3580c5aa0 0x7fd35806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:07.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.018+0000 7fd355d9b700 1 -- 172.21.15.23:0/1993530566 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd358069eb0 msgr2=0x7fd35806d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:06:07.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.018+0000 7fd355d9b700 1 --2- 172.21.15.23:0/1993530566 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd358069eb0 0x7fd35806d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:07.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.018+0000 7fd355d9b700 1 -- 172.21.15.23:0/1993530566 >> v2:172.21.15.23:3300/0 conn(0x7fd3580c5aa0 msgr2=0x7fd35806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:06:07.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.018+0000 7fd355d9b700 1 --2- 172.21.15.23:0/1993530566 >> v2:172.21.15.23:3300/0 conn(0x7fd3580c5aa0 0x7fd35806fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:07.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.018+0000 7fd355d9b700 1 -- 172.21.15.23:0/1993530566 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd358071340 con 0x7fd35815d9b0 2023-06-22T08:06:07.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.023+0000 7fd355d9b700 1 --2- 172.21.15.23:0/1993530566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35815d9b0 0x7fd35806a280 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd3580668a0 tx=0x7fd348009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b76ebdd2b128a157 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:07.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.028+0000 7fd35559a700 1 -- 172.21.15.23:0/1993530566 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348011830 con 0x7fd35815d9b0 2023-06-22T08:06:07.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.028+0000 7fd35559a700 1 -- 172.21.15.23:0/1993530566 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348011990 con 0x7fd35815d9b0 2023-06-22T08:06:07.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.028+0000 7fd35559a700 1 -- 172.21.15.23:0/1993530566 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348011d20 con 0x7fd35815d9b0 2023-06-22T08:06:07.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.029+0000 7fd35cbff700 1 -- 172.21.15.23:0/1993530566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35815d9b0 msgr2=0x7fd35806a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:07.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.029+0000 7fd35cbff700 1 --2- 172.21.15.23:0/1993530566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35815d9b0 0x7fd35806a280 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd3580668a0 tx=0x7fd348009b20 comp rx=0 tx=0).stop 2023-06-22T08:06:07.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.031+0000 7fd35cbff700 1 -- 172.21.15.23:0/1993530566 shutdown_connections 2023-06-22T08:06:07.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.031+0000 7fd35cbff700 1 --2- 172.21.15.23:0/1993530566 >> v2:172.21.15.23:3300/0 conn(0x7fd3580c5aa0 0x7fd35806fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:07.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.031+0000 7fd35cbff700 1 --2- 172.21.15.23:0/1993530566 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd358069eb0 0x7fd35806d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:07.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.031+0000 7fd35cbff700 1 --2- 172.21.15.23:0/1993530566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd35815d9b0 0x7fd35806a280 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd3580668a0 tx=0x7fd348009b20 comp rx=0 tx=0).stop 2023-06-22T08:06:07.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.031+0000 7fd35cbff700 1 -- 172.21.15.23:0/1993530566 >> 172.21.15.23:0/1993530566 conn(0x7fd3580ca3f0 msgr2=0x7fd3580ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:07.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.031+0000 7fd35cbff700 1 -- 172.21.15.23:0/1993530566 shutdown_connections 2023-06-22T08:06:07.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.032+0000 7fd35cbff700 1 -- 172.21.15.23:0/1993530566 wait complete. 2023-06-22T08:06:07.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.035+0000 7fd35cbff700 1 Processor -- start 2023-06-22T08:06:07.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.035+0000 7fd35cbff700 1 -- start start 2023-06-22T08:06:07.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.037+0000 7fd35cbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd358069eb0 0x7fd35815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:07.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.037+0000 7fd35cbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3580c5aa0 0x7fd3581520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:07.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.037+0000 7fd35cbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd35815de60 0x7fd3581f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:07.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.037+0000 7fd35cbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd358158110 con 0x7fd35815de60 2023-06-22T08:06:07.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.037+0000 7fd35cbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35815e9d0 con 0x7fd358069eb0 2023-06-22T08:06:07.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.037+0000 7fd35cbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35815a620 con 0x7fd3580c5aa0 2023-06-22T08:06:07.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.040+0000 7fd35659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd358069eb0 0x7fd35815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:07.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.040+0000 7fd35659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd358069eb0 0x7fd35815a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:60662/0 (socket says 172.21.15.23:60662) 2023-06-22T08:06:07.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.040+0000 7fd35659c700 1 -- 172.21.15.23:0/2782192891 learned_addr learned my addr 172.21.15.23:0/2782192891 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:07.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.041+0000 7fd355d9b700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3580c5aa0 0x7fd3581520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:07.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.041+0000 7fd356d9d700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd35815de60 0x7fd3581f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:07.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.042+0000 7fd356d9d700 1 -- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3580c5aa0 msgr2=0x7fd3581520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:07.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.042+0000 7fd356d9d700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3580c5aa0 0x7fd3581520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:07.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.043+0000 7fd356d9d700 1 -- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd358069eb0 msgr2=0x7fd35815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:07.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.043+0000 7fd356d9d700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd358069eb0 0x7fd35815a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:07.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.043+0000 7fd356d9d700 1 -- 172.21.15.23:0/2782192891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480094a0 con 0x7fd35815de60 2023-06-22T08:06:07.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.043+0000 7fd356d9d700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd35815de60 0x7fd3581f2860 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00e810 tx=0x7fd34c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:07.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.049+0000 7fd346ffd700 1 -- 172.21.15.23:0/2782192891 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c00c070 con 0x7fd35815de60 2023-06-22T08:06:07.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.049+0000 7fd346ffd700 1 -- 172.21.15.23:0/2782192891 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34c013610 con 0x7fd35815de60 2023-06-22T08:06:07.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.049+0000 7fd346ffd700 1 -- 172.21.15.23:0/2782192891 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34c0139a0 con 0x7fd35815de60 2023-06-22T08:06:07.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.049+0000 7fd35cbff700 1 -- 172.21.15.23:0/2782192891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35815e410 con 0x7fd35815de60 2023-06-22T08:06:07.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.050+0000 7fd35cbff700 1 -- 172.21.15.23:0/2782192891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd358157ac0 con 0x7fd35815de60 2023-06-22T08:06:07.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.059+0000 7fd346ffd700 1 -- 172.21.15.23:0/2782192891 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd34c013b00 con 0x7fd35815de60 2023-06-22T08:06:07.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.061+0000 7fd346ffd700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd33806adb0 0x7fd33806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:07.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.061+0000 7fd346ffd700 1 -- 172.21.15.23:0/2782192891 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7fd34c0895a0 con 0x7fd35815de60 2023-06-22T08:06:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.065+0000 7fd35659c700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd33806adb0 0x7fd33806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:07.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.065+0000 7fd35cbff700 1 -- 172.21.15.23:0/2782192891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd358157920 con 0x7fd35815de60 2023-06-22T08:06:07.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.074+0000 7fd35659c700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd33806adb0 0x7fd33806d260 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd350009e40 tx=0x7fd35000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:07.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.084+0000 7fd346ffd700 1 -- 172.21.15.23:0/2782192891 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd358157920 con 0x7fd35815de60 2023-06-22T08:06:07.222 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:06:07.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:06:07.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.705+0000 7fd35cbff700 1 -- 172.21.15.23:0/2782192891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd358157920 con 0x7fd35815de60 2023-06-22T08:06:07.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.706+0000 7fd346ffd700 1 -- 172.21.15.23:0/2782192891 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5492 (secure 0 0 0) 0x7fd358157920 con 0x7fd35815de60 2023-06-22T08:06:07.713 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:06:07.713 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:02.733247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:resolve","state_seq":25,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:06:07.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 -- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd33806adb0 msgr2=0x7fd33806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:07.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd33806adb0 0x7fd33806d260 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd350009e40 tx=0x7fd35000b040 comp rx=0 tx=0).stop 2023-06-22T08:06:07.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 -- 172.21.15.23:0/2782192891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd35815de60 msgr2=0x7fd3581f2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:07.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd35815de60 0x7fd3581f2860 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd34c00e810 tx=0x7fd34c00eb20 comp rx=0 tx=0).stop 2023-06-22T08:06:07.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 -- 172.21.15.23:0/2782192891 shutdown_connections 2023-06-22T08:06:07.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd35815de60 0x7fd3581f2860 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:07.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd33806adb0 0x7fd33806d260 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:07.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd358069eb0 0x7fd35815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:07.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 --2- 172.21.15.23:0/2782192891 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd3580c5aa0 0x7fd3581520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:07.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 -- 172.21.15.23:0/2782192891 >> 172.21.15.23:0/2782192891 conn(0x7fd3580ca3f0 msgr2=0x7fd3580c9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:07.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.722+0000 7fd35cbff700 1 -- 172.21.15.23:0/2782192891 shutdown_connections 2023-06-22T08:06:07.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:07.723+0000 7fd35cbff700 1 -- 172.21.15.23:0/2782192891 wait complete. 2023-06-22T08:06:07.746 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 48 2023-06-22T08:06:07.782 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:06:09.784 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:06:09.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.985+0000 7f06c51ff700 1 Processor -- start 2023-06-22T08:06:09.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.985+0000 7f06c51ff700 1 -- start start 2023-06-22T08:06:09.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06c51ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c0152e10 0x7f06c0069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:09.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06c51ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06c00c5740 0x7f06c01593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:09.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06c51ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c0158f60 con 0x7f06c00cb6b0 2023-06-22T08:06:09.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06c51ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c006bd20 con 0x7f06c00c5740 2023-06-22T08:06:09.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06c51ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c00cc450 con 0x7f06c0152e10 2023-06-22T08:06:09.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06bf59e700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06c00c5740 0x7f06c01593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:09.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06bed9d700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f06c00cb6b0 0x7f06c0152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:49620/0 (socket says 172.21.15.23:49620) 2023-06-22T08:06:09.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06be59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c0152e10 0x7f06c0069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:09.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06bed9d700 1 -- 172.21.15.23:0/995794995 learned_addr learned my addr 172.21.15.23:0/995794995 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:06:09.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06bf59e700 1 -- 172.21.15.23:0/995794995 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c0152e10 msgr2=0x7f06c0069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:06:09.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06bf59e700 1 --2- 172.21.15.23:0/995794995 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c0152e10 0x7f06c0069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:09.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06bf59e700 1 -- 172.21.15.23:0/995794995 >> v1:172.21.15.67:6789/0 conn(0x7f06c00cb6b0 legacy=0x7f06c0152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:09.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06bf59e700 1 -- 172.21.15.23:0/995794995 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c0158980 con 0x7f06c00c5740 2023-06-22T08:06:09.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.987+0000 7f06be59c700 1 --2- 172.21.15.23:0/995794995 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c0152e10 0x7f06c0069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:06:09.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.988+0000 7f06bf59e700 1 --2- 172.21.15.23:0/995794995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00c5740 0x7f06c01593f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009d90 tx=0x7f06b800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44c5b6708d839b29 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:09.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.988+0000 7f06bdd9b700 1 -- 172.21.15.23:0/995794995 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b800e070 con 0x7f06c00c5740 2023-06-22T08:06:09.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.988+0000 7f06bdd9b700 1 -- 172.21.15.23:0/995794995 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b800bb20 con 0x7f06c00c5740 2023-06-22T08:06:09.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.988+0000 7f06bdd9b700 1 -- 172.21.15.23:0/995794995 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8012430 con 0x7f06c00c5740 2023-06-22T08:06:09.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.988+0000 7f06c51ff700 1 -- 172.21.15.23:0/995794995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00c5740 msgr2=0x7f06c01593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:09.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.988+0000 7f06c51ff700 1 --2- 172.21.15.23:0/995794995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00c5740 0x7f06c01593f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009d90 tx=0x7f06b800b330 comp rx=0 tx=0).stop 2023-06-22T08:06:09.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.989+0000 7f06c51ff700 1 -- 172.21.15.23:0/995794995 shutdown_connections 2023-06-22T08:06:09.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.989+0000 7f06c51ff700 1 --2- 172.21.15.23:0/995794995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00c5740 0x7f06c01593f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:09.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.989+0000 7f06c51ff700 1 --2- 172.21.15.23:0/995794995 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c0152e10 0x7f06c0069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:09.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.989+0000 7f06c51ff700 1 -- 172.21.15.23:0/995794995 >> 172.21.15.23:0/995794995 conn(0x7f06c0070d80 msgr2=0x7f06c0071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:09.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.989+0000 7f06c51ff700 1 -- 172.21.15.23:0/995794995 shutdown_connections 2023-06-22T08:06:09.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.989+0000 7f06c51ff700 1 -- 172.21.15.23:0/995794995 wait complete. 2023-06-22T08:06:09.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.990+0000 7f06c51ff700 1 Processor -- start 2023-06-22T08:06:10.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.991+0000 7f06c51ff700 1 -- start start 2023-06-22T08:06:10.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.991+0000 7f06c51ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00cb6b0 0x7f06c015a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:10.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06c51ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c0152e10 0x7f06c015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:10.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06c51ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c01599d0 0x7f06c01e1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:10.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06c51ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c0157720 con 0x7f06c0152e10 2023-06-22T08:06:10.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06c51ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c0148ad0 con 0x7f06c00cb6b0 2023-06-22T08:06:10.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06c51ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c015b720 con 0x7f06c01599d0 2023-06-22T08:06:10.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00cb6b0 0x7f06c015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:10.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bf59e700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c01599d0 0x7f06c01e1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:10.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00cb6b0 0x7f06c015a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:60674/0 (socket says 172.21.15.23:60674) 2023-06-22T08:06:10.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bed9d700 1 -- 172.21.15.23:0/1509888605 learned_addr learned my addr 172.21.15.23:0/1509888605 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:10.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06be59c700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c0152e10 0x7f06c015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:10.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bed9d700 1 -- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c01599d0 msgr2=0x7f06c01e1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:10.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bed9d700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c01599d0 0x7f06c01e1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:10.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bed9d700 1 -- 172.21.15.23:0/1509888605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c0152e10 msgr2=0x7f06c015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:10.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bed9d700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c0152e10 0x7f06c015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:10.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bed9d700 1 -- 172.21.15.23:0/1509888605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b40094a0 con 0x7f06c00cb6b0 2023-06-22T08:06:10.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bf59e700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c01599d0 0x7f06c01e1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:06:10.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.992+0000 7f06bed9d700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00cb6b0 0x7f06c015a810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f06b000c520 tx=0x7f06b000c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:10.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.993+0000 7f06a77fe700 1 -- 172.21.15.23:0/1509888605 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b0005d70 con 0x7f06c00cb6b0 2023-06-22T08:06:10.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.993+0000 7f06a77fe700 1 -- 172.21.15.23:0/1509888605 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b00070d0 con 0x7f06c00cb6b0 2023-06-22T08:06:10.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.993+0000 7f06c51ff700 1 -- 172.21.15.23:0/1509888605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b8009a70 con 0x7f06c00cb6b0 2023-06-22T08:06:10.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.993+0000 7f06a77fe700 1 -- 172.21.15.23:0/1509888605 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b0007440 con 0x7f06c00cb6b0 2023-06-22T08:06:10.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.993+0000 7f06c51ff700 1 -- 172.21.15.23:0/1509888605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c01e0a60 con 0x7f06c00cb6b0 2023-06-22T08:06:10.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.994+0000 7f06c51ff700 1 -- 172.21.15.23:0/1509888605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c004ea20 con 0x7f06c00cb6b0 2023-06-22T08:06:10.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.995+0000 7f06a77fe700 1 -- 172.21.15.23:0/1509888605 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f06b00075a0 con 0x7f06c00cb6b0 2023-06-22T08:06:10.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.996+0000 7f06a77fe700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f06ac06ace0 0x7f06ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:10.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.996+0000 7f06a77fe700 1 -- 172.21.15.23:0/1509888605 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f06b0088bc0 con 0x7f06c00cb6b0 2023-06-22T08:06:10.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.996+0000 7f06be59c700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f06ac06ace0 0x7f06ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:10.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:09.996+0000 7f06be59c700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f06ac06ace0 0x7f06ac06d190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f06c00c9b20 tx=0x7f06b4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:10.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.000+0000 7f06a77fe700 1 -- 172.21.15.23:0/1509888605 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f06b0053890 con 0x7f06c00cb6b0 2023-06-22T08:06:10.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.251+0000 7f06c51ff700 1 -- 172.21.15.23:0/1509888605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06c00c6960 con 0x7f06c00cb6b0 2023-06-22T08:06:10.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.252+0000 7f06a77fe700 1 -- 172.21.15.23:0/1509888605 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5494 (secure 0 0 0) 0x7f06b0056eb0 con 0x7f06c00cb6b0 2023-06-22T08:06:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:06:10.255 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:08.753403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:06:10.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 -- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f06ac06ace0 msgr2=0x7f06ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:10.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f06ac06ace0 0x7f06ac06d190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f06c00c9b20 tx=0x7f06b4009210 comp rx=0 tx=0).stop 2023-06-22T08:06:10.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 -- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00cb6b0 msgr2=0x7f06c015a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:10.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00cb6b0 0x7f06c015a810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f06b000c520 tx=0x7f06b000c830 comp rx=0 tx=0).stop 2023-06-22T08:06:10.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 -- 172.21.15.23:0/1509888605 shutdown_connections 2023-06-22T08:06:10.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f06c01599d0 0x7f06c01e1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:10.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f06ac06ace0 0x7f06ac06d190 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:10.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06c00cb6b0 0x7f06c015a810 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:10.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 --2- 172.21.15.23:0/1509888605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06c0152e10 0x7f06c015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:10.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 -- 172.21.15.23:0/1509888605 >> 172.21.15.23:0/1509888605 conn(0x7f06c0070d80 msgr2=0x7f06c006a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:10.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 -- 172.21.15.23:0/1509888605 shutdown_connections 2023-06-22T08:06:10.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:10.254+0000 7f06c51ff700 1 -- 172.21.15.23:0/1509888605 wait complete. 2023-06-22T08:06:10.260 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 49 2023-06-22T08:06:10.276 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:06:12.277 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:06:12.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.484+0000 7f1353dff700 1 Processor -- start 2023-06-22T08:06:12.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.484+0000 7f1353dff700 1 -- start start 2023-06-22T08:06:12.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f1353dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f134c14f850 0x7f134c0c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:12.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f1353dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c0c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:12.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f1353dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c0c51a0 con 0x7f134c0c69e0 2023-06-22T08:06:12.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f1353dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134c0c44e0 con 0x7f134c14f850 2023-06-22T08:06:12.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f1353dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c0c5440 con 0x7f134c071070 2023-06-22T08:06:12.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f135139a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f134c0c69e0 0x7f134c155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55740/0 (socket says 172.21.15.23:55740) 2023-06-22T08:06:12.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f135139a700 1 -- 172.21.15.23:0/1667467517 learned_addr learned my addr 172.21.15.23:0/1667467517 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:06:12.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f1351b9b700 1 --2- 172.21.15.23:0/1667467517 >> v2:172.21.15.23:3300/0 conn(0x7f134c14f850 0x7f134c0c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:12.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f135239c700 1 --2- 172.21.15.23:0/1667467517 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c0c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:12.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f1350b99700 1 -- 172.21.15.23:0/1667467517 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137126901 0 0) 0x7f134c0c51a0 con 0x7f134c0c69e0 2023-06-22T08:06:12.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f135239c700 1 -- 172.21.15.23:0/1667467517 >> v2:172.21.15.23:3300/0 conn(0x7f134c14f850 msgr2=0x7f134c0c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:06:12.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f135239c700 1 --2- 172.21.15.23:0/1667467517 >> v2:172.21.15.23:3300/0 conn(0x7f134c14f850 0x7f134c0c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:12.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.486+0000 7f135239c700 1 -- 172.21.15.23:0/1667467517 >> v1:172.21.15.23:6789/0 conn(0x7f134c0c69e0 legacy=0x7f134c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:12.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.487+0000 7f135239c700 1 -- 172.21.15.23:0/1667467517 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c0c5720 con 0x7f134c071070 2023-06-22T08:06:12.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.487+0000 7f135239c700 1 --2- 172.21.15.23:0/1667467517 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c0c35c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1344009d90 tx=0x7f134400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9513dbc54135fc0d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:12.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.487+0000 7f1350b99700 1 -- 172.21.15.23:0/1667467517 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134400e070 con 0x7f134c071070 2023-06-22T08:06:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.487+0000 7f1350b99700 1 -- 172.21.15.23:0/1667467517 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134400bb20 con 0x7f134c071070 2023-06-22T08:06:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.487+0000 7f1350b99700 1 -- 172.21.15.23:0/1667467517 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1344012430 con 0x7f134c071070 2023-06-22T08:06:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.487+0000 7f1353dff700 1 -- 172.21.15.23:0/1667467517 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 msgr2=0x7f134c0c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.488+0000 7f1353dff700 1 --2- 172.21.15.23:0/1667467517 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c0c35c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1344009d90 tx=0x7f134400b330 comp rx=0 tx=0).stop 2023-06-22T08:06:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.488+0000 7f1353dff700 1 -- 172.21.15.23:0/1667467517 shutdown_connections 2023-06-22T08:06:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.488+0000 7f1353dff700 1 --2- 172.21.15.23:0/1667467517 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c0c35c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:12.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.488+0000 7f1353dff700 1 --2- 172.21.15.23:0/1667467517 >> v2:172.21.15.23:3300/0 conn(0x7f134c14f850 0x7f134c0c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:12.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.488+0000 7f1353dff700 1 -- 172.21.15.23:0/1667467517 >> 172.21.15.23:0/1667467517 conn(0x7f134c0ca3f0 msgr2=0x7f134c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:12.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.488+0000 7f1353dff700 1 -- 172.21.15.23:0/1667467517 shutdown_connections 2023-06-22T08:06:12.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.488+0000 7f1353dff700 1 -- 172.21.15.23:0/1667467517 wait complete. 2023-06-22T08:06:12.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.489+0000 7f1353dff700 1 Processor -- start 2023-06-22T08:06:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.490+0000 7f1353dff700 1 -- start start 2023-06-22T08:06:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.490+0000 7f1353dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c1e9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.490+0000 7f1353dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c0c69e0 0x7f134c14e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.490+0000 7f1353dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c14f850 0x7f134c14ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.490+0000 7f1353dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c14f0a0 con 0x7f134c0c69e0 2023-06-22T08:06:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.490+0000 7f1353dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c06fce0 con 0x7f134c14f850 2023-06-22T08:06:12.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.490+0000 7f1353dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c14e800 con 0x7f134c071070 2023-06-22T08:06:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f135139a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c0c69e0 0x7f134c14e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f1351b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c1e9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f1351b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c1e9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:41986/0 (socket says 172.21.15.23:41986) 2023-06-22T08:06:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f1351b9b700 1 -- 172.21.15.23:0/3253675571 learned_addr learned my addr 172.21.15.23:0/3253675571 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f135239c700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c14f850 0x7f134c14ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f1351b9b700 1 -- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c14f850 msgr2=0x7f134c14ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:12.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f1351b9b700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c14f850 0x7f134c14ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:12.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f1351b9b700 1 -- 172.21.15.23:0/3253675571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c0c69e0 msgr2=0x7f134c14e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:12.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f1351b9b700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c0c69e0 0x7f134c14e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:12.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f1351b9b700 1 -- 172.21.15.23:0/3253675571 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1344009a70 con 0x7f134c071070 2023-06-22T08:06:12.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.491+0000 7f1351b9b700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c1e9cd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f134800e870 tx=0x7f134800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:12.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.492+0000 7f13427fc700 1 -- 172.21.15.23:0/3253675571 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348009590 con 0x7f134c071070 2023-06-22T08:06:12.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.492+0000 7f13427fc700 1 -- 172.21.15.23:0/3253675571 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1348013680 con 0x7f134c071070 2023-06-22T08:06:12.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.492+0000 7f13427fc700 1 -- 172.21.15.23:0/3253675571 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13480139f0 con 0x7f134c071070 2023-06-22T08:06:12.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.492+0000 7f1353dff700 1 -- 172.21.15.23:0/3253675571 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c14eae0 con 0x7f134c071070 2023-06-22T08:06:12.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.492+0000 7f1353dff700 1 -- 172.21.15.23:0/3253675571 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c1ed370 con 0x7f134c071070 2023-06-22T08:06:12.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.493+0000 7f1353dff700 1 -- 172.21.15.23:0/3253675571 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134c0c2950 con 0x7f134c071070 2023-06-22T08:06:12.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.494+0000 7f13427fc700 1 -- 172.21.15.23:0/3253675571 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1348013b50 con 0x7f134c071070 2023-06-22T08:06:12.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.495+0000 7f13427fc700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f133006ace0 0x7f133006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:12.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.495+0000 7f13427fc700 1 -- 172.21.15.23:0/3253675571 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f1348089810 con 0x7f134c071070 2023-06-22T08:06:12.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.495+0000 7f135139a700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f133006ace0 0x7f133006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:12.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.496+0000 7f135139a700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f133006ace0 0x7f133006d190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f134c147430 tx=0x7f133c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:12.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.499+0000 7f13427fc700 1 -- 172.21.15.23:0/3253675571 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1348057e30 con 0x7f134c071070 2023-06-22T08:06:12.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.780+0000 7f1353dff700 1 -- 172.21.15.23:0/3253675571 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f134c06e930 con 0x7f134c071070 2023-06-22T08:06:12.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.781+0000 7f13427fc700 1 -- 172.21.15.23:0/3253675571 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5494 (secure 0 0 0) 0x7f13480579c0 con 0x7f134c071070 2023-06-22T08:06:12.784 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:06:12.784 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:08.753403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:06:12.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 -- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f133006ace0 msgr2=0x7f133006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:12.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f133006ace0 0x7f133006d190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f134c147430 tx=0x7f133c007de0 comp rx=0 tx=0).stop 2023-06-22T08:06:12.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 -- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 msgr2=0x7f134c1e9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:12.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c1e9cd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f134800e870 tx=0x7f134800eb80 comp rx=0 tx=0).stop 2023-06-22T08:06:12.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 -- 172.21.15.23:0/3253675571 shutdown_connections 2023-06-22T08:06:12.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f133006ace0 0x7f133006d190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:12.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f134c14f850 0x7f134c14ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:12.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f134c071070 0x7f134c1e9cd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:12.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 --2- 172.21.15.23:0/3253675571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f134c0c69e0 0x7f134c14e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:12.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 -- 172.21.15.23:0/3253675571 >> 172.21.15.23:0/3253675571 conn(0x7f134c0ca3f0 msgr2=0x7f134c069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:12.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 -- 172.21.15.23:0/3253675571 shutdown_connections 2023-06-22T08:06:12.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:12.783+0000 7f1353dff700 1 -- 172.21.15.23:0/3253675571 wait complete. 2023-06-22T08:06:12.788 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 49 2023-06-22T08:06:12.805 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4272": { "addr": "172.21.15.23:6835/3894174037", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 3894174037, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 3894174037, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4272, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 12 }, "gid_4279": { "addr": "172.21.15.67:6836/1962314516", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 1962314516, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 1962314516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4285": { "addr": "172.21.15.67:6839/567663545", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 567663545, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 567663545, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5060": { "addr": "172.21.15.67:6837/1551460358", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1551460358, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1551460358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5060, "incarnation": 46, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:reconnect", "state_seq": 27 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:06:08.753403+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4279, "mds_2": 4272, "mds_3": 5060, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:06:14.807 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:06:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.013+0000 7fa338fff700 1 Processor -- start 2023-06-22T08:06:15.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.014+0000 7fa338fff700 1 -- start start 2023-06-22T08:06:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.015+0000 7fa338fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3340cb260 0x7fa3340c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.015+0000 7fa338fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3340c9200 0x7fa3340cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.015+0000 7fa338fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3340cbd60 con 0x7fa3340b0fc0 2023-06-22T08:06:15.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.015+0000 7fa338fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3340cbb70 con 0x7fa3340cb260 2023-06-22T08:06:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.015+0000 7fa338fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa334158980 con 0x7fa3340c9200 2023-06-22T08:06:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.015+0000 7fa33259c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3340cb260 0x7fa3340c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.015+0000 7fa332d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa3340b0fc0 0x7fa33415be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55750/0 (socket says 172.21.15.23:55750) 2023-06-22T08:06:15.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.015+0000 7fa332d9d700 1 -- 172.21.15.23:0/3903779284 learned_addr learned my addr 172.21.15.23:0/3903779284 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:06:15.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.015+0000 7fa32bfff700 1 --2- 172.21.15.23:0/3903779284 >> v2:172.21.15.67:3300/0 conn(0x7fa3340c9200 0x7fa3340cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:15.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.016+0000 7fa33259c700 1 -- 172.21.15.23:0/3903779284 >> v2:172.21.15.67:3300/0 conn(0x7fa3340c9200 msgr2=0x7fa3340cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:06:15.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.016+0000 7fa33259c700 1 --2- 172.21.15.23:0/3903779284 >> v2:172.21.15.67:3300/0 conn(0x7fa3340c9200 0x7fa3340cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:15.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.016+0000 7fa33259c700 1 -- 172.21.15.23:0/3903779284 >> v1:172.21.15.23:6789/0 conn(0x7fa3340b0fc0 legacy=0x7fa33415be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:15.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.016+0000 7fa33259c700 1 -- 172.21.15.23:0/3903779284 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa334159550 con 0x7fa3340cb260 2023-06-22T08:06:15.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.016+0000 7fa331d9b700 1 -- 172.21.15.23:0/3903779284 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463385024 0 0) 0x7fa3340cbd60 con 0x7fa3340b0fc0 2023-06-22T08:06:15.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.016+0000 7fa33259c700 1 --2- 172.21.15.23:0/3903779284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 0x7fa3340c4a50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa33408c760 tx=0x7fa324009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e617272fdbc26061 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:15.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.016+0000 7fa331d9b700 1 -- 172.21.15.23:0/3903779284 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa324011830 con 0x7fa3340cb260 2023-06-22T08:06:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.016+0000 7fa331d9b700 1 -- 172.21.15.23:0/3903779284 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa324011990 con 0x7fa3340cb260 2023-06-22T08:06:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.016+0000 7fa331d9b700 1 -- 172.21.15.23:0/3903779284 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa324011d20 con 0x7fa3340cb260 2023-06-22T08:06:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.017+0000 7fa338fff700 1 -- 172.21.15.23:0/3903779284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 msgr2=0x7fa3340c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.017+0000 7fa338fff700 1 --2- 172.21.15.23:0/3903779284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 0x7fa3340c4a50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa33408c760 tx=0x7fa324009b20 comp rx=0 tx=0).stop 2023-06-22T08:06:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.017+0000 7fa338fff700 1 -- 172.21.15.23:0/3903779284 shutdown_connections 2023-06-22T08:06:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.017+0000 7fa338fff700 1 --2- 172.21.15.23:0/3903779284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 0x7fa3340c4a50 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.017+0000 7fa338fff700 1 --2- 172.21.15.23:0/3903779284 >> v2:172.21.15.67:3300/0 conn(0x7fa3340c9200 0x7fa3340cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:15.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.017+0000 7fa338fff700 1 -- 172.21.15.23:0/3903779284 >> 172.21.15.23:0/3903779284 conn(0x7fa334065b90 msgr2=0x7fa334065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.017+0000 7fa338fff700 1 -- 172.21.15.23:0/3903779284 shutdown_connections 2023-06-22T08:06:15.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.017+0000 7fa338fff700 1 -- 172.21.15.23:0/3903779284 wait complete. 2023-06-22T08:06:15.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.019+0000 7fa338fff700 1 Processor -- start 2023-06-22T08:06:15.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.019+0000 7fa338fff700 1 -- start start 2023-06-22T08:06:15.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa338fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3340b0fc0 0x7fa334151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:15.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa338fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3340c9200 0x7fa33414e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:15.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa338fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 0x7fa33414ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:15.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa338fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa334146e60 con 0x7fa3340cb260 2023-06-22T08:06:15.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa338fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa334158660 con 0x7fa3340c9200 2023-06-22T08:06:15.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa338fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa334157b00 con 0x7fa3340b0fc0 2023-06-22T08:06:15.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa332d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 0x7fa33414ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:15.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa332d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 0x7fa33414ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47718/0 (socket says 172.21.15.23:47718) 2023-06-22T08:06:15.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa332d9d700 1 -- 172.21.15.23:0/3743270945 learned_addr learned my addr 172.21.15.23:0/3743270945 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:15.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa32bfff700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3340c9200 0x7fa33414e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:15.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.020+0000 7fa33259c700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3340b0fc0 0x7fa334151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:15.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa332d9d700 1 -- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3340b0fc0 msgr2=0x7fa334151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:15.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa332d9d700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3340b0fc0 0x7fa334151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:15.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa332d9d700 1 -- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3340c9200 msgr2=0x7fa33414e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:15.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa332d9d700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3340c9200 0x7fa33414e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:15.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa332d9d700 1 -- 172.21.15.23:0/3743270945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3240094a0 con 0x7fa3340cb260 2023-06-22T08:06:15.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa32bfff700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3340c9200 0x7fa33414e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:06:15.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa332d9d700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 0x7fa33414ed70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa32c002230 tx=0x7fa32c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:15.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa32affd700 1 -- 172.21.15.23:0/3743270945 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c00aac0 con 0x7fa3340cb260 2023-06-22T08:06:15.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa32affd700 1 -- 172.21.15.23:0/3743270945 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c00ac40 con 0x7fa3340cb260 2023-06-22T08:06:15.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa32affd700 1 -- 172.21.15.23:0/3743270945 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c0070d0 con 0x7fa3340cb260 2023-06-22T08:06:15.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa338fff700 1 -- 172.21.15.23:0/3743270945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa334157dc0 con 0x7fa3340cb260 2023-06-22T08:06:15.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.021+0000 7fa338fff700 1 -- 172.21.15.23:0/3743270945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa334158380 con 0x7fa3340cb260 2023-06-22T08:06:15.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.022+0000 7fa338fff700 1 -- 172.21.15.23:0/3743270945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33404ea20 con 0x7fa3340cb260 2023-06-22T08:06:15.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.023+0000 7fa32affd700 1 -- 172.21.15.23:0/3743270945 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa32c007250 con 0x7fa3340cb260 2023-06-22T08:06:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.024+0000 7fa32affd700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa31406ace0 0x7fa31406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.024+0000 7fa32affd700 1 -- 172.21.15.23:0/3743270945 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7fa32c089220 con 0x7fa3340cb260 2023-06-22T08:06:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.024+0000 7fa33259c700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa31406ace0 0x7fa31406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:15.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.025+0000 7fa33259c700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa31406ace0 0x7fa31406d190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa3240090b0 tx=0x7fa32401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:15.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.029+0000 7fa32affd700 1 -- 172.21.15.23:0/3743270945 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa32c053f70 con 0x7fa3340cb260 2023-06-22T08:06:15.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.285+0000 7fa338fff700 1 -- 172.21.15.23:0/3743270945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3340c3c10 con 0x7fa3340cb260 2023-06-22T08:06:15.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.285+0000 7fa32affd700 1 -- 172.21.15.23:0/3743270945 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5491 (secure 0 0 0) 0x7fa32c057590 con 0x7fa3340cb260 2023-06-22T08:06:15.289 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:06:15.290 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:13.771134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:06:15.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 -- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa31406ace0 msgr2=0x7fa31406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:15.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa31406ace0 0x7fa31406d190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa3240090b0 tx=0x7fa32401c040 comp rx=0 tx=0).stop 2023-06-22T08:06:15.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 -- 172.21.15.23:0/3743270945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 msgr2=0x7fa33414ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:15.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 0x7fa33414ed70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa32c002230 tx=0x7fa32c00cf50 comp rx=0 tx=0).stop 2023-06-22T08:06:15.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 -- 172.21.15.23:0/3743270945 shutdown_connections 2023-06-22T08:06:15.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3340cb260 0x7fa33414ed70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:15.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3340b0fc0 0x7fa334151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:15.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa31406ace0 0x7fa31406d190 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:15.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 --2- 172.21.15.23:0/3743270945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3340c9200 0x7fa33414e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:15.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 -- 172.21.15.23:0/3743270945 >> 172.21.15.23:0/3743270945 conn(0x7fa334065b90 msgr2=0x7fa3340b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:15.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 -- 172.21.15.23:0/3743270945 shutdown_connections 2023-06-22T08:06:15.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:15.288+0000 7fa338fff700 1 -- 172.21.15.23:0/3743270945 wait complete. 2023-06-22T08:06:15.294 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 50 2023-06-22T08:06:15.310 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:06:17.311 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:06:17.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.519+0000 7fd0df5ff700 1 Processor -- start 2023-06-22T08:06:17.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.519+0000 7fd0df5ff700 1 -- start start 2023-06-22T08:06:17.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.520+0000 7fd0df5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0d80cb6b0 0x7fd0d8069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.520+0000 7fd0df5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d8152e10 0x7fd0d81593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.520+0000 7fd0df5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d8158f60 con 0x7fd0d80c5740 2023-06-22T08:06:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.520+0000 7fd0df5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d806bd20 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.520+0000 7fd0df5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80cc450 con 0x7fd0d8152e10 2023-06-22T08:06:17.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0dcb9a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd0d80c5740 0x7fd0d8152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:59768/0 (socket says 172.21.15.23:59768) 2023-06-22T08:06:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0ddb9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d8152e10 0x7fd0d81593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0dcb9a700 1 -- 172.21.15.23:0/1422315254 learned_addr learned my addr 172.21.15.23:0/1422315254 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:06:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0dd39b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0d80cb6b0 0x7fd0d8069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0cffff700 1 -- 172.21.15.23:0/1422315254 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332250368 0 0) 0x7fd0d8158f60 con 0x7fd0d80c5740 2023-06-22T08:06:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0cffff700 1 -- 172.21.15.23:0/1422315254 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c40035b0 con 0x7fd0d80c5740 2023-06-22T08:06:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0ddb9c700 1 -- 172.21.15.23:0/1422315254 >> v2:172.21.15.67:3300/0 conn(0x7fd0d80cb6b0 msgr2=0x7fd0d8069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:06:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0ddb9c700 1 --2- 172.21.15.23:0/1422315254 >> v2:172.21.15.67:3300/0 conn(0x7fd0d80cb6b0 0x7fd0d8069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0ddb9c700 1 -- 172.21.15.23:0/1422315254 >> v1:172.21.15.67:6789/0 conn(0x7fd0d80c5740 legacy=0x7fd0d8152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0ddb9c700 1 -- 172.21.15.23:0/1422315254 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d8158980 con 0x7fd0d8152e10 2023-06-22T08:06:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0dd39b700 1 --2- 172.21.15.23:0/1422315254 >> v2:172.21.15.67:3300/0 conn(0x7fd0d80cb6b0 0x7fd0d8069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:06:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.521+0000 7fd0ddb9c700 1 --2- 172.21.15.23:0/1422315254 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d8152e10 0x7fd0d81593f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009d90 tx=0x7fd0d000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35aab9a6e05ad99f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.522+0000 7fd0cffff700 1 -- 172.21.15.23:0/1422315254 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d000e070 con 0x7fd0d8152e10 2023-06-22T08:06:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.522+0000 7fd0cffff700 1 -- 172.21.15.23:0/1422315254 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d000bb20 con 0x7fd0d8152e10 2023-06-22T08:06:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.522+0000 7fd0cffff700 1 -- 172.21.15.23:0/1422315254 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0012430 con 0x7fd0d8152e10 2023-06-22T08:06:17.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.522+0000 7fd0df5ff700 1 -- 172.21.15.23:0/1422315254 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d8152e10 msgr2=0x7fd0d81593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:17.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.522+0000 7fd0df5ff700 1 --2- 172.21.15.23:0/1422315254 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d8152e10 0x7fd0d81593f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009d90 tx=0x7fd0d000b330 comp rx=0 tx=0).stop 2023-06-22T08:06:17.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.522+0000 7fd0df5ff700 1 -- 172.21.15.23:0/1422315254 shutdown_connections 2023-06-22T08:06:17.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.522+0000 7fd0df5ff700 1 --2- 172.21.15.23:0/1422315254 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d8152e10 0x7fd0d81593f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.522+0000 7fd0df5ff700 1 --2- 172.21.15.23:0/1422315254 >> v2:172.21.15.67:3300/0 conn(0x7fd0d80cb6b0 0x7fd0d8069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.522+0000 7fd0df5ff700 1 -- 172.21.15.23:0/1422315254 >> 172.21.15.23:0/1422315254 conn(0x7fd0d8070d80 msgr2=0x7fd0d8071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:17.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.523+0000 7fd0df5ff700 1 -- 172.21.15.23:0/1422315254 shutdown_connections 2023-06-22T08:06:17.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.523+0000 7fd0df5ff700 1 -- 172.21.15.23:0/1422315254 wait complete. 2023-06-22T08:06:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.525+0000 7fd0df5ff700 1 Processor -- start 2023-06-22T08:06:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.525+0000 7fd0df5ff700 1 -- start start 2023-06-22T08:06:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.526+0000 7fd0df5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0d80cb6b0 0x7fd0d80c39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.526+0000 7fd0df5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d8152e10 0x7fd0d81501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.526+0000 7fd0df5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d80c4bd0 0x7fd0d815a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.526+0000 7fd0df5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80c4070 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.526+0000 7fd0df5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d81564d0 con 0x7fd0d8152e10 2023-06-22T08:06:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.526+0000 7fd0df5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d8156ca0 con 0x7fd0d80c4bd0 2023-06-22T08:06:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0dd39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0d80cb6b0 0x7fd0d80c39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0dd39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0d80cb6b0 0x7fd0d80c39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53440/0 (socket says 172.21.15.23:53440) 2023-06-22T08:06:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0dd39b700 1 -- 172.21.15.23:0/4163350897 learned_addr learned my addr 172.21.15.23:0/4163350897 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0ddb9c700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d80c4bd0 0x7fd0d815a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0dcb9a700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d8152e10 0x7fd0d81501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0dd39b700 1 -- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d80c4bd0 msgr2=0x7fd0d815a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0dd39b700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d80c4bd0 0x7fd0d815a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0dd39b700 1 -- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d8152e10 msgr2=0x7fd0d81501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0dd39b700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d8152e10 0x7fd0d81501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0dd39b700 1 -- 172.21.15.23:0/4163350897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40094a0 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.527+0000 7fd0dd39b700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0d80cb6b0 0x7fd0d80c39b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400ebe0 tx=0x7fd0d400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.528+0000 7fd0cdffb700 1 -- 172.21.15.23:0/4163350897 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400c850 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.528+0000 7fd0cdffb700 1 -- 172.21.15.23:0/4163350897 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d400c9b0 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.528+0000 7fd0cdffb700 1 -- 172.21.15.23:0/4163350897 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400cd40 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.528+0000 7fd0df5ff700 1 -- 172.21.15.23:0/4163350897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d0009a70 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.528+0000 7fd0df5ff700 1 -- 172.21.15.23:0/4163350897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d81569c0 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.529+0000 7fd0df5ff700 1 -- 172.21.15.23:0/4163350897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d804ea20 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.530+0000 7fd0cdffb700 1 -- 172.21.15.23:0/4163350897 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd0d4013470 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.530+0000 7fd0cdffb700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd0bc06ad10 0x7fd0bc06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.531+0000 7fd0cdffb700 1 -- 172.21.15.23:0/4163350897 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7fd0d4089a30 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.531+0000 7fd0dcb9a700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd0bc06ad10 0x7fd0bc06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.531+0000 7fd0dcb9a700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd0bc06ad10 0x7fd0bc06d1c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80c9fd0 tx=0x7fd0c8007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:17.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.535+0000 7fd0cdffb700 1 -- 172.21.15.23:0/4163350897 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd0d40545c0 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.789+0000 7fd0df5ff700 1 -- 172.21.15.23:0/4163350897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0d80c27f0 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.790+0000 7fd0cdffb700 1 -- 172.21.15.23:0/4163350897 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5491 (secure 0 0 0) 0x7fd0d4017090 con 0x7fd0d80cb6b0 2023-06-22T08:06:17.794 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:06:17.794 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:13.771134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:06:17.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.792+0000 7fd0df5ff700 1 -- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd0bc06ad10 msgr2=0x7fd0bc06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:17.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.792+0000 7fd0df5ff700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd0bc06ad10 0x7fd0bc06d1c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80c9fd0 tx=0x7fd0c8007dc0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.792+0000 7fd0df5ff700 1 -- 172.21.15.23:0/4163350897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0d80cb6b0 msgr2=0x7fd0d80c39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:17.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.792+0000 7fd0df5ff700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0d80cb6b0 0x7fd0d80c39b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400ebe0 tx=0x7fd0d400eef0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.792+0000 7fd0df5ff700 1 -- 172.21.15.23:0/4163350897 shutdown_connections 2023-06-22T08:06:17.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.792+0000 7fd0df5ff700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd0d80c4bd0 0x7fd0d815a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.792+0000 7fd0df5ff700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0d80cb6b0 0x7fd0d80c39b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.792+0000 7fd0df5ff700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd0bc06ad10 0x7fd0bc06d1c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.792+0000 7fd0df5ff700 1 --2- 172.21.15.23:0/4163350897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0d8152e10 0x7fd0d81501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:17.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.792+0000 7fd0df5ff700 1 -- 172.21.15.23:0/4163350897 >> 172.21.15.23:0/4163350897 conn(0x7fd0d8070d80 msgr2=0x7fd0d81516b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:17.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.793+0000 7fd0df5ff700 1 -- 172.21.15.23:0/4163350897 shutdown_connections 2023-06-22T08:06:17.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:17.793+0000 7fd0df5ff700 1 -- 172.21.15.23:0/4163350897 wait complete. 2023-06-22T08:06:17.800 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 50 2023-06-22T08:06:17.815 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:06:19.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:06:20.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.017+0000 7f71589ff700 1 Processor -- start 2023-06-22T08:06:20.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.017+0000 7f71589ff700 1 -- start start 2023-06-22T08:06:20.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.018+0000 7f71589ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 0x7f71540b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.018+0000 7f71589ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f71540c90d0 0x7f715406b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.018+0000 7f71589ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7154158b00 con 0x7f715406bc20 2023-06-22T08:06:20.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.018+0000 7f71589ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7154152d30 con 0x7f71540c90d0 2023-06-22T08:06:20.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.018+0000 7f71589ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7154158de0 con 0x7f71540cb610 2023-06-22T08:06:20.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.019+0000 7f715259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 0x7f71540b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.019+0000 7f7151d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f71540c90d0 0x7f715406b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.019+0000 7f7152d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f715406bc20 0x7f715415c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:35272/0 (socket says 172.21.15.23:35272) 2023-06-22T08:06:20.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.019+0000 7f7152d9d700 1 -- 172.21.15.23:0/4089165223 learned_addr learned my addr 172.21.15.23:0/4089165223 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:06:20.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.019+0000 7f715259c700 1 -- 172.21.15.23:0/4089165223 >> v2:172.21.15.23:3300/0 conn(0x7f71540c90d0 msgr2=0x7f715406b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:06:20.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.019+0000 7f715259c700 1 --2- 172.21.15.23:0/4089165223 >> v2:172.21.15.23:3300/0 conn(0x7f71540c90d0 0x7f715406b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.019+0000 7f715259c700 1 -- 172.21.15.23:0/4089165223 >> v1:172.21.15.23:6789/0 conn(0x7f715406bc20 legacy=0x7f715415c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.019+0000 7f715259c700 1 -- 172.21.15.23:0/4089165223 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71541590c0 con 0x7f71540cb610 2023-06-22T08:06:20.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.019+0000 7f715259c700 1 --2- 172.21.15.23:0/4089165223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 0x7f71540b3530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7144009840 tx=0x7f7144009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc9434ef59f1b0f3 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f715159a700 1 -- 172.21.15.23:0/4089165223 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144011830 con 0x7f71540cb610 2023-06-22T08:06:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f715159a700 1 -- 172.21.15.23:0/4089165223 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7144011990 con 0x7f71540cb610 2023-06-22T08:06:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f715159a700 1 -- 172.21.15.23:0/4089165223 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144011d20 con 0x7f71540cb610 2023-06-22T08:06:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f71589ff700 1 -- 172.21.15.23:0/4089165223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 msgr2=0x7f71540b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f71589ff700 1 --2- 172.21.15.23:0/4089165223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 0x7f71540b3530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7144009840 tx=0x7f7144009b50 comp rx=0 tx=0).stop 2023-06-22T08:06:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f71589ff700 1 -- 172.21.15.23:0/4089165223 shutdown_connections 2023-06-22T08:06:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f71589ff700 1 --2- 172.21.15.23:0/4089165223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 0x7f71540b3530 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f71589ff700 1 --2- 172.21.15.23:0/4089165223 >> v2:172.21.15.23:3300/0 conn(0x7f71540c90d0 0x7f715406b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f71589ff700 1 -- 172.21.15.23:0/4089165223 >> 172.21.15.23:0/4089165223 conn(0x7f71540715e0 msgr2=0x7f71540719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f71589ff700 1 -- 172.21.15.23:0/4089165223 shutdown_connections 2023-06-22T08:06:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.020+0000 7f71589ff700 1 -- 172.21.15.23:0/4089165223 wait complete. 2023-06-22T08:06:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.022+0000 7f71589ff700 1 Processor -- start 2023-06-22T08:06:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.022+0000 7f71589ff700 1 -- start start 2023-06-22T08:06:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f71589ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f715406bc20 0x7f7154150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f71589ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71540c90d0 0x7f71540cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f71589ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 0x7f71540ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f71589ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715414e2b0 con 0x7f71540c90d0 2023-06-22T08:06:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f71589ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154156a60 con 0x7f715406bc20 2023-06-22T08:06:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f71589ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715414ca10 con 0x7f71540cb610 2023-06-22T08:06:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f715259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f715406bc20 0x7f7154150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f7152d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 0x7f71540ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f715259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f715406bc20 0x7f7154150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50920/0 (socket says 172.21.15.23:50920) 2023-06-22T08:06:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f715259c700 1 -- 172.21.15.23:0/3841412996 learned_addr learned my addr 172.21.15.23:0/3841412996 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.023+0000 7f7151d9b700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71540c90d0 0x7f71540cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.024+0000 7f715259c700 1 -- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 msgr2=0x7f71540ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.024+0000 7f715259c700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 0x7f71540ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.024+0000 7f715259c700 1 -- 172.21.15.23:0/3841412996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71540c90d0 msgr2=0x7f71540cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.024+0000 7f715259c700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71540c90d0 0x7f71540cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.024+0000 7f715259c700 1 -- 172.21.15.23:0/3841412996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71440094a0 con 0x7f715406bc20 2023-06-22T08:06:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.024+0000 7f715259c700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f715406bc20 0x7f7154150c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7144009fd0 tx=0x7f7144004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.024+0000 7f7142ffd700 1 -- 172.21.15.23:0/3841412996 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71440036a0 con 0x7f715406bc20 2023-06-22T08:06:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.024+0000 7f7142ffd700 1 -- 172.21.15.23:0/3841412996 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714401e070 con 0x7f715406bc20 2023-06-22T08:06:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.024+0000 7f7142ffd700 1 -- 172.21.15.23:0/3841412996 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144003a40 con 0x7f715406bc20 2023-06-22T08:06:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.024+0000 7f71589ff700 1 -- 172.21.15.23:0/3841412996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715414ccd0 con 0x7f715406bc20 2023-06-22T08:06:20.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.025+0000 7f71589ff700 1 -- 172.21.15.23:0/3841412996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715414df40 con 0x7f715406bc20 2023-06-22T08:06:20.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.026+0000 7f71589ff700 1 -- 172.21.15.23:0/3841412996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71540cc4f0 con 0x7f715406bc20 2023-06-22T08:06:20.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.027+0000 7f7142ffd700 1 -- 172.21.15.23:0/3841412996 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7144003ba0 con 0x7f715406bc20 2023-06-22T08:06:20.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.028+0000 7f7142ffd700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f713406ad10 0x7f713406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.028+0000 7f7142ffd700 1 -- 172.21.15.23:0/3841412996 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f714408a790 con 0x7f715406bc20 2023-06-22T08:06:20.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.028+0000 7f7151d9b700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f713406ad10 0x7f713406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.029+0000 7f7151d9b700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f713406ad10 0x7f713406d1c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f71540ca740 tx=0x7f7148009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:20.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.034+0000 7f7142ffd700 1 -- 172.21.15.23:0/3841412996 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7144055440 con 0x7f715406bc20 2023-06-22T08:06:20.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.484+0000 7f71589ff700 1 -- 172.21.15.23:0/3841412996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f715414e420 con 0x7f715406bc20 2023-06-22T08:06:20.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.485+0000 7f7142ffd700 1 -- 172.21.15.23:0/3841412996 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5491 (secure 0 0 0) 0x7f7144020090 con 0x7f715406bc20 2023-06-22T08:06:20.489 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:06:20.489 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:19.794873+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:06:20.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 -- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f713406ad10 msgr2=0x7f713406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f713406ad10 0x7f713406d1c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f71540ca740 tx=0x7f7148009210 comp rx=0 tx=0).stop 2023-06-22T08:06:20.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 -- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f715406bc20 msgr2=0x7f7154150c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f715406bc20 0x7f7154150c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7144009fd0 tx=0x7f7144004750 comp rx=0 tx=0).stop 2023-06-22T08:06:20.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 -- 172.21.15.23:0/3841412996 shutdown_connections 2023-06-22T08:06:20.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71540cb610 0x7f71540ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f713406ad10 0x7f713406d1c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f715406bc20 0x7f7154150c40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 --2- 172.21.15.23:0/3841412996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71540c90d0 0x7f71540cc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 -- 172.21.15.23:0/3841412996 >> 172.21.15.23:0/3841412996 conn(0x7f71540715e0 msgr2=0x7f715406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:20.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 -- 172.21.15.23:0/3841412996 shutdown_connections 2023-06-22T08:06:20.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.488+0000 7f71589ff700 1 -- 172.21.15.23:0/3841412996 wait complete. 2023-06-22T08:06:20.493 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 51 2023-06-22T08:06:20.510 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=3, replacing gid=4269 2023-06-22T08:06:20.510 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2023-06-22T08:06:20.510 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-06-22T08:06:20.511 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-22T08:06:20.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-22T08:06:20.554 INFO:tasks.ceph.mds.c:Started 2023-06-22T08:06:20.555 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:06:20.616 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:06:20.612+0000 7fa5334b2700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-06-22T08:06:20.706 INFO:tasks.ceph.mds.c.smithi023.stdout:starting mds.c at 2023-06-22T08:06:20.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.769+0000 7f60d0dff700 1 Processor -- start 2023-06-22T08:06:20.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.770+0000 7f60d0dff700 1 -- start start 2023-06-22T08:06:20.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.771+0000 7f60d0dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f60cc071850 0x7f60cc0c9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.771+0000 7f60d0dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60cc0c9540 con 0x7f60cc15cb00 2023-06-22T08:06:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.771+0000 7f60d0dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60cc158c10 con 0x7f60cc071c20 2023-06-22T08:06:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.771+0000 7f60d0dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60cc158ef0 con 0x7f60cc071850 2023-06-22T08:06:20.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.771+0000 7f60c9d9b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f60cc071c20 0x7f60cc15c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:59790/0 (socket says 172.21.15.23:59790) 2023-06-22T08:06:20.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.771+0000 7f60c9d9b700 1 -- 172.21.15.23:0/2482821701 learned_addr learned my addr 172.21.15.23:0/2482821701 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:06:20.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.771+0000 7f60ca59c700 1 --2- 172.21.15.23:0/2482821701 >> v2:172.21.15.23:3300/0 conn(0x7f60cc071850 0x7f60cc0c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.771+0000 7f60c959a700 1 -- 172.21.15.23:0/2482821701 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667893293 0 0) 0x7f60cc158c10 con 0x7f60cc071c20 2023-06-22T08:06:20.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.772+0000 7f60c959a700 1 -- 172.21.15.23:0/2482821701 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60b40035b0 con 0x7f60cc071c20 2023-06-22T08:06:20.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.772+0000 7f60c959a700 1 -- 172.21.15.23:0/2482821701 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032963467 0 0) 0x7f60cc0c9540 con 0x7f60cc15cb00 2023-06-22T08:06:20.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.772+0000 7f60c959a700 1 -- 172.21.15.23:0/2482821701 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60cc158c10 con 0x7f60cc15cb00 2023-06-22T08:06:20.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.772+0000 7f60ca59c700 1 -- 172.21.15.23:0/2482821701 >> v1:172.21.15.67:6789/0 conn(0x7f60cc071c20 legacy=0x7f60cc15c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.772+0000 7f60ca59c700 1 -- 172.21.15.23:0/2482821701 >> v1:172.21.15.23:6789/0 conn(0x7f60cc15cb00 legacy=0x7f60cc152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.772+0000 7f60ca59c700 1 -- 172.21.15.23:0/2482821701 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60cc1591d0 con 0x7f60cc071850 2023-06-22T08:06:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.772+0000 7f60ca59c700 1 --2- 172.21.15.23:0/2482821701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60cc071850 0x7f60cc0c9830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009840 tx=0x7f60bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66715d729f984bec server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.772+0000 7f60c959a700 1 -- 172.21.15.23:0/2482821701 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc011830 con 0x7f60cc071850 2023-06-22T08:06:20.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.772+0000 7f60c959a700 1 -- 172.21.15.23:0/2482821701 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60bc011990 con 0x7f60cc071850 2023-06-22T08:06:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.772+0000 7f60c959a700 1 -- 172.21.15.23:0/2482821701 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc011d20 con 0x7f60cc071850 2023-06-22T08:06:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.773+0000 7f60d0dff700 1 -- 172.21.15.23:0/2482821701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60cc071850 msgr2=0x7f60cc0c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.773+0000 7f60d0dff700 1 --2- 172.21.15.23:0/2482821701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60cc071850 0x7f60cc0c9830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009840 tx=0x7f60bc009b50 comp rx=0 tx=0).stop 2023-06-22T08:06:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.773+0000 7f60d0dff700 1 -- 172.21.15.23:0/2482821701 shutdown_connections 2023-06-22T08:06:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.773+0000 7f60d0dff700 1 --2- 172.21.15.23:0/2482821701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60cc071850 0x7f60cc0c9830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.773+0000 7f60d0dff700 1 -- 172.21.15.23:0/2482821701 >> 172.21.15.23:0/2482821701 conn(0x7f60cc06b890 msgr2=0x7f60cc06bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:20.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.774+0000 7f60d0dff700 1 -- 172.21.15.23:0/2482821701 shutdown_connections 2023-06-22T08:06:20.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.774+0000 7f60d0dff700 1 -- 172.21.15.23:0/2482821701 wait complete. 2023-06-22T08:06:20.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.775+0000 7f60d0dff700 1 Processor -- start 2023-06-22T08:06:20.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.776+0000 7f60d0dff700 1 -- start start 2023-06-22T08:06:20.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60d0dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60cc071850 0x7f60cc1ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60d0dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f60cc071c20 0x7f60cc14c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60d0dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60cc15cb00 0x7f60cc0b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60d0dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc1ef6f0 con 0x7f60cc15cb00 2023-06-22T08:06:20.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60d0dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc1e46e0 con 0x7f60cc071850 2023-06-22T08:06:20.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60d0dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc1eee50 con 0x7f60cc071c20 2023-06-22T08:06:20.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60ca59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60cc071850 0x7f60cc1ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60ca59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60cc071850 0x7f60cc1ee7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50936/0 (socket says 172.21.15.23:50936) 2023-06-22T08:06:20.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60ca59c700 1 -- 172.21.15.23:0/1676910347 learned_addr learned my addr 172.21.15.23:0/1676910347 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:06:20.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60c9d9b700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f60cc071c20 0x7f60cc14c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.777+0000 7f60cad9d700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60cc15cb00 0x7f60cc0b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60ca59c700 1 -- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f60cc071c20 msgr2=0x7f60cc14c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60ca59c700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f60cc071c20 0x7f60cc14c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60ca59c700 1 -- 172.21.15.23:0/1676910347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60cc15cb00 msgr2=0x7f60cc0b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:20.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60ca59c700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60cc15cb00 0x7f60cc0b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:20.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60ca59c700 1 -- 172.21.15.23:0/1676910347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc0094a0 con 0x7f60cc071850 2023-06-22T08:06:20.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60ca59c700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60cc071850 0x7f60cc1ee7a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0090b0 tx=0x7f60bc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:20.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60baffd700 1 -- 172.21.15.23:0/1676910347 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc01a760 con 0x7f60cc071850 2023-06-22T08:06:20.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60baffd700 1 -- 172.21.15.23:0/1676910347 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60bc01a8c0 con 0x7f60cc071850 2023-06-22T08:06:20.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60baffd700 1 -- 172.21.15.23:0/1676910347 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60bc01ac30 con 0x7f60cc071850 2023-06-22T08:06:20.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60d0dff700 1 -- 172.21.15.23:0/1676910347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60cc1ef130 con 0x7f60cc071850 2023-06-22T08:06:20.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.778+0000 7f60d0dff700 1 -- 172.21.15.23:0/1676910347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60cc0b3170 con 0x7f60cc071850 2023-06-22T08:06:20.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.779+0000 7f60d0dff700 1 -- 172.21.15.23:0/1676910347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60cc08bbb0 con 0x7f60cc071850 2023-06-22T08:06:20.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.780+0000 7f60baffd700 1 -- 172.21.15.23:0/1676910347 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f60bc022070 con 0x7f60cc071850 2023-06-22T08:06:20.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.781+0000 7f60baffd700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f60ac06ace0 0x7f60ac06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:06:20.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.781+0000 7f60baffd700 1 -- 172.21.15.23:0/1676910347 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f60bc08a500 con 0x7f60cc071850 2023-06-22T08:06:20.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.782+0000 7f60c9d9b700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f60ac06ace0 0x7f60ac06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:06:20.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.782+0000 7f60c9d9b700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f60ac06ace0 0x7f60ac06d190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f60cc150270 tx=0x7f60c0007e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:06:20.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:20.786+0000 7f60baffd700 1 -- 172.21.15.23:0/1676910347 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f60bc055270 con 0x7f60cc071850 2023-06-22T08:06:21.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.045+0000 7f60d0dff700 1 -- 172.21.15.23:0/1676910347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60cc04ea20 con 0x7f60cc071850 2023-06-22T08:06:21.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.045+0000 7f60baffd700 1 -- 172.21.15.23:0/1676910347 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6247 (secure 0 0 0) 0x7f60bc015090 con 0x7f60cc071850 2023-06-22T08:06:21.049 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:06:21.049 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:19.794873+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:06:21.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.048+0000 7f60d0dff700 1 -- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f60ac06ace0 msgr2=0x7f60ac06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:21.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.048+0000 7f60d0dff700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f60ac06ace0 0x7f60ac06d190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f60cc150270 tx=0x7f60c0007e00 comp rx=0 tx=0).stop 2023-06-22T08:06:21.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.048+0000 7f60d0dff700 1 -- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60cc071850 msgr2=0x7f60cc1ee7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:06:21.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.048+0000 7f60d0dff700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60cc071850 0x7f60cc1ee7a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0090b0 tx=0x7f60bc004750 comp rx=0 tx=0).stop 2023-06-22T08:06:21.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.050+0000 7f60d0dff700 1 -- 172.21.15.23:0/1676910347 shutdown_connections 2023-06-22T08:06:21.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.050+0000 7f60d0dff700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60cc15cb00 0x7f60cc0b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:21.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.050+0000 7f60d0dff700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f60ac06ace0 0x7f60ac06d190 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:21.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.050+0000 7f60d0dff700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60cc071850 0x7f60cc1ee7a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:21.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.050+0000 7f60d0dff700 1 --2- 172.21.15.23:0/1676910347 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f60cc071c20 0x7f60cc14c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:06:21.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.050+0000 7f60d0dff700 1 -- 172.21.15.23:0/1676910347 >> 172.21.15.23:0/1676910347 conn(0x7f60cc06b890 msgr2=0x7f60cc0cb3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:06:21.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.051+0000 7f60d0dff700 1 -- 172.21.15.23:0/1676910347 shutdown_connections 2023-06-22T08:06:21.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:06:21.051+0000 7f60d0dff700 1 -- 172.21.15.23:0/1676910347 wait complete. 2023-06-22T08:06:21.059 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 52 2023-06-22T08:06:21.088 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-06-22T08:06:21.088 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-06-22T08:06:37.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:06:37.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:07:00.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.454+0000 7fc626dff700 1 Processor -- start 2023-06-22T08:07:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.455+0000 7fc626dff700 1 -- start start 2023-06-22T08:07:00.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.457+0000 7fc626dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc620152ce0 0x7fc6200cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:00.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.457+0000 7fc626dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c9fd0 0x7fc62015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:00.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.457+0000 7fc626dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62015c370 con 0x7fc620158880 2023-06-22T08:07:00.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.457+0000 7fc626dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6201530b0 con 0x7fc620152ce0 2023-06-22T08:07:00.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.457+0000 7fc626dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6200cf6f0 con 0x7fc6200c9fd0 2023-06-22T08:07:00.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.457+0000 7fc62539c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc620158880 0x7fc620158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:41896/0 (socket says 172.21.15.23:41896) 2023-06-22T08:07:00.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.457+0000 7fc62539c700 1 -- 172.21.15.23:0/780397361 learned_addr learned my addr 172.21.15.23:0/780397361 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.458+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540500052 0 0) 0x7fc62015c370 con 0x7fc620158880 2023-06-22T08:07:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.458+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6080035b0 con 0x7fc620158880 2023-06-22T08:07:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.458+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1759865815 0 0) 0x7fc6080035b0 con 0x7fc620158880 2023-06-22T08:07:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.458+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c9fd0 msgr2=0x7fc62015c800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-22T08:07:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.458+0000 7fc6177fe700 1 --2- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c9fd0 0x7fc62015c800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:00.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.458+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 >> v2:172.21.15.23:3300/0 conn(0x7fc620152ce0 msgr2=0x7fc6200cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:00.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc6177fe700 1 --2- 172.21.15.23:0/780397361 >> v2:172.21.15.23:3300/0 conn(0x7fc620152ce0 0x7fc6200cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:00.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62015bda0 con 0x7fc620158880 2023-06-22T08:07:00.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7fc618001670 con 0x7fc620158880 2023-06-22T08:07:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 >> v1:172.21.15.67:6789/0 conn(0x7fc620158880 legacy=0x7fc620158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc6177fe700 1 --2- 172.21.15.23:0/780397361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc608003a00 0x7fc608003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc6177fe700 1 --2- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc608004850 0x7fc608004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc6177fe700 1 --2- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c9fd0 0x7fc60800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6201530b0 con 0x7fc608003a00 2023-06-22T08:07:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6200cf6f0 con 0x7fc608004850 2023-06-22T08:07:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc608005600 con 0x7fc6200c9fd0 2023-06-22T08:07:00.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc617fff700 1 --2- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc608004850 0x7fc608004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:00.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.459+0000 7fc62539c700 1 --2- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c9fd0 0x7fc60800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:00.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.460+0000 7fc617fff700 1 -- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c9fd0 msgr2=0x7fc60800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:00.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.460+0000 7fc617fff700 1 --2- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c9fd0 0x7fc60800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:00.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.460+0000 7fc617fff700 1 -- 172.21.15.23:0/780397361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc608003a00 msgr2=0x7fc608003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:00.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.460+0000 7fc617fff700 1 --2- 172.21.15.23:0/780397361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc608003a00 0x7fc608003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:00.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.460+0000 7fc617fff700 1 -- 172.21.15.23:0/780397361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62015bda0 con 0x7fc608004850 2023-06-22T08:07:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.460+0000 7fc617fff700 1 --2- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc608004850 0x7fc608004cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc610009860 tx=0x7fc608005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1551632ccf470987 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.461+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc610011620 con 0x7fc608004850 2023-06-22T08:07:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.461+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc610011780 con 0x7fc608004850 2023-06-22T08:07:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.461+0000 7fc6177fe700 1 -- 172.21.15.23:0/780397361 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc610011b30 con 0x7fc608004850 2023-06-22T08:07:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.462+0000 7fc626dff700 1 -- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc608004850 msgr2=0x7fc608004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.462+0000 7fc626dff700 1 --2- 172.21.15.23:0/780397361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc608004850 0x7fc608004cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc610009860 tx=0x7fc608005880 comp rx=0 tx=0).stop 2023-06-22T08:07:00.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.462+0000 7fc62539c700 1 -- 172.21.15.23:0/780397361 reap_dead start 2023-06-22T08:07:00.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.462+0000 7fc626dff700 1 -- 172.21.15.23:0/780397361 shutdown_connections 2023-06-22T08:07:00.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.462+0000 7fc626dff700 1 -- 172.21.15.23:0/780397361 >> 172.21.15.23:0/780397361 conn(0x7fc62006ae30 msgr2=0x7fc62006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:00.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.462+0000 7fc626dff700 1 -- 172.21.15.23:0/780397361 shutdown_connections 2023-06-22T08:07:00.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.462+0000 7fc626dff700 1 -- 172.21.15.23:0/780397361 wait complete. 2023-06-22T08:07:00.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.465+0000 7fc626dff700 1 Processor -- start 2023-06-22T08:07:00.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.466+0000 7fc626dff700 1 -- start start 2023-06-22T08:07:00.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.468+0000 7fc626dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc608004850 0x7fc6200cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:00.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.468+0000 7fc626dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6200c4690 0x7fc6200cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:00.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.468+0000 7fc626dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c4b00 0x7fc6200cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:00.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.468+0000 7fc626dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6200cdb80 con 0x7fc6200c4690 2023-06-22T08:07:00.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.468+0000 7fc626dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6200cf540 con 0x7fc608004850 2023-06-22T08:07:00.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.468+0000 7fc626dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6201f5b70 con 0x7fc6200c4b00 2023-06-22T08:07:00.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.468+0000 7fc62539c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c4b00 0x7fc6200cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:00.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.469+0000 7fc62539c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c4b00 0x7fc6200cef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:38626/0 (socket says 172.21.15.23:38626) 2023-06-22T08:07:00.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.469+0000 7fc62539c700 1 -- 172.21.15.23:0/1761117297 learned_addr learned my addr 172.21.15.23:0/1761117297 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:00.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.469+0000 7fc617fff700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6200c4690 0x7fc6200cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:00.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.470+0000 7fc62539c700 1 -- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc608004850 msgr2=0x7fc6200cd050 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T08:07:00.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.470+0000 7fc62539c700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc608004850 0x7fc6200cd050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:00.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.470+0000 7fc62539c700 1 -- 172.21.15.23:0/1761117297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6200c4690 msgr2=0x7fc6200cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:00.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.470+0000 7fc62539c700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6200c4690 0x7fc6200cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:00.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.471+0000 7fc62539c700 1 -- 172.21.15.23:0/1761117297 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6201f5e10 con 0x7fc6200c4b00 2023-06-22T08:07:00.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.471+0000 7fc62539c700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c4b00 0x7fc6200cef40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc61800e780 tx=0x7fc61800ea90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:00.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.472+0000 7fc6157fa700 1 -- 172.21.15.23:0/1761117297 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618010510 con 0x7fc6200c4b00 2023-06-22T08:07:00.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.473+0000 7fc6157fa700 1 -- 172.21.15.23:0/1761117297 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc618010690 con 0x7fc6200c4b00 2023-06-22T08:07:00.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.473+0000 7fc6157fa700 1 -- 172.21.15.23:0/1761117297 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618010a40 con 0x7fc6200c4b00 2023-06-22T08:07:00.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.473+0000 7fc626dff700 1 -- 172.21.15.23:0/1761117297 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6201f60f0 con 0x7fc6200c4b00 2023-06-22T08:07:00.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.473+0000 7fc626dff700 1 -- 172.21.15.23:0/1761117297 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6201f66b0 con 0x7fc6200c4b00 2023-06-22T08:07:00.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.475+0000 7fc626dff700 1 -- 172.21.15.23:0/1761117297 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6000052f0 con 0x7fc6200c4b00 2023-06-22T08:07:00.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.477+0000 7fc6157fa700 1 -- 172.21.15.23:0/1761117297 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc618010ba0 con 0x7fc6200c4b00 2023-06-22T08:07:00.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.479+0000 7fc6157fa700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc60c06ade0 0x7fc60c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:00.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.479+0000 7fc6157fa700 1 -- 172.21.15.23:0/1761117297 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7fc6180895a0 con 0x7fc6200c4b00 2023-06-22T08:07:00.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.481+0000 7fc624b9b700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc60c06ade0 0x7fc60c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:00.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.483+0000 7fc6157fa700 1 -- 172.21.15.23:0/1761117297 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc6180542d0 con 0x7fc6200c4b00 2023-06-22T08:07:00.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.485+0000 7fc624b9b700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc60c06ade0 0x7fc60c06d290 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc61c00ab10 tx=0x7fc61c009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:00.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.956+0000 7fc626dff700 1 -- 172.21.15.23:0/1761117297 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc600005870 con 0x7fc6200c4b00 2023-06-22T08:07:00.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.957+0000 7fc6157fa700 1 -- 172.21.15.23:0/1761117297 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6255 (secure 0 0 0) 0x7fc6180578f0 con 0x7fc6200c4b00 2023-06-22T08:07:00.960 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:00.961 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:51.730709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:00.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.961+0000 7fc626dff700 1 -- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc60c06ade0 msgr2=0x7fc60c06d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:00.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.961+0000 7fc626dff700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc60c06ade0 0x7fc60c06d290 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc61c00ab10 tx=0x7fc61c009250 comp rx=0 tx=0).stop 2023-06-22T08:07:00.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.961+0000 7fc626dff700 1 -- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c4b00 msgr2=0x7fc6200cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:00.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.961+0000 7fc626dff700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c4b00 0x7fc6200cef40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc61800e780 tx=0x7fc61800ea90 comp rx=0 tx=0).stop 2023-06-22T08:07:00.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.962+0000 7fc626dff700 1 -- 172.21.15.23:0/1761117297 shutdown_connections 2023-06-22T08:07:00.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.962+0000 7fc626dff700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6200c4b00 0x7fc6200cef40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:00.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.962+0000 7fc626dff700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc60c06ade0 0x7fc60c06d290 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:00.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.962+0000 7fc626dff700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc608004850 0x7fc6200cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:00.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.962+0000 7fc626dff700 1 --2- 172.21.15.23:0/1761117297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6200c4690 0x7fc6200cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:00.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.962+0000 7fc626dff700 1 -- 172.21.15.23:0/1761117297 >> 172.21.15.23:0/1761117297 conn(0x7fc62006ae30 msgr2=0x7fc6200c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:00.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.962+0000 7fc626dff700 1 -- 172.21.15.23:0/1761117297 shutdown_connections 2023-06-22T08:07:00.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:00.963+0000 7fc626dff700 1 -- 172.21.15.23:0/1761117297 wait complete. 2023-06-22T08:07:00.970 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 56 2023-06-22T08:07:01.002 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=2) 2023-06-22T08:07:01.003 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-22T08:07:01.003 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:07:01.023 INFO:tasks.ceph.mds.a:Stopped 2023-06-22T08:07:01.024 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.259+0000 7f1c7e7ff700 1 Processor -- start 2023-06-22T08:07:01.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.259+0000 7f1c7e7ff700 1 -- start start 2023-06-22T08:07:01.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.260+0000 7f1c7e7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1c78152ce0 0x7f1c780cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.260+0000 7f1c7e7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c780c9fd0 0x7f1c7815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.260+0000 7f1c7e7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c7815c370 con 0x7f1c78158880 2023-06-22T08:07:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.260+0000 7f1c7e7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c781530b0 con 0x7f1c78152ce0 2023-06-22T08:07:01.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.260+0000 7f1c7e7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c780cf6f0 con 0x7f1c780c9fd0 2023-06-22T08:07:01.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.261+0000 7f1c77fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1c78152ce0 0x7f1c780cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.261+0000 7f1c777fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c780c9fd0 0x7f1c7815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.261+0000 7f1c7cd9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1c78158880 0x7f1c78158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57222/0 (socket says 172.21.15.23:57222) 2023-06-22T08:07:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.261+0000 7f1c7cd9c700 1 -- 172.21.15.23:0/178390347 learned_addr learned my addr 172.21.15.23:0/178390347 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.261+0000 7f1c777fe700 1 -- 172.21.15.23:0/178390347 >> v2:172.21.15.67:3300/0 conn(0x7f1c78152ce0 msgr2=0x7f1c780cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.261+0000 7f1c777fe700 1 --2- 172.21.15.23:0/178390347 >> v2:172.21.15.67:3300/0 conn(0x7f1c78152ce0 0x7f1c780cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.261+0000 7f1c76ffd700 1 -- 172.21.15.23:0/178390347 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399539972 0 0) 0x7f1c7815c370 con 0x7f1c78158880 2023-06-22T08:07:01.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.261+0000 7f1c777fe700 1 -- 172.21.15.23:0/178390347 >> v1:172.21.15.23:6789/0 conn(0x7f1c78158880 legacy=0x7f1c78158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.261+0000 7f1c777fe700 1 -- 172.21.15.23:0/178390347 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7815bda0 con 0x7f1c780c9fd0 2023-06-22T08:07:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.261+0000 7f1c77fff700 1 --2- 172.21.15.23:0/178390347 >> v2:172.21.15.67:3300/0 conn(0x7f1c78152ce0 0x7f1c780cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c777fe700 1 --2- 172.21.15.23:0/178390347 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c780c9fd0 0x7f1c7815c800 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009840 tx=0x7f1c6c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1d839728de9fcce server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c76ffd700 1 -- 172.21.15.23:0/178390347 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c011830 con 0x7f1c780c9fd0 2023-06-22T08:07:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c76ffd700 1 -- 172.21.15.23:0/178390347 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c011990 con 0x7f1c780c9fd0 2023-06-22T08:07:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c76ffd700 1 -- 172.21.15.23:0/178390347 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c011d20 con 0x7f1c780c9fd0 2023-06-22T08:07:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/178390347 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c780c9fd0 msgr2=0x7f1c7815c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:01.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c7e7ff700 1 --2- 172.21.15.23:0/178390347 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c780c9fd0 0x7f1c7815c800 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009840 tx=0x7f1c6c009b50 comp rx=0 tx=0).stop 2023-06-22T08:07:01.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/178390347 shutdown_connections 2023-06-22T08:07:01.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c7e7ff700 1 --2- 172.21.15.23:0/178390347 >> v2:172.21.15.67:3300/0 conn(0x7f1c78152ce0 0x7f1c780cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:01.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c7e7ff700 1 --2- 172.21.15.23:0/178390347 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c780c9fd0 0x7f1c7815c800 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:01.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/178390347 >> 172.21.15.23:0/178390347 conn(0x7f1c7806ae30 msgr2=0x7f1c7806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:01.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/178390347 shutdown_connections 2023-06-22T08:07:01.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.262+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/178390347 wait complete. 2023-06-22T08:07:01.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.264+0000 7f1c7e7ff700 1 Processor -- start 2023-06-22T08:07:01.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.264+0000 7f1c7e7ff700 1 -- start start 2023-06-22T08:07:01.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c7e7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c780c9fd0 0x7f1c781485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:01.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c7e7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c78152ce0 0x7f1c781eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:01.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c7e7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c78158880 0x7f1c781490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:01.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c7e7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c781ee660 con 0x7f1c78158880 2023-06-22T08:07:01.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c7e7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c78148b10 con 0x7f1c780c9fd0 2023-06-22T08:07:01.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c7e7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c780b4360 con 0x7f1c78152ce0 2023-06-22T08:07:01.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c777fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c78152ce0 0x7f1c781eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:01.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c77fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c780c9fd0 0x7f1c781485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:01.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c777fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c78152ce0 0x7f1c781eeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:38652/0 (socket says 172.21.15.23:38652) 2023-06-22T08:07:01.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c7cd9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c78158880 0x7f1c781490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:01.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.265+0000 7f1c777fe700 1 -- 172.21.15.23:0/2764539341 learned_addr learned my addr 172.21.15.23:0/2764539341 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:01.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c777fe700 1 -- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c780c9fd0 msgr2=0x7f1c781485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:01.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c777fe700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c780c9fd0 0x7f1c781485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:01.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c777fe700 1 -- 172.21.15.23:0/2764539341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c78158880 msgr2=0x7f1c781490d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:01.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c777fe700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c78158880 0x7f1c781490d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:01.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c777fe700 1 -- 172.21.15.23:0/2764539341 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c680094a0 con 0x7f1c78152ce0 2023-06-22T08:07:01.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c77fff700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c780c9fd0 0x7f1c781485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:01.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c777fe700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c78152ce0 0x7f1c781eeaf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009b90 tx=0x7f1c6c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:01.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c74ff9700 1 -- 172.21.15.23:0/2764539341 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c0036a0 con 0x7f1c78152ce0 2023-06-22T08:07:01.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c74ff9700 1 -- 172.21.15.23:0/2764539341 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c01e070 con 0x7f1c78152ce0 2023-06-22T08:07:01.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/2764539341 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c0094a0 con 0x7f1c78152ce0 2023-06-22T08:07:01.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.266+0000 7f1c74ff9700 1 -- 172.21.15.23:0/2764539341 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c003a60 con 0x7f1c78152ce0 2023-06-22T08:07:01.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.267+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/2764539341 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c780b3da0 con 0x7f1c78152ce0 2023-06-22T08:07:01.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.268+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/2764539341 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c780b4710 con 0x7f1c78152ce0 2023-06-22T08:07:01.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.269+0000 7f1c74ff9700 1 -- 172.21.15.23:0/2764539341 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1c6c003bc0 con 0x7f1c78152ce0 2023-06-22T08:07:01.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.269+0000 7f1c74ff9700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6406ace0 0x7f1c6406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:01.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.270+0000 7f1c74ff9700 1 -- 172.21.15.23:0/2764539341 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f1c6c08a790 con 0x7f1c78152ce0 2023-06-22T08:07:01.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.270+0000 7f1c77fff700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6406ace0 0x7f1c6406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:01.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.270+0000 7f1c77fff700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6406ace0 0x7f1c6406d190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1c680114b0 tx=0x7f1c68009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:01.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.274+0000 7f1c74ff9700 1 -- 172.21.15.23:0/2764539341 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1c6c055440 con 0x7f1c78152ce0 2023-06-22T08:07:01.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.526+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/2764539341 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c7804ea20 con 0x7f1c78152ce0 2023-06-22T08:07:01.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.527+0000 7f1c74ff9700 1 -- 172.21.15.23:0/2764539341 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f1c6c058a60 con 0x7f1c78152ce0 2023-06-22T08:07:01.530 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:01.531 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:51.730709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:01.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6406ace0 msgr2=0x7f1c6406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:01.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6406ace0 0x7f1c6406d190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1c680114b0 tx=0x7f1c68009210 comp rx=0 tx=0).stop 2023-06-22T08:07:01.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c78152ce0 msgr2=0x7f1c781eeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:01.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c78152ce0 0x7f1c781eeaf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009b90 tx=0x7f1c6c004750 comp rx=0 tx=0).stop 2023-06-22T08:07:01.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/2764539341 shutdown_connections 2023-06-22T08:07:01.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c78158880 0x7f1c781490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:01.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6406ace0 0x7f1c6406d190 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:01.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c780c9fd0 0x7f1c781485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:01.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 --2- 172.21.15.23:0/2764539341 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c78152ce0 0x7f1c781eeaf0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:01.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/2764539341 >> 172.21.15.23:0/2764539341 conn(0x7f1c7806ae30 msgr2=0x7f1c780c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:01.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/2764539341 shutdown_connections 2023-06-22T08:07:01.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:01.529+0000 7f1c7e7ff700 1 -- 172.21.15.23:0/2764539341 wait complete. 2023-06-22T08:07:01.536 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 56 2023-06-22T08:07:01.551 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:07:03.552 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.759+0000 7f2b8cfff700 1 Processor -- start 2023-06-22T08:07:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.759+0000 7f2b8cfff700 1 -- start start 2023-06-22T08:07:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b8cfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b880c97d0 0x7f2b88159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b8cfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2b880c45f0 0x7f2b880c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b8cfff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b880c9c30 con 0x7f2b88152d70 2023-06-22T08:07:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b8cfff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b880c95e0 con 0x7f2b880c45f0 2023-06-22T08:07:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b8cfff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8815bea0 con 0x7f2b880c97d0 2023-06-22T08:07:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b8659c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2b88152d70 0x7f2b880cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57238/0 (socket says 172.21.15.23:57238) 2023-06-22T08:07:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b8659c700 1 -- 172.21.15.23:0/2336294922 learned_addr learned my addr 172.21.15.23:0/2336294922 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b86d9d700 1 --2- 172.21.15.23:0/2336294922 >> v2:172.21.15.23:3300/0 conn(0x7f2b880c45f0 0x7f2b880c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b85d9b700 1 --2- 172.21.15.23:0/2336294922 >> v2:172.21.15.67:3300/0 conn(0x7f2b880c97d0 0x7f2b88159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b8559a700 1 -- 172.21.15.23:0/2336294922 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700184169 0 0) 0x7f2b880c9c30 con 0x7f2b88152d70 2023-06-22T08:07:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b8559a700 1 -- 172.21.15.23:0/2336294922 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b700035b0 con 0x7f2b88152d70 2023-06-22T08:07:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b85d9b700 1 -- 172.21.15.23:0/2336294922 >> v2:172.21.15.23:3300/0 conn(0x7f2b880c45f0 msgr2=0x7f2b880c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b85d9b700 1 --2- 172.21.15.23:0/2336294922 >> v2:172.21.15.23:3300/0 conn(0x7f2b880c45f0 0x7f2b880c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b85d9b700 1 -- 172.21.15.23:0/2336294922 >> v1:172.21.15.23:6789/0 conn(0x7f2b88152d70 legacy=0x7f2b880cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.761+0000 7f2b85d9b700 1 -- 172.21.15.23:0/2336294922 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8815c160 con 0x7f2b880c97d0 2023-06-22T08:07:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.762+0000 7f2b85d9b700 1 --2- 172.21.15.23:0/2336294922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b880c97d0 0x7f2b88159330 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2b8808ba40 tx=0x7f2b7c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bb098afbf304c0f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.762+0000 7f2b8559a700 1 -- 172.21.15.23:0/2336294922 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c011830 con 0x7f2b880c97d0 2023-06-22T08:07:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.762+0000 7f2b8559a700 1 -- 172.21.15.23:0/2336294922 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c011990 con 0x7f2b880c97d0 2023-06-22T08:07:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.762+0000 7f2b8559a700 1 -- 172.21.15.23:0/2336294922 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c011d20 con 0x7f2b880c97d0 2023-06-22T08:07:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.762+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2336294922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b880c97d0 msgr2=0x7f2b88159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.762+0000 7f2b8cfff700 1 --2- 172.21.15.23:0/2336294922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b880c97d0 0x7f2b88159330 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2b8808ba40 tx=0x7f2b7c009b20 comp rx=0 tx=0).stop 2023-06-22T08:07:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.763+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2336294922 shutdown_connections 2023-06-22T08:07:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.763+0000 7f2b8cfff700 1 --2- 172.21.15.23:0/2336294922 >> v2:172.21.15.23:3300/0 conn(0x7f2b880c45f0 0x7f2b880c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.763+0000 7f2b8cfff700 1 --2- 172.21.15.23:0/2336294922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b880c97d0 0x7f2b88159330 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.763+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2336294922 >> 172.21.15.23:0/2336294922 conn(0x7f2b8806ae30 msgr2=0x7f2b8806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.763+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2336294922 shutdown_connections 2023-06-22T08:07:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.763+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2336294922 wait complete. 2023-06-22T08:07:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.764+0000 7f2b8cfff700 1 Processor -- start 2023-06-22T08:07:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.765+0000 7f2b8cfff700 1 -- start start 2023-06-22T08:07:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.765+0000 7f2b8cfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b880c45f0 0x7f2b880c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.765+0000 7f2b8cfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b880c97d0 0x7f2b880c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8cfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b88152d70 0x7f2b880c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8cfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880c7a60 con 0x7f2b880c45f0 2023-06-22T08:07:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8cfff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880c58a0 con 0x7f2b880c97d0 2023-06-22T08:07:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8cfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880b3f60 con 0x7f2b88152d70 2023-06-22T08:07:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b880c45f0 0x7f2b880c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b880c45f0 0x7f2b880c7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42350/0 (socket says 172.21.15.23:42350) 2023-06-22T08:07:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8659c700 1 -- 172.21.15.23:0/2247013071 learned_addr learned my addr 172.21.15.23:0/2247013071 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b85d9b700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b880c97d0 0x7f2b880c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b86d9d700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b88152d70 0x7f2b880c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:03.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8659c700 1 -- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b88152d70 msgr2=0x7f2b880c6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:03.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8659c700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b88152d70 0x7f2b880c6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:03.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8659c700 1 -- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b880c97d0 msgr2=0x7f2b880c8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:03.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8659c700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b880c97d0 0x7f2b880c8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:03.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8659c700 1 -- 172.21.15.23:0/2247013071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c0094a0 con 0x7f2b880c45f0 2023-06-22T08:07:03.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b85d9b700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b880c97d0 0x7f2b880c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:07:03.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.766+0000 7f2b8659c700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b880c45f0 0x7f2b880c7520 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2b78001eb0 tx=0x7f2b7800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:03.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.767+0000 7f2b76ffd700 1 -- 172.21.15.23:0/2247013071 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78005d70 con 0x7f2b880c45f0 2023-06-22T08:07:03.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.767+0000 7f2b76ffd700 1 -- 172.21.15.23:0/2247013071 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b780070d0 con 0x7f2b880c45f0 2023-06-22T08:07:03.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.767+0000 7f2b76ffd700 1 -- 172.21.15.23:0/2247013071 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78007460 con 0x7f2b880c45f0 2023-06-22T08:07:03.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.767+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2247013071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b880b4240 con 0x7f2b880c45f0 2023-06-22T08:07:03.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.767+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2247013071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b880c5b40 con 0x7f2b880c45f0 2023-06-22T08:07:03.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.768+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2247013071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8804ea20 con 0x7f2b880c45f0 2023-06-22T08:07:03.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.769+0000 7f2b76ffd700 1 -- 172.21.15.23:0/2247013071 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2b780075c0 con 0x7f2b880c45f0 2023-06-22T08:07:03.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.770+0000 7f2b76ffd700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b6806ace0 0x7f2b6806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:03.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.770+0000 7f2b76ffd700 1 -- 172.21.15.23:0/2247013071 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f2b78088ef0 con 0x7f2b880c45f0 2023-06-22T08:07:03.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.770+0000 7f2b85d9b700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b6806ace0 0x7f2b6806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:03.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.771+0000 7f2b85d9b700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b6806ace0 0x7f2b6806d190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0090b0 tx=0x7f2b7c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:03.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:03.774+0000 7f2b76ffd700 1 -- 172.21.15.23:0/2247013071 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2b78053a80 con 0x7f2b880c45f0 2023-06-22T08:07:04.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.026+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2247013071 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b880c9e00 con 0x7f2b880c45f0 2023-06-22T08:07:04.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.027+0000 7f2b76ffd700 1 -- 172.21.15.23:0/2247013071 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f2b78017020 con 0x7f2b880c45f0 2023-06-22T08:07:04.030 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:04.031 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:51.730709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:04.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b6806ace0 msgr2=0x7f2b6806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:04.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b6806ace0 0x7f2b6806d190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0090b0 tx=0x7f2b7c008030 comp rx=0 tx=0).stop 2023-06-22T08:07:04.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2247013071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b880c45f0 msgr2=0x7f2b880c7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:04.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b880c45f0 0x7f2b880c7520 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2b78001eb0 tx=0x7f2b7800c860 comp rx=0 tx=0).stop 2023-06-22T08:07:04.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2247013071 shutdown_connections 2023-06-22T08:07:04.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b88152d70 0x7f2b880c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:04.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b880c45f0 0x7f2b880c7520 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:04.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2b6806ace0 0x7f2b6806d190 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:04.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 --2- 172.21.15.23:0/2247013071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b880c97d0 0x7f2b880c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:04.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2247013071 >> 172.21.15.23:0/2247013071 conn(0x7f2b8806ae30 msgr2=0x7f2b880ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:04.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2247013071 shutdown_connections 2023-06-22T08:07:04.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:04.029+0000 7f2b8cfff700 1 -- 172.21.15.23:0/2247013071 wait complete. 2023-06-22T08:07:04.034 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 56 2023-06-22T08:07:04.051 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:07:06.052 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:06.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.259+0000 7ff8e1dff700 1 Processor -- start 2023-06-22T08:07:06.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.259+0000 7ff8e1dff700 1 -- start start 2023-06-22T08:07:06.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8e1dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 0x7ff8dc0c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:06.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8e1dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff8dc0c9200 0x7ff8dc0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:06.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8e1dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8dc0cb7a0 0x7ff8dc15c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:06.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8e1dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8dc0c5470 con 0x7ff8dc0c9200 2023-06-22T08:07:06.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8e1dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8dc0cbbd0 con 0x7ff8dc0cb7a0 2023-06-22T08:07:06.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8e1dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc159730 con 0x7ff8dc152a10 2023-06-22T08:07:06.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8dbfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8dc0cb7a0 0x7ff8dc15c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:06.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8db7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 0x7ff8dc0c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:06.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8daffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff8dc0c9200 0x7ff8dc0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:06.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8dbfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff8dc0cb7a0 0x7ff8dc15c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:42056/0 (socket says 172.21.15.23:42056) 2023-06-22T08:07:06.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.261+0000 7ff8dbfff700 1 -- 172.21.15.23:0/3242709439 learned_addr learned my addr 172.21.15.23:0/3242709439 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:06.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.262+0000 7ff8dbfff700 1 -- 172.21.15.23:0/3242709439 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 msgr2=0x7ff8dc0c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:06.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.262+0000 7ff8dbfff700 1 --2- 172.21.15.23:0/3242709439 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 0x7ff8dc0c4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.262+0000 7ff8dbfff700 1 -- 172.21.15.23:0/3242709439 >> v2:172.21.15.23:3300/0 conn(0x7ff8dc0c9200 msgr2=0x7ff8dc0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:06.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.262+0000 7ff8dbfff700 1 --2- 172.21.15.23:0/3242709439 >> v2:172.21.15.23:3300/0 conn(0x7ff8dc0c9200 0x7ff8dc0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.262+0000 7ff8dbfff700 1 -- 172.21.15.23:0/3242709439 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc158910 con 0x7ff8dc0cb7a0 2023-06-22T08:07:06.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.262+0000 7ff8db7fe700 1 --2- 172.21.15.23:0/3242709439 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 0x7ff8dc0c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:06.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.262+0000 7ff8dbfff700 1 --2- 172.21.15.23:0/3242709439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8dc0cb7a0 0x7ff8dc15c0b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc08c490 tx=0x7ff8d400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe90c3d82a143d57 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:06.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.262+0000 7ff8da7fc700 1 -- 172.21.15.23:0/3242709439 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d400e070 con 0x7ff8dc0cb7a0 2023-06-22T08:07:06.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.262+0000 7ff8da7fc700 1 -- 172.21.15.23:0/3242709439 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d400b910 con 0x7ff8dc0cb7a0 2023-06-22T08:07:06.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.262+0000 7ff8da7fc700 1 -- 172.21.15.23:0/3242709439 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d400bcc0 con 0x7ff8dc0cb7a0 2023-06-22T08:07:06.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.263+0000 7ff8e1dff700 1 -- 172.21.15.23:0/3242709439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8dc0cb7a0 msgr2=0x7ff8dc15c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:06.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.263+0000 7ff8e1dff700 1 --2- 172.21.15.23:0/3242709439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8dc0cb7a0 0x7ff8dc15c0b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc08c490 tx=0x7ff8d400b330 comp rx=0 tx=0).stop 2023-06-22T08:07:06.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.263+0000 7ff8e1dff700 1 -- 172.21.15.23:0/3242709439 shutdown_connections 2023-06-22T08:07:06.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.263+0000 7ff8e1dff700 1 --2- 172.21.15.23:0/3242709439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8dc0cb7a0 0x7ff8dc15c0b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.263+0000 7ff8e1dff700 1 --2- 172.21.15.23:0/3242709439 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 0x7ff8dc0c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.263+0000 7ff8e1dff700 1 --2- 172.21.15.23:0/3242709439 >> v2:172.21.15.23:3300/0 conn(0x7ff8dc0c9200 0x7ff8dc0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.263+0000 7ff8e1dff700 1 -- 172.21.15.23:0/3242709439 >> 172.21.15.23:0/3242709439 conn(0x7ff8dc065b90 msgr2=0x7ff8dc065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:06.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.263+0000 7ff8e1dff700 1 -- 172.21.15.23:0/3242709439 shutdown_connections 2023-06-22T08:07:06.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.263+0000 7ff8e1dff700 1 -- 172.21.15.23:0/3242709439 wait complete. 2023-06-22T08:07:06.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.265+0000 7ff8e1dff700 1 Processor -- start 2023-06-22T08:07:06.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.265+0000 7ff8e1dff700 1 -- start start 2023-06-22T08:07:06.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8e1dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8dc0c9200 0x7ff8dc15b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:06.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8e1dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8dc0cb7a0 0x7ff8dc1eed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:06.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8e1dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 0x7ff8dc14ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:06.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8e1dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc1ee890 con 0x7ff8dc0c9200 2023-06-22T08:07:06.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8e1dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0c8cc0 con 0x7ff8dc0cb7a0 2023-06-22T08:07:06.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8e1dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0c7a40 con 0x7ff8dc152a10 2023-06-22T08:07:06.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8db7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8dc0c9200 0x7ff8dc15b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:06.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8db7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8dc0c9200 0x7ff8dc15b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52954/0 (socket says 172.21.15.23:52954) 2023-06-22T08:07:06.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8db7fe700 1 -- 172.21.15.23:0/2375955756 learned_addr learned my addr 172.21.15.23:0/2375955756 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:06.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8daffd700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8dc0cb7a0 0x7ff8dc1eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:06.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8dbfff700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 0x7ff8dc14ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:06.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8db7fe700 1 -- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 msgr2=0x7ff8dc14ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:06.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8db7fe700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 0x7ff8dc14ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8db7fe700 1 -- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8dc0cb7a0 msgr2=0x7ff8dc1eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:06.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8db7fe700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8dc0cb7a0 0x7ff8dc1eed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.266+0000 7ff8db7fe700 1 -- 172.21.15.23:0/2375955756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc0094a0 con 0x7ff8dc0c9200 2023-06-22T08:07:06.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.267+0000 7ff8db7fe700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8dc0c9200 0x7ff8dc15b0f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00ebe0 tx=0x7ff8cc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:06.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.267+0000 7ff8c3fff700 1 -- 172.21.15.23:0/2375955756 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc009760 con 0x7ff8dc0c9200 2023-06-22T08:07:06.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.267+0000 7ff8c3fff700 1 -- 172.21.15.23:0/2375955756 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc00c850 con 0x7ff8dc0c9200 2023-06-22T08:07:06.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.267+0000 7ff8c3fff700 1 -- 172.21.15.23:0/2375955756 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc00cbe0 con 0x7ff8dc0c9200 2023-06-22T08:07:06.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.267+0000 7ff8e1dff700 1 -- 172.21.15.23:0/2375955756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d4009b30 con 0x7ff8dc0c9200 2023-06-22T08:07:06.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.267+0000 7ff8e1dff700 1 -- 172.21.15.23:0/2375955756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8dc0c8420 con 0x7ff8dc0c9200 2023-06-22T08:07:06.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.268+0000 7ff8e1dff700 1 -- 172.21.15.23:0/2375955756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8dc0ca630 con 0x7ff8dc0c9200 2023-06-22T08:07:06.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.269+0000 7ff8c3fff700 1 -- 172.21.15.23:0/2375955756 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff8cc00cd40 con 0x7ff8dc0c9200 2023-06-22T08:07:06.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.270+0000 7ff8c3fff700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff8c806ace0 0x7ff8c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:06.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.270+0000 7ff8c3fff700 1 -- 172.21.15.23:0/2375955756 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7ff8cc089820 con 0x7ff8dc0c9200 2023-06-22T08:07:06.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.271+0000 7ff8daffd700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff8c806ace0 0x7ff8c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:06.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.271+0000 7ff8daffd700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff8c806ace0 0x7ff8c806d190 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0b10b0 tx=0x7ff8d0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:06.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.275+0000 7ff8c3fff700 1 -- 172.21.15.23:0/2375955756 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff8cc054570 con 0x7ff8dc0c9200 2023-06-22T08:07:06.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.530+0000 7ff8e1dff700 1 -- 172.21.15.23:0/2375955756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8dc04ea20 con 0x7ff8dc0c9200 2023-06-22T08:07:06.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.531+0000 7ff8c3fff700 1 -- 172.21.15.23:0/2375955756 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6255 (secure 0 0 0) 0x7ff8cc057b90 con 0x7ff8dc0c9200 2023-06-22T08:07:06.534 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:06.535 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:51.730709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:06.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 -- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff8c806ace0 msgr2=0x7ff8c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:06.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff8c806ace0 0x7ff8c806d190 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0b10b0 tx=0x7ff8d0009210 comp rx=0 tx=0).stop 2023-06-22T08:07:06.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 -- 172.21.15.23:0/2375955756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8dc0c9200 msgr2=0x7ff8dc15b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:06.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8dc0c9200 0x7ff8dc15b0f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00ebe0 tx=0x7ff8cc00eef0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 -- 172.21.15.23:0/2375955756 shutdown_connections 2023-06-22T08:07:06.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8dc152a10 0x7ff8dc14ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8dc0c9200 0x7ff8dc15b0f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff8c806ace0 0x7ff8c806d190 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 --2- 172.21.15.23:0/2375955756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8dc0cb7a0 0x7ff8dc1eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:06.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 -- 172.21.15.23:0/2375955756 >> 172.21.15.23:0/2375955756 conn(0x7ff8dc065b90 msgr2=0x7ff8dc0660d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:06.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 -- 172.21.15.23:0/2375955756 shutdown_connections 2023-06-22T08:07:06.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:06.533+0000 7ff8e1dff700 1 -- 172.21.15.23:0/2375955756 wait complete. 2023-06-22T08:07:06.541 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 56 2023-06-22T08:07:06.555 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:07:07.619 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:07:07.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:07:08.556 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:08.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.764+0000 7f9baedff700 1 Processor -- start 2023-06-22T08:07:08.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.764+0000 7f9baedff700 1 -- start start 2023-06-22T08:07:08.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9baedff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ba80c69e0 0x7f9ba8155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:08.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9baedff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba80c51a0 con 0x7f9ba8071070 2023-06-22T08:07:08.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9baedff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba80c44e0 con 0x7f9ba814f850 2023-06-22T08:07:08.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9baedff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba80c5440 con 0x7f9ba80c69e0 2023-06-22T08:07:08.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9ba7fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ba80c69e0 0x7f9ba8155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:08.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9bad39c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9ba8071070 0x7f9ba815fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52122/0 (socket says 172.21.15.23:52122) 2023-06-22T08:07:08.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9bad39c700 1 -- 172.21.15.23:0/160746634 learned_addr learned my addr 172.21.15.23:0/160746634 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:08.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9bacb9b700 1 --1- 172.21.15.23:0/160746634 >> v1:172.21.15.67:6789/0 conn(0x7f9ba814f850 0x7f9ba80c3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:52426/0 (socket says 172.21.15.23:52426) 2023-06-22T08:07:08.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9ba7fff700 1 -- 172.21.15.23:0/160746634 >> v1:172.21.15.67:6789/0 conn(0x7f9ba814f850 legacy=0x7f9ba80c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:08.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9ba7fff700 1 -- 172.21.15.23:0/160746634 >> v1:172.21.15.23:6789/0 conn(0x7f9ba8071070 legacy=0x7f9ba815fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:08.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9ba7fff700 1 -- 172.21.15.23:0/160746634 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba80c5720 con 0x7f9ba80c69e0 2023-06-22T08:07:08.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.766+0000 7f9ba77fe700 1 -- 172.21.15.23:0/160746634 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073945788 0 0) 0x7f9ba80c51a0 con 0x7f9ba8071070 2023-06-22T08:07:08.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.767+0000 7f9ba7fff700 1 --2- 172.21.15.23:0/160746634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba80c69e0 0x7f9ba8155600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009840 tx=0x7f9b94009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58501da58655a386 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:08.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.767+0000 7f9ba77fe700 1 -- 172.21.15.23:0/160746634 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94011830 con 0x7f9ba80c69e0 2023-06-22T08:07:08.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.767+0000 7f9ba77fe700 1 -- 172.21.15.23:0/160746634 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b94011990 con 0x7f9ba80c69e0 2023-06-22T08:07:08.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.767+0000 7f9ba77fe700 1 -- 172.21.15.23:0/160746634 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94011d20 con 0x7f9ba80c69e0 2023-06-22T08:07:08.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.767+0000 7f9baedff700 1 -- 172.21.15.23:0/160746634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba80c69e0 msgr2=0x7f9ba8155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:08.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.767+0000 7f9baedff700 1 --2- 172.21.15.23:0/160746634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba80c69e0 0x7f9ba8155600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009840 tx=0x7f9b94009b50 comp rx=0 tx=0).stop 2023-06-22T08:07:08.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.768+0000 7f9baedff700 1 -- 172.21.15.23:0/160746634 shutdown_connections 2023-06-22T08:07:08.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.768+0000 7f9baedff700 1 --2- 172.21.15.23:0/160746634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba80c69e0 0x7f9ba8155600 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:08.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.768+0000 7f9baedff700 1 -- 172.21.15.23:0/160746634 >> 172.21.15.23:0/160746634 conn(0x7f9ba80ca3f0 msgr2=0x7f9ba80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:08.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.768+0000 7f9baedff700 1 -- 172.21.15.23:0/160746634 shutdown_connections 2023-06-22T08:07:08.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.768+0000 7f9baedff700 1 -- 172.21.15.23:0/160746634 wait complete. 2023-06-22T08:07:08.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.769+0000 7f9baedff700 1 Processor -- start 2023-06-22T08:07:08.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.770+0000 7f9baedff700 1 -- start start 2023-06-22T08:07:08.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.770+0000 7f9baedff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba8071070 0x7f9ba81e3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:08.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.770+0000 7f9baedff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9ba80c69e0 0x7f9ba814ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:08.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9baedff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ba814f850 0x7f9ba814b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:08.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9baedff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba81f2280 con 0x7f9ba814f850 2023-06-22T08:07:08.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9baedff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba814d440 con 0x7f9ba8071070 2023-06-22T08:07:08.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9baedff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba81e4530 con 0x7f9ba80c69e0 2023-06-22T08:07:08.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9bad39c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ba814f850 0x7f9ba814b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:08.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9bacb9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba8071070 0x7f9ba81e3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:08.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9ba7fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9ba80c69e0 0x7f9ba814ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:08.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9bacb9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba8071070 0x7f9ba81e3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:42074/0 (socket says 172.21.15.23:42074) 2023-06-22T08:07:08.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9bacb9b700 1 -- 172.21.15.23:0/2904402698 learned_addr learned my addr 172.21.15.23:0/2904402698 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:08.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9bacb9b700 1 -- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9ba80c69e0 msgr2=0x7f9ba814ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:08.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9bacb9b700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9ba80c69e0 0x7f9ba814ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:08.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9bacb9b700 1 -- 172.21.15.23:0/2904402698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ba814f850 msgr2=0x7f9ba814b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:08.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9bacb9b700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ba814f850 0x7f9ba814b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:08.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9bacb9b700 1 -- 172.21.15.23:0/2904402698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b940094a0 con 0x7f9ba8071070 2023-06-22T08:07:08.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.771+0000 7f9bad39c700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ba814f850 0x7f9ba814b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:08.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.772+0000 7f9bacb9b700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba8071070 0x7f9ba81e3c60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00c430 tx=0x7f9b9c00c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:08.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.772+0000 7f9ba57fa700 1 -- 172.21.15.23:0/2904402698 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c005ab0 con 0x7f9ba8071070 2023-06-22T08:07:08.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.772+0000 7f9ba57fa700 1 -- 172.21.15.23:0/2904402698 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c005c10 con 0x7f9ba8071070 2023-06-22T08:07:08.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.772+0000 7f9baedff700 1 -- 172.21.15.23:0/2904402698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba814d180 con 0x7f9ba8071070 2023-06-22T08:07:08.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.772+0000 7f9ba57fa700 1 -- 172.21.15.23:0/2904402698 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0070d0 con 0x7f9ba8071070 2023-06-22T08:07:08.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.772+0000 7f9baedff700 1 -- 172.21.15.23:0/2904402698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba81e47d0 con 0x7f9ba8071070 2023-06-22T08:07:08.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.773+0000 7f9baedff700 1 -- 172.21.15.23:0/2904402698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba814fc20 con 0x7f9ba8071070 2023-06-22T08:07:08.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.774+0000 7f9ba57fa700 1 -- 172.21.15.23:0/2904402698 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9b9c007230 con 0x7f9ba8071070 2023-06-22T08:07:08.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.775+0000 7f9ba57fa700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9b8c06ad10 0x7f9b8c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:08.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.775+0000 7f9ba57fa700 1 -- 172.21.15.23:0/2904402698 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f9b9c088b20 con 0x7f9ba8071070 2023-06-22T08:07:08.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.775+0000 7f9ba7fff700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9b8c06ad10 0x7f9b8c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:08.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.776+0000 7f9ba7fff700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9b8c06ad10 0x7f9b8c06d1c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9b940090b0 tx=0x7f9b9401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:08.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:08.780+0000 7f9ba57fa700 1 -- 172.21.15.23:0/2904402698 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9b9c053760 con 0x7f9ba8071070 2023-06-22T08:07:09.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.032+0000 7f9baedff700 1 -- 172.21.15.23:0/2904402698 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ba804ea20 con 0x7f9ba8071070 2023-06-22T08:07:09.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.033+0000 7f9ba57fa700 1 -- 172.21.15.23:0/2904402698 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f9b9c017020 con 0x7f9ba8071070 2023-06-22T08:07:09.037 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:09.037 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:51.730709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:09.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 -- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9b8c06ad10 msgr2=0x7f9b8c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:09.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9b8c06ad10 0x7f9b8c06d1c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9b940090b0 tx=0x7f9b9401c040 comp rx=0 tx=0).stop 2023-06-22T08:07:09.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 -- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba8071070 msgr2=0x7f9ba81e3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:09.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba8071070 0x7f9ba81e3c60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00c430 tx=0x7f9b9c00c740 comp rx=0 tx=0).stop 2023-06-22T08:07:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 -- 172.21.15.23:0/2904402698 shutdown_connections 2023-06-22T08:07:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ba814f850 0x7f9ba814b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9b8c06ad10 0x7f9b8c06d1c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ba8071070 0x7f9ba81e3c60 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 --2- 172.21.15.23:0/2904402698 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9ba80c69e0 0x7f9ba814ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 -- 172.21.15.23:0/2904402698 >> 172.21.15.23:0/2904402698 conn(0x7f9ba80ca3f0 msgr2=0x7f9ba80c6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 -- 172.21.15.23:0/2904402698 shutdown_connections 2023-06-22T08:07:09.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:09.036+0000 7f9baedff700 1 -- 172.21.15.23:0/2904402698 wait complete. 2023-06-22T08:07:09.041 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 56 2023-06-22T08:07:09.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:07:11.058 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:11.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.267+0000 7f8e9d9ff700 1 Processor -- start 2023-06-22T08:07:11.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.267+0000 7f8e9d9ff700 1 -- start start 2023-06-22T08:07:11.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.269+0000 7f8e9d9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8e980c4d00 0x7f8e98159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:11.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.269+0000 7f8e9d9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e980c9c30 con 0x7f8e980c97d0 2023-06-22T08:07:11.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.269+0000 7f8e9d9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9815bea0 con 0x7f8e98152d70 2023-06-22T08:07:11.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.269+0000 7f8e9d9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e980c95e0 con 0x7f8e980c4d00 2023-06-22T08:07:11.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.269+0000 7f8e967fc700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f8e980c97d0 0x7f8e980c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52136/0 (socket says 172.21.15.23:52136) 2023-06-22T08:07:11.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.269+0000 7f8e96ffd700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8e98152d70 0x7f8e980cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:52430/0 (socket says 172.21.15.23:52430) 2023-06-22T08:07:11.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.269+0000 7f8e967fc700 1 -- 172.21.15.23:0/3952642409 learned_addr learned my addr 172.21.15.23:0/3952642409 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:11.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.269+0000 7f8e977fe700 1 --2- 172.21.15.23:0/3952642409 >> v2:172.21.15.23:3300/0 conn(0x7f8e980c4d00 0x7f8e98159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:11.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.269+0000 7f8e95ffb700 1 -- 172.21.15.23:0/3952642409 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776408270 0 0) 0x7f8e9815bea0 con 0x7f8e98152d70 2023-06-22T08:07:11.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.269+0000 7f8e95ffb700 1 -- 172.21.15.23:0/3952642409 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e800035b0 con 0x7f8e98152d70 2023-06-22T08:07:11.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.270+0000 7f8e95ffb700 1 -- 172.21.15.23:0/3952642409 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121160971 0 0) 0x7f8e980c9c30 con 0x7f8e980c97d0 2023-06-22T08:07:11.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.270+0000 7f8e95ffb700 1 -- 172.21.15.23:0/3952642409 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9815bea0 con 0x7f8e980c97d0 2023-06-22T08:07:11.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.270+0000 7f8e977fe700 1 -- 172.21.15.23:0/3952642409 >> v1:172.21.15.67:6789/0 conn(0x7f8e98152d70 legacy=0x7f8e980cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.270+0000 7f8e977fe700 1 -- 172.21.15.23:0/3952642409 >> v1:172.21.15.23:6789/0 conn(0x7f8e980c97d0 legacy=0x7f8e980c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.270+0000 7f8e977fe700 1 -- 172.21.15.23:0/3952642409 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9815c160 con 0x7f8e980c4d00 2023-06-22T08:07:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.270+0000 7f8e977fe700 1 --2- 172.21.15.23:0/3952642409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c4d00 0x7f8e98159330 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009d90 tx=0x7f8e9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7958c02f7e3c18d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.270+0000 7f8e95ffb700 1 -- 172.21.15.23:0/3952642409 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9000e070 con 0x7f8e980c4d00 2023-06-22T08:07:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.270+0000 7f8e95ffb700 1 -- 172.21.15.23:0/3952642409 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9000bb20 con 0x7f8e980c4d00 2023-06-22T08:07:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.270+0000 7f8e95ffb700 1 -- 172.21.15.23:0/3952642409 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90012430 con 0x7f8e980c4d00 2023-06-22T08:07:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.271+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/3952642409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c4d00 msgr2=0x7f8e98159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:11.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.271+0000 7f8e9d9ff700 1 --2- 172.21.15.23:0/3952642409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c4d00 0x7f8e98159330 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009d90 tx=0x7f8e9000b330 comp rx=0 tx=0).stop 2023-06-22T08:07:11.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.271+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/3952642409 shutdown_connections 2023-06-22T08:07:11.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.271+0000 7f8e9d9ff700 1 --2- 172.21.15.23:0/3952642409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c4d00 0x7f8e98159330 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:11.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.271+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/3952642409 >> 172.21.15.23:0/3952642409 conn(0x7f8e9806ae30 msgr2=0x7f8e9806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:11.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.271+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/3952642409 shutdown_connections 2023-06-22T08:07:11.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.271+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/3952642409 wait complete. 2023-06-22T08:07:11.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.273+0000 7f8e9d9ff700 1 Processor -- start 2023-06-22T08:07:11.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.274+0000 7f8e9d9ff700 1 -- start start 2023-06-22T08:07:11.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e9d9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980c4d00 0x7f8e9815b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:11.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e9d9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c97d0 0x7f8e981ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:11.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e9d9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e98152d70 0x7f8e981eed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:11.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e9d9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e981ee360 con 0x7f8e980c97d0 2023-06-22T08:07:11.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e9d9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e981f2360 con 0x7f8e980c4d00 2023-06-22T08:07:11.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e9d9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e981f2620 con 0x7f8e98152d70 2023-06-22T08:07:11.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e967fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c97d0 0x7f8e981ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:11.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e967fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c97d0 0x7f8e981ee7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52972/0 (socket says 172.21.15.23:52972) 2023-06-22T08:07:11.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e967fc700 1 -- 172.21.15.23:0/1374641805 learned_addr learned my addr 172.21.15.23:0/1374641805 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:11.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e96ffd700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980c4d00 0x7f8e9815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:11.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e977fe700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e98152d70 0x7f8e981eed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:11.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e967fc700 1 -- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e98152d70 msgr2=0x7f8e981eed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:11.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e967fc700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e98152d70 0x7f8e981eed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:11.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e967fc700 1 -- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980c4d00 msgr2=0x7f8e9815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:11.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e967fc700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980c4d00 0x7f8e9815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:11.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.275+0000 7f8e967fc700 1 -- 172.21.15.23:0/1374641805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e90009a70 con 0x7f8e980c97d0 2023-06-22T08:07:11.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.276+0000 7f8e967fc700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c97d0 0x7f8e981ee7f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00c700 tx=0x7f8e8c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:11.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.276+0000 7f8e7ffff700 1 -- 172.21.15.23:0/1374641805 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c005d70 con 0x7f8e980c97d0 2023-06-22T08:07:11.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.276+0000 7f8e7ffff700 1 -- 172.21.15.23:0/1374641805 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8c0070d0 con 0x7f8e980c97d0 2023-06-22T08:07:11.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.276+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/1374641805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e980b2490 con 0x7f8e980c97d0 2023-06-22T08:07:11.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.276+0000 7f8e7ffff700 1 -- 172.21.15.23:0/1374641805 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c007460 con 0x7f8e980c97d0 2023-06-22T08:07:11.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.276+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/1374641805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e980b2a50 con 0x7f8e980c97d0 2023-06-22T08:07:11.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.277+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/1374641805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e981ee510 con 0x7f8e980c97d0 2023-06-22T08:07:11.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.278+0000 7f8e7ffff700 1 -- 172.21.15.23:0/1374641805 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8e8c0075c0 con 0x7f8e980c97d0 2023-06-22T08:07:11.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.279+0000 7f8e7ffff700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8e8406ace0 0x7f8e8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:11.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.280+0000 7f8e7ffff700 1 -- 172.21.15.23:0/1374641805 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f8e8c088da0 con 0x7f8e980c97d0 2023-06-22T08:07:11.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.280+0000 7f8e96ffd700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8e8406ace0 0x7f8e8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:11.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.281+0000 7f8e96ffd700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8e8406ace0 0x7f8e8406d190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8e88007fd0 tx=0x7f8e88007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:11.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.284+0000 7f8e7ffff700 1 -- 172.21.15.23:0/1374641805 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8e981ee510 con 0x7f8e980c97d0 2023-06-22T08:07:11.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.539+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/1374641805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e9804ea20 con 0x7f8e980c97d0 2023-06-22T08:07:11.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.539+0000 7f8e7ffff700 1 -- 172.21.15.23:0/1374641805 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f8e8c017090 con 0x7f8e980c97d0 2023-06-22T08:07:11.543 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:11.543 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:51.730709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:11.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8e8406ace0 msgr2=0x7f8e8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:11.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8e8406ace0 0x7f8e8406d190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8e88007fd0 tx=0x7f8e88007de0 comp rx=0 tx=0).stop 2023-06-22T08:07:11.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/1374641805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c97d0 msgr2=0x7f8e981ee7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:11.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c97d0 0x7f8e981ee7f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00c700 tx=0x7f8e8c00ca10 comp rx=0 tx=0).stop 2023-06-22T08:07:11.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/1374641805 shutdown_connections 2023-06-22T08:07:11.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e98152d70 0x7f8e981eed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:11.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8e8406ace0 0x7f8e8406d190 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:11.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980c4d00 0x7f8e9815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:11.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 --2- 172.21.15.23:0/1374641805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e980c97d0 0x7f8e981ee7f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:11.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/1374641805 >> 172.21.15.23:0/1374641805 conn(0x7f8e9806ae30 msgr2=0x7f8e980cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:11.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/1374641805 shutdown_connections 2023-06-22T08:07:11.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:11.542+0000 7f8e9d9ff700 1 -- 172.21.15.23:0/1374641805 wait complete. 2023-06-22T08:07:11.547 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 56 2023-06-22T08:07:11.563 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:07:13.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:13.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.773+0000 7f6ce19ff700 1 Processor -- start 2023-06-22T08:07:13.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.773+0000 7f6ce19ff700 1 -- start start 2023-06-22T08:07:13.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.774+0000 7f6ce19ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0c8bd0 0x7f6cdc0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:13.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.774+0000 7f6ce19ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0c7400 con 0x7f6cdc0cc640 2023-06-22T08:07:13.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.774+0000 7f6ce19ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0c81c0 con 0x7f6cdc152dd0 2023-06-22T08:07:13.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.774+0000 7f6ce19ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0c84a0 con 0x7f6cdc0c8bd0 2023-06-22T08:07:13.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cdb7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f6cdc0cc640 0x7f6cdc163280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52142/0 (socket says 172.21.15.23:52142) 2023-06-22T08:07:13.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cdb7fe700 1 -- 172.21.15.23:0/1363577298 learned_addr learned my addr 172.21.15.23:0/1363577298 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:13.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cda7fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0c8bd0 0x7f6cdc0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:13.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cdaffd700 1 --1- 172.21.15.23:0/1363577298 >> v1:172.21.15.67:6789/0 conn(0x7f6cdc152dd0 0x7f6cdc06afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:52444/0 (socket says 172.21.15.23:52444) 2023-06-22T08:07:13.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cd9ffb700 1 -- 172.21.15.23:0/1363577298 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224995936 0 0) 0x7f6cdc0c7400 con 0x7f6cdc0cc640 2023-06-22T08:07:13.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cd9ffb700 1 -- 172.21.15.23:0/1363577298 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc40035b0 con 0x7f6cdc0cc640 2023-06-22T08:07:13.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cd9ffb700 1 -- 172.21.15.23:0/1363577298 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541188673 0 0) 0x7f6cdc0c81c0 con 0x7f6cdc152dd0 2023-06-22T08:07:13.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cd9ffb700 1 -- 172.21.15.23:0/1363577298 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cdc0c7400 con 0x7f6cdc152dd0 2023-06-22T08:07:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cda7fc700 1 -- 172.21.15.23:0/1363577298 >> v1:172.21.15.67:6789/0 conn(0x7f6cdc152dd0 legacy=0x7f6cdc06afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cda7fc700 1 -- 172.21.15.23:0/1363577298 >> v1:172.21.15.23:6789/0 conn(0x7f6cdc0cc640 legacy=0x7f6cdc163280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.775+0000 7f6cda7fc700 1 -- 172.21.15.23:0/1363577298 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0cd370 con 0x7f6cdc0c8bd0 2023-06-22T08:07:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.776+0000 7f6cda7fc700 1 --2- 172.21.15.23:0/1363577298 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0c8bd0 0x7f6cdc0aedc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009870 tx=0x7f6cd0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed1e3be9036951b6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.776+0000 7f6cd9ffb700 1 -- 172.21.15.23:0/1363577298 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0011830 con 0x7f6cdc0c8bd0 2023-06-22T08:07:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.776+0000 7f6cd9ffb700 1 -- 172.21.15.23:0/1363577298 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd0011990 con 0x7f6cdc0c8bd0 2023-06-22T08:07:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.776+0000 7f6cd9ffb700 1 -- 172.21.15.23:0/1363577298 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0011d20 con 0x7f6cdc0c8bd0 2023-06-22T08:07:13.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.776+0000 7f6ce19ff700 1 -- 172.21.15.23:0/1363577298 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0c8bd0 msgr2=0x7f6cdc0aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:13.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.776+0000 7f6ce19ff700 1 --2- 172.21.15.23:0/1363577298 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0c8bd0 0x7f6cdc0aedc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009870 tx=0x7f6cd0009b80 comp rx=0 tx=0).stop 2023-06-22T08:07:13.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.776+0000 7f6ce19ff700 1 -- 172.21.15.23:0/1363577298 shutdown_connections 2023-06-22T08:07:13.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.776+0000 7f6ce19ff700 1 --2- 172.21.15.23:0/1363577298 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0c8bd0 0x7f6cdc0aedc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:13.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.776+0000 7f6ce19ff700 1 -- 172.21.15.23:0/1363577298 >> 172.21.15.23:0/1363577298 conn(0x7f6cdc06fb70 msgr2=0x7f6cdc06ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:13.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.777+0000 7f6ce19ff700 1 -- 172.21.15.23:0/1363577298 shutdown_connections 2023-06-22T08:07:13.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.777+0000 7f6ce19ff700 1 -- 172.21.15.23:0/1363577298 wait complete. 2023-06-22T08:07:13.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.779+0000 7f6ce19ff700 1 Processor -- start 2023-06-22T08:07:13.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.779+0000 7f6ce19ff700 1 -- start start 2023-06-22T08:07:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.780+0000 7f6ce19ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0c8bd0 0x7f6cdc15b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.780+0000 7f6ce19ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0cc640 0x7f6cdc0ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.780+0000 7f6ce19ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cdc152dd0 0x7f6cdc0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.780+0000 7f6ce19ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0b0a00 con 0x7f6cdc152dd0 2023-06-22T08:07:13.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.780+0000 7f6ce19ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0b1580 con 0x7f6cdc0c8bd0 2023-06-22T08:07:13.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.780+0000 7f6ce19ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0af800 con 0x7f6cdc0cc640 2023-06-22T08:07:13.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cdb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cdc152dd0 0x7f6cdc0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:13.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cdaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0c8bd0 0x7f6cdc15b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:13.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cda7fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0cc640 0x7f6cdc0ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:13.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cda7fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0cc640 0x7f6cdc0ae0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35916/0 (socket says 172.21.15.23:35916) 2023-06-22T08:07:13.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cdaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0c8bd0 0x7f6cdc15b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:42078/0 (socket says 172.21.15.23:42078) 2023-06-22T08:07:13.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cda7fc700 1 -- 172.21.15.23:0/175905108 learned_addr learned my addr 172.21.15.23:0/175905108 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:13.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cdb7fe700 1 -- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0cc640 msgr2=0x7f6cdc0ae0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:13.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cdb7fe700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0cc640 0x7f6cdc0ae0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:13.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cdb7fe700 1 -- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0c8bd0 msgr2=0x7f6cdc15b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:13.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cdb7fe700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0c8bd0 0x7f6cdc15b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:13.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cdb7fe700 1 -- 172.21.15.23:0/175905108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd00094a0 con 0x7f6cdc152dd0 2023-06-22T08:07:13.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.781+0000 7f6cdb7fe700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cdc152dd0 0x7f6cdc0ac8e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400caa0 tx=0x7f6cd400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:13.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.782+0000 7f6cc3fff700 1 -- 172.21.15.23:0/175905108 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd400a9a0 con 0x7f6cdc152dd0 2023-06-22T08:07:13.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.782+0000 7f6cc3fff700 1 -- 172.21.15.23:0/175905108 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd400ab00 con 0x7f6cdc152dd0 2023-06-22T08:07:13.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.782+0000 7f6ce19ff700 1 -- 172.21.15.23:0/175905108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc1f1650 con 0x7f6cdc152dd0 2023-06-22T08:07:13.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.782+0000 7f6cc3fff700 1 -- 172.21.15.23:0/175905108 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4012400 con 0x7f6cdc152dd0 2023-06-22T08:07:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.782+0000 7f6ce19ff700 1 -- 172.21.15.23:0/175905108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc0b0f30 con 0x7f6cdc152dd0 2023-06-22T08:07:13.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.783+0000 7f6ce19ff700 1 -- 172.21.15.23:0/175905108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc0742e0 con 0x7f6cdc152dd0 2023-06-22T08:07:13.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.784+0000 7f6cc3fff700 1 -- 172.21.15.23:0/175905108 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6cd40113a0 con 0x7f6cdc152dd0 2023-06-22T08:07:13.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.785+0000 7f6cc3fff700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6cc806ace0 0x7f6cc806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:13.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.785+0000 7f6cc3fff700 1 -- 172.21.15.23:0/175905108 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7f6cd40894f0 con 0x7f6cdc152dd0 2023-06-22T08:07:13.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.785+0000 7f6cdaffd700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6cc806ace0 0x7f6cc806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:13.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.786+0000 7f6cdaffd700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6cc806ace0 0x7f6cc806d190 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc001230 tx=0x7f6ccc007db0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:13.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:13.789+0000 7f6cc3fff700 1 -- 172.21.15.23:0/175905108 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6cd40541d0 con 0x7f6cdc152dd0 2023-06-22T08:07:14.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.045+0000 7f6ce19ff700 1 -- 172.21.15.23:0/175905108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cdc04ea20 con 0x7f6cdc152dd0 2023-06-22T08:07:14.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.045+0000 7f6cc3fff700 1 -- 172.21.15.23:0/175905108 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6255 (secure 0 0 0) 0x7f6cd40577f0 con 0x7f6cdc152dd0 2023-06-22T08:07:14.049 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:14.049 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:51.730709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 -- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6cc806ace0 msgr2=0x7f6cc806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6cc806ace0 0x7f6cc806d190 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc001230 tx=0x7f6ccc007db0 comp rx=0 tx=0).stop 2023-06-22T08:07:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 -- 172.21.15.23:0/175905108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cdc152dd0 msgr2=0x7f6cdc0ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:14.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cdc152dd0 0x7f6cdc0ac8e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400caa0 tx=0x7f6cd400cdb0 comp rx=0 tx=0).stop 2023-06-22T08:07:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 -- 172.21.15.23:0/175905108 shutdown_connections 2023-06-22T08:07:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6cdc152dd0 0x7f6cdc0ac8e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6cc806ace0 0x7f6cc806d190 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cdc0c8bd0 0x7f6cdc15b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 --2- 172.21.15.23:0/175905108 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6cdc0cc640 0x7f6cdc0ae0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 -- 172.21.15.23:0/175905108 >> 172.21.15.23:0/175905108 conn(0x7f6cdc06fb70 msgr2=0x7f6cdc152440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:14.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 -- 172.21.15.23:0/175905108 shutdown_connections 2023-06-22T08:07:14.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:14.048+0000 7f6ce19ff700 1 -- 172.21.15.23:0/175905108 wait complete. 2023-06-22T08:07:14.054 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 56 2023-06-22T08:07:14.070 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:07:16.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:16.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.278+0000 7fef4efff700 1 Processor -- start 2023-06-22T08:07:16.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.278+0000 7fef4efff700 1 -- start start 2023-06-22T08:07:16.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.279+0000 7fef4efff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480cb6b0 0x7fef48069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:16.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4efff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fef48152e10 0x7fef481593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:16.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4efff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef48158f60 con 0x7fef480c5740 2023-06-22T08:07:16.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4efff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef4806bd20 con 0x7fef48152e10 2023-06-22T08:07:16.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4efff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef480cc450 con 0x7fef480cb6b0 2023-06-22T08:07:16.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480cb6b0 0x7fef48069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:16.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fef48152e10 0x7fef481593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:16.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef47fff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fef480c5740 0x7fef48152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:55648/0 (socket says 172.21.15.23:55648) 2023-06-22T08:07:16.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef47fff700 1 -- 172.21.15.23:0/4208931487 learned_addr learned my addr 172.21.15.23:0/4208931487 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:16.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef477fe700 1 -- 172.21.15.23:0/4208931487 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22229335 0 0) 0x7fef48158f60 con 0x7fef480c5740 2023-06-22T08:07:16.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef477fe700 1 -- 172.21.15.23:0/4208931487 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef380035b0 con 0x7fef480c5740 2023-06-22T08:07:16.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4d59c700 1 -- 172.21.15.23:0/4208931487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480cb6b0 msgr2=0x7fef48069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:16.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4d59c700 1 --2- 172.21.15.23:0/4208931487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480cb6b0 0x7fef48069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:16.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4d59c700 1 -- 172.21.15.23:0/4208931487 >> v1:172.21.15.67:6789/0 conn(0x7fef480c5740 legacy=0x7fef48152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:16.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4d59c700 1 -- 172.21.15.23:0/4208931487 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef48158980 con 0x7fef48152e10 2023-06-22T08:07:16.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.280+0000 7fef4cd9b700 1 --2- 172.21.15.23:0/4208931487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480cb6b0 0x7fef48069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:16.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.281+0000 7fef4d59c700 1 --2- 172.21.15.23:0/4208931487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef48152e10 0x7fef481593f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fef40009d90 tx=0x7fef4000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7506f725647d61a0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:16.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.281+0000 7fef477fe700 1 -- 172.21.15.23:0/4208931487 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4000e070 con 0x7fef48152e10 2023-06-22T08:07:16.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.281+0000 7fef477fe700 1 -- 172.21.15.23:0/4208931487 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4000bb20 con 0x7fef48152e10 2023-06-22T08:07:16.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.281+0000 7fef477fe700 1 -- 172.21.15.23:0/4208931487 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40012430 con 0x7fef48152e10 2023-06-22T08:07:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.281+0000 7fef4efff700 1 -- 172.21.15.23:0/4208931487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef48152e10 msgr2=0x7fef481593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.281+0000 7fef4efff700 1 --2- 172.21.15.23:0/4208931487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef48152e10 0x7fef481593f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fef40009d90 tx=0x7fef4000b330 comp rx=0 tx=0).stop 2023-06-22T08:07:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.282+0000 7fef4efff700 1 -- 172.21.15.23:0/4208931487 shutdown_connections 2023-06-22T08:07:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.282+0000 7fef4efff700 1 --2- 172.21.15.23:0/4208931487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef48152e10 0x7fef481593f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.282+0000 7fef4efff700 1 --2- 172.21.15.23:0/4208931487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480cb6b0 0x7fef48069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.282+0000 7fef4efff700 1 -- 172.21.15.23:0/4208931487 >> 172.21.15.23:0/4208931487 conn(0x7fef48070d80 msgr2=0x7fef48071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.282+0000 7fef4efff700 1 -- 172.21.15.23:0/4208931487 shutdown_connections 2023-06-22T08:07:16.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.282+0000 7fef4efff700 1 -- 172.21.15.23:0/4208931487 wait complete. 2023-06-22T08:07:16.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.283+0000 7fef4efff700 1 Processor -- start 2023-06-22T08:07:16.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.284+0000 7fef4efff700 1 -- start start 2023-06-22T08:07:16.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef4efff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480cb6b0 0x7fef480c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:16.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef4efff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef48152e10 0x7fef4815afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:16.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef4efff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480c6b80 0x7fef481ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:16.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef4efff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef480c4b80 con 0x7fef48152e10 2023-06-22T08:07:16.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef4efff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef480c30c0 con 0x7fef480cb6b0 2023-06-22T08:07:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef4efff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef481ede30 con 0x7fef480c6b80 2023-06-22T08:07:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef47fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef48152e10 0x7fef4815afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef47fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef48152e10 0x7fef4815afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60078/0 (socket says 172.21.15.23:60078) 2023-06-22T08:07:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef47fff700 1 -- 172.21.15.23:0/2189996878 learned_addr learned my addr 172.21.15.23:0/2189996878 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef4cd9b700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480cb6b0 0x7fef480c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef4d59c700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480c6b80 0x7fef481ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef47fff700 1 -- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480c6b80 msgr2=0x7fef481ee5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef47fff700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480c6b80 0x7fef481ee5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef47fff700 1 -- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480cb6b0 msgr2=0x7fef480c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:16.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef47fff700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480cb6b0 0x7fef480c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:16.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.285+0000 7fef47fff700 1 -- 172.21.15.23:0/2189996878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c0094a0 con 0x7fef48152e10 2023-06-22T08:07:16.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.286+0000 7fef47fff700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef48152e10 0x7fef4815afd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fef3400c700 tx=0x7fef3400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:16.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.286+0000 7fef457fa700 1 -- 172.21.15.23:0/2189996878 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef34005d70 con 0x7fef48152e10 2023-06-22T08:07:16.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.286+0000 7fef457fa700 1 -- 172.21.15.23:0/2189996878 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef340070d0 con 0x7fef48152e10 2023-06-22T08:07:16.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.286+0000 7fef457fa700 1 -- 172.21.15.23:0/2189996878 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef34007460 con 0x7fef48152e10 2023-06-22T08:07:16.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.286+0000 7fef4efff700 1 -- 172.21.15.23:0/2189996878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef40009a70 con 0x7fef48152e10 2023-06-22T08:07:16.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.286+0000 7fef4efff700 1 -- 172.21.15.23:0/2189996878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef480c3640 con 0x7fef48152e10 2023-06-22T08:07:16.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.287+0000 7fef4efff700 1 -- 172.21.15.23:0/2189996878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4804ea20 con 0x7fef48152e10 2023-06-22T08:07:16.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.288+0000 7fef457fa700 1 -- 172.21.15.23:0/2189996878 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fef340075c0 con 0x7fef48152e10 2023-06-22T08:07:16.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.289+0000 7fef457fa700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef2c06ace0 0x7fef2c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:16.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.289+0000 7fef457fa700 1 -- 172.21.15.23:0/2189996878 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7fef34088ed0 con 0x7fef48152e10 2023-06-22T08:07:16.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.289+0000 7fef4cd9b700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef2c06ace0 0x7fef2c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:16.297 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.290+0000 7fef4cd9b700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef2c06ace0 0x7fef2c06d190 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009d70 tx=0x7fef3c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:16.297 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.293+0000 7fef457fa700 1 -- 172.21.15.23:0/2189996878 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fef34053af0 con 0x7fef48152e10 2023-06-22T08:07:16.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.546+0000 7fef4efff700 1 -- 172.21.15.23:0/2189996878 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef480c6960 con 0x7fef48152e10 2023-06-22T08:07:16.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.546+0000 7fef457fa700 1 -- 172.21.15.23:0/2189996878 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6255 (secure 0 0 0) 0x7fef34057110 con 0x7fef48152e10 2023-06-22T08:07:16.550 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:16.550 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:51.730709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:16.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 -- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef2c06ace0 msgr2=0x7fef2c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:16.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef2c06ace0 0x7fef2c06d190 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009d70 tx=0x7fef3c009210 comp rx=0 tx=0).stop 2023-06-22T08:07:16.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 -- 172.21.15.23:0/2189996878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef48152e10 msgr2=0x7fef4815afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:16.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef48152e10 0x7fef4815afd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fef3400c700 tx=0x7fef3400ca10 comp rx=0 tx=0).stop 2023-06-22T08:07:16.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 -- 172.21.15.23:0/2189996878 shutdown_connections 2023-06-22T08:07:16.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef480c6b80 0x7fef481ee5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:16.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fef2c06ace0 0x7fef2c06d190 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:16.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef480cb6b0 0x7fef480c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:16.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 --2- 172.21.15.23:0/2189996878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef48152e10 0x7fef4815afd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:16.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 -- 172.21.15.23:0/2189996878 >> 172.21.15.23:0/2189996878 conn(0x7fef48070d80 msgr2=0x7fef4814f490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:16.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 -- 172.21.15.23:0/2189996878 shutdown_connections 2023-06-22T08:07:16.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:16.549+0000 7fef4efff700 1 -- 172.21.15.23:0/2189996878 wait complete. 2023-06-22T08:07:16.554 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 56 2023-06-22T08:07:16.570 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:07:18.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:18.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.778+0000 7fd3239ff700 1 Processor -- start 2023-06-22T08:07:18.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.779+0000 7fd3239ff700 1 -- start start 2023-06-22T08:07:18.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.780+0000 7fd3239ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd31c152d70 0x7fd31c0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:18.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.780+0000 7fd3239ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd31c159330 0x7fd31c0cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:18.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.780+0000 7fd3239ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31c159790 con 0x7fd31c0cb800 2023-06-22T08:07:18.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.780+0000 7fd3239ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31c0c95e0 con 0x7fd31c152d70 2023-06-22T08:07:18.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.780+0000 7fd3239ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31c15bea0 con 0x7fd31c159330 2023-06-22T08:07:18.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.780+0000 7fd321f9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fd31c0cb800 0x7fd31c0c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:43034/0 (socket says 172.21.15.23:43034) 2023-06-22T08:07:18.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.780+0000 7fd321f9c700 1 -- 172.21.15.23:0/424337385 learned_addr learned my addr 172.21.15.23:0/424337385 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:18.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.780+0000 7fd32179b700 1 --2- 172.21.15.23:0/424337385 >> v2:172.21.15.23:3300/0 conn(0x7fd31c152d70 0x7fd31c0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:18.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.780+0000 7fd320f9a700 1 --2- 172.21.15.23:0/424337385 >> v2:172.21.15.67:3300/0 conn(0x7fd31c159330 0x7fd31c0cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:18.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.781+0000 7fd32179b700 1 -- 172.21.15.23:0/424337385 >> v2:172.21.15.67:3300/0 conn(0x7fd31c159330 msgr2=0x7fd31c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:18.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.781+0000 7fd32179b700 1 --2- 172.21.15.23:0/424337385 >> v2:172.21.15.67:3300/0 conn(0x7fd31c159330 0x7fd31c0cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:18.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.781+0000 7fd32179b700 1 -- 172.21.15.23:0/424337385 >> v1:172.21.15.23:6789/0 conn(0x7fd31c0cb800 legacy=0x7fd31c0c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:18.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.781+0000 7fd32179b700 1 -- 172.21.15.23:0/424337385 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c15c160 con 0x7fd31c152d70 2023-06-22T08:07:18.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.781+0000 7fd313fff700 1 -- 172.21.15.23:0/424337385 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386014070 0 0) 0x7fd31c159790 con 0x7fd31c0cb800 2023-06-22T08:07:18.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.781+0000 7fd320f9a700 1 --2- 172.21.15.23:0/424337385 >> v2:172.21.15.67:3300/0 conn(0x7fd31c159330 0x7fd31c0cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:18.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.781+0000 7fd32179b700 1 --2- 172.21.15.23:0/424337385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 0x7fd31c0c97d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd31c08ba40 tx=0x7fd318009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce92345f071324d7 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:18.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.781+0000 7fd313fff700 1 -- 172.21.15.23:0/424337385 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318011830 con 0x7fd31c152d70 2023-06-22T08:07:18.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.781+0000 7fd313fff700 1 -- 172.21.15.23:0/424337385 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd318011990 con 0x7fd31c152d70 2023-06-22T08:07:18.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.781+0000 7fd313fff700 1 -- 172.21.15.23:0/424337385 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318011d20 con 0x7fd31c152d70 2023-06-22T08:07:18.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.782+0000 7fd3239ff700 1 -- 172.21.15.23:0/424337385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 msgr2=0x7fd31c0c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:18.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.782+0000 7fd3239ff700 1 --2- 172.21.15.23:0/424337385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 0x7fd31c0c97d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd31c08ba40 tx=0x7fd318009b20 comp rx=0 tx=0).stop 2023-06-22T08:07:18.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.782+0000 7fd3239ff700 1 -- 172.21.15.23:0/424337385 shutdown_connections 2023-06-22T08:07:18.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.782+0000 7fd3239ff700 1 --2- 172.21.15.23:0/424337385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 0x7fd31c0c97d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:18.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.782+0000 7fd3239ff700 1 --2- 172.21.15.23:0/424337385 >> v2:172.21.15.67:3300/0 conn(0x7fd31c159330 0x7fd31c0cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:18.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.782+0000 7fd3239ff700 1 -- 172.21.15.23:0/424337385 >> 172.21.15.23:0/424337385 conn(0x7fd31c06ae30 msgr2=0x7fd31c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:18.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.782+0000 7fd3239ff700 1 -- 172.21.15.23:0/424337385 shutdown_connections 2023-06-22T08:07:18.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.782+0000 7fd3239ff700 1 -- 172.21.15.23:0/424337385 wait complete. 2023-06-22T08:07:18.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.784+0000 7fd3239ff700 1 Processor -- start 2023-06-22T08:07:18.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.784+0000 7fd3239ff700 1 -- start start 2023-06-22T08:07:18.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd3239ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd31c0cb800 0x7fd31c0b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:18.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd3239ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 0x7fd31c0c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:18.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd3239ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c159330 0x7fd31c14f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:18.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd3239ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c1e2000 con 0x7fd31c152d70 2023-06-22T08:07:18.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd3239ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0b3ff0 con 0x7fd31c159330 2023-06-22T08:07:18.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd3239ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0c6270 con 0x7fd31c0cb800 2023-06-22T08:07:18.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd320f9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 0x7fd31c0c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:18.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd320f9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 0x7fd31c0c7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60098/0 (socket says 172.21.15.23:60098) 2023-06-22T08:07:18.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd320f9a700 1 -- 172.21.15.23:0/4152907713 learned_addr learned my addr 172.21.15.23:0/4152907713 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:18.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd32179b700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd31c0cb800 0x7fd31c0b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:18.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd321f9c700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c159330 0x7fd31c14f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:18.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd320f9a700 1 -- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd31c0cb800 msgr2=0x7fd31c0b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:18.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd320f9a700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd31c0cb800 0x7fd31c0b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:18.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd320f9a700 1 -- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c159330 msgr2=0x7fd31c14f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:18.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd320f9a700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c159330 0x7fd31c14f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:18.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.785+0000 7fd320f9a700 1 -- 172.21.15.23:0/4152907713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c0094a0 con 0x7fd31c152d70 2023-06-22T08:07:18.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.786+0000 7fd320f9a700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 0x7fd31c0c7290 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00ec10 tx=0x7fd30c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:18.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.786+0000 7fd311ffb700 1 -- 172.21.15.23:0/4152907713 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c009760 con 0x7fd31c152d70 2023-06-22T08:07:18.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.786+0000 7fd311ffb700 1 -- 172.21.15.23:0/4152907713 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c00c850 con 0x7fd31c152d70 2023-06-22T08:07:18.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.786+0000 7fd3239ff700 1 -- 172.21.15.23:0/4152907713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3180094a0 con 0x7fd31c152d70 2023-06-22T08:07:18.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.786+0000 7fd311ffb700 1 -- 172.21.15.23:0/4152907713 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c00cbe0 con 0x7fd31c152d70 2023-06-22T08:07:18.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.786+0000 7fd3239ff700 1 -- 172.21.15.23:0/4152907713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31c0c5940 con 0x7fd31c152d70 2023-06-22T08:07:18.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.787+0000 7fd3239ff700 1 -- 172.21.15.23:0/4152907713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31c08c1f0 con 0x7fd31c152d70 2023-06-22T08:07:18.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.788+0000 7fd311ffb700 1 -- 172.21.15.23:0/4152907713 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd30c00cd40 con 0x7fd31c152d70 2023-06-22T08:07:18.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.789+0000 7fd311ffb700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd30006ace0 0x7fd30006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:18.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.789+0000 7fd311ffb700 1 -- 172.21.15.23:0/4152907713 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 7071+0+0 (secure 0 0 0) 0x7fd30c089910 con 0x7fd31c152d70 2023-06-22T08:07:18.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.789+0000 7fd32179b700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd30006ace0 0x7fd30006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:18.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.790+0000 7fd32179b700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd30006ace0 0x7fd30006d190 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd3180090b0 tx=0x7fd318008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:18.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:18.793+0000 7fd311ffb700 1 -- 172.21.15.23:0/4152907713 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd30c054530 con 0x7fd31c152d70 2023-06-22T08:07:19.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.046+0000 7fd3239ff700 1 -- 172.21.15.23:0/4152907713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd31c04ea20 con 0x7fd31c152d70 2023-06-22T08:07:19.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.047+0000 7fd311ffb700 1 -- 172.21.15.23:0/4152907713 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6255 (secure 0 0 0) 0x7fd30c057b50 con 0x7fd31c152d70 2023-06-22T08:07:19.050 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:19.050 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:06:51.730709+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":4272,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":12,"addr":"172.21.15.23:6835/3894174037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3894174037},{"type":"v1","addr":"172.21.15.23:6835","nonce":3894174037}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:19.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 -- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd30006ace0 msgr2=0x7fd30006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd30006ace0 0x7fd30006d190 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd3180090b0 tx=0x7fd318008030 comp rx=0 tx=0).stop 2023-06-22T08:07:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 -- 172.21.15.23:0/4152907713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 msgr2=0x7fd31c0c7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 0x7fd31c0c7290 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00ec10 tx=0x7fd30c00ef20 comp rx=0 tx=0).stop 2023-06-22T08:07:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 -- 172.21.15.23:0/4152907713 shutdown_connections 2023-06-22T08:07:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd30006ace0 0x7fd30006d190 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd31c159330 0x7fd31c14f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:19.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd31c0cb800 0x7fd31c0b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:19.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 --2- 172.21.15.23:0/4152907713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd31c152d70 0x7fd31c0c7290 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:19.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 -- 172.21.15.23:0/4152907713 >> 172.21.15.23:0/4152907713 conn(0x7fd31c06ae30 msgr2=0x7fd31c159c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:19.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 -- 172.21.15.23:0/4152907713 shutdown_connections 2023-06-22T08:07:19.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:19.049+0000 7fd3239ff700 1 -- 172.21.15.23:0/4152907713 wait complete. 2023-06-22T08:07:19.054 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 56 2023-06-22T08:07:19.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:07:21.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.279+0000 7f74da7ff700 1 Processor -- start 2023-06-22T08:07:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.279+0000 7f74da7ff700 1 -- start start 2023-06-22T08:07:21.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74da7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d414f850 0x7f74d40c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:21.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74da7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74d4071070 0x7f74d40c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:21.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74da7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d40c51a0 con 0x7f74d40c69e0 2023-06-22T08:07:21.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74da7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d40c44e0 con 0x7f74d4071070 2023-06-22T08:07:21.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74da7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40c5440 con 0x7f74d414f850 2023-06-22T08:07:21.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74d37fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f74d40c69e0 0x7f74d4155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:43044/0 (socket says 172.21.15.23:43044) 2023-06-22T08:07:21.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74d37fe700 1 -- 172.21.15.23:0/1349781105 learned_addr learned my addr 172.21.15.23:0/1349781105 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:21.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74d3fff700 1 --2- 172.21.15.23:0/1349781105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d414f850 0x7f74d40c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:21.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74d8d9c700 1 --2- 172.21.15.23:0/1349781105 >> v2:172.21.15.67:3300/0 conn(0x7f74d4071070 0x7f74d40c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:21.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74d2ffd700 1 -- 172.21.15.23:0/1349781105 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620967858 0 0) 0x7f74d40c51a0 con 0x7f74d40c69e0 2023-06-22T08:07:21.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.281+0000 7f74d2ffd700 1 -- 172.21.15.23:0/1349781105 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74bc0057c0 con 0x7f74d40c69e0 2023-06-22T08:07:21.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.282+0000 7f74d3fff700 1 -- 172.21.15.23:0/1349781105 >> v2:172.21.15.67:3300/0 conn(0x7f74d4071070 msgr2=0x7f74d40c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:21.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.282+0000 7f74d3fff700 1 --2- 172.21.15.23:0/1349781105 >> v2:172.21.15.67:3300/0 conn(0x7f74d4071070 0x7f74d40c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:21.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.282+0000 7f74d3fff700 1 -- 172.21.15.23:0/1349781105 >> v1:172.21.15.23:6789/0 conn(0x7f74d40c69e0 legacy=0x7f74d4155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:21.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.282+0000 7f74d3fff700 1 -- 172.21.15.23:0/1349781105 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d40c5720 con 0x7f74d414f850 2023-06-22T08:07:21.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.282+0000 7f74d8d9c700 1 --2- 172.21.15.23:0/1349781105 >> v2:172.21.15.67:3300/0 conn(0x7f74d4071070 0x7f74d40c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:21.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.282+0000 7f74d3fff700 1 --2- 172.21.15.23:0/1349781105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d414f850 0x7f74d40c3080 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009840 tx=0x7f74c4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69d32a92a5d4faa4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:21.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.282+0000 7f74d2ffd700 1 -- 172.21.15.23:0/1349781105 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4011830 con 0x7f74d414f850 2023-06-22T08:07:21.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.282+0000 7f74d2ffd700 1 -- 172.21.15.23:0/1349781105 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4011990 con 0x7f74d414f850 2023-06-22T08:07:21.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.282+0000 7f74d2ffd700 1 -- 172.21.15.23:0/1349781105 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4011d00 con 0x7f74d414f850 2023-06-22T08:07:21.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.283+0000 7f74da7ff700 1 -- 172.21.15.23:0/1349781105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d414f850 msgr2=0x7f74d40c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:21.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.283+0000 7f74da7ff700 1 --2- 172.21.15.23:0/1349781105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d414f850 0x7f74d40c3080 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009840 tx=0x7f74c4009b50 comp rx=0 tx=0).stop 2023-06-22T08:07:21.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.283+0000 7f74da7ff700 1 -- 172.21.15.23:0/1349781105 shutdown_connections 2023-06-22T08:07:21.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.283+0000 7f74da7ff700 1 --2- 172.21.15.23:0/1349781105 >> v2:172.21.15.67:3300/0 conn(0x7f74d4071070 0x7f74d40c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:21.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.283+0000 7f74da7ff700 1 --2- 172.21.15.23:0/1349781105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d414f850 0x7f74d40c3080 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:21.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.283+0000 7f74da7ff700 1 -- 172.21.15.23:0/1349781105 >> 172.21.15.23:0/1349781105 conn(0x7f74d40ca3f0 msgr2=0x7f74d40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:21.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.283+0000 7f74da7ff700 1 -- 172.21.15.23:0/1349781105 shutdown_connections 2023-06-22T08:07:21.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.283+0000 7f74da7ff700 1 -- 172.21.15.23:0/1349781105 wait complete. 2023-06-22T08:07:21.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.285+0000 7f74da7ff700 1 Processor -- start 2023-06-22T08:07:21.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.285+0000 7f74da7ff700 1 -- start start 2023-06-22T08:07:21.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74da7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4071070 0x7f74d414cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:21.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74da7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d40c69e0 0x7f74d4155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:21.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74da7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74d414f850 0x7f74d41edf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:21.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74da7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d406a8f0 con 0x7f74d414f850 2023-06-22T08:07:21.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74da7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d41de250 con 0x7f74d4071070 2023-06-22T08:07:21.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74da7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d406a610 con 0x7f74d40c69e0 2023-06-22T08:07:21.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d8d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74d414f850 0x7f74d41edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:21.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d8d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74d414f850 0x7f74d41edf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60114/0 (socket says 172.21.15.23:60114) 2023-06-22T08:07:21.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d8d9c700 1 -- 172.21.15.23:0/4157195657 learned_addr learned my addr 172.21.15.23:0/4157195657 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:21.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d3fff700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4071070 0x7f74d414cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:21.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d37fe700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d40c69e0 0x7f74d4155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:21.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d8d9c700 1 -- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d40c69e0 msgr2=0x7f74d4155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:21.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d8d9c700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d40c69e0 0x7f74d4155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:21.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d8d9c700 1 -- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4071070 msgr2=0x7f74d414cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:21.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d8d9c700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4071070 0x7f74d414cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:21.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d8d9c700 1 -- 172.21.15.23:0/4157195657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc009a70 con 0x7f74d414f850 2023-06-22T08:07:21.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.286+0000 7f74d37fe700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d40c69e0 0x7f74d4155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:07:21.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.287+0000 7f74d8d9c700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74d414f850 0x7f74d41edf80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009f00 tx=0x7f74cc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:21.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.287+0000 7f74d0ff9700 1 -- 172.21.15.23:0/4157195657 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc009d70 con 0x7f74d414f850 2023-06-22T08:07:21.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.287+0000 7f74d0ff9700 1 -- 172.21.15.23:0/4157195657 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74cc00f040 con 0x7f74d414f850 2023-06-22T08:07:21.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.287+0000 7f74d0ff9700 1 -- 172.21.15.23:0/4157195657 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc00ce20 con 0x7f74d414f850 2023-06-22T08:07:21.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.287+0000 7f74da7ff700 1 -- 172.21.15.23:0/4157195657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c40094a0 con 0x7f74d414f850 2023-06-22T08:07:21.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.287+0000 7f74da7ff700 1 -- 172.21.15.23:0/4157195657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d41ddc00 con 0x7f74d414f850 2023-06-22T08:07:21.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.288+0000 7f74da7ff700 1 -- 172.21.15.23:0/4157195657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d40c2950 con 0x7f74d414f850 2023-06-22T08:07:21.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.289+0000 7f74d0ff9700 1 -- 172.21.15.23:0/4157195657 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f74cc014480 con 0x7f74d414f850 2023-06-22T08:07:21.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.290+0000 7f74d0ff9700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f74c006ad10 0x7f74c006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:21.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.290+0000 7f74d0ff9700 1 -- 172.21.15.23:0/4157195657 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f74cc08a100 con 0x7f74d414f850 2023-06-22T08:07:21.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.290+0000 7f74d3fff700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f74c006ad10 0x7f74c006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:21.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.291+0000 7f74d3fff700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f74c006ad10 0x7f74c006d1c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009fd0 tx=0x7f74c4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:21.297 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.294+0000 7f74d0ff9700 1 -- 172.21.15.23:0/4157195657 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f74cc054c40 con 0x7f74d414f850 2023-06-22T08:07:21.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.549+0000 7f74da7ff700 1 -- 172.21.15.23:0/4157195657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74d41de5e0 con 0x7f74d414f850 2023-06-22T08:07:21.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.550+0000 7f74d0ff9700 1 -- 172.21.15.23:0/4157195657 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5498 (secure 0 0 0) 0x7f74d41de5e0 con 0x7f74d414f850 2023-06-22T08:07:21.553 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:21.553 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:19.358304+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:21.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 -- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f74c006ad10 msgr2=0x7f74c006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:21.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f74c006ad10 0x7f74c006d1c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f74c4009fd0 tx=0x7f74c4008030 comp rx=0 tx=0).stop 2023-06-22T08:07:21.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 -- 172.21.15.23:0/4157195657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74d414f850 msgr2=0x7f74d41edf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:21.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74d414f850 0x7f74d41edf80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009f00 tx=0x7f74cc00c3a0 comp rx=0 tx=0).stop 2023-06-22T08:07:21.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 -- 172.21.15.23:0/4157195657 shutdown_connections 2023-06-22T08:07:21.556 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74d414f850 0x7f74d41edf80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:21.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f74c006ad10 0x7f74c006d1c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:21.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74d4071070 0x7f74d414cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:21.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 --2- 172.21.15.23:0/4157195657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f74d40c69e0 0x7f74d4155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:21.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 -- 172.21.15.23:0/4157195657 >> 172.21.15.23:0/4157195657 conn(0x7f74d40ca3f0 msgr2=0x7f74d414d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:21.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 -- 172.21.15.23:0/4157195657 shutdown_connections 2023-06-22T08:07:21.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.552+0000 7f74da7ff700 1 -- 172.21.15.23:0/4157195657 wait complete. 2023-06-22T08:07:21.557 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 57 2023-06-22T08:07:21.573 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-06-22T08:07:21.574 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:07:21.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:23.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.773+0000 7f916b7ff700 1 Processor -- start 2023-06-22T08:07:23.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.773+0000 7f916b7ff700 1 -- start start 2023-06-22T08:07:23.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.774+0000 7f916b7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9164166380 0x7f916406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:23.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.774+0000 7f916b7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f916406dc40 0x7f916406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:23.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.774+0000 7f916b7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91640c6b20 con 0x7f9164069eb0 2023-06-22T08:07:23.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.774+0000 7f916b7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916406d050 con 0x7f9164166380 2023-06-22T08:07:23.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.774+0000 7f916b7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9164071420 con 0x7f916406dc40 2023-06-22T08:07:23.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.775+0000 7f916959b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9164166380 0x7f916406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:23.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.775+0000 7f9168d9a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9164069eb0 0x7f916406d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:55658/0 (socket says 172.21.15.23:55658) 2023-06-22T08:07:23.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.775+0000 7f9169d9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f916406dc40 0x7f916406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:23.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.775+0000 7f9168d9a700 1 -- 172.21.15.23:0/511709819 learned_addr learned my addr 172.21.15.23:0/511709819 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:23.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:21.775+0000 7f915bfff700 1 -- 172.21.15.23:0/511709819 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697260911 0 0) 0x7f91640c6b20 con 0x7f9164069eb0 2023-06-22T08:07:23.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.364+0000 7f915bfff700 1 -- 172.21.15.23:0/511709819 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9150003590 con 0x7f9164069eb0 2023-06-22T08:07:23.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.365+0000 7f916959b700 1 -- 172.21.15.23:0/511709819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f916406dc40 msgr2=0x7f916406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:23.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.365+0000 7f916959b700 1 --2- 172.21.15.23:0/511709819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f916406dc40 0x7f916406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:23.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.365+0000 7f916959b700 1 -- 172.21.15.23:0/511709819 >> v1:172.21.15.67:6789/0 conn(0x7f9164069eb0 legacy=0x7f916406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:23.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.365+0000 7f916959b700 1 -- 172.21.15.23:0/511709819 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9164071700 con 0x7f9164166380 2023-06-22T08:07:23.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.365+0000 7f916959b700 1 --2- 172.21.15.23:0/511709819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9164166380 0x7f916406fcb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f916000dc60 tx=0x7f916000df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ffe9f8f8b413219 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:23.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.366+0000 7f915bfff700 1 -- 172.21.15.23:0/511709819 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160012070 con 0x7f9164166380 2023-06-22T08:07:23.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.366+0000 7f915bfff700 1 -- 172.21.15.23:0/511709819 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916000d2e0 con 0x7f9164166380 2023-06-22T08:07:23.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.366+0000 7f915bfff700 1 -- 172.21.15.23:0/511709819 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000f900 con 0x7f9164166380 2023-06-22T08:07:23.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.367+0000 7f916b7ff700 1 -- 172.21.15.23:0/511709819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9164166380 msgr2=0x7f916406fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:23.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.367+0000 7f916b7ff700 1 --2- 172.21.15.23:0/511709819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9164166380 0x7f916406fcb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f916000dc60 tx=0x7f916000df70 comp rx=0 tx=0).stop 2023-06-22T08:07:23.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.367+0000 7f916b7ff700 1 -- 172.21.15.23:0/511709819 shutdown_connections 2023-06-22T08:07:23.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.367+0000 7f916b7ff700 1 --2- 172.21.15.23:0/511709819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f916406dc40 0x7f916406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:23.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.367+0000 7f916b7ff700 1 --2- 172.21.15.23:0/511709819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9164166380 0x7f916406fcb0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:23.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.367+0000 7f916b7ff700 1 -- 172.21.15.23:0/511709819 >> 172.21.15.23:0/511709819 conn(0x7f91640ca3f0 msgr2=0x7f91640ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:23.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.367+0000 7f916b7ff700 1 -- 172.21.15.23:0/511709819 shutdown_connections 2023-06-22T08:07:23.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.367+0000 7f916b7ff700 1 -- 172.21.15.23:0/511709819 wait complete. 2023-06-22T08:07:23.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.369+0000 7f916b7ff700 1 Processor -- start 2023-06-22T08:07:23.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.370+0000 7f916b7ff700 1 -- start start 2023-06-22T08:07:23.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f916b7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9164069eb0 0x7f91641ff480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:23.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f916b7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f916406dc40 0x7f91641ecc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:23.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f916b7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9164166380 0x7f91641ed1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:23.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f916b7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91640b7480 con 0x7f916406dc40 2023-06-22T08:07:23.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f916b7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91641ff9c0 con 0x7f9164166380 2023-06-22T08:07:23.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f916b7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91641ec7e0 con 0x7f9164069eb0 2023-06-22T08:07:23.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f9168d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f916406dc40 0x7f91641ecc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:23.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f9169d9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9164166380 0x7f91641ed1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:23.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f916959b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9164069eb0 0x7f91641ff480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:23.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f9168d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f916406dc40 0x7f91641ecc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60134/0 (socket says 172.21.15.23:60134) 2023-06-22T08:07:23.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f9169d9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9164166380 0x7f91641ed1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:35756/0 (socket says 172.21.15.23:35756) 2023-06-22T08:07:23.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.371+0000 7f9168d9a700 1 -- 172.21.15.23:0/3508564328 learned_addr learned my addr 172.21.15.23:0/3508564328 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:23.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.372+0000 7f916959b700 1 -- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9164166380 msgr2=0x7f91641ed1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:23.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.372+0000 7f916959b700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9164166380 0x7f91641ed1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:23.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.372+0000 7f916959b700 1 -- 172.21.15.23:0/3508564328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f916406dc40 msgr2=0x7f91641ecc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:23.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.372+0000 7f916959b700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f916406dc40 0x7f91641ecc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:23.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.372+0000 7f916959b700 1 -- 172.21.15.23:0/3508564328 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916000d940 con 0x7f9164069eb0 2023-06-22T08:07:23.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.372+0000 7f9169d9c700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9164166380 0x7f91641ed1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:23.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.372+0000 7f9168d9a700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f916406dc40 0x7f91641ecc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:23.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.372+0000 7f916959b700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9164069eb0 0x7f91641ff480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f916406f6c0 tx=0x7f9160004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:23.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.372+0000 7f9159ffb700 1 -- 172.21.15.23:0/3508564328 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000d7c0 con 0x7f9164069eb0 2023-06-22T08:07:23.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.372+0000 7f9159ffb700 1 -- 172.21.15.23:0/3508564328 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91600036a0 con 0x7f9164069eb0 2023-06-22T08:07:23.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.373+0000 7f9159ffb700 1 -- 172.21.15.23:0/3508564328 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160003a50 con 0x7f9164069eb0 2023-06-22T08:07:23.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.373+0000 7f916b7ff700 1 -- 172.21.15.23:0/3508564328 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91640b7a50 con 0x7f9164069eb0 2023-06-22T08:07:23.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.373+0000 7f916b7ff700 1 -- 172.21.15.23:0/3508564328 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9164163760 con 0x7f9164069eb0 2023-06-22T08:07:23.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.374+0000 7f916b7ff700 1 -- 172.21.15.23:0/3508564328 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91641ecac0 con 0x7f9164069eb0 2023-06-22T08:07:23.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.375+0000 7f9159ffb700 1 -- 172.21.15.23:0/3508564328 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9160003bb0 con 0x7f9164069eb0 2023-06-22T08:07:23.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.376+0000 7f9159ffb700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f914806ace0 0x7f914806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:23.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.376+0000 7f9159ffb700 1 -- 172.21.15.23:0/3508564328 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f916008e730 con 0x7f9164069eb0 2023-06-22T08:07:23.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.376+0000 7f9168d9a700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f914806ace0 0x7f914806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:23.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.377+0000 7f9168d9a700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f914806ace0 0x7f914806d190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f91641ec090 tx=0x7f9154009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:23.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.381+0000 7f9159ffb700 1 -- 172.21.15.23:0/3508564328 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f91600593a0 con 0x7f9164069eb0 2023-06-22T08:07:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.635+0000 7f916b7ff700 1 -- 172.21.15.23:0/3508564328 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f916404ea20 con 0x7f9164069eb0 2023-06-22T08:07:23.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.635+0000 7f9159ffb700 1 -- 172.21.15.23:0/3508564328 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5498 (secure 0 0 0) 0x7f9160003e60 con 0x7f9164069eb0 2023-06-22T08:07:23.639 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:23.639 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:19.358304+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 -- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f914806ace0 msgr2=0x7f914806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f914806ace0 0x7f914806d190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f91641ec090 tx=0x7f9154009040 comp rx=0 tx=0).stop 2023-06-22T08:07:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 -- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9164069eb0 msgr2=0x7f91641ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:23.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9164069eb0 0x7f91641ff480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f916406f6c0 tx=0x7f9160004750 comp rx=0 tx=0).stop 2023-06-22T08:07:23.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 -- 172.21.15.23:0/3508564328 shutdown_connections 2023-06-22T08:07:23.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f914806ace0 0x7f914806d190 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:23.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9164166380 0x7f91641ed1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9164069eb0 0x7f91641ff480 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 --2- 172.21.15.23:0/3508564328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f916406dc40 0x7f91641ecc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 -- 172.21.15.23:0/3508564328 >> 172.21.15.23:0/3508564328 conn(0x7f91640ca3f0 msgr2=0x7f9164165400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:23.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 -- 172.21.15.23:0/3508564328 shutdown_connections 2023-06-22T08:07:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:23.638+0000 7f916b7ff700 1 -- 172.21.15.23:0/3508564328 wait complete. 2023-06-22T08:07:23.645 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 57 2023-06-22T08:07:23.661 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 57, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 57, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4279": { "addr": "172.21.15.67:6836/1962314516", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 1962314516, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 1962314516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4285": { "addr": "172.21.15.67:6839/567663545", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 567663545, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 567663545, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5060": { "addr": "172.21.15.67:6837/1551460358", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1551460358, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1551460358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5060, "incarnation": 46, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 31 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:07:19.358304+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4279, "mds_2": 5551, "mds_3": 5060, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:07:25.662 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:25.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.859+0000 7fa01fdff700 1 Processor -- start 2023-06-22T08:07:25.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.859+0000 7fa01fdff700 1 -- start start 2023-06-22T08:07:25.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.861+0000 7fa01fdff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa018152a10 0x7fa0180c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:25.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.861+0000 7fa01fdff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa018159730 con 0x7fa0180c9200 2023-06-22T08:07:25.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.861+0000 7fa01fdff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa018158910 con 0x7fa01815c0b0 2023-06-22T08:07:25.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.861+0000 7fa01fdff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0180c5470 con 0x7fa018152a10 2023-06-22T08:07:25.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.861+0000 7fa01db9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa018152a10 0x7fa0180c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:25.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.861+0000 7fa01d39a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa0180c9200 0x7fa0180cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:43046/0 (socket says 172.21.15.23:43046) 2023-06-22T08:07:25.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.861+0000 7fa01d39a700 1 -- 172.21.15.23:0/3396160515 learned_addr learned my addr 172.21.15.23:0/3396160515 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:25.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.861+0000 7fa01db9b700 1 -- 172.21.15.23:0/3396160515 >> v1:172.21.15.67:6789/0 conn(0x7fa01815c0b0 legacy=0x7fa01815c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:25.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.861+0000 7fa01db9b700 1 -- 172.21.15.23:0/3396160515 >> v1:172.21.15.23:6789/0 conn(0x7fa0180c9200 legacy=0x7fa0180cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:25.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.861+0000 7fa01db9b700 1 -- 172.21.15.23:0/3396160515 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa018158bb0 con 0x7fa018152a10 2023-06-22T08:07:25.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.862+0000 7fa01db9b700 1 --2- 172.21.15.23:0/3396160515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa018152a10 0x7fa0180c4a30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa014009840 tx=0x7fa014009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b492fe456aa2331 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:25.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.862+0000 7fa01cb99700 1 -- 172.21.15.23:0/3396160515 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014011830 con 0x7fa018152a10 2023-06-22T08:07:25.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.862+0000 7fa01cb99700 1 -- 172.21.15.23:0/3396160515 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa014011990 con 0x7fa018152a10 2023-06-22T08:07:25.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.862+0000 7fa01cb99700 1 -- 172.21.15.23:0/3396160515 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014011d20 con 0x7fa018152a10 2023-06-22T08:07:25.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.863+0000 7fa01fdff700 1 -- 172.21.15.23:0/3396160515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa018152a10 msgr2=0x7fa0180c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:25.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.863+0000 7fa01fdff700 1 --2- 172.21.15.23:0/3396160515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa018152a10 0x7fa0180c4a30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa014009840 tx=0x7fa014009b50 comp rx=0 tx=0).stop 2023-06-22T08:07:25.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.863+0000 7fa01fdff700 1 -- 172.21.15.23:0/3396160515 shutdown_connections 2023-06-22T08:07:25.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.863+0000 7fa01fdff700 1 --2- 172.21.15.23:0/3396160515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa018152a10 0x7fa0180c4a30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:25.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.863+0000 7fa01fdff700 1 -- 172.21.15.23:0/3396160515 >> 172.21.15.23:0/3396160515 conn(0x7fa018065b90 msgr2=0x7fa018065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:25.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.863+0000 7fa01fdff700 1 -- 172.21.15.23:0/3396160515 shutdown_connections 2023-06-22T08:07:25.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.863+0000 7fa01fdff700 1 -- 172.21.15.23:0/3396160515 wait complete. 2023-06-22T08:07:25.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.865+0000 7fa01fdff700 1 Processor -- start 2023-06-22T08:07:25.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.865+0000 7fa01fdff700 1 -- start start 2023-06-22T08:07:25.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.866+0000 7fa01fdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0180c9200 0x7fa0181f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:25.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.866+0000 7fa01fdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa018152a10 0x7fa018155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:25.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01fdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa01815c0b0 0x7fa018155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:25.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01fdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa018153ed0 con 0x7fa01815c0b0 2023-06-22T08:07:25.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01fdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa018152fa0 con 0x7fa0180c9200 2023-06-22T08:07:25.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01fdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0180c8390 con 0x7fa018152a10 2023-06-22T08:07:25.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01db9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0180c9200 0x7fa0181f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:25.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01db9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0180c9200 0x7fa0181f1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:35768/0 (socket says 172.21.15.23:35768) 2023-06-22T08:07:25.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01db9b700 1 -- 172.21.15.23:0/301085125 learned_addr learned my addr 172.21.15.23:0/301085125 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:25.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01d39a700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa018152a10 0x7fa018155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:25.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01e39c700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa01815c0b0 0x7fa018155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:25.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01db9b700 1 -- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa018152a10 msgr2=0x7fa018155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:25.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01db9b700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa018152a10 0x7fa018155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:25.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01db9b700 1 -- 172.21.15.23:0/301085125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa01815c0b0 msgr2=0x7fa018155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:25.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01db9b700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa01815c0b0 0x7fa018155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:25.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01db9b700 1 -- 172.21.15.23:0/301085125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0140094a0 con 0x7fa0180c9200 2023-06-22T08:07:25.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.867+0000 7fa01db9b700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0180c9200 0x7fa0181f1ec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa0140090b0 tx=0x7fa014004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:25.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.868+0000 7fa00e7fc700 1 -- 172.21.15.23:0/301085125 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0140036a0 con 0x7fa0180c9200 2023-06-22T08:07:25.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.868+0000 7fa00e7fc700 1 -- 172.21.15.23:0/301085125 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01401f070 con 0x7fa0180c9200 2023-06-22T08:07:25.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.868+0000 7fa01fdff700 1 -- 172.21.15.23:0/301085125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0180c6460 con 0x7fa0180c9200 2023-06-22T08:07:25.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.868+0000 7fa00e7fc700 1 -- 172.21.15.23:0/301085125 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014003a60 con 0x7fa0180c9200 2023-06-22T08:07:25.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.868+0000 7fa01fdff700 1 -- 172.21.15.23:0/301085125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0180c85e0 con 0x7fa0180c9200 2023-06-22T08:07:25.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.869+0000 7fa01fdff700 1 -- 172.21.15.23:0/301085125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0180742e0 con 0x7fa0180c9200 2023-06-22T08:07:25.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.870+0000 7fa00e7fc700 1 -- 172.21.15.23:0/301085125 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa014003bc0 con 0x7fa0180c9200 2023-06-22T08:07:25.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.871+0000 7fa00e7fc700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9ffc06ace0 0x7f9ffc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:25.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.871+0000 7fa00e7fc700 1 -- 172.21.15.23:0/301085125 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7fa014021070 con 0x7fa0180c9200 2023-06-22T08:07:25.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.871+0000 7fa01d39a700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9ffc06ace0 0x7f9ffc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:25.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.872+0000 7fa01d39a700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9ffc06ace0 0x7f9ffc06d190 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa0180737d0 tx=0x7fa008009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:25.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:25.875+0000 7fa00e7fc700 1 -- 172.21.15.23:0/301085125 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa014056220 con 0x7fa0180c9200 2023-06-22T08:07:26.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.131+0000 7fa01fdff700 1 -- 172.21.15.23:0/301085125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0180c65f0 con 0x7fa0180c9200 2023-06-22T08:07:26.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.132+0000 7fa00e7fc700 1 -- 172.21.15.23:0/301085125 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5500 (secure 0 0 0) 0x7fa0180c65f0 con 0x7fa0180c9200 2023-06-22T08:07:26.136 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:26.136 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:25.362872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:resolve","state_seq":18,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:26.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 -- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9ffc06ace0 msgr2=0x7f9ffc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:26.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9ffc06ace0 0x7f9ffc06d190 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa0180737d0 tx=0x7fa008009040 comp rx=0 tx=0).stop 2023-06-22T08:07:26.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 -- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0180c9200 msgr2=0x7fa0181f1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:26.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0180c9200 0x7fa0181f1ec0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa0140090b0 tx=0x7fa014004750 comp rx=0 tx=0).stop 2023-06-22T08:07:26.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 -- 172.21.15.23:0/301085125 shutdown_connections 2023-06-22T08:07:26.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa01815c0b0 0x7fa018155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:26.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9ffc06ace0 0x7f9ffc06d190 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:26.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0180c9200 0x7fa0181f1ec0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:26.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 --2- 172.21.15.23:0/301085125 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa018152a10 0x7fa018155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:26.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 -- 172.21.15.23:0/301085125 >> 172.21.15.23:0/301085125 conn(0x7fa018065b90 msgr2=0x7fa01806adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:26.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 -- 172.21.15.23:0/301085125 shutdown_connections 2023-06-22T08:07:26.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:26.135+0000 7fa01fdff700 1 -- 172.21.15.23:0/301085125 wait complete. 2023-06-22T08:07:26.140 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 58 2023-06-22T08:07:26.156 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:07:28.157 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:28.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.363+0000 7f9fa85ff700 1 Processor -- start 2023-06-22T08:07:28.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.363+0000 7f9fa85ff700 1 -- start start 2023-06-22T08:07:28.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa85ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9fa0152a10 0x7f9fa00c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:28.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa85ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9fa00c9200 0x7f9fa00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:28.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa85ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa0159730 con 0x7f9fa00cb7a0 2023-06-22T08:07:28.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa85ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa00c5470 con 0x7f9fa0152a10 2023-06-22T08:07:28.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa85ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fa0158910 con 0x7f9fa00c9200 2023-06-22T08:07:28.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa639b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9fa0152a10 0x7f9fa00c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:28.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa6b9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9fa00cb7a0 0x7f9fa015c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33518/0 (socket says 172.21.15.23:33518) 2023-06-22T08:07:28.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa6b9c700 1 -- 172.21.15.23:0/3368411623 learned_addr learned my addr 172.21.15.23:0/3368411623 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:28.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa5b9a700 1 --2- 172.21.15.23:0/3368411623 >> v2:172.21.15.67:3300/0 conn(0x7f9fa00c9200 0x7f9fa00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:28.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa639b700 1 -- 172.21.15.23:0/3368411623 >> v2:172.21.15.67:3300/0 conn(0x7f9fa00c9200 msgr2=0x7f9fa00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:28.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa639b700 1 --2- 172.21.15.23:0/3368411623 >> v2:172.21.15.67:3300/0 conn(0x7f9fa00c9200 0x7f9fa00cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:28.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa639b700 1 -- 172.21.15.23:0/3368411623 >> v1:172.21.15.23:6789/0 conn(0x7f9fa00cb7a0 legacy=0x7f9fa015c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:28.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa639b700 1 -- 172.21.15.23:0/3368411623 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa0158bb0 con 0x7f9fa0152a10 2023-06-22T08:07:28.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.365+0000 7f9fa5399700 1 -- 172.21.15.23:0/3368411623 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526734829 0 0) 0x7f9fa0159730 con 0x7f9fa00cb7a0 2023-06-22T08:07:28.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.366+0000 7f9fa639b700 1 --2- 172.21.15.23:0/3368411623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 0x7f9fa00c4a30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9fa008c4c0 tx=0x7f9f9c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89c023d3240e8d5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:28.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.366+0000 7f9fa5399700 1 -- 172.21.15.23:0/3368411623 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c011830 con 0x7f9fa0152a10 2023-06-22T08:07:28.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.366+0000 7f9fa5399700 1 -- 172.21.15.23:0/3368411623 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9c011990 con 0x7f9fa0152a10 2023-06-22T08:07:28.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.366+0000 7f9fa5399700 1 -- 172.21.15.23:0/3368411623 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c011d20 con 0x7f9fa0152a10 2023-06-22T08:07:28.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.366+0000 7f9fa85ff700 1 -- 172.21.15.23:0/3368411623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 msgr2=0x7f9fa00c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:28.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.366+0000 7f9fa85ff700 1 --2- 172.21.15.23:0/3368411623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 0x7f9fa00c4a30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9fa008c4c0 tx=0x7f9f9c009b20 comp rx=0 tx=0).stop 2023-06-22T08:07:28.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.367+0000 7f9fa85ff700 1 -- 172.21.15.23:0/3368411623 shutdown_connections 2023-06-22T08:07:28.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.367+0000 7f9fa85ff700 1 --2- 172.21.15.23:0/3368411623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 0x7f9fa00c4a30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:28.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.367+0000 7f9fa85ff700 1 --2- 172.21.15.23:0/3368411623 >> v2:172.21.15.67:3300/0 conn(0x7f9fa00c9200 0x7f9fa00cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:28.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.367+0000 7f9fa85ff700 1 -- 172.21.15.23:0/3368411623 >> 172.21.15.23:0/3368411623 conn(0x7f9fa0065b90 msgr2=0x7f9fa0065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:28.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.367+0000 7f9fa85ff700 1 -- 172.21.15.23:0/3368411623 shutdown_connections 2023-06-22T08:07:28.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.367+0000 7f9fa85ff700 1 -- 172.21.15.23:0/3368411623 wait complete. 2023-06-22T08:07:28.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.368+0000 7f9fa85ff700 1 Processor -- start 2023-06-22T08:07:28.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.369+0000 7f9fa85ff700 1 -- start start 2023-06-22T08:07:28.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.369+0000 7f9fa85ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fa00c9200 0x7f9fa01ef910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:28.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.369+0000 7f9fa85ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa00cb7a0 0x7f9fa0159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:28.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.369+0000 7f9fa85ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 0x7f9fa015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:28.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.369+0000 7f9fa85ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa01ef4f0 con 0x7f9fa0152a10 2023-06-22T08:07:28.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.369+0000 7f9fa85ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa015ba70 con 0x7f9fa00cb7a0 2023-06-22T08:07:28.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.369+0000 7f9fa85ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa015b830 con 0x7f9fa00c9200 2023-06-22T08:07:28.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa6b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 0x7f9fa015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:28.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa6b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 0x7f9fa015b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39110/0 (socket says 172.21.15.23:39110) 2023-06-22T08:07:28.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa6b9c700 1 -- 172.21.15.23:0/2895516560 learned_addr learned my addr 172.21.15.23:0/2895516560 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:28.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa639b700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fa00c9200 0x7f9fa01ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:28.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa5b9a700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa00cb7a0 0x7f9fa0159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:28.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa6b9c700 1 -- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fa00c9200 msgr2=0x7f9fa01ef910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:28.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa6b9c700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fa00c9200 0x7f9fa01ef910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:28.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa6b9c700 1 -- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa00cb7a0 msgr2=0x7f9fa0159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:28.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa6b9c700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa00cb7a0 0x7f9fa0159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:28.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa6b9c700 1 -- 172.21.15.23:0/2895516560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c0094a0 con 0x7f9fa0152a10 2023-06-22T08:07:28.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.370+0000 7f9fa6b9c700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 0x7f9fa015b2f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9f98002230 tx=0x7f9f9800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:28.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.371+0000 7f9f96ffd700 1 -- 172.21.15.23:0/2895516560 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9800aa00 con 0x7f9fa0152a10 2023-06-22T08:07:28.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.371+0000 7f9f96ffd700 1 -- 172.21.15.23:0/2895516560 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9800ab80 con 0x7f9fa0152a10 2023-06-22T08:07:28.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.371+0000 7f9f96ffd700 1 -- 172.21.15.23:0/2895516560 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f980070d0 con 0x7f9fa0152a10 2023-06-22T08:07:28.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.371+0000 7f9fa85ff700 1 -- 172.21.15.23:0/2895516560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa0157fa0 con 0x7f9fa0152a10 2023-06-22T08:07:28.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.371+0000 7f9fa85ff700 1 -- 172.21.15.23:0/2895516560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa0158520 con 0x7f9fa0152a10 2023-06-22T08:07:28.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.372+0000 7f9fa85ff700 1 -- 172.21.15.23:0/2895516560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa01ef630 con 0x7f9fa0152a10 2023-06-22T08:07:28.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.373+0000 7f9f96ffd700 1 -- 172.21.15.23:0/2895516560 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9f98007230 con 0x7f9fa0152a10 2023-06-22T08:07:28.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.374+0000 7f9f96ffd700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f8406ace0 0x7f9f8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:28.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.374+0000 7f9f96ffd700 1 -- 172.21.15.23:0/2895516560 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f9f980893d0 con 0x7f9fa0152a10 2023-06-22T08:07:28.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.374+0000 7f9fa639b700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f8406ace0 0x7f9f8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:28.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.375+0000 7f9fa639b700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f8406ace0 0x7f9f8406d190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0090b0 tx=0x7f9f9c01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:28.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.378+0000 7f9f96ffd700 1 -- 172.21.15.23:0/2895516560 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9fa01ef630 con 0x7f9fa0152a10 2023-06-22T08:07:28.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.629+0000 7f9fa85ff700 1 -- 172.21.15.23:0/2895516560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fa01ef630 con 0x7f9fa0152a10 2023-06-22T08:07:28.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.630+0000 7f9f96ffd700 1 -- 172.21.15.23:0/2895516560 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5500 (secure 0 0 0) 0x7f9fa01ef630 con 0x7f9fa0152a10 2023-06-22T08:07:28.633 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:28.634 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:25.362872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:resolve","state_seq":18,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:28.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 -- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f8406ace0 msgr2=0x7f9f8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f8406ace0 0x7f9f8406d190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0090b0 tx=0x7f9f9c01c040 comp rx=0 tx=0).stop 2023-06-22T08:07:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 -- 172.21.15.23:0/2895516560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 msgr2=0x7f9fa015b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 0x7f9fa015b2f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9f98002230 tx=0x7f9f9800cf50 comp rx=0 tx=0).stop 2023-06-22T08:07:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 -- 172.21.15.23:0/2895516560 shutdown_connections 2023-06-22T08:07:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa0152a10 0x7f9fa015b2f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fa00c9200 0x7f9fa01ef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:28.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f8406ace0 0x7f9f8406d190 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:28.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 --2- 172.21.15.23:0/2895516560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fa00cb7a0 0x7f9fa0159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:28.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 -- 172.21.15.23:0/2895516560 >> 172.21.15.23:0/2895516560 conn(0x7f9fa0065b90 msgr2=0x7f9fa015bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:28.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 -- 172.21.15.23:0/2895516560 shutdown_connections 2023-06-22T08:07:28.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:28.632+0000 7f9fa85ff700 1 -- 172.21.15.23:0/2895516560 wait complete. 2023-06-22T08:07:28.637 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 58 2023-06-22T08:07:28.653 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:07:30.654 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:30.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.864+0000 7f89dfdff700 1 Processor -- start 2023-06-22T08:07:30.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.864+0000 7f89dfdff700 1 -- start start 2023-06-22T08:07:30.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.866+0000 7f89dfdff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89d8152ce0 0x7f89d80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:30.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.866+0000 7f89dfdff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d815c370 con 0x7f89d80c40b0 2023-06-22T08:07:30.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.866+0000 7f89dfdff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d815bda0 con 0x7f89d80c9fd0 2023-06-22T08:07:30.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.866+0000 7f89dfdff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d81530b0 con 0x7f89d8152ce0 2023-06-22T08:07:30.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.866+0000 7f89de39c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f89d80c40b0 0x7f89d80c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33526/0 (socket says 172.21.15.23:33526) 2023-06-22T08:07:30.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.866+0000 7f89de39c700 1 -- 172.21.15.23:0/139067218 learned_addr learned my addr 172.21.15.23:0/139067218 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:30.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.866+0000 7f89ddb9b700 1 --2- 172.21.15.23:0/139067218 >> v2:172.21.15.67:3300/0 conn(0x7f89d8152ce0 0x7f89d80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:30.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.866+0000 7f89dcb99700 1 -- 172.21.15.23:0/139067218 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299062073 0 0) 0x7f89d815c370 con 0x7f89d80c40b0 2023-06-22T08:07:30.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.867+0000 7f89dcb99700 1 -- 172.21.15.23:0/139067218 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89c40035b0 con 0x7f89d80c40b0 2023-06-22T08:07:30.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.867+0000 7f89ddb9b700 1 -- 172.21.15.23:0/139067218 >> v1:172.21.15.67:6789/0 conn(0x7f89d80c9fd0 legacy=0x7f89d815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:30.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.867+0000 7f89ddb9b700 1 -- 172.21.15.23:0/139067218 >> v1:172.21.15.23:6789/0 conn(0x7f89d80c40b0 legacy=0x7f89d80c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:30.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.867+0000 7f89ddb9b700 1 -- 172.21.15.23:0/139067218 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d80c9460 con 0x7f89d8152ce0 2023-06-22T08:07:30.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.867+0000 7f89ddb9b700 1 --2- 172.21.15.23:0/139067218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d8152ce0 0x7f89d80cf1b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009840 tx=0x7f89d4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f780a8c0e18f5acd server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:30.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.867+0000 7f89dcb99700 1 -- 172.21.15.23:0/139067218 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4011830 con 0x7f89d8152ce0 2023-06-22T08:07:30.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.867+0000 7f89dcb99700 1 -- 172.21.15.23:0/139067218 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d4011990 con 0x7f89d8152ce0 2023-06-22T08:07:30.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.867+0000 7f89dcb99700 1 -- 172.21.15.23:0/139067218 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4011d20 con 0x7f89d8152ce0 2023-06-22T08:07:30.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.868+0000 7f89dfdff700 1 -- 172.21.15.23:0/139067218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d8152ce0 msgr2=0x7f89d80cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:30.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.868+0000 7f89dfdff700 1 --2- 172.21.15.23:0/139067218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d8152ce0 0x7f89d80cf1b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009840 tx=0x7f89d4009b50 comp rx=0 tx=0).stop 2023-06-22T08:07:30.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.868+0000 7f89dfdff700 1 -- 172.21.15.23:0/139067218 shutdown_connections 2023-06-22T08:07:30.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.868+0000 7f89dfdff700 1 --2- 172.21.15.23:0/139067218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d8152ce0 0x7f89d80cf1b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:30.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.868+0000 7f89dfdff700 1 -- 172.21.15.23:0/139067218 >> 172.21.15.23:0/139067218 conn(0x7f89d806ae30 msgr2=0x7f89d806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:30.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.868+0000 7f89dfdff700 1 -- 172.21.15.23:0/139067218 shutdown_connections 2023-06-22T08:07:30.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.868+0000 7f89dfdff700 1 -- 172.21.15.23:0/139067218 wait complete. 2023-06-22T08:07:30.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.870+0000 7f89dfdff700 1 Processor -- start 2023-06-22T08:07:30.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.870+0000 7f89dfdff700 1 -- start start 2023-06-22T08:07:30.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dfdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89d80c40b0 0x7f89d814ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:30.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dfdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d80c9fd0 0x7f89d81474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:30.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dfdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d8152ce0 0x7f89d80c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:30.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dfdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d81eee70 con 0x7f89d80c9fd0 2023-06-22T08:07:30.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dfdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d814d440 con 0x7f89d8152ce0 2023-06-22T08:07:30.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dfdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d81ee910 con 0x7f89d80c40b0 2023-06-22T08:07:30.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dd39a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d80c9fd0 0x7f89d81474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:30.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dd39a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d80c9fd0 0x7f89d81474c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39114/0 (socket says 172.21.15.23:39114) 2023-06-22T08:07:30.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dd39a700 1 -- 172.21.15.23:0/2340144220 learned_addr learned my addr 172.21.15.23:0/2340144220 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89ddb9b700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89d80c40b0 0x7f89d814ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89de39c700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d8152ce0 0x7f89d80c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dd39a700 1 -- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89d80c40b0 msgr2=0x7f89d814ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dd39a700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89d80c40b0 0x7f89d814ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dd39a700 1 -- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d8152ce0 msgr2=0x7f89d80c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dd39a700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d8152ce0 0x7f89d80c6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.871+0000 7f89dd39a700 1 -- 172.21.15.23:0/2340144220 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d40094a0 con 0x7f89d80c9fd0 2023-06-22T08:07:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.872+0000 7f89dd39a700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d80c9fd0 0x7f89d81474c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f89c800c4e0 tx=0x7f89c800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:30.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.872+0000 7f89ce7fc700 1 -- 172.21.15.23:0/2340144220 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c80052d0 con 0x7f89d80c9fd0 2023-06-22T08:07:30.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.872+0000 7f89ce7fc700 1 -- 172.21.15.23:0/2340144220 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c8005430 con 0x7f89d80c9fd0 2023-06-22T08:07:30.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.872+0000 7f89dfdff700 1 -- 172.21.15.23:0/2340144220 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d814a760 con 0x7f89d80c9fd0 2023-06-22T08:07:30.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.872+0000 7f89ce7fc700 1 -- 172.21.15.23:0/2340144220 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c80038c0 con 0x7f89d80c9fd0 2023-06-22T08:07:30.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.872+0000 7f89dfdff700 1 -- 172.21.15.23:0/2340144220 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d80c87d0 con 0x7f89d80c9fd0 2023-06-22T08:07:30.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.873+0000 7f89dfdff700 1 -- 172.21.15.23:0/2340144220 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d814d5b0 con 0x7f89d80c9fd0 2023-06-22T08:07:30.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.874+0000 7f89ce7fc700 1 -- 172.21.15.23:0/2340144220 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f89c8003a20 con 0x7f89d80c9fd0 2023-06-22T08:07:30.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.875+0000 7f89ce7fc700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89bc06ace0 0x7f89bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:30.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.875+0000 7f89ce7fc700 1 -- 172.21.15.23:0/2340144220 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f89c8088c30 con 0x7f89d80c9fd0 2023-06-22T08:07:30.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.876+0000 7f89ddb9b700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89bc06ace0 0x7f89bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:30.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.876+0000 7f89ddb9b700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89bc06ace0 0x7f89bc06d190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f89d40090b0 tx=0x7f89d401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:30.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:30.880+0000 7f89ce7fc700 1 -- 172.21.15.23:0/2340144220 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f89c8053890 con 0x7f89d80c9fd0 2023-06-22T08:07:31.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.135+0000 7f89dfdff700 1 -- 172.21.15.23:0/2340144220 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89d804ea20 con 0x7f89d80c9fd0 2023-06-22T08:07:31.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.136+0000 7f89ce7fc700 1 -- 172.21.15.23:0/2340144220 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5502 (secure 0 0 0) 0x7f89c8056eb0 con 0x7f89d80c9fd0 2023-06-22T08:07:31.139 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:31.140 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:30.688655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:31.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 -- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89bc06ace0 msgr2=0x7f89bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:31.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89bc06ace0 0x7f89bc06d190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f89d40090b0 tx=0x7f89d401c040 comp rx=0 tx=0).stop 2023-06-22T08:07:31.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 -- 172.21.15.23:0/2340144220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d80c9fd0 msgr2=0x7f89d81474c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:31.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d80c9fd0 0x7f89d81474c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f89c800c4e0 tx=0x7f89c800c7f0 comp rx=0 tx=0).stop 2023-06-22T08:07:31.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 -- 172.21.15.23:0/2340144220 shutdown_connections 2023-06-22T08:07:31.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f89bc06ace0 0x7f89bc06d190 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:31.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89d8152ce0 0x7f89d80c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:31.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89d80c40b0 0x7f89d814ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:31.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 --2- 172.21.15.23:0/2340144220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89d80c9fd0 0x7f89d81474c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:31.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 -- 172.21.15.23:0/2340144220 >> 172.21.15.23:0/2340144220 conn(0x7f89d806ae30 msgr2=0x7f89d8150c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:31.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 -- 172.21.15.23:0/2340144220 shutdown_connections 2023-06-22T08:07:31.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:31.138+0000 7f89dfdff700 1 -- 172.21.15.23:0/2340144220 wait complete. 2023-06-22T08:07:31.143 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 59 2023-06-22T08:07:31.159 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:07:33.160 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:33.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.367+0000 7f0cb49ff700 1 Processor -- start 2023-06-22T08:07:33.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.367+0000 7f0cb49ff700 1 -- start start 2023-06-22T08:07:33.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.368+0000 7f0cb49ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb00c97d0 0x7f0cb0159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:33.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.368+0000 7f0cb49ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0cb00c45f0 0x7f0cb00c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:33.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.368+0000 7f0cb49ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb00c4fb0 con 0x7f0cb0152d70 2023-06-22T08:07:33.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.368+0000 7f0cb49ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb00c95e0 con 0x7f0cb00c45f0 2023-06-22T08:07:33.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.368+0000 7f0cb49ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00c9c00 con 0x7f0cb00c97d0 2023-06-22T08:07:33.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0caed9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0cb00c45f0 0x7f0cb00c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:33.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0cadd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb00c97d0 0x7f0cb0159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:33.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0cae59c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0cb0152d70 0x7f0cb00cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:54914/0 (socket says 172.21.15.23:54914) 2023-06-22T08:07:33.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0caed9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0cb00c45f0 0x7f0cb00c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39124/0 (socket says 172.21.15.23:39124) 2023-06-22T08:07:33.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0cae59c700 1 -- 172.21.15.23:0/3731448809 learned_addr learned my addr 172.21.15.23:0/3731448809 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:33.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0cadd9b700 1 -- 172.21.15.23:0/3731448809 >> v2:172.21.15.23:3300/0 conn(0x7f0cb00c45f0 msgr2=0x7f0cb00c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:33.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0cadd9b700 1 --2- 172.21.15.23:0/3731448809 >> v2:172.21.15.23:3300/0 conn(0x7f0cb00c45f0 0x7f0cb00c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:33.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0cadd9b700 1 -- 172.21.15.23:0/3731448809 >> v1:172.21.15.67:6789/0 conn(0x7f0cb0152d70 legacy=0x7f0cb00cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:33.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0cadd9b700 1 -- 172.21.15.23:0/3731448809 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb015bea0 con 0x7f0cb00c97d0 2023-06-22T08:07:33.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0cadd9b700 1 --2- 172.21.15.23:0/3731448809 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb00c97d0 0x7f0cb0159330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009840 tx=0x7f0ca4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=262ebe5674e9f2f8 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:33.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.369+0000 7f0cad59a700 1 -- 172.21.15.23:0/3731448809 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4011830 con 0x7f0cb00c97d0 2023-06-22T08:07:33.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.370+0000 7f0cad59a700 1 -- 172.21.15.23:0/3731448809 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca4011990 con 0x7f0cb00c97d0 2023-06-22T08:07:33.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.370+0000 7f0cad59a700 1 -- 172.21.15.23:0/3731448809 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4011d20 con 0x7f0cb00c97d0 2023-06-22T08:07:33.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.370+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3731448809 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb00c97d0 msgr2=0x7f0cb0159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:33.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.370+0000 7f0cb49ff700 1 --2- 172.21.15.23:0/3731448809 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb00c97d0 0x7f0cb0159330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009840 tx=0x7f0ca4009b50 comp rx=0 tx=0).stop 2023-06-22T08:07:33.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.370+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3731448809 shutdown_connections 2023-06-22T08:07:33.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.370+0000 7f0cb49ff700 1 --2- 172.21.15.23:0/3731448809 >> v2:172.21.15.23:3300/0 conn(0x7f0cb00c45f0 0x7f0cb00c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:33.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.370+0000 7f0cb49ff700 1 --2- 172.21.15.23:0/3731448809 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb00c97d0 0x7f0cb0159330 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:33.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.370+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3731448809 >> 172.21.15.23:0/3731448809 conn(0x7f0cb006ae30 msgr2=0x7f0cb006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:33.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.370+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3731448809 shutdown_connections 2023-06-22T08:07:33.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.370+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3731448809 wait complete. 2023-06-22T08:07:33.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.372+0000 7f0cb49ff700 1 Processor -- start 2023-06-22T08:07:33.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.372+0000 7f0cb49ff700 1 -- start start 2023-06-22T08:07:33.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0cb49ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cb00c45f0 0x7f0cb01ee160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:33.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0cb49ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb00c97d0 0x7f0cb01f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:33.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0cb49ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb0152d70 0x7f0cb0148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:33.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0cb49ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb014ee70 con 0x7f0cb00c45f0 2023-06-22T08:07:33.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0cb49ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb01f16d0 con 0x7f0cb00c97d0 2023-06-22T08:07:33.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0cb49ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0150db0 con 0x7f0cb0152d70 2023-06-22T08:07:33.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0cae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cb00c45f0 0x7f0cb01ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:33.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0cae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cb00c45f0 0x7f0cb01ee160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39136/0 (socket says 172.21.15.23:39136) 2023-06-22T08:07:33.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0cae59c700 1 -- 172.21.15.23:0/3293972396 learned_addr learned my addr 172.21.15.23:0/3293972396 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:33.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0caed9d700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb0152d70 0x7f0cb0148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.373+0000 7f0cadd9b700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb00c97d0 0x7f0cb01f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0cae59c700 1 -- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb0152d70 msgr2=0x7f0cb0148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0cae59c700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb0152d70 0x7f0cb0148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0cae59c700 1 -- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb00c97d0 msgr2=0x7f0cb01f2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0cae59c700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb00c97d0 0x7f0cb01f2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0cae59c700 1 -- 172.21.15.23:0/3293972396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca40094a0 con 0x7f0cb00c45f0 2023-06-22T08:07:33.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0cae59c700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cb00c45f0 0x7f0cb01ee160 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000c480 tx=0x7f0ca000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0c9effd700 1 -- 172.21.15.23:0/3293972396 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0005d70 con 0x7f0cb00c45f0 2023-06-22T08:07:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0c9effd700 1 -- 172.21.15.23:0/3293972396 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca000ce90 con 0x7f0cb00c45f0 2023-06-22T08:07:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0c9effd700 1 -- 172.21.15.23:0/3293972396 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca00072d0 con 0x7f0cb00c45f0 2023-06-22T08:07:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3293972396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb0151090 con 0x7f0cb00c45f0 2023-06-22T08:07:33.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.374+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3293972396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb01f1970 con 0x7f0cb00c45f0 2023-06-22T08:07:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.375+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3293972396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb004ea20 con 0x7f0cb00c45f0 2023-06-22T08:07:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.376+0000 7f0c9effd700 1 -- 172.21.15.23:0/3293972396 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0ca0007430 con 0x7f0cb00c45f0 2023-06-22T08:07:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.377+0000 7f0c9effd700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c9006ad10 0x7f0c9006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.377+0000 7f0c9effd700 1 -- 172.21.15.23:0/3293972396 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f0ca0088b80 con 0x7f0cb00c45f0 2023-06-22T08:07:33.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.377+0000 7f0cadd9b700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c9006ad10 0x7f0c9006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:33.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.378+0000 7f0cadd9b700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c9006ad10 0x7f0c9006d1c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4000c00 tx=0x7f0ca4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:33.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.382+0000 7f0c9effd700 1 -- 172.21.15.23:0/3293972396 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0ca00537f0 con 0x7f0cb00c45f0 2023-06-22T08:07:33.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.632+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3293972396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cb00c9e00 con 0x7f0cb00c45f0 2023-06-22T08:07:33.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.633+0000 7f0c9effd700 1 -- 172.21.15.23:0/3293972396 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5502 (secure 0 0 0) 0x7f0ca0056e10 con 0x7f0cb00c45f0 2023-06-22T08:07:33.641 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:33.642 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:30.688655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:33.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c9006ad10 msgr2=0x7f0c9006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:33.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c9006ad10 0x7f0c9006d1c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4000c00 tx=0x7f0ca4008030 comp rx=0 tx=0).stop 2023-06-22T08:07:33.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3293972396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cb00c45f0 msgr2=0x7f0cb01ee160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:33.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cb00c45f0 0x7f0cb01ee160 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0ca000c480 tx=0x7f0ca000c790 comp rx=0 tx=0).stop 2023-06-22T08:07:33.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3293972396 shutdown_connections 2023-06-22T08:07:33.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cb0152d70 0x7f0cb0148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:33.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cb00c45f0 0x7f0cb01ee160 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:33.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c9006ad10 0x7f0c9006d1c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:33.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 --2- 172.21.15.23:0/3293972396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cb00c97d0 0x7f0cb01f2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:33.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3293972396 >> 172.21.15.23:0/3293972396 conn(0x7f0cb006ae30 msgr2=0x7f0cb014e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:33.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3293972396 shutdown_connections 2023-06-22T08:07:33.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:33.635+0000 7f0cb49ff700 1 -- 172.21.15.23:0/3293972396 wait complete. 2023-06-22T08:07:33.644 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 59 2023-06-22T08:07:33.656 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:07:35.657 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.864+0000 7efc29fff700 1 Processor -- start 2023-06-22T08:07:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.865+0000 7efc29fff700 1 -- start start 2023-06-22T08:07:35.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.866+0000 7efc29fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc240c69e0 0x7efc24155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:35.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.866+0000 7efc29fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc240c51a0 con 0x7efc2414f850 2023-06-22T08:07:35.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.866+0000 7efc29fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc240c44e0 con 0x7efc24071070 2023-06-22T08:07:35.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.866+0000 7efc29fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc240c5440 con 0x7efc240c69e0 2023-06-22T08:07:35.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.866+0000 7efc237fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7efc2414f850 0x7efc240c3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33528/0 (socket says 172.21.15.23:33528) 2023-06-22T08:07:35.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.866+0000 7efc237fe700 1 -- 172.21.15.23:0/1602924655 learned_addr learned my addr 172.21.15.23:0/1602924655 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.866+0000 7efc1bfff700 1 --2- 172.21.15.23:0/1602924655 >> v2:172.21.15.67:3300/0 conn(0x7efc240c69e0 0x7efc24155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc22ffd700 1 -- 172.21.15.23:0/1602924655 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603238346 0 0) 0x7efc240c51a0 con 0x7efc2414f850 2023-06-22T08:07:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc22ffd700 1 -- 172.21.15.23:0/1602924655 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc100035b0 con 0x7efc2414f850 2023-06-22T08:07:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc22ffd700 1 -- 172.21.15.23:0/1602924655 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571035155 0 0) 0x7efc240c44e0 con 0x7efc24071070 2023-06-22T08:07:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc22ffd700 1 -- 172.21.15.23:0/1602924655 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc240c51a0 con 0x7efc24071070 2023-06-22T08:07:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc1bfff700 1 -- 172.21.15.23:0/1602924655 >> v1:172.21.15.67:6789/0 conn(0x7efc24071070 legacy=0x7efc2415fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc1bfff700 1 -- 172.21.15.23:0/1602924655 >> v1:172.21.15.23:6789/0 conn(0x7efc2414f850 legacy=0x7efc240c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc1bfff700 1 -- 172.21.15.23:0/1602924655 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc240c5720 con 0x7efc240c69e0 2023-06-22T08:07:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc1bfff700 1 --2- 172.21.15.23:0/1602924655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc240c69e0 0x7efc24155600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009840 tx=0x7efc0c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f22a3df51dee125b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:35.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc22ffd700 1 -- 172.21.15.23:0/1602924655 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c011830 con 0x7efc240c69e0 2023-06-22T08:07:35.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc22ffd700 1 -- 172.21.15.23:0/1602924655 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c011990 con 0x7efc240c69e0 2023-06-22T08:07:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.867+0000 7efc22ffd700 1 -- 172.21.15.23:0/1602924655 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c011d20 con 0x7efc240c69e0 2023-06-22T08:07:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.868+0000 7efc29fff700 1 -- 172.21.15.23:0/1602924655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc240c69e0 msgr2=0x7efc24155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.868+0000 7efc29fff700 1 --2- 172.21.15.23:0/1602924655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc240c69e0 0x7efc24155600 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009840 tx=0x7efc0c009b50 comp rx=0 tx=0).stop 2023-06-22T08:07:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.868+0000 7efc29fff700 1 -- 172.21.15.23:0/1602924655 shutdown_connections 2023-06-22T08:07:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.868+0000 7efc29fff700 1 --2- 172.21.15.23:0/1602924655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc240c69e0 0x7efc24155600 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.868+0000 7efc29fff700 1 -- 172.21.15.23:0/1602924655 >> 172.21.15.23:0/1602924655 conn(0x7efc240ca3f0 msgr2=0x7efc240ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.868+0000 7efc29fff700 1 -- 172.21.15.23:0/1602924655 shutdown_connections 2023-06-22T08:07:35.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.868+0000 7efc29fff700 1 -- 172.21.15.23:0/1602924655 wait complete. 2023-06-22T08:07:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.870+0000 7efc29fff700 1 Processor -- start 2023-06-22T08:07:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.871+0000 7efc29fff700 1 -- start start 2023-06-22T08:07:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc29fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc24071070 0x7efc241ea300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc29fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efc240c69e0 0x7efc241452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc29fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc2414f850 0x7efc2406d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc29fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc24156810 con 0x7efc2414f850 2023-06-22T08:07:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc29fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc241ed780 con 0x7efc24071070 2023-06-22T08:07:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc29fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc241ea840 con 0x7efc240c69e0 2023-06-22T08:07:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc237fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc24071070 0x7efc241ea300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc237fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc24071070 0x7efc241ea300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:45408/0 (socket says 172.21.15.23:45408) 2023-06-22T08:07:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc237fe700 1 -- 172.21.15.23:0/3902327760 learned_addr learned my addr 172.21.15.23:0/3902327760 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc1bfff700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efc240c69e0 0x7efc241452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.872+0000 7efc23fff700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc2414f850 0x7efc2406d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc237fe700 1 -- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efc240c69e0 msgr2=0x7efc241452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc237fe700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efc240c69e0 0x7efc241452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc237fe700 1 -- 172.21.15.23:0/3902327760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc2414f850 msgr2=0x7efc2406d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc237fe700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc2414f850 0x7efc2406d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc237fe700 1 -- 172.21.15.23:0/3902327760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0c0094a0 con 0x7efc24071070 2023-06-22T08:07:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc237fe700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc24071070 0x7efc241ea300 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efc1400c500 tx=0x7efc1400c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc20ff9700 1 -- 172.21.15.23:0/3902327760 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc14005d70 con 0x7efc24071070 2023-06-22T08:07:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc20ff9700 1 -- 172.21.15.23:0/3902327760 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1400ce90 con 0x7efc24071070 2023-06-22T08:07:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc29fff700 1 -- 172.21.15.23:0/3902327760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2406a8e0 con 0x7efc24071070 2023-06-22T08:07:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc20ff9700 1 -- 172.21.15.23:0/3902327760 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc140072f0 con 0x7efc24071070 2023-06-22T08:07:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.873+0000 7efc29fff700 1 -- 172.21.15.23:0/3902327760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc241ece50 con 0x7efc24071070 2023-06-22T08:07:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.874+0000 7efc29fff700 1 -- 172.21.15.23:0/3902327760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2404ea20 con 0x7efc24071070 2023-06-22T08:07:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.875+0000 7efc20ff9700 1 -- 172.21.15.23:0/3902327760 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efc14007450 con 0x7efc24071070 2023-06-22T08:07:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.876+0000 7efc20ff9700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc0406ace0 0x7efc0406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.876+0000 7efc20ff9700 1 -- 172.21.15.23:0/3902327760 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7efc14088dd0 con 0x7efc24071070 2023-06-22T08:07:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.876+0000 7efc1bfff700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc0406ace0 0x7efc0406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.877+0000 7efc1bfff700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc0406ace0 0x7efc0406d190 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009fd0 tx=0x7efc0c01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:35.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:35.881+0000 7efc20ff9700 1 -- 172.21.15.23:0/3902327760 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efc14053900 con 0x7efc24071070 2023-06-22T08:07:36.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.136+0000 7efc29fff700 1 -- 172.21.15.23:0/3902327760 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc240cb300 con 0x7efc24071070 2023-06-22T08:07:36.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.137+0000 7efc20ff9700 1 -- 172.21.15.23:0/3902327760 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5502 (secure 0 0 0) 0x7efc14017020 con 0x7efc24071070 2023-06-22T08:07:36.140 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:36.140 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:30.688655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 -- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc0406ace0 msgr2=0x7efc0406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc0406ace0 0x7efc0406d190 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009fd0 tx=0x7efc0c01c040 comp rx=0 tx=0).stop 2023-06-22T08:07:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 -- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc24071070 msgr2=0x7efc241ea300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc24071070 0x7efc241ea300 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efc1400c500 tx=0x7efc1400c810 comp rx=0 tx=0).stop 2023-06-22T08:07:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 -- 172.21.15.23:0/3902327760 shutdown_connections 2023-06-22T08:07:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc2414f850 0x7efc2406d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:36.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc0406ace0 0x7efc0406d190 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:36.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc24071070 0x7efc241ea300 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:36.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 --2- 172.21.15.23:0/3902327760 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efc240c69e0 0x7efc241452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:36.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 -- 172.21.15.23:0/3902327760 >> 172.21.15.23:0/3902327760 conn(0x7efc240ca3f0 msgr2=0x7efc240c5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:36.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 -- 172.21.15.23:0/3902327760 shutdown_connections 2023-06-22T08:07:36.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:36.139+0000 7efc29fff700 1 -- 172.21.15.23:0/3902327760 wait complete. 2023-06-22T08:07:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 59 2023-06-22T08:07:36.161 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 59, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 59, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4279": { "addr": "172.21.15.67:6836/1962314516", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 1962314516, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 1962314516, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4279, "incarnation": 10, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4285": { "addr": "172.21.15.67:6839/567663545", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 567663545, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 567663545, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5060": { "addr": "172.21.15.67:6837/1551460358", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1551460358, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1551460358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5060, "incarnation": 46, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 31 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:07:30.688655+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 4279, "mds_2": 5551, "mds_3": 5060, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:07:37.971 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:07:37.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:07:38.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:38.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.361+0000 7f7bfd7ff700 1 Processor -- start 2023-06-22T08:07:38.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.361+0000 7f7bfd7ff700 1 -- start start 2023-06-22T08:07:38.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.362+0000 7f7bfd7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7bf80c3a40 0x7f7bf80c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:38.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.363+0000 7f7bfd7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf80c6a60 con 0x7f7bf815a320 2023-06-22T08:07:38.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.363+0000 7f7bfd7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf814d690 con 0x7f7bf806d050 2023-06-22T08:07:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.363+0000 7f7bfd7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf8157570 con 0x7f7bf80c3a40 2023-06-22T08:07:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.363+0000 7f7bf6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7bf80c3a40 0x7f7bf80c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.363+0000 7f7bf77fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7bf815a320 0x7f7bf8160ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33682/0 (socket says 172.21.15.23:33682) 2023-06-22T08:07:38.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.363+0000 7f7bf77fe700 1 -- 172.21.15.23:0/640675393 learned_addr learned my addr 172.21.15.23:0/640675393 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.363+0000 7f7bf6ffd700 1 -- 172.21.15.23:0/640675393 >> v1:172.21.15.67:6789/0 conn(0x7f7bf806d050 legacy=0x7f7bf8159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.363+0000 7f7bf6ffd700 1 -- 172.21.15.23:0/640675393 >> v1:172.21.15.23:6789/0 conn(0x7f7bf815a320 legacy=0x7f7bf8160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:38.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.363+0000 7f7bf6ffd700 1 -- 172.21.15.23:0/640675393 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf80c6430 con 0x7f7bf80c3a40 2023-06-22T08:07:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.363+0000 7f7bf67fc700 1 -- 172.21.15.23:0/640675393 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272972690 0 0) 0x7f7bf814d690 con 0x7f7bf806d050 2023-06-22T08:07:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.364+0000 7f7bf6ffd700 1 --2- 172.21.15.23:0/640675393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bf80c3a40 0x7f7bf80c3e10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009840 tx=0x7f7be8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db557e9f2685aa14 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.364+0000 7f7bf67fc700 1 -- 172.21.15.23:0/640675393 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8011830 con 0x7f7bf80c3a40 2023-06-22T08:07:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.364+0000 7f7bf67fc700 1 -- 172.21.15.23:0/640675393 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8011990 con 0x7f7bf80c3a40 2023-06-22T08:07:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.364+0000 7f7bf67fc700 1 -- 172.21.15.23:0/640675393 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8011d20 con 0x7f7bf80c3a40 2023-06-22T08:07:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.364+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/640675393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bf80c3a40 msgr2=0x7f7bf80c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:38.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.364+0000 7f7bfd7ff700 1 --2- 172.21.15.23:0/640675393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bf80c3a40 0x7f7bf80c3e10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009840 tx=0x7f7be8009b50 comp rx=0 tx=0).stop 2023-06-22T08:07:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.364+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/640675393 shutdown_connections 2023-06-22T08:07:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.364+0000 7f7bfd7ff700 1 --2- 172.21.15.23:0/640675393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bf80c3a40 0x7f7bf80c3e10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.364+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/640675393 >> 172.21.15.23:0/640675393 conn(0x7f7bf80ca3f0 msgr2=0x7f7bf80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.365+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/640675393 shutdown_connections 2023-06-22T08:07:38.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.365+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/640675393 wait complete. 2023-06-22T08:07:38.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.366+0000 7f7bfd7ff700 1 Processor -- start 2023-06-22T08:07:38.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.366+0000 7f7bfd7ff700 1 -- start start 2023-06-22T08:07:38.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.367+0000 7f7bfd7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf806d050 0x7f7bf81ec310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:38.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.367+0000 7f7bfd7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bf80c3a40 0x7f7bf81ed780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.367+0000 7f7bfd7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7bf815a320 0x7f7bf814e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.367+0000 7f7bfd7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf814e8d0 con 0x7f7bf80c3a40 2023-06-22T08:07:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.367+0000 7f7bfd7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf814df00 con 0x7f7bf806d050 2023-06-22T08:07:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.367+0000 7f7bfd7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf81ec900 con 0x7f7bf815a320 2023-06-22T08:07:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7beffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bf80c3a40 0x7f7bf81ed780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7bf6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf806d050 0x7f7bf81ec310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:38.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7bf6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf806d050 0x7f7bf81ec310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:54428/0 (socket says 172.21.15.23:54428) 2023-06-22T08:07:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7bf6ffd700 1 -- 172.21.15.23:0/1887409310 learned_addr learned my addr 172.21.15.23:0/1887409310 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7bf77fe700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7bf815a320 0x7f7bf814e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7beffff700 1 -- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7bf815a320 msgr2=0x7f7bf814e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7beffff700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7bf815a320 0x7f7bf814e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7beffff700 1 -- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf806d050 msgr2=0x7f7bf81ec310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7beffff700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf806d050 0x7f7bf81ec310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7beffff700 1 -- 172.21.15.23:0/1887409310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be80094a0 con 0x7f7bf80c3a40 2023-06-22T08:07:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.368+0000 7f7beffff700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bf80c3a40 0x7f7bf81ed780 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7be000c550 tx=0x7f7be000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.369+0000 7f7bef7fe700 1 -- 172.21.15.23:0/1887409310 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be00054e0 con 0x7f7bf80c3a40 2023-06-22T08:07:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.369+0000 7f7bef7fe700 1 -- 172.21.15.23:0/1887409310 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be0003680 con 0x7f7bf80c3a40 2023-06-22T08:07:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.369+0000 7f7bef7fe700 1 -- 172.21.15.23:0/1887409310 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be0003a30 con 0x7f7bf80c3a40 2023-06-22T08:07:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.369+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/1887409310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf81ecbc0 con 0x7f7bf80c3a40 2023-06-22T08:07:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.369+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/1887409310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf81ed180 con 0x7f7bf80c3a40 2023-06-22T08:07:38.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.370+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/1887409310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf814e1e0 con 0x7f7bf80c3a40 2023-06-22T08:07:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.371+0000 7f7bef7fe700 1 -- 172.21.15.23:0/1887409310 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7be0003b90 con 0x7f7bf80c3a40 2023-06-22T08:07:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.372+0000 7f7bef7fe700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bd806ad10 0x7f7bd806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:38.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.372+0000 7f7bef7fe700 1 -- 172.21.15.23:0/1887409310 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f7be0088c30 con 0x7f7bf80c3a40 2023-06-22T08:07:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.372+0000 7f7bf6ffd700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bd806ad10 0x7f7bd806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:38.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.373+0000 7f7bf6ffd700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bd806ad10 0x7f7bd806d1c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009b90 tx=0x7f7be801c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:38.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.376+0000 7f7bef7fe700 1 -- 172.21.15.23:0/1887409310 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7be0053810 con 0x7f7bf80c3a40 2023-06-22T08:07:38.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.629+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/1887409310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bf804ea20 con 0x7f7bf80c3a40 2023-06-22T08:07:38.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.630+0000 7f7bef7fe700 1 -- 172.21.15.23:0/1887409310 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5499 (secure 0 0 0) 0x7f7be0017020 con 0x7f7bf80c3a40 2023-06-22T08:07:38.634 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:38.634 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:36.372964+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:38.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bd806ad10 msgr2=0x7f7bd806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:38.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bd806ad10 0x7f7bd806d1c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009b90 tx=0x7f7be801c040 comp rx=0 tx=0).stop 2023-06-22T08:07:38.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/1887409310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bf80c3a40 msgr2=0x7f7bf81ed780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bf80c3a40 0x7f7bf81ed780 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7be000c550 tx=0x7f7be000c860 comp rx=0 tx=0).stop 2023-06-22T08:07:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/1887409310 shutdown_connections 2023-06-22T08:07:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7bf815a320 0x7f7bf814e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7bd806ad10 0x7f7bd806d1c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7bf806d050 0x7f7bf81ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 --2- 172.21.15.23:0/1887409310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bf80c3a40 0x7f7bf81ed780 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/1887409310 >> 172.21.15.23:0/1887409310 conn(0x7f7bf80ca3f0 msgr2=0x7f7bf814af60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/1887409310 shutdown_connections 2023-06-22T08:07:38.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:38.633+0000 7f7bfd7ff700 1 -- 172.21.15.23:0/1887409310 wait complete. 2023-06-22T08:07:38.638 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 60 2023-06-22T08:07:38.654 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:07:40.655 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:40.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.862+0000 7fde8dbff700 1 Processor -- start 2023-06-22T08:07:40.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.862+0000 7fde8dbff700 1 -- start start 2023-06-22T08:07:40.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.863+0000 7fde8dbff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fde880c9fd0 0x7fde880cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.863+0000 7fde8dbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde880c40b0 0x7fde880c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.863+0000 7fde8dbff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8815c370 con 0x7fde88152ce0 2023-06-22T08:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.863+0000 7fde8dbff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde881530b0 con 0x7fde880c9fd0 2023-06-22T08:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.863+0000 7fde8dbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde880cf6f0 con 0x7fde880c40b0 2023-06-22T08:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.863+0000 7fde877fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fde88152ce0 0x7fde8815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33690/0 (socket says 172.21.15.23:33690) 2023-06-22T08:07:40.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.863+0000 7fde877fe700 1 -- 172.21.15.23:0/1727666663 learned_addr learned my addr 172.21.15.23:0/1727666663 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:40.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.863+0000 7fde87fff700 1 --2- 172.21.15.23:0/1727666663 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde880c40b0 0x7fde880c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:40.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.863+0000 7fde86ffd700 1 --2- 172.21.15.23:0/1727666663 >> v2:172.21.15.67:3300/0 conn(0x7fde880c9fd0 0x7fde880cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde867fc700 1 -- 172.21.15.23:0/1727666663 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037656878 0 0) 0x7fde8815c370 con 0x7fde88152ce0 2023-06-22T08:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde867fc700 1 -- 172.21.15.23:0/1727666663 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde700035b0 con 0x7fde88152ce0 2023-06-22T08:07:40.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde86ffd700 1 -- 172.21.15.23:0/1727666663 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde880c40b0 msgr2=0x7fde880c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde86ffd700 1 --2- 172.21.15.23:0/1727666663 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde880c40b0 0x7fde880c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde86ffd700 1 -- 172.21.15.23:0/1727666663 >> v1:172.21.15.23:6789/0 conn(0x7fde88152ce0 legacy=0x7fde8815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde86ffd700 1 -- 172.21.15.23:0/1727666663 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8815bda0 con 0x7fde880c9fd0 2023-06-22T08:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde87fff700 1 --2- 172.21.15.23:0/1727666663 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde880c40b0 0x7fde880c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde86ffd700 1 --2- 172.21.15.23:0/1727666663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde880c9fd0 0x7fde880cf1b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009840 tx=0x7fde7c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f40bc89c932d841 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde867fc700 1 -- 172.21.15.23:0/1727666663 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c011830 con 0x7fde880c9fd0 2023-06-22T08:07:40.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde867fc700 1 -- 172.21.15.23:0/1727666663 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c011990 con 0x7fde880c9fd0 2023-06-22T08:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.864+0000 7fde867fc700 1 -- 172.21.15.23:0/1727666663 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c011d20 con 0x7fde880c9fd0 2023-06-22T08:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.865+0000 7fde8dbff700 1 -- 172.21.15.23:0/1727666663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde880c9fd0 msgr2=0x7fde880cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.865+0000 7fde8dbff700 1 --2- 172.21.15.23:0/1727666663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde880c9fd0 0x7fde880cf1b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009840 tx=0x7fde7c009b50 comp rx=0 tx=0).stop 2023-06-22T08:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.865+0000 7fde8dbff700 1 -- 172.21.15.23:0/1727666663 shutdown_connections 2023-06-22T08:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.865+0000 7fde8dbff700 1 --2- 172.21.15.23:0/1727666663 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde880c40b0 0x7fde880c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:40.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.865+0000 7fde8dbff700 1 --2- 172.21.15.23:0/1727666663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde880c9fd0 0x7fde880cf1b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:40.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.865+0000 7fde8dbff700 1 -- 172.21.15.23:0/1727666663 >> 172.21.15.23:0/1727666663 conn(0x7fde8806ae30 msgr2=0x7fde8806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:40.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.865+0000 7fde8dbff700 1 -- 172.21.15.23:0/1727666663 shutdown_connections 2023-06-22T08:07:40.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.865+0000 7fde8dbff700 1 -- 172.21.15.23:0/1727666663 wait complete. 2023-06-22T08:07:40.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.867+0000 7fde8dbff700 1 Processor -- start 2023-06-22T08:07:40.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.867+0000 7fde8dbff700 1 -- start start 2023-06-22T08:07:40.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde8dbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde880c40b0 0x7fde881f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:40.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde8dbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde880c9fd0 0x7fde880ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:40.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde8dbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde88152ce0 0x7fde880cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:40.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde8dbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8815b700 con 0x7fde880c40b0 2023-06-22T08:07:40.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde8dbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde881f1da0 con 0x7fde880c9fd0 2023-06-22T08:07:40.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde8dbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde881599d0 con 0x7fde88152ce0 2023-06-22T08:07:40.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde877fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde880c40b0 0x7fde881f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:40.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde877fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde880c40b0 0x7fde881f1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45048/0 (socket says 172.21.15.23:45048) 2023-06-22T08:07:40.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde877fe700 1 -- 172.21.15.23:0/3787530401 learned_addr learned my addr 172.21.15.23:0/3787530401 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:40.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde86ffd700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde880c9fd0 0x7fde880ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:40.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.868+0000 7fde87fff700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde88152ce0 0x7fde880cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:40.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde877fe700 1 -- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde88152ce0 msgr2=0x7fde880cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:40.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde877fe700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde88152ce0 0x7fde880cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:40.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde877fe700 1 -- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde880c9fd0 msgr2=0x7fde880ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:40.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde877fe700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde880c9fd0 0x7fde880ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:40.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde877fe700 1 -- 172.21.15.23:0/3787530401 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde80009a70 con 0x7fde880c40b0 2023-06-22T08:07:40.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde877fe700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde880c40b0 0x7fde881f1370 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fde7800c4d0 tx=0x7fde7800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:40.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde6ffff700 1 -- 172.21.15.23:0/3787530401 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78005d70 con 0x7fde880c40b0 2023-06-22T08:07:40.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde6ffff700 1 -- 172.21.15.23:0/3787530401 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7800ce90 con 0x7fde880c40b0 2023-06-22T08:07:40.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde8dbff700 1 -- 172.21.15.23:0/3787530401 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c0094a0 con 0x7fde880c40b0 2023-06-22T08:07:40.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde6ffff700 1 -- 172.21.15.23:0/3787530401 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde780072f0 con 0x7fde880c40b0 2023-06-22T08:07:40.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.869+0000 7fde8dbff700 1 -- 172.21.15.23:0/3787530401 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8815b950 con 0x7fde880c40b0 2023-06-22T08:07:40.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.871+0000 7fde8dbff700 1 -- 172.21.15.23:0/3787530401 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8815bb40 con 0x7fde880c40b0 2023-06-22T08:07:40.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.871+0000 7fde6ffff700 1 -- 172.21.15.23:0/3787530401 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fde78007450 con 0x7fde880c40b0 2023-06-22T08:07:40.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.872+0000 7fde6ffff700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde7406ace0 0x7fde7406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:40.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.872+0000 7fde6ffff700 1 -- 172.21.15.23:0/3787530401 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7fde78088cc0 con 0x7fde880c40b0 2023-06-22T08:07:40.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.873+0000 7fde86ffd700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde7406ace0 0x7fde7406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:40.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.874+0000 7fde86ffd700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde7406ace0 0x7fde7406d190 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fde7c000c00 tx=0x7fde7c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:40.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:40.877+0000 7fde6ffff700 1 -- 172.21.15.23:0/3787530401 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fde78053910 con 0x7fde880c40b0 2023-06-22T08:07:41.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.132+0000 7fde8dbff700 1 -- 172.21.15.23:0/3787530401 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde8804ea20 con 0x7fde880c40b0 2023-06-22T08:07:41.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.132+0000 7fde6ffff700 1 -- 172.21.15.23:0/3787530401 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5499 (secure 0 0 0) 0x7fde78056f30 con 0x7fde880c40b0 2023-06-22T08:07:41.136 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:41.137 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:36.372964+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:41.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 -- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde7406ace0 msgr2=0x7fde7406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:41.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde7406ace0 0x7fde7406d190 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fde7c000c00 tx=0x7fde7c008030 comp rx=0 tx=0).stop 2023-06-22T08:07:41.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 -- 172.21.15.23:0/3787530401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde880c40b0 msgr2=0x7fde881f1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:41.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde880c40b0 0x7fde881f1370 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fde7800c4d0 tx=0x7fde7800c7e0 comp rx=0 tx=0).stop 2023-06-22T08:07:41.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 -- 172.21.15.23:0/3787530401 shutdown_connections 2023-06-22T08:07:41.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde88152ce0 0x7fde880cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:41.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde880c40b0 0x7fde881f1370 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:41.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde7406ace0 0x7fde7406d190 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:41.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 --2- 172.21.15.23:0/3787530401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde880c9fd0 0x7fde880ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:41.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 -- 172.21.15.23:0/3787530401 >> 172.21.15.23:0/3787530401 conn(0x7fde8806ae30 msgr2=0x7fde8814f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:41.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 -- 172.21.15.23:0/3787530401 shutdown_connections 2023-06-22T08:07:41.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:41.135+0000 7fde8dbff700 1 -- 172.21.15.23:0/3787530401 wait complete. 2023-06-22T08:07:41.141 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 60 2023-06-22T08:07:41.156 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:07:43.157 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:43.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.367+0000 7fe421dff700 1 Processor -- start 2023-06-22T08:07:43.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.367+0000 7fe421dff700 1 -- start start 2023-06-22T08:07:43.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.369+0000 7fe421dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 0x7fe41c0c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:43.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.369+0000 7fe421dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41c0c51a0 con 0x7fe41c0c69e0 2023-06-22T08:07:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.369+0000 7fe421dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41c0c44e0 con 0x7fe41c071070 2023-06-22T08:07:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.369+0000 7fe421dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0c5440 con 0x7fe41c14f850 2023-06-22T08:07:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.369+0000 7fe41affd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe41c0c69e0 0x7fe41c155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33702/0 (socket says 172.21.15.23:33702) 2023-06-22T08:07:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.369+0000 7fe41affd700 1 -- 172.21.15.23:0/2336886586 learned_addr learned my addr 172.21.15.23:0/2336886586 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.369+0000 7fe41b7fe700 1 --2- 172.21.15.23:0/2336886586 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 0x7fe41c0c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.370+0000 7fe41a7fc700 1 -- 172.21.15.23:0/2336886586 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714300444 0 0) 0x7fe41c0c44e0 con 0x7fe41c071070 2023-06-22T08:07:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.370+0000 7fe41a7fc700 1 -- 172.21.15.23:0/2336886586 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4040035b0 con 0x7fe41c071070 2023-06-22T08:07:43.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.370+0000 7fe41a7fc700 1 -- 172.21.15.23:0/2336886586 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758386152 0 0) 0x7fe41c0c51a0 con 0x7fe41c0c69e0 2023-06-22T08:07:43.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.370+0000 7fe41b7fe700 1 -- 172.21.15.23:0/2336886586 >> v1:172.21.15.67:6789/0 conn(0x7fe41c071070 legacy=0x7fe41c15fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:43.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.370+0000 7fe41b7fe700 1 -- 172.21.15.23:0/2336886586 >> v1:172.21.15.23:6789/0 conn(0x7fe41c0c69e0 legacy=0x7fe41c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:43.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.370+0000 7fe41b7fe700 1 -- 172.21.15.23:0/2336886586 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c0c5720 con 0x7fe41c14f850 2023-06-22T08:07:43.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.370+0000 7fe41b7fe700 1 --2- 172.21.15.23:0/2336886586 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 0x7fe41c0c3080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009870 tx=0x7fe40c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7ee9ae2c43db87a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.371+0000 7fe41a7fc700 1 -- 172.21.15.23:0/2336886586 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c011830 con 0x7fe41c14f850 2023-06-22T08:07:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.371+0000 7fe41a7fc700 1 -- 172.21.15.23:0/2336886586 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40c011990 con 0x7fe41c14f850 2023-06-22T08:07:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.371+0000 7fe41a7fc700 1 -- 172.21.15.23:0/2336886586 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c011d00 con 0x7fe41c14f850 2023-06-22T08:07:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.371+0000 7fe421dff700 1 -- 172.21.15.23:0/2336886586 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 msgr2=0x7fe41c0c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.371+0000 7fe421dff700 1 --2- 172.21.15.23:0/2336886586 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 0x7fe41c0c3080 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009870 tx=0x7fe40c009b80 comp rx=0 tx=0).stop 2023-06-22T08:07:43.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.371+0000 7fe421dff700 1 -- 172.21.15.23:0/2336886586 shutdown_connections 2023-06-22T08:07:43.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.371+0000 7fe421dff700 1 --2- 172.21.15.23:0/2336886586 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 0x7fe41c0c3080 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:43.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.371+0000 7fe421dff700 1 -- 172.21.15.23:0/2336886586 >> 172.21.15.23:0/2336886586 conn(0x7fe41c0ca3f0 msgr2=0x7fe41c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:43.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.371+0000 7fe421dff700 1 -- 172.21.15.23:0/2336886586 shutdown_connections 2023-06-22T08:07:43.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.371+0000 7fe421dff700 1 -- 172.21.15.23:0/2336886586 wait complete. 2023-06-22T08:07:43.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.373+0000 7fe421dff700 1 Processor -- start 2023-06-22T08:07:43.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.373+0000 7fe421dff700 1 -- start start 2023-06-22T08:07:43.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.374+0000 7fe421dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe41c071070 0x7fe41c0c9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:43.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.374+0000 7fe421dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe41c0c69e0 0x7fe41c1dd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:43.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.374+0000 7fe421dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 0x7fe41c069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:43.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.374+0000 7fe421dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c14d920 con 0x7fe41c0c69e0 2023-06-22T08:07:43.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.374+0000 7fe421dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0c1090 con 0x7fe41c071070 2023-06-22T08:07:43.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.374+0000 7fe421dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c0c8d80 con 0x7fe41c14f850 2023-06-22T08:07:43.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.374+0000 7fe41affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe41c0c69e0 0x7fe41c1dd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:43.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.374+0000 7fe41b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe41c071070 0x7fe41c0c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:43.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.374+0000 7fe41affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe41c0c69e0 0x7fe41c1dd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45050/0 (socket says 172.21.15.23:45050) 2023-06-22T08:07:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.374+0000 7fe41affd700 1 -- 172.21.15.23:0/3035900623 learned_addr learned my addr 172.21.15.23:0/3035900623 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.375+0000 7fe41bfff700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 0x7fe41c069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.375+0000 7fe41affd700 1 -- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 msgr2=0x7fe41c069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.375+0000 7fe41affd700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 0x7fe41c069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.375+0000 7fe41affd700 1 -- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe41c071070 msgr2=0x7fe41c0c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.375+0000 7fe41affd700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe41c071070 0x7fe41c0c9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.375+0000 7fe41affd700 1 -- 172.21.15.23:0/3035900623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c0094a0 con 0x7fe41c0c69e0 2023-06-22T08:07:43.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.375+0000 7fe41affd700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe41c0c69e0 0x7fe41c1dd9f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe41000c540 tx=0x7fe41000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:43.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.375+0000 7fe403fff700 1 -- 172.21.15.23:0/3035900623 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410004c80 con 0x7fe41c0c69e0 2023-06-22T08:07:43.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.375+0000 7fe403fff700 1 -- 172.21.15.23:0/3035900623 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe410004de0 con 0x7fe41c0c69e0 2023-06-22T08:07:43.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.375+0000 7fe421dff700 1 -- 172.21.15.23:0/3035900623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41c0c9750 con 0x7fe41c0c69e0 2023-06-22T08:07:43.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.376+0000 7fe403fff700 1 -- 172.21.15.23:0/3035900623 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410007310 con 0x7fe41c0c69e0 2023-06-22T08:07:43.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.376+0000 7fe421dff700 1 -- 172.21.15.23:0/3035900623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41c14ea70 con 0x7fe41c0c69e0 2023-06-22T08:07:43.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.377+0000 7fe421dff700 1 -- 172.21.15.23:0/3035900623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe41c1530d0 con 0x7fe41c0c69e0 2023-06-22T08:07:43.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.378+0000 7fe403fff700 1 -- 172.21.15.23:0/3035900623 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe410007470 con 0x7fe41c0c69e0 2023-06-22T08:07:43.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.379+0000 7fe403fff700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe40806ace0 0x7fe40806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.379+0000 7fe403fff700 1 -- 172.21.15.23:0/3035900623 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7fe4100894f0 con 0x7fe41c0c69e0 2023-06-22T08:07:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.379+0000 7fe41b7fe700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe40806ace0 0x7fe40806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.380+0000 7fe41b7fe700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe40806ace0 0x7fe40806d190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0097c0 tx=0x7fe40c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:43.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.383+0000 7fe403fff700 1 -- 172.21.15.23:0/3035900623 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe410054140 con 0x7fe41c0c69e0 2023-06-22T08:07:43.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.636+0000 7fe421dff700 1 -- 172.21.15.23:0/3035900623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe41c06cc00 con 0x7fe41c0c69e0 2023-06-22T08:07:43.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.637+0000 7fe403fff700 1 -- 172.21.15.23:0/3035900623 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5505 (secure 0 0 0) 0x7fe410057760 con 0x7fe41c0c69e0 2023-06-22T08:07:43.641 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:43.641 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:41.744252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.639+0000 7fe421dff700 1 -- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe40806ace0 msgr2=0x7fe40806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.639+0000 7fe421dff700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe40806ace0 0x7fe40806d190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0097c0 tx=0x7fe40c008030 comp rx=0 tx=0).stop 2023-06-22T08:07:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.639+0000 7fe421dff700 1 -- 172.21.15.23:0/3035900623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe41c0c69e0 msgr2=0x7fe41c1dd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.639+0000 7fe421dff700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe41c0c69e0 0x7fe41c1dd9f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe41000c540 tx=0x7fe41000c850 comp rx=0 tx=0).stop 2023-06-22T08:07:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.639+0000 7fe421dff700 1 -- 172.21.15.23:0/3035900623 shutdown_connections 2023-06-22T08:07:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.639+0000 7fe421dff700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe41c14f850 0x7fe41c069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.639+0000 7fe421dff700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe40806ace0 0x7fe40806d190 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.639+0000 7fe421dff700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe41c071070 0x7fe41c0c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.639+0000 7fe421dff700 1 --2- 172.21.15.23:0/3035900623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe41c0c69e0 0x7fe41c1dd9f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.639+0000 7fe421dff700 1 -- 172.21.15.23:0/3035900623 >> 172.21.15.23:0/3035900623 conn(0x7fe41c0ca3f0 msgr2=0x7fe41c1e0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.640+0000 7fe421dff700 1 -- 172.21.15.23:0/3035900623 shutdown_connections 2023-06-22T08:07:43.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:43.640+0000 7fe421dff700 1 -- 172.21.15.23:0/3035900623 wait complete. 2023-06-22T08:07:43.645 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 61 2023-06-22T08:07:43.661 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:07:45.662 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.868+0000 7f49075ff700 1 Processor -- start 2023-06-22T08:07:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.868+0000 7f49075ff700 1 -- start start 2023-06-22T08:07:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.869+0000 7f49075ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4900152ce0 0x7f49000cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.869+0000 7f49075ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f49000c9fd0 0x7f490015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.869+0000 7f49075ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490015c370 con 0x7f4900158880 2023-06-22T08:07:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.869+0000 7f49075ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49001530b0 con 0x7f49000c9fd0 2023-06-22T08:07:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.869+0000 7f49075ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49000cf6f0 con 0x7f4900152ce0 2023-06-22T08:07:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.870+0000 7f4904b9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f49000c9fd0 0x7f490015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:45.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.870+0000 7f4904b9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f49000c9fd0 0x7f490015c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45052/0 (socket says 172.21.15.23:45052) 2023-06-22T08:07:45.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.870+0000 7f4904b9a700 1 -- 172.21.15.23:0/1857708795 learned_addr learned my addr 172.21.15.23:0/1857708795 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.870+0000 7f490539b700 1 --2- 172.21.15.23:0/1857708795 >> v2:172.21.15.67:3300/0 conn(0x7f4900152ce0 0x7f49000cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.870+0000 7f4904b9a700 1 -- 172.21.15.23:0/1857708795 >> v2:172.21.15.67:3300/0 conn(0x7f4900152ce0 msgr2=0x7f49000cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:07:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.870+0000 7f4904b9a700 1 --2- 172.21.15.23:0/1857708795 >> v2:172.21.15.67:3300/0 conn(0x7f4900152ce0 0x7f49000cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.870+0000 7f4904b9a700 1 -- 172.21.15.23:0/1857708795 >> v1:172.21.15.67:6789/0 conn(0x7f4900158880 legacy=0x7f4900158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:45.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.870+0000 7f4904b9a700 1 -- 172.21.15.23:0/1857708795 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490015bda0 con 0x7f49000c9fd0 2023-06-22T08:07:45.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.870+0000 7f4904b9a700 1 --2- 172.21.15.23:0/1857708795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49000c9fd0 0x7f490015c800 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f490008ba10 tx=0x7f48f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c43a41128b88a0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:45.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.871+0000 7f48f7fff700 1 -- 172.21.15.23:0/1857708795 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f000b870 con 0x7f49000c9fd0 2023-06-22T08:07:45.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.871+0000 7f48f7fff700 1 -- 172.21.15.23:0/1857708795 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f000b9d0 con 0x7f49000c9fd0 2023-06-22T08:07:45.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.871+0000 7f48f7fff700 1 -- 172.21.15.23:0/1857708795 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f000bd60 con 0x7f49000c9fd0 2023-06-22T08:07:45.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.871+0000 7f49075ff700 1 -- 172.21.15.23:0/1857708795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49000c9fd0 msgr2=0x7f490015c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:45.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.871+0000 7f49075ff700 1 --2- 172.21.15.23:0/1857708795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49000c9fd0 0x7f490015c800 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f490008ba10 tx=0x7f48f0009bd0 comp rx=0 tx=0).stop 2023-06-22T08:07:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.871+0000 7f49075ff700 1 -- 172.21.15.23:0/1857708795 shutdown_connections 2023-06-22T08:07:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.871+0000 7f49075ff700 1 --2- 172.21.15.23:0/1857708795 >> v2:172.21.15.67:3300/0 conn(0x7f4900152ce0 0x7f49000cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.871+0000 7f49075ff700 1 --2- 172.21.15.23:0/1857708795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49000c9fd0 0x7f490015c800 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.871+0000 7f49075ff700 1 -- 172.21.15.23:0/1857708795 >> 172.21.15.23:0/1857708795 conn(0x7f490006ae30 msgr2=0x7f490006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.871+0000 7f49075ff700 1 -- 172.21.15.23:0/1857708795 shutdown_connections 2023-06-22T08:07:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.872+0000 7f49075ff700 1 -- 172.21.15.23:0/1857708795 wait complete. 2023-06-22T08:07:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.873+0000 7f49075ff700 1 Processor -- start 2023-06-22T08:07:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.873+0000 7f49075ff700 1 -- start start 2023-06-22T08:07:45.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f49075ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f49000c9fd0 0x7f4900154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:45.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f49075ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4900152ce0 0x7f49001559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:45.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f49075ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4900158880 0x7f49000cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:45.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f49075ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001583e0 con 0x7f4900152ce0 2023-06-22T08:07:45.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f49075ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4900155460 con 0x7f4900158880 2023-06-22T08:07:45.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f49075ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4900154110 con 0x7f49000c9fd0 2023-06-22T08:07:45.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f4904b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4900152ce0 0x7f49001559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:45.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f4905b9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4900158880 0x7f49000cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:45.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f4904b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4900152ce0 0x7f49001559f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45056/0 (socket says 172.21.15.23:45056) 2023-06-22T08:07:45.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f4904b9a700 1 -- 172.21.15.23:0/550705219 learned_addr learned my addr 172.21.15.23:0/550705219 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:45.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.874+0000 7f490539b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f49000c9fd0 0x7f4900154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:45.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.875+0000 7f4904b9a700 1 -- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f49000c9fd0 msgr2=0x7f4900154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:45.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.875+0000 7f4904b9a700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f49000c9fd0 0x7f4900154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:45.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.875+0000 7f4904b9a700 1 -- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4900158880 msgr2=0x7f49000cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:45.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.875+0000 7f4904b9a700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4900158880 0x7f49000cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:45.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.875+0000 7f4904b9a700 1 -- 172.21.15.23:0/550705219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f0009550 con 0x7f4900152ce0 2023-06-22T08:07:45.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.875+0000 7f4904b9a700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4900152ce0 0x7f49001559f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f490008bc30 tx=0x7f48f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:45.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.875+0000 7f48f5ffb700 1 -- 172.21.15.23:0/550705219 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0004310 con 0x7f4900152ce0 2023-06-22T08:07:45.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.875+0000 7f48f5ffb700 1 -- 172.21.15.23:0/550705219 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f001f070 con 0x7f4900152ce0 2023-06-22T08:07:45.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.875+0000 7f49075ff700 1 -- 172.21.15.23:0/550705219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4900157530 con 0x7f4900152ce0 2023-06-22T08:07:45.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.875+0000 7f48f5ffb700 1 -- 172.21.15.23:0/550705219 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f00036a0 con 0x7f4900152ce0 2023-06-22T08:07:45.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.876+0000 7f49075ff700 1 -- 172.21.15.23:0/550705219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4900157ab0 con 0x7f4900152ce0 2023-06-22T08:07:45.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.877+0000 7f49075ff700 1 -- 172.21.15.23:0/550705219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490004ea20 con 0x7f4900152ce0 2023-06-22T08:07:45.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.878+0000 7f48f5ffb700 1 -- 172.21.15.23:0/550705219 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f48f0021030 con 0x7f4900152ce0 2023-06-22T08:07:45.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.879+0000 7f48f5ffb700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48e406ace0 0x7f48e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:45.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.879+0000 7f48f5ffb700 1 -- 172.21.15.23:0/550705219 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f48f008aad0 con 0x7f4900152ce0 2023-06-22T08:07:45.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.879+0000 7f490539b700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48e406ace0 0x7f48e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:45.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.880+0000 7f490539b700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48e406ace0 0x7f48e406d190 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f48fc006b20 tx=0x7f48fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:45.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:45.883+0000 7f48f5ffb700 1 -- 172.21.15.23:0/550705219 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f48f0055720 con 0x7f4900152ce0 2023-06-22T08:07:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.138+0000 7f49075ff700 1 -- 172.21.15.23:0/550705219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49000c9db0 con 0x7f4900152ce0 2023-06-22T08:07:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.139+0000 7f48f5ffb700 1 -- 172.21.15.23:0/550705219 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+5505 (secure 0 0 0) 0x7f48f0058d40 con 0x7f4900152ce0 2023-06-22T08:07:46.142 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:46.142 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:41.744252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:clientreplay","state_seq":24,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:46.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 -- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48e406ace0 msgr2=0x7f48e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:46.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48e406ace0 0x7f48e406d190 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f48fc006b20 tx=0x7f48fc008040 comp rx=0 tx=0).stop 2023-06-22T08:07:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 -- 172.21.15.23:0/550705219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4900152ce0 msgr2=0x7f49001559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4900152ce0 0x7f49001559f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f490008bc30 tx=0x7f48f0004750 comp rx=0 tx=0).stop 2023-06-22T08:07:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 -- 172.21.15.23:0/550705219 shutdown_connections 2023-06-22T08:07:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48e406ace0 0x7f48e406d190 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4900158880 0x7f49000cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f49000c9fd0 0x7f4900154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 --2- 172.21.15.23:0/550705219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4900152ce0 0x7f49001559f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 -- 172.21.15.23:0/550705219 >> 172.21.15.23:0/550705219 conn(0x7f490006ae30 msgr2=0x7f490014b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 -- 172.21.15.23:0/550705219 shutdown_connections 2023-06-22T08:07:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:46.141+0000 7f49075ff700 1 -- 172.21.15.23:0/550705219 wait complete. 2023-06-22T08:07:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 61 2023-06-22T08:07:46.163 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:07:48.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:48.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.370+0000 7f908dbff700 1 Processor -- start 2023-06-22T08:07:48.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.370+0000 7f908dbff700 1 -- start start 2023-06-22T08:07:48.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.371+0000 7f908dbff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f90880aedc0 0x7f9088164290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:48.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.371+0000 7f908dbff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90880c7400 con 0x7f9088152dd0 2023-06-22T08:07:48.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.371+0000 7f908dbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90880c81c0 con 0x7f90880c8bd0 2023-06-22T08:07:48.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.371+0000 7f908dbff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90880c84a0 con 0x7f90880aedc0 2023-06-22T08:07:48.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.372+0000 7f9086ffd700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f90880c8bd0 0x7f90880cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:35100/0 (socket says 172.21.15.23:35100) 2023-06-22T08:07:48.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.372+0000 7f9087fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f90880aedc0 0x7f9088164290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:48.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.372+0000 7f9086ffd700 1 -- 172.21.15.23:0/314241863 learned_addr learned my addr 172.21.15.23:0/314241863 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:48.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.372+0000 7f90877fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9088152dd0 0x7f908806afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:45434/0 (socket says 172.21.15.23:45434) 2023-06-22T08:07:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.372+0000 7f90867fc700 1 -- 172.21.15.23:0/314241863 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299807571 0 0) 0x7f90880c81c0 con 0x7f90880c8bd0 2023-06-22T08:07:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.372+0000 7f90867fc700 1 -- 172.21.15.23:0/314241863 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90700035b0 con 0x7f90880c8bd0 2023-06-22T08:07:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.372+0000 7f90867fc700 1 -- 172.21.15.23:0/314241863 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843981523 0 0) 0x7f90880c7400 con 0x7f9088152dd0 2023-06-22T08:07:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.372+0000 7f9087fff700 1 -- 172.21.15.23:0/314241863 >> v1:172.21.15.67:6789/0 conn(0x7f90880c8bd0 legacy=0x7f90880cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.372+0000 7f9087fff700 1 -- 172.21.15.23:0/314241863 >> v1:172.21.15.23:6789/0 conn(0x7f9088152dd0 legacy=0x7f908806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.372+0000 7f9087fff700 1 -- 172.21.15.23:0/314241863 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90880cd370 con 0x7f90880aedc0 2023-06-22T08:07:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.373+0000 7f9087fff700 1 --2- 172.21.15.23:0/314241863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 0x7f9088164290 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9080009d90 tx=0x7f908000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e23def4d17e04b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.373+0000 7f90867fc700 1 -- 172.21.15.23:0/314241863 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908000e070 con 0x7f90880aedc0 2023-06-22T08:07:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.373+0000 7f90867fc700 1 -- 172.21.15.23:0/314241863 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908000bb20 con 0x7f90880aedc0 2023-06-22T08:07:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.373+0000 7f90867fc700 1 -- 172.21.15.23:0/314241863 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908000bed0 con 0x7f90880aedc0 2023-06-22T08:07:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.373+0000 7f908dbff700 1 -- 172.21.15.23:0/314241863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 msgr2=0x7f9088164290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.373+0000 7f908dbff700 1 --2- 172.21.15.23:0/314241863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 0x7f9088164290 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9080009d90 tx=0x7f908000b330 comp rx=0 tx=0).stop 2023-06-22T08:07:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.373+0000 7f908dbff700 1 -- 172.21.15.23:0/314241863 shutdown_connections 2023-06-22T08:07:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.373+0000 7f908dbff700 1 --2- 172.21.15.23:0/314241863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 0x7f9088164290 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.373+0000 7f908dbff700 1 -- 172.21.15.23:0/314241863 >> 172.21.15.23:0/314241863 conn(0x7f908806fb70 msgr2=0x7f908806ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.373+0000 7f908dbff700 1 -- 172.21.15.23:0/314241863 shutdown_connections 2023-06-22T08:07:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.374+0000 7f908dbff700 1 -- 172.21.15.23:0/314241863 wait complete. 2023-06-22T08:07:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.375+0000 7f908dbff700 1 Processor -- start 2023-06-22T08:07:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.375+0000 7f908dbff700 1 -- start start 2023-06-22T08:07:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.376+0000 7f908dbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 0x7f90880cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.376+0000 7f908dbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90880c8bd0 0x7f90881ee4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.376+0000 7f908dbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088152dd0 0x7f908814b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.376+0000 7f908dbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9088158890 con 0x7f90880aedc0 2023-06-22T08:07:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.376+0000 7f908dbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9088159130 con 0x7f9088152dd0 2023-06-22T08:07:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.376+0000 7f908dbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908815bf20 con 0x7f90880c8bd0 2023-06-22T08:07:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.376+0000 7f90877fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 0x7f90880cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.376+0000 7f90877fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 0x7f90880cbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36564/0 (socket says 172.21.15.23:36564) 2023-06-22T08:07:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.376+0000 7f90877fe700 1 -- 172.21.15.23:0/3448108327 learned_addr learned my addr 172.21.15.23:0/3448108327 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f9087fff700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088152dd0 0x7f908814b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f9086ffd700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90880c8bd0 0x7f90881ee4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f90877fe700 1 -- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90880c8bd0 msgr2=0x7f90881ee4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f90877fe700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90880c8bd0 0x7f90881ee4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f90877fe700 1 -- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088152dd0 msgr2=0x7f908814b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f90877fe700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088152dd0 0x7f908814b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f90877fe700 1 -- 172.21.15.23:0/3448108327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9080009a70 con 0x7f90880aedc0 2023-06-22T08:07:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f9086ffd700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90880c8bd0 0x7f90881ee4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:07:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f90877fe700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 0x7f90880cbd20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f907800c4a0 tx=0x7f907800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f906ffff700 1 -- 172.21.15.23:0/3448108327 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90780074e0 con 0x7f90880aedc0 2023-06-22T08:07:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f906ffff700 1 -- 172.21.15.23:0/3448108327 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9078007640 con 0x7f90880aedc0 2023-06-22T08:07:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f908dbff700 1 -- 172.21.15.23:0/3448108327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9088156c80 con 0x7f90880aedc0 2023-06-22T08:07:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f906ffff700 1 -- 172.21.15.23:0/3448108327 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9078005a50 con 0x7f90880aedc0 2023-06-22T08:07:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.377+0000 7f908dbff700 1 -- 172.21.15.23:0/3448108327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9088158dc0 con 0x7f90880aedc0 2023-06-22T08:07:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.378+0000 7f908dbff700 1 -- 172.21.15.23:0/3448108327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9088158f80 con 0x7f90880aedc0 2023-06-22T08:07:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.379+0000 7f906ffff700 1 -- 172.21.15.23:0/3448108327 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9078005bb0 con 0x7f90880aedc0 2023-06-22T08:07:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.380+0000 7f906ffff700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f907406ace0 0x7f907406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.380+0000 7f906ffff700 1 -- 172.21.15.23:0/3448108327 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f9078088ed0 con 0x7f90880aedc0 2023-06-22T08:07:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.380+0000 7f9086ffd700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f907406ace0 0x7f907406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.381+0000 7f9086ffd700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f907406ace0 0x7f907406d190 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f90881ef620 tx=0x7f907c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:48.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.385+0000 7f906ffff700 1 -- 172.21.15.23:0/3448108327 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9078053a00 con 0x7f90880aedc0 2023-06-22T08:07:48.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.641+0000 7f908dbff700 1 -- 172.21.15.23:0/3448108327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90880ae250 con 0x7f90880aedc0 2023-06-22T08:07:48.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.641+0000 7f906ffff700 1 -- 172.21.15.23:0/3448108327 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5499 (secure 0 0 0) 0x7f9078017070 con 0x7f90880aedc0 2023-06-22T08:07:48.645 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:48.645 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:47.381582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 -- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f907406ace0 msgr2=0x7f907406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f907406ace0 0x7f907406d190 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f90881ef620 tx=0x7f907c009040 comp rx=0 tx=0).stop 2023-06-22T08:07:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 -- 172.21.15.23:0/3448108327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 msgr2=0x7f90880cbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 0x7f90880cbd20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f907800c4a0 tx=0x7f907800c7b0 comp rx=0 tx=0).stop 2023-06-22T08:07:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 -- 172.21.15.23:0/3448108327 shutdown_connections 2023-06-22T08:07:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f907406ace0 0x7f907406d190 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9088152dd0 0x7f908814b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90880aedc0 0x7f90880cbd20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 --2- 172.21.15.23:0/3448108327 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90880c8bd0 0x7f90881ee4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 -- 172.21.15.23:0/3448108327 >> 172.21.15.23:0/3448108327 conn(0x7f908806fb70 msgr2=0x7f90880ae670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 -- 172.21.15.23:0/3448108327 shutdown_connections 2023-06-22T08:07:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:48.644+0000 7f908dbff700 1 -- 172.21.15.23:0/3448108327 wait complete. 2023-06-22T08:07:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 62 2023-06-22T08:07:48.666 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=4272 2023-06-22T08:07:48.666 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.a 2023-06-22T08:07:50.621 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:07:50.618+0000 7fa5334b2700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-06-22T08:07:53.667 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-06-22T08:07:53.668 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-22T08:07:53.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-22T08:07:53.673 INFO:tasks.ceph.mds.a:Started 2023-06-22T08:07:53.673 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:53.831 INFO:tasks.ceph.mds.a.smithi023.stdout:starting mds.a at 2023-06-22T08:07:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.886+0000 7fedfcfff700 1 Processor -- start 2023-06-22T08:07:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.887+0000 7fedfcfff700 1 -- start start 2023-06-22T08:07:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.888+0000 7fedfcfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf80c8bd0 0x7fedf80aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:53.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.888+0000 7fedfcfff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf80c7400 con 0x7fedf80cc640 2023-06-22T08:07:53.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.888+0000 7fedfcfff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf80c81c0 con 0x7fedf8152dd0 2023-06-22T08:07:53.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.888+0000 7fedfcfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80c84a0 con 0x7fedf80c8bd0 2023-06-22T08:07:53.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.888+0000 7fedf5d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf80c8bd0 0x7fedf80aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:53.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.888+0000 7fedf659c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fedf8152dd0 0x7fedf806afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:35118/0 (socket says 172.21.15.23:35118) 2023-06-22T08:07:53.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.888+0000 7fedf659c700 1 -- 172.21.15.23:0/794502765 learned_addr learned my addr 172.21.15.23:0/794502765 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:53.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.889+0000 7fedf559a700 1 -- 172.21.15.23:0/794502765 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151802655 0 0) 0x7fedf80c81c0 con 0x7fedf8152dd0 2023-06-22T08:07:53.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.889+0000 7fedf5d9b700 1 -- 172.21.15.23:0/794502765 >> v1:172.21.15.67:6789/0 conn(0x7fedf8152dd0 legacy=0x7fedf806afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:53.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.889+0000 7fedf5d9b700 1 -- 172.21.15.23:0/794502765 >> v1:172.21.15.23:6789/0 conn(0x7fedf80cc640 legacy=0x7fedf8163280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.889+0000 7fedf5d9b700 1 -- 172.21.15.23:0/794502765 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80cd370 con 0x7fedf80c8bd0 2023-06-22T08:07:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.889+0000 7fedf5d9b700 1 --2- 172.21.15.23:0/794502765 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf80c8bd0 0x7fedf80aedc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fedec009870 tx=0x7fedec009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e8fccde4b697198 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.889+0000 7fedf559a700 1 -- 172.21.15.23:0/794502765 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec011830 con 0x7fedf80c8bd0 2023-06-22T08:07:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.889+0000 7fedf559a700 1 -- 172.21.15.23:0/794502765 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec011990 con 0x7fedf80c8bd0 2023-06-22T08:07:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.889+0000 7fedf559a700 1 -- 172.21.15.23:0/794502765 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec011d20 con 0x7fedf80c8bd0 2023-06-22T08:07:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.890+0000 7fedfcfff700 1 -- 172.21.15.23:0/794502765 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf80c8bd0 msgr2=0x7fedf80aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.890+0000 7fedfcfff700 1 --2- 172.21.15.23:0/794502765 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf80c8bd0 0x7fedf80aedc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fedec009870 tx=0x7fedec009b80 comp rx=0 tx=0).stop 2023-06-22T08:07:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.890+0000 7fedfcfff700 1 -- 172.21.15.23:0/794502765 shutdown_connections 2023-06-22T08:07:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.890+0000 7fedfcfff700 1 --2- 172.21.15.23:0/794502765 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf80c8bd0 0x7fedf80aedc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.890+0000 7fedfcfff700 1 -- 172.21.15.23:0/794502765 >> 172.21.15.23:0/794502765 conn(0x7fedf806fb70 msgr2=0x7fedf806ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.890+0000 7fedfcfff700 1 -- 172.21.15.23:0/794502765 shutdown_connections 2023-06-22T08:07:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.890+0000 7fedfcfff700 1 -- 172.21.15.23:0/794502765 wait complete. 2023-06-22T08:07:53.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.892+0000 7fedfcfff700 1 Processor -- start 2023-06-22T08:07:53.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.892+0000 7fedfcfff700 1 -- start start 2023-06-22T08:07:53.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedfcfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedf80c8bd0 0x7fedf81ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedfcfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf80cc640 0x7fedf814fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedfcfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf8152dd0 0x7fedf8150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedfcfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80c5330 con 0x7fedf80c8bd0 2023-06-22T08:07:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedfcfff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80c4460 con 0x7fedf80cc640 2023-06-22T08:07:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedfcfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80c4720 con 0x7fedf8152dd0 2023-06-22T08:07:53.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedf659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedf80c8bd0 0x7fedf81ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:53.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedf5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf80cc640 0x7fedf814fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:53.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedf5d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf80cc640 0x7fedf814fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:44622/0 (socket says 172.21.15.23:44622) 2023-06-22T08:07:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedf5d9b700 1 -- 172.21.15.23:0/61797004 learned_addr learned my addr 172.21.15.23:0/61797004 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.893+0000 7fedf6d9d700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf8152dd0 0x7fedf8150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.894+0000 7fedf5d9b700 1 -- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf8152dd0 msgr2=0x7fedf8150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.894+0000 7fedf5d9b700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf8152dd0 0x7fedf8150210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.894+0000 7fedf5d9b700 1 -- 172.21.15.23:0/61797004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedf80c8bd0 msgr2=0x7fedf81ef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.894+0000 7fedf5d9b700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedf80c8bd0 0x7fedf81ef9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.894+0000 7fedf5d9b700 1 -- 172.21.15.23:0/61797004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec0094a0 con 0x7fedf80cc640 2023-06-22T08:07:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.894+0000 7fedf5d9b700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf80cc640 0x7fedf814fcd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fedec009bc0 tx=0x7fedec004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.894+0000 7fede6ffd700 1 -- 172.21.15.23:0/61797004 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec0036a0 con 0x7fedf80cc640 2023-06-22T08:07:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.894+0000 7fede6ffd700 1 -- 172.21.15.23:0/61797004 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec01f070 con 0x7fedf80cc640 2023-06-22T08:07:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.894+0000 7fedfcfff700 1 -- 172.21.15.23:0/61797004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf80c4a00 con 0x7fedf80cc640 2023-06-22T08:07:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.895+0000 7fede6ffd700 1 -- 172.21.15.23:0/61797004 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec003a40 con 0x7fedf80cc640 2023-06-22T08:07:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.895+0000 7fedfcfff700 1 -- 172.21.15.23:0/61797004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf80c4fc0 con 0x7fedf80cc640 2023-06-22T08:07:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.896+0000 7fedfcfff700 1 -- 172.21.15.23:0/61797004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf80c54e0 con 0x7fedf80cc640 2023-06-22T08:07:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.897+0000 7fede6ffd700 1 -- 172.21.15.23:0/61797004 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fedec003ba0 con 0x7fedf80cc640 2023-06-22T08:07:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.898+0000 7fede6ffd700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fedd806ad10 0x7fedd806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:53.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.898+0000 7fede6ffd700 1 -- 172.21.15.23:0/61797004 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7fedec021070 con 0x7fedf80cc640 2023-06-22T08:07:53.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.898+0000 7fedf659c700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fedd806ad10 0x7fedd806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:53.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.899+0000 7fedf659c700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fedd806ad10 0x7fedd806d1c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fede8000fb0 tx=0x7fede8000ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:53.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:53.902+0000 7fede6ffd700 1 -- 172.21.15.23:0/61797004 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fedf80c54e0 con 0x7fedf80cc640 2023-06-22T08:07:54.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.211+0000 7fedfcfff700 1 -- 172.21.15.23:0/61797004 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedf804e1d0 con 0x7fedf80cc640 2023-06-22T08:07:54.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.212+0000 7fede6ffd700 1 -- 172.21.15.23:0/61797004 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5499 (secure 0 0 0) 0x7fedec055ec0 con 0x7fedf80cc640 2023-06-22T08:07:54.217 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:54.217 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:47.381582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:54.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 -- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fedd806ad10 msgr2=0x7fedd806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:54.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fedd806ad10 0x7fedd806d1c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fede8000fb0 tx=0x7fede8000ed0 comp rx=0 tx=0).stop 2023-06-22T08:07:54.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 -- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf80cc640 msgr2=0x7fedf814fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:54.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf80cc640 0x7fedf814fcd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fedec009bc0 tx=0x7fedec004750 comp rx=0 tx=0).stop 2023-06-22T08:07:54.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 -- 172.21.15.23:0/61797004 shutdown_connections 2023-06-22T08:07:54.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fedf8152dd0 0x7fedf8150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:54.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fedf80c8bd0 0x7fedf81ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:54.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fedd806ad10 0x7fedd806d1c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 --2- 172.21.15.23:0/61797004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedf80cc640 0x7fedf814fcd0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 -- 172.21.15.23:0/61797004 >> 172.21.15.23:0/61797004 conn(0x7fedf806fb70 msgr2=0x7fedf80c9710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 -- 172.21.15.23:0/61797004 shutdown_connections 2023-06-22T08:07:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:54.215+0000 7fedfcfff700 1 -- 172.21.15.23:0/61797004 wait complete. 2023-06-22T08:07:54.222 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 62 2023-06-22T08:07:54.237 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-06-22T08:07:56.237 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:07:56.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.488+0000 7f55fd3ff700 1 Processor -- start 2023-06-22T08:07:56.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.488+0000 7f55fd3ff700 1 -- start start 2023-06-22T08:07:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.490+0000 7f55fd3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f55f815a320 0x7f55f80c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.490+0000 7f55fd3ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f80c6a60 con 0x7f55f80c3a40 2023-06-22T08:07:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.490+0000 7f55fd3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f814d690 con 0x7f55f806d050 2023-06-22T08:07:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.490+0000 7f55fd3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f8157570 con 0x7f55f815a320 2023-06-22T08:07:56.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.490+0000 7f55f6ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f55f80c3a40 0x7f55f8159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:47690/0 (socket says 172.21.15.23:47690) 2023-06-22T08:07:56.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.490+0000 7f55f6ffd700 1 -- 172.21.15.23:0/1160390017 learned_addr learned my addr 172.21.15.23:0/1160390017 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:07:56.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.490+0000 7f55f67fc700 1 --2- 172.21.15.23:0/1160390017 >> v2:172.21.15.23:3300/0 conn(0x7f55f815a320 0x7f55f80c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:56.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.490+0000 7f55f5ffb700 1 -- 172.21.15.23:0/1160390017 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374870646 0 0) 0x7f55f80c6a60 con 0x7f55f80c3a40 2023-06-22T08:07:56.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.490+0000 7f55f5ffb700 1 -- 172.21.15.23:0/1160390017 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55e00035b0 con 0x7f55f80c3a40 2023-06-22T08:07:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.490+0000 7f55f5ffb700 1 -- 172.21.15.23:0/1160390017 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159016698 0 0) 0x7f55f814d690 con 0x7f55f806d050 2023-06-22T08:07:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.491+0000 7f55f67fc700 1 -- 172.21.15.23:0/1160390017 >> v1:172.21.15.67:6789/0 conn(0x7f55f806d050 legacy=0x7f55f8160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.491+0000 7f55f67fc700 1 -- 172.21.15.23:0/1160390017 >> v1:172.21.15.23:6789/0 conn(0x7f55f80c3a40 legacy=0x7f55f8159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.491+0000 7f55f67fc700 1 -- 172.21.15.23:0/1160390017 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f80c6430 con 0x7f55f815a320 2023-06-22T08:07:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.491+0000 7f55f67fc700 1 --2- 172.21.15.23:0/1160390017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f815a320 0x7f55f80c3e10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f55ec009840 tx=0x7f55ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5b5164cb0d9e9b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.491+0000 7f55f5ffb700 1 -- 172.21.15.23:0/1160390017 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec011830 con 0x7f55f815a320 2023-06-22T08:07:56.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.491+0000 7f55f5ffb700 1 -- 172.21.15.23:0/1160390017 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ec011990 con 0x7f55f815a320 2023-06-22T08:07:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.491+0000 7f55f5ffb700 1 -- 172.21.15.23:0/1160390017 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec011d20 con 0x7f55f815a320 2023-06-22T08:07:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.492+0000 7f55fd3ff700 1 -- 172.21.15.23:0/1160390017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f815a320 msgr2=0x7f55f80c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.492+0000 7f55fd3ff700 1 --2- 172.21.15.23:0/1160390017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f815a320 0x7f55f80c3e10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f55ec009840 tx=0x7f55ec009b50 comp rx=0 tx=0).stop 2023-06-22T08:07:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.492+0000 7f55fd3ff700 1 -- 172.21.15.23:0/1160390017 shutdown_connections 2023-06-22T08:07:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.492+0000 7f55fd3ff700 1 --2- 172.21.15.23:0/1160390017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f815a320 0x7f55f80c3e10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.492+0000 7f55fd3ff700 1 -- 172.21.15.23:0/1160390017 >> 172.21.15.23:0/1160390017 conn(0x7f55f80ca3f0 msgr2=0x7f55f80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:56.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.492+0000 7f55fd3ff700 1 -- 172.21.15.23:0/1160390017 shutdown_connections 2023-06-22T08:07:56.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.492+0000 7f55fd3ff700 1 -- 172.21.15.23:0/1160390017 wait complete. 2023-06-22T08:07:56.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.494+0000 7f55fd3ff700 1 Processor -- start 2023-06-22T08:07:56.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.494+0000 7f55fd3ff700 1 -- start start 2023-06-22T08:07:56.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.495+0000 7f55fd3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f806d050 0x7f55f80c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:56.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.495+0000 7f55fd3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f55f80c3a40 0x7f55f81eeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:56.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.495+0000 7f55fd3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f815a320 0x7f55f81eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:56.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.495+0000 7f55fd3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f81ee5b0 con 0x7f55f806d050 2023-06-22T08:07:56.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.495+0000 7f55fd3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80c8630 con 0x7f55f815a320 2023-06-22T08:07:56.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.495+0000 7f55fd3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80c8910 con 0x7f55f80c3a40 2023-06-22T08:07:56.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f806d050 0x7f55f80c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:56.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f67fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f55f80c3a40 0x7f55f81eeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f815a320 0x7f55f81eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f67fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f55f80c3a40 0x7f55f81eeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:43760/0 (socket says 172.21.15.23:43760) 2023-06-22T08:07:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f806d050 0x7f55f80c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46926/0 (socket says 172.21.15.23:46926) 2023-06-22T08:07:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f67fc700 1 -- 172.21.15.23:0/2910971088 learned_addr learned my addr 172.21.15.23:0/2910971088 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:07:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f6ffd700 1 -- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f55f80c3a40 msgr2=0x7f55f81eeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f6ffd700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f55f80c3a40 0x7f55f81eeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f6ffd700 1 -- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f815a320 msgr2=0x7f55f81eeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f6ffd700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f815a320 0x7f55f81eeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f6ffd700 1 -- 172.21.15.23:0/2910971088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec0094a0 con 0x7f55f806d050 2023-06-22T08:07:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f67fc700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f55f80c3a40 0x7f55f81eeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:07:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55f6ffd700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f806d050 0x7f55f80c9a60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c420 tx=0x7f55e800c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.496+0000 7f55df7fe700 1 -- 172.21.15.23:0/2910971088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8005d70 con 0x7f55f806d050 2023-06-22T08:07:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.497+0000 7f55df7fe700 1 -- 172.21.15.23:0/2910971088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e800ce30 con 0x7f55f806d050 2023-06-22T08:07:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.497+0000 7f55fd3ff700 1 -- 172.21.15.23:0/2910971088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f80c8bf0 con 0x7f55f806d050 2023-06-22T08:07:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.497+0000 7f55df7fe700 1 -- 172.21.15.23:0/2910971088 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e80072f0 con 0x7f55f806d050 2023-06-22T08:07:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.497+0000 7f55fd3ff700 1 -- 172.21.15.23:0/2910971088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f80c91b0 con 0x7f55f806d050 2023-06-22T08:07:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.498+0000 7f55fd3ff700 1 -- 172.21.15.23:0/2910971088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f804ea20 con 0x7f55f806d050 2023-06-22T08:07:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.499+0000 7f55df7fe700 1 -- 172.21.15.23:0/2910971088 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f55e8007450 con 0x7f55f806d050 2023-06-22T08:07:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.500+0000 7f55df7fe700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f55e406ace0 0x7f55e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:07:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.500+0000 7f55df7fe700 1 -- 172.21.15.23:0/2910971088 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f55e8088c10 con 0x7f55f806d050 2023-06-22T08:07:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.500+0000 7f55f67fc700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f55e406ace0 0x7f55e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:07:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.501+0000 7f55f67fc700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f55e406ace0 0x7f55e406d190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f55ec000c00 tx=0x7f55ec01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:07:56.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.505+0000 7f55df7fe700 1 -- 172.21.15.23:0/2910971088 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f55e8053860 con 0x7f55f806d050 2023-06-22T08:07:56.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.768+0000 7f55fd3ff700 1 -- 172.21.15.23:0/2910971088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55f8147b40 con 0x7f55f806d050 2023-06-22T08:07:56.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.768+0000 7f55df7fe700 1 -- 172.21.15.23:0/2910971088 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+6254 (secure 0 0 0) 0x7f55e8017090 con 0x7f55f806d050 2023-06-22T08:07:56.772 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:07:56.772 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5707,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:07:56.760200+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:07:56.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 -- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f55e406ace0 msgr2=0x7f55e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:56.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f55e406ace0 0x7f55e406d190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f55ec000c00 tx=0x7f55ec01c040 comp rx=0 tx=0).stop 2023-06-22T08:07:56.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 -- 172.21.15.23:0/2910971088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f806d050 msgr2=0x7f55f80c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:07:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f806d050 0x7f55f80c9a60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c420 tx=0x7f55e800c730 comp rx=0 tx=0).stop 2023-06-22T08:07:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 -- 172.21.15.23:0/2910971088 shutdown_connections 2023-06-22T08:07:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f55e406ace0 0x7f55e406d190 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55f815a320 0x7f55f81eeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55f806d050 0x7f55f80c9a60 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 --2- 172.21.15.23:0/2910971088 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f55f80c3a40 0x7f55f81eeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:07:56.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 -- 172.21.15.23:0/2910971088 >> 172.21.15.23:0/2910971088 conn(0x7f55f80ca3f0 msgr2=0x7f55f8070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:07:56.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.772+0000 7f55fd3ff700 1 -- 172.21.15.23:0/2910971088 shutdown_connections 2023-06-22T08:07:56.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:07:56.773+0000 7f55fd3ff700 1 -- 172.21.15.23:0/2910971088 wait complete. 2023-06-22T08:07:56.778 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 64 2023-06-22T08:07:56.800 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-06-22T08:07:56.801 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 24 secs before thrashing 2023-06-22T08:08:03.896 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 2% (328/12587) Updating files: 3% (378/12587) Updating files: 4% (504/12587) Updating files: 4% (547/12587) Updating files: 5% (630/12587) Updating files: 5% (692/12587) Updating files: 6% (756/12587) Updating files: 7% (882/12587) Updating files: 7% (943/12587) Updating files: 8% (1007/12587) Updating files: 8% (1116/12587) Updating files: 9% (1133/12587) Updating files: 10% (1259/12587) Updating files: 10% (1369/12587) Updating files: 11% (1385/12587) Updating files: 12% (1511/12587) Updating files: 12% (1559/12587) Updating files: 13% (1637/12587) Updating files: 13% (1756/12587) Updating files: 14% (1763/12587) Updating files: 15% (1889/12587) Updating files: 15% (1976/12587) Updating files: 16% (2014/12587) Updating files: 16% (2094/12587) Updating files: 17% (2140/12587) Updating files: 18% (2266/12587) Updating files: 18% (2313/12587) Updating files: 19% (2392/12587) Updating files: 19% (2491/12587) Updating files: 20% (2518/12587) Updating files: 21% (2644/12587) Updating files: 21% (2704/12587) Updating files: 22% (2770/12587) Updating files: 23% (2896/12587) Updating files: 23% (2958/12587) Updating files: 24% (3021/12587) Updating files: 25% (3147/12587) Updating files: 25% (3192/12587) Updating files: 26% (3273/12587) Updating files: 27% (3399/12587) Updating files: 27% (3422/12587) Updating files: 28% (3525/12587) Updating files: 28% (3642/12587) Updating files: 29% (3651/12587) Updating files: 30% (3777/12587) Updating files: 30% (3852/12587) Updating files: 31% (3902/12587) Updating files: 32% (4028/12587) Updating files: 32% (4063/12587) Updating files: 33% (4154/12587) Updating files: 33% (4269/12587) Updating files: 34% (4280/12587) Updating files: 35% (4406/12587) Updating files: 36% (4532/12587) Updating files: 36% (4554/12587) Updating files: 37% (4658/12587) Updating files: 37% (4783/12587) Updating files: 38% (4784/12587) Updating files: 39% (4909/12587) Updating files: 39% (5010/12587) Updating files: 40% (5035/12587) Updating files: 41% (5161/12587) Updating files: 41% (5233/12587) Updating files: 42% (5287/12587) Updating files: 43% (5413/12587) Updating files: 43% (5492/12587) Updating files: 44% (5539/12587) Updating files: 45% (5665/12587) Updating files: 45% (5783/12587) Updating files: 46% (5791/12587) Updating files: 47% (5916/12587) Updating files: 47% (5955/12587) Updating files: 48% (6042/12587) Updating files: 49% (6168/12587) Updating files: 49% (6231/12587) Updating files: 50% (6294/12587) Updating files: 51% (6420/12587) Updating files: 51% (6520/12587) Updating files: 52% (6546/12587) Updating files: 53% (6672/12587) Updating files: 53% (6754/12587) Updating files: 54% (6797/12587) Updating files: 55% (6923/12587) Updating files: 55% (6994/12587) Updating files: 56% (7049/12587) Updating files: 57% (7175/12587) Updating files: 58% (7301/12587) Updating files: 58% (7316/12587) Updating files: 59% (7427/12587) Updating files: 59% (7551/12587) Updating files: 60% (7553/12587) Updating files: 61% (7679/12587) Updating files: 61% (7776/12587) Updating files: 62% (7804/12587) Updating files: 63% (7930/12587) Updating files: 63% (7974/12587) Updating files: 64% (8056/12587) Updating files: 64% (8122/12587) Updating files: 65% (8182/12587) Updating files: 66% (8308/12587) Updating files: 66% (8353/12587) Updating files: 67% (8434/12587) Updating files: 68% (8560/12587) Updating files: 68% (8597/12587) Updating files: 69% (8686/12587) Updating files: 70% (8811/12587) Updating files: 70% (8873/12587) Updating files: 71% (8937/12587) Updating files: 72% (9063/12587) Updating files: 72% (9182/12587) Updating files: 73% (9189/12587) Updating files: 73% (9267/12587) Updating files: 74% (9315/12587) Updating files: 75% (9441/12587) Updating files: 75% (9506/12587) Updating files: 76% (9567/12587) Updating files: 77% (9692/12587) Updating files: 78% (9818/12587) Updating files: 78% (9896/12587) Updating files: 79% (9944/12587) Updating files: 80% (10070/12587) Updating files: 80% (10111/12587) Updating files: 81% (10196/12587) Updating files: 81% (10272/12587) Updating files: 82% (10322/12587) Updating files: 83% (10448/12587) Updating files: 83% (10479/12587) Updating files: 84% (10574/12587) Updating files: 85% (10699/12587) Updating files: 85% (10709/12587) Updating files: 86% (10825/12587) Updating files: 86% (10916/12587) Updating files: 87% (10951/12587) Updating files: 88% (11077/12587) Updating files: 88% (11176/12587) Updating files: 89% (11203/12587) Updating files: 90% (11329/12587) Updating files: 90% (11430/12587) Updating files: 91% (11455/12587) Updating files: 92% (11581/12587) Updating files: 93% (11706/12587) Updating files: 93% (11812/12587) Updating files: 94% (11832/12587) Updating files: 95% (11958/12587) Updating files: 96% (12084/12587) Updating files: 97% (12210/12587) Updating files: 97% (12334/12587) Updating files: 98% (12336/12587) Updating files: 99% (12462/12587) Updating files: 100% (12587/12587) Updating files: 100% (12587/12587), done. 2023-06-22T08:08:04.081 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2023-06-22T08:08:04.090 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.1 2023-06-22T08:08:04.210 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:08:04.210 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -eq 48 ']' 2023-06-22T08:08:04.211 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.2 2023-06-22T08:08:04.211 INFO:tasks.workunit.client.0.smithi023.stdout:v0.2 2023-06-22T08:08:04.211 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.2 2023-06-22T08:08:04.211 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.2 2023-06-22T08:08:05.065 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 2b5f76f48e9 readme 2023-06-22T08:08:05.071 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.2 2023-06-22T08:08:05.186 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:08:05.187 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 3 -eq 48 ']' 2023-06-22T08:08:05.187 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.3 2023-06-22T08:08:05.187 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.3 2023-06-22T08:08:05.187 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.3 2023-06-22T08:08:05.187 INFO:tasks.workunit.client.0.smithi023.stdout:v0.3 2023-06-22T08:08:06.390 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2023-06-22T08:08:06.399 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.3 2023-06-22T08:08:06.682 INFO:tasks.workunit.client.0.smithi023.stdout:v0.4 2023-06-22T08:08:06.682 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:08:06.682 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 4 -eq 48 ']' 2023-06-22T08:08:06.683 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.4 2023-06-22T08:08:06.683 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.4 2023-06-22T08:08:06.683 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.4 2023-06-22T08:08:07.808 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2023-06-22T08:08:07.816 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.4 2023-06-22T08:08:07.841 INFO:tasks.workunit.client.0.smithi023.stdout:v0.5 2023-06-22T08:08:07.841 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:08:07.841 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 5 -eq 48 ']' 2023-06-22T08:08:07.841 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.5 2023-06-22T08:08:07.842 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.5 2023-06-22T08:08:07.842 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.5 2023-06-22T08:08:08.075 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:08:08.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:08:09.182 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2023-06-22T08:08:09.188 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.5 2023-06-22T08:08:09.255 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:08:09.256 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 6 -eq 48 ']' 2023-06-22T08:08:09.256 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.6 2023-06-22T08:08:09.256 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.6 2023-06-22T08:08:09.256 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.6 2023-06-22T08:08:09.256 INFO:tasks.workunit.client.0.smithi023.stdout:v0.6 2023-06-22T08:08:10.295 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at a4c752df713 v0.6 2023-06-22T08:08:10.300 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.6 2023-06-22T08:08:10.393 INFO:tasks.workunit.client.0.smithi023.stdout:v0.7 2023-06-22T08:08:10.393 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:08:10.393 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 7 -eq 48 ']' 2023-06-22T08:08:10.393 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.7 2023-06-22T08:08:10.394 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.7 2023-06-22T08:08:10.394 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.7 2023-06-22T08:08:11.380 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2023-06-22T08:08:11.385 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.7 2023-06-22T08:08:11.494 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:08:11.495 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 8 -eq 48 ']' 2023-06-22T08:08:11.495 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.8 2023-06-22T08:08:11.495 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.8 2023-06-22T08:08:11.496 INFO:tasks.workunit.client.0.smithi023.stdout:v0.8 2023-06-22T08:08:11.496 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.8 2023-06-22T08:08:13.013 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2023-06-22T08:08:13.021 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.8 2023-06-22T08:08:13.108 INFO:tasks.workunit.client.0.smithi023.stdout:v0.9 2023-06-22T08:08:13.108 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:08:13.109 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 9 -eq 48 ']' 2023-06-22T08:08:13.109 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.9 2023-06-22T08:08:13.109 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.9 2023-06-22T08:08:13.109 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.9 2023-06-22T08:08:20.802 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:21.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.005+0000 7f349bfff700 1 Processor -- start 2023-06-22T08:08:21.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.005+0000 7f349bfff700 1 -- start start 2023-06-22T08:08:21.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.006+0000 7f349bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06bc40 0x7f349c0c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.006+0000 7f349bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f349c06ae60 0x7f349c06b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.006+0000 7f349bfff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349c06c090 con 0x7f349c0c9890 2023-06-22T08:08:21.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.006+0000 7f349bfff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349c0ca1a0 con 0x7f349c06ae60 2023-06-22T08:08:21.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.006+0000 7f349bfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c0c8f80 con 0x7f349c06bc40 2023-06-22T08:08:21.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.006+0000 7f349affd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f349c0c9890 0x7f349c158eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34994/0 (socket says 172.21.15.23:34994) 2023-06-22T08:08:21.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.006+0000 7f349affd700 1 -- 172.21.15.23:0/879025657 learned_addr learned my addr 172.21.15.23:0/879025657 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:21.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.006+0000 7f349a7fc700 1 --2- 172.21.15.23:0/879025657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06bc40 0x7f349c0c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.006+0000 7f349b7fe700 1 --2- 172.21.15.23:0/879025657 >> v2:172.21.15.23:3300/0 conn(0x7f349c06ae60 0x7f349c06b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.007+0000 7f349a7fc700 1 -- 172.21.15.23:0/879025657 >> v2:172.21.15.23:3300/0 conn(0x7f349c06ae60 msgr2=0x7f349c06b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.007+0000 7f349a7fc700 1 --2- 172.21.15.23:0/879025657 >> v2:172.21.15.23:3300/0 conn(0x7f349c06ae60 0x7f349c06b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.007+0000 7f349a7fc700 1 -- 172.21.15.23:0/879025657 >> v1:172.21.15.23:6789/0 conn(0x7f349c0c9890 legacy=0x7f349c158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.007+0000 7f349a7fc700 1 -- 172.21.15.23:0/879025657 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c06b820 con 0x7f349c06bc40 2023-06-22T08:08:21.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.007+0000 7f3499ffb700 1 -- 172.21.15.23:0/879025657 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823326972 0 0) 0x7f348c001110 con 0x7f349c0c9890 2023-06-22T08:08:21.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.007+0000 7f349a7fc700 1 --2- 172.21.15.23:0/879025657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06bc40 0x7f349c0c9c60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3490009840 tx=0x7f3490009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a08ba53c5dee2936 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:21.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.007+0000 7f3499ffb700 1 -- 172.21.15.23:0/879025657 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490011830 con 0x7f349c06bc40 2023-06-22T08:08:21.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.007+0000 7f3499ffb700 1 -- 172.21.15.23:0/879025657 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3490011990 con 0x7f349c06bc40 2023-06-22T08:08:21.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.007+0000 7f3499ffb700 1 -- 172.21.15.23:0/879025657 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490011d20 con 0x7f349c06bc40 2023-06-22T08:08:21.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.008+0000 7f349bfff700 1 -- 172.21.15.23:0/879025657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06bc40 msgr2=0x7f349c0c9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.008+0000 7f349bfff700 1 --2- 172.21.15.23:0/879025657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06bc40 0x7f349c0c9c60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3490009840 tx=0x7f3490009b50 comp rx=0 tx=0).stop 2023-06-22T08:08:21.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.008+0000 7f349bfff700 1 -- 172.21.15.23:0/879025657 shutdown_connections 2023-06-22T08:08:21.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.008+0000 7f349bfff700 1 --2- 172.21.15.23:0/879025657 >> v2:172.21.15.23:3300/0 conn(0x7f349c06ae60 0x7f349c06b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.008+0000 7f349bfff700 1 --2- 172.21.15.23:0/879025657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06bc40 0x7f349c0c9c60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.008+0000 7f349bfff700 1 -- 172.21.15.23:0/879025657 >> 172.21.15.23:0/879025657 conn(0x7f349c065b90 msgr2=0x7f349c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:21.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.008+0000 7f349bfff700 1 -- 172.21.15.23:0/879025657 shutdown_connections 2023-06-22T08:08:21.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.008+0000 7f349bfff700 1 -- 172.21.15.23:0/879025657 wait complete. 2023-06-22T08:08:21.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.009+0000 7f349bfff700 1 Processor -- start 2023-06-22T08:08:21.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.010+0000 7f349bfff700 1 -- start start 2023-06-22T08:08:21.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.010+0000 7f349bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06ae60 0x7f349c0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.010+0000 7f349bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f349c06bc40 0x7f349c1e3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f349c0c9890 0x7f349c1ef0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349bfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c157080 con 0x7f349c0c9890 2023-06-22T08:08:21.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349bfff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c157450 con 0x7f349c06bc40 2023-06-22T08:08:21.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349bfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c1576f0 con 0x7f349c06ae60 2023-06-22T08:08:21.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f349c0c9890 0x7f349c1ef0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f349c0c9890 0x7f349c1ef0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52908/0 (socket says 172.21.15.23:52908) 2023-06-22T08:08:21.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349b7fe700 1 -- 172.21.15.23:0/1091715830 learned_addr learned my addr 172.21.15.23:0/1091715830 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:21.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349affd700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06ae60 0x7f349c0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349a7fc700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f349c06bc40 0x7f349c1e3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349b7fe700 1 -- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06ae60 msgr2=0x7f349c0c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349b7fe700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06ae60 0x7f349c0c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349b7fe700 1 -- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f349c06bc40 msgr2=0x7f349c1e3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349b7fe700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f349c06bc40 0x7f349c1e3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349b7fe700 1 -- 172.21.15.23:0/1091715830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34900094a0 con 0x7f349c0c9890 2023-06-22T08:08:21.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.011+0000 7f349b7fe700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f349c0c9890 0x7f349c1ef0b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f349400efc0 tx=0x7f349400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:21.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.012+0000 7f34837fe700 1 -- 172.21.15.23:0/1091715830 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3494009ea0 con 0x7f349c0c9890 2023-06-22T08:08:21.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.012+0000 7f34837fe700 1 -- 172.21.15.23:0/1091715830 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349400f040 con 0x7f349c0c9890 2023-06-22T08:08:21.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.012+0000 7f34837fe700 1 -- 172.21.15.23:0/1091715830 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349400ced0 con 0x7f349c0c9890 2023-06-22T08:08:21.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.012+0000 7f349bfff700 1 -- 172.21.15.23:0/1091715830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349c1579d0 con 0x7f349c0c9890 2023-06-22T08:08:21.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.012+0000 7f349bfff700 1 -- 172.21.15.23:0/1091715830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349c1565a0 con 0x7f349c0c9890 2023-06-22T08:08:21.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.013+0000 7f349bfff700 1 -- 172.21.15.23:0/1091715830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349c061b00 con 0x7f349c0c9890 2023-06-22T08:08:21.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.014+0000 7f34837fe700 1 -- 172.21.15.23:0/1091715830 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3494014480 con 0x7f349c0c9890 2023-06-22T08:08:21.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.015+0000 7f34837fe700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f348806ad10 0x7f348806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.015+0000 7f34837fe700 1 -- 172.21.15.23:0/1091715830 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f349408a1e0 con 0x7f349c0c9890 2023-06-22T08:08:21.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.015+0000 7f349affd700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f348806ad10 0x7f348806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.016+0000 7f349affd700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f348806ad10 0x7f348806d1c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f348c000f60 tx=0x7f348c007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:21.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.020+0000 7f34837fe700 1 -- 172.21.15.23:0/1091715830 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3494054da0 con 0x7f349c0c9890 2023-06-22T08:08:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.275+0000 7f349bfff700 1 -- 172.21.15.23:0/1091715830 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f349c0c9670 con 0x7f349c0c9890 2023-06-22T08:08:21.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.275+0000 7f34837fe700 1 -- 172.21.15.23:0/1091715830 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6263 (secure 0 0 0) 0x7f34940583c0 con 0x7f349c0c9890 2023-06-22T08:08:21.282 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:21.282 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5707,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:10.394986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 -- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f348806ad10 msgr2=0x7f348806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f348806ad10 0x7f348806d1c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f348c000f60 tx=0x7f348c007e70 comp rx=0 tx=0).stop 2023-06-22T08:08:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 -- 172.21.15.23:0/1091715830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f349c0c9890 msgr2=0x7f349c1ef0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f349c0c9890 0x7f349c1ef0b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f349400efc0 tx=0x7f349400c3a0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 -- 172.21.15.23:0/1091715830 shutdown_connections 2023-06-22T08:08:21.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f349c0c9890 0x7f349c1ef0b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f349c06ae60 0x7f349c0c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f348806ad10 0x7f348806d1c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 --2- 172.21.15.23:0/1091715830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f349c06bc40 0x7f349c1e3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 -- 172.21.15.23:0/1091715830 >> 172.21.15.23:0/1091715830 conn(0x7f349c065b90 msgr2=0x7f349c0cadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:21.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 -- 172.21.15.23:0/1091715830 shutdown_connections 2023-06-22T08:08:21.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.278+0000 7f349bfff700 1 -- 172.21.15.23:0/1091715830 wait complete. 2023-06-22T08:08:21.285 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 69 2023-06-22T08:08:21.299 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=1) 2023-06-22T08:08:21.300 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-22T08:08:21.300 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:08:21.311 INFO:tasks.ceph.mds.b:Stopped 2023-06-22T08:08:21.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:21.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.520+0000 7fddff9ff700 1 Processor -- start 2023-06-22T08:08:21.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.521+0000 7fddff9ff700 1 -- start start 2023-06-22T08:08:21.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.522+0000 7fddff9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddf80cb6b0 0x7fddf8069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.522+0000 7fddff9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fddf8152e10 0x7fddf81593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.522+0000 7fddff9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf8158f60 con 0x7fddf80c5740 2023-06-22T08:08:21.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.522+0000 7fddff9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf806bd20 con 0x7fddf80cb6b0 2023-06-22T08:08:21.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.522+0000 7fddff9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80cc450 con 0x7fddf8152e10 2023-06-22T08:08:21.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.522+0000 7fddfd79b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fddf80cb6b0 0x7fddf8069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.522+0000 7fddfdf9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fddf8152e10 0x7fddf81593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.522+0000 7fddfcf9a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fddf80c5740 0x7fddf8152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58766/0 (socket says 172.21.15.23:58766) 2023-06-22T08:08:21.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.522+0000 7fddfcf9a700 1 -- 172.21.15.23:0/1331004462 learned_addr learned my addr 172.21.15.23:0/1331004462 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:21.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.523+0000 7fddeffff700 1 -- 172.21.15.23:0/1331004462 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078635211 0 0) 0x7fddf8158f60 con 0x7fddf80c5740 2023-06-22T08:08:21.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.523+0000 7fddeffff700 1 -- 172.21.15.23:0/1331004462 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde40035b0 con 0x7fddf80c5740 2023-06-22T08:08:21.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.523+0000 7fddfd79b700 1 -- 172.21.15.23:0/1331004462 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fddf8152e10 msgr2=0x7fddf81593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:21.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.523+0000 7fddfd79b700 1 --2- 172.21.15.23:0/1331004462 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fddf8152e10 0x7fddf81593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.523+0000 7fddfd79b700 1 -- 172.21.15.23:0/1331004462 >> v1:172.21.15.67:6789/0 conn(0x7fddf80c5740 legacy=0x7fddf8152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.523+0000 7fddfd79b700 1 -- 172.21.15.23:0/1331004462 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8158980 con 0x7fddf80cb6b0 2023-06-22T08:08:21.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.523+0000 7fddfd79b700 1 --2- 172.21.15.23:0/1331004462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf80cb6b0 0x7fddf8069eb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fddf4009840 tx=0x7fddf4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef72ba2fec7429a0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:21.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.523+0000 7fddeffff700 1 -- 172.21.15.23:0/1331004462 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4011830 con 0x7fddf80cb6b0 2023-06-22T08:08:21.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.523+0000 7fddeffff700 1 -- 172.21.15.23:0/1331004462 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf4011990 con 0x7fddf80cb6b0 2023-06-22T08:08:21.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.523+0000 7fddeffff700 1 -- 172.21.15.23:0/1331004462 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4011d20 con 0x7fddf80cb6b0 2023-06-22T08:08:21.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.524+0000 7fddff9ff700 1 -- 172.21.15.23:0/1331004462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf80cb6b0 msgr2=0x7fddf8069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.524+0000 7fddff9ff700 1 --2- 172.21.15.23:0/1331004462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf80cb6b0 0x7fddf8069eb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fddf4009840 tx=0x7fddf4009b50 comp rx=0 tx=0).stop 2023-06-22T08:08:21.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.524+0000 7fddff9ff700 1 -- 172.21.15.23:0/1331004462 shutdown_connections 2023-06-22T08:08:21.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.524+0000 7fddff9ff700 1 --2- 172.21.15.23:0/1331004462 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fddf8152e10 0x7fddf81593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.524+0000 7fddff9ff700 1 --2- 172.21.15.23:0/1331004462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf80cb6b0 0x7fddf8069eb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.524+0000 7fddff9ff700 1 -- 172.21.15.23:0/1331004462 >> 172.21.15.23:0/1331004462 conn(0x7fddf8070d80 msgr2=0x7fddf8071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:21.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.524+0000 7fddff9ff700 1 -- 172.21.15.23:0/1331004462 shutdown_connections 2023-06-22T08:08:21.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.524+0000 7fddff9ff700 1 -- 172.21.15.23:0/1331004462 wait complete. 2023-06-22T08:08:21.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.526+0000 7fddff9ff700 1 Processor -- start 2023-06-22T08:08:21.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.526+0000 7fddff9ff700 1 -- start start 2023-06-22T08:08:21.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.526+0000 7fddff9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf80cb6b0 0x7fddf80c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddff9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8152e10 0x7fddf8157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddff9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fddf8157960 0x7fddf81f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddff9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf8157020 con 0x7fddf80cb6b0 2023-06-22T08:08:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddff9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80c7a50 con 0x7fddf8152e10 2023-06-22T08:08:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddff9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf80c8200 con 0x7fddf8157960 2023-06-22T08:08:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfd79b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf80cb6b0 0x7fddf80c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfd79b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf80cb6b0 0x7fddf80c7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52914/0 (socket says 172.21.15.23:52914) 2023-06-22T08:08:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfd79b700 1 -- 172.21.15.23:0/3954217947 learned_addr learned my addr 172.21.15.23:0/3954217947 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfcf9a700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8152e10 0x7fddf8157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfdf9c700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fddf8157960 0x7fddf81f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfd79b700 1 -- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fddf8157960 msgr2=0x7fddf81f1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfd79b700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fddf8157960 0x7fddf81f1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfd79b700 1 -- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8152e10 msgr2=0x7fddf8157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfd79b700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8152e10 0x7fddf8157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfd79b700 1 -- 172.21.15.23:0/3954217947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf40094a0 con 0x7fddf80cb6b0 2023-06-22T08:08:21.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.527+0000 7fddfd79b700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf80cb6b0 0x7fddf80c7510 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fddf40090b0 tx=0x7fddf4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:21.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.528+0000 7fddedffb700 1 -- 172.21.15.23:0/3954217947 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4004590 con 0x7fddf80cb6b0 2023-06-22T08:08:21.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.528+0000 7fddedffb700 1 -- 172.21.15.23:0/3954217947 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf401f070 con 0x7fddf80cb6b0 2023-06-22T08:08:21.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.528+0000 7fddedffb700 1 -- 172.21.15.23:0/3954217947 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf40038e0 con 0x7fddf80cb6b0 2023-06-22T08:08:21.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.528+0000 7fddff9ff700 1 -- 172.21.15.23:0/3954217947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf8157dc0 con 0x7fddf80cb6b0 2023-06-22T08:08:21.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.528+0000 7fddff9ff700 1 -- 172.21.15.23:0/3954217947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf80c7f20 con 0x7fddf80cb6b0 2023-06-22T08:08:21.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.529+0000 7fddff9ff700 1 -- 172.21.15.23:0/3954217947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf804ea20 con 0x7fddf80cb6b0 2023-06-22T08:08:21.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.530+0000 7fddedffb700 1 -- 172.21.15.23:0/3954217947 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fddf4003a40 con 0x7fddf80cb6b0 2023-06-22T08:08:21.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.531+0000 7fddedffb700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdddc06ad10 0x7fdddc06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:21.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.531+0000 7fddedffb700 1 -- 172.21.15.23:0/3954217947 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7fddf4021070 con 0x7fddf80cb6b0 2023-06-22T08:08:21.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.531+0000 7fddfcf9a700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdddc06ad10 0x7fdddc06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:21.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.531+0000 7fddfcf9a700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdddc06ad10 0x7fdddc06d1c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fddf806a020 tx=0x7fdde8007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:21.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.535+0000 7fddedffb700 1 -- 172.21.15.23:0/3954217947 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fddf4055c80 con 0x7fddf80cb6b0 2023-06-22T08:08:21.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.791+0000 7fddff9ff700 1 -- 172.21.15.23:0/3954217947 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddf8158720 con 0x7fddf80cb6b0 2023-06-22T08:08:21.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.791+0000 7fddedffb700 1 -- 172.21.15.23:0/3954217947 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6263 (secure 0 0 0) 0x7fddf40592a0 con 0x7fddf80cb6b0 2023-06-22T08:08:21.795 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:21.795 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5707,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:10.394986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:21.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 -- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdddc06ad10 msgr2=0x7fdddc06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdddc06ad10 0x7fdddc06d1c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fddf806a020 tx=0x7fdde8007dc0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 -- 172.21.15.23:0/3954217947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf80cb6b0 msgr2=0x7fddf80c7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:21.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf80cb6b0 0x7fddf80c7510 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fddf40090b0 tx=0x7fddf4004750 comp rx=0 tx=0).stop 2023-06-22T08:08:21.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 -- 172.21.15.23:0/3954217947 shutdown_connections 2023-06-22T08:08:21.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fddf8157960 0x7fddf81f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf80cb6b0 0x7fddf80c7510 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdddc06ad10 0x7fdddc06d1c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 --2- 172.21.15.23:0/3954217947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fddf8152e10 0x7fddf8157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 -- 172.21.15.23:0/3954217947 >> 172.21.15.23:0/3954217947 conn(0x7fddf8070d80 msgr2=0x7fddf81507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 -- 172.21.15.23:0/3954217947 shutdown_connections 2023-06-22T08:08:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:21.794+0000 7fddff9ff700 1 -- 172.21.15.23:0/3954217947 wait complete. 2023-06-22T08:08:21.799 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 69 2023-06-22T08:08:21.815 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-06-22T08:08:23.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:24.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.020+0000 7fe56b9ff700 1 Processor -- start 2023-06-22T08:08:24.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.020+0000 7fe56b9ff700 1 -- start start 2023-06-22T08:08:24.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.021+0000 7fe56b9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe56415a320 0x7fe5640c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:24.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.021+0000 7fe56b9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5640c6a60 con 0x7fe5640c3a40 2023-06-22T08:08:24.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.021+0000 7fe56b9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56414d690 con 0x7fe56406d050 2023-06-22T08:08:24.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.021+0000 7fe56b9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564157570 con 0x7fe56415a320 2023-06-22T08:08:24.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.021+0000 7fe56979b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe5640c3a40 0x7fe564159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:35002/0 (socket says 172.21.15.23:35002) 2023-06-22T08:08:24.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.021+0000 7fe56979b700 1 -- 172.21.15.23:0/2676931783 learned_addr learned my addr 172.21.15.23:0/2676931783 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:24.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.021+0000 7fe568f9a700 1 --2- 172.21.15.23:0/2676931783 >> v2:172.21.15.67:3300/0 conn(0x7fe56415a320 0x7fe5640c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:24.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.022+0000 7fe55bfff700 1 -- 172.21.15.23:0/2676931783 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545720121 0 0) 0x7fe56414d690 con 0x7fe56406d050 2023-06-22T08:08:24.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.022+0000 7fe55bfff700 1 -- 172.21.15.23:0/2676931783 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5500035b0 con 0x7fe56406d050 2023-06-22T08:08:24.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.022+0000 7fe55bfff700 1 -- 172.21.15.23:0/2676931783 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803398252 0 0) 0x7fe5640c6a60 con 0x7fe5640c3a40 2023-06-22T08:08:24.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.022+0000 7fe568f9a700 1 -- 172.21.15.23:0/2676931783 >> v1:172.21.15.67:6789/0 conn(0x7fe56406d050 legacy=0x7fe564160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:24.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.022+0000 7fe568f9a700 1 -- 172.21.15.23:0/2676931783 >> v1:172.21.15.23:6789/0 conn(0x7fe5640c3a40 legacy=0x7fe564159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.022+0000 7fe568f9a700 1 -- 172.21.15.23:0/2676931783 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5640c6430 con 0x7fe56415a320 2023-06-22T08:08:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.022+0000 7fe568f9a700 1 --2- 172.21.15.23:0/2676931783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe56415a320 0x7fe5640c3e10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe554009840 tx=0x7fe554009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcbd64c339529ded server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.022+0000 7fe55bfff700 1 -- 172.21.15.23:0/2676931783 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe554011830 con 0x7fe56415a320 2023-06-22T08:08:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.023+0000 7fe55bfff700 1 -- 172.21.15.23:0/2676931783 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe554011990 con 0x7fe56415a320 2023-06-22T08:08:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.023+0000 7fe55bfff700 1 -- 172.21.15.23:0/2676931783 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe554011d20 con 0x7fe56415a320 2023-06-22T08:08:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.023+0000 7fe56b9ff700 1 -- 172.21.15.23:0/2676931783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe56415a320 msgr2=0x7fe5640c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:24.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.023+0000 7fe56b9ff700 1 --2- 172.21.15.23:0/2676931783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe56415a320 0x7fe5640c3e10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe554009840 tx=0x7fe554009b50 comp rx=0 tx=0).stop 2023-06-22T08:08:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.023+0000 7fe56b9ff700 1 -- 172.21.15.23:0/2676931783 shutdown_connections 2023-06-22T08:08:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.023+0000 7fe56b9ff700 1 --2- 172.21.15.23:0/2676931783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe56415a320 0x7fe5640c3e10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.023+0000 7fe56b9ff700 1 -- 172.21.15.23:0/2676931783 >> 172.21.15.23:0/2676931783 conn(0x7fe5640ca3f0 msgr2=0x7fe5640ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.023+0000 7fe56b9ff700 1 -- 172.21.15.23:0/2676931783 shutdown_connections 2023-06-22T08:08:24.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.023+0000 7fe56b9ff700 1 -- 172.21.15.23:0/2676931783 wait complete. 2023-06-22T08:08:24.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.025+0000 7fe56b9ff700 1 Processor -- start 2023-06-22T08:08:24.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.025+0000 7fe56b9ff700 1 -- start start 2023-06-22T08:08:24.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe56b9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56406d050 0x7fe5641554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:24.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe56b9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5640c3a40 0x7fe5641f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:24.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe56b9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe56415a320 0x7fe5641f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:24.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe56b9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe564156b10 con 0x7fe56406d050 2023-06-22T08:08:24.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe56b9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe564158520 con 0x7fe56415a320 2023-06-22T08:08:24.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe56b9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564157700 con 0x7fe5640c3a40 2023-06-22T08:08:24.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe56979b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56406d050 0x7fe5641554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:24.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe569f9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe56415a320 0x7fe5641f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:24.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe56979b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56406d050 0x7fe5641554b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52920/0 (socket says 172.21.15.23:52920) 2023-06-22T08:08:24.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe56979b700 1 -- 172.21.15.23:0/3378290352 learned_addr learned my addr 172.21.15.23:0/3378290352 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:24.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.026+0000 7fe568f9a700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5640c3a40 0x7fe5641f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:24.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe569f9c700 1 -- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5640c3a40 msgr2=0x7fe5641f18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe569f9c700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5640c3a40 0x7fe5641f18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe569f9c700 1 -- 172.21.15.23:0/3378290352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56406d050 msgr2=0x7fe5641554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe569f9c700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56406d050 0x7fe5641554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe569f9c700 1 -- 172.21.15.23:0/3378290352 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5540094a0 con 0x7fe56415a320 2023-06-22T08:08:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe569f9c700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe56415a320 0x7fe5641f1e00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00cbc0 tx=0x7fe55c00ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:24.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe559ffb700 1 -- 172.21.15.23:0/3378290352 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c003c00 con 0x7fe56415a320 2023-06-22T08:08:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe559ffb700 1 -- 172.21.15.23:0/3378290352 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c003d60 con 0x7fe56415a320 2023-06-22T08:08:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe56b9ff700 1 -- 172.21.15.23:0/3378290352 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5641579a0 con 0x7fe56415a320 2023-06-22T08:08:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe559ffb700 1 -- 172.21.15.23:0/3378290352 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c00aa80 con 0x7fe56415a320 2023-06-22T08:08:24.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.027+0000 7fe56b9ff700 1 -- 172.21.15.23:0/3378290352 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe564157f60 con 0x7fe56415a320 2023-06-22T08:08:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.028+0000 7fe56b9ff700 1 -- 172.21.15.23:0/3378290352 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5641f15e0 con 0x7fe56415a320 2023-06-22T08:08:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.029+0000 7fe559ffb700 1 -- 172.21.15.23:0/3378290352 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe55c00abe0 con 0x7fe56415a320 2023-06-22T08:08:24.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.030+0000 7fe559ffb700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe54806ad10 0x7fe54806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:24.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.031+0000 7fe559ffb700 1 -- 172.21.15.23:0/3378290352 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7fe55c089910 con 0x7fe56415a320 2023-06-22T08:08:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.031+0000 7fe56979b700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe54806ad10 0x7fe54806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.031+0000 7fe56979b700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe54806ad10 0x7fe54806d1c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe560000f60 tx=0x7fe560007ea0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:24.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.035+0000 7fe559ffb700 1 -- 172.21.15.23:0/3378290352 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe5641f15e0 con 0x7fe56415a320 2023-06-22T08:08:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.286+0000 7fe56b9ff700 1 -- 172.21.15.23:0/3378290352 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5641f15e0 con 0x7fe56415a320 2023-06-22T08:08:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.287+0000 7fe559ffb700 1 -- 172.21.15.23:0/3378290352 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+6265 (secure 0 0 0) 0x7fe5641f15e0 con 0x7fe56415a320 2023-06-22T08:08:24.290 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:24.290 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5707,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:22.771994+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 -- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe54806ad10 msgr2=0x7fe54806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe54806ad10 0x7fe54806d1c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe560000f60 tx=0x7fe560007ea0 comp rx=0 tx=0).stop 2023-06-22T08:08:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 -- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe56415a320 msgr2=0x7fe5641f1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe56415a320 0x7fe5641f1e00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00cbc0 tx=0x7fe55c00ced0 comp rx=0 tx=0).stop 2023-06-22T08:08:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 -- 172.21.15.23:0/3378290352 shutdown_connections 2023-06-22T08:08:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe54806ad10 0x7fe54806d1c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:24.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe56415a320 0x7fe5641f1e00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:24.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56406d050 0x7fe5641554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:24.295 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 --2- 172.21.15.23:0/3378290352 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5640c3a40 0x7fe5641f18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:24.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 -- 172.21.15.23:0/3378290352 >> 172.21.15.23:0/3378290352 conn(0x7fe5640ca3f0 msgr2=0x7fe56414bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:24.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 -- 172.21.15.23:0/3378290352 shutdown_connections 2023-06-22T08:08:24.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:24.289+0000 7fe56b9ff700 1 -- 172.21.15.23:0/3378290352 wait complete. 2023-06-22T08:08:24.297 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 70 2023-06-22T08:08:24.312 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-06-22T08:08:26.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:26.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.513+0000 7ff31ffff700 1 Processor -- start 2023-06-22T08:08:26.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.513+0000 7ff31ffff700 1 -- start start 2023-06-22T08:08:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.514+0000 7ff31ffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff31814f850 0x7ff3180c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.514+0000 7ff31ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff318071070 0x7ff3180c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.514+0000 7ff31ffff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3180c51a0 con 0x7ff3180c69e0 2023-06-22T08:08:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.514+0000 7ff31ffff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3180c44e0 con 0x7ff31814f850 2023-06-22T08:08:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.514+0000 7ff31ffff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3180c5440 con 0x7ff318071070 2023-06-22T08:08:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31e59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff318071070 0x7ff3180c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31dd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff31814f850 0x7ff3180c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31d59a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff3180c69e0 0x7ff318155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39462/0 (socket says 172.21.15.23:39462) 2023-06-22T08:08:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31d59a700 1 -- 172.21.15.23:0/154835680 learned_addr learned my addr 172.21.15.23:0/154835680 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:26.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31dd9b700 1 -- 172.21.15.23:0/154835680 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff318071070 msgr2=0x7ff3180c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31dd9b700 1 --2- 172.21.15.23:0/154835680 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff318071070 0x7ff3180c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31dd9b700 1 -- 172.21.15.23:0/154835680 >> v1:172.21.15.67:6789/0 conn(0x7ff3180c69e0 legacy=0x7ff318155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31dd9b700 1 -- 172.21.15.23:0/154835680 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3180c5720 con 0x7ff31814f850 2023-06-22T08:08:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31cd99700 1 -- 172.21.15.23:0/154835680 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961194252 0 0) 0x7ff3180c51a0 con 0x7ff3180c69e0 2023-06-22T08:08:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31dd9b700 1 --2- 172.21.15.23:0/154835680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31814f850 0x7ff3180c3080 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff314009840 tx=0x7ff314009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8cd5ddb626834e2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.515+0000 7ff31cd99700 1 -- 172.21.15.23:0/154835680 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314011830 con 0x7ff31814f850 2023-06-22T08:08:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.516+0000 7ff31cd99700 1 -- 172.21.15.23:0/154835680 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff314011990 con 0x7ff31814f850 2023-06-22T08:08:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.516+0000 7ff31cd99700 1 -- 172.21.15.23:0/154835680 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314011d20 con 0x7ff31814f850 2023-06-22T08:08:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.516+0000 7ff31ffff700 1 -- 172.21.15.23:0/154835680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31814f850 msgr2=0x7ff3180c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.516+0000 7ff31ffff700 1 --2- 172.21.15.23:0/154835680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31814f850 0x7ff3180c3080 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff314009840 tx=0x7ff314009b50 comp rx=0 tx=0).stop 2023-06-22T08:08:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.516+0000 7ff31ffff700 1 -- 172.21.15.23:0/154835680 shutdown_connections 2023-06-22T08:08:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.516+0000 7ff31ffff700 1 --2- 172.21.15.23:0/154835680 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff318071070 0x7ff3180c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.516+0000 7ff31ffff700 1 --2- 172.21.15.23:0/154835680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31814f850 0x7ff3180c3080 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.516+0000 7ff31ffff700 1 -- 172.21.15.23:0/154835680 >> 172.21.15.23:0/154835680 conn(0x7ff3180ca3f0 msgr2=0x7ff3180ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.516+0000 7ff31ffff700 1 -- 172.21.15.23:0/154835680 shutdown_connections 2023-06-22T08:08:26.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.516+0000 7ff31ffff700 1 -- 172.21.15.23:0/154835680 wait complete. 2023-06-22T08:08:26.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.519+0000 7ff31ffff700 1 Processor -- start 2023-06-22T08:08:26.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.519+0000 7ff31ffff700 1 -- start start 2023-06-22T08:08:26.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff318071070 0x7ff318147180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3180c69e0 0x7ff318149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31814f850 0x7ff318149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31ffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3181dcc50 con 0x7ff318071070 2023-06-22T08:08:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31ffff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3181dd720 con 0x7ff31814f850 2023-06-22T08:08:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31ffff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31814adc0 con 0x7ff3180c69e0 2023-06-22T08:08:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff318071070 0x7ff318147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff318071070 0x7ff318147180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51526/0 (socket says 172.21.15.23:51526) 2023-06-22T08:08:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31dd9b700 1 -- 172.21.15.23:0/2352912912 learned_addr learned my addr 172.21.15.23:0/2352912912 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31d59a700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3180c69e0 0x7ff318149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.520+0000 7ff31e59c700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31814f850 0x7ff318149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.521+0000 7ff31dd9b700 1 -- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3180c69e0 msgr2=0x7ff318149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.521+0000 7ff31dd9b700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3180c69e0 0x7ff318149a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.521+0000 7ff31dd9b700 1 -- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31814f850 msgr2=0x7ff318149fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.521+0000 7ff31dd9b700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31814f850 0x7ff318149fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.521+0000 7ff31dd9b700 1 -- 172.21.15.23:0/2352912912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3140094a0 con 0x7ff318071070 2023-06-22T08:08:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.521+0000 7ff31dd9b700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff318071070 0x7ff318147180 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff314009fd0 tx=0x7ff314004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.521+0000 7ff30e7fc700 1 -- 172.21.15.23:0/2352912912 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3140036a0 con 0x7ff318071070 2023-06-22T08:08:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.521+0000 7ff30e7fc700 1 -- 172.21.15.23:0/2352912912 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31401e070 con 0x7ff318071070 2023-06-22T08:08:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.521+0000 7ff30e7fc700 1 -- 172.21.15.23:0/2352912912 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314003a60 con 0x7ff318071070 2023-06-22T08:08:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.521+0000 7ff31ffff700 1 -- 172.21.15.23:0/2352912912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3181476c0 con 0x7ff318071070 2023-06-22T08:08:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.522+0000 7ff31ffff700 1 -- 172.21.15.23:0/2352912912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3181ddee0 con 0x7ff318071070 2023-06-22T08:08:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.523+0000 7ff31ffff700 1 -- 172.21.15.23:0/2352912912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3180c2950 con 0x7ff318071070 2023-06-22T08:08:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.523+0000 7ff30e7fc700 1 -- 172.21.15.23:0/2352912912 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff314003bc0 con 0x7ff318071070 2023-06-22T08:08:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.524+0000 7ff30e7fc700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff2fc06ace0 0x7ff2fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.525+0000 7ff30e7fc700 1 -- 172.21.15.23:0/2352912912 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7ff31408a880 con 0x7ff318071070 2023-06-22T08:08:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.525+0000 7ff31d59a700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff2fc06ace0 0x7ff2fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.525+0000 7ff31d59a700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff2fc06ace0 0x7ff2fc06d190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff3180c11e0 tx=0x7ff308007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:26.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.529+0000 7ff30e7fc700 1 -- 172.21.15.23:0/2352912912 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff314055450 con 0x7ff318071070 2023-06-22T08:08:26.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.792+0000 7ff31ffff700 1 -- 172.21.15.23:0/2352912912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff31804ea20 con 0x7ff318071070 2023-06-22T08:08:27.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.792+0000 7ff30e7fc700 1 -- 172.21.15.23:0/2352912912 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+6265 (secure 0 0 0) 0x7ff314058a70 con 0x7ff318071070 2023-06-22T08:08:27.163 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:27.163 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5707,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:22.771994+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:27.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 -- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff2fc06ace0 msgr2=0x7ff2fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:27.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff2fc06ace0 0x7ff2fc06d190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff3180c11e0 tx=0x7ff308007de0 comp rx=0 tx=0).stop 2023-06-22T08:08:27.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 -- 172.21.15.23:0/2352912912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff318071070 msgr2=0x7ff318147180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:27.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff318071070 0x7ff318147180 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff314009fd0 tx=0x7ff314004750 comp rx=0 tx=0).stop 2023-06-22T08:08:27.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 -- 172.21.15.23:0/2352912912 shutdown_connections 2023-06-22T08:08:27.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff2fc06ace0 0x7ff2fc06d190 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:27.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff31814f850 0x7ff318149fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:27.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff318071070 0x7ff318147180 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:27.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 --2- 172.21.15.23:0/2352912912 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3180c69e0 0x7ff318149a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:27.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 -- 172.21.15.23:0/2352912912 >> 172.21.15.23:0/2352912912 conn(0x7ff3180ca3f0 msgr2=0x7ff318069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:27.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 -- 172.21.15.23:0/2352912912 shutdown_connections 2023-06-22T08:08:27.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:26.795+0000 7ff31ffff700 1 -- 172.21.15.23:0/2352912912 wait complete. 2023-06-22T08:08:27.166 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 70 2023-06-22T08:08:27.167 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-06-22T08:08:29.168 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:29.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.378+0000 7f187f3ff700 1 Processor -- start 2023-06-22T08:08:29.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.378+0000 7f187f3ff700 1 -- start start 2023-06-22T08:08:29.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.379+0000 7f187f3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1878166380 0x7f187806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:29.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.379+0000 7f187f3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878069eb0 0x7f187806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:29.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.379+0000 7f187f3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18780c6b20 con 0x7f187806d530 2023-06-22T08:08:29.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.379+0000 7f187f3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187806d050 con 0x7f1878166380 2023-06-22T08:08:29.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.379+0000 7f187f3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1878071420 con 0x7f1878069eb0 2023-06-22T08:08:29.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.379+0000 7f187c99a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878069eb0 0x7f187806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:29.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.379+0000 7f187d19b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1878166380 0x7f187806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:29.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.379+0000 7f187d99c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f187806d530 0x7f187806d990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39472/0 (socket says 172.21.15.23:39472) 2023-06-22T08:08:29.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.379+0000 7f187c99a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878069eb0 0x7f187806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:56396/0 (socket says 172.21.15.23:56396) 2023-06-22T08:08:29.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.379+0000 7f187d99c700 1 -- 172.21.15.23:0/2970981063 learned_addr learned my addr 172.21.15.23:0/2970981063 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:29.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.448+0000 7f187d19b700 1 -- 172.21.15.23:0/2970981063 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878069eb0 msgr2=0x7f187806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:29.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.448+0000 7f187d19b700 1 --2- 172.21.15.23:0/2970981063 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878069eb0 0x7f187806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:29.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.448+0000 7f187d19b700 1 -- 172.21.15.23:0/2970981063 >> v1:172.21.15.67:6789/0 conn(0x7f187806d530 legacy=0x7f187806d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:29.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.448+0000 7f187d19b700 1 -- 172.21.15.23:0/2970981063 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1878071700 con 0x7f1878166380 2023-06-22T08:08:29.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.448+0000 7f187d19b700 1 --2- 172.21.15.23:0/2970981063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1878166380 0x7f187806fcb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f187400b8f0 tx=0x7f187400bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5bc5501c8156b1 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:29.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.448+0000 7f186ffff700 1 -- 172.21.15.23:0/2970981063 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1874009030 con 0x7f1878166380 2023-06-22T08:08:29.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.448+0000 7f186ffff700 1 -- 172.21.15.23:0/2970981063 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f187400caf0 con 0x7f1878166380 2023-06-22T08:08:29.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.448+0000 7f186ffff700 1 -- 172.21.15.23:0/2970981063 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f187400cea0 con 0x7f1878166380 2023-06-22T08:08:29.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.449+0000 7f187f3ff700 1 -- 172.21.15.23:0/2970981063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1878166380 msgr2=0x7f187806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:29.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.449+0000 7f187f3ff700 1 --2- 172.21.15.23:0/2970981063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1878166380 0x7f187806fcb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f187400b8f0 tx=0x7f187400bc00 comp rx=0 tx=0).stop 2023-06-22T08:08:29.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.449+0000 7f187f3ff700 1 -- 172.21.15.23:0/2970981063 shutdown_connections 2023-06-22T08:08:29.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.449+0000 7f187f3ff700 1 --2- 172.21.15.23:0/2970981063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1878166380 0x7f187806fcb0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:29.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.449+0000 7f187f3ff700 1 --2- 172.21.15.23:0/2970981063 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878069eb0 0x7f187806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:29.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.449+0000 7f187f3ff700 1 -- 172.21.15.23:0/2970981063 >> 172.21.15.23:0/2970981063 conn(0x7f18780ca3f0 msgr2=0x7f18780ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:29.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.450+0000 7f187f3ff700 1 -- 172.21.15.23:0/2970981063 shutdown_connections 2023-06-22T08:08:29.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.450+0000 7f187f3ff700 1 -- 172.21.15.23:0/2970981063 wait complete. 2023-06-22T08:08:29.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.452+0000 7f187f3ff700 1 Processor -- start 2023-06-22T08:08:29.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.453+0000 7f187f3ff700 1 -- start start 2023-06-22T08:08:29.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.454+0000 7f187f3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1878069eb0 0x7f18781565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:29.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.454+0000 7f187f3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f187806d530 0x7f18780c3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:29.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.454+0000 7f187f3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878166380 0x7f18781fdf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:29.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.454+0000 7f187f3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187815b5c0 con 0x7f187806d530 2023-06-22T08:08:29.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.454+0000 7f187f3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18780c2860 con 0x7f1878069eb0 2023-06-22T08:08:29.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.454+0000 7f187f3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18780c2dc0 con 0x7f1878166380 2023-06-22T08:08:29.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187c99a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f187806d530 0x7f18780c3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:29.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187c99a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f187806d530 0x7f18780c3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51536/0 (socket says 172.21.15.23:51536) 2023-06-22T08:08:29.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187c99a700 1 -- 172.21.15.23:0/2215770980 learned_addr learned my addr 172.21.15.23:0/2215770980 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:29.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187d99c700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878166380 0x7f18781fdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:29.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187d19b700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1878069eb0 0x7f18781565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:29.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187c99a700 1 -- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878166380 msgr2=0x7f18781fdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:29.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187c99a700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878166380 0x7f18781fdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:29.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187c99a700 1 -- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1878069eb0 msgr2=0x7f18781565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:29.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187c99a700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1878069eb0 0x7f18781565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:29.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187c99a700 1 -- 172.21.15.23:0/2215770980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187400b550 con 0x7f187806d530 2023-06-22T08:08:29.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.455+0000 7f187c99a700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f187806d530 0x7f18780c3250 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f186800dd40 tx=0x7f18680103a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:29.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.456+0000 7f186dffb700 1 -- 172.21.15.23:0/2215770980 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186800dbb0 con 0x7f187806d530 2023-06-22T08:08:29.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.456+0000 7f186dffb700 1 -- 172.21.15.23:0/2215770980 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1868013040 con 0x7f187806d530 2023-06-22T08:08:29.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.456+0000 7f187f3ff700 1 -- 172.21.15.23:0/2215770980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1878165780 con 0x7f187806d530 2023-06-22T08:08:29.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.456+0000 7f186dffb700 1 -- 172.21.15.23:0/2215770980 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1868010ca0 con 0x7f187806d530 2023-06-22T08:08:29.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.456+0000 7f187f3ff700 1 -- 172.21.15.23:0/2215770980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18781fed50 con 0x7f187806d530 2023-06-22T08:08:29.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.457+0000 7f187f3ff700 1 -- 172.21.15.23:0/2215770980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187804ea20 con 0x7f187806d530 2023-06-22T08:08:29.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.458+0000 7f186dffb700 1 -- 172.21.15.23:0/2215770980 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1868018480 con 0x7f187806d530 2023-06-22T08:08:29.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.459+0000 7f186dffb700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f185c06ace0 0x7f185c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:29.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.459+0000 7f186dffb700 1 -- 172.21.15.23:0/2215770980 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f186808db50 con 0x7f187806d530 2023-06-22T08:08:29.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.459+0000 7f187d19b700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f185c06ace0 0x7f185c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:29.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.460+0000 7f187d19b700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f185c06ace0 0x7f185c06d190 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1874018040 tx=0x7f1874008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:29.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.464+0000 7f186dffb700 1 -- 172.21.15.23:0/2215770980 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f18680587a0 con 0x7f187806d530 2023-06-22T08:08:29.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.743+0000 7f187f3ff700 1 -- 172.21.15.23:0/2215770980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18780c2f00 con 0x7f187806d530 2023-06-22T08:08:29.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.743+0000 7f186dffb700 1 -- 172.21.15.23:0/2215770980 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6267 (secure 0 0 0) 0x7f186805bdc0 con 0x7f187806d530 2023-06-22T08:08:29.747 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:29.747 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5707,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:27.786586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:29.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 -- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f185c06ace0 msgr2=0x7f185c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:29.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f185c06ace0 0x7f185c06d190 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1874018040 tx=0x7f1874008040 comp rx=0 tx=0).stop 2023-06-22T08:08:29.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 -- 172.21.15.23:0/2215770980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f187806d530 msgr2=0x7f18780c3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:29.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f187806d530 0x7f18780c3250 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f186800dd40 tx=0x7f18680103a0 comp rx=0 tx=0).stop 2023-06-22T08:08:29.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 -- 172.21.15.23:0/2215770980 shutdown_connections 2023-06-22T08:08:29.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1878166380 0x7f18781fdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:29.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f185c06ace0 0x7f185c06d190 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:29.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1878069eb0 0x7f18781565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:29.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 --2- 172.21.15.23:0/2215770980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f187806d530 0x7f18780c3250 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:29.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 -- 172.21.15.23:0/2215770980 >> 172.21.15.23:0/2215770980 conn(0x7f18780ca3f0 msgr2=0x7f187806a280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:29.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 -- 172.21.15.23:0/2215770980 shutdown_connections 2023-06-22T08:08:29.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:29.746+0000 7f187f3ff700 1 -- 172.21.15.23:0/2215770980 wait complete. 2023-06-22T08:08:29.754 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 71 2023-06-22T08:08:29.768 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-06-22T08:08:31.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:31.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.976+0000 7f2e805ff700 1 Processor -- start 2023-06-22T08:08:31.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.976+0000 7f2e805ff700 1 -- start start 2023-06-22T08:08:31.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e805ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e78152d70 0x7f2e780c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:31.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e805ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2e780c45f0 0x7f2e780c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:31.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e805ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e780c4fb0 con 0x7f2e78159330 2023-06-22T08:08:31.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e805ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e780c95e0 con 0x7f2e780c45f0 2023-06-22T08:08:31.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e805ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7815bea0 con 0x7f2e78152d70 2023-06-22T08:08:31.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e7e39b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e78152d70 0x7f2e780c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e7db9a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2e78159330 0x7f2e780cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:46508/0 (socket says 172.21.15.23:46508) 2023-06-22T08:08:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e7e39b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e78152d70 0x7f2e780c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:56410/0 (socket says 172.21.15.23:56410) 2023-06-22T08:08:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e7eb9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2e780c45f0 0x7f2e780c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e7db9a700 1 -- 172.21.15.23:0/103644167 learned_addr learned my addr 172.21.15.23:0/103644167 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e7eb9c700 1 -- 172.21.15.23:0/103644167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e78152d70 msgr2=0x7f2e780c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e7eb9c700 1 --2- 172.21.15.23:0/103644167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e78152d70 0x7f2e780c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e7eb9c700 1 -- 172.21.15.23:0/103644167 >> v1:172.21.15.23:6789/0 conn(0x7f2e78159330 legacy=0x7f2e780cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.978+0000 7f2e7eb9c700 1 -- 172.21.15.23:0/103644167 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7815c160 con 0x7f2e780c45f0 2023-06-22T08:08:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.979+0000 7f2e7e39b700 1 --2- 172.21.15.23:0/103644167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e78152d70 0x7f2e780c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.979+0000 7f2e7eb9c700 1 --2- 172.21.15.23:0/103644167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e780c45f0 0x7f2e780c4a70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009d90 tx=0x7f2e7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7059080a08db2c60 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.979+0000 7f2e7d399700 1 -- 172.21.15.23:0/103644167 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7000e070 con 0x7f2e780c45f0 2023-06-22T08:08:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.979+0000 7f2e7d399700 1 -- 172.21.15.23:0/103644167 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7000bb20 con 0x7f2e780c45f0 2023-06-22T08:08:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.979+0000 7f2e7d399700 1 -- 172.21.15.23:0/103644167 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e70012430 con 0x7f2e780c45f0 2023-06-22T08:08:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.980+0000 7f2e805ff700 1 -- 172.21.15.23:0/103644167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e780c45f0 msgr2=0x7f2e780c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.980+0000 7f2e805ff700 1 --2- 172.21.15.23:0/103644167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e780c45f0 0x7f2e780c4a70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009d90 tx=0x7f2e7000b330 comp rx=0 tx=0).stop 2023-06-22T08:08:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.980+0000 7f2e805ff700 1 -- 172.21.15.23:0/103644167 shutdown_connections 2023-06-22T08:08:31.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.980+0000 7f2e805ff700 1 --2- 172.21.15.23:0/103644167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e78152d70 0x7f2e780c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:31.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.980+0000 7f2e805ff700 1 --2- 172.21.15.23:0/103644167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e780c45f0 0x7f2e780c4a70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:31.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.980+0000 7f2e805ff700 1 -- 172.21.15.23:0/103644167 >> 172.21.15.23:0/103644167 conn(0x7f2e7806ae30 msgr2=0x7f2e7806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:31.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.980+0000 7f2e805ff700 1 -- 172.21.15.23:0/103644167 shutdown_connections 2023-06-22T08:08:31.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.980+0000 7f2e805ff700 1 -- 172.21.15.23:0/103644167 wait complete. 2023-06-22T08:08:31.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.981+0000 7f2e805ff700 1 Processor -- start 2023-06-22T08:08:31.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.982+0000 7f2e805ff700 1 -- start start 2023-06-22T08:08:31.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e805ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e780c45f0 0x7f2e780b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:31.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e805ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e78152d70 0x7f2e7814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:31.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e805ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78159330 0x7f2e7814fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:31.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e805ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e78150950 con 0x7f2e78159330 2023-06-22T08:08:31.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e805ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e78150b80 con 0x7f2e78152d70 2023-06-22T08:08:31.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e805ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780ce200 con 0x7f2e780c45f0 2023-06-22T08:08:31.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e7eb9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78159330 0x7f2e7814fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:31.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e7db9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e78152d70 0x7f2e7814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:31.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e7eb9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78159330 0x7f2e7814fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51552/0 (socket says 172.21.15.23:51552) 2023-06-22T08:08:31.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e7eb9c700 1 -- 172.21.15.23:0/809175375 learned_addr learned my addr 172.21.15.23:0/809175375 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:31.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e7e39b700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e780c45f0 0x7f2e780b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:31.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e7eb9c700 1 -- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e780c45f0 msgr2=0x7f2e780b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e7eb9c700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e780c45f0 0x7f2e780b36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e7eb9c700 1 -- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e78152d70 msgr2=0x7f2e7814f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e7eb9c700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e78152d70 0x7f2e7814f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.983+0000 7f2e7eb9c700 1 -- 172.21.15.23:0/809175375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e740094a0 con 0x7f2e78159330 2023-06-22T08:08:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.984+0000 7f2e7eb9c700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78159330 0x7f2e7814fb60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2e7808bcd0 tx=0x7f2e70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:31.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.984+0000 7f2e6effd700 1 -- 172.21.15.23:0/809175375 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e70012430 con 0x7f2e78159330 2023-06-22T08:08:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.984+0000 7f2e6effd700 1 -- 172.21.15.23:0/809175375 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e70020070 con 0x7f2e78159330 2023-06-22T08:08:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.984+0000 7f2e6effd700 1 -- 172.21.15.23:0/809175375 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e70003ea0 con 0x7f2e78159330 2023-06-22T08:08:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.984+0000 7f2e805ff700 1 -- 172.21.15.23:0/809175375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e70009a70 con 0x7f2e78159330 2023-06-22T08:08:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.984+0000 7f2e805ff700 1 -- 172.21.15.23:0/809175375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e780cf430 con 0x7f2e78159330 2023-06-22T08:08:32.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.985+0000 7f2e805ff700 1 -- 172.21.15.23:0/809175375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7804ea20 con 0x7f2e78159330 2023-06-22T08:08:32.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.986+0000 7f2e6effd700 1 -- 172.21.15.23:0/809175375 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2e70004040 con 0x7f2e78159330 2023-06-22T08:08:32.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.988+0000 7f2e6effd700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e5c06ad10 0x7f2e5c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:32.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.988+0000 7f2e6effd700 1 -- 172.21.15.23:0/809175375 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 7157+0+0 (secure 0 0 0) 0x7f2e70022070 con 0x7f2e78159330 2023-06-22T08:08:32.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.988+0000 7f2e7e39b700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e5c06ad10 0x7f2e5c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:32.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.989+0000 7f2e7e39b700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e5c06ad10 0x7f2e5c06d1c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e74009d70 tx=0x7f2e74009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:32.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:31.992+0000 7f2e6effd700 1 -- 172.21.15.23:0/809175375 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2e70055f60 con 0x7f2e78159330 2023-06-22T08:08:32.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.243+0000 7f2e805ff700 1 -- 172.21.15.23:0/809175375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e780cf410 con 0x7f2e78159330 2023-06-22T08:08:32.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.244+0000 7f2e6effd700 1 -- 172.21.15.23:0/809175375 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6267 (secure 0 0 0) 0x7f2e70059580 con 0x7f2e78159330 2023-06-22T08:08:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:32.247 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5707,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:27.786586+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":4279,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"b","rank":1,"incarnation":10,"state":"up:active","state_seq":11,"addr":"172.21.15.67:6836/1962314516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1962314516},{"type":"v1","addr":"172.21.15.67:6836","nonce":1962314516}]},"join_fscid":-1,"export_targets":[0,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:32.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 -- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e5c06ad10 msgr2=0x7f2e5c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:32.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e5c06ad10 0x7f2e5c06d1c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e74009d70 tx=0x7f2e74009210 comp rx=0 tx=0).stop 2023-06-22T08:08:32.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 -- 172.21.15.23:0/809175375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78159330 msgr2=0x7f2e7814fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:32.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78159330 0x7f2e7814fb60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2e7808bcd0 tx=0x7f2e70004ad0 comp rx=0 tx=0).stop 2023-06-22T08:08:32.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 -- 172.21.15.23:0/809175375 shutdown_connections 2023-06-22T08:08:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78159330 0x7f2e7814fb60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e780c45f0 0x7f2e780b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e5c06ad10 0x7f2e5c06d1c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:32.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 --2- 172.21.15.23:0/809175375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e78152d70 0x7f2e7814f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:32.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 -- 172.21.15.23:0/809175375 >> 172.21.15.23:0/809175375 conn(0x7f2e7806ae30 msgr2=0x7f2e78151bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:32.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 -- 172.21.15.23:0/809175375 shutdown_connections 2023-06-22T08:08:32.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:32.246+0000 7f2e805ff700 1 -- 172.21.15.23:0/809175375 wait complete. 2023-06-22T08:08:32.253 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 71 2023-06-22T08:08:32.267 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-06-22T08:08:34.269 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:34.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.480+0000 7ffb571ff700 1 Processor -- start 2023-06-22T08:08:34.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.481+0000 7ffb571ff700 1 -- start start 2023-06-22T08:08:34.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.482+0000 7ffb571ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb50071850 0x7ffb500c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.482+0000 7ffb571ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb500c4a30 0x7ffb5015c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.482+0000 7ffb571ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb5015bf50 con 0x7ffb50071c20 2023-06-22T08:08:34.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.482+0000 7ffb571ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb500c9550 con 0x7ffb50071850 2023-06-22T08:08:34.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.482+0000 7ffb571ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb500c4eb0 con 0x7ffb500c4a30 2023-06-22T08:08:34.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.482+0000 7ffb54f9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb50071850 0x7ffb500c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.482+0000 7ffb4ffff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ffb50071c20 0x7ffb50152750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:46518/0 (socket says 172.21.15.23:46518) 2023-06-22T08:08:34.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.482+0000 7ffb4ffff700 1 -- 172.21.15.23:0/2371030708 learned_addr learned my addr 172.21.15.23:0/2371030708 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:34.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.482+0000 7ffb5579c700 1 --2- 172.21.15.23:0/2371030708 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb500c4a30 0x7ffb5015c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb4f7fe700 1 -- 172.21.15.23:0/2371030708 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235271349 0 0) 0x7ffb5015bf50 con 0x7ffb50071c20 2023-06-22T08:08:34.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb4f7fe700 1 -- 172.21.15.23:0/2371030708 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb400035b0 con 0x7ffb50071c20 2023-06-22T08:08:34.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb5579c700 1 -- 172.21.15.23:0/2371030708 >> v2:172.21.15.67:3300/0 conn(0x7ffb50071850 msgr2=0x7ffb500c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb5579c700 1 --2- 172.21.15.23:0/2371030708 >> v2:172.21.15.67:3300/0 conn(0x7ffb50071850 0x7ffb500c9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb5579c700 1 -- 172.21.15.23:0/2371030708 >> v1:172.21.15.23:6789/0 conn(0x7ffb50071c20 legacy=0x7ffb50152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb5579c700 1 -- 172.21.15.23:0/2371030708 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50158c00 con 0x7ffb500c4a30 2023-06-22T08:08:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb54f9b700 1 --2- 172.21.15.23:0/2371030708 >> v2:172.21.15.67:3300/0 conn(0x7ffb50071850 0x7ffb500c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb5579c700 1 --2- 172.21.15.23:0/2371030708 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb500c4a30 0x7ffb5015c3e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffb48009d90 tx=0x7ffb4800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97ff6a6123470c24 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb4f7fe700 1 -- 172.21.15.23:0/2371030708 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4800e070 con 0x7ffb500c4a30 2023-06-22T08:08:34.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb4f7fe700 1 -- 172.21.15.23:0/2371030708 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4800bb20 con 0x7ffb500c4a30 2023-06-22T08:08:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.483+0000 7ffb4f7fe700 1 -- 172.21.15.23:0/2371030708 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48012430 con 0x7ffb500c4a30 2023-06-22T08:08:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.484+0000 7ffb571ff700 1 -- 172.21.15.23:0/2371030708 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb500c4a30 msgr2=0x7ffb5015c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.484+0000 7ffb571ff700 1 --2- 172.21.15.23:0/2371030708 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb500c4a30 0x7ffb5015c3e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffb48009d90 tx=0x7ffb4800b330 comp rx=0 tx=0).stop 2023-06-22T08:08:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.484+0000 7ffb571ff700 1 -- 172.21.15.23:0/2371030708 shutdown_connections 2023-06-22T08:08:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.484+0000 7ffb571ff700 1 --2- 172.21.15.23:0/2371030708 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb500c4a30 0x7ffb5015c3e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.484+0000 7ffb571ff700 1 --2- 172.21.15.23:0/2371030708 >> v2:172.21.15.67:3300/0 conn(0x7ffb50071850 0x7ffb500c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.484+0000 7ffb571ff700 1 -- 172.21.15.23:0/2371030708 >> 172.21.15.23:0/2371030708 conn(0x7ffb5006b8b0 msgr2=0x7ffb5006bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:34.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.484+0000 7ffb571ff700 1 -- 172.21.15.23:0/2371030708 shutdown_connections 2023-06-22T08:08:34.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.484+0000 7ffb571ff700 1 -- 172.21.15.23:0/2371030708 wait complete. 2023-06-22T08:08:34.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.486+0000 7ffb571ff700 1 Processor -- start 2023-06-22T08:08:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.486+0000 7ffb571ff700 1 -- start start 2023-06-22T08:08:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb571ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50071850 0x7ffb501541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb571ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb50071c20 0x7ffb5014b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb571ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb500c4a30 0x7ffb5014b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb571ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb501ee8e0 con 0x7ffb500c4a30 2023-06-22T08:08:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb571ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb501eee10 con 0x7ffb50071850 2023-06-22T08:08:34.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb571ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb500ceba0 con 0x7ffb50071c20 2023-06-22T08:08:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb54f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50071850 0x7ffb501541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb5579c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb500c4a30 0x7ffb5014b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb4ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb50071c20 0x7ffb5014b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb54f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50071850 0x7ffb501541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:53828/0 (socket says 172.21.15.23:53828) 2023-06-22T08:08:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb54f9b700 1 -- 172.21.15.23:0/3310232452 learned_addr learned my addr 172.21.15.23:0/3310232452 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb5579c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb500c4a30 0x7ffb5014b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51568/0 (socket says 172.21.15.23:51568) 2023-06-22T08:08:34.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb5579c700 1 -- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb50071c20 msgr2=0x7ffb5014b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb5579c700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb50071c20 0x7ffb5014b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb5579c700 1 -- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50071850 msgr2=0x7ffb501541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.487+0000 7ffb5579c700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50071850 0x7ffb501541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.488+0000 7ffb5579c700 1 -- 172.21.15.23:0/3310232452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb440094a0 con 0x7ffb500c4a30 2023-06-22T08:08:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.488+0000 7ffb54f9b700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50071850 0x7ffb501541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.488+0000 7ffb5579c700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb500c4a30 0x7ffb5014b870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffb500cf8d0 tx=0x7ffb48004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.488+0000 7ffb4d7fa700 1 -- 172.21.15.23:0/3310232452 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48012430 con 0x7ffb500c4a30 2023-06-22T08:08:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.488+0000 7ffb4d7fa700 1 -- 172.21.15.23:0/3310232452 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb48020070 con 0x7ffb500c4a30 2023-06-22T08:08:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.488+0000 7ffb4d7fa700 1 -- 172.21.15.23:0/3310232452 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb480098f0 con 0x7ffb500c4a30 2023-06-22T08:08:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.488+0000 7ffb571ff700 1 -- 172.21.15.23:0/3310232452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb48009a70 con 0x7ffb500c4a30 2023-06-22T08:08:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.488+0000 7ffb571ff700 1 -- 172.21.15.23:0/3310232452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb500ce270 con 0x7ffb500c4a30 2023-06-22T08:08:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.489+0000 7ffb571ff700 1 -- 172.21.15.23:0/3310232452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb500cbfc0 con 0x7ffb500c4a30 2023-06-22T08:08:34.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.490+0000 7ffb4d7fa700 1 -- 172.21.15.23:0/3310232452 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffb48004040 con 0x7ffb500c4a30 2023-06-22T08:08:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.492+0000 7ffb4d7fa700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb3406ace0 0x7ffb3406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.492+0000 7ffb4d7fa700 1 -- 172.21.15.23:0/3310232452 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7ffb48022070 con 0x7ffb500c4a30 2023-06-22T08:08:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.492+0000 7ffb54f9b700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb3406ace0 0x7ffb3406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.493+0000 7ffb54f9b700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb3406ace0 0x7ffb3406d190 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ffb44011530 tx=0x7ffb44009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:34.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.496+0000 7ffb4d7fa700 1 -- 172.21.15.23:0/3310232452 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffb480560e0 con 0x7ffb500c4a30 2023-06-22T08:08:34.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.753+0000 7ffb571ff700 1 -- 172.21.15.23:0/3310232452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb50159190 con 0x7ffb500c4a30 2023-06-22T08:08:34.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.754+0000 7ffb4d7fa700 1 -- 172.21.15.23:0/3310232452 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5507 (secure 0 0 0) 0x7ffb48059700 con 0x7ffb500c4a30 2023-06-22T08:08:34.758 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:34.758 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:34.378619+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:34.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.756+0000 7ffb571ff700 1 -- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb3406ace0 msgr2=0x7ffb3406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:34.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.756+0000 7ffb571ff700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb3406ace0 0x7ffb3406d190 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ffb44011530 tx=0x7ffb44009210 comp rx=0 tx=0).stop 2023-06-22T08:08:34.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.756+0000 7ffb571ff700 1 -- 172.21.15.23:0/3310232452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb500c4a30 msgr2=0x7ffb5014b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:34.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.756+0000 7ffb571ff700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb500c4a30 0x7ffb5014b870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffb500cf8d0 tx=0x7ffb48004ad0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.756+0000 7ffb571ff700 1 -- 172.21.15.23:0/3310232452 shutdown_connections 2023-06-22T08:08:34.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.756+0000 7ffb571ff700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb500c4a30 0x7ffb5014b870 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.757+0000 7ffb571ff700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb3406ace0 0x7ffb3406d190 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.757+0000 7ffb571ff700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb50071850 0x7ffb501541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.757+0000 7ffb571ff700 1 --2- 172.21.15.23:0/3310232452 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb50071c20 0x7ffb5014b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.757+0000 7ffb571ff700 1 -- 172.21.15.23:0/3310232452 >> 172.21.15.23:0/3310232452 conn(0x7ffb5006b8b0 msgr2=0x7ffb500b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:34.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.757+0000 7ffb571ff700 1 -- 172.21.15.23:0/3310232452 shutdown_connections 2023-06-22T08:08:34.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.757+0000 7ffb571ff700 1 -- 172.21.15.23:0/3310232452 wait complete. 2023-06-22T08:08:34.762 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 72 2023-06-22T08:08:34.778 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-06-22T08:08:34.778 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:08:34.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:34.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.976+0000 7f3d527ff700 1 Processor -- start 2023-06-22T08:08:34.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.976+0000 7f3d527ff700 1 -- start start 2023-06-22T08:08:34.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.977+0000 7f3d527ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d4c152d70 0x7f3d4c0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.977+0000 7f3d527ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 0x7f3d4c0c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.977+0000 7f3d527ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4c0c4f90 con 0x7f3d4c159330 2023-06-22T08:08:34.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.977+0000 7f3d527ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4c0c95e0 con 0x7f3d4c152d70 2023-06-22T08:08:34.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.977+0000 7f3d527ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c159790 con 0x7f3d4c0c45f0 2023-06-22T08:08:34.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d50d9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 0x7f3d4c0c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d4b7fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3d4c159330 0x7f3d4c0cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39488/0 (socket says 172.21.15.23:39488) 2023-06-22T08:08:34.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d4bfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d4c152d70 0x7f3d4c0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d4b7fe700 1 -- 172.21.15.23:0/316954986 learned_addr learned my addr 172.21.15.23:0/316954986 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:34.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d4affd700 1 -- 172.21.15.23:0/316954986 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667860797 0 0) 0x7f3d4c0c4f90 con 0x7f3d4c159330 2023-06-22T08:08:34.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d4affd700 1 -- 172.21.15.23:0/316954986 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d340035b0 con 0x7f3d4c159330 2023-06-22T08:08:34.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d4bfff700 1 -- 172.21.15.23:0/316954986 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 msgr2=0x7f3d4c0c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:34.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d4bfff700 1 --2- 172.21.15.23:0/316954986 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 0x7f3d4c0c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d4bfff700 1 -- 172.21.15.23:0/316954986 >> v1:172.21.15.67:6789/0 conn(0x7f3d4c159330 legacy=0x7f3d4c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:34.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d4bfff700 1 -- 172.21.15.23:0/316954986 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c15bea0 con 0x7f3d4c152d70 2023-06-22T08:08:34.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d50d9c700 1 --2- 172.21.15.23:0/316954986 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 0x7f3d4c0c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:34.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.978+0000 7f3d4bfff700 1 --2- 172.21.15.23:0/316954986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152d70 0x7f3d4c0c97d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009840 tx=0x7f3d3c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47c423539cc740d9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:34.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d4affd700 1 -- 172.21.15.23:0/316954986 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c011830 con 0x7f3d4c152d70 2023-06-22T08:08:34.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d4affd700 1 -- 172.21.15.23:0/316954986 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3c011990 con 0x7f3d4c152d70 2023-06-22T08:08:34.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d4affd700 1 -- 172.21.15.23:0/316954986 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c011d20 con 0x7f3d4c152d70 2023-06-22T08:08:34.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d527ff700 1 -- 172.21.15.23:0/316954986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152d70 msgr2=0x7f3d4c0c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:34.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d527ff700 1 --2- 172.21.15.23:0/316954986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152d70 0x7f3d4c0c97d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009840 tx=0x7f3d3c009b50 comp rx=0 tx=0).stop 2023-06-22T08:08:34.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d527ff700 1 -- 172.21.15.23:0/316954986 shutdown_connections 2023-06-22T08:08:34.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d527ff700 1 --2- 172.21.15.23:0/316954986 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 0x7f3d4c0c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d527ff700 1 --2- 172.21.15.23:0/316954986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c152d70 0x7f3d4c0c97d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d527ff700 1 -- 172.21.15.23:0/316954986 >> 172.21.15.23:0/316954986 conn(0x7f3d4c06ae30 msgr2=0x7f3d4c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:34.985 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d527ff700 1 -- 172.21.15.23:0/316954986 shutdown_connections 2023-06-22T08:08:34.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.979+0000 7f3d527ff700 1 -- 172.21.15.23:0/316954986 wait complete. 2023-06-22T08:08:34.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.981+0000 7f3d527ff700 1 Processor -- start 2023-06-22T08:08:34.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.981+0000 7f3d527ff700 1 -- start start 2023-06-22T08:08:34.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.982+0000 7f3d527ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 0x7f3d4c14e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.982+0000 7f3d527ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d4c152d70 0x7f3d4c15b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.988 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.982+0000 7f3d527ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c159330 0x7f3d4c157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.982+0000 7f3d527ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c14cb10 con 0x7f3d4c152d70 2023-06-22T08:08:34.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.982+0000 7f3d527ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c14f880 con 0x7f3d4c159330 2023-06-22T08:08:34.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.982+0000 7f3d527ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0c8cc0 con 0x7f3d4c0c45f0 2023-06-22T08:08:34.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.982+0000 7f3d4b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d4c152d70 0x7f3d4c15b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.982+0000 7f3d4b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d4c152d70 0x7f3d4c15b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51580/0 (socket says 172.21.15.23:51580) 2023-06-22T08:08:34.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.982+0000 7f3d4bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 0x7f3d4c14e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.982+0000 7f3d4b7fe700 1 -- 172.21.15.23:0/3735285244 learned_addr learned my addr 172.21.15.23:0/3735285244 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:34.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d50d9c700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c159330 0x7f3d4c157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d4b7fe700 1 -- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 msgr2=0x7f3d4c14e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:34.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d4b7fe700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 0x7f3d4c14e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d4b7fe700 1 -- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c159330 msgr2=0x7f3d4c157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:34.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d4b7fe700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c159330 0x7f3d4c157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:34.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d4b7fe700 1 -- 172.21.15.23:0/3735285244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d44009a70 con 0x7f3d4c152d70 2023-06-22T08:08:34.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d4b7fe700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d4c152d70 0x7f3d4c15b0a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3d40001eb0 tx=0x7f3d4000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:34.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d48ff9700 1 -- 172.21.15.23:0/3735285244 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4000cea0 con 0x7f3d4c152d70 2023-06-22T08:08:34.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d48ff9700 1 -- 172.21.15.23:0/3735285244 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d40005d70 con 0x7f3d4c152d70 2023-06-22T08:08:34.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d48ff9700 1 -- 172.21.15.23:0/3735285244 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d400072f0 con 0x7f3d4c152d70 2023-06-22T08:08:34.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.983+0000 7f3d527ff700 1 -- 172.21.15.23:0/3735285244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c0094a0 con 0x7f3d4c152d70 2023-06-22T08:08:34.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.984+0000 7f3d527ff700 1 -- 172.21.15.23:0/3735285244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c14cfe0 con 0x7f3d4c152d70 2023-06-22T08:08:34.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.985+0000 7f3d527ff700 1 -- 172.21.15.23:0/3735285244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c04ea20 con 0x7f3d4c152d70 2023-06-22T08:08:34.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.986+0000 7f3d48ff9700 1 -- 172.21.15.23:0/3735285244 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3d40007450 con 0x7f3d4c152d70 2023-06-22T08:08:34.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.987+0000 7f3d48ff9700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d3806ad10 0x7f3d3806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:34.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.987+0000 7f3d48ff9700 1 -- 172.21.15.23:0/3735285244 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f3d40088dc0 con 0x7f3d4c152d70 2023-06-22T08:08:34.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.987+0000 7f3d4bfff700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d3806ad10 0x7f3d3806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:34.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.988+0000 7f3d4bfff700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d3806ad10 0x7f3d3806d1c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c000c00 tx=0x7f3d3c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:34.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:34.992+0000 7f3d48ff9700 1 -- 172.21.15.23:0/3735285244 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3d400538a0 con 0x7f3d4c152d70 2023-06-22T08:08:35.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.245+0000 7f3d527ff700 1 -- 172.21.15.23:0/3735285244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d4c0c9e00 con 0x7f3d4c152d70 2023-06-22T08:08:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.246+0000 7f3d48ff9700 1 -- 172.21.15.23:0/3735285244 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5507 (secure 0 0 0) 0x7f3d40017020 con 0x7f3d4c152d70 2023-06-22T08:08:35.249 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:35.250 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:34.378619+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.248+0000 7f3d527ff700 1 -- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d3806ad10 msgr2=0x7f3d3806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.248+0000 7f3d527ff700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d3806ad10 0x7f3d3806d1c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c000c00 tx=0x7f3d3c008030 comp rx=0 tx=0).stop 2023-06-22T08:08:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.248+0000 7f3d527ff700 1 -- 172.21.15.23:0/3735285244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d4c152d70 msgr2=0x7f3d4c15b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.248+0000 7f3d527ff700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d4c152d70 0x7f3d4c15b0a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3d40001eb0 tx=0x7f3d4000c7b0 comp rx=0 tx=0).stop 2023-06-22T08:08:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.249+0000 7f3d527ff700 1 -- 172.21.15.23:0/3735285244 shutdown_connections 2023-06-22T08:08:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.249+0000 7f3d527ff700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3d3806ad10 0x7f3d3806d1c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.249+0000 7f3d527ff700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d4c159330 0x7f3d4c157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.249+0000 7f3d527ff700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3d4c0c45f0 0x7f3d4c14e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.249+0000 7f3d527ff700 1 --2- 172.21.15.23:0/3735285244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d4c152d70 0x7f3d4c15b0a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.249+0000 7f3d527ff700 1 -- 172.21.15.23:0/3735285244 >> 172.21.15.23:0/3735285244 conn(0x7f3d4c06ae30 msgr2=0x7f3d4c151500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.249+0000 7f3d527ff700 1 -- 172.21.15.23:0/3735285244 shutdown_connections 2023-06-22T08:08:35.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:35.249+0000 7f3d527ff700 1 -- 172.21.15.23:0/3735285244 wait complete. 2023-06-22T08:08:35.254 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 72 2023-06-22T08:08:35.271 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 72, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 72, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4285": { "addr": "172.21.15.67:6839/567663545", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 567663545, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 567663545, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5060": { "addr": "172.21.15.67:6837/1551460358", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1551460358, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1551460358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5060, "incarnation": 46, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 31 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 26 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:08:34.378619+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 5707, "mds_2": 5551, "mds_3": 5060, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:08:37.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:37.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.497+0000 7fc2eb5ff700 1 Processor -- start 2023-06-22T08:08:37.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.497+0000 7fc2eb5ff700 1 -- start start 2023-06-22T08:08:37.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.498+0000 7fc2eb5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2e40cb6b0 0x7fc2e4069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:37.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.498+0000 7fc2eb5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40c5740 0x7fc2e41593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:37.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.498+0000 7fc2eb5ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e4158f60 con 0x7fc2e40cba80 2023-06-22T08:08:37.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.498+0000 7fc2eb5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e406bd20 con 0x7fc2e40cb6b0 2023-06-22T08:08:37.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.498+0000 7fc2eb5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e40cc450 con 0x7fc2e40c5740 2023-06-22T08:08:37.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.498+0000 7fc2e9b9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc2e40cba80 0x7fc2e4152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55310/0 (socket says 172.21.15.23:55310) 2023-06-22T08:08:37.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.498+0000 7fc2e9b9c700 1 -- 172.21.15.23:0/313619374 learned_addr learned my addr 172.21.15.23:0/313619374 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:37.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.498+0000 7fc2e939b700 1 --2- 172.21.15.23:0/313619374 >> v2:172.21.15.67:3300/0 conn(0x7fc2e40cb6b0 0x7fc2e4069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:37.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.498+0000 7fc2e8b9a700 1 --2- 172.21.15.23:0/313619374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40c5740 0x7fc2e41593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:37.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.499+0000 7fc2dbfff700 1 -- 172.21.15.23:0/313619374 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214954339 0 0) 0x7fc2e4158f60 con 0x7fc2e40cba80 2023-06-22T08:08:37.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.499+0000 7fc2e8b9a700 1 -- 172.21.15.23:0/313619374 >> v2:172.21.15.67:3300/0 conn(0x7fc2e40cb6b0 msgr2=0x7fc2e4069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.499+0000 7fc2e8b9a700 1 --2- 172.21.15.23:0/313619374 >> v2:172.21.15.67:3300/0 conn(0x7fc2e40cb6b0 0x7fc2e4069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.499+0000 7fc2e8b9a700 1 -- 172.21.15.23:0/313619374 >> v1:172.21.15.23:6789/0 conn(0x7fc2e40cba80 legacy=0x7fc2e4152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.499+0000 7fc2e8b9a700 1 -- 172.21.15.23:0/313619374 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e4158980 con 0x7fc2e40c5740 2023-06-22T08:08:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.499+0000 7fc2e939b700 1 --2- 172.21.15.23:0/313619374 >> v2:172.21.15.67:3300/0 conn(0x7fc2e40cb6b0 0x7fc2e4069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:37.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.499+0000 7fc2e8b9a700 1 --2- 172.21.15.23:0/313619374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40c5740 0x7fc2e41593f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009840 tx=0x7fc2d4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca3ed3929c707fca server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:37.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2dbfff700 1 -- 172.21.15.23:0/313619374 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4011830 con 0x7fc2e40c5740 2023-06-22T08:08:37.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2dbfff700 1 -- 172.21.15.23:0/313619374 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d4011990 con 0x7fc2e40c5740 2023-06-22T08:08:37.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2dbfff700 1 -- 172.21.15.23:0/313619374 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4011d20 con 0x7fc2e40c5740 2023-06-22T08:08:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/313619374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40c5740 msgr2=0x7fc2e41593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2eb5ff700 1 --2- 172.21.15.23:0/313619374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40c5740 0x7fc2e41593f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009840 tx=0x7fc2d4009b50 comp rx=0 tx=0).stop 2023-06-22T08:08:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/313619374 shutdown_connections 2023-06-22T08:08:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2eb5ff700 1 --2- 172.21.15.23:0/313619374 >> v2:172.21.15.67:3300/0 conn(0x7fc2e40cb6b0 0x7fc2e4069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2eb5ff700 1 --2- 172.21.15.23:0/313619374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40c5740 0x7fc2e41593f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/313619374 >> 172.21.15.23:0/313619374 conn(0x7fc2e4070d80 msgr2=0x7fc2e4071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/313619374 shutdown_connections 2023-06-22T08:08:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.500+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/313619374 wait complete. 2023-06-22T08:08:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.502+0000 7fc2eb5ff700 1 Processor -- start 2023-06-22T08:08:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.502+0000 7fc2eb5ff700 1 -- start start 2023-06-22T08:08:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2eb5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2e40cb6b0 0x7fc2e415ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2eb5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40cba80 0x7fc2e4157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2eb5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2e41eee20 0x7fc2e414b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2eb5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e415a140 con 0x7fc2e41eee20 2023-06-22T08:08:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2eb5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e415b970 con 0x7fc2e40cb6b0 2023-06-22T08:08:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2eb5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e41ee130 con 0x7fc2e40cba80 2023-06-22T08:08:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2e9b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2e41eee20 0x7fc2e414b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2e9b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2e41eee20 0x7fc2e414b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51948/0 (socket says 172.21.15.23:51948) 2023-06-22T08:08:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2e9b9c700 1 -- 172.21.15.23:0/1774285541 learned_addr learned my addr 172.21.15.23:0/1774285541 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2e939b700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2e40cb6b0 0x7fc2e415ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.503+0000 7fc2e8b9a700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40cba80 0x7fc2e4157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2e9b9c700 1 -- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40cba80 msgr2=0x7fc2e4157730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2e9b9c700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40cba80 0x7fc2e4157730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2e9b9c700 1 -- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2e40cb6b0 msgr2=0x7fc2e415ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2e9b9c700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2e40cb6b0 0x7fc2e415ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2e9b9c700 1 -- 172.21.15.23:0/1774285541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e00094a0 con 0x7fc2e41eee20 2023-06-22T08:08:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2e9b9c700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2e41eee20 0x7fc2e414b2f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00cc70 tx=0x7fc2dc00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2d9ffb700 1 -- 172.21.15.23:0/1774285541 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc00aaa0 con 0x7fc2e41eee20 2023-06-22T08:08:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2d9ffb700 1 -- 172.21.15.23:0/1774285541 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc00ac20 con 0x7fc2e41eee20 2023-06-22T08:08:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2d9ffb700 1 -- 172.21.15.23:0/1774285541 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0070d0 con 0x7fc2e41eee20 2023-06-22T08:08:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/1774285541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d40094a0 con 0x7fc2e41eee20 2023-06-22T08:08:37.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.504+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/1774285541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e40c27f0 con 0x7fc2e41eee20 2023-06-22T08:08:37.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.505+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/1774285541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e415b6f0 con 0x7fc2e41eee20 2023-06-22T08:08:37.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.506+0000 7fc2d9ffb700 1 -- 172.21.15.23:0/1774285541 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc2dc007230 con 0x7fc2e41eee20 2023-06-22T08:08:37.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.507+0000 7fc2d9ffb700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc2c806ace0 0x7fc2c806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:37.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.507+0000 7fc2d9ffb700 1 -- 172.21.15.23:0/1774285541 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7fc2dc089590 con 0x7fc2e41eee20 2023-06-22T08:08:37.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.507+0000 7fc2e939b700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc2c806ace0 0x7fc2c806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:37.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.508+0000 7fc2e939b700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc2c806ace0 0x7fc2c806d190 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009d70 tx=0x7fc2e0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:37.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.512+0000 7fc2d9ffb700 1 -- 172.21.15.23:0/1774285541 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc2dc054070 con 0x7fc2e41eee20 2023-06-22T08:08:37.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.765+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/1774285541 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2e404ea20 con 0x7fc2e41eee20 2023-06-22T08:08:37.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.766+0000 7fc2d9ffb700 1 -- 172.21.15.23:0/1774285541 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5507 (secure 0 0 0) 0x7fc2dc057690 con 0x7fc2e41eee20 2023-06-22T08:08:37.770 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:37.770 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:34.378619+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:37.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.768+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc2c806ace0 msgr2=0x7fc2c806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:37.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.768+0000 7fc2eb5ff700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc2c806ace0 0x7fc2c806d190 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009d70 tx=0x7fc2e0009210 comp rx=0 tx=0).stop 2023-06-22T08:08:37.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.768+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/1774285541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2e41eee20 msgr2=0x7fc2e414b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:37.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.768+0000 7fc2eb5ff700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2e41eee20 0x7fc2e414b2f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00cc70 tx=0x7fc2dc00cf80 comp rx=0 tx=0).stop 2023-06-22T08:08:37.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.768+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/1774285541 shutdown_connections 2023-06-22T08:08:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.768+0000 7fc2eb5ff700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2e41eee20 0x7fc2e414b2f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.768+0000 7fc2eb5ff700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc2c806ace0 0x7fc2c806d190 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.768+0000 7fc2eb5ff700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2e40cb6b0 0x7fc2e415ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.768+0000 7fc2eb5ff700 1 --2- 172.21.15.23:0/1774285541 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc2e40cba80 0x7fc2e4157730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.768+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/1774285541 >> 172.21.15.23:0/1774285541 conn(0x7fc2e4070d80 msgr2=0x7fc2e4150800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.769+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/1774285541 shutdown_connections 2023-06-22T08:08:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:37.769+0000 7fc2eb5ff700 1 -- 172.21.15.23:0/1774285541 wait complete. 2023-06-22T08:08:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 72 2023-06-22T08:08:37.791 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:08:38.184 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:08:38.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:08:39.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:39.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.993+0000 7f2b04f9d700 1 Processor -- start 2023-06-22T08:08:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.993+0000 7f2b04f9d700 1 -- start start 2023-06-22T08:08:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.994+0000 7f2b04f9d700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b001523b0 0x7f2b000b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.994+0000 7f2b04f9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b0006f060 0x7f2b00158b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.994+0000 7f2b04f9d700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b000b2330 con 0x7f2b000c6cf0 2023-06-22T08:08:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.994+0000 7f2b04f9d700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b000b2110 con 0x7f2b001523b0 2023-06-22T08:08:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.994+0000 7f2b04f9d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b00158700 con 0x7f2b0006f060 2023-06-22T08:08:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2afeffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b0006f060 0x7f2b00158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2aff7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2b001523b0 0x7f2b000b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2afffff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2b000c6cf0 0x7f2b000cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:57746/0 (socket says 172.21.15.23:57746) 2023-06-22T08:08:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2afeffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b0006f060 0x7f2b00158b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:46394/0 (socket says 172.21.15.23:46394) 2023-06-22T08:08:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2afffff700 1 -- 172.21.15.23:0/1781854115 learned_addr learned my addr 172.21.15.23:0/1781854115 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2afe7fc700 1 -- 172.21.15.23:0/1781854115 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339788721 0 0) 0x7f2b000b2330 con 0x7f2b000c6cf0 2023-06-22T08:08:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2afe7fc700 1 -- 172.21.15.23:0/1781854115 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae80035b0 con 0x7f2b000c6cf0 2023-06-22T08:08:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2afeffd700 1 -- 172.21.15.23:0/1781854115 >> v2:172.21.15.67:3300/0 conn(0x7f2b001523b0 msgr2=0x7f2b000b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2afeffd700 1 --2- 172.21.15.23:0/1781854115 >> v2:172.21.15.67:3300/0 conn(0x7f2b001523b0 0x7f2b000b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2afeffd700 1 -- 172.21.15.23:0/1781854115 >> v1:172.21.15.67:6789/0 conn(0x7f2b000c6cf0 legacy=0x7f2b000cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2afeffd700 1 -- 172.21.15.23:0/1781854115 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b00152780 con 0x7f2b0006f060 2023-06-22T08:08:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.995+0000 7f2aff7fe700 1 --2- 172.21.15.23:0/1781854115 >> v2:172.21.15.67:3300/0 conn(0x7f2b001523b0 0x7f2b000b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:40.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.996+0000 7f2afeffd700 1 --2- 172.21.15.23:0/1781854115 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b0006f060 0x7f2b00158b90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009840 tx=0x7f2af4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2fff59678ac69ed server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:40.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.996+0000 7f2afe7fc700 1 -- 172.21.15.23:0/1781854115 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4011830 con 0x7f2b0006f060 2023-06-22T08:08:40.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.996+0000 7f2afe7fc700 1 -- 172.21.15.23:0/1781854115 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af4011990 con 0x7f2b0006f060 2023-06-22T08:08:40.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.996+0000 7f2afe7fc700 1 -- 172.21.15.23:0/1781854115 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4011d20 con 0x7f2b0006f060 2023-06-22T08:08:40.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.996+0000 7f2b04f9d700 1 -- 172.21.15.23:0/1781854115 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b0006f060 msgr2=0x7f2b00158b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:40.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.996+0000 7f2b04f9d700 1 --2- 172.21.15.23:0/1781854115 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b0006f060 0x7f2b00158b90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009840 tx=0x7f2af4009b50 comp rx=0 tx=0).stop 2023-06-22T08:08:40.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.996+0000 7f2b04f9d700 1 -- 172.21.15.23:0/1781854115 shutdown_connections 2023-06-22T08:08:40.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.996+0000 7f2b04f9d700 1 --2- 172.21.15.23:0/1781854115 >> v2:172.21.15.67:3300/0 conn(0x7f2b001523b0 0x7f2b000b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:40.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.996+0000 7f2b04f9d700 1 --2- 172.21.15.23:0/1781854115 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b0006f060 0x7f2b00158b90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:40.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.996+0000 7f2b04f9d700 1 -- 172.21.15.23:0/1781854115 >> 172.21.15.23:0/1781854115 conn(0x7f2b00065b90 msgr2=0x7f2b00065f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:40.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.997+0000 7f2b04f9d700 1 -- 172.21.15.23:0/1781854115 shutdown_connections 2023-06-22T08:08:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.997+0000 7f2b04f9d700 1 -- 172.21.15.23:0/1781854115 wait complete. 2023-06-22T08:08:40.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.998+0000 7f2b04f9d700 1 Processor -- start 2023-06-22T08:08:40.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.999+0000 7f2b04f9d700 1 -- start start 2023-06-22T08:08:40.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.999+0000 7f2b04f9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0006f060 0x7f2b001ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:40.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:39.999+0000 7f2b04f9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b000c6cf0 0x7f2b001507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:40.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2b04f9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b001523b0 0x7f2b00153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:40.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2b04f9d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b00153a90 con 0x7f2b001523b0 2023-06-22T08:08:40.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2b04f9d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b00153460 con 0x7f2b0006f060 2023-06-22T08:08:40.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2b04f9d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b000c65f0 con 0x7f2b000c6cf0 2023-06-22T08:08:40.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2afeffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b000c6cf0 0x7f2b001507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:40.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2afffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b001523b0 0x7f2b00153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:40.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2aff7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0006f060 0x7f2b001ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:40.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2afeffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b000c6cf0 0x7f2b001507c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:46410/0 (socket says 172.21.15.23:46410) 2023-06-22T08:08:40.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2afeffd700 1 -- 172.21.15.23:0/2486593105 learned_addr learned my addr 172.21.15.23:0/2486593105 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:40.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2afeffd700 1 -- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0006f060 msgr2=0x7f2b001ee9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:40.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2afeffd700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0006f060 0x7f2b001ee9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:40.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2afeffd700 1 -- 172.21.15.23:0/2486593105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b001523b0 msgr2=0x7f2b00153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:40.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2afeffd700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b001523b0 0x7f2b00153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:40.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2afeffd700 1 -- 172.21.15.23:0/2486593105 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af00094a0 con 0x7f2b000c6cf0 2023-06-22T08:08:40.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2aff7fe700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0006f060 0x7f2b001ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:40.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.000+0000 7f2afffff700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b001523b0 0x7f2b00153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:40.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.001+0000 7f2afeffd700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b000c6cf0 0x7f2b001507c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2af40090b0 tx=0x7f2af4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:40.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.001+0000 7f2ae7fff700 1 -- 172.21.15.23:0/2486593105 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4004590 con 0x7f2b000c6cf0 2023-06-22T08:08:40.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.001+0000 7f2ae7fff700 1 -- 172.21.15.23:0/2486593105 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af401e070 con 0x7f2b000c6cf0 2023-06-22T08:08:40.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.001+0000 7f2ae7fff700 1 -- 172.21.15.23:0/2486593105 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af40038c0 con 0x7f2b000c6cf0 2023-06-22T08:08:40.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.001+0000 7f2b04f9d700 1 -- 172.21.15.23:0/2486593105 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af40094a0 con 0x7f2b000c6cf0 2023-06-22T08:08:40.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.001+0000 7f2b04f9d700 1 -- 172.21.15.23:0/2486593105 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b000c9660 con 0x7f2b000c6cf0 2023-06-22T08:08:40.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.002+0000 7f2b04f9d700 1 -- 172.21.15.23:0/2486593105 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b00153bd0 con 0x7f2b000c6cf0 2023-06-22T08:08:40.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.003+0000 7f2ae7fff700 1 -- 172.21.15.23:0/2486593105 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2af4003a20 con 0x7f2b000c6cf0 2023-06-22T08:08:40.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.004+0000 7f2ae7fff700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2aec06ad10 0x7f2aec06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:40.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.004+0000 7f2ae7fff700 1 -- 172.21.15.23:0/2486593105 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f2af408a6e0 con 0x7f2b000c6cf0 2023-06-22T08:08:40.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.004+0000 7f2aff7fe700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2aec06ad10 0x7f2aec06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:40.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.005+0000 7f2aff7fe700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2aec06ad10 0x7f2aec06d1c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2af0011380 tx=0x7f2af0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:40.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.009+0000 7f2ae7fff700 1 -- 172.21.15.23:0/2486593105 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2b00153bd0 con 0x7f2b000c6cf0 2023-06-22T08:08:40.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.265+0000 7f2b04f9d700 1 -- 172.21.15.23:0/2486593105 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b00153bd0 con 0x7f2b000c6cf0 2023-06-22T08:08:40.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.266+0000 7f2ae7fff700 1 -- 172.21.15.23:0/2486593105 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5507 (secure 0 0 0) 0x7f2b00153bd0 con 0x7f2b000c6cf0 2023-06-22T08:08:40.270 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:40.270 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:34.378619+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:40.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 -- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2aec06ad10 msgr2=0x7f2aec06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:40.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2aec06ad10 0x7f2aec06d1c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2af0011380 tx=0x7f2af0009210 comp rx=0 tx=0).stop 2023-06-22T08:08:40.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 -- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b000c6cf0 msgr2=0x7f2b001507c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:40.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b000c6cf0 0x7f2b001507c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2af40090b0 tx=0x7f2af4004750 comp rx=0 tx=0).stop 2023-06-22T08:08:40.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 -- 172.21.15.23:0/2486593105 shutdown_connections 2023-06-22T08:08:40.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b001523b0 0x7f2b00153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:40.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2aec06ad10 0x7f2aec06d1c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:40.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b0006f060 0x7f2b001ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:40.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 --2- 172.21.15.23:0/2486593105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2b000c6cf0 0x7f2b001507c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:40.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 -- 172.21.15.23:0/2486593105 >> 172.21.15.23:0/2486593105 conn(0x7f2b00065b90 msgr2=0x7f2b0006afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:40.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 -- 172.21.15.23:0/2486593105 shutdown_connections 2023-06-22T08:08:40.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:40.269+0000 7f2b04f9d700 1 -- 172.21.15.23:0/2486593105 wait complete. 2023-06-22T08:08:40.276 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 72 2023-06-22T08:08:40.291 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:08:42.292 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.501+0000 7f1968dff700 1 Processor -- start 2023-06-22T08:08:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.501+0000 7f1968dff700 1 -- start start 2023-06-22T08:08:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1968dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19640c8bd0 0x7f19640aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1968dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f19640cc640 0x7f19640cca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1968dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19640c7400 con 0x7f1964152dd0 2023-06-22T08:08:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1968dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19640c81c0 con 0x7f19640cc640 2023-06-22T08:08:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1968dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19640c84a0 con 0x7f19640c8bd0 2023-06-22T08:08:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1962d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f19640cc640 0x7f19640cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1961d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19640c8bd0 0x7f19640aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1962d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f19640cc640 0x7f19640cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51976/0 (socket says 172.21.15.23:51976) 2023-06-22T08:08:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1962d9d700 1 -- 172.21.15.23:0/3827550038 learned_addr learned my addr 172.21.15.23:0/3827550038 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f196259c700 1 --1- 172.21.15.23:0/3827550038 >> v1:172.21.15.67:6789/0 conn(0x7f1964152dd0 0x7f196406afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:57754/0 (socket says 172.21.15.23:57754) 2023-06-22T08:08:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1962d9d700 1 -- 172.21.15.23:0/3827550038 >> v2:172.21.15.67:3300/0 conn(0x7f19640c8bd0 msgr2=0x7f19640aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.503+0000 7f1962d9d700 1 --2- 172.21.15.23:0/3827550038 >> v2:172.21.15.67:3300/0 conn(0x7f19640c8bd0 0x7f19640aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.504+0000 7f1962d9d700 1 -- 172.21.15.23:0/3827550038 >> v1:172.21.15.67:6789/0 conn(0x7f1964152dd0 legacy=0x7f196406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.504+0000 7f1962d9d700 1 -- 172.21.15.23:0/3827550038 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19640cd370 con 0x7f19640cc640 2023-06-22T08:08:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.504+0000 7f1962d9d700 1 --2- 172.21.15.23:0/3827550038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640cc640 0x7f19640cca80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1964071cb0 tx=0x7f195c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e03e385aa99416 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.504+0000 7f196159a700 1 -- 172.21.15.23:0/3827550038 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c00e070 con 0x7f19640cc640 2023-06-22T08:08:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.504+0000 7f196159a700 1 -- 172.21.15.23:0/3827550038 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195c00b910 con 0x7f19640cc640 2023-06-22T08:08:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.504+0000 7f196159a700 1 -- 172.21.15.23:0/3827550038 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c00bcc0 con 0x7f19640cc640 2023-06-22T08:08:42.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.504+0000 7f1968dff700 1 -- 172.21.15.23:0/3827550038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640cc640 msgr2=0x7f19640cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:42.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.505+0000 7f1968dff700 1 --2- 172.21.15.23:0/3827550038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640cc640 0x7f19640cca80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1964071cb0 tx=0x7f195c00b330 comp rx=0 tx=0).stop 2023-06-22T08:08:42.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.505+0000 7f1968dff700 1 -- 172.21.15.23:0/3827550038 shutdown_connections 2023-06-22T08:08:42.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.505+0000 7f1968dff700 1 --2- 172.21.15.23:0/3827550038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640cc640 0x7f19640cca80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:42.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.505+0000 7f1968dff700 1 --2- 172.21.15.23:0/3827550038 >> v2:172.21.15.67:3300/0 conn(0x7f19640c8bd0 0x7f19640aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:42.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.505+0000 7f1968dff700 1 -- 172.21.15.23:0/3827550038 >> 172.21.15.23:0/3827550038 conn(0x7f196406fb70 msgr2=0x7f196406ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.505+0000 7f1968dff700 1 -- 172.21.15.23:0/3827550038 shutdown_connections 2023-06-22T08:08:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.505+0000 7f1968dff700 1 -- 172.21.15.23:0/3827550038 wait complete. 2023-06-22T08:08:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.506+0000 7f1968dff700 1 Processor -- start 2023-06-22T08:08:42.513 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.507+0000 7f1968dff700 1 -- start start 2023-06-22T08:08:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.507+0000 7f1968dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640c8bd0 0x7f1964150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f1968dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19640cc640 0x7f19640b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f1968dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1964152dd0 0x7f19640b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f1968dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19640c4b80 con 0x7f19640c8bd0 2023-06-22T08:08:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f1968dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19641f1720 con 0x7f1964152dd0 2023-06-22T08:08:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f1968dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19640af980 con 0x7f19640cc640 2023-06-22T08:08:42.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f196259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640c8bd0 0x7f1964150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:42.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f196259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640c8bd0 0x7f1964150eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51982/0 (socket says 172.21.15.23:51982) 2023-06-22T08:08:42.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f196259c700 1 -- 172.21.15.23:0/3036081694 learned_addr learned my addr 172.21.15.23:0/3036081694 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:42.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f1962d9d700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1964152dd0 0x7f19640b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:42.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f1961d9b700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19640cc640 0x7f19640b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:42.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f196259c700 1 -- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19640cc640 msgr2=0x7f19640b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:42.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f196259c700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19640cc640 0x7f19640b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:42.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f196259c700 1 -- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1964152dd0 msgr2=0x7f19640b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:42.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f196259c700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1964152dd0 0x7f19640b0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:42.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f196259c700 1 -- 172.21.15.23:0/3036081694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c009b30 con 0x7f19640c8bd0 2023-06-22T08:08:42.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.508+0000 7f196259c700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640c8bd0 0x7f1964150eb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f195400c460 tx=0x7f195400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:42.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.509+0000 7f1952ffd700 1 -- 172.21.15.23:0/3036081694 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954005d00 con 0x7f19640c8bd0 2023-06-22T08:08:42.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.509+0000 7f1952ffd700 1 -- 172.21.15.23:0/3036081694 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1954005e60 con 0x7f19640c8bd0 2023-06-22T08:08:42.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.509+0000 7f1968dff700 1 -- 172.21.15.23:0/3036081694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19641f1440 con 0x7f19640c8bd0 2023-06-22T08:08:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.509+0000 7f1952ffd700 1 -- 172.21.15.23:0/3036081694 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954007310 con 0x7f19640c8bd0 2023-06-22T08:08:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.509+0000 7f1968dff700 1 -- 172.21.15.23:0/3036081694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19640b0030 con 0x7f19640c8bd0 2023-06-22T08:08:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.510+0000 7f1968dff700 1 -- 172.21.15.23:0/3036081694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196404ea20 con 0x7f19640c8bd0 2023-06-22T08:08:42.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.511+0000 7f1952ffd700 1 -- 172.21.15.23:0/3036081694 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1954007470 con 0x7f19640c8bd0 2023-06-22T08:08:42.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.512+0000 7f1952ffd700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f194406ace0 0x7f194406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:42.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.513+0000 7f1952ffd700 1 -- 172.21.15.23:0/3036081694 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f1954088d90 con 0x7f19640c8bd0 2023-06-22T08:08:42.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.513+0000 7f1961d9b700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f194406ace0 0x7f194406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:42.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.513+0000 7f1961d9b700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f194406ace0 0x7f194406d190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f19640c6f50 tx=0x7f1958008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:42.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.517+0000 7f1952ffd700 1 -- 172.21.15.23:0/3036081694 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1954053990 con 0x7f19640c8bd0 2023-06-22T08:08:42.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.774+0000 7f1968dff700 1 -- 172.21.15.23:0/3036081694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19640c35f0 con 0x7f19640c8bd0 2023-06-22T08:08:42.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.774+0000 7f1952ffd700 1 -- 172.21.15.23:0/3036081694 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5509 (secure 0 0 0) 0x7f1954017090 con 0x7f19640c8bd0 2023-06-22T08:08:42.778 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:42.779 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:40.833204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:resolve","state_seq":13,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:42.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 -- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f194406ace0 msgr2=0x7f194406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:42.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f194406ace0 0x7f194406d190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f19640c6f50 tx=0x7f1958008040 comp rx=0 tx=0).stop 2023-06-22T08:08:42.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 -- 172.21.15.23:0/3036081694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640c8bd0 msgr2=0x7f1964150eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:42.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640c8bd0 0x7f1964150eb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f195400c460 tx=0x7f195400c820 comp rx=0 tx=0).stop 2023-06-22T08:08:42.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 -- 172.21.15.23:0/3036081694 shutdown_connections 2023-06-22T08:08:42.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f194406ace0 0x7f194406d190 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:42.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1964152dd0 0x7f19640b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:42.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19640c8bd0 0x7f1964150eb0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:42.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 --2- 172.21.15.23:0/3036081694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f19640cc640 0x7f19640b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:42.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 -- 172.21.15.23:0/3036081694 >> 172.21.15.23:0/3036081694 conn(0x7f196406fb70 msgr2=0x7f196406ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:42.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 -- 172.21.15.23:0/3036081694 shutdown_connections 2023-06-22T08:08:42.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:42.777+0000 7f1968dff700 1 -- 172.21.15.23:0/3036081694 wait complete. 2023-06-22T08:08:42.783 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 73 2023-06-22T08:08:42.799 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:08:44.800 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.015+0000 7ff5addff700 1 Processor -- start 2023-06-22T08:08:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.015+0000 7ff5addff700 1 -- start start 2023-06-22T08:08:45.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.016+0000 7ff5addff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5a80cf1b0 0x7ff5a80c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.016+0000 7ff5addff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a815c370 con 0x7ff5a8152ce0 2023-06-22T08:08:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.016+0000 7ff5addff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a815bda0 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.016+0000 7ff5addff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a81530b0 con 0x7ff5a80cf1b0 2023-06-22T08:08:45.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.016+0000 7ff5a77fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff5a8152ce0 0x7ff5a815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55320/0 (socket says 172.21.15.23:55320) 2023-06-22T08:08:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.016+0000 7ff5a77fe700 1 -- 172.21.15.23:0/3365228794 learned_addr learned my addr 172.21.15.23:0/3365228794 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:45.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.017+0000 7ff5a7fff700 1 --2- 172.21.15.23:0/3365228794 >> v2:172.21.15.67:3300/0 conn(0x7ff5a80cf1b0 0x7ff5a80c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:45.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.017+0000 7ff5a67fc700 1 -- 172.21.15.23:0/3365228794 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844906765 0 0) 0x7ff5a815c370 con 0x7ff5a8152ce0 2023-06-22T08:08:45.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.017+0000 7ff5a67fc700 1 -- 172.21.15.23:0/3365228794 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5900035b0 con 0x7ff5a8152ce0 2023-06-22T08:08:45.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.017+0000 7ff5a67fc700 1 -- 172.21.15.23:0/3365228794 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230405447 0 0) 0x7ff5a815bda0 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.017+0000 7ff5a67fc700 1 -- 172.21.15.23:0/3365228794 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a815c370 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.017+0000 7ff5a7fff700 1 -- 172.21.15.23:0/3365228794 >> v1:172.21.15.67:6789/0 conn(0x7ff5a80c9fd0 legacy=0x7ff5a80c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:45.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.017+0000 7ff5a7fff700 1 -- 172.21.15.23:0/3365228794 >> v1:172.21.15.23:6789/0 conn(0x7ff5a8152ce0 legacy=0x7ff5a815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:45.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.017+0000 7ff5a7fff700 1 -- 172.21.15.23:0/3365228794 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a80cf630 con 0x7ff5a80cf1b0 2023-06-22T08:08:45.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.017+0000 7ff5a7fff700 1 --2- 172.21.15.23:0/3365228794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a80cf1b0 0x7ff5a80c47c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009d90 tx=0x7ff5a000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=451a1923d469bc3d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:45.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.018+0000 7ff5a67fc700 1 -- 172.21.15.23:0/3365228794 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a000e070 con 0x7ff5a80cf1b0 2023-06-22T08:08:45.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.018+0000 7ff5a67fc700 1 -- 172.21.15.23:0/3365228794 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a000bb20 con 0x7ff5a80cf1b0 2023-06-22T08:08:45.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.018+0000 7ff5a67fc700 1 -- 172.21.15.23:0/3365228794 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0012430 con 0x7ff5a80cf1b0 2023-06-22T08:08:45.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.018+0000 7ff5addff700 1 -- 172.21.15.23:0/3365228794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a80cf1b0 msgr2=0x7ff5a80c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:45.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.018+0000 7ff5addff700 1 --2- 172.21.15.23:0/3365228794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a80cf1b0 0x7ff5a80c47c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009d90 tx=0x7ff5a000b330 comp rx=0 tx=0).stop 2023-06-22T08:08:45.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.018+0000 7ff5addff700 1 -- 172.21.15.23:0/3365228794 shutdown_connections 2023-06-22T08:08:45.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.018+0000 7ff5addff700 1 --2- 172.21.15.23:0/3365228794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a80cf1b0 0x7ff5a80c47c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:45.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.018+0000 7ff5addff700 1 -- 172.21.15.23:0/3365228794 >> 172.21.15.23:0/3365228794 conn(0x7ff5a806ae30 msgr2=0x7ff5a806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:45.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.018+0000 7ff5addff700 1 -- 172.21.15.23:0/3365228794 shutdown_connections 2023-06-22T08:08:45.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.018+0000 7ff5addff700 1 -- 172.21.15.23:0/3365228794 wait complete. 2023-06-22T08:08:45.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.020+0000 7ff5addff700 1 Processor -- start 2023-06-22T08:08:45.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.020+0000 7ff5addff700 1 -- start start 2023-06-22T08:08:45.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5addff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5a80c9fd0 0x7ff5a814c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:45.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5addff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff5a80cf1b0 0x7ff5a81e3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:45.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5addff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8152ce0 0x7ff5a81ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:45.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5addff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a814cad0 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5addff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a81f2330 con 0x7ff5a8152ce0 2023-06-22T08:08:45.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5addff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a81f1720 con 0x7ff5a80cf1b0 2023-06-22T08:08:45.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5a77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5a80c9fd0 0x7ff5a814c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:45.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5a77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5a80c9fd0 0x7ff5a814c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51998/0 (socket says 172.21.15.23:51998) 2023-06-22T08:08:45.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5a77fe700 1 -- 172.21.15.23:0/1897424509 learned_addr learned my addr 172.21.15.23:0/1897424509 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:45.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5a6ffd700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff5a80cf1b0 0x7ff5a81e3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:45.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.021+0000 7ff5a7fff700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8152ce0 0x7ff5a81ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:45.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.022+0000 7ff5a77fe700 1 -- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff5a80cf1b0 msgr2=0x7ff5a81e3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:45.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.022+0000 7ff5a77fe700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff5a80cf1b0 0x7ff5a81e3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:45.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.022+0000 7ff5a77fe700 1 -- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8152ce0 msgr2=0x7ff5a81ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:45.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.022+0000 7ff5a77fe700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8152ce0 0x7ff5a81ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:45.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.022+0000 7ff5a77fe700 1 -- 172.21.15.23:0/1897424509 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a0009a70 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.022+0000 7ff5a77fe700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5a80c9fd0 0x7ff5a814c290 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff598001eb0 tx=0x7ff59800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:45.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.022+0000 7ff58ffff700 1 -- 172.21.15.23:0/1897424509 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598005d70 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.022+0000 7ff58ffff700 1 -- 172.21.15.23:0/1897424509 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5980070d0 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.022+0000 7ff58ffff700 1 -- 172.21.15.23:0/1897424509 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598007440 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.022+0000 7ff5addff700 1 -- 172.21.15.23:0/1897424509 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a81f1a00 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.023+0000 7ff5addff700 1 -- 172.21.15.23:0/1897424509 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a81f1fc0 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.023+0000 7ff5addff700 1 -- 172.21.15.23:0/1897424509 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a81ee080 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.025+0000 7ff58ffff700 1 -- 172.21.15.23:0/1897424509 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff5980075a0 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.025+0000 7ff58ffff700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff59406ace0 0x7ff59406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:45.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.025+0000 7ff58ffff700 1 -- 172.21.15.23:0/1897424509 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7ff598088c20 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.026+0000 7ff5a6ffd700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff59406ace0 0x7ff59406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.026+0000 7ff5a6ffd700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff59406ace0 0x7ff59406d190 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff5a80c8e40 tx=0x7ff59c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.030+0000 7ff58ffff700 1 -- 172.21.15.23:0/1897424509 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff598053850 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.284+0000 7ff5addff700 1 -- 172.21.15.23:0/1897424509 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5a804ea20 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.284+0000 7ff58ffff700 1 -- 172.21.15.23:0/1897424509 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5509 (secure 0 0 0) 0x7ff598017090 con 0x7ff5a80c9fd0 2023-06-22T08:08:45.288 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:45.288 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:40.833204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:resolve","state_seq":13,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:45.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 -- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff59406ace0 msgr2=0x7ff59406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:45.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff59406ace0 0x7ff59406d190 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff5a80c8e40 tx=0x7ff59c009040 comp rx=0 tx=0).stop 2023-06-22T08:08:45.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 -- 172.21.15.23:0/1897424509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5a80c9fd0 msgr2=0x7ff5a814c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:45.291 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5a80c9fd0 0x7ff5a814c290 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff598001eb0 tx=0x7ff59800c7b0 comp rx=0 tx=0).stop 2023-06-22T08:08:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 -- 172.21.15.23:0/1897424509 shutdown_connections 2023-06-22T08:08:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff59406ace0 0x7ff59406d190 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8152ce0 0x7ff5a81ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5a80c9fd0 0x7ff5a814c290 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 --2- 172.21.15.23:0/1897424509 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff5a80cf1b0 0x7ff5a81e3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 -- 172.21.15.23:0/1897424509 >> 172.21.15.23:0/1897424509 conn(0x7ff5a806ae30 msgr2=0x7ff5a80b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 -- 172.21.15.23:0/1897424509 shutdown_connections 2023-06-22T08:08:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:45.287+0000 7ff5addff700 1 -- 172.21.15.23:0/1897424509 wait complete. 2023-06-22T08:08:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 73 2023-06-22T08:08:45.308 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:08:47.310 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:47.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.517+0000 7f4c38bff700 1 Processor -- start 2023-06-22T08:08:47.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.517+0000 7f4c38bff700 1 -- start start 2023-06-22T08:08:47.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.518+0000 7f4c38bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4c34071070 0x7f4c34160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:47.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.518+0000 7f4c38bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c340c51a0 con 0x7f4c3414f850 2023-06-22T08:08:47.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.518+0000 7f4c38bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c340c44e0 con 0x7f4c340c69e0 2023-06-22T08:08:47.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.518+0000 7f4c38bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c340c5440 con 0x7f4c34071070 2023-06-22T08:08:47.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.519+0000 7f4c32d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4c34071070 0x7f4c34160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:47.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.519+0000 7f4c3259c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4c3414f850 0x7f4c340c3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58700/0 (socket says 172.21.15.23:58700) 2023-06-22T08:08:47.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.519+0000 7f4c3259c700 1 -- 172.21.15.23:0/160959860 learned_addr learned my addr 172.21.15.23:0/160959860 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:47.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.519+0000 7f4c32d9d700 1 -- 172.21.15.23:0/160959860 >> v1:172.21.15.67:6789/0 conn(0x7f4c340c69e0 legacy=0x7f4c34155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:47.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.519+0000 7f4c32d9d700 1 -- 172.21.15.23:0/160959860 >> v1:172.21.15.23:6789/0 conn(0x7f4c3414f850 legacy=0x7f4c340c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:47.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.519+0000 7f4c3159a700 1 -- 172.21.15.23:0/160959860 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517343167 0 0) 0x7f4c340c44e0 con 0x7f4c340c69e0 2023-06-22T08:08:47.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.519+0000 7f4c32d9d700 1 -- 172.21.15.23:0/160959860 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c340c5720 con 0x7f4c34071070 2023-06-22T08:08:47.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.519+0000 7f4c32d9d700 1 --2- 172.21.15.23:0/160959860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 0x7f4c34160eb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009d90 tx=0x7f4c2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd867c81ba99a53d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:47.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.520+0000 7f4c3159a700 1 -- 172.21.15.23:0/160959860 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00e070 con 0x7f4c34071070 2023-06-22T08:08:47.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.520+0000 7f4c3159a700 1 -- 172.21.15.23:0/160959860 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2c00bad0 con 0x7f4c34071070 2023-06-22T08:08:47.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.520+0000 7f4c3159a700 1 -- 172.21.15.23:0/160959860 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00be80 con 0x7f4c34071070 2023-06-22T08:08:47.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.520+0000 7f4c38bff700 1 -- 172.21.15.23:0/160959860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 msgr2=0x7f4c34160eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:47.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.520+0000 7f4c38bff700 1 --2- 172.21.15.23:0/160959860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 0x7f4c34160eb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009d90 tx=0x7f4c2c00b330 comp rx=0 tx=0).stop 2023-06-22T08:08:47.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.520+0000 7f4c38bff700 1 -- 172.21.15.23:0/160959860 shutdown_connections 2023-06-22T08:08:47.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.520+0000 7f4c38bff700 1 --2- 172.21.15.23:0/160959860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 0x7f4c34160eb0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:47.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.520+0000 7f4c38bff700 1 -- 172.21.15.23:0/160959860 >> 172.21.15.23:0/160959860 conn(0x7f4c340ca3f0 msgr2=0x7f4c340ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:47.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.520+0000 7f4c38bff700 1 -- 172.21.15.23:0/160959860 shutdown_connections 2023-06-22T08:08:47.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.520+0000 7f4c38bff700 1 -- 172.21.15.23:0/160959860 wait complete. 2023-06-22T08:08:47.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.522+0000 7f4c38bff700 1 Processor -- start 2023-06-22T08:08:47.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.522+0000 7f4c38bff700 1 -- start start 2023-06-22T08:08:47.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c38bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 0x7f4c341e2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:47.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c38bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340c69e0 0x7f4c34150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:47.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c38bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c3414f850 0x7f4c341510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:47.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c38bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341f1a40 con 0x7f4c34071070 2023-06-22T08:08:47.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c38bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341f1d00 con 0x7f4c340c69e0 2023-06-22T08:08:47.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c38bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341f1fe0 con 0x7f4c3414f850 2023-06-22T08:08:47.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c3259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 0x7f4c341e2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:47.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c3259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 0x7f4c341e2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53590/0 (socket says 172.21.15.23:53590) 2023-06-22T08:08:47.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c31d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340c69e0 0x7f4c34150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:47.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c32d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c3414f850 0x7f4c341510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:47.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.523+0000 7f4c3259c700 1 -- 172.21.15.23:0/2261081789 learned_addr learned my addr 172.21.15.23:0/2261081789 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:47.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.524+0000 7f4c3259c700 1 -- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c3414f850 msgr2=0x7f4c341510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:47.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.524+0000 7f4c3259c700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c3414f850 0x7f4c341510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:47.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.524+0000 7f4c3259c700 1 -- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340c69e0 msgr2=0x7f4c34150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:47.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.524+0000 7f4c3259c700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340c69e0 0x7f4c34150b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:47.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.524+0000 7f4c3259c700 1 -- 172.21.15.23:0/2261081789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c009a70 con 0x7f4c34071070 2023-06-22T08:08:47.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.524+0000 7f4c3259c700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 0x7f4c341e2ee0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400c780 tx=0x7f4c2400ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:47.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.524+0000 7f4c22ffd700 1 -- 172.21.15.23:0/2261081789 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24004c80 con 0x7f4c34071070 2023-06-22T08:08:47.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.524+0000 7f4c22ffd700 1 -- 172.21.15.23:0/2261081789 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c24004de0 con 0x7f4c34071070 2023-06-22T08:08:47.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.524+0000 7f4c22ffd700 1 -- 172.21.15.23:0/2261081789 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c240072f0 con 0x7f4c34071070 2023-06-22T08:08:47.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.524+0000 7f4c38bff700 1 -- 172.21.15.23:0/2261081789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c341f22c0 con 0x7f4c34071070 2023-06-22T08:08:47.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.525+0000 7f4c38bff700 1 -- 172.21.15.23:0/2261081789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c341526e0 con 0x7f4c34071070 2023-06-22T08:08:47.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.526+0000 7f4c38bff700 1 -- 172.21.15.23:0/2261081789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3404ea20 con 0x7f4c34071070 2023-06-22T08:08:47.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.527+0000 7f4c22ffd700 1 -- 172.21.15.23:0/2261081789 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4c24007450 con 0x7f4c34071070 2023-06-22T08:08:47.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.527+0000 7f4c22ffd700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c1406ace0 0x7f4c1406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:47.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.528+0000 7f4c22ffd700 1 -- 172.21.15.23:0/2261081789 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f4c24089650 con 0x7f4c34071070 2023-06-22T08:08:47.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.528+0000 7f4c31d9b700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c1406ace0 0x7f4c1406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:47.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.528+0000 7f4c31d9b700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c1406ace0 0x7f4c1406d190 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4c341f29e0 tx=0x7f4c28009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:47.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.532+0000 7f4c22ffd700 1 -- 172.21.15.23:0/2261081789 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4c24054260 con 0x7f4c34071070 2023-06-22T08:08:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.783+0000 7f4c38bff700 1 -- 172.21.15.23:0/2261081789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c3406ae60 con 0x7f4c34071070 2023-06-22T08:08:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.784+0000 7f4c22ffd700 1 -- 172.21.15.23:0/2261081789 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5511 (secure 0 0 0) 0x7f4c24057880 con 0x7f4c34071070 2023-06-22T08:08:47.788 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:47.789 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:46.391996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:47.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 -- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c1406ace0 msgr2=0x7f4c1406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:47.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c1406ace0 0x7f4c1406d190 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4c341f29e0 tx=0x7f4c28009040 comp rx=0 tx=0).stop 2023-06-22T08:08:47.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 -- 172.21.15.23:0/2261081789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 msgr2=0x7f4c341e2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:47.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 0x7f4c341e2ee0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400c780 tx=0x7f4c2400ca90 comp rx=0 tx=0).stop 2023-06-22T08:08:47.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 -- 172.21.15.23:0/2261081789 shutdown_connections 2023-06-22T08:08:47.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c3414f850 0x7f4c341510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:47.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c34071070 0x7f4c341e2ee0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:47.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c1406ace0 0x7f4c1406d190 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:47.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 --2- 172.21.15.23:0/2261081789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c340c69e0 0x7f4c34150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:47.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 -- 172.21.15.23:0/2261081789 >> 172.21.15.23:0/2261081789 conn(0x7f4c340ca3f0 msgr2=0x7f4c34069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:47.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 -- 172.21.15.23:0/2261081789 shutdown_connections 2023-06-22T08:08:47.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:47.786+0000 7f4c38bff700 1 -- 172.21.15.23:0/2261081789 wait complete. 2023-06-22T08:08:47.792 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2023-06-22T08:08:47.809 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 74, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 74, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4285": { "addr": "172.21.15.67:6839/567663545", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 567663545, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 567663545, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5060": { "addr": "172.21.15.67:6837/1551460358", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1551460358, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1551460358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5060, "incarnation": 46, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 31 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 26 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:reconnect", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:08:46.391996+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4285, "mds_1": 5707, "mds_2": 5551, "mds_3": 5060, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:08:49.811 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:50.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.018+0000 7f28037ff700 1 Processor -- start 2023-06-22T08:08:50.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.018+0000 7f28037ff700 1 -- start start 2023-06-22T08:08:50.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.019+0000 7f28037ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f27fc0c4d00 0x7f27fc159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:50.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.019+0000 7f28037ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc0c9c30 con 0x7f27fc152d70 2023-06-22T08:08:50.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.019+0000 7f28037ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc15bea0 con 0x7f27fc0c97d0 2023-06-22T08:08:50.024 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.019+0000 7f28037ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27fc0c95e0 con 0x7f27fc0c4d00 2023-06-22T08:08:50.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.019+0000 7f2801d9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f27fc0c4d00 0x7f27fc159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:50.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.019+0000 7f280159b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f27fc152d70 0x7f27fc0cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58714/0 (socket says 172.21.15.23:58714) 2023-06-22T08:08:50.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.020+0000 7f280159b700 1 -- 172.21.15.23:0/1458772216 learned_addr learned my addr 172.21.15.23:0/1458772216 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:50.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.020+0000 7f2801d9c700 1 -- 172.21.15.23:0/1458772216 >> v1:172.21.15.67:6789/0 conn(0x7f27fc0c97d0 legacy=0x7f27fc0c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:50.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.020+0000 7f2801d9c700 1 -- 172.21.15.23:0/1458772216 >> v1:172.21.15.23:6789/0 conn(0x7f27fc152d70 legacy=0x7f27fc0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:50.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.020+0000 7f27f3fff700 1 -- 172.21.15.23:0/1458772216 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348488962 0 0) 0x7f27fc15bea0 con 0x7f27fc0c97d0 2023-06-22T08:08:50.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.020+0000 7f2801d9c700 1 -- 172.21.15.23:0/1458772216 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc15c160 con 0x7f27fc0c4d00 2023-06-22T08:08:50.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.020+0000 7f2801d9c700 1 --2- 172.21.15.23:0/1458772216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc0c4d00 0x7f27fc159330 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f27f4009d90 tx=0x7f27f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1474d414324c2210 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:50.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.021+0000 7f27f3fff700 1 -- 172.21.15.23:0/1458772216 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f400e070 con 0x7f27fc0c4d00 2023-06-22T08:08:50.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.021+0000 7f27f3fff700 1 -- 172.21.15.23:0/1458772216 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f400bad0 con 0x7f27fc0c4d00 2023-06-22T08:08:50.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.021+0000 7f27f3fff700 1 -- 172.21.15.23:0/1458772216 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f400be80 con 0x7f27fc0c4d00 2023-06-22T08:08:50.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.021+0000 7f28037ff700 1 -- 172.21.15.23:0/1458772216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc0c4d00 msgr2=0x7f27fc159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:50.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.021+0000 7f28037ff700 1 --2- 172.21.15.23:0/1458772216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc0c4d00 0x7f27fc159330 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f27f4009d90 tx=0x7f27f400b330 comp rx=0 tx=0).stop 2023-06-22T08:08:50.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.021+0000 7f28037ff700 1 -- 172.21.15.23:0/1458772216 shutdown_connections 2023-06-22T08:08:50.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.021+0000 7f28037ff700 1 --2- 172.21.15.23:0/1458772216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc0c4d00 0x7f27fc159330 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:50.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.021+0000 7f28037ff700 1 -- 172.21.15.23:0/1458772216 >> 172.21.15.23:0/1458772216 conn(0x7f27fc06ae30 msgr2=0x7f27fc06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:50.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.021+0000 7f28037ff700 1 -- 172.21.15.23:0/1458772216 shutdown_connections 2023-06-22T08:08:50.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.021+0000 7f28037ff700 1 -- 172.21.15.23:0/1458772216 wait complete. 2023-06-22T08:08:50.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.023+0000 7f28037ff700 1 Processor -- start 2023-06-22T08:08:50.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.023+0000 7f28037ff700 1 -- start start 2023-06-22T08:08:50.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f28037ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f27fc0c4d00 0x7f27fc154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:50.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f28037ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27fc0c97d0 0x7f27fc14e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f28037ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc152d70 0x7f27fc0b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f28037ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc1f2830 con 0x7f27fc152d70 2023-06-22T08:08:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f28037ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc1ee580 con 0x7f27fc0c97d0 2023-06-22T08:08:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f28037ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc1ee860 con 0x7f27fc0c4d00 2023-06-22T08:08:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f2801d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc152d70 0x7f27fc0b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f2801d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc152d70 0x7f27fc0b3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53616/0 (socket says 172.21.15.23:53616) 2023-06-22T08:08:50.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f2801d9c700 1 -- 172.21.15.23:0/27974494 learned_addr learned my addr 172.21.15.23:0/27974494 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f280159b700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f27fc0c4d00 0x7f27fc154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.024+0000 7f2800d9a700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27fc0c97d0 0x7f27fc14e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f2801d9c700 1 -- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f27fc0c4d00 msgr2=0x7f27fc154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f2801d9c700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f27fc0c4d00 0x7f27fc154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f2801d9c700 1 -- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27fc0c97d0 msgr2=0x7f27fc14e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f2801d9c700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27fc0c97d0 0x7f27fc14e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f2801d9c700 1 -- 172.21.15.23:0/27974494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f4009a70 con 0x7f27fc152d70 2023-06-22T08:08:50.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f2801d9c700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc152d70 0x7f27fc0b3950 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f27fc08bcd0 tx=0x7f27f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:50.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f27f1ffb700 1 -- 172.21.15.23:0/27974494 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f4012630 con 0x7f27fc152d70 2023-06-22T08:08:50.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f27f1ffb700 1 -- 172.21.15.23:0/27974494 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f4012790 con 0x7f27fc152d70 2023-06-22T08:08:50.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f27f1ffb700 1 -- 172.21.15.23:0/27974494 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f4012b40 con 0x7f27fc152d70 2023-06-22T08:08:50.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f28037ff700 1 -- 172.21.15.23:0/27974494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc1eeb40 con 0x7f27fc152d70 2023-06-22T08:08:50.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.025+0000 7f28037ff700 1 -- 172.21.15.23:0/27974494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc1ef100 con 0x7f27fc152d70 2023-06-22T08:08:50.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.026+0000 7f28037ff700 1 -- 172.21.15.23:0/27974494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc1ef2f0 con 0x7f27fc152d70 2023-06-22T08:08:50.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.028+0000 7f27f1ffb700 1 -- 172.21.15.23:0/27974494 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f27f4012ca0 con 0x7f27fc152d70 2023-06-22T08:08:50.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.028+0000 7f27f1ffb700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f27e006ad10 0x7f27e006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:50.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.029+0000 7f27f1ffb700 1 -- 172.21.15.23:0/27974494 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f27f408ba60 con 0x7f27fc152d70 2023-06-22T08:08:50.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.029+0000 7f280159b700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f27e006ad10 0x7f27e006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:50.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.029+0000 7f280159b700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f27e006ad10 0x7f27e006d1c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f27f8000f60 tx=0x7f27f8004a50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:50.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.033+0000 7f27f1ffb700 1 -- 172.21.15.23:0/27974494 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f27f4056700 con 0x7f27fc152d70 2023-06-22T08:08:50.289 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.284+0000 7f28037ff700 1 -- 172.21.15.23:0/27974494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27fc04ea20 con 0x7f27fc152d70 2023-06-22T08:08:50.290 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.285+0000 7f27f1ffb700 1 -- 172.21.15.23:0/27974494 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5511 (secure 0 0 0) 0x7f27f4059d20 con 0x7f27fc152d70 2023-06-22T08:08:50.290 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:50.290 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:46.391996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:50.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 -- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f27e006ad10 msgr2=0x7f27e006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:50.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f27e006ad10 0x7f27e006d1c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f27f8000f60 tx=0x7f27f8004a50 comp rx=0 tx=0).stop 2023-06-22T08:08:50.292 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 -- 172.21.15.23:0/27974494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc152d70 msgr2=0x7f27fc0b3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc152d70 0x7f27fc0b3950 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f27fc08bcd0 tx=0x7f27f4004ad0 comp rx=0 tx=0).stop 2023-06-22T08:08:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 -- 172.21.15.23:0/27974494 shutdown_connections 2023-06-22T08:08:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27fc152d70 0x7f27fc0b3950 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f27fc0c4d00 0x7f27fc154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f27e006ad10 0x7f27e006d1c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 --2- 172.21.15.23:0/27974494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27fc0c97d0 0x7f27fc14e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 -- 172.21.15.23:0/27974494 >> 172.21.15.23:0/27974494 conn(0x7f27fc06ae30 msgr2=0x7f27fc06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:50.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.287+0000 7f28037ff700 1 -- 172.21.15.23:0/27974494 shutdown_connections 2023-06-22T08:08:50.294 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:50.288+0000 7f28037ff700 1 -- 172.21.15.23:0/27974494 wait complete. 2023-06-22T08:08:50.294 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 74 2023-06-22T08:08:50.311 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:08:52.312 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:52.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.519+0000 7f09b4dff700 1 Processor -- start 2023-06-22T08:08:52.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.519+0000 7f09b4dff700 1 -- start start 2023-06-22T08:08:52.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.520+0000 7f09b4dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00c4a30 0x7f09b00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:52.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.520+0000 7f09b4dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09b00cb7a0 0x7f09b015c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:52.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.520+0000 7f09b4dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b0159730 con 0x7f09b0152a10 2023-06-22T08:08:52.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.520+0000 7f09b4dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b00c5470 con 0x7f09b00cb7a0 2023-06-22T08:08:52.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.520+0000 7f09b4dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0158910 con 0x7f09b00c4a30 2023-06-22T08:08:52.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09aed9d700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09b00cb7a0 0x7f09b015c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:52.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09ae59c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f09b0152a10 0x7f09b00c9200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58806/0 (socket says 172.21.15.23:58806) 2023-06-22T08:08:52.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09ae59c700 1 -- 172.21.15.23:0/1025706767 learned_addr learned my addr 172.21.15.23:0/1025706767 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:52.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09a7fff700 1 --2- 172.21.15.23:0/1025706767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00c4a30 0x7f09b00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:52.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09add9b700 1 -- 172.21.15.23:0/1025706767 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317719167 0 0) 0x7f09b0159730 con 0x7f09b0152a10 2023-06-22T08:08:52.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09add9b700 1 -- 172.21.15.23:0/1025706767 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f099c0035b0 con 0x7f09b0152a10 2023-06-22T08:08:52.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09a7fff700 1 -- 172.21.15.23:0/1025706767 >> v2:172.21.15.67:3300/0 conn(0x7f09b00cb7a0 msgr2=0x7f09b015c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:52.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09a7fff700 1 --2- 172.21.15.23:0/1025706767 >> v2:172.21.15.67:3300/0 conn(0x7f09b00cb7a0 0x7f09b015c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:52.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09a7fff700 1 -- 172.21.15.23:0/1025706767 >> v1:172.21.15.67:6789/0 conn(0x7f09b0152a10 legacy=0x7f09b00c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:52.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09a7fff700 1 -- 172.21.15.23:0/1025706767 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b0158bb0 con 0x7f09b00c4a30 2023-06-22T08:08:52.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.521+0000 7f09aed9d700 1 --2- 172.21.15.23:0/1025706767 >> v2:172.21.15.67:3300/0 conn(0x7f09b00cb7a0 0x7f09b015c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:52.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.522+0000 7f09a7fff700 1 --2- 172.21.15.23:0/1025706767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00c4a30 0x7f09b00cb260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0998009840 tx=0x7f0998009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc7ea01fe88e0b80 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:52.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.522+0000 7f09add9b700 1 -- 172.21.15.23:0/1025706767 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0998011830 con 0x7f09b00c4a30 2023-06-22T08:08:52.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.522+0000 7f09add9b700 1 -- 172.21.15.23:0/1025706767 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0998011990 con 0x7f09b00c4a30 2023-06-22T08:08:52.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.522+0000 7f09add9b700 1 -- 172.21.15.23:0/1025706767 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0998011d20 con 0x7f09b00c4a30 2023-06-22T08:08:52.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.522+0000 7f09b4dff700 1 -- 172.21.15.23:0/1025706767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00c4a30 msgr2=0x7f09b00cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:52.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.522+0000 7f09b4dff700 1 --2- 172.21.15.23:0/1025706767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00c4a30 0x7f09b00cb260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0998009840 tx=0x7f0998009b50 comp rx=0 tx=0).stop 2023-06-22T08:08:52.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.522+0000 7f09b4dff700 1 -- 172.21.15.23:0/1025706767 shutdown_connections 2023-06-22T08:08:52.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.522+0000 7f09b4dff700 1 --2- 172.21.15.23:0/1025706767 >> v2:172.21.15.67:3300/0 conn(0x7f09b00cb7a0 0x7f09b015c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:52.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.522+0000 7f09b4dff700 1 --2- 172.21.15.23:0/1025706767 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00c4a30 0x7f09b00cb260 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:52.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.523+0000 7f09b4dff700 1 -- 172.21.15.23:0/1025706767 >> 172.21.15.23:0/1025706767 conn(0x7f09b0065b90 msgr2=0x7f09b0065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:52.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.523+0000 7f09b4dff700 1 -- 172.21.15.23:0/1025706767 shutdown_connections 2023-06-22T08:08:52.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.523+0000 7f09b4dff700 1 -- 172.21.15.23:0/1025706767 wait complete. 2023-06-22T08:08:52.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.524+0000 7f09b4dff700 1 Processor -- start 2023-06-22T08:08:52.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.525+0000 7f09b4dff700 1 -- start start 2023-06-22T08:08:52.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.525+0000 7f09b4dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b00c4a30 0x7f09b01e4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:52.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.525+0000 7f09b4dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00cb7a0 0x7f09b00b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:52.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09b4dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09b0152a10 0x7f09b0157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:52.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09b4dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0149bf0 con 0x7f09b0152a10 2023-06-22T08:08:52.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09b4dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b015a5f0 con 0x7f09b00c4a30 2023-06-22T08:08:52.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09b4dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00b2590 con 0x7f09b00cb7a0 2023-06-22T08:08:52.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09ae59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b00c4a30 0x7f09b01e4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:52.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09a7fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00cb7a0 0x7f09b00b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:52.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09a7fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00cb7a0 0x7f09b00b3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:50068/0 (socket says 172.21.15.23:50068) 2023-06-22T08:08:52.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09a7fff700 1 -- 172.21.15.23:0/3008743904 learned_addr learned my addr 172.21.15.23:0/3008743904 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:52.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09aed9d700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09b0152a10 0x7f09b0157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:52.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09a7fff700 1 -- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b00c4a30 msgr2=0x7f09b01e4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:52.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09a7fff700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b00c4a30 0x7f09b01e4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:52.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09a7fff700 1 -- 172.21.15.23:0/3008743904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09b0152a10 msgr2=0x7f09b0157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:52.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09a7fff700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09b0152a10 0x7f09b0157bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:52.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09a7fff700 1 -- 172.21.15.23:0/3008743904 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a8009a70 con 0x7f09b00cb7a0 2023-06-22T08:08:52.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09aed9d700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09b0152a10 0x7f09b0157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:52.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.526+0000 7f09a7fff700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00cb7a0 0x7f09b00b3da0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f09980090b0 tx=0x7f0998004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:52.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.527+0000 7f09a6ffd700 1 -- 172.21.15.23:0/3008743904 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0998004550 con 0x7f09b00cb7a0 2023-06-22T08:08:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.527+0000 7f09a6ffd700 1 -- 172.21.15.23:0/3008743904 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099801f070 con 0x7f09b00cb7a0 2023-06-22T08:08:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.527+0000 7f09a6ffd700 1 -- 172.21.15.23:0/3008743904 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09980038e0 con 0x7f09b00cb7a0 2023-06-22T08:08:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.527+0000 7f09b4dff700 1 -- 172.21.15.23:0/3008743904 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09980094a0 con 0x7f09b00cb7a0 2023-06-22T08:08:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.527+0000 7f09b4dff700 1 -- 172.21.15.23:0/3008743904 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b0159fa0 con 0x7f09b00cb7a0 2023-06-22T08:08:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.528+0000 7f09b4dff700 1 -- 172.21.15.23:0/3008743904 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b00c3b10 con 0x7f09b00cb7a0 2023-06-22T08:08:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.529+0000 7f09a6ffd700 1 -- 172.21.15.23:0/3008743904 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0998003a40 con 0x7f09b00cb7a0 2023-06-22T08:08:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.530+0000 7f09a6ffd700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f099006ace0 0x7f099006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:52.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.530+0000 7f09a6ffd700 1 -- 172.21.15.23:0/3008743904 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f0998021070 con 0x7f09b00cb7a0 2023-06-22T08:08:52.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.530+0000 7f09ae59c700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f099006ace0 0x7f099006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:52.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.530+0000 7f09ae59c700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f099006ace0 0x7f099006d190 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f09a0007fd0 tx=0x7f09a0007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:52.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.534+0000 7f09a6ffd700 1 -- 172.21.15.23:0/3008743904 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0998055c60 con 0x7f09b00cb7a0 2023-06-22T08:08:52.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.791+0000 7f09b4dff700 1 -- 172.21.15.23:0/3008743904 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09b00c4f00 con 0x7f09b00cb7a0 2023-06-22T08:08:52.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.792+0000 7f09a6ffd700 1 -- 172.21.15.23:0/3008743904 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5508 (secure 0 0 0) 0x7f0998059280 con 0x7f09b00cb7a0 2023-06-22T08:08:52.796 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:52.797 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:51.421737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.794+0000 7f09b4dff700 1 -- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f099006ace0 msgr2=0x7f099006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.794+0000 7f09b4dff700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f099006ace0 0x7f099006d190 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f09a0007fd0 tx=0x7f09a0007dc0 comp rx=0 tx=0).stop 2023-06-22T08:08:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.794+0000 7f09b4dff700 1 -- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00cb7a0 msgr2=0x7f09b00b3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.794+0000 7f09b4dff700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00cb7a0 0x7f09b00b3da0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f09980090b0 tx=0x7f0998004750 comp rx=0 tx=0).stop 2023-06-22T08:08:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.794+0000 7f09b4dff700 1 -- 172.21.15.23:0/3008743904 shutdown_connections 2023-06-22T08:08:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.794+0000 7f09b4dff700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09b0152a10 0x7f09b0157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.794+0000 7f09b4dff700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f099006ace0 0x7f099006d190 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.794+0000 7f09b4dff700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b00c4a30 0x7f09b01e4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.794+0000 7f09b4dff700 1 --2- 172.21.15.23:0/3008743904 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09b00cb7a0 0x7f09b00b3da0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.794+0000 7f09b4dff700 1 -- 172.21.15.23:0/3008743904 >> 172.21.15.23:0/3008743904 conn(0x7f09b0065b90 msgr2=0x7f09b00ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.795+0000 7f09b4dff700 1 -- 172.21.15.23:0/3008743904 shutdown_connections 2023-06-22T08:08:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:52.795+0000 7f09b4dff700 1 -- 172.21.15.23:0/3008743904 wait complete. 2023-06-22T08:08:52.802 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 75 2023-06-22T08:08:52.817 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:08:54.818 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.023+0000 7f7fe27ff700 1 Processor -- start 2023-06-22T08:08:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.023+0000 7f7fe27ff700 1 -- start start 2023-06-22T08:08:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.024+0000 7f7fe27ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7fdc0c97d0 0x7f7fdc159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.024+0000 7f7fe27ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc0c9c30 con 0x7f7fdc152d70 2023-06-22T08:08:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.024+0000 7f7fe27ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc15bea0 con 0x7f7fdc0c45f0 2023-06-22T08:08:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.024+0000 7f7fe27ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fdc0c95e0 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fe0d9c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7fdc0c45f0 0x7f7fdc0c4a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58814/0 (socket says 172.21.15.23:58814) 2023-06-22T08:08:55.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fe0d9c700 1 -- 172.21.15.23:0/1147742357 learned_addr learned my addr 172.21.15.23:0/1147742357 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:55.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fdb7fe700 1 --2- 172.21.15.23:0/1147742357 >> v2:172.21.15.23:3300/0 conn(0x7f7fdc0c97d0 0x7f7fdc159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:55.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fdaffd700 1 -- 172.21.15.23:0/1147742357 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566029767 0 0) 0x7f7fdc15bea0 con 0x7f7fdc0c45f0 2023-06-22T08:08:55.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fdaffd700 1 -- 172.21.15.23:0/1147742357 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc40035b0 con 0x7f7fdc0c45f0 2023-06-22T08:08:55.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fdaffd700 1 -- 172.21.15.23:0/1147742357 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151502192 0 0) 0x7f7fdc0c9c30 con 0x7f7fdc152d70 2023-06-22T08:08:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fdaffd700 1 -- 172.21.15.23:0/1147742357 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fdc15bea0 con 0x7f7fdc152d70 2023-06-22T08:08:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fdb7fe700 1 -- 172.21.15.23:0/1147742357 >> v1:172.21.15.67:6789/0 conn(0x7f7fdc0c45f0 legacy=0x7f7fdc0c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fdb7fe700 1 -- 172.21.15.23:0/1147742357 >> v1:172.21.15.23:6789/0 conn(0x7f7fdc152d70 legacy=0x7f7fdc0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fdb7fe700 1 -- 172.21.15.23:0/1147742357 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc15c160 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.025+0000 7f7fdb7fe700 1 --2- 172.21.15.23:0/1147742357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fdc0c97d0 0x7f7fdc159330 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009840 tx=0x7f7fd0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19e0c89181d742a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.026+0000 7f7fdaffd700 1 -- 172.21.15.23:0/1147742357 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0011830 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.026+0000 7f7fdaffd700 1 -- 172.21.15.23:0/1147742357 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0011990 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.026+0000 7f7fdaffd700 1 -- 172.21.15.23:0/1147742357 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0011d20 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.026+0000 7f7fe27ff700 1 -- 172.21.15.23:0/1147742357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fdc0c97d0 msgr2=0x7f7fdc159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:55.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.026+0000 7f7fe27ff700 1 --2- 172.21.15.23:0/1147742357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fdc0c97d0 0x7f7fdc159330 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009840 tx=0x7f7fd0009b50 comp rx=0 tx=0).stop 2023-06-22T08:08:55.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.026+0000 7f7fe27ff700 1 -- 172.21.15.23:0/1147742357 shutdown_connections 2023-06-22T08:08:55.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.026+0000 7f7fe27ff700 1 --2- 172.21.15.23:0/1147742357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fdc0c97d0 0x7f7fdc159330 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:55.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.026+0000 7f7fe27ff700 1 -- 172.21.15.23:0/1147742357 >> 172.21.15.23:0/1147742357 conn(0x7f7fdc06ae30 msgr2=0x7f7fdc06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:55.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.026+0000 7f7fe27ff700 1 -- 172.21.15.23:0/1147742357 shutdown_connections 2023-06-22T08:08:55.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.026+0000 7f7fe27ff700 1 -- 172.21.15.23:0/1147742357 wait complete. 2023-06-22T08:08:55.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.029+0000 7f7fe27ff700 1 Processor -- start 2023-06-22T08:08:55.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.029+0000 7f7fe27ff700 1 -- start start 2023-06-22T08:08:55.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.030+0000 7f7fe27ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fdc0c45f0 0x7f7fdc1ef040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:55.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.030+0000 7f7fe27ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fdc0c97d0 0x7f7fdc151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:55.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.030+0000 7f7fe27ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc152d70 0x7f7fdc14c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:55.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.030+0000 7f7fe27ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc1e3680 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.030+0000 7f7fe27ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc1e1f70 con 0x7f7fdc152d70 2023-06-22T08:08:55.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.030+0000 7f7fe27ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc15b290 con 0x7f7fdc0c45f0 2023-06-22T08:08:55.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdbfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fdc0c45f0 0x7f7fdc1ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:55.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fe0d9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc152d70 0x7f7fdc14c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:55.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fdc0c97d0 0x7f7fdc151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:55.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdbfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fdc0c45f0 0x7f7fdc1ef040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:50072/0 (socket says 172.21.15.23:50072) 2023-06-22T08:08:55.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fe0d9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc152d70 0x7f7fdc14c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:42088/0 (socket says 172.21.15.23:42088) 2023-06-22T08:08:55.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdbfff700 1 -- 172.21.15.23:0/3883322866 learned_addr learned my addr 172.21.15.23:0/3883322866 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdb7fe700 1 -- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fdc0c45f0 msgr2=0x7f7fdc1ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdb7fe700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fdc0c45f0 0x7f7fdc1ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:55.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdb7fe700 1 -- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc152d70 msgr2=0x7f7fdc14c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:55.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdb7fe700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc152d70 0x7f7fdc14c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:55.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdb7fe700 1 -- 172.21.15.23:0/3883322866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd00094a0 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fe0d9c700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc152d70 0x7f7fdc14c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:55.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdbfff700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fdc0c45f0 0x7f7fdc1ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:08:55.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.031+0000 7f7fdb7fe700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fdc0c97d0 0x7f7fdc151620 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0c8ef0 tx=0x7f7fd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:55.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.032+0000 7f7fd8ff9700 1 -- 172.21.15.23:0/3883322866 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd001a8f0 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.032+0000 7f7fd8ff9700 1 -- 172.21.15.23:0/3883322866 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd001aa70 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.032+0000 7f7fd8ff9700 1 -- 172.21.15.23:0/3883322866 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd001ae20 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.032+0000 7f7fe27ff700 1 -- 172.21.15.23:0/3883322866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fdc15b570 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.032+0000 7f7fe27ff700 1 -- 172.21.15.23:0/3883322866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fdc1e21c0 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.033+0000 7f7fe27ff700 1 -- 172.21.15.23:0/3883322866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc04ea20 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.034+0000 7f7fd8ff9700 1 -- 172.21.15.23:0/3883322866 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7fd002d450 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.035+0000 7f7fd8ff9700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7fc806ace0 0x7f7fc806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.035+0000 7f7fd8ff9700 1 -- 172.21.15.23:0/3883322866 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f7fd008ace0 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.035+0000 7f7fdbfff700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7fc806ace0 0x7f7fc806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.036+0000 7f7fdbfff700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7fc806ace0 0x7f7fc806d190 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0011e0 tx=0x7f7fcc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:55.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.039+0000 7f7fd8ff9700 1 -- 172.21.15.23:0/3883322866 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7fd0055850 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.303+0000 7f7fe27ff700 1 -- 172.21.15.23:0/3883322866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fdc15bc10 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.304+0000 7f7fd8ff9700 1 -- 172.21.15.23:0/3883322866 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5508 (secure 0 0 0) 0x7f7fd0058e70 con 0x7f7fdc0c97d0 2023-06-22T08:08:55.308 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:55.309 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:51.421737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:55.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 -- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7fc806ace0 msgr2=0x7f7fc806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:55.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7fc806ace0 0x7f7fc806d190 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0011e0 tx=0x7f7fcc009040 comp rx=0 tx=0).stop 2023-06-22T08:08:55.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 -- 172.21.15.23:0/3883322866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fdc0c97d0 msgr2=0x7f7fdc151620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:55.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fdc0c97d0 0x7f7fdc151620 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0c8ef0 tx=0x7f7fd0004750 comp rx=0 tx=0).stop 2023-06-22T08:08:55.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 -- 172.21.15.23:0/3883322866 shutdown_connections 2023-06-22T08:08:55.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7fc806ace0 0x7f7fc806d190 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:55.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc152d70 0x7f7fdc14c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:55.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fdc0c45f0 0x7f7fdc1ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:55.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 --2- 172.21.15.23:0/3883322866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fdc0c97d0 0x7f7fdc151620 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:55.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 -- 172.21.15.23:0/3883322866 >> 172.21.15.23:0/3883322866 conn(0x7f7fdc06ae30 msgr2=0x7f7fdc0cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:55.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 -- 172.21.15.23:0/3883322866 shutdown_connections 2023-06-22T08:08:55.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:55.306+0000 7f7fe27ff700 1 -- 172.21.15.23:0/3883322866 wait complete. 2023-06-22T08:08:55.315 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 75 2023-06-22T08:08:55.329 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:08:57.330 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:08:57.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.538+0000 7f85467ff700 1 Processor -- start 2023-06-22T08:08:57.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.538+0000 7f85467ff700 1 -- start start 2023-06-22T08:08:57.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f85467ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85400c9fd0 0x7f85400cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:57.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f85467ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f85400c40b0 0x7f85400c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:57.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f85467ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854015c370 con 0x7f8540152ce0 2023-06-22T08:08:57.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f85467ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85401530b0 con 0x7f85400c40b0 2023-06-22T08:08:57.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f85467ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85400cf6f0 con 0x7f85400c9fd0 2023-06-22T08:08:57.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f853f7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85400c9fd0 0x7f85400cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:57.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f8544d9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f85400c40b0 0x7f85400c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:57.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f853ffff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8540152ce0 0x7f854015c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:57396/0 (socket says 172.21.15.23:57396) 2023-06-22T08:08:57.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f853ffff700 1 -- 172.21.15.23:0/1454140594 learned_addr learned my addr 172.21.15.23:0/1454140594 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:08:57.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f8544d9c700 1 -- 172.21.15.23:0/1454140594 >> v2:172.21.15.67:3300/0 conn(0x7f85400c9fd0 msgr2=0x7f85400cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:08:57.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f8544d9c700 1 --2- 172.21.15.23:0/1454140594 >> v2:172.21.15.67:3300/0 conn(0x7f85400c9fd0 0x7f85400cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:57.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f8544d9c700 1 -- 172.21.15.23:0/1454140594 >> v1:172.21.15.67:6789/0 conn(0x7f8540152ce0 legacy=0x7f854015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:57.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.540+0000 7f8544d9c700 1 -- 172.21.15.23:0/1454140594 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854015bda0 con 0x7f85400c40b0 2023-06-22T08:08:57.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.541+0000 7f8544d9c700 1 --2- 172.21.15.23:0/1454140594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c40b0 0x7f85400c4530 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f85400cc380 tx=0x7f853800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e44ee988619a49 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:57.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.541+0000 7f853effd700 1 -- 172.21.15.23:0/1454140594 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853800e070 con 0x7f85400c40b0 2023-06-22T08:08:57.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.541+0000 7f853effd700 1 -- 172.21.15.23:0/1454140594 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853800bb20 con 0x7f85400c40b0 2023-06-22T08:08:57.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.541+0000 7f853effd700 1 -- 172.21.15.23:0/1454140594 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538012430 con 0x7f85400c40b0 2023-06-22T08:08:57.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.542+0000 7f85467ff700 1 -- 172.21.15.23:0/1454140594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c40b0 msgr2=0x7f85400c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:57.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.542+0000 7f85467ff700 1 --2- 172.21.15.23:0/1454140594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c40b0 0x7f85400c4530 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f85400cc380 tx=0x7f853800b330 comp rx=0 tx=0).stop 2023-06-22T08:08:57.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.542+0000 7f85467ff700 1 -- 172.21.15.23:0/1454140594 shutdown_connections 2023-06-22T08:08:57.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.542+0000 7f85467ff700 1 --2- 172.21.15.23:0/1454140594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c40b0 0x7f85400c4530 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:57.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.542+0000 7f85467ff700 1 --2- 172.21.15.23:0/1454140594 >> v2:172.21.15.67:3300/0 conn(0x7f85400c9fd0 0x7f85400cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:57.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.542+0000 7f85467ff700 1 -- 172.21.15.23:0/1454140594 >> 172.21.15.23:0/1454140594 conn(0x7f854006ae30 msgr2=0x7f854006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:57.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.542+0000 7f85467ff700 1 -- 172.21.15.23:0/1454140594 shutdown_connections 2023-06-22T08:08:57.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.542+0000 7f85467ff700 1 -- 172.21.15.23:0/1454140594 wait complete. 2023-06-22T08:08:57.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.544+0000 7f85467ff700 1 Processor -- start 2023-06-22T08:08:57.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.544+0000 7f85467ff700 1 -- start start 2023-06-22T08:08:57.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f85467ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85400c40b0 0x7f85401e3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:57.558 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f85467ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c9fd0 0x7f85401e3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:57.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f85467ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8540152ce0 0x7f85401ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:57.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f85467ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85400cd8a0 con 0x7f85400c9fd0 2023-06-22T08:08:57.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f85467ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85400ce3e0 con 0x7f8540152ce0 2023-06-22T08:08:57.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f85467ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85401eee00 con 0x7f85400c40b0 2023-06-22T08:08:57.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f853f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c9fd0 0x7f85401e3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:57.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f853f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c9fd0 0x7f85401e3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36398/0 (socket says 172.21.15.23:36398) 2023-06-22T08:08:57.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f853f7fe700 1 -- 172.21.15.23:0/3152017426 learned_addr learned my addr 172.21.15.23:0/3152017426 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:08:57.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f8544d9c700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8540152ce0 0x7f85401ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:57.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f853ffff700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85400c40b0 0x7f85401e3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:57.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f853f7fe700 1 -- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85400c40b0 msgr2=0x7f85401e3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:57.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f853f7fe700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85400c40b0 0x7f85401e3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:57.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f853f7fe700 1 -- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8540152ce0 msgr2=0x7f85401ee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:57.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f853f7fe700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8540152ce0 0x7f85401ee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:57.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.545+0000 7f853f7fe700 1 -- 172.21.15.23:0/3152017426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8538009a70 con 0x7f85400c9fd0 2023-06-22T08:08:57.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.546+0000 7f853f7fe700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c9fd0 0x7f85401e3eb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f853400e9b0 tx=0x7f853400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:57.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.546+0000 7f853cff9700 1 -- 172.21.15.23:0/3152017426 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534009710 con 0x7f85400c9fd0 2023-06-22T08:08:57.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.546+0000 7f853cff9700 1 -- 172.21.15.23:0/3152017426 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853400ee50 con 0x7f85400c9fd0 2023-06-22T08:08:57.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.546+0000 7f853cff9700 1 -- 172.21.15.23:0/3152017426 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853400ca50 con 0x7f85400c9fd0 2023-06-22T08:08:57.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.546+0000 7f85467ff700 1 -- 172.21.15.23:0/3152017426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85400cdb40 con 0x7f85400c9fd0 2023-06-22T08:08:57.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.546+0000 7f85467ff700 1 -- 172.21.15.23:0/3152017426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85400ce100 con 0x7f85400c9fd0 2023-06-22T08:08:57.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.547+0000 7f85467ff700 1 -- 172.21.15.23:0/3152017426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854004ea20 con 0x7f85400c9fd0 2023-06-22T08:08:57.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.548+0000 7f853cff9700 1 -- 172.21.15.23:0/3152017426 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f853400cbb0 con 0x7f85400c9fd0 2023-06-22T08:08:57.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.549+0000 7f853cff9700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f852c06ace0 0x7f852c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:08:57.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.549+0000 7f853cff9700 1 -- 172.21.15.23:0/3152017426 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f85340899d0 con 0x7f85400c9fd0 2023-06-22T08:08:57.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.549+0000 7f853ffff700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f852c06ace0 0x7f852c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:08:57.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.550+0000 7f853ffff700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f852c06ace0 0x7f852c06d190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8530005910 tx=0x7f8530009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:08:57.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.554+0000 7f853cff9700 1 -- 172.21.15.23:0/3152017426 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8534057fe0 con 0x7f85400c9fd0 2023-06-22T08:08:57.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.810+0000 7f85467ff700 1 -- 172.21.15.23:0/3152017426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85400c9db0 con 0x7f85400c9fd0 2023-06-22T08:08:57.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.811+0000 7f853cff9700 1 -- 172.21.15.23:0/3152017426 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5508 (secure 0 0 0) 0x7f8534017090 con 0x7f85400c9fd0 2023-06-22T08:08:57.816 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:08:57.816 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:56.516995+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:08:57.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.813+0000 7f85467ff700 1 -- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f852c06ace0 msgr2=0x7f852c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:57.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.813+0000 7f85467ff700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f852c06ace0 0x7f852c06d190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8530005910 tx=0x7f8530009040 comp rx=0 tx=0).stop 2023-06-22T08:08:57.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.814+0000 7f85467ff700 1 -- 172.21.15.23:0/3152017426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c9fd0 msgr2=0x7f85401e3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:08:57.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.814+0000 7f85467ff700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c9fd0 0x7f85401e3eb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f853400e9b0 tx=0x7f853400ecc0 comp rx=0 tx=0).stop 2023-06-22T08:08:57.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.814+0000 7f85467ff700 1 -- 172.21.15.23:0/3152017426 shutdown_connections 2023-06-22T08:08:57.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.814+0000 7f85467ff700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f852c06ace0 0x7f852c06d190 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:57.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.814+0000 7f85467ff700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8540152ce0 0x7f85401ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:57.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.814+0000 7f85467ff700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f85400c40b0 0x7f85401e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:57.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.814+0000 7f85467ff700 1 --2- 172.21.15.23:0/3152017426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85400c9fd0 0x7f85401e3eb0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:08:57.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.814+0000 7f85467ff700 1 -- 172.21.15.23:0/3152017426 >> 172.21.15.23:0/3152017426 conn(0x7f854006ae30 msgr2=0x7f85400cb760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:08:57.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.814+0000 7f85467ff700 1 -- 172.21.15.23:0/3152017426 shutdown_connections 2023-06-22T08:08:57.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:08:57.814+0000 7f85467ff700 1 -- 172.21.15.23:0/3152017426 wait complete. 2023-06-22T08:08:57.820 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 76 2023-06-22T08:08:57.836 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=4279 2023-06-22T08:08:57.836 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2023-06-22T08:09:00.625 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:09:00.622+0000 7fa5334b2700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-06-22T08:09:02.406 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 1% (3/193) Updating files: 2% (4/193) Updating files: 3% (6/193) Updating files: 4% (8/193) Updating files: 5% (10/193) Updating files: 6% (12/193) Updating files: 7% (14/193) Updating files: 8% (16/193) Updating files: 9% (18/193) Updating files: 10% (20/193) Updating files: 11% (22/193) Updating files: 12% (24/193) Updating files: 13% (26/193) Updating files: 14% (28/193) Updating files: 15% (29/193) Updating files: 16% (31/193) Updating files: 17% (33/193) Updating files: 18% (35/193) Updating files: 19% (37/193) Updating files: 20% (39/193) Updating files: 21% (41/193) Updating files: 22% (43/193) Updating files: 23% (45/193) Updating files: 24% (47/193) Updating files: 25% (49/193) Updating files: 26% (51/193) Updating files: 27% (53/193) Updating files: 28% (55/193) Updating files: 29% (56/193) Updating files: 30% (58/193) Updating files: 31% (60/193) Updating files: 32% (62/193) Updating files: 33% (64/193) Updating files: 34% (66/193) Updating files: 35% (68/193) Updating files: 36% (70/193) Updating files: 37% (72/193) Updating files: 38% (74/193) Updating files: 39% (76/193) Updating files: 40% (78/193) Updating files: 41% (80/193) Updating files: 42% (82/193) Updating files: 43% (83/193) Updating files: 44% (85/193) Updating files: 45% (87/193) Updating files: 46% (89/193) Updating files: 47% (91/193) Updating files: 48% (93/193) Updating files: 49% (95/193) Updating files: 50% (97/193) Updating files: 51% (99/193) Updating files: 52% (101/193) Updating files: 53% (103/193) Updating files: 54% (105/193) Updating files: 55% (107/193) Updating files: 56% (109/193) Updating files: 57% (111/193) Updating files: 58% (112/193) Updating files: 59% (114/193) Updating files: 60% (116/193) Updating files: 61% (118/193) Updating files: 62% (120/193) Updating files: 63% (122/193) Updating files: 64% (124/193) Updating files: 65% (126/193) Updating files: 66% (128/193) Updating files: 67% (130/193) Updating files: 68% (132/193) Updating files: 69% (134/193) Updating files: 70% (136/193) Updating files: 71% (138/193) Updating files: 72% (139/193) Updating files: 73% (141/193) Updating files: 74% (143/193) Updating files: 75% (145/193) Updating files: 76% (147/193) Updating files: 77% (149/193) Updating files: 78% (151/193) Updating files: 79% (153/193) Updating files: 80% (155/193) Updating files: 81% (157/193) Updating files: 82% (159/193) Updating files: 83% (161/193) Updating files: 84% (163/193) Updating files: 85% (165/193) Updating files: 86% (166/193) Updating files: 87% (168/193) Updating files: 88% (170/193) Updating files: 89% (172/193) Updating files: 90% (174/193) Updating files: 91% (176/193) Updating files: 92% (178/193) Updating files: 93% (180/193) Updating files: 94% (182/193) Updating files: 95% (184/193) Updating files: 96% (186/193) Updating files: 97% (188/193) Updating files: 98% (190/193) Updating files: 99% (192/193) Updating files: 100% (193/193) Updating files: 100% (193/193), done. 2023-06-22T08:09:02.691 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2023-06-22T08:09:02.697 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.9 2023-06-22T08:09:02.732 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:02.733 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 10 -eq 48 ']' 2023-06-22T08:09:02.733 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.10 2023-06-22T08:09:02.733 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.10 2023-06-22T08:09:02.733 INFO:tasks.workunit.client.0.smithi023.stdout:v0.10 2023-06-22T08:09:02.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.10 2023-06-22T08:09:03.837 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-06-22T08:09:03.838 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-22T08:09:03.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-22T08:09:03.840 INFO:tasks.ceph.mds.b:Started 2023-06-22T08:09:03.840 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:09:03.920 INFO:tasks.ceph.mds.b.smithi067.stdout:starting mds.b at 2023-06-22T08:09:04.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.057+0000 7f6e789ff700 1 Processor -- start 2023-06-22T08:09:04.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.057+0000 7f6e789ff700 1 -- start start 2023-06-22T08:09:04.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.059+0000 7f6e789ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6e740cb6b0 0x7f6e74069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.059+0000 7f6e789ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e740c5740 0x7f6e741593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.059+0000 7f6e789ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e740cba80 0x7f6e74159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.059+0000 7f6e789ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e7406bd20 con 0x7f6e740cb6b0 2023-06-22T08:09:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.059+0000 7f6e789ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e74158f60 con 0x7f6e740c5740 2023-06-22T08:09:04.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.059+0000 7f6e789ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e740cc450 con 0x7f6e740cba80 2023-06-22T08:09:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e72d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e740cba80 0x7f6e74159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e72d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e740cba80 0x7f6e74159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:46198/0 (socket says 172.21.15.23:46198) 2023-06-22T08:09:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e72d9d700 1 -- 172.21.15.23:0/4274486674 learned_addr learned my addr 172.21.15.23:0/4274486674 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:09:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e71d9b700 1 --2- 172.21.15.23:0/4274486674 >> v2:172.21.15.67:3300/0 conn(0x7f6e740c5740 0x7f6e741593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e7259c700 1 --2- 172.21.15.23:0/4274486674 >> v2:172.21.15.23:3300/0 conn(0x7f6e740cb6b0 0x7f6e74069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e72d9d700 1 -- 172.21.15.23:0/4274486674 >> v2:172.21.15.67:3300/0 conn(0x7f6e740c5740 msgr2=0x7f6e741593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:09:04.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e72d9d700 1 --2- 172.21.15.23:0/4274486674 >> v2:172.21.15.67:3300/0 conn(0x7f6e740c5740 0x7f6e741593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.067 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e72d9d700 1 -- 172.21.15.23:0/4274486674 >> v2:172.21.15.23:3300/0 conn(0x7f6e740cb6b0 msgr2=0x7f6e74069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:09:04.067 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e72d9d700 1 --2- 172.21.15.23:0/4274486674 >> v2:172.21.15.23:3300/0 conn(0x7f6e740cb6b0 0x7f6e74069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.067 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e72d9d700 1 -- 172.21.15.23:0/4274486674 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74158980 con 0x7f6e740cba80 2023-06-22T08:09:04.067 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.060+0000 7f6e72d9d700 1 --2- 172.21.15.23:0/4274486674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e740cba80 0x7f6e74159ce0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6e7415bd10 tx=0x7f6e6c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb5fb837a4f3bc2c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:09:04.067 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.061+0000 7f6e7159a700 1 -- 172.21.15.23:0/4274486674 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c00e070 con 0x7f6e740cba80 2023-06-22T08:09:04.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.061+0000 7f6e7159a700 1 -- 172.21.15.23:0/4274486674 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c00b910 con 0x7f6e740cba80 2023-06-22T08:09:04.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.061+0000 7f6e7159a700 1 -- 172.21.15.23:0/4274486674 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c00bc80 con 0x7f6e740cba80 2023-06-22T08:09:04.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.061+0000 7f6e789ff700 1 -- 172.21.15.23:0/4274486674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e740cba80 msgr2=0x7f6e74159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:04.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.061+0000 7f6e789ff700 1 --2- 172.21.15.23:0/4274486674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e740cba80 0x7f6e74159ce0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6e7415bd10 tx=0x7f6e6c00b330 comp rx=0 tx=0).stop 2023-06-22T08:09:04.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.061+0000 7f6e789ff700 1 -- 172.21.15.23:0/4274486674 shutdown_connections 2023-06-22T08:09:04.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.061+0000 7f6e789ff700 1 --2- 172.21.15.23:0/4274486674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e740cba80 0x7f6e74159ce0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.069 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.061+0000 7f6e789ff700 1 --2- 172.21.15.23:0/4274486674 >> v2:172.21.15.23:3300/0 conn(0x7f6e740cb6b0 0x7f6e74069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.069 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.061+0000 7f6e789ff700 1 --2- 172.21.15.23:0/4274486674 >> v2:172.21.15.67:3300/0 conn(0x7f6e740c5740 0x7f6e741593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.069 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.061+0000 7f6e789ff700 1 -- 172.21.15.23:0/4274486674 >> 172.21.15.23:0/4274486674 conn(0x7f6e74070d80 msgr2=0x7f6e74071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:09:04.069 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.062+0000 7f6e789ff700 1 -- 172.21.15.23:0/4274486674 shutdown_connections 2023-06-22T08:09:04.069 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.062+0000 7f6e789ff700 1 -- 172.21.15.23:0/4274486674 wait complete. 2023-06-22T08:09:04.070 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.064+0000 7f6e789ff700 1 Processor -- start 2023-06-22T08:09:04.071 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.064+0000 7f6e789ff700 1 -- start start 2023-06-22T08:09:04.071 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 007313fe844 v0.10 2023-06-22T08:09:04.073 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e789ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e740cb6b0 0x7f6e741ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:04.073 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e789ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e740cba80 0x7f6e74156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:04.073 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e789ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e74147550 0x7f6e741578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:04.073 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e789ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e741f2390 con 0x7f6e740cba80 2023-06-22T08:09:04.073 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e789ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e741f28c0 con 0x7f6e740cb6b0 2023-06-22T08:09:04.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e789ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e741470c0 con 0x7f6e74147550 2023-06-22T08:09:04.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e71d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e740cba80 0x7f6e74156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:04.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e72d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e74147550 0x7f6e741578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:04.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e7259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e740cb6b0 0x7f6e741ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:04.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e72d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e74147550 0x7f6e741578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:46210/0 (socket says 172.21.15.23:46210) 2023-06-22T08:09:04.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.065+0000 7f6e72d9d700 1 -- 172.21.15.23:0/2608091864 learned_addr learned my addr 172.21.15.23:0/2608091864 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:09:04.074 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e72d9d700 1 -- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e740cb6b0 msgr2=0x7f6e741ee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:04.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e72d9d700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e740cb6b0 0x7f6e741ee410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e72d9d700 1 -- 172.21.15.23:0/2608091864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e740cba80 msgr2=0x7f6e74156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:04.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e72d9d700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e740cba80 0x7f6e74156840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e72d9d700 1 -- 172.21.15.23:0/2608091864 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c009b30 con 0x7f6e74147550 2023-06-22T08:09:04.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e72d9d700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e74147550 0x7f6e741578b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00b680 tx=0x7f6e6c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:09:04.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e62ffd700 1 -- 172.21.15.23:0/2608091864 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c0092e0 con 0x7f6e74147550 2023-06-22T08:09:04.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e62ffd700 1 -- 172.21.15.23:0/2608091864 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c0094d0 con 0x7f6e74147550 2023-06-22T08:09:04.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e62ffd700 1 -- 172.21.15.23:0/2608091864 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c004470 con 0x7f6e74147550 2023-06-22T08:09:04.076 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e789ff700 1 -- 172.21.15.23:0/2608091864 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e740c8840 con 0x7f6e74147550 2023-06-22T08:09:04.076 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.066+0000 7f6e789ff700 1 -- 172.21.15.23:0/2608091864 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e74146d50 con 0x7f6e74147550 2023-06-22T08:09:04.076 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.068+0000 7f6e789ff700 1 -- 172.21.15.23:0/2608091864 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e74146f10 con 0x7f6e74147550 2023-06-22T08:09:04.076 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.069+0000 7f6e62ffd700 1 -- 172.21.15.23:0/2608091864 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6e6c0045d0 con 0x7f6e74147550 2023-06-22T08:09:04.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.070+0000 7f6e62ffd700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ad10 0x7f6e5406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:04.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.070+0000 7f6e62ffd700 1 -- 172.21.15.23:0/2608091864 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f6e6c08a9e0 con 0x7f6e74147550 2023-06-22T08:09:04.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.070+0000 7f6e7259c700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ad10 0x7f6e5406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:04.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.070+0000 7f6e7259c700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ad10 0x7f6e5406d1c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6e64006b20 tx=0x7f6e64008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:09:04.078 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.10 2023-06-22T08:09:04.078 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.074+0000 7f6e62ffd700 1 -- 172.21.15.23:0/2608091864 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6e6c058ff0 con 0x7f6e74147550 2023-06-22T08:09:04.127 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:04.128 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 11 -eq 48 ']' 2023-06-22T08:09:04.128 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.11 2023-06-22T08:09:04.129 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.11 2023-06-22T08:09:04.130 INFO:tasks.workunit.client.0.smithi023.stdout:v0.11 2023-06-22T08:09:04.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.11 2023-06-22T08:09:04.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.460+0000 7f6e789ff700 1 -- 172.21.15.23:0/2608091864 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e7404ea20 con 0x7f6e74147550 2023-06-22T08:09:04.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.461+0000 7f6e62ffd700 1 -- 172.21.15.23:0/2608091864 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+5508 (secure 0 0 0) 0x7f6e6c058b80 con 0x7f6e74147550 2023-06-22T08:09:04.465 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:09:04.465 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:56.516995+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:09:04.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 -- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ad10 msgr2=0x7f6e5406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:04.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ad10 0x7f6e5406d1c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6e64006b20 tx=0x7f6e64008040 comp rx=0 tx=0).stop 2023-06-22T08:09:04.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 -- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e74147550 msgr2=0x7f6e741578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:04.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e74147550 0x7f6e741578b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00b680 tx=0x7f6e6c004ad0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 -- 172.21.15.23:0/2608091864 shutdown_connections 2023-06-22T08:09:04.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6e74147550 0x7f6e741578b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6e5406ad10 0x7f6e5406d1c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e740cb6b0 0x7f6e741ee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 --2- 172.21.15.23:0/2608091864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e740cba80 0x7f6e74156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:04.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 -- 172.21.15.23:0/2608091864 >> 172.21.15.23:0/2608091864 conn(0x7f6e74070d80 msgr2=0x7f6e740ca860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:09:04.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 -- 172.21.15.23:0/2608091864 shutdown_connections 2023-06-22T08:09:04.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:04.463+0000 7f6e789ff700 1 -- 172.21.15.23:0/2608091864 wait complete. 2023-06-22T08:09:04.472 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 76 2023-06-22T08:09:04.485 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-06-22T08:09:04.737 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2023-06-22T08:09:04.741 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.11 2023-06-22T08:09:04.776 INFO:tasks.workunit.client.0.smithi023.stdout:v0.12 2023-06-22T08:09:04.776 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:04.777 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 12 -eq 48 ']' 2023-06-22T08:09:04.777 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.12 2023-06-22T08:09:04.777 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.12 2023-06-22T08:09:04.778 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.12 2023-06-22T08:09:05.354 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at f524150822a v0.12 2023-06-22T08:09:05.361 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.12 2023-06-22T08:09:05.447 INFO:tasks.workunit.client.0.smithi023.stdout:v0.13 2023-06-22T08:09:05.447 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:05.447 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 13 -eq 48 ']' 2023-06-22T08:09:05.447 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.13 2023-06-22T08:09:05.448 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.13 2023-06-22T08:09:05.448 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.13 2023-06-22T08:09:06.206 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 2b32297267d v0.13 2023-06-22T08:09:06.211 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.13 2023-06-22T08:09:06.268 INFO:tasks.workunit.client.0.smithi023.stdout:v0.14 2023-06-22T08:09:06.269 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:06.269 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 14 -eq 48 ']' 2023-06-22T08:09:06.269 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.14 2023-06-22T08:09:06.270 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.14 2023-06-22T08:09:06.270 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.14 2023-06-22T08:09:06.487 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:09:06.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.716+0000 7fe5a97ff700 1 Processor -- start 2023-06-22T08:09:06.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.716+0000 7fe5a97ff700 1 -- start start 2023-06-22T08:09:06.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a97ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5a414f850 0x7fe5a40c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:06.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a97ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe5a40c69e0 0x7fe5a40c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:06.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a97ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40c51a0 con 0x7fe5a4071070 2023-06-22T08:09:06.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a97ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a40c44e0 con 0x7fe5a40c69e0 2023-06-22T08:09:06.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a97ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a40c5440 con 0x7fe5a414f850 2023-06-22T08:09:06.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a2ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5a414f850 0x7fe5a40c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:06.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a27fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe5a40c69e0 0x7fe5a40c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:06.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a37fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe5a4071070 0x7fe5a4155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:49822/0 (socket says 172.21.15.23:49822) 2023-06-22T08:09:06.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a37fe700 1 -- 172.21.15.23:0/3193989999 learned_addr learned my addr 172.21.15.23:0/3193989999 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:09:06.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a27fc700 1 -- 172.21.15.23:0/3193989999 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5a414f850 msgr2=0x7fe5a40c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:09:06.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a27fc700 1 --2- 172.21.15.23:0/3193989999 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5a414f850 0x7fe5a40c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:06.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a27fc700 1 -- 172.21.15.23:0/3193989999 >> v1:172.21.15.67:6789/0 conn(0x7fe5a4071070 legacy=0x7fe5a4155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:06.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a27fc700 1 -- 172.21.15.23:0/3193989999 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40c5720 con 0x7fe5a40c69e0 2023-06-22T08:09:06.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.718+0000 7fe5a1ffb700 1 -- 172.21.15.23:0/3193989999 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652187999 0 0) 0x7fe5a40c51a0 con 0x7fe5a4071070 2023-06-22T08:09:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.719+0000 7fe5a27fc700 1 --2- 172.21.15.23:0/3193989999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 0x7fe5a40c35c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe598009840 tx=0x7fe598009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76697e8b3641b888 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:09:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.719+0000 7fe5a1ffb700 1 -- 172.21.15.23:0/3193989999 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598011830 con 0x7fe5a40c69e0 2023-06-22T08:09:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.719+0000 7fe5a1ffb700 1 -- 172.21.15.23:0/3193989999 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe598011990 con 0x7fe5a40c69e0 2023-06-22T08:09:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.719+0000 7fe5a1ffb700 1 -- 172.21.15.23:0/3193989999 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598011d20 con 0x7fe5a40c69e0 2023-06-22T08:09:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.720+0000 7fe5a97ff700 1 -- 172.21.15.23:0/3193989999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 msgr2=0x7fe5a40c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.720+0000 7fe5a97ff700 1 --2- 172.21.15.23:0/3193989999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 0x7fe5a40c35c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe598009840 tx=0x7fe598009b50 comp rx=0 tx=0).stop 2023-06-22T08:09:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.720+0000 7fe5a97ff700 1 -- 172.21.15.23:0/3193989999 shutdown_connections 2023-06-22T08:09:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.720+0000 7fe5a97ff700 1 --2- 172.21.15.23:0/3193989999 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5a414f850 0x7fe5a40c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.720+0000 7fe5a97ff700 1 --2- 172.21.15.23:0/3193989999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 0x7fe5a40c35c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.720+0000 7fe5a97ff700 1 -- 172.21.15.23:0/3193989999 >> 172.21.15.23:0/3193989999 conn(0x7fe5a40ca3f0 msgr2=0x7fe5a40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:09:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.720+0000 7fe5a97ff700 1 -- 172.21.15.23:0/3193989999 shutdown_connections 2023-06-22T08:09:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.720+0000 7fe5a97ff700 1 -- 172.21.15.23:0/3193989999 wait complete. 2023-06-22T08:09:06.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.722+0000 7fe5a97ff700 1 Processor -- start 2023-06-22T08:09:06.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.722+0000 7fe5a97ff700 1 -- start start 2023-06-22T08:09:06.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a97ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5a4071070 0x7fe5a414adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:06.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a97ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 0x7fe5a41df170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:06.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a97ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5a414f850 0x7fe5a41df6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:06.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a97ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a414d3a0 con 0x7fe5a4071070 2023-06-22T08:09:06.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a97ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41ee7b0 con 0x7fe5a40c69e0 2023-06-22T08:09:06.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a97ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a41ee9a0 con 0x7fe5a414f850 2023-06-22T08:09:06.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a27fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 0x7fe5a41df170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:06.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5a4071070 0x7fe5a414adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:06.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a37fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5a414f850 0x7fe5a41df6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:06.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a27fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 0x7fe5a41df170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:56390/0 (socket says 172.21.15.23:56390) 2023-06-22T08:09:06.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.723+0000 7fe5a27fc700 1 -- 172.21.15.23:0/1354172716 learned_addr learned my addr 172.21.15.23:0/1354172716 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:09:06.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.724+0000 7fe5a27fc700 1 -- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5a414f850 msgr2=0x7fe5a41df6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:06.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.724+0000 7fe5a27fc700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5a414f850 0x7fe5a41df6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:06.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.724+0000 7fe5a27fc700 1 -- 172.21.15.23:0/1354172716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5a4071070 msgr2=0x7fe5a414adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:06.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.724+0000 7fe5a27fc700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5a4071070 0x7fe5a414adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:06.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.724+0000 7fe5a27fc700 1 -- 172.21.15.23:0/1354172716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5980094a0 con 0x7fe5a40c69e0 2023-06-22T08:09:06.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.724+0000 7fe5a27fc700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 0x7fe5a41df170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe598009fd0 tx=0x7fe598004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:09:06.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.724+0000 7fe58b7fe700 1 -- 172.21.15.23:0/1354172716 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5980036a0 con 0x7fe5a40c69e0 2023-06-22T08:09:06.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.724+0000 7fe58b7fe700 1 -- 172.21.15.23:0/1354172716 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59801f070 con 0x7fe5a40c69e0 2023-06-22T08:09:06.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.724+0000 7fe5a97ff700 1 -- 172.21.15.23:0/1354172716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a41eec80 con 0x7fe5a40c69e0 2023-06-22T08:09:06.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.724+0000 7fe58b7fe700 1 -- 172.21.15.23:0/1354172716 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598003a20 con 0x7fe5a40c69e0 2023-06-22T08:09:06.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.725+0000 7fe5a97ff700 1 -- 172.21.15.23:0/1354172716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a41ef240 con 0x7fe5a40c69e0 2023-06-22T08:09:06.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.726+0000 7fe5a97ff700 1 -- 172.21.15.23:0/1354172716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a40c2950 con 0x7fe5a40c69e0 2023-06-22T08:09:06.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.727+0000 7fe58b7fe700 1 -- 172.21.15.23:0/1354172716 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe598003b80 con 0x7fe5a40c69e0 2023-06-22T08:09:06.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.728+0000 7fe58b7fe700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe59006ace0 0x7fe59006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:09:06.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.729+0000 7fe58b7fe700 1 -- 172.21.15.23:0/1354172716 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7fe598021070 con 0x7fe5a40c69e0 2023-06-22T08:09:06.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.729+0000 7fe5a2ffd700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe59006ace0 0x7fe59006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:09:06.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.729+0000 7fe5a2ffd700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe59006ace0 0x7fe59006d190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe594009b40 tx=0x7fe594009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:09:06.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.737+0000 7fe58b7fe700 1 -- 172.21.15.23:0/1354172716 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe598055c60 con 0x7fe5a40c69e0 2023-06-22T08:09:07.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.997+0000 7fe5a97ff700 1 -- 172.21.15.23:0/1354172716 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5a404ea20 con 0x7fe5a40c69e0 2023-06-22T08:09:07.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:06.998+0000 7fe58b7fe700 1 -- 172.21.15.23:0/1354172716 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6264 (secure 0 0 0) 0x7fe598015090 con 0x7fe5a40c69e0 2023-06-22T08:09:07.002 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:09:07.003 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:08:56.516995+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:09:07.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.000+0000 7fe5a97ff700 1 -- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe59006ace0 msgr2=0x7fe59006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:07.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.000+0000 7fe5a97ff700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe59006ace0 0x7fe59006d190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe594009b40 tx=0x7fe594009210 comp rx=0 tx=0).stop 2023-06-22T08:09:07.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.000+0000 7fe5a97ff700 1 -- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 msgr2=0x7fe5a41df170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:09:07.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.000+0000 7fe5a97ff700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 0x7fe5a41df170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe598009fd0 tx=0x7fe598004750 comp rx=0 tx=0).stop 2023-06-22T08:09:07.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.000+0000 7fe5a97ff700 1 -- 172.21.15.23:0/1354172716 shutdown_connections 2023-06-22T08:09:07.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.000+0000 7fe5a97ff700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe5a414f850 0x7fe5a41df6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:07.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.000+0000 7fe5a97ff700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5a4071070 0x7fe5a414adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:07.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.000+0000 7fe5a97ff700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe59006ace0 0x7fe59006d190 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:07.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.000+0000 7fe5a97ff700 1 --2- 172.21.15.23:0/1354172716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe5a40c69e0 0x7fe5a41df170 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:09:07.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.000+0000 7fe5a97ff700 1 -- 172.21.15.23:0/1354172716 >> 172.21.15.23:0/1354172716 conn(0x7fe5a40ca3f0 msgr2=0x7fe5a40c77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:09:07.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.001+0000 7fe5a97ff700 1 -- 172.21.15.23:0/1354172716 shutdown_connections 2023-06-22T08:09:07.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:09:07.001+0000 7fe5a97ff700 1 -- 172.21.15.23:0/1354172716 wait complete. 2023-06-22T08:09:07.008 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 77 2023-06-22T08:09:07.022 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-06-22T08:09:07.022 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-06-22T08:09:07.101 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2023-06-22T08:09:07.108 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.14 2023-06-22T08:09:07.152 INFO:tasks.workunit.client.0.smithi023.stdout:v0.15 2023-06-22T08:09:07.152 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:07.153 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 15 -eq 48 ']' 2023-06-22T08:09:07.153 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.15 2023-06-22T08:09:07.153 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.15 2023-06-22T08:09:07.154 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.15 2023-06-22T08:09:08.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:09:08.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:09:31.434 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 5% (6/112) Updating files: 6% (7/112) Updating files: 7% (8/112) Updating files: 8% (9/112) Updating files: 9% (11/112) Updating files: 10% (12/112) Updating files: 11% (13/112) Updating files: 12% (14/112) Updating files: 13% (15/112) Updating files: 14% (16/112) Updating files: 15% (17/112) Updating files: 16% (18/112) Updating files: 17% (20/112) Updating files: 18% (21/112) Updating files: 19% (22/112) Updating files: 20% (23/112) Updating files: 21% (24/112) Updating files: 22% (25/112) Updating files: 23% (26/112) Updating files: 24% (27/112) Updating files: 25% (28/112) Updating files: 26% (30/112) Updating files: 27% (31/112) Updating files: 28% (32/112) Updating files: 29% (33/112) Updating files: 30% (34/112) Updating files: 31% (35/112) Updating files: 32% (36/112) Updating files: 33% (37/112) Updating files: 34% (39/112) Updating files: 35% (40/112) Updating files: 36% (41/112) Updating files: 37% (42/112) Updating files: 38% (43/112) Updating files: 39% (44/112) Updating files: 40% (45/112) Updating files: 41% (46/112) Updating files: 42% (48/112) Updating files: 43% (49/112) Updating files: 44% (50/112) Updating files: 45% (51/112) Updating files: 46% (52/112) Updating files: 47% (53/112) Updating files: 48% (54/112) Updating files: 49% (55/112) Updating files: 50% (56/112) Updating files: 51% (58/112) Updating files: 52% (59/112) Updating files: 53% (60/112) Updating files: 54% (61/112) Updating files: 55% (62/112) Updating files: 56% (63/112) Updating files: 57% (64/112) Updating files: 58% (65/112) Updating files: 59% (67/112) Updating files: 60% (68/112) Updating files: 61% (69/112) Updating files: 62% (70/112) Updating files: 63% (71/112) Updating files: 64% (72/112) Updating files: 65% (73/112) Updating files: 66% (74/112) Updating files: 67% (76/112) Updating files: 68% (77/112) Updating files: 69% (78/112) Updating files: 70% (79/112) Updating files: 71% (80/112) Updating files: 72% (81/112) Updating files: 73% (82/112) Updating files: 74% (83/112) Updating files: 75% (84/112) Updating files: 76% (86/112) Updating files: 77% (87/112) Updating files: 78% (88/112) Updating files: 79% (89/112) Updating files: 80% (90/112) Updating files: 81% (91/112) Updating files: 82% (92/112) Updating files: 83% (93/112) Updating files: 84% (95/112) Updating files: 85% (96/112) Updating files: 86% (97/112) Updating files: 87% (98/112) Updating files: 88% (99/112) Updating files: 89% (100/112) Updating files: 90% (101/112) Updating files: 91% (102/112) Updating files: 92% (104/112) Updating files: 93% (105/112) Updating files: 94% (106/112) Updating files: 95% (107/112) Updating files: 96% (108/112) Updating files: 97% (109/112) Updating files: 98% (110/112) Updating files: 99% (111/112) Updating files: 100% (112/112) Updating files: 100% (112/112), done. 2023-06-22T08:09:31.506 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at ab3c421eb95 v0.15 2023-06-22T08:09:31.510 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.15 2023-06-22T08:09:31.597 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:31.597 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 16 -eq 48 ']' 2023-06-22T08:09:31.598 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.16 2023-06-22T08:09:31.598 INFO:tasks.workunit.client.0.smithi023.stdout:v0.16 2023-06-22T08:09:31.598 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.16 2023-06-22T08:09:31.598 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.16 2023-06-22T08:09:32.126 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 38dbaa5175c v0.16 2023-06-22T08:09:32.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.16 2023-06-22T08:09:32.194 INFO:tasks.workunit.client.0.smithi023.stdout:v0.17 2023-06-22T08:09:32.194 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:32.195 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 17 -eq 48 ']' 2023-06-22T08:09:32.195 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.17 2023-06-22T08:09:32.195 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.17 2023-06-22T08:09:32.195 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.17 2023-06-22T08:09:34.868 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 89% (195/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-06-22T08:09:34.933 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at dac3a8c35fb v0.17 2023-06-22T08:09:34.937 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.17 2023-06-22T08:09:34.993 INFO:tasks.workunit.client.0.smithi023.stdout:v0.18 2023-06-22T08:09:34.994 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:34.994 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 18 -eq 48 ']' 2023-06-22T08:09:34.994 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.18 2023-06-22T08:09:34.994 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.18 2023-06-22T08:09:34.994 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.18 2023-06-22T08:09:38.051 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 37% (295/779) Updating files: 38% (297/779) Updating files: 39% (304/779) Updating files: 40% (312/779) Updating files: 41% (320/779) Updating files: 42% (328/779) Updating files: 43% (335/779) Updating files: 44% (343/779) Updating files: 45% (351/779) Updating files: 46% (359/779) Updating files: 47% (367/779) Updating files: 48% (374/779) Updating files: 49% (382/779) Updating files: 50% (390/779) Updating files: 51% (398/779) Updating files: 52% (406/779) Updating files: 53% (413/779) Updating files: 54% (421/779) Updating files: 55% (429/779) Updating files: 56% (437/779) Updating files: 57% (445/779) Updating files: 58% (452/779) Updating files: 59% (460/779) Updating files: 60% (468/779) Updating files: 61% (476/779) Updating files: 62% (483/779) Updating files: 63% (491/779) Updating files: 64% (499/779) Updating files: 65% (507/779) Updating files: 66% (515/779) Updating files: 67% (522/779) Updating files: 68% (530/779) Updating files: 69% (538/779) Updating files: 70% (546/779) Updating files: 71% (554/779) Updating files: 72% (561/779) Updating files: 73% (569/779) Updating files: 74% (577/779) Updating files: 75% (585/779) Updating files: 76% (593/779) Updating files: 77% (600/779) Updating files: 78% (608/779) Updating files: 79% (616/779) Updating files: 80% (624/779) Updating files: 81% (631/779) Updating files: 82% (639/779) Updating files: 83% (647/779) Updating files: 84% (655/779) Updating files: 85% (663/779) Updating files: 86% (670/779) Updating files: 87% (678/779) Updating files: 88% (686/779) Updating files: 89% (694/779) Updating files: 90% (702/779) Updating files: 91% (709/779) Updating files: 92% (717/779) Updating files: 93% (725/779) Updating files: 94% (733/779) Updating files: 95% (741/779) Updating files: 96% (748/779) Updating files: 97% (756/779) Updating files: 98% (764/779) Updating files: 99% (772/779) Updating files: 100% (779/779) Updating files: 100% (779/779), done. 2023-06-22T08:09:38.134 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 2f10137986a v0.18 2023-06-22T08:09:38.139 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.18 2023-06-22T08:09:38.284 INFO:tasks.workunit.client.0.smithi023.stdout:v0.19 2023-06-22T08:09:38.284 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:38.284 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 19 -eq 48 ']' 2023-06-22T08:09:38.285 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.19 2023-06-22T08:09:38.285 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.19 2023-06-22T08:09:38.285 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.19 2023-06-22T08:09:38.403 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:09:38.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:09:39.391 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 98f5be53f85 v0.19 2023-06-22T08:09:39.395 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.19 2023-06-22T08:09:39.558 INFO:tasks.workunit.client.0.smithi023.stdout:v0.20 2023-06-22T08:09:39.559 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:39.559 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 20 -eq 48 ']' 2023-06-22T08:09:39.559 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.20 2023-06-22T08:09:39.559 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.20 2023-06-22T08:09:39.559 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.20 2023-06-22T08:09:40.859 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 266d2076b6c debian: fix up debian scripts 2023-06-22T08:09:40.862 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.20 2023-06-22T08:09:41.060 INFO:tasks.workunit.client.0.smithi023.stdout:v0.21 2023-06-22T08:09:41.060 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:41.061 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 21 -eq 48 ']' 2023-06-22T08:09:41.061 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.21 2023-06-22T08:09:41.061 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.21 2023-06-22T08:09:41.061 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.21 2023-06-22T08:09:43.598 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 39% (210/537) Updating files: 40% (215/537) Updating files: 41% (221/537) Updating files: 42% (226/537) Updating files: 43% (231/537) Updating files: 44% (237/537) Updating files: 45% (242/537) Updating files: 46% (248/537) Updating files: 47% (253/537) Updating files: 48% (258/537) Updating files: 49% (264/537) Updating files: 50% (269/537) Updating files: 51% (274/537) Updating files: 52% (280/537) Updating files: 53% (285/537) Updating files: 54% (290/537) Updating files: 55% (296/537) Updating files: 56% (301/537) Updating files: 57% (307/537) Updating files: 58% (312/537) Updating files: 59% (317/537) Updating files: 60% (323/537) Updating files: 61% (328/537) Updating files: 62% (333/537) Updating files: 63% (339/537) Updating files: 64% (344/537) Updating files: 65% (350/537) Updating files: 66% (355/537) Updating files: 67% (360/537) Updating files: 68% (366/537) Updating files: 69% (371/537) Updating files: 70% (376/537) Updating files: 71% (382/537) Updating files: 72% (387/537) Updating files: 73% (393/537) Updating files: 74% (398/537) Updating files: 75% (403/537) Updating files: 76% (409/537) Updating files: 77% (414/537) Updating files: 78% (419/537) Updating files: 79% (425/537) Updating files: 80% (430/537) Updating files: 81% (435/537) Updating files: 82% (441/537) Updating files: 83% (446/537) Updating files: 84% (452/537) Updating files: 85% (457/537) Updating files: 86% (462/537) Updating files: 87% (468/537) Updating files: 88% (473/537) Updating files: 89% (478/537) Updating files: 90% (484/537) Updating files: 91% (489/537) Updating files: 92% (495/537) Updating files: 93% (500/537) Updating files: 94% (505/537) Updating files: 95% (511/537) Updating files: 96% (516/537) Updating files: 97% (521/537) Updating files: 98% (527/537) Updating files: 99% (532/537) Updating files: 100% (537/537) Updating files: 100% (537/537), done. 2023-06-22T08:09:43.694 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 090436f56ac v0.21 2023-06-22T08:09:43.701 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.21 2023-06-22T08:09:43.787 INFO:tasks.workunit.client.0.smithi023.stdout:v0.22 2023-06-22T08:09:43.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:43.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 22 -eq 48 ']' 2023-06-22T08:09:43.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.22 2023-06-22T08:09:43.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.22 2023-06-22T08:09:43.789 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.22 2023-06-22T08:09:44.913 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 8a7c95f60ad v0.22 2023-06-22T08:09:44.919 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.22 2023-06-22T08:09:44.940 INFO:tasks.workunit.client.0.smithi023.stdout:v0.23 2023-06-22T08:09:44.941 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:44.941 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 23 -eq 48 ']' 2023-06-22T08:09:44.941 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.23 2023-06-22T08:09:44.941 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.23 2023-06-22T08:09:44.941 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.23 2023-06-22T08:09:57.930 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 14% (23/163) Updating files: 15% (25/163) Updating files: 16% (27/163) Updating files: 17% (28/163) Updating files: 18% (30/163) Updating files: 19% (31/163) Updating files: 20% (33/163) Updating files: 21% (35/163) Updating files: 22% (36/163) Updating files: 23% (38/163) Updating files: 24% (40/163) Updating files: 25% (41/163) Updating files: 25% (42/163) Updating files: 26% (43/163) Updating files: 27% (45/163) Updating files: 28% (46/163) Updating files: 29% (48/163) Updating files: 30% (49/163) Updating files: 31% (51/163) Updating files: 32% (53/163) Updating files: 33% (54/163) Updating files: 34% (56/163) Updating files: 35% (58/163) Updating files: 36% (59/163) Updating files: 37% (61/163) Updating files: 38% (62/163) Updating files: 39% (64/163) Updating files: 40% (66/163) Updating files: 41% (67/163) Updating files: 42% (69/163) Updating files: 43% (71/163) Updating files: 44% (72/163) Updating files: 45% (74/163) Updating files: 46% (75/163) Updating files: 47% (77/163) Updating files: 48% (79/163) Updating files: 49% (80/163) Updating files: 50% (82/163) Updating files: 51% (84/163) Updating files: 52% (85/163) Updating files: 53% (87/163) Updating files: 54% (89/163) Updating files: 55% (90/163) Updating files: 56% (92/163) Updating files: 57% (93/163) Updating files: 58% (95/163) Updating files: 59% (97/163) Updating files: 60% (98/163) Updating files: 61% (100/163) Updating files: 62% (102/163) Updating files: 63% (103/163) Updating files: 64% (105/163) Updating files: 65% (106/163) Updating files: 66% (108/163) Updating files: 67% (110/163) Updating files: 68% (111/163) Updating files: 69% (113/163) Updating files: 70% (115/163) Updating files: 71% (116/163) Updating files: 72% (118/163) Updating files: 73% (119/163) Updating files: 74% (121/163) Updating files: 75% (123/163) Updating files: 76% (124/163) Updating files: 77% (126/163) Updating files: 78% (128/163) Updating files: 79% (129/163) Updating files: 80% (131/163) Updating files: 81% (133/163) Updating files: 82% (134/163) Updating files: 83% (136/163) Updating files: 84% (137/163) Updating files: 85% (139/163) Updating files: 86% (141/163) Updating files: 87% (142/163) Updating files: 88% (144/163) Updating files: 89% (146/163) Updating files: 90% (147/163) Updating files: 91% (149/163) Updating files: 92% (150/163) Updating files: 93% (152/163) Updating files: 94% (154/163) Updating files: 95% (155/163) Updating files: 96% (157/163) Updating files: 97% (159/163) Updating files: 98% (160/163) Updating files: 99% (162/163) Updating files: 100% (163/163) Updating files: 100% (163/163), done. 2023-06-22T08:09:58.001 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 5d1d8d0c460 v0.23 2023-06-22T08:09:58.006 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.23 2023-06-22T08:09:58.134 INFO:tasks.workunit.client.0.smithi023.stdout:v0.24 2023-06-22T08:09:58.135 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:58.135 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 24 -eq 48 ']' 2023-06-22T08:09:58.135 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.24 2023-06-22T08:09:58.136 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.24 2023-06-22T08:09:58.136 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.24 2023-06-22T08:09:59.270 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 180a4176035 v0.24 2023-06-22T08:09:59.274 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.24 2023-06-22T08:09:59.392 INFO:tasks.workunit.client.0.smithi023.stdout:v0.25 2023-06-22T08:09:59.392 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:09:59.393 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 25 -eq 48 ']' 2023-06-22T08:09:59.393 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.25 2023-06-22T08:09:59.393 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.25 2023-06-22T08:09:59.393 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.25 2023-06-22T08:10:01.359 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 56% (351/622) Updating files: 57% (355/622) Updating files: 58% (361/622) Updating files: 59% (367/622) Updating files: 60% (374/622) Updating files: 61% (380/622) Updating files: 62% (386/622) Updating files: 63% (392/622) Updating files: 64% (399/622) Updating files: 65% (405/622) Updating files: 66% (411/622) Updating files: 67% (417/622) Updating files: 68% (423/622) Updating files: 69% (430/622) Updating files: 70% (436/622) Updating files: 71% (442/622) Updating files: 72% (448/622) Updating files: 73% (455/622) Updating files: 74% (461/622) Updating files: 75% (467/622) Updating files: 76% (473/622) Updating files: 77% (479/622) Updating files: 78% (486/622) Updating files: 79% (492/622) Updating files: 80% (498/622) Updating files: 81% (504/622) Updating files: 82% (511/622) Updating files: 83% (517/622) Updating files: 84% (523/622) Updating files: 85% (529/622) Updating files: 86% (535/622) Updating files: 87% (542/622) Updating files: 88% (548/622) Updating files: 89% (554/622) Updating files: 90% (560/622) Updating files: 91% (567/622) Updating files: 92% (573/622) Updating files: 93% (579/622) Updating files: 94% (585/622) Updating files: 95% (591/622) Updating files: 96% (598/622) Updating files: 97% (604/622) Updating files: 98% (610/622) Updating files: 99% (616/622) Updating files: 100% (622/622) Updating files: 100% (622/622), done. 2023-06-22T08:10:01.524 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 3fa121bf0a8 v0.25 2023-06-22T08:10:01.529 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.25 2023-06-22T08:10:01.684 INFO:tasks.workunit.client.0.smithi023.stdout:v0.26 2023-06-22T08:10:01.685 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:01.685 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 26 -eq 48 ']' 2023-06-22T08:10:01.685 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.26 2023-06-22T08:10:01.686 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.26 2023-06-22T08:10:01.686 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.26 2023-06-22T08:10:02.873 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 9981ff90968 v0.26 2023-06-22T08:10:02.878 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.26 2023-06-22T08:10:02.936 INFO:tasks.workunit.client.0.smithi023.stdout:v0.27 2023-06-22T08:10:02.936 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:02.937 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 27 -eq 48 ']' 2023-06-22T08:10:02.937 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.27 2023-06-22T08:10:02.937 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.27 2023-06-22T08:10:02.937 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.27 2023-06-22T08:10:08.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:10:08.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:10:21.797 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 37% (95/253) Updating files: 38% (97/253) Updating files: 39% (99/253) Updating files: 40% (102/253) Updating files: 41% (104/253) Updating files: 42% (107/253) Updating files: 43% (109/253) Updating files: 44% (112/253) Updating files: 45% (114/253) Updating files: 46% (117/253) Updating files: 47% (119/253) Updating files: 48% (122/253) Updating files: 49% (124/253) Updating files: 50% (127/253) Updating files: 51% (130/253) Updating files: 52% (132/253) Updating files: 53% (135/253) Updating files: 54% (137/253) Updating files: 55% (140/253) Updating files: 56% (142/253) Updating files: 57% (145/253) Updating files: 58% (147/253) Updating files: 59% (150/253) Updating files: 60% (152/253) Updating files: 61% (155/253) Updating files: 62% (157/253) Updating files: 63% (160/253) Updating files: 64% (162/253) Updating files: 65% (165/253) Updating files: 66% (167/253) Updating files: 67% (170/253) Updating files: 68% (173/253) Updating files: 69% (175/253) Updating files: 70% (178/253) Updating files: 71% (180/253) Updating files: 72% (183/253) Updating files: 73% (185/253) Updating files: 74% (188/253) Updating files: 75% (190/253) Updating files: 76% (193/253) Updating files: 77% (195/253) Updating files: 77% (196/253) Updating files: 78% (198/253) Updating files: 79% (200/253) Updating files: 80% (203/253) Updating files: 81% (205/253) Updating files: 82% (208/253) Updating files: 83% (210/253) Updating files: 84% (213/253) Updating files: 85% (216/253) Updating files: 86% (218/253) Updating files: 87% (221/253) Updating files: 88% (223/253) Updating files: 89% (226/253) Updating files: 90% (228/253) Updating files: 91% (231/253) Updating files: 92% (233/253) Updating files: 93% (236/253) Updating files: 94% (238/253) Updating files: 95% (241/253) Updating files: 96% (243/253) Updating files: 97% (246/253) Updating files: 98% (248/253) Updating files: 99% (251/253) Updating files: 100% (253/253) Updating files: 100% (253/253), done. 2023-06-22T08:10:21.872 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 793034c62c8 v0.27 2023-06-22T08:10:21.878 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.27 2023-06-22T08:10:21.926 INFO:tasks.workunit.client.0.smithi023.stdout:v0.28 2023-06-22T08:10:21.926 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:21.927 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 28 -eq 48 ']' 2023-06-22T08:10:21.927 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.28 2023-06-22T08:10:21.927 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.28 2023-06-22T08:10:21.927 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.28 2023-06-22T08:10:23.177 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 96% (219/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2023-06-22T08:10:23.273 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 071881d7e55 v0.28 2023-06-22T08:10:23.278 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.28 2023-06-22T08:10:23.355 INFO:tasks.workunit.client.0.smithi023.stdout:v0.29 2023-06-22T08:10:23.356 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:23.356 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 29 -eq 48 ']' 2023-06-22T08:10:23.356 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.29 2023-06-22T08:10:23.356 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.29 2023-06-22T08:10:23.356 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.29 2023-06-22T08:10:24.479 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 8e69c39f699 v0.29 2023-06-22T08:10:24.484 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.29 2023-06-22T08:10:24.570 INFO:tasks.workunit.client.0.smithi023.stdout:v0.30 2023-06-22T08:10:24.570 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:24.571 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 30 -eq 48 ']' 2023-06-22T08:10:24.571 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.30 2023-06-22T08:10:24.571 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.30 2023-06-22T08:10:24.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.30 2023-06-22T08:10:25.880 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 94% (268/285) Updating files: 95% (271/285) Updating files: 96% (274/285) Updating files: 97% (277/285) Updating files: 98% (280/285) Updating files: 99% (283/285) Updating files: 100% (285/285) Updating files: 100% (285/285), done. 2023-06-22T08:10:25.954 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 64b1b2c70f0 v0.30 2023-06-22T08:10:25.958 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.30 2023-06-22T08:10:26.073 INFO:tasks.workunit.client.0.smithi023.stdout:v0.31 2023-06-22T08:10:26.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:26.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 31 -eq 48 ']' 2023-06-22T08:10:26.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.31 2023-06-22T08:10:26.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.31 2023-06-22T08:10:26.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.31 2023-06-22T08:10:27.815 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 56% (224/398) Updating files: 57% (227/398) Updating files: 58% (231/398) Updating files: 59% (235/398) Updating files: 60% (239/398) Updating files: 61% (243/398) Updating files: 62% (247/398) Updating files: 63% (251/398) Updating files: 64% (255/398) Updating files: 65% (259/398) Updating files: 66% (263/398) Updating files: 67% (267/398) Updating files: 68% (271/398) Updating files: 69% (275/398) Updating files: 70% (279/398) Updating files: 71% (283/398) Updating files: 72% (287/398) Updating files: 73% (291/398) Updating files: 74% (295/398) Updating files: 75% (299/398) Updating files: 76% (303/398) Updating files: 77% (307/398) Updating files: 78% (311/398) Updating files: 79% (315/398) Updating files: 80% (319/398) Updating files: 81% (323/398) Updating files: 82% (327/398) Updating files: 83% (331/398) Updating files: 84% (335/398) Updating files: 85% (339/398) Updating files: 86% (343/398) Updating files: 87% (347/398) Updating files: 88% (351/398) Updating files: 89% (355/398) Updating files: 90% (359/398) Updating files: 91% (363/398) Updating files: 92% (367/398) Updating files: 93% (371/398) Updating files: 94% (375/398) Updating files: 95% (379/398) Updating files: 96% (383/398) Updating files: 97% (387/398) Updating files: 98% (391/398) Updating files: 99% (395/398) Updating files: 100% (398/398) Updating files: 100% (398/398), done. 2023-06-22T08:10:27.977 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 9019c6ce640 v0.31 2023-06-22T08:10:27.981 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.31 2023-06-22T08:10:28.062 INFO:tasks.workunit.client.0.smithi023.stdout:v0.32 2023-06-22T08:10:28.062 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:28.062 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 32 -eq 48 ']' 2023-06-22T08:10:28.062 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.32 2023-06-22T08:10:28.063 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.32 2023-06-22T08:10:28.063 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.32 2023-06-22T08:10:38.637 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:10:38.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:10:40.532 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 7% (14/195) Updating files: 7% (15/195) Updating files: 8% (16/195) Updating files: 9% (18/195) Updating files: 10% (20/195) Updating files: 11% (22/195) Updating files: 12% (24/195) Updating files: 13% (26/195) Updating files: 14% (28/195) Updating files: 15% (30/195) Updating files: 16% (32/195) Updating files: 17% (34/195) Updating files: 18% (36/195) Updating files: 19% (38/195) Updating files: 20% (39/195) Updating files: 21% (41/195) Updating files: 22% (43/195) Updating files: 23% (45/195) Updating files: 24% (47/195) Updating files: 25% (49/195) Updating files: 26% (51/195) Updating files: 27% (53/195) Updating files: 28% (55/195) Updating files: 29% (57/195) Updating files: 30% (59/195) Updating files: 31% (61/195) Updating files: 32% (63/195) Updating files: 33% (65/195) Updating files: 34% (67/195) Updating files: 35% (69/195) Updating files: 36% (71/195) Updating files: 37% (73/195) Updating files: 38% (75/195) Updating files: 39% (77/195) Updating files: 40% (78/195) Updating files: 41% (80/195) Updating files: 42% (82/195) Updating files: 43% (84/195) Updating files: 44% (86/195) Updating files: 45% (88/195) Updating files: 46% (90/195) Updating files: 47% (92/195) Updating files: 48% (94/195) Updating files: 49% (96/195) Updating files: 50% (98/195) Updating files: 51% (100/195) Updating files: 52% (102/195) Updating files: 53% (104/195) Updating files: 54% (106/195) Updating files: 55% (108/195) Updating files: 56% (110/195) Updating files: 57% (112/195) Updating files: 58% (114/195) Updating files: 59% (116/195) Updating files: 60% (117/195) Updating files: 61% (119/195) Updating files: 62% (121/195) Updating files: 63% (123/195) Updating files: 64% (125/195) Updating files: 65% (127/195) Updating files: 66% (129/195) Updating files: 67% (131/195) Updating files: 68% (133/195) Updating files: 69% (135/195) Updating files: 70% (137/195) Updating files: 71% (139/195) Updating files: 72% (141/195) Updating files: 73% (143/195) Updating files: 74% (145/195) Updating files: 75% (147/195) Updating files: 76% (149/195) Updating files: 77% (151/195) Updating files: 78% (153/195) Updating files: 79% (155/195) Updating files: 80% (156/195) Updating files: 81% (158/195) Updating files: 82% (160/195) Updating files: 83% (162/195) Updating files: 84% (164/195) Updating files: 85% (166/195) Updating files: 86% (168/195) Updating files: 87% (170/195) Updating files: 88% (172/195) Updating files: 89% (174/195) Updating files: 90% (176/195) Updating files: 91% (178/195) Updating files: 92% (180/195) Updating files: 93% (182/195) Updating files: 94% (184/195) Updating files: 95% (186/195) Updating files: 96% (188/195) Updating files: 97% (190/195) Updating files: 98% (192/195) Updating files: 99% (194/195) Updating files: 100% (195/195) Updating files: 100% (195/195), done. 2023-06-22T08:10:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at c08d08baa6a v0.32 2023-06-22T08:10:40.649 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.32 2023-06-22T08:10:40.765 INFO:tasks.workunit.client.0.smithi023.stdout:v0.33 2023-06-22T08:10:40.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:40.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 33 -eq 48 ']' 2023-06-22T08:10:40.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.33 2023-06-22T08:10:40.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.33 2023-06-22T08:10:40.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.33 2023-06-22T08:10:42.060 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 340b4348bf6 v0.33 2023-06-22T08:10:42.063 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.33 2023-06-22T08:10:42.135 INFO:tasks.workunit.client.0.smithi023.stdout:v0.34 2023-06-22T08:10:42.136 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:42.136 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 34 -eq 48 ']' 2023-06-22T08:10:42.136 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.34 2023-06-22T08:10:42.136 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.34 2023-06-22T08:10:42.136 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.34 2023-06-22T08:10:43.041 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 2f039eeeb74 v0.34 2023-06-22T08:10:43.044 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.34 2023-06-22T08:10:43.138 INFO:tasks.workunit.client.0.smithi023.stdout:v0.35 2023-06-22T08:10:43.140 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:43.140 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 35 -eq 48 ']' 2023-06-22T08:10:43.140 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.35 2023-06-22T08:10:43.141 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.35 2023-06-22T08:10:43.141 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.35 2023-06-22T08:10:44.454 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 94% (315/334) Updating files: 95% (318/334) Updating files: 96% (321/334) Updating files: 97% (324/334) Updating files: 98% (328/334) Updating files: 99% (331/334) Updating files: 100% (334/334) Updating files: 100% (334/334), done. 2023-06-22T08:10:44.552 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 0afda378124 v0.35 2023-06-22T08:10:44.555 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.35 2023-06-22T08:10:44.650 INFO:tasks.workunit.client.0.smithi023.stdout:v0.36 2023-06-22T08:10:44.651 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:44.651 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 36 -eq 48 ']' 2023-06-22T08:10:44.651 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.36 2023-06-22T08:10:44.651 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.36 2023-06-22T08:10:44.651 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.36 2023-06-22T08:10:45.826 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 877cacbbdbb v0.36 2023-06-22T08:10:45.828 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.36 2023-06-22T08:10:45.867 INFO:tasks.workunit.client.0.smithi023.stdout:v0.37 2023-06-22T08:10:45.867 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:10:45.868 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 37 -eq 48 ']' 2023-06-22T08:10:45.868 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.37 2023-06-22T08:10:45.868 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.37 2023-06-22T08:10:45.868 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.37 2023-06-22T08:10:49.023 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:10:49.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.229+0000 7ff7049ff700 1 Processor -- start 2023-06-22T08:10:49.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.229+0000 7ff7049ff700 1 -- start start 2023-06-22T08:10:49.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff7049ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 0x7ff7000c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff7049ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff7000c69e0 0x7ff7000c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff7049ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7000c51a0 con 0x7ff700071070 2023-06-22T08:10:49.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff7049ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7000c44e0 con 0x7ff7000c69e0 2023-06-22T08:10:49.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff7049ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7000c5440 con 0x7ff70014f850 2023-06-22T08:10:49.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff6fe59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 0x7ff7000c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff6fdd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff7000c69e0 0x7ff7000c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff6fed9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff700071070 0x7ff700155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40610/0 (socket says 172.21.15.23:40610) 2023-06-22T08:10:49.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff6fed9d700 1 -- 172.21.15.23:0/1504088568 learned_addr learned my addr 172.21.15.23:0/1504088568 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:10:49.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff6fdd9b700 1 -- 172.21.15.23:0/1504088568 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 msgr2=0x7ff7000c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:10:49.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff6fdd9b700 1 --2- 172.21.15.23:0/1504088568 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 0x7ff7000c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff6fdd9b700 1 -- 172.21.15.23:0/1504088568 >> v1:172.21.15.23:6789/0 conn(0x7ff700071070 legacy=0x7ff700155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:49.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.231+0000 7ff6fdd9b700 1 -- 172.21.15.23:0/1504088568 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7000c5720 con 0x7ff7000c69e0 2023-06-22T08:10:49.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.232+0000 7ff6fdd9b700 1 --2- 172.21.15.23:0/1504088568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7000c69e0 0x7ff7000c35c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009840 tx=0x7ff6f4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4a7b1daa8b1111 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:49.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.232+0000 7ff6fd59a700 1 -- 172.21.15.23:0/1504088568 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4011830 con 0x7ff7000c69e0 2023-06-22T08:10:49.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.232+0000 7ff6fd59a700 1 -- 172.21.15.23:0/1504088568 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f4011990 con 0x7ff7000c69e0 2023-06-22T08:10:49.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.232+0000 7ff6fd59a700 1 -- 172.21.15.23:0/1504088568 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4011d20 con 0x7ff7000c69e0 2023-06-22T08:10:49.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.232+0000 7ff7049ff700 1 -- 172.21.15.23:0/1504088568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7000c69e0 msgr2=0x7ff7000c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:49.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.232+0000 7ff7049ff700 1 --2- 172.21.15.23:0/1504088568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7000c69e0 0x7ff7000c35c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009840 tx=0x7ff6f4009b50 comp rx=0 tx=0).stop 2023-06-22T08:10:49.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.233+0000 7ff7049ff700 1 -- 172.21.15.23:0/1504088568 shutdown_connections 2023-06-22T08:10:49.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.233+0000 7ff7049ff700 1 --2- 172.21.15.23:0/1504088568 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 0x7ff7000c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.233+0000 7ff7049ff700 1 --2- 172.21.15.23:0/1504088568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7000c69e0 0x7ff7000c35c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.233+0000 7ff7049ff700 1 -- 172.21.15.23:0/1504088568 >> 172.21.15.23:0/1504088568 conn(0x7ff7000ca3f0 msgr2=0x7ff7000ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:49.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.233+0000 7ff7049ff700 1 -- 172.21.15.23:0/1504088568 shutdown_connections 2023-06-22T08:10:49.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.233+0000 7ff7049ff700 1 -- 172.21.15.23:0/1504088568 wait complete. 2023-06-22T08:10:49.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.235+0000 7ff7049ff700 1 Processor -- start 2023-06-22T08:10:49.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.235+0000 7ff7049ff700 1 -- start start 2023-06-22T08:10:49.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff7049ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff700071070 0x7ff7000c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff7049ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7000c69e0 0x7ff700146b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff7049ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 0x7ff7001db900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff7049ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7000c8360 con 0x7ff700071070 2023-06-22T08:10:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff7049ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7001f0610 con 0x7ff7000c69e0 2023-06-22T08:10:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff7049ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7001dd510 con 0x7ff70014f850 2023-06-22T08:10:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fe59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff700071070 0x7ff7000c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fe59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff700071070 0x7ff7000c8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34790/0 (socket says 172.21.15.23:34790) 2023-06-22T08:10:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fe59c700 1 -- 172.21.15.23:0/3269104693 learned_addr learned my addr 172.21.15.23:0/3269104693 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:10:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fdd9b700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7000c69e0 0x7ff700146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fed9d700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 0x7ff7001db900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fe59c700 1 -- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 msgr2=0x7ff7001db900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fe59c700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 0x7ff7001db900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fe59c700 1 -- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7000c69e0 msgr2=0x7ff700146b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fe59c700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7000c69e0 0x7ff700146b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fe59c700 1 -- 172.21.15.23:0/3269104693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f40094a0 con 0x7ff700071070 2023-06-22T08:10:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.236+0000 7ff6fed9d700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 0x7ff7001db900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T08:10:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.237+0000 7ff6fe59c700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff700071070 0x7ff7000c8dd0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff6f000e870 tx=0x7ff6f000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.237+0000 7ff6eeffd700 1 -- 172.21.15.23:0/3269104693 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0009590 con 0x7ff700071070 2023-06-22T08:10:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.237+0000 7ff6eeffd700 1 -- 172.21.15.23:0/3269104693 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f000ee70 con 0x7ff700071070 2023-06-22T08:10:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.237+0000 7ff6eeffd700 1 -- 172.21.15.23:0/3269104693 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f000ca50 con 0x7ff700071070 2023-06-22T08:10:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.237+0000 7ff7049ff700 1 -- 172.21.15.23:0/3269104693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7001ef850 con 0x7ff700071070 2023-06-22T08:10:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.237+0000 7ff7049ff700 1 -- 172.21.15.23:0/3269104693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7001efe10 con 0x7ff700071070 2023-06-22T08:10:49.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.238+0000 7ff7049ff700 1 -- 172.21.15.23:0/3269104693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7000c2950 con 0x7ff700071070 2023-06-22T08:10:49.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.239+0000 7ff6eeffd700 1 -- 172.21.15.23:0/3269104693 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff6f000cbb0 con 0x7ff700071070 2023-06-22T08:10:49.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.240+0000 7ff6eeffd700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff6e006ace0 0x7ff6e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.240+0000 7ff6eeffd700 1 -- 172.21.15.23:0/3269104693 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7ff6f0089900 con 0x7ff700071070 2023-06-22T08:10:49.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.240+0000 7ff6fdd9b700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff6e006ace0 0x7ff6e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.241+0000 7ff6fdd9b700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff6e006ace0 0x7ff6e006d190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff7000c6db0 tx=0x7ff6f4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:49.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.245+0000 7ff6eeffd700 1 -- 172.21.15.23:0/3269104693 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff6f0057f90 con 0x7ff700071070 2023-06-22T08:10:49.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.495+0000 7ff7049ff700 1 -- 172.21.15.23:0/3269104693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7000742e0 con 0x7ff700071070 2023-06-22T08:10:49.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.496+0000 7ff6eeffd700 1 -- 172.21.15.23:0/3269104693 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6267 (secure 0 0 0) 0x7ff6f0021090 con 0x7ff700071070 2023-06-22T08:10:49.500 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:10:49.500 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:10:46.771362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:10:49.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 -- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff6e006ace0 msgr2=0x7ff6e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:49.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff6e006ace0 0x7ff6e006d190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff7000c6db0 tx=0x7ff6f4008030 comp rx=0 tx=0).stop 2023-06-22T08:10:49.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 -- 172.21.15.23:0/3269104693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff700071070 msgr2=0x7ff7000c8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:49.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff700071070 0x7ff7000c8dd0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff6f000e870 tx=0x7ff6f000ec30 comp rx=0 tx=0).stop 2023-06-22T08:10:49.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 -- 172.21.15.23:0/3269104693 shutdown_connections 2023-06-22T08:10:49.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff70014f850 0x7ff7001db900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff700071070 0x7ff7000c8dd0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff6e006ace0 0x7ff6e006d190 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 --2- 172.21.15.23:0/3269104693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7000c69e0 0x7ff700146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 -- 172.21.15.23:0/3269104693 >> 172.21.15.23:0/3269104693 conn(0x7ff7000ca3f0 msgr2=0x7ff7001de360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:49.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 -- 172.21.15.23:0/3269104693 shutdown_connections 2023-06-22T08:10:49.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.498+0000 7ff7049ff700 1 -- 172.21.15.23:0/3269104693 wait complete. 2023-06-22T08:10:49.503 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 85 2023-06-22T08:10:49.520 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-06-22T08:10:49.520 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-06-22T08:10:49.520 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:10:49.536 INFO:tasks.ceph.mds.f:Stopped 2023-06-22T08:10:49.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:10:49.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.733+0000 7f7a0bbff700 1 Processor -- start 2023-06-22T08:10:49.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.733+0000 7f7a0bbff700 1 -- start start 2023-06-22T08:10:49.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.734+0000 7f7a0bbff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a04152ce0 0x7f7a040cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.734+0000 7f7a0bbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a040c9fd0 0x7f7a0415c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.734+0000 7f7a0bbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0415c370 con 0x7f7a04158880 2023-06-22T08:10:49.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.734+0000 7f7a0bbff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a041530b0 con 0x7f7a04152ce0 2023-06-22T08:10:49.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.734+0000 7f7a0bbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040cf6f0 con 0x7f7a040c9fd0 2023-06-22T08:10:49.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.735+0000 7f7a0999b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a04152ce0 0x7f7a040cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.735+0000 7f7a0919a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a040c9fd0 0x7f7a0415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.735+0000 7f7a0999b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a04152ce0 0x7f7a040cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:43832/0 (socket says 172.21.15.23:43832) 2023-06-22T08:10:49.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.735+0000 7f7a0999b700 1 -- 172.21.15.23:0/1304079569 learned_addr learned my addr 172.21.15.23:0/1304079569 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:10:49.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.735+0000 7f7a0919a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a040c9fd0 0x7f7a0415c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:43806/0 (socket says 172.21.15.23:43806) 2023-06-22T08:10:49.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.735+0000 7f7a0919a700 1 -- 172.21.15.23:0/1304079569 >> v2:172.21.15.67:3300/0 conn(0x7f7a04152ce0 msgr2=0x7f7a040cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:10:49.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.735+0000 7f7a0919a700 1 --2- 172.21.15.23:0/1304079569 >> v2:172.21.15.67:3300/0 conn(0x7f7a04152ce0 0x7f7a040cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.735+0000 7f7a0919a700 1 -- 172.21.15.23:0/1304079569 >> v1:172.21.15.67:6789/0 conn(0x7f7a04158880 legacy=0x7f7a04158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:49.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.735+0000 7f7a0919a700 1 -- 172.21.15.23:0/1304079569 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0415bda0 con 0x7f7a040c9fd0 2023-06-22T08:10:49.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.735+0000 7f7a0999b700 1 --2- 172.21.15.23:0/1304079569 >> v2:172.21.15.67:3300/0 conn(0x7f7a04152ce0 0x7f7a040cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:10:49.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.736+0000 7f7a0919a700 1 --2- 172.21.15.23:0/1304079569 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a040c9fd0 0x7f7a0415c800 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009840 tx=0x7f79f4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c56184b5f535fb2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:49.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.736+0000 7f7a08999700 1 -- 172.21.15.23:0/1304079569 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4011830 con 0x7f7a040c9fd0 2023-06-22T08:10:49.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.736+0000 7f7a08999700 1 -- 172.21.15.23:0/1304079569 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f4011990 con 0x7f7a040c9fd0 2023-06-22T08:10:49.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.736+0000 7f7a08999700 1 -- 172.21.15.23:0/1304079569 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4011d20 con 0x7f7a040c9fd0 2023-06-22T08:10:49.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.736+0000 7f7a0bbff700 1 -- 172.21.15.23:0/1304079569 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a040c9fd0 msgr2=0x7f7a0415c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:49.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.736+0000 7f7a0bbff700 1 --2- 172.21.15.23:0/1304079569 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a040c9fd0 0x7f7a0415c800 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009840 tx=0x7f79f4009b50 comp rx=0 tx=0).stop 2023-06-22T08:10:49.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.737+0000 7f7a0bbff700 1 -- 172.21.15.23:0/1304079569 shutdown_connections 2023-06-22T08:10:49.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.737+0000 7f7a0bbff700 1 --2- 172.21.15.23:0/1304079569 >> v2:172.21.15.67:3300/0 conn(0x7f7a04152ce0 0x7f7a040cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.737+0000 7f7a0bbff700 1 --2- 172.21.15.23:0/1304079569 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a040c9fd0 0x7f7a0415c800 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.737+0000 7f7a0bbff700 1 -- 172.21.15.23:0/1304079569 >> 172.21.15.23:0/1304079569 conn(0x7f7a0406ae30 msgr2=0x7f7a0406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:49.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.737+0000 7f7a0bbff700 1 -- 172.21.15.23:0/1304079569 shutdown_connections 2023-06-22T08:10:49.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.737+0000 7f7a0bbff700 1 -- 172.21.15.23:0/1304079569 wait complete. 2023-06-22T08:10:49.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.738+0000 7f7a0bbff700 1 Processor -- start 2023-06-22T08:10:49.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.739+0000 7f7a0bbff700 1 -- start start 2023-06-22T08:10:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.739+0000 7f7a0bbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a040c9fd0 0x7f7a040c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.739+0000 7f7a0bbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a04152ce0 0x7f7a040c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.739+0000 7f7a0bbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a04158880 0x7f7a04148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.739+0000 7f7a0bbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040c7390 con 0x7f7a040c9fd0 2023-06-22T08:10:49.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.739+0000 7f7a0bbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040c6fa0 con 0x7f7a04158880 2023-06-22T08:10:49.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.739+0000 7f7a0bbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040c5d90 con 0x7f7a04152ce0 2023-06-22T08:10:49.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0999b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a040c9fd0 0x7f7a040c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0999b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a040c9fd0 0x7f7a040c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34796/0 (socket says 172.21.15.23:34796) 2023-06-22T08:10:49.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0999b700 1 -- 172.21.15.23:0/791504775 learned_addr learned my addr 172.21.15.23:0/791504775 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:10:49.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0919a700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a04152ce0 0x7f7a040c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0a19c700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a04158880 0x7f7a04148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0999b700 1 -- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a04152ce0 msgr2=0x7f7a040c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0999b700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a04152ce0 0x7f7a040c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0999b700 1 -- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a04158880 msgr2=0x7f7a04148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0999b700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a04158880 0x7f7a04148150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0999b700 1 -- 172.21.15.23:0/791504775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a00009550 con 0x7f7a040c9fd0 2023-06-22T08:10:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.740+0000 7f7a0999b700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a040c9fd0 0x7f7a040c6a60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000ebe0 tx=0x7f7a0000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:49.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.741+0000 7f79fa7fc700 1 -- 172.21.15.23:0/791504775 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0000c850 con 0x7f7a040c9fd0 2023-06-22T08:10:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.741+0000 7f79fa7fc700 1 -- 172.21.15.23:0/791504775 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0000c9b0 con 0x7f7a040c9fd0 2023-06-22T08:10:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.741+0000 7f7a0bbff700 1 -- 172.21.15.23:0/791504775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f40094a0 con 0x7f7a040c9fd0 2023-06-22T08:10:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.741+0000 7f79fa7fc700 1 -- 172.21.15.23:0/791504775 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0000cd40 con 0x7f7a040c9fd0 2023-06-22T08:10:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.741+0000 7f7a0bbff700 1 -- 172.21.15.23:0/791504775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a040cdb90 con 0x7f7a040c9fd0 2023-06-22T08:10:49.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.742+0000 7f7a0bbff700 1 -- 172.21.15.23:0/791504775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a04159ed0 con 0x7f7a040c9fd0 2023-06-22T08:10:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.743+0000 7f79fa7fc700 1 -- 172.21.15.23:0/791504775 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7a00013470 con 0x7f7a040c9fd0 2023-06-22T08:10:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.744+0000 7f79fa7fc700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f79e806ace0 0x7f79e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.744+0000 7f79fa7fc700 1 -- 172.21.15.23:0/791504775 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f7a00089e30 con 0x7f7a040c9fd0 2023-06-22T08:10:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.744+0000 7f7a0919a700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f79e806ace0 0x7f79e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.745+0000 7f7a0919a700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f79e806ace0 0x7f79e806d190 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f79f4000c00 tx=0x7f79f401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:49.748+0000 7f79fa7fc700 1 -- 172.21.15.23:0/791504775 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7a00054a40 con 0x7f7a040c9fd0 2023-06-22T08:10:50.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.009+0000 7f7a0bbff700 1 -- 172.21.15.23:0/791504775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a0404ea20 con 0x7f7a040c9fd0 2023-06-22T08:10:50.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.010+0000 7f79fa7fc700 1 -- 172.21.15.23:0/791504775 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6267 (secure 0 0 0) 0x7f7a00058060 con 0x7f7a040c9fd0 2023-06-22T08:10:50.014 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:10:50.014 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:10:46.771362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:10:50.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.012+0000 7f7a0bbff700 1 -- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f79e806ace0 msgr2=0x7f79e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:50.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.012+0000 7f7a0bbff700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f79e806ace0 0x7f79e806d190 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f79f4000c00 tx=0x7f79f401c040 comp rx=0 tx=0).stop 2023-06-22T08:10:50.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.012+0000 7f7a0bbff700 1 -- 172.21.15.23:0/791504775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a040c9fd0 msgr2=0x7f7a040c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:50.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.012+0000 7f7a0bbff700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a040c9fd0 0x7f7a040c6a60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000ebe0 tx=0x7f7a0000eef0 comp rx=0 tx=0).stop 2023-06-22T08:10:50.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.012+0000 7f7a0bbff700 1 -- 172.21.15.23:0/791504775 shutdown_connections 2023-06-22T08:10:50.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.012+0000 7f7a0bbff700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f79e806ace0 0x7f79e806d190 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:50.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.012+0000 7f7a0bbff700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a04158880 0x7f7a04148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:50.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.012+0000 7f7a0bbff700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a040c9fd0 0x7f7a040c6a60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:50.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.012+0000 7f7a0bbff700 1 --2- 172.21.15.23:0/791504775 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a04152ce0 0x7f7a040c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:50.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.012+0000 7f7a0bbff700 1 -- 172.21.15.23:0/791504775 >> 172.21.15.23:0/791504775 conn(0x7f7a0406ae30 msgr2=0x7f7a0406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:50.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.013+0000 7f7a0bbff700 1 -- 172.21.15.23:0/791504775 shutdown_connections 2023-06-22T08:10:50.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:50.013+0000 7f7a0bbff700 1 -- 172.21.15.23:0/791504775 wait complete. 2023-06-22T08:10:50.018 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 85 2023-06-22T08:10:50.034 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:10:52.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:10:52.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.244+0000 7f440abff700 1 Processor -- start 2023-06-22T08:10:52.248 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.245+0000 7f440abff700 1 -- start start 2023-06-22T08:10:52.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.246+0000 7f440abff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4404152ce0 0x7f44040cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.246+0000 7f440abff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f440415c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.246+0000 7f440abff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440415c370 con 0x7f4404158880 2023-06-22T08:10:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.246+0000 7f440abff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44041530b0 con 0x7f4404152ce0 2023-06-22T08:10:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.246+0000 7f440abff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44040cf6f0 con 0x7f44040c9fd0 2023-06-22T08:10:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.246+0000 7f440919c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4404158880 0x7f4404158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40626/0 (socket says 172.21.15.23:40626) 2023-06-22T08:10:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.246+0000 7f440919c700 1 -- 172.21.15.23:0/737960694 learned_addr learned my addr 172.21.15.23:0/737960694 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:10:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.246+0000 7f440899b700 1 --2- 172.21.15.23:0/737960694 >> v2:172.21.15.67:3300/0 conn(0x7f4404152ce0 0x7f44040cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:52.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.246+0000 7f4403fff700 1 --2- 172.21.15.23:0/737960694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f440415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:52.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f44037fe700 1 -- 172.21.15.23:0/737960694 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684638135 0 0) 0x7f440415c370 con 0x7f4404158880 2023-06-22T08:10:52.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f44037fe700 1 -- 172.21.15.23:0/737960694 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f40035b0 con 0x7f4404158880 2023-06-22T08:10:52.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f440899b700 1 -- 172.21.15.23:0/737960694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 msgr2=0x7f440415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:10:52.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f440899b700 1 --2- 172.21.15.23:0/737960694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f440415c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:52.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f440899b700 1 -- 172.21.15.23:0/737960694 >> v1:172.21.15.23:6789/0 conn(0x7f4404158880 legacy=0x7f4404158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f440899b700 1 -- 172.21.15.23:0/737960694 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440415bda0 con 0x7f4404152ce0 2023-06-22T08:10:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f4403fff700 1 --2- 172.21.15.23:0/737960694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f440415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:10:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f440899b700 1 --2- 172.21.15.23:0/737960694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4404152ce0 0x7f44040cf1b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009840 tx=0x7f43f8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9026edac16e21720 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f44037fe700 1 -- 172.21.15.23:0/737960694 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8011830 con 0x7f4404152ce0 2023-06-22T08:10:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f44037fe700 1 -- 172.21.15.23:0/737960694 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f8011990 con 0x7f4404152ce0 2023-06-22T08:10:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.247+0000 7f44037fe700 1 -- 172.21.15.23:0/737960694 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8011d20 con 0x7f4404152ce0 2023-06-22T08:10:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.248+0000 7f440abff700 1 -- 172.21.15.23:0/737960694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4404152ce0 msgr2=0x7f44040cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.248+0000 7f440abff700 1 --2- 172.21.15.23:0/737960694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4404152ce0 0x7f44040cf1b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009840 tx=0x7f43f8009b50 comp rx=0 tx=0).stop 2023-06-22T08:10:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.248+0000 7f440abff700 1 -- 172.21.15.23:0/737960694 shutdown_connections 2023-06-22T08:10:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.248+0000 7f440abff700 1 --2- 172.21.15.23:0/737960694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4404152ce0 0x7f44040cf1b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.248+0000 7f440abff700 1 --2- 172.21.15.23:0/737960694 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f440415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.248+0000 7f440abff700 1 -- 172.21.15.23:0/737960694 >> 172.21.15.23:0/737960694 conn(0x7f440406ae30 msgr2=0x7f440406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.248+0000 7f440abff700 1 -- 172.21.15.23:0/737960694 shutdown_connections 2023-06-22T08:10:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.248+0000 7f440abff700 1 -- 172.21.15.23:0/737960694 wait complete. 2023-06-22T08:10:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.250+0000 7f440abff700 1 Processor -- start 2023-06-22T08:10:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.251+0000 7f440abff700 1 -- start start 2023-06-22T08:10:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f440abff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f44041477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f440abff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4404152ce0 0x7f4404150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f440abff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4404158880 0x7f4404151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f440abff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44041eeec0 con 0x7f4404152ce0 2023-06-22T08:10:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f440abff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44041ee010 con 0x7f4404158880 2023-06-22T08:10:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f440abff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44040ccac0 con 0x7f44040c9fd0 2023-06-22T08:10:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f440899b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f44041477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f440899b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f44041477c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:43828/0 (socket says 172.21.15.23:43828) 2023-06-22T08:10:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f440899b700 1 -- 172.21.15.23:0/3105110348 learned_addr learned my addr 172.21.15.23:0/3105110348 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:10:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f440919c700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4404158880 0x7f4404151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.252+0000 7f4403fff700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4404152ce0 0x7f4404150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f440899b700 1 -- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4404158880 msgr2=0x7f4404151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f440899b700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4404158880 0x7f4404151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f440899b700 1 -- 172.21.15.23:0/3105110348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4404152ce0 msgr2=0x7f4404150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f440899b700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4404152ce0 0x7f4404150bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f440899b700 1 -- 172.21.15.23:0/3105110348 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f00094a0 con 0x7f44040c9fd0 2023-06-22T08:10:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f440899b700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f44041477c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f43f80090b0 tx=0x7f43f8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f44017fa700 1 -- 172.21.15.23:0/3105110348 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8004590 con 0x7f44040c9fd0 2023-06-22T08:10:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f44017fa700 1 -- 172.21.15.23:0/3105110348 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f801f070 con 0x7f44040c9fd0 2023-06-22T08:10:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f440abff700 1 -- 172.21.15.23:0/3105110348 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f80094a0 con 0x7f44040c9fd0 2023-06-22T08:10:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f44017fa700 1 -- 172.21.15.23:0/3105110348 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f80038e0 con 0x7f44040c9fd0 2023-06-22T08:10:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.253+0000 7f440abff700 1 -- 172.21.15.23:0/3105110348 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44041ee2b0 con 0x7f44040c9fd0 2023-06-22T08:10:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.254+0000 7f440abff700 1 -- 172.21.15.23:0/3105110348 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4404150a20 con 0x7f44040c9fd0 2023-06-22T08:10:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.255+0000 7f44017fa700 1 -- 172.21.15.23:0/3105110348 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f43f8003a40 con 0x7f44040c9fd0 2023-06-22T08:10:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.257+0000 7f44017fa700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f43e806ace0 0x7f43e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.257+0000 7f44017fa700 1 -- 172.21.15.23:0/3105110348 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f43f8021070 con 0x7f44040c9fd0 2023-06-22T08:10:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.257+0000 7f4403fff700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f43e806ace0 0x7f43e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.257+0000 7f4403fff700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f43e806ace0 0x7f43e806d190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4404152540 tx=0x7f43f0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:52.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.261+0000 7f44017fa700 1 -- 172.21.15.23:0/3105110348 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f43f8055cc0 con 0x7f44040c9fd0 2023-06-22T08:10:52.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.556+0000 7f440abff700 1 -- 172.21.15.23:0/3105110348 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4404158600 con 0x7f44040c9fd0 2023-06-22T08:10:52.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.557+0000 7f44017fa700 1 -- 172.21.15.23:0/3105110348 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6267 (secure 0 0 0) 0x7f43f80592e0 con 0x7f44040c9fd0 2023-06-22T08:10:52.561 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:10:52.562 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:10:46.771362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:10:52.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 -- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f43e806ace0 msgr2=0x7f43e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:52.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f43e806ace0 0x7f43e806d190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4404152540 tx=0x7f43f0009210 comp rx=0 tx=0).stop 2023-06-22T08:10:52.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 -- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 msgr2=0x7f44041477c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:52.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f44041477c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f43f80090b0 tx=0x7f43f8004750 comp rx=0 tx=0).stop 2023-06-22T08:10:52.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 -- 172.21.15.23:0/3105110348 shutdown_connections 2023-06-22T08:10:52.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f43e806ace0 0x7f43e806d190 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:52.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4404158880 0x7f4404151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:52.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44040c9fd0 0x7f44041477c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:52.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 --2- 172.21.15.23:0/3105110348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4404152ce0 0x7f4404150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:52.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 -- 172.21.15.23:0/3105110348 >> 172.21.15.23:0/3105110348 conn(0x7f440406ae30 msgr2=0x7f44040c4620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:52.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 -- 172.21.15.23:0/3105110348 shutdown_connections 2023-06-22T08:10:52.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:52.560+0000 7f440abff700 1 -- 172.21.15.23:0/3105110348 wait complete. 2023-06-22T08:10:52.567 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 85 2023-06-22T08:10:52.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:10:54.583 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:10:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.790+0000 7fab505ff700 1 Processor -- start 2023-06-22T08:10:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.791+0000 7fab505ff700 1 -- start start 2023-06-22T08:10:54.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.792+0000 7fab505ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48152d70 0x7fab480c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.792+0000 7fab505ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4815bea0 con 0x7fab480c45f0 2023-06-22T08:10:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.792+0000 7fab505ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4815c160 con 0x7fab48159330 2023-06-22T08:10:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.792+0000 7fab505ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab480c95e0 con 0x7fab48152d70 2023-06-22T08:10:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.792+0000 7fab4eb9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fab480c45f0 0x7fab480c4a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40628/0 (socket says 172.21.15.23:40628) 2023-06-22T08:10:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.792+0000 7fab4eb9c700 1 -- 172.21.15.23:0/502409151 learned_addr learned my addr 172.21.15.23:0/502409151 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:10:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.792+0000 7fab4e39b700 1 --2- 172.21.15.23:0/502409151 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48152d70 0x7fab480c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:54.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4d399700 1 -- 172.21.15.23:0/502409151 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794803216 0 0) 0x7fab4815bea0 con 0x7fab480c45f0 2023-06-22T08:10:54.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4d399700 1 -- 172.21.15.23:0/502409151 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab340035b0 con 0x7fab480c45f0 2023-06-22T08:10:54.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4d399700 1 -- 172.21.15.23:0/502409151 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864814634 0 0) 0x7fab4815c160 con 0x7fab48159330 2023-06-22T08:10:54.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4d399700 1 -- 172.21.15.23:0/502409151 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4815bea0 con 0x7fab48159330 2023-06-22T08:10:54.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4e39b700 1 -- 172.21.15.23:0/502409151 >> v1:172.21.15.67:6789/0 conn(0x7fab48159330 legacy=0x7fab480cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:54.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4e39b700 1 -- 172.21.15.23:0/502409151 >> v1:172.21.15.23:6789/0 conn(0x7fab480c45f0 legacy=0x7fab480c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4e39b700 1 -- 172.21.15.23:0/502409151 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4815c440 con 0x7fab48152d70 2023-06-22T08:10:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4e39b700 1 --2- 172.21.15.23:0/502409151 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48152d70 0x7fab480c97d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fab44009870 tx=0x7fab44009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e03f093d141c4e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4d399700 1 -- 172.21.15.23:0/502409151 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44011830 con 0x7fab48152d70 2023-06-22T08:10:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4d399700 1 -- 172.21.15.23:0/502409151 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab44011990 con 0x7fab48152d70 2023-06-22T08:10:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.793+0000 7fab4d399700 1 -- 172.21.15.23:0/502409151 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44011d20 con 0x7fab48152d70 2023-06-22T08:10:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.794+0000 7fab505ff700 1 -- 172.21.15.23:0/502409151 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48152d70 msgr2=0x7fab480c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:54.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.794+0000 7fab505ff700 1 --2- 172.21.15.23:0/502409151 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48152d70 0x7fab480c97d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fab44009870 tx=0x7fab44009b80 comp rx=0 tx=0).stop 2023-06-22T08:10:54.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.794+0000 7fab505ff700 1 -- 172.21.15.23:0/502409151 shutdown_connections 2023-06-22T08:10:54.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.794+0000 7fab505ff700 1 --2- 172.21.15.23:0/502409151 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48152d70 0x7fab480c97d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:54.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.794+0000 7fab505ff700 1 -- 172.21.15.23:0/502409151 >> 172.21.15.23:0/502409151 conn(0x7fab4806ae30 msgr2=0x7fab4806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:54.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.794+0000 7fab505ff700 1 -- 172.21.15.23:0/502409151 shutdown_connections 2023-06-22T08:10:54.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.794+0000 7fab505ff700 1 -- 172.21.15.23:0/502409151 wait complete. 2023-06-22T08:10:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.796+0000 7fab505ff700 1 Processor -- start 2023-06-22T08:10:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.796+0000 7fab505ff700 1 -- start start 2023-06-22T08:10:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab505ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab480c45f0 0x7fab48158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab505ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab48152d70 0x7fab481484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab505ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48159330 0x7fab481565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab505ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab480ccb30 con 0x7fab480c45f0 2023-06-22T08:10:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab505ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab480b2ac0 con 0x7fab48152d70 2023-06-22T08:10:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab505ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab480b2da0 con 0x7fab48159330 2023-06-22T08:10:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab4e39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab480c45f0 0x7fab48158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:54.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab4e39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab480c45f0 0x7fab48158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34812/0 (socket says 172.21.15.23:34812) 2023-06-22T08:10:54.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab4e39b700 1 -- 172.21.15.23:0/173963934 learned_addr learned my addr 172.21.15.23:0/173963934 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:10:54.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab4eb9c700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48159330 0x7fab481565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:54.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab4db9a700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab48152d70 0x7fab481484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:54.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab4e39b700 1 -- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48159330 msgr2=0x7fab481565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:54.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab4e39b700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48159330 0x7fab481565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:54.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab4e39b700 1 -- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab48152d70 msgr2=0x7fab481484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:54.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab4e39b700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab48152d70 0x7fab481484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:54.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.797+0000 7fab4e39b700 1 -- 172.21.15.23:0/173963934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab440094a0 con 0x7fab480c45f0 2023-06-22T08:10:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.798+0000 7fab4e39b700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab480c45f0 0x7fab48158420 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fab440090b0 tx=0x7fab44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.798+0000 7fab3effd700 1 -- 172.21.15.23:0/173963934 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4401a6e0 con 0x7fab480c45f0 2023-06-22T08:10:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.798+0000 7fab3effd700 1 -- 172.21.15.23:0/173963934 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4401a840 con 0x7fab480c45f0 2023-06-22T08:10:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.798+0000 7fab3effd700 1 -- 172.21.15.23:0/173963934 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4401abb0 con 0x7fab480c45f0 2023-06-22T08:10:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.798+0000 7fab505ff700 1 -- 172.21.15.23:0/173963934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab481f1300 con 0x7fab480c45f0 2023-06-22T08:10:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.798+0000 7fab505ff700 1 -- 172.21.15.23:0/173963934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab481f18c0 con 0x7fab480c45f0 2023-06-22T08:10:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.799+0000 7fab505ff700 1 -- 172.21.15.23:0/173963934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4804ea20 con 0x7fab480c45f0 2023-06-22T08:10:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.800+0000 7fab3effd700 1 -- 172.21.15.23:0/173963934 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fab44022070 con 0x7fab480c45f0 2023-06-22T08:10:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.801+0000 7fab3effd700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab2c06ad10 0x7fab2c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.801+0000 7fab3effd700 1 -- 172.21.15.23:0/173963934 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7fab4408a8f0 con 0x7fab480c45f0 2023-06-22T08:10:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.802+0000 7fab4db9a700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab2c06ad10 0x7fab2c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.802+0000 7fab4db9a700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab2c06ad10 0x7fab2c06d1c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fab480cf170 tx=0x7fab38009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:54.806+0000 7fab3effd700 1 -- 172.21.15.23:0/173963934 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fab44055630 con 0x7fab480c45f0 2023-06-22T08:10:55.071 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.068+0000 7fab505ff700 1 -- 172.21.15.23:0/173963934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab480c9e00 con 0x7fab480c45f0 2023-06-22T08:10:55.072 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.068+0000 7fab3effd700 1 -- 172.21.15.23:0/173963934 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6267 (secure 0 0 0) 0x7fab44058c50 con 0x7fab480c45f0 2023-06-22T08:10:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:10:55.072 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:10:46.771362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:10:55.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 -- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab2c06ad10 msgr2=0x7fab2c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:55.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab2c06ad10 0x7fab2c06d1c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fab480cf170 tx=0x7fab38009040 comp rx=0 tx=0).stop 2023-06-22T08:10:55.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 -- 172.21.15.23:0/173963934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab480c45f0 msgr2=0x7fab48158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:55.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab480c45f0 0x7fab48158420 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fab440090b0 tx=0x7fab44004750 comp rx=0 tx=0).stop 2023-06-22T08:10:55.075 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 -- 172.21.15.23:0/173963934 shutdown_connections 2023-06-22T08:10:55.076 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fab48159330 0x7fab481565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:55.076 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab480c45f0 0x7fab48158420 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:55.076 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fab2c06ad10 0x7fab2c06d1c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:55.076 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 --2- 172.21.15.23:0/173963934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab48152d70 0x7fab481484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:55.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 -- 172.21.15.23:0/173963934 >> 172.21.15.23:0/173963934 conn(0x7fab4806ae30 msgr2=0x7fab4814e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:55.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 -- 172.21.15.23:0/173963934 shutdown_connections 2023-06-22T08:10:55.077 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:55.071+0000 7fab505ff700 1 -- 172.21.15.23:0/173963934 wait complete. 2023-06-22T08:10:55.077 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 85 2023-06-22T08:10:55.093 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:10:57.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:10:57.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.324+0000 7f117ebff700 1 Processor -- start 2023-06-22T08:10:57.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.324+0000 7f117ebff700 1 -- start start 2023-06-22T08:10:57.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.325+0000 7f117ebff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1178152a10 0x7f11780c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:57.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.325+0000 7f117ebff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f117815c0b0 0x7f117815c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:57.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f117ebff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1178159730 con 0x7f11780c9200 2023-06-22T08:10:57.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f117ebff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11780c5470 con 0x7f1178152a10 2023-06-22T08:10:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f117ebff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1178158910 con 0x7f117815c0b0 2023-06-22T08:10:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f117c99b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1178152a10 0x7f11780c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f1177fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f11780c9200 0x7f11780cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34104/0 (socket says 172.21.15.23:34104) 2023-06-22T08:10:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f1177fff700 1 -- 172.21.15.23:0/4019688170 learned_addr learned my addr 172.21.15.23:0/4019688170 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:10:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f117d19c700 1 --2- 172.21.15.23:0/4019688170 >> v2:172.21.15.67:3300/0 conn(0x7f117815c0b0 0x7f117815c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f117c99b700 1 -- 172.21.15.23:0/4019688170 >> v2:172.21.15.67:3300/0 conn(0x7f117815c0b0 msgr2=0x7f117815c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:10:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f117c99b700 1 --2- 172.21.15.23:0/4019688170 >> v2:172.21.15.67:3300/0 conn(0x7f117815c0b0 0x7f117815c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f117c99b700 1 -- 172.21.15.23:0/4019688170 >> v1:172.21.15.23:6789/0 conn(0x7f11780c9200 legacy=0x7f11780cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f117c99b700 1 -- 172.21.15.23:0/4019688170 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117815ca50 con 0x7f1178152a10 2023-06-22T08:10:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.326+0000 7f11777fe700 1 -- 172.21.15.23:0/4019688170 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424519057 0 0) 0x7f1178159730 con 0x7f11780c9200 2023-06-22T08:10:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.327+0000 7f117c99b700 1 --2- 172.21.15.23:0/4019688170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1178152a10 0x7f11780c4a30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f117806c330 tx=0x7f116c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66838b54f28b162 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.327+0000 7f11777fe700 1 -- 172.21.15.23:0/4019688170 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c011830 con 0x7f1178152a10 2023-06-22T08:10:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.327+0000 7f11777fe700 1 -- 172.21.15.23:0/4019688170 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c011990 con 0x7f1178152a10 2023-06-22T08:10:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.327+0000 7f11777fe700 1 -- 172.21.15.23:0/4019688170 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c011d20 con 0x7f1178152a10 2023-06-22T08:10:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.327+0000 7f117ebff700 1 -- 172.21.15.23:0/4019688170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1178152a10 msgr2=0x7f11780c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.327+0000 7f117ebff700 1 --2- 172.21.15.23:0/4019688170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1178152a10 0x7f11780c4a30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f117806c330 tx=0x7f116c009b20 comp rx=0 tx=0).stop 2023-06-22T08:10:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.327+0000 7f117ebff700 1 -- 172.21.15.23:0/4019688170 shutdown_connections 2023-06-22T08:10:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.327+0000 7f117ebff700 1 --2- 172.21.15.23:0/4019688170 >> v2:172.21.15.67:3300/0 conn(0x7f117815c0b0 0x7f117815c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.327+0000 7f117ebff700 1 --2- 172.21.15.23:0/4019688170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1178152a10 0x7f11780c4a30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.327+0000 7f117ebff700 1 -- 172.21.15.23:0/4019688170 >> 172.21.15.23:0/4019688170 conn(0x7f1178065b90 msgr2=0x7f1178065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.328+0000 7f117ebff700 1 -- 172.21.15.23:0/4019688170 shutdown_connections 2023-06-22T08:10:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.328+0000 7f117ebff700 1 -- 172.21.15.23:0/4019688170 wait complete. 2023-06-22T08:10:57.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.329+0000 7f117ebff700 1 Processor -- start 2023-06-22T08:10:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.329+0000 7f117ebff700 1 -- start start 2023-06-22T08:10:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.330+0000 7f117ebff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11780c9200 0x7f11781eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.330+0000 7f117ebff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178152a10 0x7f11780ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.330+0000 7f117ebff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f117815c0b0 0x7f117814e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.330+0000 7f117ebff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11780b2490 con 0x7f117815c0b0 2023-06-22T08:10:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.330+0000 7f117ebff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1178152150 con 0x7f1178152a10 2023-06-22T08:10:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.330+0000 7f117ebff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11781ee390 con 0x7f11780c9200 2023-06-22T08:10:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f117d19c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f117815c0b0 0x7f117814e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f117d19c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f117815c0b0 0x7f117814e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38898/0 (socket says 172.21.15.23:38898) 2023-06-22T08:10:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f1177fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178152a10 0x7f11780ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f117d19c700 1 -- 172.21.15.23:0/3336955137 learned_addr learned my addr 172.21.15.23:0/3336955137 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:10:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f117c99b700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11780c9200 0x7f11781eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f117d19c700 1 -- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11780c9200 msgr2=0x7f11781eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f117d19c700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11780c9200 0x7f11781eeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f117d19c700 1 -- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178152a10 msgr2=0x7f11780ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f117d19c700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178152a10 0x7f11780ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f117d19c700 1 -- 172.21.15.23:0/3336955137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c0094a0 con 0x7f117815c0b0 2023-06-22T08:10:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f1177fff700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178152a10 0x7f11780ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:10:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.331+0000 7f117d19c700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f117815c0b0 0x7f117814e2b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f117000efc0 tx=0x7f117000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.332+0000 7f11757fa700 1 -- 172.21.15.23:0/3336955137 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117000cad0 con 0x7f117815c0b0 2023-06-22T08:10:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.332+0000 7f11757fa700 1 -- 172.21.15.23:0/3336955137 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117000f040 con 0x7f117815c0b0 2023-06-22T08:10:57.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.332+0000 7f117ebff700 1 -- 172.21.15.23:0/3336955137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11781515b0 con 0x7f117815c0b0 2023-06-22T08:10:57.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.332+0000 7f11757fa700 1 -- 172.21.15.23:0/3336955137 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117000ce70 con 0x7f117815c0b0 2023-06-22T08:10:57.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.332+0000 7f117ebff700 1 -- 172.21.15.23:0/3336955137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1178151b90 con 0x7f117815c0b0 2023-06-22T08:10:57.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.333+0000 7f117ebff700 1 -- 172.21.15.23:0/3336955137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11780cca90 con 0x7f117815c0b0 2023-06-22T08:10:57.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.334+0000 7f11757fa700 1 -- 172.21.15.23:0/3336955137 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1170014480 con 0x7f117815c0b0 2023-06-22T08:10:57.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.336+0000 7f11757fa700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f115c06ace0 0x7f115c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:57.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.336+0000 7f11757fa700 1 -- 172.21.15.23:0/3336955137 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f117008a3e0 con 0x7f117815c0b0 2023-06-22T08:10:57.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.336+0000 7f117c99b700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f115c06ace0 0x7f115c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:57.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.337+0000 7f117c99b700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f115c06ace0 0x7f115c06d190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f116c0090b0 tx=0x7f116c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:57.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.340+0000 7f11757fa700 1 -- 172.21.15.23:0/3336955137 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1170054f50 con 0x7f117815c0b0 2023-06-22T08:10:57.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.607+0000 7f117ebff700 1 -- 172.21.15.23:0/3336955137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11780669c0 con 0x7f117815c0b0 2023-06-22T08:10:57.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.607+0000 7f11757fa700 1 -- 172.21.15.23:0/3336955137 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6267 (secure 0 0 0) 0x7f1170058570 con 0x7f117815c0b0 2023-06-22T08:10:57.611 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:10:57.611 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:10:46.771362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:10:57.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 -- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f115c06ace0 msgr2=0x7f115c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:57.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f115c06ace0 0x7f115c06d190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f116c0090b0 tx=0x7f116c008030 comp rx=0 tx=0).stop 2023-06-22T08:10:57.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 -- 172.21.15.23:0/3336955137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f117815c0b0 msgr2=0x7f117814e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:57.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f117815c0b0 0x7f117814e2b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f117000efc0 tx=0x7f117000c0c0 comp rx=0 tx=0).stop 2023-06-22T08:10:57.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 -- 172.21.15.23:0/3336955137 shutdown_connections 2023-06-22T08:10:57.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f117815c0b0 0x7f117814e2b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:57.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11780c9200 0x7f11781eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:57.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f115c06ace0 0x7f115c06d190 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:57.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 --2- 172.21.15.23:0/3336955137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1178152a10 0x7f11780ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:57.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 -- 172.21.15.23:0/3336955137 >> 172.21.15.23:0/3336955137 conn(0x7f1178065b90 msgr2=0x7f117806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:57.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 -- 172.21.15.23:0/3336955137 shutdown_connections 2023-06-22T08:10:57.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:57.610+0000 7f117ebff700 1 -- 172.21.15.23:0/3336955137 wait complete. 2023-06-22T08:10:57.615 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 85 2023-06-22T08:10:57.631 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:10:59.632 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:10:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.843+0000 7f6d27bff700 1 Processor -- start 2023-06-22T08:10:59.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.843+0000 7f6d27bff700 1 -- start start 2023-06-22T08:10:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d27bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d2014f0c0 0x7f6d201558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d27bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d200c61e0 0x7f6d2006df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d27bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6d2006e450 0x7f6d2006d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d27bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d200c1f70 con 0x7f6d2006e450 2023-06-22T08:10:59.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d27bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d200c4ea0 con 0x7f6d2014f0c0 2023-06-22T08:10:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d27bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d200c5140 con 0x7f6d200c61e0 2023-06-22T08:10:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d2619c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6d2006e450 0x7f6d2006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d2619c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6d2006e450 0x7f6d2006d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38912/0 (socket says 172.21.15.23:38912) 2023-06-22T08:10:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d2619c700 1 -- 172.21.15.23:0/2554708567 learned_addr learned my addr 172.21.15.23:0/2554708567 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:10:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d2519a700 1 --2- 172.21.15.23:0/2554708567 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d200c61e0 0x7f6d2006df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:59.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.845+0000 7f6d2599b700 1 --2- 172.21.15.23:0/2554708567 >> v2:172.21.15.67:3300/0 conn(0x7f6d2014f0c0 0x7f6d201558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.846+0000 7f6d2619c700 1 -- 172.21.15.23:0/2554708567 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d200c61e0 msgr2=0x7f6d2006df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:10:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.846+0000 7f6d2619c700 1 --2- 172.21.15.23:0/2554708567 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d200c61e0 0x7f6d2006df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.846+0000 7f6d2619c700 1 -- 172.21.15.23:0/2554708567 >> v2:172.21.15.67:3300/0 conn(0x7f6d2014f0c0 msgr2=0x7f6d201558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:10:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.846+0000 7f6d2619c700 1 --2- 172.21.15.23:0/2554708567 >> v2:172.21.15.67:3300/0 conn(0x7f6d2014f0c0 0x7f6d201558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:59.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.846+0000 7f6d2619c700 1 -- 172.21.15.23:0/2554708567 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d200c5420 con 0x7f6d2006e450 2023-06-22T08:10:59.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.846+0000 7f6d2619c700 1 --2- 172.21.15.23:0/2554708567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2006e450 0x7f6d2006d050 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6d2006fab0 tx=0x7f6d1800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be814558b7dd6754 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:59.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.846+0000 7f6d24999700 1 -- 172.21.15.23:0/2554708567 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18010070 con 0x7f6d2006e450 2023-06-22T08:10:59.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.846+0000 7f6d24999700 1 -- 172.21.15.23:0/2554708567 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1800dce0 con 0x7f6d2006e450 2023-06-22T08:10:59.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.846+0000 7f6d24999700 1 -- 172.21.15.23:0/2554708567 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1800b310 con 0x7f6d2006e450 2023-06-22T08:10:59.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.847+0000 7f6d27bff700 1 -- 172.21.15.23:0/2554708567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2006e450 msgr2=0x7f6d2006d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:59.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.847+0000 7f6d27bff700 1 --2- 172.21.15.23:0/2554708567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2006e450 0x7f6d2006d050 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6d2006fab0 tx=0x7f6d1800d430 comp rx=0 tx=0).stop 2023-06-22T08:10:59.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.847+0000 7f6d27bff700 1 -- 172.21.15.23:0/2554708567 shutdown_connections 2023-06-22T08:10:59.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.847+0000 7f6d27bff700 1 --2- 172.21.15.23:0/2554708567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2006e450 0x7f6d2006d050 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:59.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.847+0000 7f6d27bff700 1 --2- 172.21.15.23:0/2554708567 >> v2:172.21.15.67:3300/0 conn(0x7f6d2014f0c0 0x7f6d201558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:59.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.847+0000 7f6d27bff700 1 --2- 172.21.15.23:0/2554708567 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d200c61e0 0x7f6d2006df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.847+0000 7f6d27bff700 1 -- 172.21.15.23:0/2554708567 >> 172.21.15.23:0/2554708567 conn(0x7f6d200ca3f0 msgr2=0x7f6d200ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:10:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.847+0000 7f6d27bff700 1 -- 172.21.15.23:0/2554708567 shutdown_connections 2023-06-22T08:10:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.847+0000 7f6d27bff700 1 -- 172.21.15.23:0/2554708567 wait complete. 2023-06-22T08:10:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.849+0000 7f6d27bff700 1 Processor -- start 2023-06-22T08:10:59.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.849+0000 7f6d27bff700 1 -- start start 2023-06-22T08:10:59.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d27bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006e450 0x7f6d201f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:59.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d27bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d200c61e0 0x7f6d2013e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:59.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d27bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2014f0c0 0x7f6d2014aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:59.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d27bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2013f400 con 0x7f6d2014f0c0 2023-06-22T08:10:59.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d27bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d20149620 con 0x7f6d2006e450 2023-06-22T08:10:59.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d27bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2014a960 con 0x7f6d200c61e0 2023-06-22T08:10:59.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d2619c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2014f0c0 0x7f6d2014aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:59.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d2619c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2014f0c0 0x7f6d2014aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38928/0 (socket says 172.21.15.23:38928) 2023-06-22T08:10:59.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d2619c700 1 -- 172.21.15.23:0/4151603373 learned_addr learned my addr 172.21.15.23:0/4151603373 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:10:59.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d2519a700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d200c61e0 0x7f6d2013e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:59.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.850+0000 7f6d2599b700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006e450 0x7f6d201f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:59.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d2619c700 1 -- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d200c61e0 msgr2=0x7f6d2013e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:59.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d2619c700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d200c61e0 0x7f6d2013e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:59.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d2619c700 1 -- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006e450 msgr2=0x7f6d201f1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:10:59.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d2619c700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006e450 0x7f6d201f1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:10:59.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d2619c700 1 -- 172.21.15.23:0/4151603373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1800bd40 con 0x7f6d2014f0c0 2023-06-22T08:10:59.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d2619c700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2014f0c0 0x7f6d2014aca0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6d18014000 tx=0x7f6d18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:59.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d167fc700 1 -- 172.21.15.23:0/4151603373 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18009a10 con 0x7f6d2014f0c0 2023-06-22T08:10:59.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d167fc700 1 -- 172.21.15.23:0/4151603373 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d18009b70 con 0x7f6d2014f0c0 2023-06-22T08:10:59.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d167fc700 1 -- 172.21.15.23:0/4151603373 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18003e50 con 0x7f6d2014f0c0 2023-06-22T08:10:59.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d27bff700 1 -- 172.21.15.23:0/4151603373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d201498c0 con 0x7f6d2014f0c0 2023-06-22T08:10:59.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.851+0000 7f6d27bff700 1 -- 172.21.15.23:0/4151603373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d20149e80 con 0x7f6d2014f0c0 2023-06-22T08:10:59.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.852+0000 7f6d27bff700 1 -- 172.21.15.23:0/4151603373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2004ea20 con 0x7f6d2014f0c0 2023-06-22T08:10:59.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.853+0000 7f6d167fc700 1 -- 172.21.15.23:0/4151603373 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6d18004040 con 0x7f6d2014f0c0 2023-06-22T08:10:59.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.854+0000 7f6d167fc700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d0406ad10 0x7f6d0406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:10:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.854+0000 7f6d167fc700 1 -- 172.21.15.23:0/4151603373 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f6d1808b140 con 0x7f6d2014f0c0 2023-06-22T08:10:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.855+0000 7f6d2599b700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d0406ad10 0x7f6d0406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:10:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.855+0000 7f6d2599b700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d0406ad10 0x7f6d0406d1c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009ac0 tx=0x7f6d1c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:10:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:10:59.859+0000 7f6d167fc700 1 -- 172.21.15.23:0/4151603373 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6d18055d40 con 0x7f6d2014f0c0 2023-06-22T08:11:00.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.108+0000 7f6d27bff700 1 -- 172.21.15.23:0/4151603373 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d200cb300 con 0x7f6d2014f0c0 2023-06-22T08:11:00.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.109+0000 7f6d167fc700 1 -- 172.21.15.23:0/4151603373 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6267 (secure 0 0 0) 0x7f6d18059360 con 0x7f6d2014f0c0 2023-06-22T08:11:00.113 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:00.113 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:10:46.771362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:00.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.111+0000 7f6d27bff700 1 -- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d0406ad10 msgr2=0x7f6d0406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:00.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.111+0000 7f6d27bff700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d0406ad10 0x7f6d0406d1c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009ac0 tx=0x7f6d1c009210 comp rx=0 tx=0).stop 2023-06-22T08:11:00.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.111+0000 7f6d27bff700 1 -- 172.21.15.23:0/4151603373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2014f0c0 msgr2=0x7f6d2014aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:00.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.111+0000 7f6d27bff700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2014f0c0 0x7f6d2014aca0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6d18014000 tx=0x7f6d18004ad0 comp rx=0 tx=0).stop 2023-06-22T08:11:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.112+0000 7f6d27bff700 1 -- 172.21.15.23:0/4151603373 shutdown_connections 2023-06-22T08:11:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.112+0000 7f6d27bff700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d2014f0c0 0x7f6d2014aca0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.112+0000 7f6d27bff700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d0406ad10 0x7f6d0406d1c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.112+0000 7f6d27bff700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006e450 0x7f6d201f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:00.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.112+0000 7f6d27bff700 1 --2- 172.21.15.23:0/4151603373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d200c61e0 0x7f6d2013e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:00.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.112+0000 7f6d27bff700 1 -- 172.21.15.23:0/4151603373 >> 172.21.15.23:0/4151603373 conn(0x7f6d200ca3f0 msgr2=0x7f6d2014c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:00.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.112+0000 7f6d27bff700 1 -- 172.21.15.23:0/4151603373 shutdown_connections 2023-06-22T08:11:00.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:00.112+0000 7f6d27bff700 1 -- 172.21.15.23:0/4151603373 wait complete. 2023-06-22T08:11:00.117 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 85 2023-06-22T08:11:00.133 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:11:02.134 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:02.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.400+0000 7f0c799ff700 1 Processor -- start 2023-06-22T08:11:02.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.400+0000 7f0c799ff700 1 -- start start 2023-06-22T08:11:02.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.402+0000 7f0c799ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0c740c69e0 0x7f0c74155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:02.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.402+0000 7f0c799ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c740c51a0 con 0x7f0c74071070 2023-06-22T08:11:02.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.402+0000 7f0c799ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c740c44e0 con 0x7f0c7414f850 2023-06-22T08:11:02.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.402+0000 7f0c799ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c740c5440 con 0x7f0c740c69e0 2023-06-22T08:11:02.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.402+0000 7f0c72ffd700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0c7414f850 0x7f0c740c3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:36132/0 (socket says 172.21.15.23:36132) 2023-06-22T08:11:02.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.402+0000 7f0c72ffd700 1 -- 172.21.15.23:0/2559480724 learned_addr learned my addr 172.21.15.23:0/2559480724 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:02.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.402+0000 7f0c727fc700 1 --2- 172.21.15.23:0/2559480724 >> v2:172.21.15.23:3300/0 conn(0x7f0c740c69e0 0x7f0c74155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:02.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.402+0000 7f0c71ffb700 1 -- 172.21.15.23:0/2559480724 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621003351 0 0) 0x7f0c740c44e0 con 0x7f0c7414f850 2023-06-22T08:11:02.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.403+0000 7f0c727fc700 1 -- 172.21.15.23:0/2559480724 >> v1:172.21.15.67:6789/0 conn(0x7f0c7414f850 legacy=0x7f0c740c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:02.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.403+0000 7f0c727fc700 1 -- 172.21.15.23:0/2559480724 >> v1:172.21.15.23:6789/0 conn(0x7f0c74071070 legacy=0x7f0c7415fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:02.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.403+0000 7f0c727fc700 1 -- 172.21.15.23:0/2559480724 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c740c5720 con 0x7f0c740c69e0 2023-06-22T08:11:02.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.403+0000 7f0c727fc700 1 --2- 172.21.15.23:0/2559480724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740c69e0 0x7f0c74155600 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009840 tx=0x7f0c68009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eedad9a508a61fd6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:02.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.403+0000 7f0c71ffb700 1 -- 172.21.15.23:0/2559480724 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68011830 con 0x7f0c740c69e0 2023-06-22T08:11:02.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.403+0000 7f0c71ffb700 1 -- 172.21.15.23:0/2559480724 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c68011990 con 0x7f0c740c69e0 2023-06-22T08:11:02.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.403+0000 7f0c71ffb700 1 -- 172.21.15.23:0/2559480724 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68011d00 con 0x7f0c740c69e0 2023-06-22T08:11:02.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.404+0000 7f0c799ff700 1 -- 172.21.15.23:0/2559480724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740c69e0 msgr2=0x7f0c74155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:02.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.404+0000 7f0c799ff700 1 --2- 172.21.15.23:0/2559480724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740c69e0 0x7f0c74155600 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009840 tx=0x7f0c68009b50 comp rx=0 tx=0).stop 2023-06-22T08:11:02.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.404+0000 7f0c799ff700 1 -- 172.21.15.23:0/2559480724 shutdown_connections 2023-06-22T08:11:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.404+0000 7f0c799ff700 1 --2- 172.21.15.23:0/2559480724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740c69e0 0x7f0c74155600 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.404+0000 7f0c799ff700 1 -- 172.21.15.23:0/2559480724 >> 172.21.15.23:0/2559480724 conn(0x7f0c740ca3f0 msgr2=0x7f0c740ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.404+0000 7f0c799ff700 1 -- 172.21.15.23:0/2559480724 shutdown_connections 2023-06-22T08:11:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.404+0000 7f0c799ff700 1 -- 172.21.15.23:0/2559480724 wait complete. 2023-06-22T08:11:02.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.406+0000 7f0c799ff700 1 Processor -- start 2023-06-22T08:11:02.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.406+0000 7f0c799ff700 1 -- start start 2023-06-22T08:11:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.407+0000 7f0c799ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c74071070 0x7f0c741de080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.407+0000 7f0c799ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7414f850 0x7f0c740c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.407+0000 7f0c799ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c741e1010 0x7f0c74146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.407+0000 7f0c799ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7414f130 con 0x7f0c741e1010 2023-06-22T08:11:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.407+0000 7f0c799ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741de5c0 con 0x7f0c7414f850 2023-06-22T08:11:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.407+0000 7f0c799ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740c14b0 con 0x7f0c74071070 2023-06-22T08:11:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.407+0000 7f0c737fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c741e1010 0x7f0c74146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.407+0000 7f0c727fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7414f850 0x7f0c740c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.407+0000 7f0c737fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c741e1010 0x7f0c74146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38946/0 (socket says 172.21.15.23:38946) 2023-06-22T08:11:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.407+0000 7f0c737fe700 1 -- 172.21.15.23:0/3910051597 learned_addr learned my addr 172.21.15.23:0/3910051597 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.408+0000 7f0c72ffd700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c74071070 0x7f0c741de080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.408+0000 7f0c727fc700 1 -- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c74071070 msgr2=0x7f0c741de080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.408+0000 7f0c727fc700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c74071070 0x7f0c741de080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.408+0000 7f0c727fc700 1 -- 172.21.15.23:0/3910051597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c741e1010 msgr2=0x7f0c74146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.408+0000 7f0c727fc700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c741e1010 0x7f0c74146a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:02.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.408+0000 7f0c727fc700 1 -- 172.21.15.23:0/3910051597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c680094a0 con 0x7f0c7414f850 2023-06-22T08:11:02.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.408+0000 7f0c737fe700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c741e1010 0x7f0c74146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:11:02.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.408+0000 7f0c727fc700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7414f850 0x7f0c740c9490 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0c68016000 tx=0x7f0c68004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:02.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.408+0000 7f0c5bfff700 1 -- 172.21.15.23:0/3910051597 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c680117e0 con 0x7f0c7414f850 2023-06-22T08:11:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.409+0000 7f0c5bfff700 1 -- 172.21.15.23:0/3910051597 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6801a750 con 0x7f0c7414f850 2023-06-22T08:11:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.409+0000 7f0c5bfff700 1 -- 172.21.15.23:0/3910051597 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6801ab20 con 0x7f0c7414f850 2023-06-22T08:11:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.409+0000 7f0c799ff700 1 -- 172.21.15.23:0/3910051597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7414ab00 con 0x7f0c7414f850 2023-06-22T08:11:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.409+0000 7f0c799ff700 1 -- 172.21.15.23:0/3910051597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c740c11f0 con 0x7f0c7414f850 2023-06-22T08:11:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.410+0000 7f0c799ff700 1 -- 172.21.15.23:0/3910051597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7414ee30 con 0x7f0c7414f850 2023-06-22T08:11:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.412+0000 7f0c5bfff700 1 -- 172.21.15.23:0/3910051597 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0c68022070 con 0x7f0c7414f850 2023-06-22T08:11:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.413+0000 7f0c5bfff700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c6006ace0 0x7f0c6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.413+0000 7f0c5bfff700 1 -- 172.21.15.23:0/3910051597 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 7243+0+0 (secure 0 0 0) 0x7f0c6808a3a0 con 0x7f0c7414f850 2023-06-22T08:11:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.413+0000 7f0c72ffd700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c6006ace0 0x7f0c6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.414+0000 7f0c72ffd700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c6006ace0 0x7f0c6006d190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0c640018b0 tx=0x7f0c64001840 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.418+0000 7f0c5bfff700 1 -- 172.21.15.23:0/3910051597 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0c680550f0 con 0x7f0c7414f850 2023-06-22T08:11:02.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.668+0000 7f0c799ff700 1 -- 172.21.15.23:0/3910051597 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c7404ea20 con 0x7f0c7414f850 2023-06-22T08:11:02.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.669+0000 7f0c5bfff700 1 -- 172.21.15.23:0/3910051597 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6267 (secure 0 0 0) 0x7f0c68058710 con 0x7f0c7414f850 2023-06-22T08:11:02.674 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:02.675 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5845,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:10:46.771362+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4285,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/567663545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":567663545},{"type":"v1","addr":"172.21.15.67:6839","nonce":567663545}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:02.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 -- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c6006ace0 msgr2=0x7f0c6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:02.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c6006ace0 0x7f0c6006d190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0c640018b0 tx=0x7f0c64001840 comp rx=0 tx=0).stop 2023-06-22T08:11:02.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 -- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7414f850 msgr2=0x7f0c740c9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:02.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7414f850 0x7f0c740c9490 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0c68016000 tx=0x7f0c68004750 comp rx=0 tx=0).stop 2023-06-22T08:11:02.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 -- 172.21.15.23:0/3910051597 shutdown_connections 2023-06-22T08:11:02.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c741e1010 0x7f0c74146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:02.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0c74071070 0x7f0c741de080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:02.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0c6006ace0 0x7f0c6006d190 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:02.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 --2- 172.21.15.23:0/3910051597 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c7414f850 0x7f0c740c9490 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:02.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 -- 172.21.15.23:0/3910051597 >> 172.21.15.23:0/3910051597 conn(0x7f0c740ca3f0 msgr2=0x7f0c740c3010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:02.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 -- 172.21.15.23:0/3910051597 shutdown_connections 2023-06-22T08:11:02.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:02.672+0000 7f0c799ff700 1 -- 172.21.15.23:0/3910051597 wait complete. 2023-06-22T08:11:02.680 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 85 2023-06-22T08:11:02.695 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:11:04.696 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:04.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.911+0000 7f479afff700 1 Processor -- start 2023-06-22T08:11:04.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.911+0000 7f479afff700 1 -- start start 2023-06-22T08:11:04.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.912+0000 7f479afff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4794152ce0 0x7f47940cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.912+0000 7f479afff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479415c370 con 0x7f47940c40b0 2023-06-22T08:11:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.912+0000 7f479afff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479415bda0 con 0x7f47940c9fd0 2023-06-22T08:11:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.912+0000 7f479afff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47941530b0 con 0x7f4794152ce0 2023-06-22T08:11:04.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.912+0000 7f479959c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f47940c40b0 0x7f47940c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34130/0 (socket says 172.21.15.23:34130) 2023-06-22T08:11:04.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.912+0000 7f479959c700 1 -- 172.21.15.23:0/2142028580 learned_addr learned my addr 172.21.15.23:0/2142028580 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:04.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.912+0000 7f4798d9b700 1 --2- 172.21.15.23:0/2142028580 >> v2:172.21.15.67:3300/0 conn(0x7f4794152ce0 0x7f47940cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:04.918 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.913+0000 7f47937fe700 1 -- 172.21.15.23:0/2142028580 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462203468 0 0) 0x7f479415bda0 con 0x7f47940c9fd0 2023-06-22T08:11:04.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.913+0000 7f47937fe700 1 -- 172.21.15.23:0/2142028580 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47840035b0 con 0x7f47940c9fd0 2023-06-22T08:11:04.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.913+0000 7f47937fe700 1 -- 172.21.15.23:0/2142028580 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656258895 0 0) 0x7f479415c370 con 0x7f47940c40b0 2023-06-22T08:11:04.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.913+0000 7f4798d9b700 1 -- 172.21.15.23:0/2142028580 >> v1:172.21.15.67:6789/0 conn(0x7f47940c9fd0 legacy=0x7f479415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:04.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.913+0000 7f4798d9b700 1 -- 172.21.15.23:0/2142028580 >> v1:172.21.15.23:6789/0 conn(0x7f47940c40b0 legacy=0x7f47940c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:04.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.913+0000 7f4798d9b700 1 -- 172.21.15.23:0/2142028580 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47940c9460 con 0x7f4794152ce0 2023-06-22T08:11:04.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.913+0000 7f4798d9b700 1 --2- 172.21.15.23:0/2142028580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4794152ce0 0x7f47940cf1b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4788009840 tx=0x7f4788009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=497f8ff4058aca85 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:04.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.913+0000 7f47937fe700 1 -- 172.21.15.23:0/2142028580 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788011830 con 0x7f4794152ce0 2023-06-22T08:11:04.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.914+0000 7f47937fe700 1 -- 172.21.15.23:0/2142028580 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788011990 con 0x7f4794152ce0 2023-06-22T08:11:04.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.914+0000 7f47937fe700 1 -- 172.21.15.23:0/2142028580 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788011d20 con 0x7f4794152ce0 2023-06-22T08:11:04.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.914+0000 7f479afff700 1 -- 172.21.15.23:0/2142028580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4794152ce0 msgr2=0x7f47940cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:04.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.914+0000 7f479afff700 1 --2- 172.21.15.23:0/2142028580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4794152ce0 0x7f47940cf1b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4788009840 tx=0x7f4788009b50 comp rx=0 tx=0).stop 2023-06-22T08:11:04.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.914+0000 7f479afff700 1 -- 172.21.15.23:0/2142028580 shutdown_connections 2023-06-22T08:11:04.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.914+0000 7f479afff700 1 --2- 172.21.15.23:0/2142028580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4794152ce0 0x7f47940cf1b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:04.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.914+0000 7f479afff700 1 -- 172.21.15.23:0/2142028580 >> 172.21.15.23:0/2142028580 conn(0x7f479406ae30 msgr2=0x7f479406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:04.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.914+0000 7f479afff700 1 -- 172.21.15.23:0/2142028580 shutdown_connections 2023-06-22T08:11:04.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.914+0000 7f479afff700 1 -- 172.21.15.23:0/2142028580 wait complete. 2023-06-22T08:11:04.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.916+0000 7f479afff700 1 Processor -- start 2023-06-22T08:11:04.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.916+0000 7f479afff700 1 -- start start 2023-06-22T08:11:04.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f479afff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47940c40b0 0x7f479415b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:04.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f479afff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f47940c9fd0 0x7f4794157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:04.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f479afff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4794152ce0 0x7f4794157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:04.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f479afff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47941ee070 con 0x7f4794152ce0 2023-06-22T08:11:04.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f479afff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47941eeed0 con 0x7f47940c40b0 2023-06-22T08:11:04.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f479afff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47941ee2c0 con 0x7f47940c9fd0 2023-06-22T08:11:04.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f479959c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4794152ce0 0x7f4794157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:04.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f479959c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4794152ce0 0x7f4794157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38962/0 (socket says 172.21.15.23:38962) 2023-06-22T08:11:04.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f479959c700 1 -- 172.21.15.23:0/1687527623 learned_addr learned my addr 172.21.15.23:0/1687527623 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:04.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f4798d9b700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47940c40b0 0x7f479415b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:04.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.917+0000 7f4793fff700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f47940c9fd0 0x7f4794157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:04.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f4798d9b700 1 -- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f47940c9fd0 msgr2=0x7f4794157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:04.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f4798d9b700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f47940c9fd0 0x7f4794157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:04.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f4798d9b700 1 -- 172.21.15.23:0/1687527623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4794152ce0 msgr2=0x7f4794157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f4798d9b700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4794152ce0 0x7f4794157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f4798d9b700 1 -- 172.21.15.23:0/1687527623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880094a0 con 0x7f47940c40b0 2023-06-22T08:11:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f479959c700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4794152ce0 0x7f4794157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:11:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f4798d9b700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47940c40b0 0x7f479415b330 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4788016000 tx=0x7f4788004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f47917fa700 1 -- 172.21.15.23:0/1687527623 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478801a7e0 con 0x7f47940c40b0 2023-06-22T08:11:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f47917fa700 1 -- 172.21.15.23:0/1687527623 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f478801a980 con 0x7f47940c40b0 2023-06-22T08:11:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f47917fa700 1 -- 172.21.15.23:0/1687527623 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478801ad30 con 0x7f47940c40b0 2023-06-22T08:11:04.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.918+0000 7f479afff700 1 -- 172.21.15.23:0/1687527623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47941ee5a0 con 0x7f47940c40b0 2023-06-22T08:11:04.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.919+0000 7f479afff700 1 -- 172.21.15.23:0/1687527623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47941eeb60 con 0x7f47940c40b0 2023-06-22T08:11:04.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.919+0000 7f479afff700 1 -- 172.21.15.23:0/1687527623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47941ef040 con 0x7f47940c40b0 2023-06-22T08:11:04.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.921+0000 7f47917fa700 1 -- 172.21.15.23:0/1687527623 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4788022070 con 0x7f47940c40b0 2023-06-22T08:11:04.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.922+0000 7f47917fa700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f477806ace0 0x7f477806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:04.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.922+0000 7f47917fa700 1 -- 172.21.15.23:0/1687527623 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f478808a870 con 0x7f47940c40b0 2023-06-22T08:11:04.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.922+0000 7f4793fff700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f477806ace0 0x7f477806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:04.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.923+0000 7f4793fff700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f477806ace0 0x7f477806d190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f47940b3640 tx=0x7f4780009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:04.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:04.926+0000 7f47917fa700 1 -- 172.21.15.23:0/1687527623 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4788055540 con 0x7f47940c40b0 2023-06-22T08:11:05.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.181+0000 7f479afff700 1 -- 172.21.15.23:0/1687527623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f479404ea20 con 0x7f47940c40b0 2023-06-22T08:11:05.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.182+0000 7f47917fa700 1 -- 172.21.15.23:0/1687527623 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5508 (secure 0 0 0) 0x7f4788029090 con 0x7f47940c40b0 2023-06-22T08:11:05.186 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:05.186 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:04.427789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:05.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 -- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f477806ace0 msgr2=0x7f477806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:05.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f477806ace0 0x7f477806d190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f47940b3640 tx=0x7f4780009040 comp rx=0 tx=0).stop 2023-06-22T08:11:05.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 -- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47940c40b0 msgr2=0x7f479415b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:05.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47940c40b0 0x7f479415b330 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4788016000 tx=0x7f4788004750 comp rx=0 tx=0).stop 2023-06-22T08:11:05.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 -- 172.21.15.23:0/1687527623 shutdown_connections 2023-06-22T08:11:05.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4794152ce0 0x7f4794157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f477806ace0 0x7f477806d190 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47940c40b0 0x7f479415b330 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 --2- 172.21.15.23:0/1687527623 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f47940c9fd0 0x7f4794157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 -- 172.21.15.23:0/1687527623 >> 172.21.15.23:0/1687527623 conn(0x7f479406ae30 msgr2=0x7f47940b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:05.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 -- 172.21.15.23:0/1687527623 shutdown_connections 2023-06-22T08:11:05.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.184+0000 7f479afff700 1 -- 172.21.15.23:0/1687527623 wait complete. 2023-06-22T08:11:05.191 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 86 2023-06-22T08:11:05.207 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-06-22T08:11:05.207 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:11:05.208 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:05.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.403+0000 7fa3807ff700 1 Processor -- start 2023-06-22T08:11:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.403+0000 7fa3807ff700 1 -- start start 2023-06-22T08:11:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa3807ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa378152ce0 0x7fa3780cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa3807ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3780c9fd0 0x7fa37815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa3807ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa378158880 0x7fa378158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa3807ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3781530b0 con 0x7fa3780c9fd0 2023-06-22T08:11:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa3807ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3780cf6f0 con 0x7fa378152ce0 2023-06-22T08:11:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa3807ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37815c370 con 0x7fa378158880 2023-06-22T08:11:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa37dd9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3780c9fd0 0x7fa37815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:05.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa37dd9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3780c9fd0 0x7fa37815c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38970/0 (socket says 172.21.15.23:38970) 2023-06-22T08:11:05.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa37dd9a700 1 -- 172.21.15.23:0/802569622 learned_addr learned my addr 172.21.15.23:0/802569622 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:05.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa37e59b700 1 --2- 172.21.15.23:0/802569622 >> v2:172.21.15.67:3300/0 conn(0x7fa378152ce0 0x7fa3780cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:05.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.404+0000 7fa37ed9c700 1 --2- 172.21.15.23:0/802569622 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa378158880 0x7fa378158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:05.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.405+0000 7fa37dd9a700 1 -- 172.21.15.23:0/802569622 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa378158880 msgr2=0x7fa378158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.405+0000 7fa37dd9a700 1 --2- 172.21.15.23:0/802569622 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa378158880 0x7fa378158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.405+0000 7fa37dd9a700 1 -- 172.21.15.23:0/802569622 >> v2:172.21.15.67:3300/0 conn(0x7fa378152ce0 msgr2=0x7fa3780cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.405+0000 7fa37dd9a700 1 --2- 172.21.15.23:0/802569622 >> v2:172.21.15.67:3300/0 conn(0x7fa378152ce0 0x7fa3780cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.405+0000 7fa37dd9a700 1 -- 172.21.15.23:0/802569622 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37815bda0 con 0x7fa3780c9fd0 2023-06-22T08:11:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.405+0000 7fa37dd9a700 1 --2- 172.21.15.23:0/802569622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3780c9fd0 0x7fa37815c800 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa37808ba10 tx=0x7fa368009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c767c99dc3deae server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.405+0000 7fa37d599700 1 -- 172.21.15.23:0/802569622 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368011830 con 0x7fa3780c9fd0 2023-06-22T08:11:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.405+0000 7fa37d599700 1 -- 172.21.15.23:0/802569622 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa368011990 con 0x7fa3780c9fd0 2023-06-22T08:11:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.405+0000 7fa37d599700 1 -- 172.21.15.23:0/802569622 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368011d20 con 0x7fa3780c9fd0 2023-06-22T08:11:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.406+0000 7fa3807ff700 1 -- 172.21.15.23:0/802569622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3780c9fd0 msgr2=0x7fa37815c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.406+0000 7fa3807ff700 1 --2- 172.21.15.23:0/802569622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3780c9fd0 0x7fa37815c800 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa37808ba10 tx=0x7fa368009bd0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.406+0000 7fa3807ff700 1 -- 172.21.15.23:0/802569622 shutdown_connections 2023-06-22T08:11:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.406+0000 7fa3807ff700 1 --2- 172.21.15.23:0/802569622 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa378158880 0x7fa378158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.406+0000 7fa3807ff700 1 --2- 172.21.15.23:0/802569622 >> v2:172.21.15.67:3300/0 conn(0x7fa378152ce0 0x7fa3780cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.406+0000 7fa3807ff700 1 --2- 172.21.15.23:0/802569622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3780c9fd0 0x7fa37815c800 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.406+0000 7fa3807ff700 1 -- 172.21.15.23:0/802569622 >> 172.21.15.23:0/802569622 conn(0x7fa37806ae30 msgr2=0x7fa37806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.406+0000 7fa3807ff700 1 -- 172.21.15.23:0/802569622 shutdown_connections 2023-06-22T08:11:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.406+0000 7fa3807ff700 1 -- 172.21.15.23:0/802569622 wait complete. 2023-06-22T08:11:05.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.408+0000 7fa3807ff700 1 Processor -- start 2023-06-22T08:11:05.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.408+0000 7fa3807ff700 1 -- start start 2023-06-22T08:11:05.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.409+0000 7fa3807ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3780c9fd0 0x7fa3781ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:05.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.409+0000 7fa3807ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa378152ce0 0x7fa378150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:05.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.409+0000 7fa3807ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa378158880 0x7fa37815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:05.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.409+0000 7fa3807ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3781ef390 con 0x7fa378158880 2023-06-22T08:11:05.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.409+0000 7fa3807ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3781ee910 con 0x7fa3780c9fd0 2023-06-22T08:11:05.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.409+0000 7fa3807ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa378150990 con 0x7fa378152ce0 2023-06-22T08:11:05.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37ed9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa378158880 0x7fa37815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:05.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37dd9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa378152ce0 0x7fa378150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:05.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37e59b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3780c9fd0 0x7fa3781ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37ed9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa378158880 0x7fa37815aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38982/0 (socket says 172.21.15.23:38982) 2023-06-22T08:11:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37ed9c700 1 -- 172.21.15.23:0/2131134523 learned_addr learned my addr 172.21.15.23:0/2131134523 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37ed9c700 1 -- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa378152ce0 msgr2=0x7fa378150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37ed9c700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa378152ce0 0x7fa378150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37ed9c700 1 -- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3780c9fd0 msgr2=0x7fa3781ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37ed9c700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3780c9fd0 0x7fa3781ee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37ed9c700 1 -- 172.21.15.23:0/2131134523 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368009550 con 0x7fa378158880 2023-06-22T08:11:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa37ed9c700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa378158880 0x7fa37815aeb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa370009d30 tx=0x7fa37000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.410+0000 7fa36effd700 1 -- 172.21.15.23:0/2131134523 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37000cce0 con 0x7fa378158880 2023-06-22T08:11:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.411+0000 7fa36effd700 1 -- 172.21.15.23:0/2131134523 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37000f040 con 0x7fa378158880 2023-06-22T08:11:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.411+0000 7fa3807ff700 1 -- 172.21.15.23:0/2131134523 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3780cf500 con 0x7fa378158880 2023-06-22T08:11:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.411+0000 7fa36effd700 1 -- 172.21.15.23:0/2131134523 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa370009e80 con 0x7fa378158880 2023-06-22T08:11:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.411+0000 7fa3807ff700 1 -- 172.21.15.23:0/2131134523 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3781eedd0 con 0x7fa378158880 2023-06-22T08:11:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.412+0000 7fa3807ff700 1 -- 172.21.15.23:0/2131134523 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37808bf10 con 0x7fa378158880 2023-06-22T08:11:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.413+0000 7fa36effd700 1 -- 172.21.15.23:0/2131134523 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa370014480 con 0x7fa378158880 2023-06-22T08:11:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.414+0000 7fa36effd700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa35c06ad10 0x7fa35c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:05.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.414+0000 7fa36effd700 1 -- 172.21.15.23:0/2131134523 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7fa37008a200 con 0x7fa378158880 2023-06-22T08:11:05.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.414+0000 7fa37e59b700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa35c06ad10 0x7fa35c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:05.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.415+0000 7fa37e59b700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa35c06ad10 0x7fa35c06d1c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa374009be0 tx=0x7fa374009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:05.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.418+0000 7fa36effd700 1 -- 172.21.15.23:0/2131134523 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa370054c80 con 0x7fa378158880 2023-06-22T08:11:05.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.671+0000 7fa3807ff700 1 -- 172.21.15.23:0/2131134523 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa37804ea20 con 0x7fa378158880 2023-06-22T08:11:05.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.672+0000 7fa36effd700 1 -- 172.21.15.23:0/2131134523 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5508 (secure 0 0 0) 0x7fa370018090 con 0x7fa378158880 2023-06-22T08:11:05.677 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:05.677 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:04.427789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:05.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.674+0000 7fa3807ff700 1 -- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa35c06ad10 msgr2=0x7fa35c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:05.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.674+0000 7fa3807ff700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa35c06ad10 0x7fa35c06d1c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa374009be0 tx=0x7fa374009210 comp rx=0 tx=0).stop 2023-06-22T08:11:05.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.674+0000 7fa3807ff700 1 -- 172.21.15.23:0/2131134523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa378158880 msgr2=0x7fa37815aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:05.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.674+0000 7fa3807ff700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa378158880 0x7fa37815aeb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa370009d30 tx=0x7fa37000c3a0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.674+0000 7fa3807ff700 1 -- 172.21.15.23:0/2131134523 shutdown_connections 2023-06-22T08:11:05.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.674+0000 7fa3807ff700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa378158880 0x7fa37815aeb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.674+0000 7fa3807ff700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa35c06ad10 0x7fa35c06d1c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.674+0000 7fa3807ff700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3780c9fd0 0x7fa3781ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.674+0000 7fa3807ff700 1 --2- 172.21.15.23:0/2131134523 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa378152ce0 0x7fa378150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:05.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.674+0000 7fa3807ff700 1 -- 172.21.15.23:0/2131134523 >> 172.21.15.23:0/2131134523 conn(0x7fa37806ae30 msgr2=0x7fa3780c4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:05.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.675+0000 7fa3807ff700 1 -- 172.21.15.23:0/2131134523 shutdown_connections 2023-06-22T08:11:05.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:05.675+0000 7fa3807ff700 1 -- 172.21.15.23:0/2131134523 wait complete. 2023-06-22T08:11:05.681 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 86 2023-06-22T08:11:05.697 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 86, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 86, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5060": { "addr": "172.21.15.67:6837/1551460358", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1551460358, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1551460358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5060, "incarnation": 46, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 31 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 26 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:11:04.427789+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 5551, "mds_3": 5060, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:11:07.698 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:07.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.905+0000 7fb5acfff700 1 Processor -- start 2023-06-22T08:11:07.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.905+0000 7fb5acfff700 1 -- start start 2023-06-22T08:11:07.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.906+0000 7fb5acfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb5a8152ce0 0x7fb5a80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:07.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.906+0000 7fb5acfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5a80c9fd0 0x7fb5a815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:07.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.906+0000 7fb5acfff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a815c370 con 0x7fb5a8158880 2023-06-22T08:11:07.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.906+0000 7fb5acfff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a81530b0 con 0x7fb5a8152ce0 2023-06-22T08:11:07.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.906+0000 7fb5acfff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a80cf6f0 con 0x7fb5a80c9fd0 2023-06-22T08:11:07.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.906+0000 7fb5a659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb5a8152ce0 0x7fb5a80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:07.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.907+0000 7fb5a5d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5a80c9fd0 0x7fb5a815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:07.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.907+0000 7fb5a659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb5a8152ce0 0x7fb5a80cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51848/0 (socket says 172.21.15.23:51848) 2023-06-22T08:11:07.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.907+0000 7fb5a659c700 1 -- 172.21.15.23:0/3203011725 learned_addr learned my addr 172.21.15.23:0/3203011725 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:07.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.907+0000 7fb5a659c700 1 -- 172.21.15.23:0/3203011725 >> v2:172.21.15.67:3300/0 conn(0x7fb5a80c9fd0 msgr2=0x7fb5a815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:07.919 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.907+0000 7fb5a659c700 1 --2- 172.21.15.23:0/3203011725 >> v2:172.21.15.67:3300/0 conn(0x7fb5a80c9fd0 0x7fb5a815c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:07.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.907+0000 7fb5a659c700 1 -- 172.21.15.23:0/3203011725 >> v1:172.21.15.67:6789/0 conn(0x7fb5a8158880 legacy=0x7fb5a8158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:07.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.907+0000 7fb5a659c700 1 -- 172.21.15.23:0/3203011725 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a815bda0 con 0x7fb5a8152ce0 2023-06-22T08:11:07.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.907+0000 7fb5a659c700 1 --2- 172.21.15.23:0/3203011725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 0x7fb5a80cf1b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80cc380 tx=0x7fb598009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e4a0c1e65d2c05 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:07.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.907+0000 7fb5a559a700 1 -- 172.21.15.23:0/3203011725 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598011830 con 0x7fb5a8152ce0 2023-06-22T08:11:07.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.907+0000 7fb5a559a700 1 -- 172.21.15.23:0/3203011725 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb598011990 con 0x7fb5a8152ce0 2023-06-22T08:11:07.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.908+0000 7fb5a559a700 1 -- 172.21.15.23:0/3203011725 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598011d20 con 0x7fb5a8152ce0 2023-06-22T08:11:07.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.908+0000 7fb5acfff700 1 -- 172.21.15.23:0/3203011725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 msgr2=0x7fb5a80cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:07.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.908+0000 7fb5acfff700 1 --2- 172.21.15.23:0/3203011725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 0x7fb5a80cf1b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80cc380 tx=0x7fb598009bd0 comp rx=0 tx=0).stop 2023-06-22T08:11:07.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.908+0000 7fb5acfff700 1 -- 172.21.15.23:0/3203011725 shutdown_connections 2023-06-22T08:11:07.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.908+0000 7fb5acfff700 1 --2- 172.21.15.23:0/3203011725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 0x7fb5a80cf1b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:07.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.908+0000 7fb5acfff700 1 --2- 172.21.15.23:0/3203011725 >> v2:172.21.15.67:3300/0 conn(0x7fb5a80c9fd0 0x7fb5a815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:07.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.908+0000 7fb5acfff700 1 -- 172.21.15.23:0/3203011725 >> 172.21.15.23:0/3203011725 conn(0x7fb5a806ae30 msgr2=0x7fb5a806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:07.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.908+0000 7fb5acfff700 1 -- 172.21.15.23:0/3203011725 shutdown_connections 2023-06-22T08:11:07.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.908+0000 7fb5acfff700 1 -- 172.21.15.23:0/3203011725 wait complete. 2023-06-22T08:11:07.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.910+0000 7fb5acfff700 1 Processor -- start 2023-06-22T08:11:07.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.910+0000 7fb5acfff700 1 -- start start 2023-06-22T08:11:07.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5acfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5a80c9fd0 0x7fb5a81eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:07.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5acfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 0x7fb5a80ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:07.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5acfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8158880 0x7fb5a815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:07.922 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5acfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a80ce1c0 con 0x7fb5a8152ce0 2023-06-22T08:11:07.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5acfff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a81ef010 con 0x7fb5a8158880 2023-06-22T08:11:07.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5acfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a80cc9c0 con 0x7fb5a80c9fd0 2023-06-22T08:11:07.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5a5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 0x7fb5a80ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:07.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5a659c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5a80c9fd0 0x7fb5a81eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:07.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5a5d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 0x7fb5a80ce5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51856/0 (socket says 172.21.15.23:51856) 2023-06-22T08:11:07.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5a5d9b700 1 -- 172.21.15.23:0/325205522 learned_addr learned my addr 172.21.15.23:0/325205522 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:07.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.911+0000 7fb5a6d9d700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8158880 0x7fb5a815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:07.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb5a5d9b700 1 -- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5a80c9fd0 msgr2=0x7fb5a81eead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:07.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb5a5d9b700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5a80c9fd0 0x7fb5a81eead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:07.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb5a5d9b700 1 -- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8158880 msgr2=0x7fb5a815aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:07.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb5a5d9b700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8158880 0x7fb5a815aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:07.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb5a5d9b700 1 -- 172.21.15.23:0/325205522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb598009550 con 0x7fb5a8152ce0 2023-06-22T08:11:07.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb5a5d9b700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 0x7fb5a80ce5e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00e9e0 tx=0x7fb59c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:07.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb596ffd700 1 -- 172.21.15.23:0/325205522 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c009710 con 0x7fb5a8152ce0 2023-06-22T08:11:07.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb596ffd700 1 -- 172.21.15.23:0/325205522 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c00ee80 con 0x7fb5a8152ce0 2023-06-22T08:11:07.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb596ffd700 1 -- 172.21.15.23:0/325205522 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c00ca50 con 0x7fb5a8152ce0 2023-06-22T08:11:07.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb5acfff700 1 -- 172.21.15.23:0/325205522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a80ccc10 con 0x7fb5a8152ce0 2023-06-22T08:11:07.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.912+0000 7fb5acfff700 1 -- 172.21.15.23:0/325205522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a80cd1d0 con 0x7fb5a8152ce0 2023-06-22T08:11:07.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.914+0000 7fb5acfff700 1 -- 172.21.15.23:0/325205522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a80ce300 con 0x7fb5a8152ce0 2023-06-22T08:11:07.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.915+0000 7fb596ffd700 1 -- 172.21.15.23:0/325205522 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb59c00cbb0 con 0x7fb5a8152ce0 2023-06-22T08:11:07.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.915+0000 7fb596ffd700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb58806ad10 0x7fb58806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:07.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.916+0000 7fb596ffd700 1 -- 172.21.15.23:0/325205522 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7fb59c089c70 con 0x7fb5a8152ce0 2023-06-22T08:11:07.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.916+0000 7fb5a659c700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb58806ad10 0x7fb58806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:07.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.916+0000 7fb5a659c700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb58806ad10 0x7fb58806d1c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb598000c00 tx=0x7fb598008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:07.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:07.920+0000 7fb596ffd700 1 -- 172.21.15.23:0/325205522 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb5a80ce300 con 0x7fb5a8152ce0 2023-06-22T08:11:08.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.175+0000 7fb5acfff700 1 -- 172.21.15.23:0/325205522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5a80ce300 con 0x7fb5a8152ce0 2023-06-22T08:11:08.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.176+0000 7fb596ffd700 1 -- 172.21.15.23:0/325205522 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5508 (secure 0 0 0) 0x7fb5a80ce300 con 0x7fb5a8152ce0 2023-06-22T08:11:08.181 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:08.182 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:04.427789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:08.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.178+0000 7fb5acfff700 1 -- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb58806ad10 msgr2=0x7fb58806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:08.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.178+0000 7fb5acfff700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb58806ad10 0x7fb58806d1c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb598000c00 tx=0x7fb598008030 comp rx=0 tx=0).stop 2023-06-22T08:11:08.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.178+0000 7fb5acfff700 1 -- 172.21.15.23:0/325205522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 msgr2=0x7fb5a80ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:08.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.178+0000 7fb5acfff700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 0x7fb5a80ce5e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00e9e0 tx=0x7fb59c00ecf0 comp rx=0 tx=0).stop 2023-06-22T08:11:08.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.179+0000 7fb5acfff700 1 -- 172.21.15.23:0/325205522 shutdown_connections 2023-06-22T08:11:08.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.179+0000 7fb5acfff700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb58806ad10 0x7fb58806d1c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:08.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.179+0000 7fb5acfff700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5a8158880 0x7fb5a815aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:08.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.179+0000 7fb5acfff700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5a80c9fd0 0x7fb5a81eead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:08.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.179+0000 7fb5acfff700 1 --2- 172.21.15.23:0/325205522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a8152ce0 0x7fb5a80ce5e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:08.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.179+0000 7fb5acfff700 1 -- 172.21.15.23:0/325205522 >> 172.21.15.23:0/325205522 conn(0x7fb5a806ae30 msgr2=0x7fb5a80cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:08.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.179+0000 7fb5acfff700 1 -- 172.21.15.23:0/325205522 shutdown_connections 2023-06-22T08:11:08.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:08.179+0000 7fb5acfff700 1 -- 172.21.15.23:0/325205522 wait complete. 2023-06-22T08:11:08.186 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 86 2023-06-22T08:11:08.201 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:11:08.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:11:08.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:11:10.202 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.409+0000 7f39cffff700 1 Processor -- start 2023-06-22T08:11:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.409+0000 7f39cffff700 1 -- start start 2023-06-22T08:11:10.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.410+0000 7f39cffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f39c80cb6b0 0x7f39c8069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:10.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.410+0000 7f39cffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f39c8152e10 0x7f39c81593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.410+0000 7f39cffff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c8158f60 con 0x7f39c80c5740 2023-06-22T08:11:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.410+0000 7f39cffff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c806bd20 con 0x7f39c8152e10 2023-06-22T08:11:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.410+0000 7f39cffff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c80cc450 con 0x7f39c80cb6b0 2023-06-22T08:11:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.411+0000 7f39ce59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f39c8152e10 0x7f39c81593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.411+0000 7f39cd59a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f39c80c5740 0x7f39c8152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:38920/0 (socket says 172.21.15.23:38920) 2023-06-22T08:11:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.411+0000 7f39cd59a700 1 -- 172.21.15.23:0/2204904490 learned_addr learned my addr 172.21.15.23:0/2204904490 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.411+0000 7f39cdd9b700 1 --2- 172.21.15.23:0/2204904490 >> v2:172.21.15.67:3300/0 conn(0x7f39c80cb6b0 0x7f39c8069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.411+0000 7f39ce59c700 1 -- 172.21.15.23:0/2204904490 >> v2:172.21.15.67:3300/0 conn(0x7f39c80cb6b0 msgr2=0x7f39c8069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.411+0000 7f39ce59c700 1 --2- 172.21.15.23:0/2204904490 >> v2:172.21.15.67:3300/0 conn(0x7f39c80cb6b0 0x7f39c8069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.411+0000 7f39ce59c700 1 -- 172.21.15.23:0/2204904490 >> v1:172.21.15.23:6789/0 conn(0x7f39c80c5740 legacy=0x7f39c8152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.411+0000 7f39ce59c700 1 -- 172.21.15.23:0/2204904490 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c8158980 con 0x7f39c8152e10 2023-06-22T08:11:10.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.411+0000 7f39ce59c700 1 --2- 172.21.15.23:0/2204904490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c8152e10 0x7f39c81593f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f39c80cc230 tx=0x7f39c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd9940863a9195 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:10.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39ccd99700 1 -- 172.21.15.23:0/2204904490 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c000e070 con 0x7f39c8152e10 2023-06-22T08:11:10.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39ccd99700 1 -- 172.21.15.23:0/2204904490 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c000bb20 con 0x7f39c8152e10 2023-06-22T08:11:10.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39ccd99700 1 -- 172.21.15.23:0/2204904490 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0012430 con 0x7f39c8152e10 2023-06-22T08:11:10.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39cffff700 1 -- 172.21.15.23:0/2204904490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c8152e10 msgr2=0x7f39c81593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:10.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39cffff700 1 --2- 172.21.15.23:0/2204904490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c8152e10 0x7f39c81593f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f39c80cc230 tx=0x7f39c000b330 comp rx=0 tx=0).stop 2023-06-22T08:11:10.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39cffff700 1 -- 172.21.15.23:0/2204904490 shutdown_connections 2023-06-22T08:11:10.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39cffff700 1 --2- 172.21.15.23:0/2204904490 >> v2:172.21.15.67:3300/0 conn(0x7f39c80cb6b0 0x7f39c8069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:10.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39cffff700 1 --2- 172.21.15.23:0/2204904490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c8152e10 0x7f39c81593f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:10.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39cffff700 1 -- 172.21.15.23:0/2204904490 >> 172.21.15.23:0/2204904490 conn(0x7f39c8070d80 msgr2=0x7f39c8071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:10.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39cffff700 1 -- 172.21.15.23:0/2204904490 shutdown_connections 2023-06-22T08:11:10.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.412+0000 7f39cffff700 1 -- 172.21.15.23:0/2204904490 wait complete. 2023-06-22T08:11:10.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.414+0000 7f39cffff700 1 Processor -- start 2023-06-22T08:11:10.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.414+0000 7f39cffff700 1 -- start start 2023-06-22T08:11:10.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39cffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c80cb6b0 0x7f39c80c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:10.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39cffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f39c8152e10 0x7f39c80c3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:10.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39cffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c80c4510 0x7f39c80c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:10.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39cffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c8152210 con 0x7f39c80cb6b0 2023-06-22T08:11:10.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39cffff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c80c4eb0 con 0x7f39c80c4510 2023-06-22T08:11:10.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39cffff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c80c5190 con 0x7f39c8152e10 2023-06-22T08:11:10.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39cdd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c80cb6b0 0x7f39c80c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:10.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39cdd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c80cb6b0 0x7f39c80c8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51872/0 (socket says 172.21.15.23:51872) 2023-06-22T08:11:10.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39cdd9b700 1 -- 172.21.15.23:0/567079304 learned_addr learned my addr 172.21.15.23:0/567079304 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:10.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39cd59a700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f39c8152e10 0x7f39c80c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:10.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.415+0000 7f39ce59c700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c80c4510 0x7f39c80c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:10.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.416+0000 7f39cdd9b700 1 -- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f39c8152e10 msgr2=0x7f39c80c3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:10.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.416+0000 7f39cdd9b700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f39c8152e10 0x7f39c80c3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:10.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.416+0000 7f39cdd9b700 1 -- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c80c4510 msgr2=0x7f39c80c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:10.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.416+0000 7f39cdd9b700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c80c4510 0x7f39c80c7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:10.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.416+0000 7f39cdd9b700 1 -- 172.21.15.23:0/567079304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c0009a70 con 0x7f39c80cb6b0 2023-06-22T08:11:10.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.416+0000 7f39cdd9b700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c80cb6b0 0x7f39c80c8720 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f39c400e9b0 tx=0x7f39c400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:10.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.416+0000 7f39be7fc700 1 -- 172.21.15.23:0/567079304 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4009710 con 0x7f39c80cb6b0 2023-06-22T08:11:10.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.416+0000 7f39be7fc700 1 -- 172.21.15.23:0/567079304 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c400ee50 con 0x7f39c80cb6b0 2023-06-22T08:11:10.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.416+0000 7f39be7fc700 1 -- 172.21.15.23:0/567079304 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c400ca50 con 0x7f39c80cb6b0 2023-06-22T08:11:10.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.416+0000 7f39cffff700 1 -- 172.21.15.23:0/567079304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c80c5470 con 0x7f39c80cb6b0 2023-06-22T08:11:10.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.417+0000 7f39cffff700 1 -- 172.21.15.23:0/567079304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c8151b50 con 0x7f39c80cb6b0 2023-06-22T08:11:10.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.418+0000 7f39cffff700 1 -- 172.21.15.23:0/567079304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c804ea20 con 0x7f39c80cb6b0 2023-06-22T08:11:10.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.419+0000 7f39be7fc700 1 -- 172.21.15.23:0/567079304 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f39c400cbb0 con 0x7f39c80cb6b0 2023-06-22T08:11:10.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.420+0000 7f39be7fc700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f39ac06ad10 0x7f39ac06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:10.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.420+0000 7f39be7fc700 1 -- 172.21.15.23:0/567079304 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f39c4089930 con 0x7f39c80cb6b0 2023-06-22T08:11:10.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.420+0000 7f39cd59a700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f39ac06ad10 0x7f39ac06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:10.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.420+0000 7f39cd59a700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f39ac06ad10 0x7f39ac06d1c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f39c8144aa0 tx=0x7f39b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:10.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.424+0000 7f39be7fc700 1 -- 172.21.15.23:0/567079304 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f39c4057f70 con 0x7f39c80cb6b0 2023-06-22T08:11:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.692+0000 7f39cffff700 1 -- 172.21.15.23:0/567079304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39c80c6960 con 0x7f39c80cb6b0 2023-06-22T08:11:10.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.693+0000 7f39be7fc700 1 -- 172.21.15.23:0/567079304 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5508 (secure 0 0 0) 0x7f39c400ce60 con 0x7f39c80cb6b0 2023-06-22T08:11:10.698 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:10.699 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:04.427789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:10.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.695+0000 7f39cffff700 1 -- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f39ac06ad10 msgr2=0x7f39ac06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:10.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.695+0000 7f39cffff700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f39ac06ad10 0x7f39ac06d1c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f39c8144aa0 tx=0x7f39b8009040 comp rx=0 tx=0).stop 2023-06-22T08:11:10.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.695+0000 7f39cffff700 1 -- 172.21.15.23:0/567079304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c80cb6b0 msgr2=0x7f39c80c8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:10.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.695+0000 7f39cffff700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c80cb6b0 0x7f39c80c8720 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f39c400e9b0 tx=0x7f39c400ecc0 comp rx=0 tx=0).stop 2023-06-22T08:11:10.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.695+0000 7f39cffff700 1 -- 172.21.15.23:0/567079304 shutdown_connections 2023-06-22T08:11:10.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.695+0000 7f39cffff700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f39ac06ad10 0x7f39ac06d1c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:10.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.695+0000 7f39cffff700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39c80c4510 0x7f39c80c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.695+0000 7f39cffff700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39c80cb6b0 0x7f39c80c8720 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.695+0000 7f39cffff700 1 --2- 172.21.15.23:0/567079304 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f39c8152e10 0x7f39c80c3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.695+0000 7f39cffff700 1 -- 172.21.15.23:0/567079304 >> 172.21.15.23:0/567079304 conn(0x7f39c8070d80 msgr2=0x7f39c806a590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:10.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.696+0000 7f39cffff700 1 -- 172.21.15.23:0/567079304 shutdown_connections 2023-06-22T08:11:10.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:10.696+0000 7f39cffff700 1 -- 172.21.15.23:0/567079304 wait complete. 2023-06-22T08:11:10.704 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 86 2023-06-22T08:11:10.718 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:11:12.719 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:12.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.925+0000 7ff44f7ff700 1 Processor -- start 2023-06-22T08:11:12.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.925+0000 7ff44f7ff700 1 -- start start 2023-06-22T08:11:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.926+0000 7ff44f7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff448152a90 0x7ff4480c9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.926+0000 7ff44f7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4480c9680 0x7ff4480c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:12.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.926+0000 7ff44f7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4480c4c50 con 0x7ff4480ca470 2023-06-22T08:11:12.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.926+0000 7ff44f7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff448158ac0 con 0x7ff448152a90 2023-06-22T08:11:12.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.926+0000 7ff44f7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4480ca8f0 con 0x7ff4480c9680 2023-06-22T08:11:12.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.926+0000 7ff44d59b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff448152a90 0x7ff4480c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:12.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.926+0000 7ff44dd9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff4480ca470 0x7ff44815bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:38934/0 (socket says 172.21.15.23:38934) 2023-06-22T08:11:12.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.926+0000 7ff44dd9c700 1 -- 172.21.15.23:0/2229800031 learned_addr learned my addr 172.21.15.23:0/2229800031 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:12.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.926+0000 7ff44cd9a700 1 --2- 172.21.15.23:0/2229800031 >> v2:172.21.15.67:3300/0 conn(0x7ff4480c9680 0x7ff4480c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:12.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.927+0000 7ff44d59b700 1 -- 172.21.15.23:0/2229800031 >> v2:172.21.15.67:3300/0 conn(0x7ff4480c9680 msgr2=0x7ff4480c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:12.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.927+0000 7ff44d59b700 1 --2- 172.21.15.23:0/2229800031 >> v2:172.21.15.67:3300/0 conn(0x7ff4480c9680 0x7ff4480c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:12.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.927+0000 7ff44d59b700 1 -- 172.21.15.23:0/2229800031 >> v1:172.21.15.23:6789/0 conn(0x7ff4480ca470 legacy=0x7ff44815bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:12.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.927+0000 7ff44d59b700 1 -- 172.21.15.23:0/2229800031 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44815cba0 con 0x7ff448152a90 2023-06-22T08:11:12.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.927+0000 7ff43ffff700 1 -- 172.21.15.23:0/2229800031 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076997950 0 0) 0x7ff4480c4c50 con 0x7ff4480ca470 2023-06-22T08:11:12.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.927+0000 7ff44d59b700 1 --2- 172.21.15.23:0/2229800031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448152a90 0x7ff4480c9140 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff444009840 tx=0x7ff444009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc6064ca6789f05 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:12.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.927+0000 7ff43ffff700 1 -- 172.21.15.23:0/2229800031 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444011830 con 0x7ff448152a90 2023-06-22T08:11:12.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.927+0000 7ff43ffff700 1 -- 172.21.15.23:0/2229800031 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff444011990 con 0x7ff448152a90 2023-06-22T08:11:12.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.927+0000 7ff43ffff700 1 -- 172.21.15.23:0/2229800031 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444011d20 con 0x7ff448152a90 2023-06-22T08:11:12.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.928+0000 7ff44f7ff700 1 -- 172.21.15.23:0/2229800031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448152a90 msgr2=0x7ff4480c9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:12.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.928+0000 7ff44f7ff700 1 --2- 172.21.15.23:0/2229800031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448152a90 0x7ff4480c9140 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff444009840 tx=0x7ff444009b50 comp rx=0 tx=0).stop 2023-06-22T08:11:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.928+0000 7ff44f7ff700 1 -- 172.21.15.23:0/2229800031 shutdown_connections 2023-06-22T08:11:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.928+0000 7ff44f7ff700 1 --2- 172.21.15.23:0/2229800031 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448152a90 0x7ff4480c9140 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.928+0000 7ff44f7ff700 1 --2- 172.21.15.23:0/2229800031 >> v2:172.21.15.67:3300/0 conn(0x7ff4480c9680 0x7ff4480c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.928+0000 7ff44f7ff700 1 -- 172.21.15.23:0/2229800031 >> 172.21.15.23:0/2229800031 conn(0x7ff44806ace0 msgr2=0x7ff44806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.928+0000 7ff44f7ff700 1 -- 172.21.15.23:0/2229800031 shutdown_connections 2023-06-22T08:11:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.928+0000 7ff44f7ff700 1 -- 172.21.15.23:0/2229800031 wait complete. 2023-06-22T08:11:12.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.930+0000 7ff44f7ff700 1 Processor -- start 2023-06-22T08:11:12.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.931+0000 7ff44f7ff700 1 -- start start 2023-06-22T08:11:12.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44f7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c9680 0x7ff4480cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:12.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44f7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4480ca470 0x7ff44814e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44f7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448152a90 0x7ff44814c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44f7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481ee320 con 0x7ff448152a90 2023-06-22T08:11:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44f7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481ee5c0 con 0x7ff4480c9680 2023-06-22T08:11:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44f7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481ee8a0 con 0x7ff4480ca470 2023-06-22T08:11:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44d59b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c9680 0x7ff4480cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44cd9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4480ca470 0x7ff44814e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44d59b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c9680 0x7ff4480cc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50420/0 (socket says 172.21.15.23:50420) 2023-06-22T08:11:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44d59b700 1 -- 172.21.15.23:0/640372885 learned_addr learned my addr 172.21.15.23:0/640372885 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.932+0000 7ff44dd9c700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448152a90 0x7ff44814c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff44d59b700 1 -- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4480ca470 msgr2=0x7ff44814e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff44d59b700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4480ca470 0x7ff44814e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff44d59b700 1 -- 172.21.15.23:0/640372885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448152a90 msgr2=0x7ff44814c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff44d59b700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448152a90 0x7ff44814c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff44d59b700 1 -- 172.21.15.23:0/640372885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4440094a0 con 0x7ff4480c9680 2023-06-22T08:11:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff44dd9c700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448152a90 0x7ff44814c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:11:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff44d59b700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c9680 0x7ff4480cc610 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff4440090b0 tx=0x7ff444004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff43dffb700 1 -- 172.21.15.23:0/640372885 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44401a6b0 con 0x7ff4480c9680 2023-06-22T08:11:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff43dffb700 1 -- 172.21.15.23:0/640372885 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44401a810 con 0x7ff4480c9680 2023-06-22T08:11:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff44f7ff700 1 -- 172.21.15.23:0/640372885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4481eeb80 con 0x7ff4480c9680 2023-06-22T08:11:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.933+0000 7ff43dffb700 1 -- 172.21.15.23:0/640372885 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44401aba0 con 0x7ff4480c9680 2023-06-22T08:11:12.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.934+0000 7ff44f7ff700 1 -- 172.21.15.23:0/640372885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4481ef140 con 0x7ff4480c9680 2023-06-22T08:11:12.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.934+0000 7ff44f7ff700 1 -- 172.21.15.23:0/640372885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4480b3310 con 0x7ff4480c9680 2023-06-22T08:11:12.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.936+0000 7ff43dffb700 1 -- 172.21.15.23:0/640372885 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff444022070 con 0x7ff4480c9680 2023-06-22T08:11:12.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.937+0000 7ff43dffb700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ad10 0x7ff42c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:12.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.937+0000 7ff43dffb700 1 -- 172.21.15.23:0/640372885 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7ff44408ac10 con 0x7ff4480c9680 2023-06-22T08:11:12.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.937+0000 7ff44cd9a700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ad10 0x7ff42c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:12.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.937+0000 7ff44cd9a700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ad10 0x7ff42c06d1c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff4480b4a70 tx=0x7ff438009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:12.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:12.945+0000 7ff43dffb700 1 -- 172.21.15.23:0/640372885 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff4440591a0 con 0x7ff4480c9680 2023-06-22T08:11:13.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.195+0000 7ff44f7ff700 1 -- 172.21.15.23:0/640372885 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4480b1e00 con 0x7ff4480c9680 2023-06-22T08:11:13.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.198+0000 7ff43dffb700 1 -- 172.21.15.23:0/640372885 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5508 (secure 0 0 0) 0x7ff444029070 con 0x7ff4480c9680 2023-06-22T08:11:13.201 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:13.201 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:04.427789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 -- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ad10 msgr2=0x7ff42c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ad10 0x7ff42c06d1c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff4480b4a70 tx=0x7ff438009210 comp rx=0 tx=0).stop 2023-06-22T08:11:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 -- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c9680 msgr2=0x7ff4480cc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c9680 0x7ff4480cc610 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff4440090b0 tx=0x7ff444004750 comp rx=0 tx=0).stop 2023-06-22T08:11:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 -- 172.21.15.23:0/640372885 shutdown_connections 2023-06-22T08:11:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448152a90 0x7ff44814c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff42c06ad10 0x7ff42c06d1c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:13.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480c9680 0x7ff4480cc610 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:13.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 --2- 172.21.15.23:0/640372885 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff4480ca470 0x7ff44814e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:13.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 -- 172.21.15.23:0/640372885 >> 172.21.15.23:0/640372885 conn(0x7ff44806ace0 msgr2=0x7ff44814d860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:13.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 -- 172.21.15.23:0/640372885 shutdown_connections 2023-06-22T08:11:13.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:13.199+0000 7ff44f7ff700 1 -- 172.21.15.23:0/640372885 wait complete. 2023-06-22T08:11:13.210 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 86 2023-06-22T08:11:13.222 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:11:15.223 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:15.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.430+0000 7f107b9ff700 1 Processor -- start 2023-06-22T08:11:15.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.430+0000 7f107b9ff700 1 -- start start 2023-06-22T08:11:15.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.431+0000 7f107b9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f10740c4a30 0x7f10740cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:15.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.431+0000 7f107b9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f10740cb7a0 0x7f107415c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:15.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f107b9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1074159730 con 0x7f1074152a10 2023-06-22T08:11:15.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f107b9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10740c5470 con 0x7f10740c4a30 2023-06-22T08:11:15.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f107b9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10740cbbd0 con 0x7f10740cb7a0 2023-06-22T08:11:15.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f1078f9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f10740c4a30 0x7f10740cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:15.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f107979b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1074152a10 0x7f10740c9200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:38950/0 (socket says 172.21.15.23:38950) 2023-06-22T08:11:15.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f107979b700 1 -- 172.21.15.23:0/1940739231 learned_addr learned my addr 172.21.15.23:0/1940739231 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:15.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f1079f9c700 1 --2- 172.21.15.23:0/1940739231 >> v2:172.21.15.67:3300/0 conn(0x7f10740cb7a0 0x7f107415c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:15.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f1079f9c700 1 -- 172.21.15.23:0/1940739231 >> v2:172.21.15.23:3300/0 conn(0x7f10740c4a30 msgr2=0x7f10740cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:15.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f1079f9c700 1 --2- 172.21.15.23:0/1940739231 >> v2:172.21.15.23:3300/0 conn(0x7f10740c4a30 0x7f10740cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:15.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f1079f9c700 1 -- 172.21.15.23:0/1940739231 >> v1:172.21.15.23:6789/0 conn(0x7f1074152a10 legacy=0x7f10740c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:15.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.432+0000 7f1079f9c700 1 -- 172.21.15.23:0/1940739231 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1074158910 con 0x7f10740cb7a0 2023-06-22T08:11:15.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.433+0000 7f1079f9c700 1 --2- 172.21.15.23:0/1940739231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10740cb7a0 0x7f107415c0b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f107406c300 tx=0x7f106c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c803d95fc96404e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:15.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.434+0000 7f106bfff700 1 -- 172.21.15.23:0/1940739231 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c00e070 con 0x7f10740cb7a0 2023-06-22T08:11:15.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.434+0000 7f106bfff700 1 -- 172.21.15.23:0/1940739231 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c00bb20 con 0x7f10740cb7a0 2023-06-22T08:11:15.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.434+0000 7f106bfff700 1 -- 172.21.15.23:0/1940739231 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c012430 con 0x7f10740cb7a0 2023-06-22T08:11:15.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.435+0000 7f107b9ff700 1 -- 172.21.15.23:0/1940739231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10740cb7a0 msgr2=0x7f107415c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:15.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.435+0000 7f107b9ff700 1 --2- 172.21.15.23:0/1940739231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10740cb7a0 0x7f107415c0b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f107406c300 tx=0x7f106c00b330 comp rx=0 tx=0).stop 2023-06-22T08:11:15.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.435+0000 7f107b9ff700 1 -- 172.21.15.23:0/1940739231 shutdown_connections 2023-06-22T08:11:15.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.435+0000 7f107b9ff700 1 --2- 172.21.15.23:0/1940739231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10740cb7a0 0x7f107415c0b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:15.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.435+0000 7f107b9ff700 1 --2- 172.21.15.23:0/1940739231 >> v2:172.21.15.23:3300/0 conn(0x7f10740c4a30 0x7f10740cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:15.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.435+0000 7f107b9ff700 1 -- 172.21.15.23:0/1940739231 >> 172.21.15.23:0/1940739231 conn(0x7f1074065b90 msgr2=0x7f1074065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:15.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.435+0000 7f107b9ff700 1 -- 172.21.15.23:0/1940739231 shutdown_connections 2023-06-22T08:11:15.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.435+0000 7f107b9ff700 1 -- 172.21.15.23:0/1940739231 wait complete. 2023-06-22T08:11:15.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.437+0000 7f107b9ff700 1 Processor -- start 2023-06-22T08:11:15.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.437+0000 7f107b9ff700 1 -- start start 2023-06-22T08:11:15.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.438+0000 7f107b9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10740c4a30 0x7f107415a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:15.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.438+0000 7f107b9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10740cb7a0 0x7f10740b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:15.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.438+0000 7f107b9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1074152a10 0x7f10740b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:15.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.438+0000 7f107b9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10740b3970 con 0x7f10740c4a30 2023-06-22T08:11:15.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.438+0000 7f107b9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107415b3e0 con 0x7f1074152a10 2023-06-22T08:11:15.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.438+0000 7f107b9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10740b2490 con 0x7f10740cb7a0 2023-06-22T08:11:15.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.438+0000 7f1078f9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10740cb7a0 0x7f10740b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:15.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f1079f9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1074152a10 0x7f10740b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:15.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f107979b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10740c4a30 0x7f107415a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f1078f9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10740cb7a0 0x7f10740b2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:39836/0 (socket says 172.21.15.23:39836) 2023-06-22T08:11:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f1078f9a700 1 -- 172.21.15.23:0/1227830758 learned_addr learned my addr 172.21.15.23:0/1227830758 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f1079f9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1074152a10 0x7f10740b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50430/0 (socket says 172.21.15.23:50430) 2023-06-22T08:11:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f1078f9a700 1 -- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1074152a10 msgr2=0x7f10740b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f1078f9a700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1074152a10 0x7f10740b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f1078f9a700 1 -- 172.21.15.23:0/1227830758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10740c4a30 msgr2=0x7f107415a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f1078f9a700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10740c4a30 0x7f107415a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:15.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f1078f9a700 1 -- 172.21.15.23:0/1227830758 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c009a70 con 0x7f10740cb7a0 2023-06-22T08:11:15.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.439+0000 7f1078f9a700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10740cb7a0 0x7f10740b2920 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f106400e9b0 tx=0x7f106400ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:15.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.443+0000 7f1069ffb700 1 -- 172.21.15.23:0/1227830758 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064009710 con 0x7f10740cb7a0 2023-06-22T08:11:15.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.443+0000 7f1069ffb700 1 -- 172.21.15.23:0/1227830758 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106400ee50 con 0x7f10740cb7a0 2023-06-22T08:11:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.443+0000 7f1069ffb700 1 -- 172.21.15.23:0/1227830758 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106400c9f0 con 0x7f10740cb7a0 2023-06-22T08:11:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.443+0000 7f107b9ff700 1 -- 172.21.15.23:0/1227830758 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10741eed30 con 0x7f10740cb7a0 2023-06-22T08:11:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.443+0000 7f107b9ff700 1 -- 172.21.15.23:0/1227830758 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107415b630 con 0x7f10740cb7a0 2023-06-22T08:11:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.444+0000 7f107b9ff700 1 -- 172.21.15.23:0/1227830758 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107404ea20 con 0x7f10740cb7a0 2023-06-22T08:11:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.445+0000 7f1069ffb700 1 -- 172.21.15.23:0/1227830758 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f106400cb50 con 0x7f10740cb7a0 2023-06-22T08:11:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.446+0000 7f1069ffb700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f105806ad30 0x7f105806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.446+0000 7f1069ffb700 1 -- 172.21.15.23:0/1227830758 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f1064089cb0 con 0x7f10740cb7a0 2023-06-22T08:11:15.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.446+0000 7f107979b700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f105806ad30 0x7f105806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:15.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.447+0000 7f107979b700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f105806ad30 0x7f105806d1e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1070005910 tx=0x7f1070009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:15.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.451+0000 7f1069ffb700 1 -- 172.21.15.23:0/1227830758 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f10640582d0 con 0x7f10740cb7a0 2023-06-22T08:11:15.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.703+0000 7f107b9ff700 1 -- 172.21.15.23:0/1227830758 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10740cbd20 con 0x7f10740cb7a0 2023-06-22T08:11:15.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.704+0000 7f1069ffb700 1 -- 172.21.15.23:0/1227830758 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5508 (secure 0 0 0) 0x7f1064017090 con 0x7f10740cb7a0 2023-06-22T08:11:15.707 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:15.707 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:04.427789+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:15.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 -- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f105806ad30 msgr2=0x7f105806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:15.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f105806ad30 0x7f105806d1e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1070005910 tx=0x7f1070009040 comp rx=0 tx=0).stop 2023-06-22T08:11:15.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 -- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10740cb7a0 msgr2=0x7f10740b2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:15.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10740cb7a0 0x7f10740b2920 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f106400e9b0 tx=0x7f106400ecc0 comp rx=0 tx=0).stop 2023-06-22T08:11:15.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 -- 172.21.15.23:0/1227830758 shutdown_connections 2023-06-22T08:11:15.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f105806ad30 0x7f105806d1e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:15.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1074152a10 0x7f10740b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:15.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10740c4a30 0x7f107415a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:15.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 --2- 172.21.15.23:0/1227830758 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f10740cb7a0 0x7f10740b2920 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:15.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 -- 172.21.15.23:0/1227830758 >> 172.21.15.23:0/1227830758 conn(0x7f1074065b90 msgr2=0x7f10741519a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:15.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 -- 172.21.15.23:0/1227830758 shutdown_connections 2023-06-22T08:11:15.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:15.706+0000 7f107b9ff700 1 -- 172.21.15.23:0/1227830758 wait complete. 2023-06-22T08:11:15.712 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 86 2023-06-22T08:11:15.728 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:11:17.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:17.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.937+0000 7ff8395ff700 1 Processor -- start 2023-06-22T08:11:17.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.937+0000 7ff8395ff700 1 -- start start 2023-06-22T08:11:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.939+0000 7ff8395ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff83414f850 0x7ff8340c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.939+0000 7ff8395ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8340c51a0 con 0x7ff834071070 2023-06-22T08:11:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.939+0000 7ff8395ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8340c44e0 con 0x7ff8340c69e0 2023-06-22T08:11:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.939+0000 7ff8395ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8340c5440 con 0x7ff83414f850 2023-06-22T08:11:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.939+0000 7ff8337fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff834071070 0x7ff83415fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40510/0 (socket says 172.21.15.23:40510) 2023-06-22T08:11:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.939+0000 7ff8337fe700 1 -- 172.21.15.23:0/894266872 learned_addr learned my addr 172.21.15.23:0/894266872 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.939+0000 7ff832ffd700 1 --2- 172.21.15.23:0/894266872 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff83414f850 0x7ff8340c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.939+0000 7ff8327fc700 1 -- 172.21.15.23:0/894266872 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260768511 0 0) 0x7ff8340c51a0 con 0x7ff834071070 2023-06-22T08:11:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.939+0000 7ff8327fc700 1 -- 172.21.15.23:0/894266872 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8200035b0 con 0x7ff834071070 2023-06-22T08:11:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.939+0000 7ff8327fc700 1 -- 172.21.15.23:0/894266872 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032710314 0 0) 0x7ff8340c44e0 con 0x7ff8340c69e0 2023-06-22T08:11:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.940+0000 7ff8327fc700 1 -- 172.21.15.23:0/894266872 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8340c51a0 con 0x7ff8340c69e0 2023-06-22T08:11:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.940+0000 7ff832ffd700 1 -- 172.21.15.23:0/894266872 >> v1:172.21.15.67:6789/0 conn(0x7ff8340c69e0 legacy=0x7ff834155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.940+0000 7ff832ffd700 1 -- 172.21.15.23:0/894266872 >> v1:172.21.15.23:6789/0 conn(0x7ff834071070 legacy=0x7ff83415fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.940+0000 7ff832ffd700 1 -- 172.21.15.23:0/894266872 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8340c5720 con 0x7ff83414f850 2023-06-22T08:11:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.940+0000 7ff832ffd700 1 --2- 172.21.15.23:0/894266872 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff83414f850 0x7ff8340c3080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff824009870 tx=0x7ff824009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4df185c89f479c01 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.940+0000 7ff8327fc700 1 -- 172.21.15.23:0/894266872 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824011830 con 0x7ff83414f850 2023-06-22T08:11:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.940+0000 7ff8327fc700 1 -- 172.21.15.23:0/894266872 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff824011990 con 0x7ff83414f850 2023-06-22T08:11:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.940+0000 7ff8327fc700 1 -- 172.21.15.23:0/894266872 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824011d00 con 0x7ff83414f850 2023-06-22T08:11:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.941+0000 7ff8395ff700 1 -- 172.21.15.23:0/894266872 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff83414f850 msgr2=0x7ff8340c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.941+0000 7ff8395ff700 1 --2- 172.21.15.23:0/894266872 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff83414f850 0x7ff8340c3080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff824009870 tx=0x7ff824009b80 comp rx=0 tx=0).stop 2023-06-22T08:11:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.941+0000 7ff8395ff700 1 -- 172.21.15.23:0/894266872 shutdown_connections 2023-06-22T08:11:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.941+0000 7ff8395ff700 1 --2- 172.21.15.23:0/894266872 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff83414f850 0x7ff8340c3080 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:17.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.941+0000 7ff8395ff700 1 -- 172.21.15.23:0/894266872 >> 172.21.15.23:0/894266872 conn(0x7ff8340ca3f0 msgr2=0x7ff8340ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:17.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.941+0000 7ff8395ff700 1 -- 172.21.15.23:0/894266872 shutdown_connections 2023-06-22T08:11:17.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.941+0000 7ff8395ff700 1 -- 172.21.15.23:0/894266872 wait complete. 2023-06-22T08:11:17.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.942+0000 7ff8395ff700 1 Processor -- start 2023-06-22T08:11:17.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.943+0000 7ff8395ff700 1 -- start start 2023-06-22T08:11:17.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.943+0000 7ff8395ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff834071070 0x7ff834145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:17.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.943+0000 7ff8395ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8340c69e0 0x7ff8341df940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:17.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff8395ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff83414f850 0x7ff8341dfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:17.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff8395ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8341df4f0 con 0x7ff834071070 2023-06-22T08:11:17.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff8395ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8340c8b00 con 0x7ff83414f850 2023-06-22T08:11:17.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff8395ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8341f1430 con 0x7ff8340c69e0 2023-06-22T08:11:17.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff832ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff834071070 0x7ff834145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff832ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff834071070 0x7ff834145640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56588/0 (socket says 172.21.15.23:56588) 2023-06-22T08:11:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff832ffd700 1 -- 172.21.15.23:0/264216415 learned_addr learned my addr 172.21.15.23:0/264216415 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff8337fe700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff83414f850 0x7ff8341dfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff82bfff700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8340c69e0 0x7ff8341df940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff832ffd700 1 -- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8340c69e0 msgr2=0x7ff8341df940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff832ffd700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8340c69e0 0x7ff8341df940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff832ffd700 1 -- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff83414f850 msgr2=0x7ff8341dfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff832ffd700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff83414f850 0x7ff8341dfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff832ffd700 1 -- 172.21.15.23:0/264216415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8240094a0 con 0x7ff834071070 2023-06-22T08:11:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff82bfff700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8340c69e0 0x7ff8341df940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:11:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.944+0000 7ff832ffd700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff834071070 0x7ff834145640 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff824014040 tx=0x7ff824004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.945+0000 7ff82b7fe700 1 -- 172.21.15.23:0/264216415 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824004550 con 0x7ff834071070 2023-06-22T08:11:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.945+0000 7ff82b7fe700 1 -- 172.21.15.23:0/264216415 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82401f070 con 0x7ff834071070 2023-06-22T08:11:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.945+0000 7ff8395ff700 1 -- 172.21.15.23:0/264216415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8341f1680 con 0x7ff834071070 2023-06-22T08:11:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.945+0000 7ff82b7fe700 1 -- 172.21.15.23:0/264216415 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8240038e0 con 0x7ff834071070 2023-06-22T08:11:17.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.945+0000 7ff8395ff700 1 -- 172.21.15.23:0/264216415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8341f1c40 con 0x7ff834071070 2023-06-22T08:11:17.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.946+0000 7ff8395ff700 1 -- 172.21.15.23:0/264216415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8341df790 con 0x7ff834071070 2023-06-22T08:11:17.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.947+0000 7ff82b7fe700 1 -- 172.21.15.23:0/264216415 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff824003a40 con 0x7ff834071070 2023-06-22T08:11:17.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.948+0000 7ff82b7fe700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff81406ace0 0x7ff81406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:17.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.948+0000 7ff82b7fe700 1 -- 172.21.15.23:0/264216415 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7ff824021070 con 0x7ff834071070 2023-06-22T08:11:17.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.948+0000 7ff82bfff700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff81406ace0 0x7ff81406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:17.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.949+0000 7ff82bfff700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff81406ace0 0x7ff81406d190 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff8341ebaf0 tx=0x7ff81c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:17.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:17.953+0000 7ff82b7fe700 1 -- 172.21.15.23:0/264216415 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff824055cb0 con 0x7ff834071070 2023-06-22T08:11:18.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.209+0000 7ff8395ff700 1 -- 172.21.15.23:0/264216415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff83404ea20 con 0x7ff834071070 2023-06-22T08:11:18.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.210+0000 7ff82b7fe700 1 -- 172.21.15.23:0/264216415 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5510 (secure 0 0 0) 0x7ff8240592d0 con 0x7ff834071070 2023-06-22T08:11:18.214 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:18.214 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:16.439390+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:resolve","state_seq":34,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:18.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.212+0000 7ff8395ff700 1 -- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff81406ace0 msgr2=0x7ff81406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:18.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.212+0000 7ff8395ff700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff81406ace0 0x7ff81406d190 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff8341ebaf0 tx=0x7ff81c009040 comp rx=0 tx=0).stop 2023-06-22T08:11:18.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.212+0000 7ff8395ff700 1 -- 172.21.15.23:0/264216415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff834071070 msgr2=0x7ff834145640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:18.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.212+0000 7ff8395ff700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff834071070 0x7ff834145640 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff824014040 tx=0x7ff824004750 comp rx=0 tx=0).stop 2023-06-22T08:11:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.212+0000 7ff8395ff700 1 -- 172.21.15.23:0/264216415 shutdown_connections 2023-06-22T08:11:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.212+0000 7ff8395ff700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff81406ace0 0x7ff81406d190 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.212+0000 7ff8395ff700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff83414f850 0x7ff8341dfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.212+0000 7ff8395ff700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff834071070 0x7ff834145640 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.212+0000 7ff8395ff700 1 --2- 172.21.15.23:0/264216415 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8340c69e0 0x7ff8341df940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.212+0000 7ff8395ff700 1 -- 172.21.15.23:0/264216415 >> 172.21.15.23:0/264216415 conn(0x7ff8340ca3f0 msgr2=0x7ff834070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.213+0000 7ff8395ff700 1 -- 172.21.15.23:0/264216415 shutdown_connections 2023-06-22T08:11:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:18.213+0000 7ff8395ff700 1 -- 172.21.15.23:0/264216415 wait complete. 2023-06-22T08:11:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 87 2023-06-22T08:11:18.236 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 87, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 87, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5060": { "addr": "172.21.15.67:6837/1551460358", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1551460358, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1551460358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5060, "incarnation": 46, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 31 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 26 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:resolve", "state_seq": 34 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:11:16.439390+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 5551, "mds_3": 5060, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:11:20.238 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:20.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.444+0000 7fc363dff700 1 Processor -- start 2023-06-22T08:11:20.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.445+0000 7fc363dff700 1 -- start start 2023-06-22T08:11:20.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.446+0000 7fc363dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc35c0cb6b0 0x7fc35c069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:20.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.446+0000 7fc363dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc35c0c5740 0x7fc35c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:20.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.446+0000 7fc363dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c158f60 con 0x7fc35c0cba80 2023-06-22T08:11:20.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.446+0000 7fc363dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35c06bd20 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.446+0000 7fc363dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0cc450 con 0x7fc35c0c5740 2023-06-22T08:11:20.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.446+0000 7fc361b9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc35c0cb6b0 0x7fc35c069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:20.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.446+0000 7fc36139a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc35c0c5740 0x7fc35c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:20.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.446+0000 7fc361b9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc35c0cb6b0 0x7fc35c069eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56598/0 (socket says 172.21.15.23:56598) 2023-06-22T08:11:20.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.446+0000 7fc361b9b700 1 -- 172.21.15.23:0/1135268890 learned_addr learned my addr 172.21.15.23:0/1135268890 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:20.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.447+0000 7fc361b9b700 1 -- 172.21.15.23:0/1135268890 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc35c0c5740 msgr2=0x7fc35c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:20.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.447+0000 7fc361b9b700 1 --2- 172.21.15.23:0/1135268890 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc35c0c5740 0x7fc35c1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:20.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.447+0000 7fc361b9b700 1 -- 172.21.15.23:0/1135268890 >> v1:172.21.15.67:6789/0 conn(0x7fc35c0cba80 legacy=0x7fc35c152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:20.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.447+0000 7fc361b9b700 1 -- 172.21.15.23:0/1135268890 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c158980 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.447+0000 7fc361b9b700 1 --2- 172.21.15.23:0/1135268890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc35c0cb6b0 0x7fc35c069eb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc3580098f0 tx=0x7fc358009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41903f6a4426dc7d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:20.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.447+0000 7fc360b99700 1 -- 172.21.15.23:0/1135268890 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358011830 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.447+0000 7fc360b99700 1 -- 172.21.15.23:0/1135268890 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc358011990 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.447+0000 7fc360b99700 1 -- 172.21.15.23:0/1135268890 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358011d20 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.448+0000 7fc363dff700 1 -- 172.21.15.23:0/1135268890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc35c0cb6b0 msgr2=0x7fc35c069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:20.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.448+0000 7fc363dff700 1 --2- 172.21.15.23:0/1135268890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc35c0cb6b0 0x7fc35c069eb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc3580098f0 tx=0x7fc358009c00 comp rx=0 tx=0).stop 2023-06-22T08:11:20.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.448+0000 7fc363dff700 1 -- 172.21.15.23:0/1135268890 shutdown_connections 2023-06-22T08:11:20.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.448+0000 7fc363dff700 1 --2- 172.21.15.23:0/1135268890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc35c0cb6b0 0x7fc35c069eb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:20.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.448+0000 7fc363dff700 1 --2- 172.21.15.23:0/1135268890 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc35c0c5740 0x7fc35c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:20.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.448+0000 7fc363dff700 1 -- 172.21.15.23:0/1135268890 >> 172.21.15.23:0/1135268890 conn(0x7fc35c070d80 msgr2=0x7fc35c071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:20.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.448+0000 7fc363dff700 1 -- 172.21.15.23:0/1135268890 shutdown_connections 2023-06-22T08:11:20.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.448+0000 7fc363dff700 1 -- 172.21.15.23:0/1135268890 wait complete. 2023-06-22T08:11:20.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.449+0000 7fc363dff700 1 Processor -- start 2023-06-22T08:11:20.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.450+0000 7fc363dff700 1 -- start start 2023-06-22T08:11:20.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc363dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc35c0cb6b0 0x7fc35c14bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:20.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc363dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc35c0cba80 0x7fc35c0c4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc363dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc35c1564d0 0x7fc35c06a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc363dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c1e29c0 con 0x7fc35c1564d0 2023-06-22T08:11:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc363dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c1ef5d0 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc363dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c1ee700 con 0x7fc35c0cba80 2023-06-22T08:11:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc361b9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc35c0cb6b0 0x7fc35c14bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc36139a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc35c0cba80 0x7fc35c0c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:20.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc361b9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc35c0cb6b0 0x7fc35c14bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:43288/0 (socket says 172.21.15.23:43288) 2023-06-22T08:11:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc361b9b700 1 -- 172.21.15.23:0/155408897 learned_addr learned my addr 172.21.15.23:0/155408897 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc36239c700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc35c1564d0 0x7fc35c06a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc361b9b700 1 -- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc35c0cba80 msgr2=0x7fc35c0c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc361b9b700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc35c0cba80 0x7fc35c0c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc361b9b700 1 -- 172.21.15.23:0/155408897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc35c1564d0 msgr2=0x7fc35c06a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc361b9b700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc35c1564d0 0x7fc35c06a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc361b9b700 1 -- 172.21.15.23:0/155408897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc358009550 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.451+0000 7fc36239c700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc35c1564d0 0x7fc35c06a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:11:20.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.452+0000 7fc361b9b700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc35c0cb6b0 0x7fc35c14bb60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc358009c40 tx=0x7fc358004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:20.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.452+0000 7fc3527fc700 1 -- 172.21.15.23:0/155408897 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358004400 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.452+0000 7fc3527fc700 1 -- 172.21.15.23:0/155408897 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35801f070 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.452+0000 7fc363dff700 1 -- 172.21.15.23:0/155408897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c1ee9c0 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.452+0000 7fc3527fc700 1 -- 172.21.15.23:0/155408897 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3580038e0 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.452+0000 7fc363dff700 1 -- 172.21.15.23:0/155408897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c1eef80 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.453+0000 7fc363dff700 1 -- 172.21.15.23:0/155408897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c04ea20 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.454+0000 7fc3527fc700 1 -- 172.21.15.23:0/155408897 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc358021030 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.455+0000 7fc3527fc700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc34006ace0 0x7fc34006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:20.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.455+0000 7fc3527fc700 1 -- 172.21.15.23:0/155408897 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7fc35808b0f0 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.456+0000 7fc36139a700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc34006ace0 0x7fc34006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:20.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.456+0000 7fc36139a700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc34006ace0 0x7fc34006d190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc35c159510 tx=0x7fc34c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:20.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.459+0000 7fc3527fc700 1 -- 172.21.15.23:0/155408897 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc358055d10 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.710+0000 7fc363dff700 1 -- 172.21.15.23:0/155408897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc35c14ff70 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.711+0000 7fc3527fc700 1 -- 172.21.15.23:0/155408897 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5510 (secure 0 0 0) 0x7fc358015020 con 0x7fc35c0cb6b0 2023-06-22T08:11:20.714 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:20.714 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:16.439390+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:resolve","state_seq":34,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:20.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 -- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc34006ace0 msgr2=0x7fc34006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:20.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc34006ace0 0x7fc34006d190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc35c159510 tx=0x7fc34c009210 comp rx=0 tx=0).stop 2023-06-22T08:11:20.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 -- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc35c0cb6b0 msgr2=0x7fc35c14bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:20.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc35c0cb6b0 0x7fc35c14bb60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc358009c40 tx=0x7fc358004750 comp rx=0 tx=0).stop 2023-06-22T08:11:20.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 -- 172.21.15.23:0/155408897 shutdown_connections 2023-06-22T08:11:20.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc35c1564d0 0x7fc35c06a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:20.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc34006ace0 0x7fc34006d190 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:20.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc35c0cb6b0 0x7fc35c14bb60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:20.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 --2- 172.21.15.23:0/155408897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc35c0cba80 0x7fc35c0c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:20.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 -- 172.21.15.23:0/155408897 >> 172.21.15.23:0/155408897 conn(0x7fc35c070d80 msgr2=0x7fc35c1524d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:20.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 -- 172.21.15.23:0/155408897 shutdown_connections 2023-06-22T08:11:20.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:20.713+0000 7fc363dff700 1 -- 172.21.15.23:0/155408897 wait complete. 2023-06-22T08:11:20.718 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 87 2023-06-22T08:11:20.734 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:11:22.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:22.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.943+0000 7fddaa9ff700 1 Processor -- start 2023-06-22T08:11:22.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.944+0000 7fddaa9ff700 1 -- start start 2023-06-22T08:11:22.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.945+0000 7fddaa9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 0x7fdda4155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:22.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.945+0000 7fddaa9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdda4071070 0x7fdda4155b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:22.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.945+0000 7fddaa9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda4156080 con 0x7fdda414f850 2023-06-22T08:11:22.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.945+0000 7fddaa9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda40c51a0 con 0x7fdda4071070 2023-06-22T08:11:22.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.945+0000 7fddaa9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda40c44e0 con 0x7fdda40c69e0 2023-06-22T08:11:22.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.945+0000 7fdda8f9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdda4071070 0x7fdda4155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:22.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.945+0000 7fdda3fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fdda414f850 0x7fdda40c3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40516/0 (socket says 172.21.15.23:40516) 2023-06-22T08:11:22.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.945+0000 7fdda3fff700 1 -- 172.21.15.23:0/2087126358 learned_addr learned my addr 172.21.15.23:0/2087126358 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:22.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.945+0000 7fdd9b7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 0x7fdda4155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:22.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.946+0000 7fdda8f9c700 1 -- 172.21.15.23:0/2087126358 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 msgr2=0x7fdda4155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:22.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.946+0000 7fdda8f9c700 1 --2- 172.21.15.23:0/2087126358 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 0x7fdda4155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:22.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.946+0000 7fdda8f9c700 1 -- 172.21.15.23:0/2087126358 >> v1:172.21.15.23:6789/0 conn(0x7fdda414f850 legacy=0x7fdda40c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:22.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.946+0000 7fdda8f9c700 1 -- 172.21.15.23:0/2087126358 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda40c5440 con 0x7fdda4071070 2023-06-22T08:11:22.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.946+0000 7fdda37fe700 1 -- 172.21.15.23:0/2087126358 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218146562 0 0) 0x7fdda4156080 con 0x7fdda414f850 2023-06-22T08:11:22.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.946+0000 7fdda8f9c700 1 --2- 172.21.15.23:0/2087126358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdda4071070 0x7fdda4155b40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c009d90 tx=0x7fdd9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ddb44a6f5b999 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:22.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.946+0000 7fdda37fe700 1 -- 172.21.15.23:0/2087126358 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c00e070 con 0x7fdda4071070 2023-06-22T08:11:22.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.946+0000 7fdda37fe700 1 -- 172.21.15.23:0/2087126358 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9c00bb20 con 0x7fdda4071070 2023-06-22T08:11:22.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.946+0000 7fdda37fe700 1 -- 172.21.15.23:0/2087126358 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c012430 con 0x7fdda4071070 2023-06-22T08:11:22.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.947+0000 7fddaa9ff700 1 -- 172.21.15.23:0/2087126358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdda4071070 msgr2=0x7fdda4155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:22.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.947+0000 7fddaa9ff700 1 --2- 172.21.15.23:0/2087126358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdda4071070 0x7fdda4155b40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c009d90 tx=0x7fdd9c00b330 comp rx=0 tx=0).stop 2023-06-22T08:11:22.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.947+0000 7fddaa9ff700 1 -- 172.21.15.23:0/2087126358 shutdown_connections 2023-06-22T08:11:22.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.947+0000 7fddaa9ff700 1 --2- 172.21.15.23:0/2087126358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdda4071070 0x7fdda4155b40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:22.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.947+0000 7fddaa9ff700 1 --2- 172.21.15.23:0/2087126358 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 0x7fdda4155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:22.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.947+0000 7fddaa9ff700 1 -- 172.21.15.23:0/2087126358 >> 172.21.15.23:0/2087126358 conn(0x7fdda40ca3f0 msgr2=0x7fdda40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:22.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.947+0000 7fddaa9ff700 1 -- 172.21.15.23:0/2087126358 shutdown_connections 2023-06-22T08:11:22.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.947+0000 7fddaa9ff700 1 -- 172.21.15.23:0/2087126358 wait complete. 2023-06-22T08:11:22.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.949+0000 7fddaa9ff700 1 Processor -- start 2023-06-22T08:11:22.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.949+0000 7fddaa9ff700 1 -- start start 2023-06-22T08:11:22.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fddaa9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdda4071070 0x7fdda41edfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:22.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fddaa9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 0x7fdda4148190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:22.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fddaa9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdda414f850 0x7fdda41486d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:22.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fddaa9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda41f1540 con 0x7fdda4071070 2023-06-22T08:11:22.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fddaa9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda4147660 con 0x7fdda414f850 2023-06-22T08:11:22.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fddaa9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda4147940 con 0x7fdda40c69e0 2023-06-22T08:11:22.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fdda3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdda4071070 0x7fdda41edfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:22.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fdda3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdda4071070 0x7fdda41edfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56630/0 (socket says 172.21.15.23:56630) 2023-06-22T08:11:22.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fdda3fff700 1 -- 172.21.15.23:0/3526775985 learned_addr learned my addr 172.21.15.23:0/3526775985 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:22.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fdd9b7fe700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 0x7fdda4148190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:22.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fdda8f9c700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdda414f850 0x7fdda41486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:22.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fdd9b7fe700 1 -- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdda414f850 msgr2=0x7fdda41486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:22.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fdd9b7fe700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdda414f850 0x7fdda41486d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:22.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fdd9b7fe700 1 -- 172.21.15.23:0/3526775985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdda4071070 msgr2=0x7fdda41edfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:22.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fdd9b7fe700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdda4071070 0x7fdda41edfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:22.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.950+0000 7fdd9b7fe700 1 -- 172.21.15.23:0/3526775985 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c009a70 con 0x7fdda40c69e0 2023-06-22T08:11:22.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.951+0000 7fdd9b7fe700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 0x7fdda4148190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00e8d0 tx=0x7fdd8c00ebe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:22.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.951+0000 7fdda17fa700 1 -- 172.21.15.23:0/3526775985 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c013680 con 0x7fdda40c69e0 2023-06-22T08:11:22.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.951+0000 7fdda17fa700 1 -- 172.21.15.23:0/3526775985 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c0137e0 con 0x7fdda40c69e0 2023-06-22T08:11:22.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.951+0000 7fdda17fa700 1 -- 172.21.15.23:0/3526775985 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c013b50 con 0x7fdda40c69e0 2023-06-22T08:11:22.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.951+0000 7fddaa9ff700 1 -- 172.21.15.23:0/3526775985 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda41f1790 con 0x7fdda40c69e0 2023-06-22T08:11:22.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.951+0000 7fddaa9ff700 1 -- 172.21.15.23:0/3526775985 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda41f1d50 con 0x7fdda40c69e0 2023-06-22T08:11:22.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.952+0000 7fddaa9ff700 1 -- 172.21.15.23:0/3526775985 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda40c2950 con 0x7fdda40c69e0 2023-06-22T08:11:22.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.953+0000 7fdda17fa700 1 -- 172.21.15.23:0/3526775985 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdd8c013cb0 con 0x7fdda40c69e0 2023-06-22T08:11:22.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.954+0000 7fdda17fa700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd8406ace0 0x7fdd8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:22.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.954+0000 7fdda17fa700 1 -- 172.21.15.23:0/3526775985 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7fdd8c0899b0 con 0x7fdda40c69e0 2023-06-22T08:11:22.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.955+0000 7fdda3fff700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd8406ace0 0x7fdd8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:22.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.955+0000 7fdda3fff700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd8406ace0 0x7fdd8406d190 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdd94000f60 tx=0x7fdd94007f30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:22.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:22.959+0000 7fdda17fa700 1 -- 172.21.15.23:0/3526775985 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdd8c057ec0 con 0x7fdda40c69e0 2023-06-22T08:11:23.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.214+0000 7fddaa9ff700 1 -- 172.21.15.23:0/3526775985 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdda41f1f70 con 0x7fdda40c69e0 2023-06-22T08:11:23.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.215+0000 7fdda17fa700 1 -- 172.21.15.23:0/3526775985 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5512 (secure 0 0 0) 0x7fdd8c017090 con 0x7fdda40c69e0 2023-06-22T08:11:23.219 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:23.219 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:21.812622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:23.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.217+0000 7fddaa9ff700 1 -- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd8406ace0 msgr2=0x7fdd8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:23.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.217+0000 7fddaa9ff700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd8406ace0 0x7fdd8406d190 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdd94000f60 tx=0x7fdd94007f30 comp rx=0 tx=0).stop 2023-06-22T08:11:23.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.217+0000 7fddaa9ff700 1 -- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 msgr2=0x7fdda4148190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:23.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.217+0000 7fddaa9ff700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 0x7fdda4148190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00e8d0 tx=0x7fdd8c00ebe0 comp rx=0 tx=0).stop 2023-06-22T08:11:23.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.218+0000 7fddaa9ff700 1 -- 172.21.15.23:0/3526775985 shutdown_connections 2023-06-22T08:11:23.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.218+0000 7fddaa9ff700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd8406ace0 0x7fdd8406d190 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:23.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.218+0000 7fddaa9ff700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdda414f850 0x7fdda41486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:23.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.218+0000 7fddaa9ff700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdda4071070 0x7fdda41edfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:23.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.218+0000 7fddaa9ff700 1 --2- 172.21.15.23:0/3526775985 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdda40c69e0 0x7fdda4148190 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:23.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.218+0000 7fddaa9ff700 1 -- 172.21.15.23:0/3526775985 >> 172.21.15.23:0/3526775985 conn(0x7fdda40ca3f0 msgr2=0x7fdda406a160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:23.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.218+0000 7fddaa9ff700 1 -- 172.21.15.23:0/3526775985 shutdown_connections 2023-06-22T08:11:23.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:23.218+0000 7fddaa9ff700 1 -- 172.21.15.23:0/3526775985 wait complete. 2023-06-22T08:11:23.226 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 88 2023-06-22T08:11:23.239 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:11:25.241 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.453+0000 7f5b033ff700 1 Processor -- start 2023-06-22T08:11:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.454+0000 7f5b033ff700 1 -- start start 2023-06-22T08:11:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.455+0000 7f5b033ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5afc1558b0 0x7f5afc06df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.455+0000 7f5b033ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0c4ea0 con 0x7f5afc14f0c0 2023-06-22T08:11:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.455+0000 7f5b033ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc155ce0 con 0x7f5afc06e450 2023-06-22T08:11:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.455+0000 7f5b033ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5afc0c1f70 con 0x7f5afc1558b0 2023-06-22T08:11:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.455+0000 7f5b0119b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5afc14f0c0 0x7f5afc0c61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40526/0 (socket says 172.21.15.23:40526) 2023-06-22T08:11:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.455+0000 7f5b0119b700 1 -- 172.21.15.23:0/215053680 learned_addr learned my addr 172.21.15.23:0/215053680 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.455+0000 7f5b0099a700 1 --2- 172.21.15.23:0/215053680 >> v2:172.21.15.67:3300/0 conn(0x7f5afc1558b0 0x7f5afc06df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.455+0000 7f5af3fff700 1 -- 172.21.15.23:0/215053680 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573920855 0 0) 0x7f5afc0c4ea0 con 0x7f5afc14f0c0 2023-06-22T08:11:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.455+0000 7f5af3fff700 1 -- 172.21.15.23:0/215053680 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae80035b0 con 0x7f5afc14f0c0 2023-06-22T08:11:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.456+0000 7f5b0099a700 1 -- 172.21.15.23:0/215053680 >> v1:172.21.15.67:6789/0 conn(0x7f5afc06e450 legacy=0x7f5afc06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.456+0000 7f5af3fff700 1 -- 172.21.15.23:0/215053680 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703847777 0 0) 0x7f5afc155ce0 con 0x7f5afc06e450 2023-06-22T08:11:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.456+0000 7f5b0099a700 1 -- 172.21.15.23:0/215053680 >> v1:172.21.15.23:6789/0 conn(0x7f5afc14f0c0 legacy=0x7f5afc0c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.456+0000 7f5b0099a700 1 -- 172.21.15.23:0/215053680 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc0c5140 con 0x7f5afc1558b0 2023-06-22T08:11:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.456+0000 7f5b0099a700 1 --2- 172.21.15.23:0/215053680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc1558b0 0x7f5afc06df10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5aec009840 tx=0x7f5aec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbeac324f3bc2c3c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.456+0000 7f5af3fff700 1 -- 172.21.15.23:0/215053680 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec011830 con 0x7f5afc1558b0 2023-06-22T08:11:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.456+0000 7f5af3fff700 1 -- 172.21.15.23:0/215053680 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec011990 con 0x7f5afc1558b0 2023-06-22T08:11:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.456+0000 7f5af3fff700 1 -- 172.21.15.23:0/215053680 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec011d20 con 0x7f5afc1558b0 2023-06-22T08:11:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.457+0000 7f5b033ff700 1 -- 172.21.15.23:0/215053680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc1558b0 msgr2=0x7f5afc06df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:25.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.457+0000 7f5b033ff700 1 --2- 172.21.15.23:0/215053680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc1558b0 0x7f5afc06df10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5aec009840 tx=0x7f5aec009b50 comp rx=0 tx=0).stop 2023-06-22T08:11:25.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.457+0000 7f5b033ff700 1 -- 172.21.15.23:0/215053680 shutdown_connections 2023-06-22T08:11:25.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.457+0000 7f5b033ff700 1 --2- 172.21.15.23:0/215053680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc1558b0 0x7f5afc06df10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:25.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.457+0000 7f5b033ff700 1 -- 172.21.15.23:0/215053680 >> 172.21.15.23:0/215053680 conn(0x7f5afc0ca3f0 msgr2=0x7f5afc0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:25.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.457+0000 7f5b033ff700 1 -- 172.21.15.23:0/215053680 shutdown_connections 2023-06-22T08:11:25.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.457+0000 7f5b033ff700 1 -- 172.21.15.23:0/215053680 wait complete. 2023-06-22T08:11:25.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.459+0000 7f5b033ff700 1 Processor -- start 2023-06-22T08:11:25.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.459+0000 7f5b033ff700 1 -- start start 2023-06-22T08:11:25.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.459+0000 7f5b033ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5afc06e450 0x7f5afc1546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:25.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b033ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc14f0c0 0x7f5afc13d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:25.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b033ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5afc1558b0 0x7f5afc14d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:25.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b033ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc152f70 con 0x7f5afc06e450 2023-06-22T08:11:25.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b033ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc154bf0 con 0x7f5afc14f0c0 2023-06-22T08:11:25.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b033ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc13d030 con 0x7f5afc1558b0 2023-06-22T08:11:25.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b0119b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5afc06e450 0x7f5afc1546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:25.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b0119b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5afc06e450 0x7f5afc1546b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56644/0 (socket says 172.21.15.23:56644) 2023-06-22T08:11:25.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b0119b700 1 -- 172.21.15.23:0/1739802123 learned_addr learned my addr 172.21.15.23:0/1739802123 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:25.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b0099a700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc14f0c0 0x7f5afc13d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:25.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b0199c700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5afc1558b0 0x7f5afc14d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:25.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b0119b700 1 -- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5afc1558b0 msgr2=0x7f5afc14d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:25.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b0119b700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5afc1558b0 0x7f5afc14d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:25.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b0119b700 1 -- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc14f0c0 msgr2=0x7f5afc13d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:25.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b0119b700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc14f0c0 0x7f5afc13d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:25.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.460+0000 7f5b0119b700 1 -- 172.21.15.23:0/1739802123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0094a0 con 0x7f5afc06e450 2023-06-22T08:11:25.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.461+0000 7f5b0119b700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5afc06e450 0x7f5afc1546b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5af800c500 tx=0x7f5af800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:25.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.461+0000 7f5af1ffb700 1 -- 172.21.15.23:0/1739802123 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af8005d70 con 0x7f5afc06e450 2023-06-22T08:11:25.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.461+0000 7f5af1ffb700 1 -- 172.21.15.23:0/1739802123 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af80070d0 con 0x7f5afc06e450 2023-06-22T08:11:25.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.461+0000 7f5af1ffb700 1 -- 172.21.15.23:0/1739802123 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af8007440 con 0x7f5afc06e450 2023-06-22T08:11:25.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.461+0000 7f5b033ff700 1 -- 172.21.15.23:0/1739802123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc152c90 con 0x7f5afc06e450 2023-06-22T08:11:25.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.461+0000 7f5b033ff700 1 -- 172.21.15.23:0/1739802123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc13c9e0 con 0x7f5afc06e450 2023-06-22T08:11:25.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.462+0000 7f5b033ff700 1 -- 172.21.15.23:0/1739802123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc13d310 con 0x7f5afc06e450 2023-06-22T08:11:25.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.463+0000 7f5af1ffb700 1 -- 172.21.15.23:0/1739802123 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5af80075a0 con 0x7f5afc06e450 2023-06-22T08:11:25.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.464+0000 7f5af1ffb700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ae006ace0 0x7f5ae006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:25.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.464+0000 7f5af1ffb700 1 -- 172.21.15.23:0/1739802123 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f5af8088ca0 con 0x7f5afc06e450 2023-06-22T08:11:25.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.465+0000 7f5b0099a700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ae006ace0 0x7f5ae006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:25.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.465+0000 7f5b0099a700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ae006ace0 0x7f5ae006d190 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5aec009fd0 tx=0x7f5aec01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:25.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.470+0000 7f5af1ffb700 1 -- 172.21.15.23:0/1739802123 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5af80538f0 con 0x7f5afc06e450 2023-06-22T08:11:25.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.723+0000 7f5b033ff700 1 -- 172.21.15.23:0/1739802123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5afc04ea20 con 0x7f5afc06e450 2023-06-22T08:11:25.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.723+0000 7f5af1ffb700 1 -- 172.21.15.23:0/1739802123 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5512 (secure 0 0 0) 0x7f5af8056f10 con 0x7f5afc06e450 2023-06-22T08:11:25.727 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:25.727 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:21.812622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:25.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 -- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ae006ace0 msgr2=0x7f5ae006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:25.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ae006ace0 0x7f5ae006d190 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5aec009fd0 tx=0x7f5aec01c040 comp rx=0 tx=0).stop 2023-06-22T08:11:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 -- 172.21.15.23:0/1739802123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5afc06e450 msgr2=0x7f5afc1546b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5afc06e450 0x7f5afc1546b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5af800c500 tx=0x7f5af800c810 comp rx=0 tx=0).stop 2023-06-22T08:11:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 -- 172.21.15.23:0/1739802123 shutdown_connections 2023-06-22T08:11:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5afc1558b0 0x7f5afc14d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5afc06e450 0x7f5afc1546b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ae006ace0 0x7f5ae006d190 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 --2- 172.21.15.23:0/1739802123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5afc14f0c0 0x7f5afc13d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 -- 172.21.15.23:0/1739802123 >> 172.21.15.23:0/1739802123 conn(0x7f5afc0ca3f0 msgr2=0x7f5afc165250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 -- 172.21.15.23:0/1739802123 shutdown_connections 2023-06-22T08:11:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:25.726+0000 7f5b033ff700 1 -- 172.21.15.23:0/1739802123 wait complete. 2023-06-22T08:11:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 88 2023-06-22T08:11:25.748 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:11:27.748 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:27.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.956+0000 7f4f767ff700 1 Processor -- start 2023-06-22T08:11:27.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.956+0000 7f4f767ff700 1 -- start start 2023-06-22T08:11:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.957+0000 7f4f767ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f700c9fd0 0x7f4f700cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.957+0000 7f4f767ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 0x7f4f700c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.957+0000 7f4f767ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7015c370 con 0x7f4f70152ce0 2023-06-22T08:11:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.957+0000 7f4f767ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f701530b0 con 0x7f4f700c9fd0 2023-06-22T08:11:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.957+0000 7f4f767ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700cf6f0 con 0x7f4f700c40b0 2023-06-22T08:11:27.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f6ffff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4f70152ce0 0x7f4f7015c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54098/0 (socket says 172.21.15.23:54098) 2023-06-22T08:11:27.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f6ffff700 1 -- 172.21.15.23:0/1971593386 learned_addr learned my addr 172.21.15.23:0/1971593386 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:27.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f74d9c700 1 --2- 172.21.15.23:0/1971593386 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 0x7f4f700c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:27.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f6f7fe700 1 --2- 172.21.15.23:0/1971593386 >> v2:172.21.15.67:3300/0 conn(0x7f4f700c9fd0 0x7f4f700cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:27.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f6effd700 1 -- 172.21.15.23:0/1971593386 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910147826 0 0) 0x7f4f7015c370 con 0x7f4f70152ce0 2023-06-22T08:11:27.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f6effd700 1 -- 172.21.15.23:0/1971593386 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f580035b0 con 0x7f4f70152ce0 2023-06-22T08:11:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f74d9c700 1 -- 172.21.15.23:0/1971593386 >> v2:172.21.15.67:3300/0 conn(0x7f4f700c9fd0 msgr2=0x7f4f700cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f74d9c700 1 --2- 172.21.15.23:0/1971593386 >> v2:172.21.15.67:3300/0 conn(0x7f4f700c9fd0 0x7f4f700cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f74d9c700 1 -- 172.21.15.23:0/1971593386 >> v1:172.21.15.23:6789/0 conn(0x7f4f70152ce0 legacy=0x7f4f7015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f74d9c700 1 -- 172.21.15.23:0/1971593386 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7015bda0 con 0x7f4f700c40b0 2023-06-22T08:11:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.958+0000 7f4f6f7fe700 1 --2- 172.21.15.23:0/1971593386 >> v2:172.21.15.67:3300/0 conn(0x7f4f700c9fd0 0x7f4f700cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:11:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f74d9c700 1 --2- 172.21.15.23:0/1971593386 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 0x7f4f700c4530 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009d90 tx=0x7f4f6800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2c48c6d80b008d6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f6effd700 1 -- 172.21.15.23:0/1971593386 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6800e070 con 0x7f4f700c40b0 2023-06-22T08:11:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f6effd700 1 -- 172.21.15.23:0/1971593386 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6800bb20 con 0x7f4f700c40b0 2023-06-22T08:11:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f6effd700 1 -- 172.21.15.23:0/1971593386 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68012430 con 0x7f4f700c40b0 2023-06-22T08:11:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f767ff700 1 -- 172.21.15.23:0/1971593386 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 msgr2=0x7f4f700c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f767ff700 1 --2- 172.21.15.23:0/1971593386 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 0x7f4f700c4530 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009d90 tx=0x7f4f6800b330 comp rx=0 tx=0).stop 2023-06-22T08:11:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f767ff700 1 -- 172.21.15.23:0/1971593386 shutdown_connections 2023-06-22T08:11:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f767ff700 1 --2- 172.21.15.23:0/1971593386 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 0x7f4f700c4530 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f767ff700 1 --2- 172.21.15.23:0/1971593386 >> v2:172.21.15.67:3300/0 conn(0x7f4f700c9fd0 0x7f4f700cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f767ff700 1 -- 172.21.15.23:0/1971593386 >> 172.21.15.23:0/1971593386 conn(0x7f4f7006ae30 msgr2=0x7f4f7006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.959+0000 7f4f767ff700 1 -- 172.21.15.23:0/1971593386 shutdown_connections 2023-06-22T08:11:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.960+0000 7f4f767ff700 1 -- 172.21.15.23:0/1971593386 wait complete. 2023-06-22T08:11:27.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.961+0000 7f4f767ff700 1 Processor -- start 2023-06-22T08:11:27.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.961+0000 7f4f767ff700 1 -- start start 2023-06-22T08:11:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f767ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 0x7f4f701579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f767ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700c9fd0 0x7f4f700cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f767ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f70152ce0 0x7f4f7015aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f767ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f701eef90 con 0x7f4f70152ce0 2023-06-22T08:11:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f767ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f700cef00 con 0x7f4f700c9fd0 2023-06-22T08:11:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f767ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f701ee6c0 con 0x7f4f700c40b0 2023-06-22T08:11:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f74d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f70152ce0 0x7f4f7015aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:27.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f74d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f70152ce0 0x7f4f7015aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58256/0 (socket says 172.21.15.23:58256) 2023-06-22T08:11:27.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f74d9c700 1 -- 172.21.15.23:0/1907180022 learned_addr learned my addr 172.21.15.23:0/1907180022 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:27.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f6ffff700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 0x7f4f701579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:27.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.962+0000 7f4f6f7fe700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700c9fd0 0x7f4f700cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:27.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.963+0000 7f4f74d9c700 1 -- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 msgr2=0x7f4f701579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:27.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.963+0000 7f4f74d9c700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 0x7f4f701579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:27.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.963+0000 7f4f74d9c700 1 -- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700c9fd0 msgr2=0x7f4f700cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:27.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.963+0000 7f4f74d9c700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700c9fd0 0x7f4f700cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.963+0000 7f4f74d9c700 1 -- 172.21.15.23:0/1907180022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f640094a0 con 0x7f4f70152ce0 2023-06-22T08:11:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.963+0000 7f4f74d9c700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f70152ce0 0x7f4f7015aeb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4f68014040 tx=0x7f4f68004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.963+0000 7f4f6cff9700 1 -- 172.21.15.23:0/1907180022 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f680129f0 con 0x7f4f70152ce0 2023-06-22T08:11:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.963+0000 7f4f6cff9700 1 -- 172.21.15.23:0/1907180022 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f68020070 con 0x7f4f70152ce0 2023-06-22T08:11:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.963+0000 7f4f6cff9700 1 -- 172.21.15.23:0/1907180022 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68012dd0 con 0x7f4f70152ce0 2023-06-22T08:11:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.963+0000 7f4f767ff700 1 -- 172.21.15.23:0/1907180022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f68009a70 con 0x7f4f70152ce0 2023-06-22T08:11:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.964+0000 7f4f767ff700 1 -- 172.21.15.23:0/1907180022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f700cd700 con 0x7f4f70152ce0 2023-06-22T08:11:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.965+0000 7f4f767ff700 1 -- 172.21.15.23:0/1907180022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f701ef0d0 con 0x7f4f70152ce0 2023-06-22T08:11:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.965+0000 7f4f6cff9700 1 -- 172.21.15.23:0/1907180022 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4f68004040 con 0x7f4f70152ce0 2023-06-22T08:11:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.967+0000 7f4f6cff9700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4f5c06ad10 0x7f4f5c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.967+0000 7f4f6cff9700 1 -- 172.21.15.23:0/1907180022 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f4f68022070 con 0x7f4f70152ce0 2023-06-22T08:11:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.967+0000 7f4f6ffff700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4f5c06ad10 0x7f4f5c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.968+0000 7f4f6ffff700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4f5c06ad10 0x7f4f5c06d1c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4f60007fd0 tx=0x7f4f60007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:27.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:27.972+0000 7f4f6cff9700 1 -- 172.21.15.23:0/1907180022 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4f701ef0d0 con 0x7f4f70152ce0 2023-06-22T08:11:28.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.226+0000 7f4f767ff700 1 -- 172.21.15.23:0/1907180022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f701ef0d0 con 0x7f4f70152ce0 2023-06-22T08:11:28.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.227+0000 7f4f6cff9700 1 -- 172.21.15.23:0/1907180022 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5509 (secure 0 0 0) 0x7f4f701ef0d0 con 0x7f4f70152ce0 2023-06-22T08:11:28.234 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:28.234 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:27.443458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:28.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.229+0000 7f4f767ff700 1 -- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4f5c06ad10 msgr2=0x7f4f5c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:28.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.229+0000 7f4f767ff700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4f5c06ad10 0x7f4f5c06d1c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4f60007fd0 tx=0x7f4f60007dc0 comp rx=0 tx=0).stop 2023-06-22T08:11:28.235 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.229+0000 7f4f767ff700 1 -- 172.21.15.23:0/1907180022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f70152ce0 msgr2=0x7f4f7015aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:28.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.229+0000 7f4f767ff700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f70152ce0 0x7f4f7015aeb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4f68014040 tx=0x7f4f68004ad0 comp rx=0 tx=0).stop 2023-06-22T08:11:28.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.229+0000 7f4f767ff700 1 -- 172.21.15.23:0/1907180022 shutdown_connections 2023-06-22T08:11:28.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.229+0000 7f4f767ff700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f70152ce0 0x7f4f7015aeb0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:28.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.229+0000 7f4f767ff700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4f700c40b0 0x7f4f701579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:28.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.229+0000 7f4f767ff700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4f5c06ad10 0x7f4f5c06d1c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:28.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.229+0000 7f4f767ff700 1 --2- 172.21.15.23:0/1907180022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f700c9fd0 0x7f4f700cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:28.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.229+0000 7f4f767ff700 1 -- 172.21.15.23:0/1907180022 >> 172.21.15.23:0/1907180022 conn(0x7f4f7006ae30 msgr2=0x7f4f7014f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:28.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.230+0000 7f4f767ff700 1 -- 172.21.15.23:0/1907180022 shutdown_connections 2023-06-22T08:11:28.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:28.230+0000 7f4f767ff700 1 -- 172.21.15.23:0/1907180022 wait complete. 2023-06-22T08:11:28.237 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 89 2023-06-22T08:11:28.251 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:11:30.252 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:30.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.458+0000 7f9ff39ff700 1 Processor -- start 2023-06-22T08:11:30.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.458+0000 7f9ff39ff700 1 -- start start 2023-06-22T08:11:30.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.460+0000 7f9ff39ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:30.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.460+0000 7f9ff39ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9fec0c40b0 0x7f9fec0c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.460+0000 7f9ff39ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec15c370 con 0x7f9fec152ce0 2023-06-22T08:11:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.460+0000 7f9ff39ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fec1530b0 con 0x7f9fec0c40b0 2023-06-22T08:11:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.460+0000 7f9ff39ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec0cf6f0 con 0x7f9fec0c9fd0 2023-06-22T08:11:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.460+0000 7f9ff0f9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.460+0000 7f9ff1f9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9fec0c40b0 0x7f9fec0c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.460+0000 7f9ff0f9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec0cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:43068/0 (socket says 172.21.15.23:43068) 2023-06-22T08:11:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.460+0000 7f9ff1f9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9fec0c40b0 0x7f9fec0c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:36550/0 (socket says 172.21.15.23:36550) 2023-06-22T08:11:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.460+0000 7f9ff0f9a700 1 -- 172.21.15.23:0/639926094 learned_addr learned my addr 172.21.15.23:0/639926094 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.461+0000 7f9ff0f9a700 1 -- 172.21.15.23:0/639926094 >> v2:172.21.15.67:3300/0 conn(0x7f9fec0c40b0 msgr2=0x7f9fec0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.461+0000 7f9ff0f9a700 1 --2- 172.21.15.23:0/639926094 >> v2:172.21.15.67:3300/0 conn(0x7f9fec0c40b0 0x7f9fec0c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.461+0000 7f9ff0f9a700 1 -- 172.21.15.23:0/639926094 >> v1:172.21.15.67:6789/0 conn(0x7f9fec152ce0 legacy=0x7f9fec15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.461+0000 7f9ff0f9a700 1 -- 172.21.15.23:0/639926094 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec15bda0 con 0x7f9fec0c9fd0 2023-06-22T08:11:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.461+0000 7f9ff0f9a700 1 --2- 172.21.15.23:0/639926094 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec0cf1b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0098f0 tx=0x7f9fdc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c98a38a6003519 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.461+0000 7f9fe3fff700 1 -- 172.21.15.23:0/639926094 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc011830 con 0x7f9fec0c9fd0 2023-06-22T08:11:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.461+0000 7f9fe3fff700 1 -- 172.21.15.23:0/639926094 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc011990 con 0x7f9fec0c9fd0 2023-06-22T08:11:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.461+0000 7f9fe3fff700 1 -- 172.21.15.23:0/639926094 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc011d20 con 0x7f9fec0c9fd0 2023-06-22T08:11:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.462+0000 7f9ff39ff700 1 -- 172.21.15.23:0/639926094 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 msgr2=0x7f9fec0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.462+0000 7f9ff39ff700 1 --2- 172.21.15.23:0/639926094 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec0cf1b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0098f0 tx=0x7f9fdc009c00 comp rx=0 tx=0).stop 2023-06-22T08:11:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.462+0000 7f9ff39ff700 1 -- 172.21.15.23:0/639926094 shutdown_connections 2023-06-22T08:11:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.462+0000 7f9ff39ff700 1 --2- 172.21.15.23:0/639926094 >> v2:172.21.15.67:3300/0 conn(0x7f9fec0c40b0 0x7f9fec0c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.462+0000 7f9ff39ff700 1 --2- 172.21.15.23:0/639926094 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec0cf1b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.462+0000 7f9ff39ff700 1 -- 172.21.15.23:0/639926094 >> 172.21.15.23:0/639926094 conn(0x7f9fec06ae30 msgr2=0x7f9fec06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:30.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.462+0000 7f9ff39ff700 1 -- 172.21.15.23:0/639926094 shutdown_connections 2023-06-22T08:11:30.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.462+0000 7f9ff39ff700 1 -- 172.21.15.23:0/639926094 wait complete. 2023-06-22T08:11:30.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.464+0000 7f9ff39ff700 1 Processor -- start 2023-06-22T08:11:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.464+0000 7f9ff39ff700 1 -- start start 2023-06-22T08:11:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff39ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fec0c40b0 0x7f9fec1edd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff39ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec15a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff39ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fec152ce0 0x7f9fec15a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff39ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec15bb30 con 0x7f9fec0c40b0 2023-06-22T08:11:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff39ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec15adb0 con 0x7f9fec152ce0 2023-06-22T08:11:30.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff39ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec15b5d0 con 0x7f9fec0c9fd0 2023-06-22T08:11:30.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff179b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fec0c40b0 0x7f9fec1edd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:30.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff179b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fec0c40b0 0x7f9fec1edd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58270/0 (socket says 172.21.15.23:58270) 2023-06-22T08:11:30.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff179b700 1 -- 172.21.15.23:0/3468207424 learned_addr learned my addr 172.21.15.23:0/3468207424 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:30.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff0f9a700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec15a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:30.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff1f9c700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fec152ce0 0x7f9fec15a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:30.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff179b700 1 -- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 msgr2=0x7f9fec15a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:30.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff179b700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec15a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:30.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff179b700 1 -- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fec152ce0 msgr2=0x7f9fec15a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:30.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff179b700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fec152ce0 0x7f9fec15a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:30.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.465+0000 7f9ff179b700 1 -- 172.21.15.23:0/3468207424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc009550 con 0x7f9fec0c40b0 2023-06-22T08:11:30.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.466+0000 7f9ff0f9a700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec15a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:11:30.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.466+0000 7f9ff179b700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fec0c40b0 0x7f9fec1edd20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800c450 tx=0x7f9fe800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:30.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.466+0000 7f9fe1ffb700 1 -- 172.21.15.23:0/3468207424 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe80052d0 con 0x7f9fec0c40b0 2023-06-22T08:11:30.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.466+0000 7f9fe1ffb700 1 -- 172.21.15.23:0/3468207424 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe8005430 con 0x7f9fec0c40b0 2023-06-22T08:11:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.466+0000 7f9fe1ffb700 1 -- 172.21.15.23:0/3468207424 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe80038c0 con 0x7f9fec0c40b0 2023-06-22T08:11:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.466+0000 7f9ff39ff700 1 -- 172.21.15.23:0/3468207424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec15b850 con 0x7f9fec0c40b0 2023-06-22T08:11:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.466+0000 7f9ff39ff700 1 -- 172.21.15.23:0/3468207424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec1f1710 con 0x7f9fec0c40b0 2023-06-22T08:11:30.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.467+0000 7f9ff39ff700 1 -- 172.21.15.23:0/3468207424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec1f1290 con 0x7f9fec0c40b0 2023-06-22T08:11:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.468+0000 7f9fe1ffb700 1 -- 172.21.15.23:0/3468207424 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9fe8003a20 con 0x7f9fec0c40b0 2023-06-22T08:11:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.469+0000 7f9fe1ffb700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fd006ace0 0x7f9fd006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.469+0000 7f9fe1ffb700 1 -- 172.21.15.23:0/3468207424 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f9fe8088df0 con 0x7f9fec0c40b0 2023-06-22T08:11:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.469+0000 7f9ff0f9a700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fd006ace0 0x7f9fd006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:30.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.470+0000 7f9ff0f9a700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fd006ace0 0x7f9fd006d190 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc009c40 tx=0x7f9fdc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:30.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.474+0000 7f9fe1ffb700 1 -- 172.21.15.23:0/3468207424 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9fec1f1290 con 0x7f9fec0c40b0 2023-06-22T08:11:30.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.725+0000 7f9ff39ff700 1 -- 172.21.15.23:0/3468207424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fec1f1290 con 0x7f9fec0c40b0 2023-06-22T08:11:30.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.726+0000 7f9fe1ffb700 1 -- 172.21.15.23:0/3468207424 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5509 (secure 0 0 0) 0x7f9fec1f1290 con 0x7f9fec0c40b0 2023-06-22T08:11:30.730 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:30.730 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:27.443458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 -- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fd006ace0 msgr2=0x7f9fd006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fd006ace0 0x7f9fd006d190 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc009c40 tx=0x7f9fdc008030 comp rx=0 tx=0).stop 2023-06-22T08:11:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 -- 172.21.15.23:0/3468207424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fec0c40b0 msgr2=0x7f9fec1edd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fec0c40b0 0x7f9fec1edd20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800c450 tx=0x7f9fe800c810 comp rx=0 tx=0).stop 2023-06-22T08:11:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 -- 172.21.15.23:0/3468207424 shutdown_connections 2023-06-22T08:11:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9fd006ace0 0x7f9fd006d190 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fec152ce0 0x7f9fec15a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fec0c40b0 0x7f9fec1edd20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 --2- 172.21.15.23:0/3468207424 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9fec0c9fd0 0x7f9fec15a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 -- 172.21.15.23:0/3468207424 >> 172.21.15.23:0/3468207424 conn(0x7f9fec06ae30 msgr2=0x7f9fec151050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 -- 172.21.15.23:0/3468207424 shutdown_connections 2023-06-22T08:11:30.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:30.729+0000 7f9ff39ff700 1 -- 172.21.15.23:0/3468207424 wait complete. 2023-06-22T08:11:30.734 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 89 2023-06-22T08:11:30.751 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5060": { "addr": "172.21.15.67:6837/1551460358", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1551460358, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1551460358, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5060, "incarnation": 46, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 31 }, "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 26 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:11:27.443458+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 5551, "mds_3": 5060, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:11:32.751 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.955+0000 7fcb623ff700 1 Processor -- start 2023-06-22T08:11:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.956+0000 7fcb623ff700 1 -- start start 2023-06-22T08:11:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.957+0000 7fcb623ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcb5c152ce0 0x7fcb5c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.957+0000 7fcb623ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb5c0c9fd0 0x7fcb5c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.957+0000 7fcb623ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5c15c370 con 0x7fcb5c158880 2023-06-22T08:11:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.957+0000 7fcb623ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5c1530b0 con 0x7fcb5c152ce0 2023-06-22T08:11:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.957+0000 7fcb623ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5c0cf6f0 con 0x7fcb5c0c9fd0 2023-06-22T08:11:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.957+0000 7fcb5bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcb5c152ce0 0x7fcb5c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.957+0000 7fcb6099c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fcb5c158880 0x7fcb5c158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54112/0 (socket says 172.21.15.23:54112) 2023-06-22T08:11:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.957+0000 7fcb6099c700 1 -- 172.21.15.23:0/4082813001 learned_addr learned my addr 172.21.15.23:0/4082813001 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.957+0000 7fcb53fff700 1 --2- 172.21.15.23:0/4082813001 >> v2:172.21.15.67:3300/0 conn(0x7fcb5c0c9fd0 0x7fcb5c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:32.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.958+0000 7fcb5bfff700 1 -- 172.21.15.23:0/4082813001 >> v2:172.21.15.67:3300/0 conn(0x7fcb5c0c9fd0 msgr2=0x7fcb5c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.958+0000 7fcb5bfff700 1 --2- 172.21.15.23:0/4082813001 >> v2:172.21.15.67:3300/0 conn(0x7fcb5c0c9fd0 0x7fcb5c15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.958+0000 7fcb5bfff700 1 -- 172.21.15.23:0/4082813001 >> v1:172.21.15.23:6789/0 conn(0x7fcb5c158880 legacy=0x7fcb5c158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.958+0000 7fcb5bfff700 1 -- 172.21.15.23:0/4082813001 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c15bda0 con 0x7fcb5c152ce0 2023-06-22T08:11:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.958+0000 7fcb5b7fe700 1 -- 172.21.15.23:0/4082813001 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939837236 0 0) 0x7fcb5c15c370 con 0x7fcb5c158880 2023-06-22T08:11:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.958+0000 7fcb5bfff700 1 --2- 172.21.15.23:0/4082813001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c152ce0 0x7fcb5c0cf1b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0cc380 tx=0x7fcb4c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73500d9bcc1c819f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.958+0000 7fcb5b7fe700 1 -- 172.21.15.23:0/4082813001 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c011830 con 0x7fcb5c152ce0 2023-06-22T08:11:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.958+0000 7fcb5b7fe700 1 -- 172.21.15.23:0/4082813001 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4c011990 con 0x7fcb5c152ce0 2023-06-22T08:11:32.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.958+0000 7fcb5b7fe700 1 -- 172.21.15.23:0/4082813001 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c011d20 con 0x7fcb5c152ce0 2023-06-22T08:11:32.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.959+0000 7fcb623ff700 1 -- 172.21.15.23:0/4082813001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c152ce0 msgr2=0x7fcb5c0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:32.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.959+0000 7fcb623ff700 1 --2- 172.21.15.23:0/4082813001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c152ce0 0x7fcb5c0cf1b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0cc380 tx=0x7fcb4c009b20 comp rx=0 tx=0).stop 2023-06-22T08:11:32.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.959+0000 7fcb623ff700 1 -- 172.21.15.23:0/4082813001 shutdown_connections 2023-06-22T08:11:32.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.959+0000 7fcb623ff700 1 --2- 172.21.15.23:0/4082813001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c152ce0 0x7fcb5c0cf1b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:32.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.959+0000 7fcb623ff700 1 --2- 172.21.15.23:0/4082813001 >> v2:172.21.15.67:3300/0 conn(0x7fcb5c0c9fd0 0x7fcb5c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:32.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.959+0000 7fcb623ff700 1 -- 172.21.15.23:0/4082813001 >> 172.21.15.23:0/4082813001 conn(0x7fcb5c06ae30 msgr2=0x7fcb5c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.959+0000 7fcb623ff700 1 -- 172.21.15.23:0/4082813001 shutdown_connections 2023-06-22T08:11:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.959+0000 7fcb623ff700 1 -- 172.21.15.23:0/4082813001 wait complete. 2023-06-22T08:11:32.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.960+0000 7fcb623ff700 1 Processor -- start 2023-06-22T08:11:32.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.961+0000 7fcb623ff700 1 -- start start 2023-06-22T08:11:32.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.961+0000 7fcb623ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c0c9fd0 0x7fcb5c1510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:32.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.961+0000 7fcb623ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcb5c152ce0 0x7fcb5c151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:32.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb623ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c158880 0x7fcb5c147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:32.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb623ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c1506b0 con 0x7fcb5c158880 2023-06-22T08:11:32.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb623ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c0b2290 con 0x7fcb5c0c9fd0 2023-06-22T08:11:32.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb623ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c1f1400 con 0x7fcb5c152ce0 2023-06-22T08:11:32.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb6099c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c158880 0x7fcb5c147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:32.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb6099c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c158880 0x7fcb5c147ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58282/0 (socket says 172.21.15.23:58282) 2023-06-22T08:11:32.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb6099c700 1 -- 172.21.15.23:0/2228729270 learned_addr learned my addr 172.21.15.23:0/2228729270 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:32.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb5bfff700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c0c9fd0 0x7fcb5c1510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:32.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb53fff700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcb5c152ce0 0x7fcb5c151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:32.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb6099c700 1 -- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcb5c152ce0 msgr2=0x7fcb5c151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:32.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb6099c700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcb5c152ce0 0x7fcb5c151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:32.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb6099c700 1 -- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c0c9fd0 msgr2=0x7fcb5c1510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:32.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb6099c700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c0c9fd0 0x7fcb5c1510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:32.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.962+0000 7fcb6099c700 1 -- 172.21.15.23:0/2228729270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c0094a0 con 0x7fcb5c158880 2023-06-22T08:11:32.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.963+0000 7fcb6099c700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c158880 0x7fcb5c147ec0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400cba0 tx=0x7fcb5400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:32.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.963+0000 7fcb597fa700 1 -- 172.21.15.23:0/2228729270 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5400aaa0 con 0x7fcb5c158880 2023-06-22T08:11:32.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.963+0000 7fcb597fa700 1 -- 172.21.15.23:0/2228729270 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5400ac20 con 0x7fcb5c158880 2023-06-22T08:11:32.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.963+0000 7fcb623ff700 1 -- 172.21.15.23:0/2228729270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5c147a30 con 0x7fcb5c158880 2023-06-22T08:11:32.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.963+0000 7fcb597fa700 1 -- 172.21.15.23:0/2228729270 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb540070d0 con 0x7fcb5c158880 2023-06-22T08:11:32.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.963+0000 7fcb623ff700 1 -- 172.21.15.23:0/2228729270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5c1476c0 con 0x7fcb5c158880 2023-06-22T08:11:32.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.964+0000 7fcb623ff700 1 -- 172.21.15.23:0/2228729270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5c147bc0 con 0x7fcb5c158880 2023-06-22T08:11:32.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.965+0000 7fcb597fa700 1 -- 172.21.15.23:0/2228729270 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcb54007230 con 0x7fcb5c158880 2023-06-22T08:11:32.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.966+0000 7fcb597fa700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcb3c06ace0 0x7fcb3c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:32.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.966+0000 7fcb597fa700 1 -- 172.21.15.23:0/2228729270 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7fcb54089450 con 0x7fcb5c158880 2023-06-22T08:11:32.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.966+0000 7fcb5bfff700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcb3c06ace0 0x7fcb3c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:32.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.967+0000 7fcb5bfff700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcb3c06ace0 0x7fcb3c06d190 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0090b0 tx=0x7fcb4c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:32.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:32.971+0000 7fcb597fa700 1 -- 172.21.15.23:0/2228729270 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcb54053ff0 con 0x7fcb5c158880 2023-06-22T08:11:33.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.222+0000 7fcb623ff700 1 -- 172.21.15.23:0/2228729270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb5c04ea20 con 0x7fcb5c158880 2023-06-22T08:11:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.222+0000 7fcb597fa700 1 -- 172.21.15.23:0/2228729270 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5509 (secure 0 0 0) 0x7fcb54057610 con 0x7fcb5c158880 2023-06-22T08:11:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:33.226 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:32.817830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 -- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcb3c06ace0 msgr2=0x7fcb3c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcb3c06ace0 0x7fcb3c06d190 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0090b0 tx=0x7fcb4c008030 comp rx=0 tx=0).stop 2023-06-22T08:11:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 -- 172.21.15.23:0/2228729270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c158880 msgr2=0x7fcb5c147ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c158880 0x7fcb5c147ec0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400cba0 tx=0x7fcb5400ceb0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 -- 172.21.15.23:0/2228729270 shutdown_connections 2023-06-22T08:11:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb5c158880 0x7fcb5c147ec0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcb3c06ace0 0x7fcb3c06d190 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb5c0c9fd0 0x7fcb5c1510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 --2- 172.21.15.23:0/2228729270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcb5c152ce0 0x7fcb5c151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 -- 172.21.15.23:0/2228729270 >> 172.21.15.23:0/2228729270 conn(0x7fcb5c06ae30 msgr2=0x7fcb5c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:33.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 -- 172.21.15.23:0/2228729270 shutdown_connections 2023-06-22T08:11:33.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.226+0000 7fcb623ff700 1 -- 172.21.15.23:0/2228729270 wait complete. 2023-06-22T08:11:33.232 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 90 2023-06-22T08:11:33.247 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4285 2023-06-22T08:11:33.248 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.f 2023-06-22T08:11:33.248 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-06-22T08:11:33.248 INFO:tasks.ceph.mds.f:Restarting daemon 2023-06-22T08:11:33.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-06-22T08:11:33.251 INFO:tasks.ceph.mds.f:Started 2023-06-22T08:11:33.251 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:33.341 INFO:tasks.ceph.mds.f.smithi067.stdout:starting mds.f at 2023-06-22T08:11:33.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.447+0000 7fa1285ff700 1 Processor -- start 2023-06-22T08:11:33.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.447+0000 7fa1285ff700 1 -- start start 2023-06-22T08:11:33.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.448+0000 7fa1285ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1200c8bd0 0x7fa1200aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.448+0000 7fa1285ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1200c7400 con 0x7fa1200cc640 2023-06-22T08:11:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.448+0000 7fa1285ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1200c81c0 con 0x7fa120152dd0 2023-06-22T08:11:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.448+0000 7fa1285ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1200c84a0 con 0x7fa1200c8bd0 2023-06-22T08:11:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.448+0000 7fa125b9a700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1200c8bd0 0x7fa1200aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.448+0000 7fa12639b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa120152dd0 0x7fa12006afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:48888/0 (socket says 172.21.15.23:48888) 2023-06-22T08:11:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.448+0000 7fa12639b700 1 -- 172.21.15.23:0/2504640449 learned_addr learned my addr 172.21.15.23:0/2504640449 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:11:33.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.448+0000 7fa126b9c700 1 --1- 172.21.15.23:0/2504640449 >> v1:172.21.15.23:6789/0 conn(0x7fa1200cc640 0x7fa120163280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54120/0 (socket says 172.21.15.23:54120) 2023-06-22T08:11:33.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.449+0000 7fa125b9a700 1 -- 172.21.15.23:0/2504640449 >> v1:172.21.15.67:6789/0 conn(0x7fa120152dd0 legacy=0x7fa12006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:33.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.449+0000 7fa125b9a700 1 -- 172.21.15.23:0/2504640449 >> v1:172.21.15.23:6789/0 conn(0x7fa1200cc640 legacy=0x7fa120163280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:33.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.449+0000 7fa125b9a700 1 -- 172.21.15.23:0/2504640449 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1200cd370 con 0x7fa1200c8bd0 2023-06-22T08:11:33.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.449+0000 7fa125b9a700 1 --2- 172.21.15.23:0/2504640449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1200c8bd0 0x7fa1200aedc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa110009840 tx=0x7fa110009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d271b95506ccf74 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:33.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.449+0000 7fa125399700 1 -- 172.21.15.23:0/2504640449 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110011830 con 0x7fa1200c8bd0 2023-06-22T08:11:33.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.449+0000 7fa125399700 1 -- 172.21.15.23:0/2504640449 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa110011990 con 0x7fa1200c8bd0 2023-06-22T08:11:33.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.449+0000 7fa125399700 1 -- 172.21.15.23:0/2504640449 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110011d20 con 0x7fa1200c8bd0 2023-06-22T08:11:33.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.450+0000 7fa1285ff700 1 -- 172.21.15.23:0/2504640449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1200c8bd0 msgr2=0x7fa1200aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:33.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.450+0000 7fa1285ff700 1 --2- 172.21.15.23:0/2504640449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1200c8bd0 0x7fa1200aedc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa110009840 tx=0x7fa110009b50 comp rx=0 tx=0).stop 2023-06-22T08:11:33.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.450+0000 7fa1285ff700 1 -- 172.21.15.23:0/2504640449 shutdown_connections 2023-06-22T08:11:33.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.450+0000 7fa1285ff700 1 --2- 172.21.15.23:0/2504640449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1200c8bd0 0x7fa1200aedc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.450+0000 7fa1285ff700 1 -- 172.21.15.23:0/2504640449 >> 172.21.15.23:0/2504640449 conn(0x7fa12006fb70 msgr2=0x7fa12006ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:33.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.450+0000 7fa1285ff700 1 -- 172.21.15.23:0/2504640449 shutdown_connections 2023-06-22T08:11:33.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.450+0000 7fa1285ff700 1 -- 172.21.15.23:0/2504640449 wait complete. 2023-06-22T08:11:33.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.452+0000 7fa1285ff700 1 Processor -- start 2023-06-22T08:11:33.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.452+0000 7fa1285ff700 1 -- start start 2023-06-22T08:11:33.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa1285ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1200c8bd0 0x7fa1200b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:33.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa1285ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa1200cc640 0x7fa120150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:33.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa1285ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa120152dd0 0x7fa120151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa1285ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1201ee690 con 0x7fa1200c8bd0 2023-06-22T08:11:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa1285ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12015c530 con 0x7fa120152dd0 2023-06-22T08:11:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa1285ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1201ee950 con 0x7fa1200cc640 2023-06-22T08:11:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa12639b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1200c8bd0 0x7fa1200b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa12639b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1200c8bd0 0x7fa1200b0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58290/0 (socket says 172.21.15.23:58290) 2023-06-22T08:11:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa12639b700 1 -- 172.21.15.23:0/2443190313 learned_addr learned my addr 172.21.15.23:0/2443190313 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa125b9a700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa1200cc640 0x7fa120150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:33.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa126b9c700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa120152dd0 0x7fa120151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa12639b700 1 -- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa1200cc640 msgr2=0x7fa120150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa12639b700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa1200cc640 0x7fa120150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa12639b700 1 -- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa120152dd0 msgr2=0x7fa120151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa12639b700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa120152dd0 0x7fa120151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.453+0000 7fa12639b700 1 -- 172.21.15.23:0/2443190313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1100094a0 con 0x7fa1200c8bd0 2023-06-22T08:11:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.454+0000 7fa12639b700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1200c8bd0 0x7fa1200b0d60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa11c001f00 tx=0x7fa11c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:33.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.454+0000 7fa116ffd700 1 -- 172.21.15.23:0/2443190313 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c005dc0 con 0x7fa1200c8bd0 2023-06-22T08:11:33.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.454+0000 7fa116ffd700 1 -- 172.21.15.23:0/2443190313 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11c0070d0 con 0x7fa1200c8bd0 2023-06-22T08:11:33.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.454+0000 7fa116ffd700 1 -- 172.21.15.23:0/2443190313 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c007440 con 0x7fa1200c8bd0 2023-06-22T08:11:33.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.454+0000 7fa1285ff700 1 -- 172.21.15.23:0/2443190313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1201eec30 con 0x7fa1200c8bd0 2023-06-22T08:11:33.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.454+0000 7fa1285ff700 1 -- 172.21.15.23:0/2443190313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1201ef1f0 con 0x7fa1200c8bd0 2023-06-22T08:11:33.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.455+0000 7fa1285ff700 1 -- 172.21.15.23:0/2443190313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12006bfc0 con 0x7fa1200c8bd0 2023-06-22T08:11:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.456+0000 7fa116ffd700 1 -- 172.21.15.23:0/2443190313 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa11c0075a0 con 0x7fa1200c8bd0 2023-06-22T08:11:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.457+0000 7fa116ffd700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa10406ace0 0x7fa10406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.457+0000 7fa116ffd700 1 -- 172.21.15.23:0/2443190313 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7fa11c088c10 con 0x7fa1200c8bd0 2023-06-22T08:11:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.457+0000 7fa125b9a700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa10406ace0 0x7fa10406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.458+0000 7fa125b9a700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa10406ace0 0x7fa10406d190 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa1100090b0 tx=0x7fa110008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:33.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.461+0000 7fa116ffd700 1 -- 172.21.15.23:0/2443190313 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa11c053860 con 0x7fa1200c8bd0 2023-06-22T08:11:33.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.714+0000 7fa1285ff700 1 -- 172.21.15.23:0/2443190313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1200c7b70 con 0x7fa1200c8bd0 2023-06-22T08:11:33.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.715+0000 7fa116ffd700 1 -- 172.21.15.23:0/2443190313 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5509 (secure 0 0 0) 0x7fa11c086e60 con 0x7fa1200c8bd0 2023-06-22T08:11:33.719 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:33.719 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:32.817830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:33.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 -- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa10406ace0 msgr2=0x7fa10406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:33.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa10406ace0 0x7fa10406d190 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa1100090b0 tx=0x7fa110008030 comp rx=0 tx=0).stop 2023-06-22T08:11:33.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 -- 172.21.15.23:0/2443190313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1200c8bd0 msgr2=0x7fa1200b0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:33.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1200c8bd0 0x7fa1200b0d60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa11c001f00 tx=0x7fa11c00c7b0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 -- 172.21.15.23:0/2443190313 shutdown_connections 2023-06-22T08:11:33.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa10406ace0 0x7fa10406d190 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa120152dd0 0x7fa120151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1200c8bd0 0x7fa1200b0d60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 --2- 172.21.15.23:0/2443190313 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa1200cc640 0x7fa120150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:33.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 -- 172.21.15.23:0/2443190313 >> 172.21.15.23:0/2443190313 conn(0x7fa12006fb70 msgr2=0x7fa1200c99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:33.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 -- 172.21.15.23:0/2443190313 shutdown_connections 2023-06-22T08:11:33.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:33.718+0000 7fa1285ff700 1 -- 172.21.15.23:0/2443190313 wait complete. 2023-06-22T08:11:33.723 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 90 2023-06-22T08:11:33.739 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-06-22T08:11:35.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:11:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.949+0000 7f6d83fff700 1 Processor -- start 2023-06-22T08:11:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.949+0000 7f6d83fff700 1 -- start start 2023-06-22T08:11:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.950+0000 7f6d83fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d7c152d70 0x7f6d7c0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.951+0000 7f6d83fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c0c45f0 0x7f6d7c0c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.951+0000 7f6d83fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7c0c4f90 con 0x7f6d7c159330 2023-06-22T08:11:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.951+0000 7f6d83fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d7c0c95e0 con 0x7f6d7c152d70 2023-06-22T08:11:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.951+0000 7f6d83fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c159790 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.951+0000 7f6d8259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c0c45f0 0x7f6d7c0c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.951+0000 7f6d81d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d7c152d70 0x7f6d7c0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.951+0000 7f6d81d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d7c152d70 0x7f6d7c0c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:42178/0 (socket says 172.21.15.23:42178) 2023-06-22T08:11:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.951+0000 7f6d81d9b700 1 -- 172.21.15.23:0/633344733 learned_addr learned my addr 172.21.15.23:0/633344733 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.952+0000 7f6d8259c700 1 -- 172.21.15.23:0/633344733 >> v2:172.21.15.67:3300/0 conn(0x7f6d7c152d70 msgr2=0x7f6d7c0c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:11:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.952+0000 7f6d8259c700 1 --2- 172.21.15.23:0/633344733 >> v2:172.21.15.67:3300/0 conn(0x7f6d7c152d70 0x7f6d7c0c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.952+0000 7f6d8259c700 1 -- 172.21.15.23:0/633344733 >> v1:172.21.15.67:6789/0 conn(0x7f6d7c159330 legacy=0x7f6d7c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.952+0000 7f6d8259c700 1 -- 172.21.15.23:0/633344733 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c15bea0 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.952+0000 7f6d8259c700 1 --2- 172.21.15.23:0/633344733 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c0c45f0 0x7f6d7c0c4a50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d74009d90 tx=0x7f6d7400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=572d3a8cd80a73c7 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.952+0000 7f6d80d99700 1 -- 172.21.15.23:0/633344733 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7400e070 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.952+0000 7f6d80d99700 1 -- 172.21.15.23:0/633344733 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7400bb20 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.952+0000 7f6d80d99700 1 -- 172.21.15.23:0/633344733 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d74012430 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.953+0000 7f6d83fff700 1 -- 172.21.15.23:0/633344733 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c0c45f0 msgr2=0x7f6d7c0c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.953+0000 7f6d83fff700 1 --2- 172.21.15.23:0/633344733 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c0c45f0 0x7f6d7c0c4a50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6d74009d90 tx=0x7f6d7400b330 comp rx=0 tx=0).stop 2023-06-22T08:11:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.953+0000 7f6d83fff700 1 -- 172.21.15.23:0/633344733 shutdown_connections 2023-06-22T08:11:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.953+0000 7f6d83fff700 1 --2- 172.21.15.23:0/633344733 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c0c45f0 0x7f6d7c0c4a50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.953+0000 7f6d83fff700 1 --2- 172.21.15.23:0/633344733 >> v2:172.21.15.67:3300/0 conn(0x7f6d7c152d70 0x7f6d7c0c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.953+0000 7f6d83fff700 1 -- 172.21.15.23:0/633344733 >> 172.21.15.23:0/633344733 conn(0x7f6d7c06ae30 msgr2=0x7f6d7c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:35.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.953+0000 7f6d83fff700 1 -- 172.21.15.23:0/633344733 shutdown_connections 2023-06-22T08:11:35.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.953+0000 7f6d83fff700 1 -- 172.21.15.23:0/633344733 wait complete. 2023-06-22T08:11:35.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.955+0000 7f6d83fff700 1 Processor -- start 2023-06-22T08:11:35.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.955+0000 7f6d83fff700 1 -- start start 2023-06-22T08:11:35.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.956+0000 7f6d83fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d7c0c45f0 0x7f6d7c1509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:35.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d83fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d7c152d70 0x7f6d7c15b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:35.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d83fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c159330 0x7f6d7c1e3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:35.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d83fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c1e4cc0 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d83fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c1e49e0 con 0x7f6d7c152d70 2023-06-22T08:11:35.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d83fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c1e2d80 con 0x7f6d7c159330 2023-06-22T08:11:35.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d81d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d7c0c45f0 0x7f6d7c1509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:35.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d81d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d7c0c45f0 0x7f6d7c1509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52854/0 (socket says 172.21.15.23:52854) 2023-06-22T08:11:35.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d81d9b700 1 -- 172.21.15.23:0/2659753839 learned_addr learned my addr 172.21.15.23:0/2659753839 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:11:35.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d8159a700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d7c152d70 0x7f6d7c15b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:35.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d8259c700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c159330 0x7f6d7c1e3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:35.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d81d9b700 1 -- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c159330 msgr2=0x7f6d7c1e3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:35.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d81d9b700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c159330 0x7f6d7c1e3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:35.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d81d9b700 1 -- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d7c152d70 msgr2=0x7f6d7c15b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d81d9b700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d7c152d70 0x7f6d7c15b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.957+0000 7f6d81d9b700 1 -- 172.21.15.23:0/2659753839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d74009a70 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.958+0000 7f6d81d9b700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d7c0c45f0 0x7f6d7c1509b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800e910 tx=0x7f6d7800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.958+0000 7f6d727fc700 1 -- 172.21.15.23:0/2659753839 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d780096a0 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.958+0000 7f6d727fc700 1 -- 172.21.15.23:0/2659753839 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d78013680 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.958+0000 7f6d83fff700 1 -- 172.21.15.23:0/2659753839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7c1e3060 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.958+0000 7f6d727fc700 1 -- 172.21.15.23:0/2659753839 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78013a30 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.958+0000 7f6d83fff700 1 -- 172.21.15.23:0/2659753839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7c1e3620 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.959+0000 7f6d83fff700 1 -- 172.21.15.23:0/2659753839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d7c1e3840 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.960+0000 7f6d727fc700 1 -- 172.21.15.23:0/2659753839 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6d78013b90 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.962+0000 7f6d727fc700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d6006ad10 0x7f6d6006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:11:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.962+0000 7f6d727fc700 1 -- 172.21.15.23:0/2659753839 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f6d78089bf0 con 0x7f6d7c0c45f0 2023-06-22T08:11:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.962+0000 7f6d8159a700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d6006ad10 0x7f6d6006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:11:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.963+0000 7f6d8159a700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d6006ad10 0x7f6d6006d1c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c146630 tx=0x7f6d6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:11:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:35.966+0000 7f6d727fc700 1 -- 172.21.15.23:0/2659753839 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6d78054790 con 0x7f6d7c0c45f0 2023-06-22T08:11:36.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.221+0000 7f6d83fff700 1 -- 172.21.15.23:0/2659753839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d7c04ea20 con 0x7f6d7c0c45f0 2023-06-22T08:11:36.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.222+0000 7f6d727fc700 1 -- 172.21.15.23:0/2659753839 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6262 (secure 0 0 0) 0x7f6d78057db0 con 0x7f6d7c0c45f0 2023-06-22T08:11:36.226 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:11:36.226 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5995,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:11:32.817830+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:11:36.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.224+0000 7f6d83fff700 1 -- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d6006ad10 msgr2=0x7f6d6006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:36.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.224+0000 7f6d83fff700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d6006ad10 0x7f6d6006d1c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c146630 tx=0x7f6d6c009040 comp rx=0 tx=0).stop 2023-06-22T08:11:36.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.224+0000 7f6d83fff700 1 -- 172.21.15.23:0/2659753839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d7c0c45f0 msgr2=0x7f6d7c1509b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:11:36.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.224+0000 7f6d83fff700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d7c0c45f0 0x7f6d7c1509b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800e910 tx=0x7f6d7800ec20 comp rx=0 tx=0).stop 2023-06-22T08:11:36.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.225+0000 7f6d83fff700 1 -- 172.21.15.23:0/2659753839 shutdown_connections 2023-06-22T08:11:36.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.225+0000 7f6d83fff700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6d7c159330 0x7f6d7c1e3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:36.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.225+0000 7f6d83fff700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d7c0c45f0 0x7f6d7c1509b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:36.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.225+0000 7f6d83fff700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6d6006ad10 0x7f6d6006d1c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:36.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.225+0000 7f6d83fff700 1 --2- 172.21.15.23:0/2659753839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d7c152d70 0x7f6d7c15b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:11:36.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.225+0000 7f6d83fff700 1 -- 172.21.15.23:0/2659753839 >> 172.21.15.23:0/2659753839 conn(0x7f6d7c06ae30 msgr2=0x7f6d7c0cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:11:36.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.225+0000 7f6d83fff700 1 -- 172.21.15.23:0/2659753839 shutdown_connections 2023-06-22T08:11:36.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:11:36.225+0000 7f6d83fff700 1 -- 172.21.15.23:0/2659753839 wait complete. 2023-06-22T08:11:36.230 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 91 2023-06-22T08:11:36.246 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-06-22T08:11:36.247 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 75 secs before thrashing 2023-06-22T08:11:38.847 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 16% (30/181) Updating files: 17% (31/181) Updating files: 18% (33/181) Updating files: 19% (35/181) Updating files: 20% (37/181) Updating files: 21% (39/181) Updating files: 22% (40/181) Updating files: 23% (42/181) Updating files: 24% (44/181) Updating files: 25% (46/181) Updating files: 26% (48/181) Updating files: 27% (49/181) Updating files: 28% (51/181) Updating files: 29% (53/181) Updating files: 30% (55/181) Updating files: 31% (57/181) Updating files: 32% (58/181) Updating files: 33% (60/181) Updating files: 34% (62/181) Updating files: 35% (64/181) Updating files: 36% (66/181) Updating files: 37% (67/181) Updating files: 38% (69/181) Updating files: 39% (71/181) Updating files: 40% (73/181) Updating files: 41% (75/181) Updating files: 42% (77/181) Updating files: 43% (78/181) Updating files: 44% (80/181) Updating files: 45% (82/181) Updating files: 46% (84/181) Updating files: 47% (86/181) Updating files: 48% (87/181) Updating files: 49% (89/181) Updating files: 50% (91/181) Updating files: 51% (93/181) Updating files: 52% (95/181) Updating files: 53% (96/181) Updating files: 54% (98/181) Updating files: 55% (100/181) Updating files: 56% (102/181) Updating files: 57% (104/181) Updating files: 58% (105/181) Updating files: 59% (107/181) Updating files: 60% (109/181) Updating files: 61% (111/181) Updating files: 62% (113/181) Updating files: 63% (115/181) Updating files: 64% (116/181) Updating files: 65% (118/181) Updating files: 66% (120/181) Updating files: 67% (122/181) Updating files: 68% (124/181) Updating files: 69% (125/181) Updating files: 70% (127/181) Updating files: 71% (129/181) Updating files: 72% (131/181) Updating files: 73% (133/181) Updating files: 74% (134/181) Updating files: 75% (136/181) Updating files: 76% (138/181) Updating files: 77% (140/181) Updating files: 78% (142/181) Updating files: 79% (143/181) Updating files: 80% (145/181) Updating files: 81% (147/181) Updating files: 82% (149/181) Updating files: 83% (151/181) Updating files: 84% (153/181) Updating files: 85% (154/181) Updating files: 86% (156/181) Updating files: 87% (158/181) Updating files: 88% (160/181) Updating files: 89% (162/181) Updating files: 90% (163/181) Updating files: 91% (165/181) Updating files: 92% (167/181) Updating files: 93% (169/181) Updating files: 94% (171/181) Updating files: 95% (172/181) Updating files: 96% (174/181) Updating files: 97% (176/181) Updating files: 98% (178/181) Updating files: 99% (180/181) Updating files: 100% (181/181) Updating files: 100% (181/181), done. 2023-06-22T08:11:38.848 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:11:38.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:11:39.293 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at a6f3bbb744a v0.37 2023-06-22T08:11:39.295 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.37 2023-06-22T08:11:39.387 INFO:tasks.workunit.client.0.smithi023.stdout:v0.38 2023-06-22T08:11:39.387 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:11:39.388 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 38 -eq 48 ']' 2023-06-22T08:11:39.388 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.38 2023-06-22T08:11:39.388 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.38 2023-06-22T08:11:39.388 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.38 2023-06-22T08:11:39.819 INFO:tasks.ceph.mds.b.smithi067.stderr:2023-06-22T08:11:39.815+0000 7f06546cf700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-22T08:11:39.819 INFO:tasks.ceph.mds.b.smithi067.stderr:2023-06-22T08:11:39.815+0000 7f06546cf700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-06-22T08:11:39.820 INFO:tasks.ceph.mds.b.smithi067.stderr:2023-06-22T08:11:39.815+0000 7f06546cf700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-06-22T08:11:39.820 INFO:tasks.ceph.mds.b.smithi067.stderr:2023-06-22T08:11:39.815+0000 7f06546cf700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-06-22T08:11:39.820 INFO:tasks.ceph.mds.b.smithi067.stderr:2023-06-22T08:11:39.815+0000 7f06546cf700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-06-22T08:11:41.074 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at b600ec2ac7c v0.38 2023-06-22T08:11:41.077 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.38 2023-06-22T08:11:41.155 INFO:tasks.workunit.client.0.smithi023.stdout:v0.39 2023-06-22T08:11:41.155 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:11:41.155 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 39 -eq 48 ']' 2023-06-22T08:11:41.155 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.39 2023-06-22T08:11:41.155 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.39 2023-06-22T08:11:41.156 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.39 2023-06-22T08:11:42.622 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 75% (194/258) Updating files: 76% (197/258) Updating files: 77% (199/258) Updating files: 78% (202/258) Updating files: 79% (204/258) Updating files: 80% (207/258) Updating files: 81% (209/258) Updating files: 82% (212/258) Updating files: 83% (215/258) Updating files: 84% (217/258) Updating files: 85% (220/258) Updating files: 86% (222/258) Updating files: 87% (225/258) Updating files: 88% (228/258) Updating files: 89% (230/258) Updating files: 90% (233/258) Updating files: 91% (235/258) Updating files: 92% (238/258) Updating files: 93% (240/258) Updating files: 94% (243/258) Updating files: 95% (246/258) Updating files: 96% (248/258) Updating files: 97% (251/258) Updating files: 98% (253/258) Updating files: 99% (256/258) Updating files: 100% (258/258) Updating files: 100% (258/258), done. 2023-06-22T08:11:42.725 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 321ecdaba2c v0.39 2023-06-22T08:11:42.729 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.39 2023-06-22T08:11:42.882 INFO:tasks.workunit.client.0.smithi023.stdout:v0.40 2023-06-22T08:11:42.883 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:11:42.883 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 40 -eq 48 ']' 2023-06-22T08:11:42.883 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.40 2023-06-22T08:11:42.883 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.40 2023-06-22T08:11:42.883 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.40 2023-06-22T08:11:53.370 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 52% (118/226) Updating files: 53% (120/226) Updating files: 54% (123/226) Updating files: 55% (125/226) Updating files: 56% (127/226) Updating files: 57% (129/226) Updating files: 58% (132/226) Updating files: 59% (134/226) Updating files: 60% (136/226) Updating files: 61% (138/226) Updating files: 62% (141/226) Updating files: 63% (143/226) Updating files: 64% (145/226) Updating files: 65% (147/226) Updating files: 66% (150/226) Updating files: 67% (152/226) Updating files: 68% (154/226) Updating files: 69% (156/226) Updating files: 70% (159/226) Updating files: 71% (161/226) Updating files: 72% (163/226) Updating files: 73% (165/226) Updating files: 74% (168/226) Updating files: 75% (170/226) Updating files: 76% (172/226) Updating files: 77% (175/226) Updating files: 78% (177/226) Updating files: 79% (179/226) Updating files: 80% (181/226) Updating files: 81% (184/226) Updating files: 82% (186/226) Updating files: 83% (188/226) Updating files: 84% (190/226) Updating files: 85% (193/226) Updating files: 86% (195/226) Updating files: 87% (197/226) Updating files: 88% (199/226) Updating files: 89% (202/226) Updating files: 90% (204/226) Updating files: 91% (206/226) Updating files: 92% (208/226) Updating files: 93% (211/226) Updating files: 94% (213/226) Updating files: 95% (215/226) Updating files: 96% (217/226) Updating files: 97% (220/226) Updating files: 98% (222/226) Updating files: 99% (224/226) Updating files: 100% (226/226) Updating files: 100% (226/226), done. 2023-06-22T08:11:53.461 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 7eea40ea37f v0.40 2023-06-22T08:11:53.464 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.40 2023-06-22T08:11:53.553 INFO:tasks.workunit.client.0.smithi023.stdout:v0.41 2023-06-22T08:11:53.553 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:11:53.554 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 41 -eq 48 ']' 2023-06-22T08:11:53.554 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.41 2023-06-22T08:11:53.554 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.41 2023-06-22T08:11:53.554 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.41 2023-06-22T08:11:59.145 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 19% (17/87) Updating files: 20% (18/87) Updating files: 21% (19/87) Updating files: 22% (20/87) Updating files: 24% (21/87) Updating files: 25% (22/87) Updating files: 26% (23/87) Updating files: 27% (24/87) Updating files: 28% (25/87) Updating files: 29% (26/87) Updating files: 31% (27/87) Updating files: 32% (28/87) Updating files: 33% (29/87) Updating files: 34% (30/87) Updating files: 35% (31/87) Updating files: 36% (32/87) Updating files: 37% (33/87) Updating files: 39% (34/87) Updating files: 40% (35/87) Updating files: 41% (36/87) Updating files: 42% (37/87) Updating files: 43% (38/87) Updating files: 44% (39/87) Updating files: 45% (40/87) Updating files: 47% (41/87) Updating files: 48% (42/87) Updating files: 49% (43/87) Updating files: 50% (44/87) Updating files: 51% (45/87) Updating files: 52% (46/87) Updating files: 54% (47/87) Updating files: 55% (48/87) Updating files: 56% (49/87) Updating files: 57% (50/87) Updating files: 58% (51/87) Updating files: 59% (52/87) Updating files: 60% (53/87) Updating files: 62% (54/87) Updating files: 63% (55/87) Updating files: 64% (56/87) Updating files: 65% (57/87) Updating files: 66% (58/87) Updating files: 67% (59/87) Updating files: 68% (60/87) Updating files: 70% (61/87) Updating files: 71% (62/87) Updating files: 72% (63/87) Updating files: 73% (64/87) Updating files: 74% (65/87) Updating files: 75% (66/87) Updating files: 77% (67/87) Updating files: 78% (68/87) Updating files: 79% (69/87) Updating files: 80% (70/87) Updating files: 81% (71/87) Updating files: 82% (72/87) Updating files: 83% (73/87) Updating files: 85% (74/87) Updating files: 86% (75/87) Updating files: 87% (76/87) Updating files: 88% (77/87) Updating files: 89% (78/87) Updating files: 90% (79/87) Updating files: 91% (80/87) Updating files: 93% (81/87) Updating files: 94% (82/87) Updating files: 95% (83/87) Updating files: 96% (84/87) Updating files: 97% (85/87) Updating files: 98% (86/87) Updating files: 100% (87/87) Updating files: 100% (87/87), done. 2023-06-22T08:11:59.245 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at c1345f7136a v0.41 2023-06-22T08:11:59.248 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.41 2023-06-22T08:11:59.306 INFO:tasks.workunit.client.0.smithi023.stdout:v0.42 2023-06-22T08:11:59.307 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:11:59.307 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 42 -eq 48 ']' 2023-06-22T08:11:59.307 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.42 2023-06-22T08:11:59.307 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.42 2023-06-22T08:11:59.307 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.42 2023-06-22T08:12:01.080 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 61% (153/250) Updating files: 62% (155/250) Updating files: 63% (158/250) Updating files: 64% (160/250) Updating files: 65% (163/250) Updating files: 66% (165/250) Updating files: 67% (168/250) Updating files: 68% (170/250) Updating files: 69% (173/250) Updating files: 70% (175/250) Updating files: 71% (178/250) Updating files: 72% (180/250) Updating files: 73% (183/250) Updating files: 74% (185/250) Updating files: 75% (188/250) Updating files: 76% (190/250) Updating files: 77% (193/250) Updating files: 78% (195/250) Updating files: 79% (198/250) Updating files: 80% (200/250) Updating files: 81% (203/250) Updating files: 82% (205/250) Updating files: 83% (208/250) Updating files: 84% (210/250) Updating files: 85% (213/250) Updating files: 86% (215/250) Updating files: 87% (218/250) Updating files: 88% (220/250) Updating files: 89% (223/250) Updating files: 90% (225/250) Updating files: 91% (228/250) Updating files: 92% (230/250) Updating files: 93% (233/250) Updating files: 94% (235/250) Updating files: 95% (238/250) Updating files: 96% (240/250) Updating files: 97% (243/250) Updating files: 98% (245/250) Updating files: 99% (248/250) Updating files: 100% (250/250) Updating files: 100% (250/250), done. 2023-06-22T08:12:01.229 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at b205c64c687 v0.42 2023-06-22T08:12:01.232 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.42 2023-06-22T08:12:01.282 INFO:tasks.workunit.client.0.smithi023.stdout:v0.43 2023-06-22T08:12:01.283 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:01.283 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 43 -eq 48 ']' 2023-06-22T08:12:01.283 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.43 2023-06-22T08:12:01.283 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.43 2023-06-22T08:12:01.283 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.43 2023-06-22T08:12:02.029 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 9fa8781c014 v0.43 2023-06-22T08:12:02.031 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.43 2023-06-22T08:12:02.091 INFO:tasks.workunit.client.0.smithi023.stdout:v0.44 2023-06-22T08:12:02.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:02.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 44 -eq 48 ']' 2023-06-22T08:12:02.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.44 2023-06-22T08:12:02.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.44 2023-06-22T08:12:02.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.44 2023-06-22T08:12:03.234 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at cdd5298d276 v0.44 2023-06-22T08:12:03.236 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.44 2023-06-22T08:12:03.325 INFO:tasks.workunit.client.0.smithi023.stdout:v0.45 2023-06-22T08:12:03.326 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:03.326 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 45 -eq 48 ']' 2023-06-22T08:12:03.326 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.45 2023-06-22T08:12:03.326 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.45 2023-06-22T08:12:03.326 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.45 2023-06-22T08:12:04.372 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 0aea1cb1df5 v0.45 2023-06-22T08:12:04.375 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.45 2023-06-22T08:12:04.448 INFO:tasks.workunit.client.0.smithi023.stdout:v0.46 2023-06-22T08:12:04.448 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:04.448 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 46 -eq 48 ']' 2023-06-22T08:12:04.449 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.46 2023-06-22T08:12:04.449 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.46 2023-06-22T08:12:04.449 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.46 2023-06-22T08:12:08.967 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:12:08.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:12:22.060 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 42% (130/309) Updating files: 43% (133/309) Updating files: 44% (136/309) Updating files: 45% (140/309) Updating files: 46% (143/309) Updating files: 47% (146/309) Updating files: 48% (149/309) Updating files: 49% (152/309) Updating files: 50% (155/309) Updating files: 51% (158/309) Updating files: 52% (161/309) Updating files: 53% (164/309) Updating files: 54% (167/309) Updating files: 55% (170/309) Updating files: 56% (174/309) Updating files: 57% (177/309) Updating files: 58% (180/309) Updating files: 58% (181/309) Updating files: 59% (183/309) Updating files: 60% (186/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2023-06-22T08:12:22.152 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at cb7f1c9c752 v0.46 2023-06-22T08:12:22.156 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.46 2023-06-22T08:12:22.231 INFO:tasks.workunit.client.0.smithi023.stdout:v0.47 2023-06-22T08:12:22.232 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:22.232 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 47 -eq 48 ']' 2023-06-22T08:12:22.232 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.47 2023-06-22T08:12:22.232 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.47 2023-06-22T08:12:22.233 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.47 2023-06-22T08:12:23.504 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at ad663d58c89 v0.47 2023-06-22T08:12:23.507 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.47 2023-06-22T08:12:23.591 INFO:tasks.workunit.client.0.smithi023.stdout:v0.49 2023-06-22T08:12:23.591 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:23.591 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 48 -eq 48 ']' 2023-06-22T08:12:23.592 INFO:tasks.workunit.client.0.smithi023.stderr:+ continue 2023-06-22T08:12:23.592 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:23.592 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 49 -eq 48 ']' 2023-06-22T08:12:23.592 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.49 2023-06-22T08:12:23.592 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.49 2023-06-22T08:12:23.592 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.49 2023-06-22T08:12:26.158 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 28% (139/481) Updating files: 29% (140/481) Updating files: 30% (145/481) Updating files: 31% (150/481) Updating files: 32% (154/481) Updating files: 33% (159/481) Updating files: 34% (164/481) Updating files: 35% (169/481) Updating files: 36% (174/481) Updating files: 37% (178/481) Updating files: 38% (183/481) Updating files: 39% (188/481) Updating files: 40% (193/481) Updating files: 41% (198/481) Updating files: 42% (203/481) Updating files: 43% (207/481) Updating files: 44% (212/481) Updating files: 45% (217/481) Updating files: 46% (222/481) Updating files: 47% (227/481) Updating files: 48% (231/481) Updating files: 49% (236/481) Updating files: 50% (241/481) Updating files: 51% (246/481) Updating files: 52% (251/481) Updating files: 53% (255/481) Updating files: 54% (260/481) Updating files: 55% (265/481) Updating files: 56% (270/481) Updating files: 57% (275/481) Updating files: 58% (279/481) Updating files: 59% (284/481) Updating files: 60% (289/481) Updating files: 61% (294/481) Updating files: 62% (299/481) Updating files: 63% (304/481) Updating files: 64% (308/481) Updating files: 65% (313/481) Updating files: 66% (318/481) Updating files: 67% (323/481) Updating files: 68% (328/481) Updating files: 69% (332/481) Updating files: 70% (337/481) Updating files: 71% (342/481) Updating files: 72% (347/481) Updating files: 73% (352/481) Updating files: 74% (356/481) Updating files: 75% (361/481) Updating files: 76% (366/481) Updating files: 77% (371/481) Updating files: 78% (376/481) Updating files: 79% (380/481) Updating files: 80% (385/481) Updating files: 81% (390/481) Updating files: 81% (393/481) Updating files: 82% (395/481) Updating files: 83% (400/481) Updating files: 84% (405/481) Updating files: 85% (409/481) Updating files: 86% (414/481) Updating files: 87% (419/481) Updating files: 88% (424/481) Updating files: 89% (429/481) Updating files: 90% (433/481) Updating files: 91% (438/481) Updating files: 92% (443/481) Updating files: 93% (448/481) Updating files: 94% (453/481) Updating files: 95% (457/481) Updating files: 96% (462/481) Updating files: 97% (467/481) Updating files: 98% (472/481) Updating files: 99% (477/481) Updating files: 100% (481/481) Updating files: 100% (481/481), done. 2023-06-22T08:12:26.273 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at ca6265d0f4d v0.49 2023-06-22T08:12:26.279 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.49 2023-06-22T08:12:26.471 INFO:tasks.workunit.client.0.smithi023.stdout:v0.50 2023-06-22T08:12:26.471 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:26.471 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 50 -eq 48 ']' 2023-06-22T08:12:26.472 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.50 2023-06-22T08:12:26.472 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.50 2023-06-22T08:12:26.472 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.50 2023-06-22T08:12:39.088 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:12:39.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:12:42.617 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 79% (109/137) Updating files: 80% (110/137) Updating files: 81% (111/137) Updating files: 82% (113/137) Updating files: 83% (114/137) Updating files: 84% (116/137) Updating files: 85% (117/137) Updating files: 86% (118/137) Updating files: 87% (120/137) Updating files: 88% (121/137) Updating files: 89% (122/137) Updating files: 90% (124/137) Updating files: 91% (125/137) Updating files: 92% (127/137) Updating files: 93% (128/137) Updating files: 94% (129/137) Updating files: 95% (131/137) Updating files: 96% (132/137) Updating files: 97% (133/137) Updating files: 98% (135/137) Updating files: 99% (136/137) Updating files: 100% (137/137) Updating files: 100% (137/137), done. 2023-06-22T08:12:42.683 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at a4428bd0e19 v0.50 2023-06-22T08:12:42.686 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.50 2023-06-22T08:12:42.738 INFO:tasks.workunit.client.0.smithi023.stdout:v0.51 2023-06-22T08:12:42.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:42.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 51 -eq 48 ']' 2023-06-22T08:12:42.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.51 2023-06-22T08:12:42.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.51 2023-06-22T08:12:42.740 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.51 2023-06-22T08:12:43.784 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at c03ca95d235 v0.51 2023-06-22T08:12:43.787 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.51 2023-06-22T08:12:43.873 INFO:tasks.workunit.client.0.smithi023.stdout:v0.52 2023-06-22T08:12:43.873 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:43.873 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 52 -eq 48 ']' 2023-06-22T08:12:43.873 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.52 2023-06-22T08:12:43.873 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.52 2023-06-22T08:12:43.874 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.52 2023-06-22T08:12:45.056 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at e48859474c4 v0.52 2023-06-22T08:12:45.060 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.52 2023-06-22T08:12:45.094 INFO:tasks.workunit.client.0.smithi023.stdout:v0.53 2023-06-22T08:12:45.094 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:45.095 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 53 -eq 48 ']' 2023-06-22T08:12:45.095 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.53 2023-06-22T08:12:45.095 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.53 2023-06-22T08:12:45.095 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.53 2023-06-22T08:12:46.737 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 68% (264/386) Updating files: 69% (267/386) Updating files: 70% (271/386) Updating files: 71% (275/386) Updating files: 72% (278/386) Updating files: 73% (282/386) Updating files: 74% (286/386) Updating files: 75% (290/386) Updating files: 76% (294/386) Updating files: 77% (298/386) Updating files: 78% (302/386) Updating files: 79% (305/386) Updating files: 80% (309/386) Updating files: 81% (313/386) Updating files: 82% (317/386) Updating files: 83% (321/386) Updating files: 84% (325/386) Updating files: 85% (329/386) Updating files: 86% (332/386) Updating files: 87% (336/386) Updating files: 88% (340/386) Updating files: 89% (344/386) Updating files: 90% (348/386) Updating files: 91% (352/386) Updating files: 92% (356/386) Updating files: 93% (359/386) Updating files: 94% (363/386) Updating files: 95% (367/386) Updating files: 96% (371/386) Updating files: 97% (375/386) Updating files: 98% (379/386) Updating files: 99% (383/386) Updating files: 100% (386/386) Updating files: 100% (386/386), done. 2023-06-22T08:12:46.830 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 2528b5ee105 v0.43 2023-06-22T08:12:46.835 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.53 2023-06-22T08:12:46.952 INFO:tasks.workunit.client.0.smithi023.stdout:v0.54 2023-06-22T08:12:46.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:46.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 54 -eq 48 ']' 2023-06-22T08:12:46.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.54 2023-06-22T08:12:46.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.54 2023-06-22T08:12:46.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.54 2023-06-22T08:12:48.326 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 88% (187/212) Updating files: 89% (189/212) Updating files: 90% (191/212) Updating files: 91% (193/212) Updating files: 92% (196/212) Updating files: 93% (198/212) Updating files: 94% (200/212) Updating files: 95% (202/212) Updating files: 96% (204/212) Updating files: 97% (206/212) Updating files: 98% (208/212) Updating files: 99% (210/212) Updating files: 100% (212/212) Updating files: 100% (212/212), done. 2023-06-22T08:12:48.468 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 60b84b095b1 v0.54 2023-06-22T08:12:48.473 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.54 2023-06-22T08:12:48.537 INFO:tasks.workunit.client.0.smithi023.stdout:v0.55 2023-06-22T08:12:48.538 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:48.538 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 55 -eq 48 ']' 2023-06-22T08:12:48.538 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.55 2023-06-22T08:12:48.538 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.55 2023-06-22T08:12:48.538 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.55 2023-06-22T08:12:51.021 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 32% (178/540) Updating files: 33% (179/540) Updating files: 34% (184/540) Updating files: 35% (189/540) Updating files: 36% (195/540) Updating files: 37% (200/540) Updating files: 38% (206/540) Updating files: 39% (211/540) Updating files: 40% (216/540) Updating files: 41% (222/540) Updating files: 42% (227/540) Updating files: 43% (233/540) Updating files: 44% (238/540) Updating files: 45% (243/540) Updating files: 46% (249/540) Updating files: 47% (254/540) Updating files: 48% (260/540) Updating files: 49% (265/540) Updating files: 50% (270/540) Updating files: 51% (276/540) Updating files: 52% (281/540) Updating files: 53% (287/540) Updating files: 54% (292/540) Updating files: 55% (297/540) Updating files: 56% (303/540) Updating files: 57% (308/540) Updating files: 58% (314/540) Updating files: 59% (319/540) Updating files: 60% (324/540) Updating files: 61% (330/540) Updating files: 62% (335/540) Updating files: 63% (341/540) Updating files: 64% (346/540) Updating files: 65% (351/540) Updating files: 66% (357/540) Updating files: 67% (362/540) Updating files: 68% (368/540) Updating files: 69% (373/540) Updating files: 70% (378/540) Updating files: 71% (384/540) Updating files: 72% (389/540) Updating files: 73% (395/540) Updating files: 74% (400/540) Updating files: 75% (405/540) Updating files: 76% (411/540) Updating files: 77% (416/540) Updating files: 78% (422/540) Updating files: 79% (427/540) Updating files: 79% (428/540) Updating files: 80% (432/540) Updating files: 81% (438/540) Updating files: 82% (443/540) Updating files: 83% (449/540) Updating files: 84% (454/540) Updating files: 85% (459/540) Updating files: 86% (465/540) Updating files: 87% (470/540) Updating files: 88% (476/540) Updating files: 89% (481/540) Updating files: 90% (486/540) Updating files: 91% (492/540) Updating files: 92% (497/540) Updating files: 93% (503/540) Updating files: 94% (508/540) Updating files: 95% (513/540) Updating files: 96% (519/540) Updating files: 97% (524/540) Updating files: 98% (530/540) Updating files: 99% (535/540) Updating files: 100% (540/540) Updating files: 100% (540/540), done. 2023-06-22T08:12:51.160 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 690f8175606 v0.55 2023-06-22T08:12:51.167 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.55 2023-06-22T08:12:51.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:12:51.290 INFO:tasks.workunit.client.0.smithi023.stdout:v0.56 2023-06-22T08:12:51.290 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:12:51.291 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 56 -eq 48 ']' 2023-06-22T08:12:51.291 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.56 2023-06-22T08:12:51.291 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.56 2023-06-22T08:12:51.291 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.56 2023-06-22T08:12:51.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.644+0000 7fecae7ff700 1 Processor -- start 2023-06-22T08:12:51.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.644+0000 7fecae7ff700 1 -- start start 2023-06-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.645+0000 7fecae7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feca8071070 0x7feca8160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.645+0000 7fecae7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca80c51a0 con 0x7feca80c69e0 2023-06-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.645+0000 7fecae7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca80c44e0 con 0x7feca814f850 2023-06-22T08:12:51.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.645+0000 7fecae7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca80c5440 con 0x7feca8071070 2023-06-22T08:12:51.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.646+0000 7fecacd9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feca8071070 0x7feca8160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:51.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.646+0000 7feca7fff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feca814f850 0x7feca80c3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:47116/0 (socket says 172.21.15.23:47116) 2023-06-22T08:12:51.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.646+0000 7feca7fff700 1 -- 172.21.15.23:0/4255218471 learned_addr learned my addr 172.21.15.23:0/4255218471 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:12:51.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.646+0000 7feca77fe700 1 --1- 172.21.15.23:0/4255218471 >> v1:172.21.15.23:6789/0 conn(0x7feca80c69e0 0x7feca8155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34908/0 (socket says 172.21.15.23:34908) 2023-06-22T08:12:51.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.646+0000 7feca6ffd700 1 -- 172.21.15.23:0/4255218471 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636094948 0 0) 0x7feca80c44e0 con 0x7feca814f850 2023-06-22T08:12:51.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.646+0000 7feca6ffd700 1 -- 172.21.15.23:0/4255218471 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec900057c0 con 0x7feca814f850 2023-06-22T08:12:51.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.646+0000 7feca6ffd700 1 -- 172.21.15.23:0/4255218471 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619219336 0 0) 0x7feca80c51a0 con 0x7feca80c69e0 2023-06-22T08:12:51.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.646+0000 7fecacd9c700 1 -- 172.21.15.23:0/4255218471 >> v1:172.21.15.67:6789/0 conn(0x7feca814f850 legacy=0x7feca80c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:51.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.646+0000 7fecacd9c700 1 -- 172.21.15.23:0/4255218471 >> v1:172.21.15.23:6789/0 conn(0x7feca80c69e0 legacy=0x7feca8155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:51.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.646+0000 7fecacd9c700 1 -- 172.21.15.23:0/4255218471 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca80c5720 con 0x7feca8071070 2023-06-22T08:12:51.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.647+0000 7fecacd9c700 1 --2- 172.21.15.23:0/4255218471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca8071070 0x7feca8160eb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7feca0009d90 tx=0x7feca000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=444df7da245f7fad server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:51.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.647+0000 7feca6ffd700 1 -- 172.21.15.23:0/4255218471 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca000e070 con 0x7feca8071070 2023-06-22T08:12:51.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.647+0000 7feca6ffd700 1 -- 172.21.15.23:0/4255218471 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca000bb20 con 0x7feca8071070 2023-06-22T08:12:51.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.647+0000 7feca6ffd700 1 -- 172.21.15.23:0/4255218471 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca000bed0 con 0x7feca8071070 2023-06-22T08:12:51.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.647+0000 7fecae7ff700 1 -- 172.21.15.23:0/4255218471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca8071070 msgr2=0x7feca8160eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:51.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.647+0000 7fecae7ff700 1 --2- 172.21.15.23:0/4255218471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca8071070 0x7feca8160eb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7feca0009d90 tx=0x7feca000b330 comp rx=0 tx=0).stop 2023-06-22T08:12:51.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.647+0000 7fecae7ff700 1 -- 172.21.15.23:0/4255218471 shutdown_connections 2023-06-22T08:12:51.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.647+0000 7fecae7ff700 1 --2- 172.21.15.23:0/4255218471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca8071070 0x7feca8160eb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:51.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.647+0000 7fecae7ff700 1 -- 172.21.15.23:0/4255218471 >> 172.21.15.23:0/4255218471 conn(0x7feca80ca3f0 msgr2=0x7feca80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:12:51.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.647+0000 7fecae7ff700 1 -- 172.21.15.23:0/4255218471 shutdown_connections 2023-06-22T08:12:51.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.648+0000 7fecae7ff700 1 -- 172.21.15.23:0/4255218471 wait complete. 2023-06-22T08:12:51.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.650+0000 7fecae7ff700 1 Processor -- start 2023-06-22T08:12:51.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.650+0000 7fecae7ff700 1 -- start start 2023-06-22T08:12:51.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.651+0000 7fecae7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feca8071070 0x7feca814dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:51.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.651+0000 7fecae7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca80c69e0 0x7feca814aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:51.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.651+0000 7fecae7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7feca814f850 0x7feca814ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:51.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.651+0000 7fecae7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca81eac20 con 0x7feca8071070 2023-06-22T08:12:51.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.651+0000 7fecae7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca814ec40 con 0x7feca80c69e0 2023-06-22T08:12:51.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.651+0000 7fecae7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca81ea000 con 0x7feca814f850 2023-06-22T08:12:51.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7feca7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feca8071070 0x7feca814dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:51.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7feca7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feca8071070 0x7feca814dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60842/0 (socket says 172.21.15.23:60842) 2023-06-22T08:12:51.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7feca7fff700 1 -- 172.21.15.23:0/1027684980 learned_addr learned my addr 172.21.15.23:0/1027684980 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:12:51.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7fecacd9c700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7feca814f850 0x7feca814ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:51.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7feca77fe700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca80c69e0 0x7feca814aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:51.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7feca7fff700 1 -- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7feca814f850 msgr2=0x7feca814ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7feca7fff700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7feca814f850 0x7feca814ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7feca7fff700 1 -- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca80c69e0 msgr2=0x7feca814aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7feca7fff700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca80c69e0 0x7feca814aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7feca7fff700 1 -- 172.21.15.23:0/1027684980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca0009a70 con 0x7feca8071070 2023-06-22T08:12:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.652+0000 7feca77fe700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca80c69e0 0x7feca814aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:12:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.653+0000 7feca7fff700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feca8071070 0x7feca814dfe0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fec9800c450 tx=0x7fec9800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.653+0000 7feca4ff9700 1 -- 172.21.15.23:0/1027684980 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98005d70 con 0x7feca8071070 2023-06-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.653+0000 7feca4ff9700 1 -- 172.21.15.23:0/1027684980 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9800ce20 con 0x7feca8071070 2023-06-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.653+0000 7fecae7ff700 1 -- 172.21.15.23:0/1027684980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca814e690 con 0x7feca8071070 2023-06-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.653+0000 7feca4ff9700 1 -- 172.21.15.23:0/1027684980 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec980072d0 con 0x7feca8071070 2023-06-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.653+0000 7fecae7ff700 1 -- 172.21.15.23:0/1027684980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca81ea560 con 0x7feca8071070 2023-06-22T08:12:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.654+0000 7fecae7ff700 1 -- 172.21.15.23:0/1027684980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca804ea20 con 0x7feca8071070 2023-06-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.655+0000 7feca4ff9700 1 -- 172.21.15.23:0/1027684980 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fec98007430 con 0x7feca8071070 2023-06-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.656+0000 7feca4ff9700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fec9406ad10 0x7fec9406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.656+0000 7feca4ff9700 1 -- 172.21.15.23:0/1027684980 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7fec98088be0 con 0x7feca8071070 2023-06-22T08:12:51.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.656+0000 7feca77fe700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fec9406ad10 0x7fec9406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:51.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.657+0000 7feca77fe700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fec9406ad10 0x7fec9406d1c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7feca8070cc0 tx=0x7fec9c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:51.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.661+0000 7feca4ff9700 1 -- 172.21.15.23:0/1027684980 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fec980537a0 con 0x7feca8071070 2023-06-22T08:12:51.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.925+0000 7fecae7ff700 1 -- 172.21.15.23:0/1027684980 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feca80cb300 con 0x7feca8071070 2023-06-22T08:12:51.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.925+0000 7feca4ff9700 1 -- 172.21.15.23:0/1027684980 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+6275 (secure 0 0 0) 0x7fec98056dc0 con 0x7feca8071070 2023-06-22T08:12:51.930 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:12:51.930 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5995,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:12:37.469779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:12:51.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 -- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fec9406ad10 msgr2=0x7fec9406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:51.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fec9406ad10 0x7fec9406d1c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7feca8070cc0 tx=0x7fec9c007de0 comp rx=0 tx=0).stop 2023-06-22T08:12:51.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 -- 172.21.15.23:0/1027684980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feca8071070 msgr2=0x7feca814dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:51.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feca8071070 0x7feca814dfe0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fec9800c450 tx=0x7fec9800c760 comp rx=0 tx=0).stop 2023-06-22T08:12:51.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 -- 172.21.15.23:0/1027684980 shutdown_connections 2023-06-22T08:12:51.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7feca814f850 0x7feca814ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:51.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feca8071070 0x7feca814dfe0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:51.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fec9406ad10 0x7fec9406d1c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:51.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 --2- 172.21.15.23:0/1027684980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca80c69e0 0x7feca814aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:51.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 -- 172.21.15.23:0/1027684980 >> 172.21.15.23:0/1027684980 conn(0x7feca80ca3f0 msgr2=0x7feca8069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:12:51.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 -- 172.21.15.23:0/1027684980 shutdown_connections 2023-06-22T08:12:51.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:51.928+0000 7fecae7ff700 1 -- 172.21.15.23:0/1027684980 wait complete. 2023-06-22T08:12:51.935 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 100 2023-06-22T08:12:51.950 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=3) 2023-06-22T08:12:51.950 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-22T08:12:51.951 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:12:51.963 INFO:tasks.ceph.mds.d:Stopped 2023-06-22T08:12:51.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:12:52.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.186+0000 7f31269ff700 1 Processor -- start 2023-06-22T08:12:52.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.187+0000 7f31269ff700 1 -- start start 2023-06-22T08:12:52.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.188+0000 7f31269ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3120152ce0 0x7f31200cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:52.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.188+0000 7f31269ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312015c370 con 0x7f31200c9fd0 2023-06-22T08:12:52.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.188+0000 7f31269ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312015bda0 con 0x7f31200c40b0 2023-06-22T08:12:52.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.188+0000 7f31269ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31201530b0 con 0x7f3120152ce0 2023-06-22T08:12:52.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.188+0000 7f311f7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f31200c9fd0 0x7f312015c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34916/0 (socket says 172.21.15.23:34916) 2023-06-22T08:12:52.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.188+0000 7f311f7fe700 1 -- 172.21.15.23:0/3729007957 learned_addr learned my addr 172.21.15.23:0/3729007957 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:12:52.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.188+0000 7f311ffff700 1 --2- 172.21.15.23:0/3729007957 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3120152ce0 0x7f31200cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:52.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.188+0000 7f311effd700 1 -- 172.21.15.23:0/3729007957 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064808016 0 0) 0x7f312015bda0 con 0x7f31200c40b0 2023-06-22T08:12:52.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.189+0000 7f311effd700 1 -- 172.21.15.23:0/3729007957 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31080035b0 con 0x7f31200c40b0 2023-06-22T08:12:52.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.189+0000 7f311effd700 1 -- 172.21.15.23:0/3729007957 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632593129 0 0) 0x7f312015c370 con 0x7f31200c9fd0 2023-06-22T08:12:52.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.189+0000 7f311effd700 1 -- 172.21.15.23:0/3729007957 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f312015bda0 con 0x7f31200c9fd0 2023-06-22T08:12:52.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.189+0000 7f311ffff700 1 -- 172.21.15.23:0/3729007957 >> v1:172.21.15.67:6789/0 conn(0x7f31200c40b0 legacy=0x7f31200c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:52.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.189+0000 7f311ffff700 1 -- 172.21.15.23:0/3729007957 >> v1:172.21.15.23:6789/0 conn(0x7f31200c9fd0 legacy=0x7f312015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:52.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.189+0000 7f311ffff700 1 -- 172.21.15.23:0/3729007957 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31200c9460 con 0x7f3120152ce0 2023-06-22T08:12:52.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.189+0000 7f311ffff700 1 --2- 172.21.15.23:0/3729007957 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3120152ce0 0x7f31200cf1b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3110009870 tx=0x7f3110009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d22bdb7731febd78 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:52.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.189+0000 7f311effd700 1 -- 172.21.15.23:0/3729007957 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110011830 con 0x7f3120152ce0 2023-06-22T08:12:52.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.189+0000 7f311effd700 1 -- 172.21.15.23:0/3729007957 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110011990 con 0x7f3120152ce0 2023-06-22T08:12:52.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.189+0000 7f311effd700 1 -- 172.21.15.23:0/3729007957 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110011d20 con 0x7f3120152ce0 2023-06-22T08:12:52.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.190+0000 7f31269ff700 1 -- 172.21.15.23:0/3729007957 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3120152ce0 msgr2=0x7f31200cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:52.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.190+0000 7f31269ff700 1 --2- 172.21.15.23:0/3729007957 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3120152ce0 0x7f31200cf1b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3110009870 tx=0x7f3110009b80 comp rx=0 tx=0).stop 2023-06-22T08:12:52.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.190+0000 7f31269ff700 1 -- 172.21.15.23:0/3729007957 shutdown_connections 2023-06-22T08:12:52.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.190+0000 7f31269ff700 1 --2- 172.21.15.23:0/3729007957 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3120152ce0 0x7f31200cf1b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:52.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.190+0000 7f31269ff700 1 -- 172.21.15.23:0/3729007957 >> 172.21.15.23:0/3729007957 conn(0x7f312006ae30 msgr2=0x7f312006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:12:52.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.190+0000 7f31269ff700 1 -- 172.21.15.23:0/3729007957 shutdown_connections 2023-06-22T08:12:52.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.190+0000 7f31269ff700 1 -- 172.21.15.23:0/3729007957 wait complete. 2023-06-22T08:12:52.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.192+0000 7f31269ff700 1 Processor -- start 2023-06-22T08:12:52.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.192+0000 7f31269ff700 1 -- start start 2023-06-22T08:12:52.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f31269ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f31200c40b0 0x7f31201f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:52.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f31269ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31200c9fd0 0x7f312015ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:52.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f31269ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120152ce0 0x7f312015b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:52.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f31269ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3120157610 con 0x7f31200c9fd0 2023-06-22T08:12:52.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f31269ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31201f1be0 con 0x7f3120152ce0 2023-06-22T08:12:52.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f31269ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3120157ba0 con 0x7f31200c40b0 2023-06-22T08:12:52.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f311f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31200c9fd0 0x7f312015ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:52.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f311f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31200c9fd0 0x7f312015ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60852/0 (socket says 172.21.15.23:60852) 2023-06-22T08:12:52.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f311f7fe700 1 -- 172.21.15.23:0/1865127502 learned_addr learned my addr 172.21.15.23:0/1865127502 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:12:52.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f311ffff700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f31200c40b0 0x7f31201f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:52.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f3124f9c700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120152ce0 0x7f312015b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:52.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f311f7fe700 1 -- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f31200c40b0 msgr2=0x7f31201f2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:52.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f311f7fe700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f31200c40b0 0x7f31201f2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:52.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f311f7fe700 1 -- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120152ce0 msgr2=0x7f312015b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:52.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f311f7fe700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120152ce0 0x7f312015b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:52.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.193+0000 7f311f7fe700 1 -- 172.21.15.23:0/1865127502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100094a0 con 0x7f31200c9fd0 2023-06-22T08:12:52.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.194+0000 7f311f7fe700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31200c9fd0 0x7f312015ac40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f311400c700 tx=0x7f311400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:52.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.194+0000 7f311cff9700 1 -- 172.21.15.23:0/1865127502 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114005d70 con 0x7f31200c9fd0 2023-06-22T08:12:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.194+0000 7f311cff9700 1 -- 172.21.15.23:0/1865127502 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31140070d0 con 0x7f31200c9fd0 2023-06-22T08:12:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.194+0000 7f31269ff700 1 -- 172.21.15.23:0/1865127502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31201f25b0 con 0x7f31200c9fd0 2023-06-22T08:12:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.194+0000 7f311cff9700 1 -- 172.21.15.23:0/1865127502 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114007460 con 0x7f31200c9fd0 2023-06-22T08:12:52.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.194+0000 7f31269ff700 1 -- 172.21.15.23:0/1865127502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3120158120 con 0x7f31200c9fd0 2023-06-22T08:12:52.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.195+0000 7f31269ff700 1 -- 172.21.15.23:0/1865127502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3120158340 con 0x7f31200c9fd0 2023-06-22T08:12:52.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.196+0000 7f311cff9700 1 -- 172.21.15.23:0/1865127502 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f31140075c0 con 0x7f31200c9fd0 2023-06-22T08:12:52.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.197+0000 7f311cff9700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f310c06ad10 0x7f310c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:52.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.197+0000 7f311cff9700 1 -- 172.21.15.23:0/1865127502 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f3114088e20 con 0x7f31200c9fd0 2023-06-22T08:12:52.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.197+0000 7f311ffff700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f310c06ad10 0x7f310c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:52.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.198+0000 7f311ffff700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f310c06ad10 0x7f310c06d1c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f31100090b0 tx=0x7f311001d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:52.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.201+0000 7f311cff9700 1 -- 172.21.15.23:0/1865127502 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3114053930 con 0x7f31200c9fd0 2023-06-22T08:12:52.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.454+0000 7f31269ff700 1 -- 172.21.15.23:0/1865127502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f312008bf10 con 0x7f31200c9fd0 2023-06-22T08:12:52.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.455+0000 7f311cff9700 1 -- 172.21.15.23:0/1865127502 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+6275 (secure 0 0 0) 0x7f3114017090 con 0x7f31200c9fd0 2023-06-22T08:12:52.459 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:12:52.459 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5995,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:12:37.469779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:12:52.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.457+0000 7f31269ff700 1 -- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f310c06ad10 msgr2=0x7f310c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:52.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.457+0000 7f31269ff700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f310c06ad10 0x7f310c06d1c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f31100090b0 tx=0x7f311001d040 comp rx=0 tx=0).stop 2023-06-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.458+0000 7f31269ff700 1 -- 172.21.15.23:0/1865127502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31200c9fd0 msgr2=0x7f312015ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.458+0000 7f31269ff700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31200c9fd0 0x7f312015ac40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f311400c700 tx=0x7f311400ca10 comp rx=0 tx=0).stop 2023-06-22T08:12:52.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.458+0000 7f31269ff700 1 -- 172.21.15.23:0/1865127502 shutdown_connections 2023-06-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.458+0000 7f31269ff700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f310c06ad10 0x7f310c06d1c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.458+0000 7f31269ff700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120152ce0 0x7f312015b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.458+0000 7f31269ff700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f31200c40b0 0x7f31201f2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.458+0000 7f31269ff700 1 --2- 172.21.15.23:0/1865127502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31200c9fd0 0x7f312015ac40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:52.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.458+0000 7f31269ff700 1 -- 172.21.15.23:0/1865127502 >> 172.21.15.23:0/1865127502 conn(0x7f312006ae30 msgr2=0x7f312014c530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:12:52.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.458+0000 7f31269ff700 1 -- 172.21.15.23:0/1865127502 shutdown_connections 2023-06-22T08:12:52.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:52.458+0000 7f31269ff700 1 -- 172.21.15.23:0/1865127502 wait complete. 2023-06-22T08:12:52.465 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 100 2023-06-22T08:12:52.479 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:12:54.481 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:12:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.728+0000 7fc03e7ff700 1 Processor -- start 2023-06-22T08:12:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.728+0000 7fc03e7ff700 1 -- start start 2023-06-22T08:12:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc03e7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 0x7fc0380c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc03e7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc03806d050 0x7fc038159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc03e7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0380c6a60 con 0x7fc038156fc0 2023-06-22T08:12:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc03e7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc038157570 con 0x7fc03806d050 2023-06-22T08:12:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc03e7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0380c4350 con 0x7fc0380c3a40 2023-06-22T08:12:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc03cd9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc038156fc0 0x7fc038069eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34928/0 (socket says 172.21.15.23:34928) 2023-06-22T08:12:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc03cd9c700 1 -- 172.21.15.23:0/1580474557 learned_addr learned my addr 172.21.15.23:0/1580474557 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:12:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc037fff700 1 --2- 172.21.15.23:0/1580474557 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 0x7fc0380c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc0377fe700 1 --2- 172.21.15.23:0/1580474557 >> v2:172.21.15.67:3300/0 conn(0x7fc03806d050 0x7fc038159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc036ffd700 1 -- 172.21.15.23:0/1580474557 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590891261 0 0) 0x7fc0380c6a60 con 0x7fc038156fc0 2023-06-22T08:12:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.730+0000 7fc036ffd700 1 -- 172.21.15.23:0/1580474557 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0200035b0 con 0x7fc038156fc0 2023-06-22T08:12:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.731+0000 7fc037fff700 1 -- 172.21.15.23:0/1580474557 >> v2:172.21.15.67:3300/0 conn(0x7fc03806d050 msgr2=0x7fc038159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:12:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.731+0000 7fc037fff700 1 --2- 172.21.15.23:0/1580474557 >> v2:172.21.15.67:3300/0 conn(0x7fc03806d050 0x7fc038159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.731+0000 7fc037fff700 1 -- 172.21.15.23:0/1580474557 >> v1:172.21.15.23:6789/0 conn(0x7fc038156fc0 legacy=0x7fc038069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.731+0000 7fc037fff700 1 -- 172.21.15.23:0/1580474557 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03814d690 con 0x7fc0380c3a40 2023-06-22T08:12:54.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.731+0000 7fc0377fe700 1 --2- 172.21.15.23:0/1580474557 >> v2:172.21.15.67:3300/0 conn(0x7fc03806d050 0x7fc038159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:12:54.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.731+0000 7fc037fff700 1 --2- 172.21.15.23:0/1580474557 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 0x7fc0380c3e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc028009840 tx=0x7fc028009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=824bc9bc7225d589 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:54.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.731+0000 7fc036ffd700 1 -- 172.21.15.23:0/1580474557 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028011830 con 0x7fc0380c3a40 2023-06-22T08:12:54.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.731+0000 7fc036ffd700 1 -- 172.21.15.23:0/1580474557 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc028011990 con 0x7fc0380c3a40 2023-06-22T08:12:54.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.731+0000 7fc036ffd700 1 -- 172.21.15.23:0/1580474557 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028011d20 con 0x7fc0380c3a40 2023-06-22T08:12:54.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.732+0000 7fc03e7ff700 1 -- 172.21.15.23:0/1580474557 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 msgr2=0x7fc0380c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:54.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.732+0000 7fc03e7ff700 1 --2- 172.21.15.23:0/1580474557 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 0x7fc0380c3e10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc028009840 tx=0x7fc028009b50 comp rx=0 tx=0).stop 2023-06-22T08:12:54.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.732+0000 7fc03e7ff700 1 -- 172.21.15.23:0/1580474557 shutdown_connections 2023-06-22T08:12:54.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.732+0000 7fc03e7ff700 1 --2- 172.21.15.23:0/1580474557 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 0x7fc0380c3e10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:54.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.732+0000 7fc03e7ff700 1 --2- 172.21.15.23:0/1580474557 >> v2:172.21.15.67:3300/0 conn(0x7fc03806d050 0x7fc038159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:54.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.732+0000 7fc03e7ff700 1 -- 172.21.15.23:0/1580474557 >> 172.21.15.23:0/1580474557 conn(0x7fc0380ca3f0 msgr2=0x7fc0380ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:12:54.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.732+0000 7fc03e7ff700 1 -- 172.21.15.23:0/1580474557 shutdown_connections 2023-06-22T08:12:54.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.732+0000 7fc03e7ff700 1 -- 172.21.15.23:0/1580474557 wait complete. 2023-06-22T08:12:54.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.734+0000 7fc03e7ff700 1 Processor -- start 2023-06-22T08:12:54.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.734+0000 7fc03e7ff700 1 -- start start 2023-06-22T08:12:54.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc03e7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03806d050 0x7fc0381e04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:54.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc03e7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 0x7fc0381f15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:54.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc03e7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc038156fc0 0x7fc0380c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:54.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc03e7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0381e0010 con 0x7fc03806d050 2023-06-22T08:12:54.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc03e7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03814b260 con 0x7fc038156fc0 2023-06-22T08:12:54.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc03e7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc038070440 con 0x7fc0380c3a40 2023-06-22T08:12:54.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc037fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03806d050 0x7fc0381e04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc037fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03806d050 0x7fc0381e04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60862/0 (socket says 172.21.15.23:60862) 2023-06-22T08:12:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc037fff700 1 -- 172.21.15.23:0/755542229 learned_addr learned my addr 172.21.15.23:0/755542229 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:12:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc0377fe700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 0x7fc0381f15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.735+0000 7fc03cd9c700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc038156fc0 0x7fc0380c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc037fff700 1 -- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 msgr2=0x7fc0381f15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc037fff700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 0x7fc0381f15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc037fff700 1 -- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc038156fc0 msgr2=0x7fc0380c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:54.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc037fff700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc038156fc0 0x7fc0380c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:54.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc037fff700 1 -- 172.21.15.23:0/755542229 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c0094a0 con 0x7fc03806d050 2023-06-22T08:12:54.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc0377fe700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 0x7fc0381f15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:12:54.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc037fff700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03806d050 0x7fc0381e04a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc028000c00 tx=0x7fc028004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:54.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc034ff9700 1 -- 172.21.15.23:0/755542229 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028004590 con 0x7fc03806d050 2023-06-22T08:12:54.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc034ff9700 1 -- 172.21.15.23:0/755542229 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02801f070 con 0x7fc03806d050 2023-06-22T08:12:54.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc034ff9700 1 -- 172.21.15.23:0/755542229 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0280038e0 con 0x7fc03806d050 2023-06-22T08:12:54.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc03e7ff700 1 -- 172.21.15.23:0/755542229 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0280094a0 con 0x7fc03806d050 2023-06-22T08:12:54.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.736+0000 7fc03e7ff700 1 -- 172.21.15.23:0/755542229 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03814aca0 con 0x7fc03806d050 2023-06-22T08:12:54.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.738+0000 7fc03e7ff700 1 -- 172.21.15.23:0/755542229 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0381e02f0 con 0x7fc03806d050 2023-06-22T08:12:54.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.738+0000 7fc034ff9700 1 -- 172.21.15.23:0/755542229 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc028003a40 con 0x7fc03806d050 2023-06-22T08:12:54.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.739+0000 7fc034ff9700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc02406ace0 0x7fc02406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:54.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.739+0000 7fc034ff9700 1 -- 172.21.15.23:0/755542229 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7fc028021070 con 0x7fc03806d050 2023-06-22T08:12:54.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.739+0000 7fc0377fe700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc02406ace0 0x7fc02406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:54.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.740+0000 7fc0377fe700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc02406ace0 0x7fc02406d190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc02c011380 tx=0x7fc02c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:54.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:54.744+0000 7fc034ff9700 1 -- 172.21.15.23:0/755542229 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc028055cd0 con 0x7fc03806d050 2023-06-22T08:12:55.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.092+0000 7fc03e7ff700 1 -- 172.21.15.23:0/755542229 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc03806f580 con 0x7fc03806d050 2023-06-22T08:12:55.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.092+0000 7fc034ff9700 1 -- 172.21.15.23:0/755542229 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+6275 (secure 0 0 0) 0x7fc0280592f0 con 0x7fc03806d050 2023-06-22T08:12:55.097 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:12:55.097 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5995,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:12:37.469779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:12:55.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 -- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc02406ace0 msgr2=0x7fc02406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:55.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc02406ace0 0x7fc02406d190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc02c011380 tx=0x7fc02c009210 comp rx=0 tx=0).stop 2023-06-22T08:12:55.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 -- 172.21.15.23:0/755542229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03806d050 msgr2=0x7fc0381e04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:55.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03806d050 0x7fc0381e04a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc028000c00 tx=0x7fc028004750 comp rx=0 tx=0).stop 2023-06-22T08:12:55.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 -- 172.21.15.23:0/755542229 shutdown_connections 2023-06-22T08:12:55.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc02406ace0 0x7fc02406d190 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:55.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc038156fc0 0x7fc0380c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:55.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc03806d050 0x7fc0381e04a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:55.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 --2- 172.21.15.23:0/755542229 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc0380c3a40 0x7fc0381f15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:55.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 -- 172.21.15.23:0/755542229 >> 172.21.15.23:0/755542229 conn(0x7fc0380ca3f0 msgr2=0x7fc0380c6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:12:55.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 -- 172.21.15.23:0/755542229 shutdown_connections 2023-06-22T08:12:55.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:55.095+0000 7fc03e7ff700 1 -- 172.21.15.23:0/755542229 wait complete. 2023-06-22T08:12:55.100 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 100 2023-06-22T08:12:55.116 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:12:57.118 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:12:57.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.321+0000 7f66aedff700 1 Processor -- start 2023-06-22T08:12:57.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.321+0000 7f66aedff700 1 -- start start 2023-06-22T08:12:57.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.322+0000 7f66aedff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f66a8152a10 0x7f66a80c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:57.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.322+0000 7f66aedff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a8159730 con 0x7f66a80c9200 2023-06-22T08:12:57.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.322+0000 7f66aedff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a8158910 con 0x7f66a815c0b0 2023-06-22T08:12:57.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.322+0000 7f66aedff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a80c5470 con 0x7f66a8152a10 2023-06-22T08:12:57.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.322+0000 7f66a7fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f66a80c9200 0x7f66a80cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:44802/0 (socket says 172.21.15.23:44802) 2023-06-22T08:12:57.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.322+0000 7f66a7fff700 1 -- 172.21.15.23:0/1090787981 learned_addr learned my addr 172.21.15.23:0/1090787981 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:12:57.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.322+0000 7f66acb9b700 1 --2- 172.21.15.23:0/1090787981 >> v2:172.21.15.67:3300/0 conn(0x7f66a8152a10 0x7f66a80c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.323+0000 7f66a77fe700 1 -- 172.21.15.23:0/1090787981 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049852510 0 0) 0x7f66a8158910 con 0x7f66a815c0b0 2023-06-22T08:12:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.323+0000 7f66a77fe700 1 -- 172.21.15.23:0/1090787981 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66980035b0 con 0x7f66a815c0b0 2023-06-22T08:12:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.323+0000 7f66acb9b700 1 -- 172.21.15.23:0/1090787981 >> v1:172.21.15.67:6789/0 conn(0x7f66a815c0b0 legacy=0x7f66a815c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:57.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.323+0000 7f66acb9b700 1 -- 172.21.15.23:0/1090787981 >> v1:172.21.15.23:6789/0 conn(0x7f66a80c9200 legacy=0x7f66a80cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:57.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.323+0000 7f66acb9b700 1 -- 172.21.15.23:0/1090787981 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8158bb0 con 0x7f66a8152a10 2023-06-22T08:12:57.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.323+0000 7f66acb9b700 1 --2- 172.21.15.23:0/1090787981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a8152a10 0x7f66a80c4a30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f669c009840 tx=0x7f669c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=265568d0495366be server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:57.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.324+0000 7f66a77fe700 1 -- 172.21.15.23:0/1090787981 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c011830 con 0x7f66a8152a10 2023-06-22T08:12:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.324+0000 7f66a77fe700 1 -- 172.21.15.23:0/1090787981 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c011990 con 0x7f66a8152a10 2023-06-22T08:12:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.324+0000 7f66a77fe700 1 -- 172.21.15.23:0/1090787981 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c011d20 con 0x7f66a8152a10 2023-06-22T08:12:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.324+0000 7f66aedff700 1 -- 172.21.15.23:0/1090787981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a8152a10 msgr2=0x7f66a80c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:57.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.324+0000 7f66aedff700 1 --2- 172.21.15.23:0/1090787981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a8152a10 0x7f66a80c4a30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f669c009840 tx=0x7f669c009b50 comp rx=0 tx=0).stop 2023-06-22T08:12:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.324+0000 7f66aedff700 1 -- 172.21.15.23:0/1090787981 shutdown_connections 2023-06-22T08:12:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.324+0000 7f66aedff700 1 --2- 172.21.15.23:0/1090787981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a8152a10 0x7f66a80c4a30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.324+0000 7f66aedff700 1 -- 172.21.15.23:0/1090787981 >> 172.21.15.23:0/1090787981 conn(0x7f66a8065b90 msgr2=0x7f66a8065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:12:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.324+0000 7f66aedff700 1 -- 172.21.15.23:0/1090787981 shutdown_connections 2023-06-22T08:12:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.324+0000 7f66aedff700 1 -- 172.21.15.23:0/1090787981 wait complete. 2023-06-22T08:12:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.326+0000 7f66aedff700 1 Processor -- start 2023-06-22T08:12:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.326+0000 7f66aedff700 1 -- start start 2023-06-22T08:12:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.327+0000 7f66aedff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66a80c9200 0x7f66a814eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.327+0000 7f66aedff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66a8152a10 0x7f66a81e2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.327+0000 7f66aedff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a815c0b0 0x7f66a81e2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.327+0000 7f66aedff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80b3580 con 0x7f66a80c9200 2023-06-22T08:12:57.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.327+0000 7f66aedff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80b45c0 con 0x7f66a815c0b0 2023-06-22T08:12:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.327+0000 7f66aedff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a80b2620 con 0x7f66a8152a10 2023-06-22T08:12:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.327+0000 7f66acb9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66a80c9200 0x7f66a814eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.327+0000 7f66a7fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66a8152a10 0x7f66a81e2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:57.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.327+0000 7f66ad39c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a815c0b0 0x7f66a81e2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:57.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66a7fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66a8152a10 0x7f66a81e2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35676/0 (socket says 172.21.15.23:35676) 2023-06-22T08:12:57.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66a7fff700 1 -- 172.21.15.23:0/3525021430 learned_addr learned my addr 172.21.15.23:0/3525021430 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:12:57.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66acb9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66a80c9200 0x7f66a814eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36186/0 (socket says 172.21.15.23:36186) 2023-06-22T08:12:57.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66acb9b700 1 -- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66a8152a10 msgr2=0x7f66a81e2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:57.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66acb9b700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66a8152a10 0x7f66a81e2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:57.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66acb9b700 1 -- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a815c0b0 msgr2=0x7f66a81e2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:57.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66acb9b700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a815c0b0 0x7f66a81e2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:57.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66acb9b700 1 -- 172.21.15.23:0/3525021430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c0094a0 con 0x7f66a80c9200 2023-06-22T08:12:57.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66a7fff700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66a8152a10 0x7f66a81e2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:12:57.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66acb9b700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66a80c9200 0x7f66a814eeb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f669c016000 tx=0x7f669c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:57.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66a57fa700 1 -- 172.21.15.23:0/3525021430 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c0036a0 con 0x7f66a80c9200 2023-06-22T08:12:57.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.328+0000 7f66a57fa700 1 -- 172.21.15.23:0/3525021430 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c01f070 con 0x7f66a80c9200 2023-06-22T08:12:57.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.329+0000 7f66aedff700 1 -- 172.21.15.23:0/3525021430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a80b2900 con 0x7f66a80c9200 2023-06-22T08:12:57.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.329+0000 7f66a57fa700 1 -- 172.21.15.23:0/3525021430 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c003a20 con 0x7f66a80c9200 2023-06-22T08:12:57.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.329+0000 7f66aedff700 1 -- 172.21.15.23:0/3525021430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a80b2ec0 con 0x7f66a80c9200 2023-06-22T08:12:57.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.330+0000 7f66aedff700 1 -- 172.21.15.23:0/3525021430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a804ea20 con 0x7f66a80c9200 2023-06-22T08:12:57.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.331+0000 7f66a57fa700 1 -- 172.21.15.23:0/3525021430 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f669c003b80 con 0x7f66a80c9200 2023-06-22T08:12:57.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.332+0000 7f66a57fa700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f668c06ace0 0x7f668c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:57.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.332+0000 7f66a57fa700 1 -- 172.21.15.23:0/3525021430 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f669c021070 con 0x7f66a80c9200 2023-06-22T08:12:57.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.332+0000 7f66a7fff700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f668c06ace0 0x7f668c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:57.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.333+0000 7f66a7fff700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f668c06ace0 0x7f668c06d190 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f66a815b740 tx=0x7f6694009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:57.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.336+0000 7f66a57fa700 1 -- 172.21.15.23:0/3525021430 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f669c055cd0 con 0x7f66a80c9200 2023-06-22T08:12:57.601 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.598+0000 7f66aedff700 1 -- 172.21.15.23:0/3525021430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66a80cbd20 con 0x7f66a80c9200 2023-06-22T08:12:57.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.599+0000 7f66a57fa700 1 -- 172.21.15.23:0/3525021430 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+6275 (secure 0 0 0) 0x7f669c015020 con 0x7f66a80c9200 2023-06-22T08:12:57.602 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:12:57.602 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5995,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:12:37.469779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:12:57.604 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 -- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f668c06ace0 msgr2=0x7f668c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:57.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f668c06ace0 0x7f668c06d190 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f66a815b740 tx=0x7f6694009040 comp rx=0 tx=0).stop 2023-06-22T08:12:57.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 -- 172.21.15.23:0/3525021430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66a80c9200 msgr2=0x7f66a814eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:57.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66a80c9200 0x7f66a814eeb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f669c016000 tx=0x7f669c004750 comp rx=0 tx=0).stop 2023-06-22T08:12:57.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 -- 172.21.15.23:0/3525021430 shutdown_connections 2023-06-22T08:12:57.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f668c06ace0 0x7f668c06d190 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:57.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66a815c0b0 0x7f66a81e2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:57.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66a80c9200 0x7f66a814eeb0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:57.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 --2- 172.21.15.23:0/3525021430 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66a8152a10 0x7f66a81e2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:57.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 -- 172.21.15.23:0/3525021430 >> 172.21.15.23:0/3525021430 conn(0x7f66a8065b90 msgr2=0x7f66a806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:12:57.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 -- 172.21.15.23:0/3525021430 shutdown_connections 2023-06-22T08:12:57.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:57.601+0000 7f66aedff700 1 -- 172.21.15.23:0/3525021430 wait complete. 2023-06-22T08:12:57.606 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 100 2023-06-22T08:12:57.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:12:59.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:12:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.864+0000 7f693f3ff700 1 Processor -- start 2023-06-22T08:12:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.864+0000 7f693f3ff700 1 -- start start 2023-06-22T08:12:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.865+0000 7f693f3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6938152d70 0x7f69380c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.865+0000 7f693f3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 0x7f69380c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.865+0000 7f693f3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69380c4f90 con 0x7f6938159330 2023-06-22T08:12:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.865+0000 7f693f3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69380c95e0 con 0x7f6938152d70 2023-06-22T08:12:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.865+0000 7f693f3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6938159790 con 0x7f69380c45f0 2023-06-22T08:12:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.865+0000 7f693d19b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6938152d70 0x7f69380c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.865+0000 7f693d19b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6938152d70 0x7f69380c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36194/0 (socket says 172.21.15.23:36194) 2023-06-22T08:12:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.865+0000 7f693d19b700 1 -- 172.21.15.23:0/70904853 learned_addr learned my addr 172.21.15.23:0/70904853 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:12:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.865+0000 7f693d99c700 1 --2- 172.21.15.23:0/70904853 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 0x7f69380c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.866+0000 7f693d19b700 1 -- 172.21.15.23:0/70904853 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 msgr2=0x7f69380c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:12:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.866+0000 7f693d19b700 1 --2- 172.21.15.23:0/70904853 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 0x7f69380c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.866+0000 7f693d19b700 1 -- 172.21.15.23:0/70904853 >> v1:172.21.15.67:6789/0 conn(0x7f6938159330 legacy=0x7f69380cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.866+0000 7f693d19b700 1 -- 172.21.15.23:0/70904853 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693815bea0 con 0x7f6938152d70 2023-06-22T08:12:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.866+0000 7f693d99c700 1 --2- 172.21.15.23:0/70904853 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 0x7f69380c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:12:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.866+0000 7f693d19b700 1 --2- 172.21.15.23:0/70904853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938152d70 0x7f69380c97d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f69340098f0 tx=0x7f6934009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9a57e335999fe3 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.866+0000 7f692ffff700 1 -- 172.21.15.23:0/70904853 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934011830 con 0x7f6938152d70 2023-06-22T08:12:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.866+0000 7f692ffff700 1 -- 172.21.15.23:0/70904853 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6934011990 con 0x7f6938152d70 2023-06-22T08:12:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.866+0000 7f692ffff700 1 -- 172.21.15.23:0/70904853 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934011d20 con 0x7f6938152d70 2023-06-22T08:12:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.867+0000 7f693f3ff700 1 -- 172.21.15.23:0/70904853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938152d70 msgr2=0x7f69380c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.867+0000 7f693f3ff700 1 --2- 172.21.15.23:0/70904853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938152d70 0x7f69380c97d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f69340098f0 tx=0x7f6934009c00 comp rx=0 tx=0).stop 2023-06-22T08:12:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.867+0000 7f693f3ff700 1 -- 172.21.15.23:0/70904853 shutdown_connections 2023-06-22T08:12:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.867+0000 7f693f3ff700 1 --2- 172.21.15.23:0/70904853 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 0x7f69380c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.867+0000 7f693f3ff700 1 --2- 172.21.15.23:0/70904853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938152d70 0x7f69380c97d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.867+0000 7f693f3ff700 1 -- 172.21.15.23:0/70904853 >> 172.21.15.23:0/70904853 conn(0x7f693806ae30 msgr2=0x7f693806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:12:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.867+0000 7f693f3ff700 1 -- 172.21.15.23:0/70904853 shutdown_connections 2023-06-22T08:12:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.867+0000 7f693f3ff700 1 -- 172.21.15.23:0/70904853 wait complete. 2023-06-22T08:12:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.869+0000 7f693f3ff700 1 Processor -- start 2023-06-22T08:12:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.869+0000 7f693f3ff700 1 -- start start 2023-06-22T08:12:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693f3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 0x7f69380cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693f3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6938152d70 0x7f69380c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693f3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938159330 0x7f69380c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693f3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6938151740 con 0x7f6938159330 2023-06-22T08:12:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693f3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69381ee470 con 0x7f6938152d70 2023-06-22T08:12:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693f3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69380ce180 con 0x7f69380c45f0 2023-06-22T08:12:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693d19b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 0x7f69380cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693c99a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6938152d70 0x7f69380c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693d99c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938159330 0x7f69380c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693d19b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 0x7f69380cdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35698/0 (socket says 172.21.15.23:35698) 2023-06-22T08:12:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693c99a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6938152d70 0x7f69380c7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51038/0 (socket says 172.21.15.23:51038) 2023-06-22T08:12:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.870+0000 7f693d19b700 1 -- 172.21.15.23:0/1161039502 learned_addr learned my addr 172.21.15.23:0/1161039502 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:12:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f693c99a700 1 -- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 msgr2=0x7f69380cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f693c99a700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 0x7f69380cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f693c99a700 1 -- 172.21.15.23:0/1161039502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938159330 msgr2=0x7f69380c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:12:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f693c99a700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938159330 0x7f69380c58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:12:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f693c99a700 1 -- 172.21.15.23:0/1161039502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6930009a70 con 0x7f6938152d70 2023-06-22T08:12:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f693c99a700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6938152d70 0x7f69380c7240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6928000f60 tx=0x7f692800c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f692dffb700 1 -- 172.21.15.23:0/1161039502 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280052d0 con 0x7f6938152d70 2023-06-22T08:12:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f692dffb700 1 -- 172.21.15.23:0/1161039502 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6928005430 con 0x7f6938152d70 2023-06-22T08:12:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f692dffb700 1 -- 172.21.15.23:0/1161039502 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280038a0 con 0x7f6938152d70 2023-06-22T08:12:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f693f3ff700 1 -- 172.21.15.23:0/1161039502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6934009550 con 0x7f6938152d70 2023-06-22T08:12:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.871+0000 7f693f3ff700 1 -- 172.21.15.23:0/1161039502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69380cf4e0 con 0x7f6938152d70 2023-06-22T08:12:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.872+0000 7f693f3ff700 1 -- 172.21.15.23:0/1161039502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693804ea20 con 0x7f6938152d70 2023-06-22T08:12:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.873+0000 7f692dffb700 1 -- 172.21.15.23:0/1161039502 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6928003a00 con 0x7f6938152d70 2023-06-22T08:12:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.874+0000 7f692dffb700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f691c06ace0 0x7f691c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:12:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.874+0000 7f692dffb700 1 -- 172.21.15.23:0/1161039502 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f6928088c80 con 0x7f6938152d70 2023-06-22T08:12:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.874+0000 7f693d19b700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f691c06ace0 0x7f691c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:12:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.875+0000 7f693d19b700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f691c06ace0 0x7f691c06d190 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6934009130 tx=0x7f6934008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:12:59.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:12:59.879+0000 7f692dffb700 1 -- 172.21.15.23:0/1161039502 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6928053790 con 0x7f6938152d70 2023-06-22T08:13:00.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.167+0000 7f693f3ff700 1 -- 172.21.15.23:0/1161039502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f693815c9f0 con 0x7f6938152d70 2023-06-22T08:13:00.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.168+0000 7f692dffb700 1 -- 172.21.15.23:0/1161039502 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+6275 (secure 0 0 0) 0x7f6928003cb0 con 0x7f6938152d70 2023-06-22T08:13:00.172 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:00.172 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5995,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:12:37.469779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.170+0000 7f693f3ff700 1 -- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f691c06ace0 msgr2=0x7f691c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.170+0000 7f693f3ff700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f691c06ace0 0x7f691c06d190 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6934009130 tx=0x7f6934008030 comp rx=0 tx=0).stop 2023-06-22T08:13:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.170+0000 7f693f3ff700 1 -- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6938152d70 msgr2=0x7f69380c7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.170+0000 7f693f3ff700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6938152d70 0x7f69380c7240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6928000f60 tx=0x7f692800c6e0 comp rx=0 tx=0).stop 2023-06-22T08:13:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.170+0000 7f693f3ff700 1 -- 172.21.15.23:0/1161039502 shutdown_connections 2023-06-22T08:13:00.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.171+0000 7f693f3ff700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938159330 0x7f69380c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:00.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.171+0000 7f693f3ff700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f69380c45f0 0x7f69380cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:00.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.171+0000 7f693f3ff700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f691c06ace0 0x7f691c06d190 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:00.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.171+0000 7f693f3ff700 1 --2- 172.21.15.23:0/1161039502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6938152d70 0x7f69380c7240 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:00.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.171+0000 7f693f3ff700 1 -- 172.21.15.23:0/1161039502 >> 172.21.15.23:0/1161039502 conn(0x7f693806ae30 msgr2=0x7f69380ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:00.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.171+0000 7f693f3ff700 1 -- 172.21.15.23:0/1161039502 shutdown_connections 2023-06-22T08:13:00.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:00.171+0000 7f693f3ff700 1 -- 172.21.15.23:0/1161039502 wait complete. 2023-06-22T08:13:00.176 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 100 2023-06-22T08:13:00.192 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:13:02.193 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.410+0000 7f22abdff700 1 Processor -- start 2023-06-22T08:13:02.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.411+0000 7f22abdff700 1 -- start start 2023-06-22T08:13:02.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.413+0000 7f22abdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 0x7f22a406d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.413+0000 7f22abdff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22a406a280 0x7f22a40c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.413+0000 7f22abdff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a40c4d70 con 0x7f22a415d9b0 2023-06-22T08:13:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.413+0000 7f22abdff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a40703c0 con 0x7f22a406a280 2023-06-22T08:13:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.413+0000 7f22abdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40c5610 con 0x7f22a4069eb0 2023-06-22T08:13:02.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.414+0000 7f22a939a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f22a415d9b0 0x7f22a406fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:35526/0 (socket says 172.21.15.23:35526) 2023-06-22T08:13:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.414+0000 7f22a939a700 1 -- 172.21.15.23:0/1148661956 learned_addr learned my addr 172.21.15.23:0/1148661956 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.414+0000 7f22aa39c700 1 --2- 172.21.15.23:0/1148661956 >> v2:172.21.15.67:3300/0 conn(0x7f22a406a280 0x7f22a40c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.414+0000 7f22a9b9b700 1 --2- 172.21.15.23:0/1148661956 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 0x7f22a406d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:02.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.415+0000 7f22aa39c700 1 -- 172.21.15.23:0/1148661956 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 msgr2=0x7f22a406d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.415+0000 7f22a8b99700 1 -- 172.21.15.23:0/1148661956 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80338617 0 0) 0x7f22a40c4d70 con 0x7f22a415d9b0 2023-06-22T08:13:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.415+0000 7f22aa39c700 1 --2- 172.21.15.23:0/1148661956 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 0x7f22a406d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.415+0000 7f22aa39c700 1 -- 172.21.15.23:0/1148661956 >> v1:172.21.15.67:6789/0 conn(0x7f22a415d9b0 legacy=0x7f22a406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.415+0000 7f22aa39c700 1 -- 172.21.15.23:0/1148661956 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a40c5fe0 con 0x7f22a406a280 2023-06-22T08:13:02.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.417+0000 7f22aa39c700 1 --2- 172.21.15.23:0/1148661956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a406a280 0x7f22a40c5aa0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f229c009d90 tx=0x7f229c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=444f3e8e4bec0bb6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.417+0000 7f22a8b99700 1 -- 172.21.15.23:0/1148661956 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c00e070 con 0x7f22a406a280 2023-06-22T08:13:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.417+0000 7f22a8b99700 1 -- 172.21.15.23:0/1148661956 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229c00bb20 con 0x7f22a406a280 2023-06-22T08:13:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.417+0000 7f22a8b99700 1 -- 172.21.15.23:0/1148661956 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f229c012430 con 0x7f22a406a280 2023-06-22T08:13:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.419+0000 7f22abdff700 1 -- 172.21.15.23:0/1148661956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a406a280 msgr2=0x7f22a40c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.419+0000 7f22abdff700 1 --2- 172.21.15.23:0/1148661956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a406a280 0x7f22a40c5aa0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f229c009d90 tx=0x7f229c00b330 comp rx=0 tx=0).stop 2023-06-22T08:13:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.419+0000 7f22abdff700 1 -- 172.21.15.23:0/1148661956 shutdown_connections 2023-06-22T08:13:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.419+0000 7f22abdff700 1 --2- 172.21.15.23:0/1148661956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a406a280 0x7f22a40c5aa0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.419+0000 7f22abdff700 1 --2- 172.21.15.23:0/1148661956 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 0x7f22a406d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.419+0000 7f22abdff700 1 -- 172.21.15.23:0/1148661956 >> 172.21.15.23:0/1148661956 conn(0x7f22a40ca3f0 msgr2=0x7f22a40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:02.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.421+0000 7f22abdff700 1 -- 172.21.15.23:0/1148661956 shutdown_connections 2023-06-22T08:13:02.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.422+0000 7f22abdff700 1 -- 172.21.15.23:0/1148661956 wait complete. 2023-06-22T08:13:02.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.425+0000 7f22abdff700 1 Processor -- start 2023-06-22T08:13:02.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.425+0000 7f22abdff700 1 -- start start 2023-06-22T08:13:02.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.426+0000 7f22abdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 0x7f22a41e9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:02.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22abdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a406a280 0x7f22a40c2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:02.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22abdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22a415d9b0 0x7f22a41508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:02.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22abdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4152f30 con 0x7f22a415d9b0 2023-06-22T08:13:02.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22abdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4152320 con 0x7f22a406a280 2023-06-22T08:13:02.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22abdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4152600 con 0x7f22a4069eb0 2023-06-22T08:13:02.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22a9b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 0x7f22a41e9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22a9b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 0x7f22a41e9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35718/0 (socket says 172.21.15.23:35718) 2023-06-22T08:13:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22a9b9b700 1 -- 172.21.15.23:0/1483709984 learned_addr learned my addr 172.21.15.23:0/1483709984 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22a939a700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a406a280 0x7f22a40c2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22aa39c700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22a415d9b0 0x7f22a41508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22a9b9b700 1 -- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a406a280 msgr2=0x7f22a40c2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22a9b9b700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a406a280 0x7f22a40c2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22a9b9b700 1 -- 172.21.15.23:0/1483709984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22a415d9b0 msgr2=0x7f22a41508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22a9b9b700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22a415d9b0 0x7f22a41508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.427+0000 7f22a9b9b700 1 -- 172.21.15.23:0/1483709984 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229c009a70 con 0x7f22a4069eb0 2023-06-22T08:13:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.428+0000 7f22a9b9b700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 0x7f22a41e9aa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22a000e9b0 tx=0x7f22a000ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.428+0000 7f229a7fc700 1 -- 172.21.15.23:0/1483709984 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a0009710 con 0x7f22a4069eb0 2023-06-22T08:13:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.428+0000 7f229a7fc700 1 -- 172.21.15.23:0/1483709984 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a000c850 con 0x7f22a4069eb0 2023-06-22T08:13:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.428+0000 7f22abdff700 1 -- 172.21.15.23:0/1483709984 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a41528e0 con 0x7f22a4069eb0 2023-06-22T08:13:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.428+0000 7f229a7fc700 1 -- 172.21.15.23:0/1483709984 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a000cbc0 con 0x7f22a4069eb0 2023-06-22T08:13:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.428+0000 7f22abdff700 1 -- 172.21.15.23:0/1483709984 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a40c42e0 con 0x7f22a4069eb0 2023-06-22T08:13:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.430+0000 7f22abdff700 1 -- 172.21.15.23:0/1483709984 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a404ea20 con 0x7f22a4069eb0 2023-06-22T08:13:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.430+0000 7f229a7fc700 1 -- 172.21.15.23:0/1483709984 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f22a000cd20 con 0x7f22a4069eb0 2023-06-22T08:13:02.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.432+0000 7f229a7fc700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f228806ace0 0x7f228806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:02.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.432+0000 7f229a7fc700 1 -- 172.21.15.23:0/1483709984 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f22a0089b30 con 0x7f22a4069eb0 2023-06-22T08:13:02.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.432+0000 7f22a939a700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f228806ace0 0x7f228806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:02.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.433+0000 7f22a939a700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f228806ace0 0x7f228806d190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f22a40c45e0 tx=0x7f2294007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:02.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.436+0000 7f229a7fc700 1 -- 172.21.15.23:0/1483709984 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f22a00545b0 con 0x7f22a4069eb0 2023-06-22T08:13:02.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.695+0000 7f22abdff700 1 -- 172.21.15.23:0/1483709984 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22a40cb300 con 0x7f22a4069eb0 2023-06-22T08:13:02.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.696+0000 7f229a7fc700 1 -- 172.21.15.23:0/1483709984 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+6275 (secure 0 0 0) 0x7f22a0017090 con 0x7f22a4069eb0 2023-06-22T08:13:02.700 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:02.700 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5995,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:12:37.469779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:02.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 -- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f228806ace0 msgr2=0x7f228806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:02.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f228806ace0 0x7f228806d190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f22a40c45e0 tx=0x7f2294007e70 comp rx=0 tx=0).stop 2023-06-22T08:13:02.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 -- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 msgr2=0x7f22a41e9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:02.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 0x7f22a41e9aa0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f22a000e9b0 tx=0x7f22a000ecc0 comp rx=0 tx=0).stop 2023-06-22T08:13:02.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 -- 172.21.15.23:0/1483709984 shutdown_connections 2023-06-22T08:13:02.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22a415d9b0 0x7f22a41508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:02.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f22a4069eb0 0x7f22a41e9aa0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:02.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f228806ace0 0x7f228806d190 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:02.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 --2- 172.21.15.23:0/1483709984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22a406a280 0x7f22a40c2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:02.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 -- 172.21.15.23:0/1483709984 >> 172.21.15.23:0/1483709984 conn(0x7f22a40ca3f0 msgr2=0x7f22a406d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:02.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 -- 172.21.15.23:0/1483709984 shutdown_connections 2023-06-22T08:13:02.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:02.699+0000 7f22abdff700 1 -- 172.21.15.23:0/1483709984 wait complete. 2023-06-22T08:13:02.705 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 100 2023-06-22T08:13:02.720 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:13:04.722 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:04.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.935+0000 7fb1b3fff700 1 Processor -- start 2023-06-22T08:13:04.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.935+0000 7fb1b3fff700 1 -- start start 2023-06-22T08:13:04.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.937+0000 7fb1b3fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac152e10 0x7fb1ac069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:04.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.937+0000 7fb1b3fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac158f60 con 0x7fb1ac0cb6b0 2023-06-22T08:13:04.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.937+0000 7fb1b3fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac0cc450 con 0x7fb1ac0c5740 2023-06-22T08:13:04.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.937+0000 7fb1b3fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac06bd20 con 0x7fb1ac152e10 2023-06-22T08:13:04.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.937+0000 7fb1b1d9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb1ac0cb6b0 0x7fb1ac152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:44812/0 (socket says 172.21.15.23:44812) 2023-06-22T08:13:04.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.937+0000 7fb1b1d9b700 1 -- 172.21.15.23:0/1354784117 learned_addr learned my addr 172.21.15.23:0/1354784117 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:04.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.937+0000 7fb1b159a700 1 --2- 172.21.15.23:0/1354784117 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac152e10 0x7fb1ac069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:04.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.937+0000 7fb1b0d99700 1 -- 172.21.15.23:0/1354784117 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045096191 0 0) 0x7fb1ac158f60 con 0x7fb1ac0cb6b0 2023-06-22T08:13:04.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.938+0000 7fb1b0d99700 1 -- 172.21.15.23:0/1354784117 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1980035b0 con 0x7fb1ac0cb6b0 2023-06-22T08:13:04.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.938+0000 7fb1b0d99700 1 -- 172.21.15.23:0/1354784117 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225313918 0 0) 0x7fb1ac0cc450 con 0x7fb1ac0c5740 2023-06-22T08:13:04.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.938+0000 7fb1b159a700 1 -- 172.21.15.23:0/1354784117 >> v1:172.21.15.67:6789/0 conn(0x7fb1ac0c5740 legacy=0x7fb1ac163110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:04.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.938+0000 7fb1b159a700 1 -- 172.21.15.23:0/1354784117 >> v1:172.21.15.23:6789/0 conn(0x7fb1ac0cb6b0 legacy=0x7fb1ac152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:04.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.938+0000 7fb1b159a700 1 -- 172.21.15.23:0/1354784117 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac158980 con 0x7fb1ac152e10 2023-06-22T08:13:04.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.938+0000 7fb1b159a700 1 --2- 172.21.15.23:0/1354784117 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac152e10 0x7fb1ac069eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb19c009870 tx=0x7fb19c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46967cf569730e78 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:04.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.938+0000 7fb1b0d99700 1 -- 172.21.15.23:0/1354784117 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c011830 con 0x7fb1ac152e10 2023-06-22T08:13:04.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.938+0000 7fb1b0d99700 1 -- 172.21.15.23:0/1354784117 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c011990 con 0x7fb1ac152e10 2023-06-22T08:13:04.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.938+0000 7fb1b0d99700 1 -- 172.21.15.23:0/1354784117 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c011d20 con 0x7fb1ac152e10 2023-06-22T08:13:04.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.939+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1354784117 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac152e10 msgr2=0x7fb1ac069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:04.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.939+0000 7fb1b3fff700 1 --2- 172.21.15.23:0/1354784117 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac152e10 0x7fb1ac069eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb19c009870 tx=0x7fb19c009b80 comp rx=0 tx=0).stop 2023-06-22T08:13:04.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.939+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1354784117 shutdown_connections 2023-06-22T08:13:04.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.939+0000 7fb1b3fff700 1 --2- 172.21.15.23:0/1354784117 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac152e10 0x7fb1ac069eb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:04.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.939+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1354784117 >> 172.21.15.23:0/1354784117 conn(0x7fb1ac070d80 msgr2=0x7fb1ac071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:04.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.939+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1354784117 shutdown_connections 2023-06-22T08:13:04.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.939+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1354784117 wait complete. 2023-06-22T08:13:04.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.941+0000 7fb1b3fff700 1 Processor -- start 2023-06-22T08:13:04.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.941+0000 7fb1b3fff700 1 -- start start 2023-06-22T08:13:04.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b3fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac0cb6b0 0x7fb1ac157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:04.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1ac152e10 0x7fb1ac146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:04.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ac1e2490 0x7fb1ac1568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:04.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b3fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac15b760 con 0x7fb1ac152e10 2023-06-22T08:13:04.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b3fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac1e1080 con 0x7fb1ac1e2490 2023-06-22T08:13:04.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b3fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac15b950 con 0x7fb1ac0cb6b0 2023-06-22T08:13:04.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b1d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac0cb6b0 0x7fb1ac157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:04.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b1d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac0cb6b0 0x7fb1ac157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35728/0 (socket says 172.21.15.23:35728) 2023-06-22T08:13:04.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b1d9b700 1 -- 172.21.15.23:0/1448360407 learned_addr learned my addr 172.21.15.23:0/1448360407 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:04.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b159a700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1ac152e10 0x7fb1ac146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:04.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b259c700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ac1e2490 0x7fb1ac1568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:04.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b159a700 1 -- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac0cb6b0 msgr2=0x7fb1ac157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:04.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b159a700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac0cb6b0 0x7fb1ac157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:04.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b159a700 1 -- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ac1e2490 msgr2=0x7fb1ac1568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:04.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b159a700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ac1e2490 0x7fb1ac1568c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:04.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.942+0000 7fb1b159a700 1 -- 172.21.15.23:0/1448360407 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c0094a0 con 0x7fb1ac152e10 2023-06-22T08:13:04.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.943+0000 7fb1b159a700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1ac152e10 0x7fb1ac146d90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb19c016000 tx=0x7fb19c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:04.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.943+0000 7fb1a27fc700 1 -- 172.21.15.23:0/1448360407 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c01a7e0 con 0x7fb1ac152e10 2023-06-22T08:13:04.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.943+0000 7fb1a27fc700 1 -- 172.21.15.23:0/1448360407 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c01a960 con 0x7fb1ac152e10 2023-06-22T08:13:04.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.943+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1448360407 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac1e0da0 con 0x7fb1ac152e10 2023-06-22T08:13:04.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.943+0000 7fb1a27fc700 1 -- 172.21.15.23:0/1448360407 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c01ad10 con 0x7fb1ac152e10 2023-06-22T08:13:04.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.943+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1448360407 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ac14b680 con 0x7fb1ac152e10 2023-06-22T08:13:04.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.944+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1448360407 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ac14b1d0 con 0x7fb1ac152e10 2023-06-22T08:13:04.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.945+0000 7fb1a27fc700 1 -- 172.21.15.23:0/1448360407 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb19c022070 con 0x7fb1ac152e10 2023-06-22T08:13:04.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.946+0000 7fb1a27fc700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb19006ace0 0x7fb19006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:04.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.946+0000 7fb1a27fc700 1 -- 172.21.15.23:0/1448360407 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7fb19c08a710 con 0x7fb1ac152e10 2023-06-22T08:13:04.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.946+0000 7fb1b1d9b700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb19006ace0 0x7fb19006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:04.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.947+0000 7fb1b1d9b700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb19006ace0 0x7fb19006d190 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8005910 tx=0x7fb1a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:04.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:04.951+0000 7fb1a27fc700 1 -- 172.21.15.23:0/1448360407 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb19c0553d0 con 0x7fb1ac152e10 2023-06-22T08:13:05.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.206+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1448360407 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1ac14b870 con 0x7fb1ac152e10 2023-06-22T08:13:05.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.206+0000 7fb1a27fc700 1 -- 172.21.15.23:0/1448360407 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+6275 (secure 0 0 0) 0x7fb19c0589f0 con 0x7fb1ac152e10 2023-06-22T08:13:05.210 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:05.210 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5995,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:12:37.469779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:05.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb19006ace0 msgr2=0x7fb19006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:05.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb19006ace0 0x7fb19006d190 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8005910 tx=0x7fb1a8009040 comp rx=0 tx=0).stop 2023-06-22T08:13:05.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1448360407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1ac152e10 msgr2=0x7fb1ac146d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:05.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1ac152e10 0x7fb1ac146d90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb19c016000 tx=0x7fb19c004750 comp rx=0 tx=0).stop 2023-06-22T08:13:05.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1448360407 shutdown_connections 2023-06-22T08:13:05.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb19006ace0 0x7fb19006d190 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:05.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1ac1e2490 0x7fb1ac1568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:05.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1ac0cb6b0 0x7fb1ac157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:05.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 --2- 172.21.15.23:0/1448360407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1ac152e10 0x7fb1ac146d90 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:05.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1448360407 >> 172.21.15.23:0/1448360407 conn(0x7fb1ac070d80 msgr2=0x7fb1ac150830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:05.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1448360407 shutdown_connections 2023-06-22T08:13:05.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:05.209+0000 7fb1b3fff700 1 -- 172.21.15.23:0/1448360407 wait complete. 2023-06-22T08:13:05.214 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 100 2023-06-22T08:13:05.230 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:13:07.232 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:07.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.439+0000 7f72bd3ff700 1 Processor -- start 2023-06-22T08:13:07.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.439+0000 7f72bd3ff700 1 -- start start 2023-06-22T08:13:07.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.440+0000 7f72bd3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f72b8152ce0 0x7f72b80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:07.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.440+0000 7f72bd3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f72b80c9fd0 0x7f72b815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:07.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.440+0000 7f72bd3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b815c370 con 0x7f72b8158880 2023-06-22T08:13:07.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.440+0000 7f72bd3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b81530b0 con 0x7f72b8152ce0 2023-06-22T08:13:07.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.440+0000 7f72bd3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b80cf6f0 con 0x7f72b80c9fd0 2023-06-22T08:13:07.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.440+0000 7f72b6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f72b8152ce0 0x7f72b80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:07.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.440+0000 7f72b6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f72b8152ce0 0x7f72b80cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55820/0 (socket says 172.21.15.23:55820) 2023-06-22T08:13:07.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.440+0000 7f72b67fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f72b80c9fd0 0x7f72b815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:07.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.440+0000 7f72b6ffd700 1 -- 172.21.15.23:0/3769456275 learned_addr learned my addr 172.21.15.23:0/3769456275 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:07.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.441+0000 7f72b6ffd700 1 -- 172.21.15.23:0/3769456275 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f72b80c9fd0 msgr2=0x7f72b815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.441+0000 7f72b6ffd700 1 --2- 172.21.15.23:0/3769456275 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f72b80c9fd0 0x7f72b815c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.441+0000 7f72b6ffd700 1 -- 172.21.15.23:0/3769456275 >> v1:172.21.15.67:6789/0 conn(0x7f72b8158880 legacy=0x7f72b8158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.441+0000 7f72b6ffd700 1 -- 172.21.15.23:0/3769456275 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b815bda0 con 0x7f72b8152ce0 2023-06-22T08:13:07.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.441+0000 7f72b5ffb700 1 -- 172.21.15.23:0/3769456275 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560437534 0 0) 0x7f72b815c370 con 0x7f72b8158880 2023-06-22T08:13:07.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.441+0000 7f72b6ffd700 1 --2- 172.21.15.23:0/3769456275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8152ce0 0x7f72b80cf1b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f72a80098f0 tx=0x7f72a8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb6c1094f9c5d7d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:07.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.441+0000 7f72b5ffb700 1 -- 172.21.15.23:0/3769456275 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8011830 con 0x7f72b8152ce0 2023-06-22T08:13:07.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.441+0000 7f72b5ffb700 1 -- 172.21.15.23:0/3769456275 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a8011990 con 0x7f72b8152ce0 2023-06-22T08:13:07.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.441+0000 7f72b5ffb700 1 -- 172.21.15.23:0/3769456275 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a8011d20 con 0x7f72b8152ce0 2023-06-22T08:13:07.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.442+0000 7f72bd3ff700 1 -- 172.21.15.23:0/3769456275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8152ce0 msgr2=0x7f72b80cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:07.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.442+0000 7f72bd3ff700 1 --2- 172.21.15.23:0/3769456275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8152ce0 0x7f72b80cf1b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f72a80098f0 tx=0x7f72a8009c00 comp rx=0 tx=0).stop 2023-06-22T08:13:07.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.442+0000 7f72bd3ff700 1 -- 172.21.15.23:0/3769456275 shutdown_connections 2023-06-22T08:13:07.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.442+0000 7f72bd3ff700 1 --2- 172.21.15.23:0/3769456275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8152ce0 0x7f72b80cf1b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:07.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.442+0000 7f72bd3ff700 1 --2- 172.21.15.23:0/3769456275 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f72b80c9fd0 0x7f72b815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:07.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.442+0000 7f72bd3ff700 1 -- 172.21.15.23:0/3769456275 >> 172.21.15.23:0/3769456275 conn(0x7f72b806ae30 msgr2=0x7f72b806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.442+0000 7f72bd3ff700 1 -- 172.21.15.23:0/3769456275 shutdown_connections 2023-06-22T08:13:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.442+0000 7f72bd3ff700 1 -- 172.21.15.23:0/3769456275 wait complete. 2023-06-22T08:13:07.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.444+0000 7f72bd3ff700 1 Processor -- start 2023-06-22T08:13:07.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.445+0000 7f72bd3ff700 1 -- start start 2023-06-22T08:13:07.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72bd3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f72b80c9fd0 0x7f72b80cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:07.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72bd3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72b8152ce0 0x7f72b80b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:07.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72bd3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8158880 0x7f72b80b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:07.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72bd3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b80b2a50 con 0x7f72b8158880 2023-06-22T08:13:07.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72bd3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b8152490 con 0x7f72b8152ce0 2023-06-22T08:13:07.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72bd3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b80b12c0 con 0x7f72b80c9fd0 2023-06-22T08:13:07.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72b77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8158880 0x7f72b80b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:07.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72b77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8158880 0x7f72b80b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55830/0 (socket says 172.21.15.23:55830) 2023-06-22T08:13:07.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72b77fe700 1 -- 172.21.15.23:0/1846419824 learned_addr learned my addr 172.21.15.23:0/1846419824 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:07.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72b67fc700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72b8152ce0 0x7f72b80b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:07.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.446+0000 7f72b6ffd700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f72b80c9fd0 0x7f72b80cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:07.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f72b77fe700 1 -- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f72b80c9fd0 msgr2=0x7f72b80cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:07.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f72b77fe700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f72b80c9fd0 0x7f72b80cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:07.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f72b77fe700 1 -- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72b8152ce0 msgr2=0x7f72b80b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:07.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f72b77fe700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72b8152ce0 0x7f72b80b2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:07.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f72b77fe700 1 -- 172.21.15.23:0/1846419824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a8009550 con 0x7f72b8158880 2023-06-22T08:13:07.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f72b77fe700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8158880 0x7f72b80b34d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f72b0001560 tx=0x7f72b000cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:07.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f729f7fe700 1 -- 172.21.15.23:0/1846419824 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b0003e30 con 0x7f72b8158880 2023-06-22T08:13:07.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f729f7fe700 1 -- 172.21.15.23:0/1846419824 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b0005070 con 0x7f72b8158880 2023-06-22T08:13:07.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f729f7fe700 1 -- 172.21.15.23:0/1846419824 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b0005420 con 0x7f72b8158880 2023-06-22T08:13:07.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f72bd3ff700 1 -- 172.21.15.23:0/1846419824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b80b2260 con 0x7f72b8158880 2023-06-22T08:13:07.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.447+0000 7f72bd3ff700 1 -- 172.21.15.23:0/1846419824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b80b4160 con 0x7f72b8158880 2023-06-22T08:13:07.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.448+0000 7f72bd3ff700 1 -- 172.21.15.23:0/1846419824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b804ea20 con 0x7f72b8158880 2023-06-22T08:13:07.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.449+0000 7f729f7fe700 1 -- 172.21.15.23:0/1846419824 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f72b0005580 con 0x7f72b8158880 2023-06-22T08:13:07.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.450+0000 7f729f7fe700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f72a406ad10 0x7f72a406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:07.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.450+0000 7f729f7fe700 1 -- 172.21.15.23:0/1846419824 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 7329+0+0 (secure 0 0 0) 0x7f72b0088fc0 con 0x7f72b8158880 2023-06-22T08:13:07.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.451+0000 7f72b6ffd700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f72a406ad10 0x7f72a406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:07.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.451+0000 7f72b6ffd700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f72a406ad10 0x7f72a406d1c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f72a8009c40 tx=0x7f72a8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:07.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.455+0000 7f729f7fe700 1 -- 172.21.15.23:0/1846419824 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f72b0053b60 con 0x7f72b8158880 2023-06-22T08:13:07.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.706+0000 7f72bd3ff700 1 -- 172.21.15.23:0/1846419824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72b80b2be0 con 0x7f72b8158880 2023-06-22T08:13:07.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.707+0000 7f729f7fe700 1 -- 172.21.15.23:0/1846419824 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+6275 (secure 0 0 0) 0x7f72b0017020 con 0x7f72b8158880 2023-06-22T08:13:07.710 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:07.710 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5995,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:12:37.469779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5060,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5060":{"gid":5060,"name":"d","rank":3,"incarnation":46,"state":"up:active","state_seq":31,"addr":"172.21.15.67:6837/1551460358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1551460358},{"type":"v1","addr":"172.21.15.67:6837","nonce":1551460358}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:07.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 -- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f72a406ad10 msgr2=0x7f72a406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:07.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f72a406ad10 0x7f72a406d1c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f72a8009c40 tx=0x7f72a8008030 comp rx=0 tx=0).stop 2023-06-22T08:13:07.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 -- 172.21.15.23:0/1846419824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8158880 msgr2=0x7f72b80b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:07.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8158880 0x7f72b80b34d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f72b0001560 tx=0x7f72b000cb50 comp rx=0 tx=0).stop 2023-06-22T08:13:07.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 -- 172.21.15.23:0/1846419824 shutdown_connections 2023-06-22T08:13:07.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72b8158880 0x7f72b80b34d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:07.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f72b80c9fd0 0x7f72b80cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:07.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f72a406ad10 0x7f72a406d1c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:07.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 --2- 172.21.15.23:0/1846419824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72b8152ce0 0x7f72b80b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:07.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 -- 172.21.15.23:0/1846419824 >> 172.21.15.23:0/1846419824 conn(0x7f72b806ae30 msgr2=0x7f72b80c3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:07.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 -- 172.21.15.23:0/1846419824 shutdown_connections 2023-06-22T08:13:07.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:07.709+0000 7f72bd3ff700 1 -- 172.21.15.23:0/1846419824 wait complete. 2023-06-22T08:13:07.715 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 100 2023-06-22T08:13:07.730 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:13:09.211 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:13:09.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:13:09.731 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:09.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.938+0000 7fe8793ff700 1 Processor -- start 2023-06-22T08:13:09.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.939+0000 7fe8793ff700 1 -- start start 2023-06-22T08:13:09.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.940+0000 7fe8793ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe87414f850 0x7fe8740c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:09.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.940+0000 7fe8793ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe874071070 0x7fe8740c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:09.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.940+0000 7fe8793ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8740c51a0 con 0x7fe8740c69e0 2023-06-22T08:13:09.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.940+0000 7fe8793ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8740c44e0 con 0x7fe874071070 2023-06-22T08:13:09.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.940+0000 7fe8793ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740c5440 con 0x7fe87414f850 2023-06-22T08:13:09.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.940+0000 7fe8737fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe874071070 0x7fe8740c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:09.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.940+0000 7fe8737fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe874071070 0x7fe8740c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55840/0 (socket says 172.21.15.23:55840) 2023-06-22T08:13:09.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.940+0000 7fe8737fe700 1 -- 172.21.15.23:0/1311876626 learned_addr learned my addr 172.21.15.23:0/1311876626 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:09.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.940+0000 7fe872ffd700 1 --2- 172.21.15.23:0/1311876626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe87414f850 0x7fe8740c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:09.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.941+0000 7fe8737fe700 1 -- 172.21.15.23:0/1311876626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe87414f850 msgr2=0x7fe8740c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:09.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.941+0000 7fe8737fe700 1 --2- 172.21.15.23:0/1311876626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe87414f850 0x7fe8740c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:09.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.941+0000 7fe8737fe700 1 -- 172.21.15.23:0/1311876626 >> v1:172.21.15.67:6789/0 conn(0x7fe8740c69e0 legacy=0x7fe874155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:09.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.941+0000 7fe8737fe700 1 -- 172.21.15.23:0/1311876626 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8740c5720 con 0x7fe874071070 2023-06-22T08:13:09.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.941+0000 7fe8727fc700 1 -- 172.21.15.23:0/1311876626 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826861484 0 0) 0x7fe8740c51a0 con 0x7fe8740c69e0 2023-06-22T08:13:09.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.941+0000 7fe8737fe700 1 --2- 172.21.15.23:0/1311876626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe874071070 0x7fe8740c35c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009e50 tx=0x7fe86c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebdaef5335f044f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:09.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.941+0000 7fe8727fc700 1 -- 172.21.15.23:0/1311876626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c00e070 con 0x7fe874071070 2023-06-22T08:13:09.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.941+0000 7fe8727fc700 1 -- 172.21.15.23:0/1311876626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86c00b910 con 0x7fe874071070 2023-06-22T08:13:09.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.941+0000 7fe8727fc700 1 -- 172.21.15.23:0/1311876626 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c00bcc0 con 0x7fe874071070 2023-06-22T08:13:09.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.942+0000 7fe8793ff700 1 -- 172.21.15.23:0/1311876626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe874071070 msgr2=0x7fe8740c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:09.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.942+0000 7fe8793ff700 1 --2- 172.21.15.23:0/1311876626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe874071070 0x7fe8740c35c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009e50 tx=0x7fe86c00b330 comp rx=0 tx=0).stop 2023-06-22T08:13:09.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.942+0000 7fe8793ff700 1 -- 172.21.15.23:0/1311876626 shutdown_connections 2023-06-22T08:13:09.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.942+0000 7fe8793ff700 1 --2- 172.21.15.23:0/1311876626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe874071070 0x7fe8740c35c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:09.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.942+0000 7fe8793ff700 1 --2- 172.21.15.23:0/1311876626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe87414f850 0x7fe8740c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:09.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.942+0000 7fe8793ff700 1 -- 172.21.15.23:0/1311876626 >> 172.21.15.23:0/1311876626 conn(0x7fe8740ca3f0 msgr2=0x7fe8740ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:09.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.942+0000 7fe8793ff700 1 -- 172.21.15.23:0/1311876626 shutdown_connections 2023-06-22T08:13:09.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.942+0000 7fe8793ff700 1 -- 172.21.15.23:0/1311876626 wait complete. 2023-06-22T08:13:09.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.944+0000 7fe8793ff700 1 Processor -- start 2023-06-22T08:13:09.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.944+0000 7fe8793ff700 1 -- start start 2023-06-22T08:13:09.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe8793ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe874071070 0x7fe8741eef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:09.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe8793ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8740c69e0 0x7fe874142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:09.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe8793ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe87414f850 0x7fe8741e12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:09.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe8793ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8741ef4d0 con 0x7fe87414f850 2023-06-22T08:13:09.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe8793ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe874142d50 con 0x7fe874071070 2023-06-22T08:13:09.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe8793ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8741dc130 con 0x7fe8740c69e0 2023-06-22T08:13:09.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe8737fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe87414f850 0x7fe8741e12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:09.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe86bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8740c69e0 0x7fe874142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:09.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe872ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe874071070 0x7fe8741eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:09.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe8737fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe87414f850 0x7fe8741e12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55854/0 (socket says 172.21.15.23:55854) 2023-06-22T08:13:09.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.945+0000 7fe8737fe700 1 -- 172.21.15.23:0/194108032 learned_addr learned my addr 172.21.15.23:0/194108032 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:09.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe8737fe700 1 -- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8740c69e0 msgr2=0x7fe874142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:09.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe8737fe700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8740c69e0 0x7fe874142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:09.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe8737fe700 1 -- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe874071070 msgr2=0x7fe8741eef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:09.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe8737fe700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe874071070 0x7fe8741eef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:09.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe8737fe700 1 -- 172.21.15.23:0/194108032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86c009b30 con 0x7fe87414f850 2023-06-22T08:13:09.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe8737fe700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe87414f850 0x7fe8741e12c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe86c004270 tx=0x7fe86c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:09.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe86b7fe700 1 -- 172.21.15.23:0/194108032 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0048a0 con 0x7fe87414f850 2023-06-22T08:13:09.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe86b7fe700 1 -- 172.21.15.23:0/194108032 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86c0092e0 con 0x7fe87414f850 2023-06-22T08:13:09.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe86b7fe700 1 -- 172.21.15.23:0/194108032 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c01c660 con 0x7fe87414f850 2023-06-22T08:13:09.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe8793ff700 1 -- 172.21.15.23:0/194108032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87406ac40 con 0x7fe87414f850 2023-06-22T08:13:09.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.946+0000 7fe8793ff700 1 -- 172.21.15.23:0/194108032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8741dbae0 con 0x7fe87414f850 2023-06-22T08:13:09.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.947+0000 7fe8793ff700 1 -- 172.21.15.23:0/194108032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8740c2950 con 0x7fe87414f850 2023-06-22T08:13:09.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.948+0000 7fe86b7fe700 1 -- 172.21.15.23:0/194108032 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe86c01c7c0 con 0x7fe87414f850 2023-06-22T08:13:09.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.949+0000 7fe86b7fe700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe85406ace0 0x7fe85406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:09.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.950+0000 7fe86b7fe700 1 -- 172.21.15.23:0/194108032 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7fe86c08b2a0 con 0x7fe87414f850 2023-06-22T08:13:09.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.950+0000 7fe872ffd700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe85406ace0 0x7fe85406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:09.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.951+0000 7fe872ffd700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe85406ace0 0x7fe85406d190 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe864005d10 tx=0x7fe86400b750 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:09.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:09.954+0000 7fe86b7fe700 1 -- 172.21.15.23:0/194108032 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe86c055df0 con 0x7fe87414f850 2023-06-22T08:13:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.203+0000 7fe8793ff700 1 -- 172.21.15.23:0/194108032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe87404ea20 con 0x7fe87414f850 2023-06-22T08:13:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.204+0000 7fe86b7fe700 1 -- 172.21.15.23:0/194108032 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5514 (secure 0 0 0) 0x7fe86c059410 con 0x7fe87414f850 2023-06-22T08:13:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:10.208 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:09.469620+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 -- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe85406ace0 msgr2=0x7fe85406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe85406ace0 0x7fe85406d190 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe864005d10 tx=0x7fe86400b750 comp rx=0 tx=0).stop 2023-06-22T08:13:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 -- 172.21.15.23:0/194108032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe87414f850 msgr2=0x7fe8741e12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe87414f850 0x7fe8741e12c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe86c004270 tx=0x7fe86c004ad0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 -- 172.21.15.23:0/194108032 shutdown_connections 2023-06-22T08:13:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe87414f850 0x7fe8741e12c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe85406ace0 0x7fe85406d190 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe874071070 0x7fe8741eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 --2- 172.21.15.23:0/194108032 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe8740c69e0 0x7fe874142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 -- 172.21.15.23:0/194108032 >> 172.21.15.23:0/194108032 conn(0x7fe8740ca3f0 msgr2=0x7fe87414e950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:10.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 -- 172.21.15.23:0/194108032 shutdown_connections 2023-06-22T08:13:10.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.206+0000 7fe8793ff700 1 -- 172.21.15.23:0/194108032 wait complete. 2023-06-22T08:13:10.212 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 101 2023-06-22T08:13:10.227 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-06-22T08:13:10.228 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:13:10.228 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:10.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.424+0000 7f8d229ff700 1 Processor -- start 2023-06-22T08:13:10.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.424+0000 7f8d229ff700 1 -- start start 2023-06-22T08:13:10.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.425+0000 7f8d229ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8d1c152ce0 0x7f8d1c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:10.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.425+0000 7f8d229ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c0c9fd0 0x7f8d1c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:10.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.425+0000 7f8d229ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c15c370 con 0x7f8d1c158880 2023-06-22T08:13:10.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.425+0000 7f8d229ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d1c1530b0 con 0x7f8d1c152ce0 2023-06-22T08:13:10.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.425+0000 7f8d229ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c0cf6f0 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d20f9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f8d1c158880 0x7f8d1c158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60558/0 (socket says 172.21.15.23:60558) 2023-06-22T08:13:10.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d20f9c700 1 -- 172.21.15.23:0/1332801688 learned_addr learned my addr 172.21.15.23:0/1332801688 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:10.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d1bfff700 1 --2- 172.21.15.23:0/1332801688 >> v2:172.21.15.67:3300/0 conn(0x7f8d1c152ce0 0x7f8d1c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:10.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d1b7fe700 1 --2- 172.21.15.23:0/1332801688 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c0c9fd0 0x7f8d1c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:10.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d1affd700 1 -- 172.21.15.23:0/1332801688 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299930644 0 0) 0x7f8d1c15c370 con 0x7f8d1c158880 2023-06-22T08:13:10.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d1bfff700 1 -- 172.21.15.23:0/1332801688 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c0c9fd0 msgr2=0x7f8d1c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:10.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d1bfff700 1 --2- 172.21.15.23:0/1332801688 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c0c9fd0 0x7f8d1c15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d1bfff700 1 -- 172.21.15.23:0/1332801688 >> v1:172.21.15.23:6789/0 conn(0x7f8d1c158880 legacy=0x7f8d1c158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:10.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d1bfff700 1 -- 172.21.15.23:0/1332801688 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c15bda0 con 0x7f8d1c152ce0 2023-06-22T08:13:10.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d1b7fe700 1 --2- 172.21.15.23:0/1332801688 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c0c9fd0 0x7f8d1c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:13:10.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.426+0000 7f8d1bfff700 1 --2- 172.21.15.23:0/1332801688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c152ce0 0x7f8d1c0cf1b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009840 tx=0x7f8d0c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77d5443cc592f58 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:10.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d1affd700 1 -- 172.21.15.23:0/1332801688 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c011830 con 0x7f8d1c152ce0 2023-06-22T08:13:10.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d1affd700 1 -- 172.21.15.23:0/1332801688 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0c011990 con 0x7f8d1c152ce0 2023-06-22T08:13:10.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d1affd700 1 -- 172.21.15.23:0/1332801688 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c011d20 con 0x7f8d1c152ce0 2023-06-22T08:13:10.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d229ff700 1 -- 172.21.15.23:0/1332801688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c152ce0 msgr2=0x7f8d1c0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:10.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d229ff700 1 --2- 172.21.15.23:0/1332801688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c152ce0 0x7f8d1c0cf1b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009840 tx=0x7f8d0c009b50 comp rx=0 tx=0).stop 2023-06-22T08:13:10.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d229ff700 1 -- 172.21.15.23:0/1332801688 shutdown_connections 2023-06-22T08:13:10.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d229ff700 1 --2- 172.21.15.23:0/1332801688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c152ce0 0x7f8d1c0cf1b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d229ff700 1 --2- 172.21.15.23:0/1332801688 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c0c9fd0 0x7f8d1c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d229ff700 1 -- 172.21.15.23:0/1332801688 >> 172.21.15.23:0/1332801688 conn(0x7f8d1c06ae30 msgr2=0x7f8d1c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:10.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d229ff700 1 -- 172.21.15.23:0/1332801688 shutdown_connections 2023-06-22T08:13:10.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.427+0000 7f8d229ff700 1 -- 172.21.15.23:0/1332801688 wait complete. 2023-06-22T08:13:10.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.429+0000 7f8d229ff700 1 Processor -- start 2023-06-22T08:13:10.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.429+0000 7f8d229ff700 1 -- start start 2023-06-22T08:13:10.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d229ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d1c0c9fd0 0x7f8d1c1e4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:10.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d229ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c152ce0 0x7f8d1c0cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:10.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d229ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c158880 0x7f8d1c1e2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:10.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d229ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c1525e0 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d229ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c0cdfa0 con 0x7f8d1c152ce0 2023-06-22T08:13:10.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d229ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c0cf310 con 0x7f8d1c158880 2023-06-22T08:13:10.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d1bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d1c0c9fd0 0x7f8d1c1e4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:10.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d1bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d1c0c9fd0 0x7f8d1c1e4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55856/0 (socket says 172.21.15.23:55856) 2023-06-22T08:13:10.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d1bfff700 1 -- 172.21.15.23:0/1879680775 learned_addr learned my addr 172.21.15.23:0/1879680775 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:10.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d20f9c700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c158880 0x7f8d1c1e2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:10.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.430+0000 7f8d1b7fe700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c152ce0 0x7f8d1c0cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:10.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d1bfff700 1 -- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c158880 msgr2=0x7f8d1c1e2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:10.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d1bfff700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c158880 0x7f8d1c1e2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d1bfff700 1 -- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c152ce0 msgr2=0x7f8d1c0cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:10.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d1bfff700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c152ce0 0x7f8d1c0cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d1bfff700 1 -- 172.21.15.23:0/1879680775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d100094a0 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d1b7fe700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c152ce0 0x7f8d1c0cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:13:10.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d1bfff700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d1c0c9fd0 0x7f8d1c1e4470 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0090b0 tx=0x7f8d0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:10.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d18ff9700 1 -- 172.21.15.23:0/1879680775 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c0036a0 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d18ff9700 1 -- 172.21.15.23:0/1879680775 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0c01f070 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d18ff9700 1 -- 172.21.15.23:0/1879680775 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c003a40 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d229ff700 1 -- 172.21.15.23:0/1879680775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0c0094a0 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.431+0000 7f8d229ff700 1 -- 172.21.15.23:0/1879680775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1c0ce700 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.432+0000 7f8d229ff700 1 -- 172.21.15.23:0/1879680775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d1c04ea20 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.433+0000 7f8d18ff9700 1 -- 172.21.15.23:0/1879680775 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8d0c003ba0 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.434+0000 7f8d18ff9700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8d0806ace0 0x7f8d0806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:10.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.434+0000 7f8d18ff9700 1 -- 172.21.15.23:0/1879680775 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f8d0c021070 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.435+0000 7f8d1b7fe700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8d0806ace0 0x7f8d0806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:10.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.435+0000 7f8d1b7fe700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8d0806ace0 0x7f8d0806d190 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c14a440 tx=0x7f8d10008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:10.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.439+0000 7f8d18ff9700 1 -- 172.21.15.23:0/1879680775 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8d0c055e10 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.695+0000 7f8d229ff700 1 -- 172.21.15.23:0/1879680775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d1c0c9db0 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.696+0000 7f8d18ff9700 1 -- 172.21.15.23:0/1879680775 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5514 (secure 0 0 0) 0x7f8d0c059430 con 0x7f8d1c0c9fd0 2023-06-22T08:13:10.700 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:10.700 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:09.469620+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:10.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.698+0000 7f8d229ff700 1 -- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8d0806ace0 msgr2=0x7f8d0806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:10.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.698+0000 7f8d229ff700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8d0806ace0 0x7f8d0806d190 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c14a440 tx=0x7f8d10008040 comp rx=0 tx=0).stop 2023-06-22T08:13:10.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.698+0000 7f8d229ff700 1 -- 172.21.15.23:0/1879680775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d1c0c9fd0 msgr2=0x7f8d1c1e4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:10.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.698+0000 7f8d229ff700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d1c0c9fd0 0x7f8d1c1e4470 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c0090b0 tx=0x7f8d0c004750 comp rx=0 tx=0).stop 2023-06-22T08:13:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.698+0000 7f8d229ff700 1 -- 172.21.15.23:0/1879680775 shutdown_connections 2023-06-22T08:13:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.698+0000 7f8d229ff700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8d1c158880 0x7f8d1c1e2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.698+0000 7f8d229ff700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d1c0c9fd0 0x7f8d1c1e4470 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.698+0000 7f8d229ff700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8d0806ace0 0x7f8d0806d190 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.699+0000 7f8d229ff700 1 --2- 172.21.15.23:0/1879680775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c152ce0 0x7f8d1c0cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.699+0000 7f8d229ff700 1 -- 172.21.15.23:0/1879680775 >> 172.21.15.23:0/1879680775 conn(0x7f8d1c06ae30 msgr2=0x7f8d1c0c4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.699+0000 7f8d229ff700 1 -- 172.21.15.23:0/1879680775 shutdown_connections 2023-06-22T08:13:10.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:10.699+0000 7f8d229ff700 1 -- 172.21.15.23:0/1879680775 wait complete. 2023-06-22T08:13:10.704 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 101 2023-06-22T08:13:10.720 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 26 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:13:09.469620+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 5551, "mds_3": 5995, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:13:12.722 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:12.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.932+0000 7f70ad9ff700 1 Processor -- start 2023-06-22T08:13:12.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.932+0000 7f70ad9ff700 1 -- start start 2023-06-22T08:13:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.933+0000 7f70ad9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70a8152ce0 0x7f70a80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.933+0000 7f70ad9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f70a80c40b0 0x7f70a80c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:12.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.933+0000 7f70ad9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a815c370 con 0x7f70a80c9fd0 2023-06-22T08:13:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.933+0000 7f70ad9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a81530b0 con 0x7f70a80c40b0 2023-06-22T08:13:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.933+0000 7f70ad9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a80cf6f0 con 0x7f70a8152ce0 2023-06-22T08:13:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.933+0000 7f70a77fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f70a80c40b0 0x7f70a80c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:12.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.934+0000 7f70a77fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f70a80c40b0 0x7f70a80c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55862/0 (socket says 172.21.15.23:55862) 2023-06-22T08:13:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.934+0000 7f70a77fe700 1 -- 172.21.15.23:0/2072761036 learned_addr learned my addr 172.21.15.23:0/2072761036 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.934+0000 7f70a6ffd700 1 --2- 172.21.15.23:0/2072761036 >> v2:172.21.15.67:3300/0 conn(0x7f70a8152ce0 0x7f70a80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.934+0000 7f70a77fe700 1 -- 172.21.15.23:0/2072761036 >> v2:172.21.15.67:3300/0 conn(0x7f70a8152ce0 msgr2=0x7f70a80cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:12.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.934+0000 7f70a77fe700 1 --2- 172.21.15.23:0/2072761036 >> v2:172.21.15.67:3300/0 conn(0x7f70a8152ce0 0x7f70a80cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.934+0000 7f70a77fe700 1 -- 172.21.15.23:0/2072761036 >> v1:172.21.15.67:6789/0 conn(0x7f70a80c9fd0 legacy=0x7f70a815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.934+0000 7f70a77fe700 1 -- 172.21.15.23:0/2072761036 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a815bda0 con 0x7f70a80c40b0 2023-06-22T08:13:12.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.934+0000 7f70a5ffb700 1 -- 172.21.15.23:0/2072761036 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250232537 0 0) 0x7f70a815c370 con 0x7f70a80c9fd0 2023-06-22T08:13:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.934+0000 7f70a77fe700 1 --2- 172.21.15.23:0/2072761036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 0x7f70a80c4530 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f70a808ba10 tx=0x7f70a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7498ba23db67fa40 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.934+0000 7f70a5ffb700 1 -- 172.21.15.23:0/2072761036 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a000e070 con 0x7f70a80c40b0 2023-06-22T08:13:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.935+0000 7f70a5ffb700 1 -- 172.21.15.23:0/2072761036 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a000b910 con 0x7f70a80c40b0 2023-06-22T08:13:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.935+0000 7f70a5ffb700 1 -- 172.21.15.23:0/2072761036 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a000bcc0 con 0x7f70a80c40b0 2023-06-22T08:13:12.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.935+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2072761036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 msgr2=0x7f70a80c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:12.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.935+0000 7f70ad9ff700 1 --2- 172.21.15.23:0/2072761036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 0x7f70a80c4530 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f70a808ba10 tx=0x7f70a000b330 comp rx=0 tx=0).stop 2023-06-22T08:13:12.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.935+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2072761036 shutdown_connections 2023-06-22T08:13:12.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.935+0000 7f70ad9ff700 1 --2- 172.21.15.23:0/2072761036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 0x7f70a80c4530 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:12.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.935+0000 7f70ad9ff700 1 --2- 172.21.15.23:0/2072761036 >> v2:172.21.15.67:3300/0 conn(0x7f70a8152ce0 0x7f70a80cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:12.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.935+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2072761036 >> 172.21.15.23:0/2072761036 conn(0x7f70a806ae30 msgr2=0x7f70a806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:12.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.935+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2072761036 shutdown_connections 2023-06-22T08:13:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.935+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2072761036 wait complete. 2023-06-22T08:13:12.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.937+0000 7f70ad9ff700 1 Processor -- start 2023-06-22T08:13:12.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.937+0000 7f70ad9ff700 1 -- start start 2023-06-22T08:13:12.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70ad9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 0x7f70a81edee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:12.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70ad9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a80c9fd0 0x7f70a80b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:12.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70ad9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a8152ce0 0x7f70a814a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:12.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70ad9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a80cc9e0 con 0x7f70a80c40b0 2023-06-22T08:13:12.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70ad9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a81561c0 con 0x7f70a80c9fd0 2023-06-22T08:13:12.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70ad9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a80cc690 con 0x7f70a8152ce0 2023-06-22T08:13:12.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70a6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 0x7f70a81edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:12.949 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70a6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 0x7f70a81edee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55864/0 (socket says 172.21.15.23:55864) 2023-06-22T08:13:12.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70a77fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a8152ce0 0x7f70a814a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:12.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70a6ffd700 1 -- 172.21.15.23:0/2519588674 learned_addr learned my addr 172.21.15.23:0/2519588674 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:12.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.938+0000 7f70a67fc700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a80c9fd0 0x7f70a80b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:12.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f70a6ffd700 1 -- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a8152ce0 msgr2=0x7f70a814a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:12.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f70a6ffd700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a8152ce0 0x7f70a814a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:12.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f70a6ffd700 1 -- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a80c9fd0 msgr2=0x7f70a80b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:12.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f70a6ffd700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a80c9fd0 0x7f70a80b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:12.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f70a6ffd700 1 -- 172.21.15.23:0/2519588674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a0009b30 con 0x7f70a80c40b0 2023-06-22T08:13:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f70a6ffd700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 0x7f70a81edee0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f709800e880 tx=0x7f709800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f708ffff700 1 -- 172.21.15.23:0/2519588674 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098013680 con 0x7f70a80c40b0 2023-06-22T08:13:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f708ffff700 1 -- 172.21.15.23:0/2519588674 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7098013800 con 0x7f70a80c40b0 2023-06-22T08:13:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2519588674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a8156b60 con 0x7f70a80c40b0 2023-06-22T08:13:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f708ffff700 1 -- 172.21.15.23:0/2519588674 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098013bb0 con 0x7f70a80c40b0 2023-06-22T08:13:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.939+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2519588674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a8157120 con 0x7f70a80c40b0 2023-06-22T08:13:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.940+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2519588674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a80ccb20 con 0x7f70a80c40b0 2023-06-22T08:13:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.941+0000 7f708ffff700 1 -- 172.21.15.23:0/2519588674 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7098013d10 con 0x7f70a80c40b0 2023-06-22T08:13:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.943+0000 7f708ffff700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f709406ace0 0x7f709406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.943+0000 7f708ffff700 1 -- 172.21.15.23:0/2519588674 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f7098089910 con 0x7f70a80c40b0 2023-06-22T08:13:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.943+0000 7f70a67fc700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f709406ace0 0x7f709406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.944+0000 7f70a67fc700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f709406ace0 0x7f709406d190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f70a8150c40 tx=0x7f709c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:12.948+0000 7f708ffff700 1 -- 172.21.15.23:0/2519588674 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f70a80ccb20 con 0x7f70a80c40b0 2023-06-22T08:13:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.205+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2519588674 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70a80ccb20 con 0x7f70a80c40b0 2023-06-22T08:13:13.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.206+0000 7f708ffff700 1 -- 172.21.15.23:0/2519588674 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5514 (secure 0 0 0) 0x7f70a80ccb20 con 0x7f70a80c40b0 2023-06-22T08:13:13.210 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:13.210 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:09.469620+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f709406ace0 msgr2=0x7f709406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f709406ace0 0x7f709406d190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f70a8150c40 tx=0x7f709c009040 comp rx=0 tx=0).stop 2023-06-22T08:13:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2519588674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 msgr2=0x7f70a81edee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 0x7f70a81edee0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f709800e880 tx=0x7f709800eb90 comp rx=0 tx=0).stop 2023-06-22T08:13:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2519588674 shutdown_connections 2023-06-22T08:13:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a8152ce0 0x7f70a814a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70a80c40b0 0x7f70a81edee0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f709406ace0 0x7f709406d190 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 --2- 172.21.15.23:0/2519588674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a80c9fd0 0x7f70a80b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2519588674 >> 172.21.15.23:0/2519588674 conn(0x7f70a806ae30 msgr2=0x7f70a806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2519588674 shutdown_connections 2023-06-22T08:13:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:13.208+0000 7f70ad9ff700 1 -- 172.21.15.23:0/2519588674 wait complete. 2023-06-22T08:13:13.216 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 101 2023-06-22T08:13:13.230 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:13:15.231 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:15.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.435+0000 7f0f191ff700 1 Processor -- start 2023-06-22T08:13:15.439 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.435+0000 7f0f191ff700 1 -- start start 2023-06-22T08:13:15.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.436+0000 7f0f191ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f1414f850 0x7f0f140c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:15.440 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.436+0000 7f0f191ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f140c69e0 0x7f0f140c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:15.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f191ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f140c51a0 con 0x7f0f14071070 2023-06-22T08:13:15.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f191ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f140c44e0 con 0x7f0f1414f850 2023-06-22T08:13:15.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f191ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f140c5440 con 0x7f0f140c69e0 2023-06-22T08:13:15.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f12d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f1414f850 0x7f0f140c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:15.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f0bfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f140c69e0 0x7f0f140c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:15.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f12d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f1414f850 0x7f0f140c3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55876/0 (socket says 172.21.15.23:55876) 2023-06-22T08:13:15.441 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f12d9d700 1 -- 172.21.15.23:0/1029787977 learned_addr learned my addr 172.21.15.23:0/1029787977 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:15.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f1359e700 1 --1- 172.21.15.23:0/1029787977 >> v1:172.21.15.67:6789/0 conn(0x7f0f14071070 0x7f0f14155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:46830/0 (socket says 172.21.15.23:46830) 2023-06-22T08:13:15.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f12d9d700 1 -- 172.21.15.23:0/1029787977 >> v2:172.21.15.67:3300/0 conn(0x7f0f140c69e0 msgr2=0x7f0f140c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:15.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f12d9d700 1 --2- 172.21.15.23:0/1029787977 >> v2:172.21.15.67:3300/0 conn(0x7f0f140c69e0 0x7f0f140c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f12d9d700 1 -- 172.21.15.23:0/1029787977 >> v1:172.21.15.67:6789/0 conn(0x7f0f14071070 legacy=0x7f0f14155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:15.442 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f12d9d700 1 -- 172.21.15.23:0/1029787977 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f140c5720 con 0x7f0f1414f850 2023-06-22T08:13:15.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.437+0000 7f0f12d9d700 1 --2- 172.21.15.23:0/1029787977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1414f850 0x7f0f140c3080 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0f1406f9a0 tx=0x7f0f04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27a37e181ef3f89 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:15.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f1259c700 1 -- 172.21.15.23:0/1029787977 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04011830 con 0x7f0f1414f850 2023-06-22T08:13:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f1259c700 1 -- 172.21.15.23:0/1029787977 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f04011990 con 0x7f0f1414f850 2023-06-22T08:13:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f1259c700 1 -- 172.21.15.23:0/1029787977 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04011d20 con 0x7f0f1414f850 2023-06-22T08:13:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f191ff700 1 -- 172.21.15.23:0/1029787977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1414f850 msgr2=0x7f0f140c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f191ff700 1 --2- 172.21.15.23:0/1029787977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1414f850 0x7f0f140c3080 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0f1406f9a0 tx=0x7f0f04009bd0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f191ff700 1 -- 172.21.15.23:0/1029787977 shutdown_connections 2023-06-22T08:13:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f191ff700 1 --2- 172.21.15.23:0/1029787977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1414f850 0x7f0f140c3080 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f191ff700 1 --2- 172.21.15.23:0/1029787977 >> v2:172.21.15.67:3300/0 conn(0x7f0f140c69e0 0x7f0f140c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f191ff700 1 -- 172.21.15.23:0/1029787977 >> 172.21.15.23:0/1029787977 conn(0x7f0f140ca3f0 msgr2=0x7f0f140ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:15.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f191ff700 1 -- 172.21.15.23:0/1029787977 shutdown_connections 2023-06-22T08:13:15.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.438+0000 7f0f191ff700 1 -- 172.21.15.23:0/1029787977 wait complete. 2023-06-22T08:13:15.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.440+0000 7f0f191ff700 1 Processor -- start 2023-06-22T08:13:15.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.440+0000 7f0f191ff700 1 -- start start 2023-06-22T08:13:15.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.441+0000 7f0f191ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f14071070 0x7f0f14161680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.441+0000 7f0f191ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f140c69e0 0x7f0f140c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.441+0000 7f0f191ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1414f850 0x7f0f141f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.441+0000 7f0f191ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f14161f10 con 0x7f0f1414f850 2023-06-22T08:13:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.441+0000 7f0f191ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f14154220 con 0x7f0f14071070 2023-06-22T08:13:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.441+0000 7f0f191ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f14153610 con 0x7f0f140c69e0 2023-06-22T08:13:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.441+0000 7f0f0bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f140c69e0 0x7f0f140c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:15.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.441+0000 7f0f12d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f14071070 0x7f0f14161680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.441+0000 7f0f0bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f140c69e0 0x7f0f140c8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:58008/0 (socket says 172.21.15.23:58008) 2023-06-22T08:13:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.441+0000 7f0f0bfff700 1 -- 172.21.15.23:0/1904475486 learned_addr learned my addr 172.21.15.23:0/1904475486 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f1359e700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1414f850 0x7f0f141f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f0bfff700 1 -- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f14071070 msgr2=0x7f0f14161680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f0bfff700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f14071070 0x7f0f14161680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f0bfff700 1 -- 172.21.15.23:0/1904475486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1414f850 msgr2=0x7f0f141f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f0bfff700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1414f850 0x7f0f141f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f0bfff700 1 -- 172.21.15.23:0/1904475486 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f04009550 con 0x7f0f140c69e0 2023-06-22T08:13:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f0bfff700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f140c69e0 0x7f0f140c8440 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00e8a0 tx=0x7f0efc00ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f0b7fe700 1 -- 172.21.15.23:0/1904475486 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc009590 con 0x7f0f140c69e0 2023-06-22T08:13:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f0b7fe700 1 -- 172.21.15.23:0/1904475486 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0efc013680 con 0x7f0f140c69e0 2023-06-22T08:13:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f0b7fe700 1 -- 172.21.15.23:0/1904475486 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc0139f0 con 0x7f0f140c69e0 2023-06-22T08:13:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.442+0000 7f0f191ff700 1 -- 172.21.15.23:0/1904475486 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f141538f0 con 0x7f0f140c69e0 2023-06-22T08:13:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.443+0000 7f0f191ff700 1 -- 172.21.15.23:0/1904475486 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f14153eb0 con 0x7f0f140c69e0 2023-06-22T08:13:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.443+0000 7f0f191ff700 1 -- 172.21.15.23:0/1904475486 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f140c2950 con 0x7f0f140c69e0 2023-06-22T08:13:15.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.445+0000 7f0f0b7fe700 1 -- 172.21.15.23:0/1904475486 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0efc013b50 con 0x7f0f140c69e0 2023-06-22T08:13:15.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.445+0000 7f0f0b7fe700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ef406ad10 0x7f0ef406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:15.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.446+0000 7f0f0b7fe700 1 -- 172.21.15.23:0/1904475486 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f0efc089c20 con 0x7f0f140c69e0 2023-06-22T08:13:15.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.446+0000 7f0f12d9d700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ef406ad10 0x7f0ef406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:15.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.446+0000 7f0f12d9d700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ef406ad10 0x7f0ef406d1c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009c10 tx=0x7f0f04008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:15.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.450+0000 7f0f0b7fe700 1 -- 172.21.15.23:0/1904475486 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0efc0546f0 con 0x7f0f140c69e0 2023-06-22T08:13:15.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.707+0000 7f0f191ff700 1 -- 172.21.15.23:0/1904475486 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f1404ea20 con 0x7f0f140c69e0 2023-06-22T08:13:15.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.709+0000 7f0f0b7fe700 1 -- 172.21.15.23:0/1904475486 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5514 (secure 0 0 0) 0x7f0efc017090 con 0x7f0f140c69e0 2023-06-22T08:13:15.711 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:15.712 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:09.469620+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:15.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.710+0000 7f0f191ff700 1 -- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ef406ad10 msgr2=0x7f0ef406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:15.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.710+0000 7f0f191ff700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ef406ad10 0x7f0ef406d1c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009c10 tx=0x7f0f04008030 comp rx=0 tx=0).stop 2023-06-22T08:13:15.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.710+0000 7f0f191ff700 1 -- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f140c69e0 msgr2=0x7f0f140c8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:15.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.710+0000 7f0f191ff700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f140c69e0 0x7f0f140c8440 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00e8a0 tx=0x7f0efc00ebb0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.710+0000 7f0f191ff700 1 -- 172.21.15.23:0/1904475486 shutdown_connections 2023-06-22T08:13:15.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.710+0000 7f0f191ff700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f1414f850 0x7f0f141f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.710+0000 7f0f191ff700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ef406ad10 0x7f0ef406d1c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.710+0000 7f0f191ff700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f14071070 0x7f0f14161680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.710+0000 7f0f191ff700 1 --2- 172.21.15.23:0/1904475486 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f140c69e0 0x7f0f140c8440 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:15.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.710+0000 7f0f191ff700 1 -- 172.21.15.23:0/1904475486 >> 172.21.15.23:0/1904475486 conn(0x7f0f140ca3f0 msgr2=0x7f0f1406a880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:15.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.711+0000 7f0f191ff700 1 -- 172.21.15.23:0/1904475486 shutdown_connections 2023-06-22T08:13:15.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:15.711+0000 7f0f191ff700 1 -- 172.21.15.23:0/1904475486 wait complete. 2023-06-22T08:13:15.719 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 101 2023-06-22T08:13:15.732 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:13:17.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.938+0000 7ffb3cbff700 1 Processor -- start 2023-06-22T08:13:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.938+0000 7ffb3cbff700 1 -- start start 2023-06-22T08:13:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.939+0000 7ffb3cbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb380cf1b0 0x7ffb380c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.939+0000 7ffb3cbff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3815c370 con 0x7ffb38152ce0 2023-06-22T08:13:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.939+0000 7ffb3cbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3815bda0 con 0x7ffb380c9fd0 2023-06-22T08:13:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.939+0000 7ffb3cbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb381530b0 con 0x7ffb380cf1b0 2023-06-22T08:13:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.939+0000 7ffb3659c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ffb38152ce0 0x7ffb3815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60912/0 (socket says 172.21.15.23:60912) 2023-06-22T08:13:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.939+0000 7ffb2ffff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffb380c9fd0 0x7ffb380c40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:53504/0 (socket says 172.21.15.23:53504) 2023-06-22T08:13:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.939+0000 7ffb3659c700 1 -- 172.21.15.23:0/1325299373 learned_addr learned my addr 172.21.15.23:0/1325299373 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.939+0000 7ffb36d9d700 1 --2- 172.21.15.23:0/1325299373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb380cf1b0 0x7ffb380c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.940+0000 7ffb35d9b700 1 -- 172.21.15.23:0/1325299373 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220235511 0 0) 0x7ffb3815bda0 con 0x7ffb380c9fd0 2023-06-22T08:13:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.940+0000 7ffb35d9b700 1 -- 172.21.15.23:0/1325299373 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb240035b0 con 0x7ffb380c9fd0 2023-06-22T08:13:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.940+0000 7ffb36d9d700 1 -- 172.21.15.23:0/1325299373 >> v1:172.21.15.67:6789/0 conn(0x7ffb380c9fd0 legacy=0x7ffb380c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.940+0000 7ffb36d9d700 1 -- 172.21.15.23:0/1325299373 >> v1:172.21.15.23:6789/0 conn(0x7ffb38152ce0 legacy=0x7ffb3815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.940+0000 7ffb35d9b700 1 -- 172.21.15.23:0/1325299373 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587226070 0 0) 0x7ffb3815c370 con 0x7ffb38152ce0 2023-06-22T08:13:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.940+0000 7ffb36d9d700 1 -- 172.21.15.23:0/1325299373 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb380cf630 con 0x7ffb380cf1b0 2023-06-22T08:13:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.940+0000 7ffb36d9d700 1 --2- 172.21.15.23:0/1325299373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb380cf1b0 0x7ffb380c47c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009dc0 tx=0x7ffb3000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33aa21317e2aa51a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.940+0000 7ffb35d9b700 1 -- 172.21.15.23:0/1325299373 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3000e070 con 0x7ffb380cf1b0 2023-06-22T08:13:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.940+0000 7ffb35d9b700 1 -- 172.21.15.23:0/1325299373 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3000bb20 con 0x7ffb380cf1b0 2023-06-22T08:13:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.940+0000 7ffb35d9b700 1 -- 172.21.15.23:0/1325299373 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3000be90 con 0x7ffb380cf1b0 2023-06-22T08:13:17.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.941+0000 7ffb3cbff700 1 -- 172.21.15.23:0/1325299373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb380cf1b0 msgr2=0x7ffb380c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:17.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.941+0000 7ffb3cbff700 1 --2- 172.21.15.23:0/1325299373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb380cf1b0 0x7ffb380c47c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009dc0 tx=0x7ffb3000b330 comp rx=0 tx=0).stop 2023-06-22T08:13:17.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.941+0000 7ffb3cbff700 1 -- 172.21.15.23:0/1325299373 shutdown_connections 2023-06-22T08:13:17.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.941+0000 7ffb3cbff700 1 --2- 172.21.15.23:0/1325299373 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb380cf1b0 0x7ffb380c47c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:17.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.941+0000 7ffb3cbff700 1 -- 172.21.15.23:0/1325299373 >> 172.21.15.23:0/1325299373 conn(0x7ffb3806ae30 msgr2=0x7ffb3806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:17.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.941+0000 7ffb3cbff700 1 -- 172.21.15.23:0/1325299373 shutdown_connections 2023-06-22T08:13:17.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.941+0000 7ffb3cbff700 1 -- 172.21.15.23:0/1325299373 wait complete. 2023-06-22T08:13:17.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.943+0000 7ffb3cbff700 1 Processor -- start 2023-06-22T08:13:17.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.943+0000 7ffb3cbff700 1 -- start start 2023-06-22T08:13:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb3cbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb380c9fd0 0x7ffb380b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb3cbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb380cf1b0 0x7ffb3814bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb3cbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb38152ce0 0x7ffb381507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb3cbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb380cdc60 con 0x7ffb380c9fd0 2023-06-22T08:13:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb3cbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb380cd060 con 0x7ffb380cf1b0 2023-06-22T08:13:17.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb3cbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb381ee970 con 0x7ffb38152ce0 2023-06-22T08:13:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb3659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb380c9fd0 0x7ffb380b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb2ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb380cf1b0 0x7ffb3814bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb3659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb380c9fd0 0x7ffb380b3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40270/0 (socket says 172.21.15.23:40270) 2023-06-22T08:13:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb36d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb38152ce0 0x7ffb381507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb3659c700 1 -- 172.21.15.23:0/3068500687 learned_addr learned my addr 172.21.15.23:0/3068500687 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb2ffff700 1 -- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb38152ce0 msgr2=0x7ffb381507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb2ffff700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb38152ce0 0x7ffb381507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:17.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb2ffff700 1 -- 172.21.15.23:0/3068500687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb380c9fd0 msgr2=0x7ffb380b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:17.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb2ffff700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb380c9fd0 0x7ffb380b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:17.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.944+0000 7ffb2ffff700 1 -- 172.21.15.23:0/3068500687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb30009a70 con 0x7ffb380cf1b0 2023-06-22T08:13:17.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.945+0000 7ffb2ffff700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb380cf1b0 0x7ffb3814bcb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000c420 tx=0x7ffb2000c730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:17.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.945+0000 7ffb2effd700 1 -- 172.21.15.23:0/3068500687 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb20004a70 con 0x7ffb380cf1b0 2023-06-22T08:13:17.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.945+0000 7ffb2effd700 1 -- 172.21.15.23:0/3068500687 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb20004bf0 con 0x7ffb380cf1b0 2023-06-22T08:13:17.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.945+0000 7ffb3cbff700 1 -- 172.21.15.23:0/3068500687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb380cc730 con 0x7ffb380cf1b0 2023-06-22T08:13:17.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.945+0000 7ffb2effd700 1 -- 172.21.15.23:0/3068500687 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb200070d0 con 0x7ffb380cf1b0 2023-06-22T08:13:17.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.945+0000 7ffb3cbff700 1 -- 172.21.15.23:0/3068500687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb380cccf0 con 0x7ffb380cf1b0 2023-06-22T08:13:17.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.946+0000 7ffb3cbff700 1 -- 172.21.15.23:0/3068500687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb380cceb0 con 0x7ffb380cf1b0 2023-06-22T08:13:17.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.947+0000 7ffb2effd700 1 -- 172.21.15.23:0/3068500687 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ffb20007230 con 0x7ffb380cf1b0 2023-06-22T08:13:17.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.948+0000 7ffb2effd700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb1806ace0 0x7ffb1806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:17.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.948+0000 7ffb2effd700 1 -- 172.21.15.23:0/3068500687 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7ffb20089220 con 0x7ffb380cf1b0 2023-06-22T08:13:17.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.948+0000 7ffb3659c700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb1806ace0 0x7ffb1806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:17.954 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.949+0000 7ffb3659c700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb1806ace0 0x7ffb1806d190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ffb28000f60 tx=0x7ffb28007f30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:17.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:17.952+0000 7ffb2effd700 1 -- 172.21.15.23:0/3068500687 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ffb20053ce0 con 0x7ffb380cf1b0 2023-06-22T08:13:18.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.209+0000 7ffb3cbff700 1 -- 172.21.15.23:0/3068500687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb3804ea20 con 0x7ffb380cf1b0 2023-06-22T08:13:18.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.211+0000 7ffb2effd700 1 -- 172.21.15.23:0/3068500687 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5514 (secure 0 0 0) 0x7ffb20057300 con 0x7ffb380cf1b0 2023-06-22T08:13:18.214 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:18.214 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:09.469620+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:18.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 -- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb1806ace0 msgr2=0x7ffb1806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb1806ace0 0x7ffb1806d190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ffb28000f60 tx=0x7ffb28007f30 comp rx=0 tx=0).stop 2023-06-22T08:13:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 -- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb380cf1b0 msgr2=0x7ffb3814bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb380cf1b0 0x7ffb3814bcb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000c420 tx=0x7ffb2000c730 comp rx=0 tx=0).stop 2023-06-22T08:13:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 -- 172.21.15.23:0/3068500687 shutdown_connections 2023-06-22T08:13:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ffb38152ce0 0x7ffb381507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb380c9fd0 0x7ffb380b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ffb1806ace0 0x7ffb1806d190 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 --2- 172.21.15.23:0/3068500687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb380cf1b0 0x7ffb3814bcb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 -- 172.21.15.23:0/3068500687 >> 172.21.15.23:0/3068500687 conn(0x7ffb3806ae30 msgr2=0x7ffb380c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 -- 172.21.15.23:0/3068500687 shutdown_connections 2023-06-22T08:13:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:18.212+0000 7ffb3cbff700 1 -- 172.21.15.23:0/3068500687 wait complete. 2023-06-22T08:13:18.218 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 101 2023-06-22T08:13:18.235 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:13:20.237 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:20.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.490+0000 7f388bbff700 1 Processor -- start 2023-06-22T08:13:20.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.490+0000 7f388bbff700 1 -- start start 2023-06-22T08:13:20.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.491+0000 7f388bbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f38840c97d0 0x7f3884159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:20.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.491+0000 7f388bbff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38840c45f0 0x7f38840c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:20.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.491+0000 7f388bbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38840c4fb0 con 0x7f3884152d70 2023-06-22T08:13:20.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.491+0000 7f388bbff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38840c95e0 con 0x7f38840c45f0 2023-06-22T08:13:20.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.491+0000 7f388bbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38840c9c00 con 0x7f38840c97d0 2023-06-22T08:13:20.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.491+0000 7f388a19c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f38840c45f0 0x7f38840c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:20.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.491+0000 7f388999b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3884152d70 0x7f38840cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:53512/0 (socket says 172.21.15.23:53512) 2023-06-22T08:13:20.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.491+0000 7f388999b700 1 -- 172.21.15.23:0/1654661863 learned_addr learned my addr 172.21.15.23:0/1654661863 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:20.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.491+0000 7f388919a700 1 --2- 172.21.15.23:0/1654661863 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f38840c97d0 0x7f3884159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:20.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.492+0000 7f388a19c700 1 -- 172.21.15.23:0/1654661863 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f38840c97d0 msgr2=0x7f3884159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:20.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.492+0000 7f388a19c700 1 --2- 172.21.15.23:0/1654661863 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f38840c97d0 0x7f3884159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:20.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.492+0000 7f388a19c700 1 -- 172.21.15.23:0/1654661863 >> v1:172.21.15.67:6789/0 conn(0x7f3884152d70 legacy=0x7f38840cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:20.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.492+0000 7f388a19c700 1 -- 172.21.15.23:0/1654661863 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388415bea0 con 0x7f38840c45f0 2023-06-22T08:13:20.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.492+0000 7f3888999700 1 -- 172.21.15.23:0/1654661863 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609825192 0 0) 0x7f38840c4fb0 con 0x7f3884152d70 2023-06-22T08:13:20.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.492+0000 7f388a19c700 1 --2- 172.21.15.23:0/1654661863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38840c45f0 0x7f38840c4a70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f387c009d90 tx=0x7f387c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8401306944200c4c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:20.501 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.492+0000 7f3888999700 1 -- 172.21.15.23:0/1654661863 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c00e070 con 0x7f38840c45f0 2023-06-22T08:13:20.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.492+0000 7f3888999700 1 -- 172.21.15.23:0/1654661863 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c00bb20 con 0x7f38840c45f0 2023-06-22T08:13:20.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.492+0000 7f3888999700 1 -- 172.21.15.23:0/1654661863 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c012430 con 0x7f38840c45f0 2023-06-22T08:13:20.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.493+0000 7f388bbff700 1 -- 172.21.15.23:0/1654661863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38840c45f0 msgr2=0x7f38840c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:20.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.493+0000 7f388bbff700 1 --2- 172.21.15.23:0/1654661863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38840c45f0 0x7f38840c4a70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f387c009d90 tx=0x7f387c00b330 comp rx=0 tx=0).stop 2023-06-22T08:13:20.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.493+0000 7f388bbff700 1 -- 172.21.15.23:0/1654661863 shutdown_connections 2023-06-22T08:13:20.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.493+0000 7f388bbff700 1 --2- 172.21.15.23:0/1654661863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38840c45f0 0x7f38840c4a70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:20.502 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.493+0000 7f388bbff700 1 --2- 172.21.15.23:0/1654661863 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f38840c97d0 0x7f3884159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:20.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.493+0000 7f388bbff700 1 -- 172.21.15.23:0/1654661863 >> 172.21.15.23:0/1654661863 conn(0x7f388406ae30 msgr2=0x7f388406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:20.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.493+0000 7f388bbff700 1 -- 172.21.15.23:0/1654661863 shutdown_connections 2023-06-22T08:13:20.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.493+0000 7f388bbff700 1 -- 172.21.15.23:0/1654661863 wait complete. 2023-06-22T08:13:20.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.495+0000 7f388bbff700 1 Processor -- start 2023-06-22T08:13:20.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.495+0000 7f388bbff700 1 -- start start 2023-06-22T08:13:20.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388bbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38840c45f0 0x7f38841f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:20.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388bbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38840c97d0 0x7f38840ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:20.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388bbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3884152d70 0x7f388415b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:20.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388bbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840ccac0 con 0x7f38840c97d0 2023-06-22T08:13:20.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388bbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38840cdb80 con 0x7f38840c45f0 2023-06-22T08:13:20.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388bbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38840cc770 con 0x7f3884152d70 2023-06-22T08:13:20.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388919a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38840c97d0 0x7f38840ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:20.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388919a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38840c97d0 0x7f38840ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40274/0 (socket says 172.21.15.23:40274) 2023-06-22T08:13:20.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388919a700 1 -- 172.21.15.23:0/1298665586 learned_addr learned my addr 172.21.15.23:0/1298665586 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:20.506 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388999b700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38840c45f0 0x7f38841f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:20.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388a19c700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3884152d70 0x7f388415b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:20.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388919a700 1 -- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3884152d70 msgr2=0x7f388415b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:20.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388919a700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3884152d70 0x7f388415b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:20.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388919a700 1 -- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38840c45f0 msgr2=0x7f38841f1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:20.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388919a700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38840c45f0 0x7f38841f1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:20.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.496+0000 7f388919a700 1 -- 172.21.15.23:0/1298665586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c009a70 con 0x7f38840c97d0 2023-06-22T08:13:20.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.497+0000 7f388919a700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38840c97d0 0x7f38840ccee0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f387400e9b0 tx=0x7f387400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:20.507 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.497+0000 7f387a7fc700 1 -- 172.21.15.23:0/1298665586 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3874009710 con 0x7f38840c97d0 2023-06-22T08:13:20.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.497+0000 7f387a7fc700 1 -- 172.21.15.23:0/1298665586 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387400ee50 con 0x7f38840c97d0 2023-06-22T08:13:20.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.497+0000 7f387a7fc700 1 -- 172.21.15.23:0/1298665586 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387400ca50 con 0x7f38840c97d0 2023-06-22T08:13:20.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.497+0000 7f388bbff700 1 -- 172.21.15.23:0/1298665586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38840cddd0 con 0x7f38840c97d0 2023-06-22T08:13:20.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.497+0000 7f388bbff700 1 -- 172.21.15.23:0/1298665586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38840ce390 con 0x7f38840c97d0 2023-06-22T08:13:20.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.498+0000 7f388bbff700 1 -- 172.21.15.23:0/1298665586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38840ccc00 con 0x7f38840c97d0 2023-06-22T08:13:20.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.499+0000 7f387a7fc700 1 -- 172.21.15.23:0/1298665586 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f387400cbb0 con 0x7f38840c97d0 2023-06-22T08:13:20.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.500+0000 7f387a7fc700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f386806ace0 0x7f386806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:20.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.500+0000 7f387a7fc700 1 -- 172.21.15.23:0/1298665586 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f3874089970 con 0x7f38840c97d0 2023-06-22T08:13:20.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.500+0000 7f388999b700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f386806ace0 0x7f386806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:20.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.501+0000 7f388999b700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f386806ace0 0x7f386806d190 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3880000f60 tx=0x7f3880007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:20.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.505+0000 7f387a7fc700 1 -- 172.21.15.23:0/1298665586 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f38840ccc00 con 0x7f38840c97d0 2023-06-22T08:13:20.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.756+0000 7f388bbff700 1 -- 172.21.15.23:0/1298665586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38840ccc00 con 0x7f38840c97d0 2023-06-22T08:13:20.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.756+0000 7f387a7fc700 1 -- 172.21.15.23:0/1298665586 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5516 (secure 0 0 0) 0x7f38840ccc00 con 0x7f38840c97d0 2023-06-22T08:13:20.761 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:20.761 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:18.496661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:resolve","state_seq":28,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:20.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 -- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f386806ace0 msgr2=0x7f386806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:20.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f386806ace0 0x7f386806d190 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3880000f60 tx=0x7f3880007e70 comp rx=0 tx=0).stop 2023-06-22T08:13:20.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 -- 172.21.15.23:0/1298665586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38840c97d0 msgr2=0x7f38840ccee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:20.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38840c97d0 0x7f38840ccee0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f387400e9b0 tx=0x7f387400ecc0 comp rx=0 tx=0).stop 2023-06-22T08:13:20.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 -- 172.21.15.23:0/1298665586 shutdown_connections 2023-06-22T08:13:20.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3884152d70 0x7f388415b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:20.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f386806ace0 0x7f386806d190 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:20.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38840c45f0 0x7f38841f1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:20.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 --2- 172.21.15.23:0/1298665586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38840c97d0 0x7f38840ccee0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:20.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 -- 172.21.15.23:0/1298665586 >> 172.21.15.23:0/1298665586 conn(0x7f388406ae30 msgr2=0x7f3884150360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:20.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 -- 172.21.15.23:0/1298665586 shutdown_connections 2023-06-22T08:13:20.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:20.759+0000 7f388bbff700 1 -- 172.21.15.23:0/1298665586 wait complete. 2023-06-22T08:13:20.766 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 102 2023-06-22T08:13:20.781 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:13:22.782 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:22.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.988+0000 7fa0bf5ff700 1 Processor -- start 2023-06-22T08:13:22.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.988+0000 7fa0bf5ff700 1 -- start start 2023-06-22T08:13:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.989+0000 7fa0bf5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0b814f850 0x7fa0b80c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.989+0000 7fa0bf5ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa0b8071070 0x7fa0b80c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.989+0000 7fa0bf5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b80c51a0 con 0x7fa0b80c69e0 2023-06-22T08:13:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.989+0000 7fa0bf5ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b80c44e0 con 0x7fa0b8071070 2023-06-22T08:13:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.989+0000 7fa0bf5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b80c5440 con 0x7fa0b814f850 2023-06-22T08:13:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.990+0000 7fa0bdb9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa0b8071070 0x7fa0b80c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.990+0000 7fa0bd39b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0b814f850 0x7fa0b80c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.990+0000 7fa0bdb9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa0b8071070 0x7fa0b80c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40280/0 (socket says 172.21.15.23:40280) 2023-06-22T08:13:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.990+0000 7fa0bdb9c700 1 -- 172.21.15.23:0/1859983889 learned_addr learned my addr 172.21.15.23:0/1859983889 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:22.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.990+0000 7fa0bdb9c700 1 -- 172.21.15.23:0/1859983889 >> v2:172.21.15.67:3300/0 conn(0x7fa0b814f850 msgr2=0x7fa0b80c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:22.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.990+0000 7fa0bdb9c700 1 --2- 172.21.15.23:0/1859983889 >> v2:172.21.15.67:3300/0 conn(0x7fa0b814f850 0x7fa0b80c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:22.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.990+0000 7fa0bdb9c700 1 -- 172.21.15.23:0/1859983889 >> v1:172.21.15.67:6789/0 conn(0x7fa0b80c69e0 legacy=0x7fa0b8155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:22.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.990+0000 7fa0bdb9c700 1 -- 172.21.15.23:0/1859983889 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b80c5720 con 0x7fa0b8071070 2023-06-22T08:13:22.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.990+0000 7fa0bdb9c700 1 --2- 172.21.15.23:0/1859983889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 0x7fa0b80c35c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa0b806f240 tx=0x7fa0b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ca68898202f65c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:22.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0affff700 1 -- 172.21.15.23:0/1859983889 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b000e070 con 0x7fa0b8071070 2023-06-22T08:13:22.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0affff700 1 -- 172.21.15.23:0/1859983889 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b000b910 con 0x7fa0b8071070 2023-06-22T08:13:22.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0affff700 1 -- 172.21.15.23:0/1859983889 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b000bcc0 con 0x7fa0b8071070 2023-06-22T08:13:22.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/1859983889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 msgr2=0x7fa0b80c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:22.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0bf5ff700 1 --2- 172.21.15.23:0/1859983889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 0x7fa0b80c35c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa0b806f240 tx=0x7fa0b000b330 comp rx=0 tx=0).stop 2023-06-22T08:13:22.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/1859983889 shutdown_connections 2023-06-22T08:13:22.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0bf5ff700 1 --2- 172.21.15.23:0/1859983889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 0x7fa0b80c35c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:22.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0bf5ff700 1 --2- 172.21.15.23:0/1859983889 >> v2:172.21.15.67:3300/0 conn(0x7fa0b814f850 0x7fa0b80c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:22.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/1859983889 >> 172.21.15.23:0/1859983889 conn(0x7fa0b80ca3f0 msgr2=0x7fa0b80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:22.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/1859983889 shutdown_connections 2023-06-22T08:13:22.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.991+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/1859983889 wait complete. 2023-06-22T08:13:23.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.993+0000 7fa0bf5ff700 1 Processor -- start 2023-06-22T08:13:23.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.993+0000 7fa0bf5ff700 1 -- start start 2023-06-22T08:13:23.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.994+0000 7fa0bf5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 0x7fa0b81e0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:23.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.994+0000 7fa0bf5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0b80c69e0 0x7fa0b8150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:23.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.994+0000 7fa0bf5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b814f850 0x7fa0b8151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:23.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.994+0000 7fa0bf5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b8070410 con 0x7fa0b8071070 2023-06-22T08:13:23.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.994+0000 7fa0bf5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b81ee450 con 0x7fa0b814f850 2023-06-22T08:13:23.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.994+0000 7fa0bf5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b81ee730 con 0x7fa0b80c69e0 2023-06-22T08:13:23.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.994+0000 7fa0bd39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 0x7fa0b81e0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:23.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.994+0000 7fa0bd39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 0x7fa0b81e0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40284/0 (socket says 172.21.15.23:40284) 2023-06-22T08:13:23.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.994+0000 7fa0bd39b700 1 -- 172.21.15.23:0/813397375 learned_addr learned my addr 172.21.15.23:0/813397375 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:23.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0bcb9a700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0b80c69e0 0x7fa0b8150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:23.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0bdb9c700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b814f850 0x7fa0b8151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:23.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0bd39b700 1 -- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0b80c69e0 msgr2=0x7fa0b8150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:23.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0bd39b700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0b80c69e0 0x7fa0b8150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:23.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0bd39b700 1 -- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b814f850 msgr2=0x7fa0b8151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:23.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0bd39b700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b814f850 0x7fa0b8151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:23.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0bd39b700 1 -- 172.21.15.23:0/813397375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b0009b30 con 0x7fa0b8071070 2023-06-22T08:13:23.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0bd39b700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 0x7fa0b81e0b20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400e870 tx=0x7fa0b400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:23.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0adffb700 1 -- 172.21.15.23:0/813397375 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4009590 con 0x7fa0b8071070 2023-06-22T08:13:23.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0adffb700 1 -- 172.21.15.23:0/813397375 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b4013680 con 0x7fa0b8071070 2023-06-22T08:13:23.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0adffb700 1 -- 172.21.15.23:0/813397375 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b40139f0 con 0x7fa0b8071070 2023-06-22T08:13:23.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.995+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/813397375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b81eea10 con 0x7fa0b8071070 2023-06-22T08:13:23.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.996+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/813397375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b81eefd0 con 0x7fa0b8071070 2023-06-22T08:13:23.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.997+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/813397375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b8150950 con 0x7fa0b8071070 2023-06-22T08:13:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.998+0000 7fa0adffb700 1 -- 172.21.15.23:0/813397375 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa0b4013b50 con 0x7fa0b8071070 2023-06-22T08:13:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.999+0000 7fa0adffb700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa09c06ace0 0x7fa09c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.999+0000 7fa0adffb700 1 -- 172.21.15.23:0/813397375 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7fa0b4089930 con 0x7fa0b8071070 2023-06-22T08:13:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:22.999+0000 7fa0bcb9a700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa09c06ace0 0x7fa09c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.000+0000 7fa0bcb9a700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa09c06ace0 0x7fa09c06d190 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa0b81e1800 tx=0x7fa0a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:23.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.004+0000 7fa0adffb700 1 -- 172.21.15.23:0/813397375 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa0b4057e80 con 0x7fa0b8071070 2023-06-22T08:13:23.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.259+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/813397375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0b80c2950 con 0x7fa0b8071070 2023-06-22T08:13:23.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.259+0000 7fa0adffb700 1 -- 172.21.15.23:0/813397375 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5516 (secure 0 0 0) 0x7fa0b4057a10 con 0x7fa0b8071070 2023-06-22T08:13:23.265 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:23.265 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:18.496661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:resolve","state_seq":28,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:23.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa09c06ace0 msgr2=0x7fa09c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:23.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa09c06ace0 0x7fa09c06d190 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa0b81e1800 tx=0x7fa0a8009040 comp rx=0 tx=0).stop 2023-06-22T08:13:23.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/813397375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 msgr2=0x7fa0b81e0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:23.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 0x7fa0b81e0b20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400e870 tx=0x7fa0b400eb80 comp rx=0 tx=0).stop 2023-06-22T08:13:23.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/813397375 shutdown_connections 2023-06-22T08:13:23.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa09c06ace0 0x7fa09c06d190 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:23.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b814f850 0x7fa0b8151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:23.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa0b8071070 0x7fa0b81e0b20 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:23.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 --2- 172.21.15.23:0/813397375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa0b80c69e0 0x7fa0b8150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:23.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/813397375 >> 172.21.15.23:0/813397375 conn(0x7fa0b80ca3f0 msgr2=0x7fa0b80c3010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:23.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/813397375 shutdown_connections 2023-06-22T08:13:23.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:23.262+0000 7fa0bf5ff700 1 -- 172.21.15.23:0/813397375 wait complete. 2023-06-22T08:13:23.269 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 102 2023-06-22T08:13:23.291 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 102, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 102, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5211": { "addr": "172.21.15.23:6837/2058668008", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 2058668008, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 2058668008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5211, "incarnation": 35, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 26 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:resolve", "state_seq": 28 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:13:18.496661+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 5551, "mds_3": 5995, "mds_4": 5211 } } } ], "standbys": [] } 2023-06-22T08:13:25.292 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:25.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.624+0000 7f7375bff700 1 Processor -- start 2023-06-22T08:13:25.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.624+0000 7f7375bff700 1 -- start start 2023-06-22T08:13:25.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.625+0000 7f7375bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f73700c3a40 0x7f73700c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:25.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.625+0000 7f7375bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f737015a320 0x7f7370160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:25.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.625+0000 7f7375bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73700c6a60 con 0x7f737006d050 2023-06-22T08:13:25.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.625+0000 7f7375bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7370157570 con 0x7f737015a320 2023-06-22T08:13:25.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.625+0000 7f7375bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73700c4350 con 0x7f73700c3a40 2023-06-22T08:13:25.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.625+0000 7f736ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f737015a320 0x7f7370160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:25.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.625+0000 7f736effd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f737006d050 0x7f7370159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60920/0 (socket says 172.21.15.23:60920) 2023-06-22T08:13:25.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.625+0000 7f736effd700 1 -- 172.21.15.23:0/2004199418 learned_addr learned my addr 172.21.15.23:0/2004199418 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.625+0000 7f736f7fe700 1 --2- 172.21.15.23:0/2004199418 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f73700c3a40 0x7f73700c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.626+0000 7f736ffff700 1 -- 172.21.15.23:0/2004199418 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f73700c3a40 msgr2=0x7f73700c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.626+0000 7f736ffff700 1 --2- 172.21.15.23:0/2004199418 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f73700c3a40 0x7f73700c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.626+0000 7f736ffff700 1 -- 172.21.15.23:0/2004199418 >> v1:172.21.15.23:6789/0 conn(0x7f737006d050 legacy=0x7f7370159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.626+0000 7f736ffff700 1 -- 172.21.15.23:0/2004199418 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737014d690 con 0x7f737015a320 2023-06-22T08:13:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.626+0000 7f736ffff700 1 --2- 172.21.15.23:0/2004199418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737015a320 0x7f7370160ec0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7368009d90 tx=0x7f736800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca71fe60da6016d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.626+0000 7f736e7fc700 1 -- 172.21.15.23:0/2004199418 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736800e070 con 0x7f737015a320 2023-06-22T08:13:25.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.626+0000 7f736e7fc700 1 -- 172.21.15.23:0/2004199418 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736800bb20 con 0x7f737015a320 2023-06-22T08:13:25.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.626+0000 7f736e7fc700 1 -- 172.21.15.23:0/2004199418 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368012430 con 0x7f737015a320 2023-06-22T08:13:25.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.627+0000 7f7375bff700 1 -- 172.21.15.23:0/2004199418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737015a320 msgr2=0x7f7370160ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:25.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.627+0000 7f7375bff700 1 --2- 172.21.15.23:0/2004199418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737015a320 0x7f7370160ec0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7368009d90 tx=0x7f736800b330 comp rx=0 tx=0).stop 2023-06-22T08:13:25.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.627+0000 7f7375bff700 1 -- 172.21.15.23:0/2004199418 shutdown_connections 2023-06-22T08:13:25.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.627+0000 7f7375bff700 1 --2- 172.21.15.23:0/2004199418 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f73700c3a40 0x7f73700c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:25.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.627+0000 7f7375bff700 1 --2- 172.21.15.23:0/2004199418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737015a320 0x7f7370160ec0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:25.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.627+0000 7f7375bff700 1 -- 172.21.15.23:0/2004199418 >> 172.21.15.23:0/2004199418 conn(0x7f73700ca3f0 msgr2=0x7f73700ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:25.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.627+0000 7f7375bff700 1 -- 172.21.15.23:0/2004199418 shutdown_connections 2023-06-22T08:13:25.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.627+0000 7f7375bff700 1 -- 172.21.15.23:0/2004199418 wait complete. 2023-06-22T08:13:25.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.629+0000 7f7375bff700 1 Processor -- start 2023-06-22T08:13:25.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.629+0000 7f7375bff700 1 -- start start 2023-06-22T08:13:25.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f7375bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737006d050 0x7f73701554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:25.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f7375bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73700c3a40 0x7f73701504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:25.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f7375bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f737015a320 0x7f737014e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:25.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f7375bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7370070980 con 0x7f737006d050 2023-06-22T08:13:25.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f7375bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73700c7080 con 0x7f73700c3a40 2023-06-22T08:13:25.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f7375bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73700c7360 con 0x7f737015a320 2023-06-22T08:13:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f736f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737006d050 0x7f73701554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f736f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737006d050 0x7f73701554b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40300/0 (socket says 172.21.15.23:40300) 2023-06-22T08:13:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f736f7fe700 1 -- 172.21.15.23:0/3011874961 learned_addr learned my addr 172.21.15.23:0/3011874961 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f736effd700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73700c3a40 0x7f73701504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.630+0000 7f736ffff700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f737015a320 0x7f737014e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f736f7fe700 1 -- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f737015a320 msgr2=0x7f737014e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f736f7fe700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f737015a320 0x7f737014e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f736f7fe700 1 -- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73700c3a40 msgr2=0x7f73701504b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f736f7fe700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73700c3a40 0x7f73701504b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f736f7fe700 1 -- 172.21.15.23:0/3011874961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7368009a70 con 0x7f737006d050 2023-06-22T08:13:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f736f7fe700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737006d050 0x7f73701554b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f736000e9b0 tx=0x7f736000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f7357fff700 1 -- 172.21.15.23:0/3011874961 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360009710 con 0x7f737006d050 2023-06-22T08:13:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f7357fff700 1 -- 172.21.15.23:0/3011874961 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736000ee50 con 0x7f737006d050 2023-06-22T08:13:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f7375bff700 1 -- 172.21.15.23:0/3011874961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73700c7640 con 0x7f737006d050 2023-06-22T08:13:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f7357fff700 1 -- 172.21.15.23:0/3011874961 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736000ca50 con 0x7f737006d050 2023-06-22T08:13:25.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.631+0000 7f7375bff700 1 -- 172.21.15.23:0/3011874961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73700c7c00 con 0x7f737006d050 2023-06-22T08:13:25.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.632+0000 7f7375bff700 1 -- 172.21.15.23:0/3011874961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f737014e320 con 0x7f737006d050 2023-06-22T08:13:25.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.633+0000 7f7357fff700 1 -- 172.21.15.23:0/3011874961 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f736000cbb0 con 0x7f737006d050 2023-06-22T08:13:25.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.634+0000 7f7357fff700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f735c06ace0 0x7f735c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:25.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.634+0000 7f7357fff700 1 -- 172.21.15.23:0/3011874961 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f7360089e80 con 0x7f737006d050 2023-06-22T08:13:25.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.635+0000 7f736effd700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f735c06ace0 0x7f735c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:25.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.635+0000 7f736effd700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f735c06ace0 0x7f735c06d190 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f73700c3a10 tx=0x7f7364009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:25.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.639+0000 7f7357fff700 1 -- 172.21.15.23:0/3011874961 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7360058340 con 0x7f737006d050 2023-06-22T08:13:25.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.894+0000 7f7375bff700 1 -- 172.21.15.23:0/3011874961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f737014e040 con 0x7f737006d050 2023-06-22T08:13:25.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.895+0000 7f7357fff700 1 -- 172.21.15.23:0/3011874961 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5518 (secure 0 0 0) 0x7f7360021090 con 0x7f737006d050 2023-06-22T08:13:25.899 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:25.900 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:23.853713+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:25.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 -- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f735c06ace0 msgr2=0x7f735c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:25.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f735c06ace0 0x7f735c06d190 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f73700c3a10 tx=0x7f7364009040 comp rx=0 tx=0).stop 2023-06-22T08:13:25.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 -- 172.21.15.23:0/3011874961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737006d050 msgr2=0x7f73701554b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:25.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737006d050 0x7f73701554b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f736000e9b0 tx=0x7f736000ecc0 comp rx=0 tx=0).stop 2023-06-22T08:13:25.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 -- 172.21.15.23:0/3011874961 shutdown_connections 2023-06-22T08:13:25.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f737015a320 0x7f737014e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:25.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f737006d050 0x7f73701554b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:25.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f735c06ace0 0x7f735c06d190 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:25.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 --2- 172.21.15.23:0/3011874961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73700c3a40 0x7f73701504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:25.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 -- 172.21.15.23:0/3011874961 >> 172.21.15.23:0/3011874961 conn(0x7f73700ca3f0 msgr2=0x7f7370070100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:25.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 -- 172.21.15.23:0/3011874961 shutdown_connections 2023-06-22T08:13:25.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:25.897+0000 7f7375bff700 1 -- 172.21.15.23:0/3011874961 wait complete. 2023-06-22T08:13:25.904 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 103 2023-06-22T08:13:25.920 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:13:27.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:28.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.129+0000 7f07ffbff700 1 Processor -- start 2023-06-22T08:13:28.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.129+0000 7f07ffbff700 1 -- start start 2023-06-22T08:13:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.130+0000 7f07ffbff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07f815a320 0x7f07f80c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.130+0000 7f07ffbff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f80c6a60 con 0x7f07f80c3a40 2023-06-22T08:13:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.130+0000 7f07ffbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07f814d690 con 0x7f07f806d050 2023-06-22T08:13:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.130+0000 7f07ffbff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07f8157570 con 0x7f07f815a320 2023-06-22T08:13:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.131+0000 7f07fd99b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f07f80c3a40 0x7f07f8159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:32924/0 (socket says 172.21.15.23:32924) 2023-06-22T08:13:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.131+0000 7f07fd99b700 1 -- 172.21.15.23:0/1997377368 learned_addr learned my addr 172.21.15.23:0/1997377368 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:28.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.131+0000 7f07fd19a700 1 --2- 172.21.15.23:0/1997377368 >> v2:172.21.15.67:3300/0 conn(0x7f07f815a320 0x7f07f80c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:28.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.131+0000 7f07fc999700 1 -- 172.21.15.23:0/1997377368 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382442880 0 0) 0x7f07f80c6a60 con 0x7f07f80c3a40 2023-06-22T08:13:28.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.131+0000 7f07fc999700 1 -- 172.21.15.23:0/1997377368 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07e40035b0 con 0x7f07f80c3a40 2023-06-22T08:13:28.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.131+0000 7f07fd19a700 1 -- 172.21.15.23:0/1997377368 >> v1:172.21.15.67:6789/0 conn(0x7f07f806d050 legacy=0x7f07f8160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:28.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.131+0000 7f07fd19a700 1 -- 172.21.15.23:0/1997377368 >> v1:172.21.15.23:6789/0 conn(0x7f07f80c3a40 legacy=0x7f07f8159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:28.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.131+0000 7f07fd19a700 1 -- 172.21.15.23:0/1997377368 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f80c6430 con 0x7f07f815a320 2023-06-22T08:13:28.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.131+0000 7f07fd19a700 1 --2- 172.21.15.23:0/1997377368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f815a320 0x7f07f80c3e10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009840 tx=0x7f07e8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266aebc95190adce server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:28.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.132+0000 7f07fc999700 1 -- 172.21.15.23:0/1997377368 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8011830 con 0x7f07f815a320 2023-06-22T08:13:28.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.132+0000 7f07fc999700 1 -- 172.21.15.23:0/1997377368 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e8011990 con 0x7f07f815a320 2023-06-22T08:13:28.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.132+0000 7f07fc999700 1 -- 172.21.15.23:0/1997377368 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e8011d20 con 0x7f07f815a320 2023-06-22T08:13:28.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.132+0000 7f07ffbff700 1 -- 172.21.15.23:0/1997377368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f815a320 msgr2=0x7f07f80c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:28.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.132+0000 7f07ffbff700 1 --2- 172.21.15.23:0/1997377368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f815a320 0x7f07f80c3e10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009840 tx=0x7f07e8009b50 comp rx=0 tx=0).stop 2023-06-22T08:13:28.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.132+0000 7f07ffbff700 1 -- 172.21.15.23:0/1997377368 shutdown_connections 2023-06-22T08:13:28.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.132+0000 7f07ffbff700 1 --2- 172.21.15.23:0/1997377368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f815a320 0x7f07f80c3e10 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:28.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.132+0000 7f07ffbff700 1 -- 172.21.15.23:0/1997377368 >> 172.21.15.23:0/1997377368 conn(0x7f07f80ca3f0 msgr2=0x7f07f80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:28.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.132+0000 7f07ffbff700 1 -- 172.21.15.23:0/1997377368 shutdown_connections 2023-06-22T08:13:28.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.133+0000 7f07ffbff700 1 -- 172.21.15.23:0/1997377368 wait complete. 2023-06-22T08:13:28.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.134+0000 7f07ffbff700 1 Processor -- start 2023-06-22T08:13:28.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.134+0000 7f07ffbff700 1 -- start start 2023-06-22T08:13:28.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.135+0000 7f07ffbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07f806d050 0x7f07f8155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:28.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.135+0000 7f07ffbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07f80c3a40 0x7f07f81559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:28.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.135+0000 7f07ffbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f815a320 0x7f07f81f1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:28.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.135+0000 7f07ffbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f80c9f40 con 0x7f07f80c3a40 2023-06-22T08:13:28.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.135+0000 7f07ffbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f8140b40 con 0x7f07f815a320 2023-06-22T08:13:28.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.135+0000 7f07ffbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f81449b0 con 0x7f07f806d050 2023-06-22T08:13:28.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.135+0000 7f07fd99b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07f806d050 0x7f07f8155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:28.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.135+0000 7f07fd19a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07f80c3a40 0x7f07f81559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:28.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.135+0000 7f07fe19c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f815a320 0x7f07f81f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:28.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07fd99b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07f806d050 0x7f07f8155490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:41960/0 (socket says 172.21.15.23:41960) 2023-06-22T08:13:28.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07fd99b700 1 -- 172.21.15.23:0/341612080 learned_addr learned my addr 172.21.15.23:0/341612080 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:28.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07fe19c700 1 -- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07f806d050 msgr2=0x7f07f8155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:28.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07fe19c700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07f806d050 0x7f07f8155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:28.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07fe19c700 1 -- 172.21.15.23:0/341612080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07f80c3a40 msgr2=0x7f07f81559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:28.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07fe19c700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07f80c3a40 0x7f07f81559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:28.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07fe19c700 1 -- 172.21.15.23:0/341612080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e80094a0 con 0x7f07f815a320 2023-06-22T08:13:28.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07fd99b700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07f806d050 0x7f07f8155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:13:28.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07fd19a700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07f80c3a40 0x7f07f81559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:13:28.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07fe19c700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f815a320 0x7f07f81f1720 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f07f000cb90 tx=0x7f07f000cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:28.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07ee7fc700 1 -- 172.21.15.23:0/341612080 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f0003c00 con 0x7f07f815a320 2023-06-22T08:13:28.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.136+0000 7f07ee7fc700 1 -- 172.21.15.23:0/341612080 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f0003d60 con 0x7f07f815a320 2023-06-22T08:13:28.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.137+0000 7f07ee7fc700 1 -- 172.21.15.23:0/341612080 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f000aa80 con 0x7f07f815a320 2023-06-22T08:13:28.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.137+0000 7f07ffbff700 1 -- 172.21.15.23:0/341612080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f8145b60 con 0x7f07f815a320 2023-06-22T08:13:28.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.137+0000 7f07ffbff700 1 -- 172.21.15.23:0/341612080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f80c81d0 con 0x7f07f815a320 2023-06-22T08:13:28.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.138+0000 7f07ffbff700 1 -- 172.21.15.23:0/341612080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f804ea20 con 0x7f07f815a320 2023-06-22T08:13:28.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.139+0000 7f07ee7fc700 1 -- 172.21.15.23:0/341612080 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f07f000abe0 con 0x7f07f815a320 2023-06-22T08:13:28.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.140+0000 7f07ee7fc700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f07dc06ace0 0x7f07dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:28.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.140+0000 7f07ee7fc700 1 -- 172.21.15.23:0/341612080 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f07f0089d90 con 0x7f07f815a320 2023-06-22T08:13:28.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.140+0000 7f07fd99b700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f07dc06ace0 0x7f07dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:28.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.141+0000 7f07fd99b700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f07dc06ace0 0x7f07dc06d190 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f07f40011e0 tx=0x7f07f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:28.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.143+0000 7f07ee7fc700 1 -- 172.21.15.23:0/341612080 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f07f0054850 con 0x7f07f815a320 2023-06-22T08:13:28.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.397+0000 7f07ffbff700 1 -- 172.21.15.23:0/341612080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07f80c5f30 con 0x7f07f815a320 2023-06-22T08:13:28.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.398+0000 7f07ee7fc700 1 -- 172.21.15.23:0/341612080 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5518 (secure 0 0 0) 0x7f07f0057e70 con 0x7f07f815a320 2023-06-22T08:13:28.403 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:28.403 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:23.853713+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:28.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 -- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f07dc06ace0 msgr2=0x7f07dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:28.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f07dc06ace0 0x7f07dc06d190 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f07f40011e0 tx=0x7f07f4009040 comp rx=0 tx=0).stop 2023-06-22T08:13:28.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 -- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f815a320 msgr2=0x7f07f81f1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:28.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f815a320 0x7f07f81f1720 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f07f000cb90 tx=0x7f07f000cea0 comp rx=0 tx=0).stop 2023-06-22T08:13:28.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 -- 172.21.15.23:0/341612080 shutdown_connections 2023-06-22T08:13:28.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f07dc06ace0 0x7f07dc06d190 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:28.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07f815a320 0x7f07f81f1720 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:28.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07f806d050 0x7f07f8155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:28.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 --2- 172.21.15.23:0/341612080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07f80c3a40 0x7f07f81559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:28.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 -- 172.21.15.23:0/341612080 >> 172.21.15.23:0/341612080 conn(0x7f07f80ca3f0 msgr2=0x7f07f8148e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:28.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 -- 172.21.15.23:0/341612080 shutdown_connections 2023-06-22T08:13:28.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:28.401+0000 7f07ffbff700 1 -- 172.21.15.23:0/341612080 wait complete. 2023-06-22T08:13:28.408 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 103 2023-06-22T08:13:28.423 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:13:30.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:30.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.630+0000 7f3e1c5ff700 1 Processor -- start 2023-06-22T08:13:30.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.630+0000 7f3e1c5ff700 1 -- start start 2023-06-22T08:13:30.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e1c5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e140c4a30 0x7f3e140cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:30.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e1c5ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e14159730 con 0x7f3e140cb7a0 2023-06-22T08:13:30.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e1c5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e14158910 con 0x7f3e14152a10 2023-06-22T08:13:30.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e1c5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e140c5470 con 0x7f3e140c4a30 2023-06-22T08:13:30.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e1ab9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f3e140cb7a0 0x7f3e1415c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:32932/0 (socket says 172.21.15.23:32932) 2023-06-22T08:13:30.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e1ab9c700 1 -- 172.21.15.23:0/942131209 learned_addr learned my addr 172.21.15.23:0/942131209 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:30.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e19b9a700 1 --2- 172.21.15.23:0/942131209 >> v2:172.21.15.67:3300/0 conn(0x7f3e140c4a30 0x7f3e140cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:30.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e19399700 1 -- 172.21.15.23:0/942131209 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836862325 0 0) 0x7f3e14159730 con 0x7f3e140cb7a0 2023-06-22T08:13:30.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e19b9a700 1 -- 172.21.15.23:0/942131209 >> v1:172.21.15.67:6789/0 conn(0x7f3e14152a10 legacy=0x7f3e140c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:30.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e19b9a700 1 -- 172.21.15.23:0/942131209 >> v1:172.21.15.23:6789/0 conn(0x7f3e140cb7a0 legacy=0x7f3e1415c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:30.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.632+0000 7f3e19b9a700 1 -- 172.21.15.23:0/942131209 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e14158bb0 con 0x7f3e140c4a30 2023-06-22T08:13:30.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.633+0000 7f3e19b9a700 1 --2- 172.21.15.23:0/942131209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e140c4a30 0x7f3e140cb260 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3e04009840 tx=0x7f3e04009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2af1c7ce2322dae5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:30.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.633+0000 7f3e19399700 1 -- 172.21.15.23:0/942131209 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e04011830 con 0x7f3e140c4a30 2023-06-22T08:13:30.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.633+0000 7f3e19399700 1 -- 172.21.15.23:0/942131209 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e04011990 con 0x7f3e140c4a30 2023-06-22T08:13:30.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.633+0000 7f3e19399700 1 -- 172.21.15.23:0/942131209 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e04011d20 con 0x7f3e140c4a30 2023-06-22T08:13:30.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.633+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/942131209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e140c4a30 msgr2=0x7f3e140cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:30.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.633+0000 7f3e1c5ff700 1 --2- 172.21.15.23:0/942131209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e140c4a30 0x7f3e140cb260 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3e04009840 tx=0x7f3e04009b50 comp rx=0 tx=0).stop 2023-06-22T08:13:30.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.635+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/942131209 shutdown_connections 2023-06-22T08:13:30.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.635+0000 7f3e1c5ff700 1 --2- 172.21.15.23:0/942131209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e140c4a30 0x7f3e140cb260 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:30.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.635+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/942131209 >> 172.21.15.23:0/942131209 conn(0x7f3e14065b90 msgr2=0x7f3e14065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:30.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.635+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/942131209 shutdown_connections 2023-06-22T08:13:30.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.635+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/942131209 wait complete. 2023-06-22T08:13:30.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.635+0000 7f3e1c5ff700 1 Processor -- start 2023-06-22T08:13:30.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.635+0000 7f3e1c5ff700 1 -- start start 2023-06-22T08:13:30.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.636+0000 7f3e1c5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e140c4a30 0x7f3e14157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:30.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.636+0000 7f3e1c5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e140cb7a0 0x7f3e140cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:30.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.636+0000 7f3e1c5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e14152a10 0x7f3e141e5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:30.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.636+0000 7f3e1c5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e141580c0 con 0x7f3e140c4a30 2023-06-22T08:13:30.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.636+0000 7f3e1c5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e141ef2f0 con 0x7f3e140cb7a0 2023-06-22T08:13:30.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.636+0000 7f3e1c5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e141e44e0 con 0x7f3e14152a10 2023-06-22T08:13:30.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e1a39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e140c4a30 0x7f3e14157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:30.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e1a39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e140c4a30 0x7f3e14157b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56652/0 (socket says 172.21.15.23:56652) 2023-06-22T08:13:30.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e1a39b700 1 -- 172.21.15.23:0/4087889755 learned_addr learned my addr 172.21.15.23:0/4087889755 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:30.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e19b9a700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e140cb7a0 0x7f3e140cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:30.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e1ab9c700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e14152a10 0x7f3e141e5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:30.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e1a39b700 1 -- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e14152a10 msgr2=0x7f3e141e5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:30.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e1a39b700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e14152a10 0x7f3e141e5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:30.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e1a39b700 1 -- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e140cb7a0 msgr2=0x7f3e140cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:30.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e1a39b700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e140cb7a0 0x7f3e140cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:30.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e1a39b700 1 -- 172.21.15.23:0/4087889755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e040094a0 con 0x7f3e140c4a30 2023-06-22T08:13:30.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e19b9a700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e140cb7a0 0x7f3e140cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:13:30.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.637+0000 7f3e1a39b700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e140c4a30 0x7f3e14157b80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000c450 tx=0x7f3e1000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:30.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.638+0000 7f3e0affd700 1 -- 172.21.15.23:0/4087889755 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e100050d0 con 0x7f3e140c4a30 2023-06-22T08:13:30.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.638+0000 7f3e0affd700 1 -- 172.21.15.23:0/4087889755 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e10005230 con 0x7f3e140c4a30 2023-06-22T08:13:30.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.638+0000 7f3e0affd700 1 -- 172.21.15.23:0/4087889755 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e10003680 con 0x7f3e140c4a30 2023-06-22T08:13:30.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.638+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/4087889755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e14150050 con 0x7f3e140c4a30 2023-06-22T08:13:30.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.638+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/4087889755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e141e4a60 con 0x7f3e140c4a30 2023-06-22T08:13:30.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.640+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/4087889755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e1404ea20 con 0x7f3e140c4a30 2023-06-22T08:13:30.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.640+0000 7f3e0affd700 1 -- 172.21.15.23:0/4087889755 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3e100037e0 con 0x7f3e140c4a30 2023-06-22T08:13:30.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.641+0000 7f3e0affd700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3df806ad10 0x7f3df806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:30.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.641+0000 7f3e0affd700 1 -- 172.21.15.23:0/4087889755 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f3e10088cb0 con 0x7f3e140c4a30 2023-06-22T08:13:30.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.641+0000 7f3e19b9a700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3df806ad10 0x7f3df806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:30.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.642+0000 7f3e19b9a700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3df806ad10 0x7f3df806d1c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3e040090b0 tx=0x7f3e0401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:30.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.645+0000 7f3e0affd700 1 -- 172.21.15.23:0/4087889755 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3e10053770 con 0x7f3e140c4a30 2023-06-22T08:13:30.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.906+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/4087889755 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e1406bbf0 con 0x7f3e140c4a30 2023-06-22T08:13:30.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.907+0000 7f3e0affd700 1 -- 172.21.15.23:0/4087889755 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5515 (secure 0 0 0) 0x7f3e10056d90 con 0x7f3e140c4a30 2023-06-22T08:13:30.910 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:30.911 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:28.891263+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:30.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3df806ad10 msgr2=0x7f3df806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:30.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3df806ad10 0x7f3df806d1c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3e040090b0 tx=0x7f3e0401c040 comp rx=0 tx=0).stop 2023-06-22T08:13:30.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/4087889755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e140c4a30 msgr2=0x7f3e14157b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:30.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e140c4a30 0x7f3e14157b80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000c450 tx=0x7f3e1000c760 comp rx=0 tx=0).stop 2023-06-22T08:13:30.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/4087889755 shutdown_connections 2023-06-22T08:13:30.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e14152a10 0x7f3e141e5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:30.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e140c4a30 0x7f3e14157b80 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:30.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3df806ad10 0x7f3df806d1c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:30.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 --2- 172.21.15.23:0/4087889755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e140cb7a0 0x7f3e140cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:30.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/4087889755 >> 172.21.15.23:0/4087889755 conn(0x7f3e14065b90 msgr2=0x7f3e14065f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:30.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/4087889755 shutdown_connections 2023-06-22T08:13:30.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:30.909+0000 7f3e1c5ff700 1 -- 172.21.15.23:0/4087889755 wait complete. 2023-06-22T08:13:30.915 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 104 2023-06-22T08:13:30.932 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:13:32.933 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:33.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.142+0000 7f3381dff700 1 Processor -- start 2023-06-22T08:13:33.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.142+0000 7f3381dff700 1 -- start start 2023-06-22T08:13:33.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f3381dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f337c0c97d0 0x7f337c159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:33.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f3381dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f337c0c45f0 0x7f337c0c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:33.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f3381dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c0c9c30 con 0x7f337c152d70 2023-06-22T08:13:33.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f3381dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337c0c95e0 con 0x7f337c0c45f0 2023-06-22T08:13:33.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f3381dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337c15bea0 con 0x7f337c0c97d0 2023-06-22T08:13:33.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f337bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f337c0c45f0 0x7f337c0c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:33.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f337affd700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f337c0c97d0 0x7f337c159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:33.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f337b7fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f337c152d70 0x7f337c0cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:43056/0 (socket says 172.21.15.23:43056) 2023-06-22T08:13:33.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f337bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f337c0c45f0 0x7f337c0c4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56668/0 (socket says 172.21.15.23:56668) 2023-06-22T08:13:33.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f337b7fe700 1 -- 172.21.15.23:0/27009324 learned_addr learned my addr 172.21.15.23:0/27009324 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:33.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f337bfff700 1 -- 172.21.15.23:0/27009324 >> v2:172.21.15.67:3300/0 conn(0x7f337c0c97d0 msgr2=0x7f337c159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:33.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f337bfff700 1 --2- 172.21.15.23:0/27009324 >> v2:172.21.15.67:3300/0 conn(0x7f337c0c97d0 0x7f337c159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:33.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f337bfff700 1 -- 172.21.15.23:0/27009324 >> v1:172.21.15.67:6789/0 conn(0x7f337c152d70 legacy=0x7f337c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:33.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.144+0000 7f337bfff700 1 -- 172.21.15.23:0/27009324 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c15c160 con 0x7f337c0c45f0 2023-06-22T08:13:33.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.145+0000 7f337bfff700 1 --2- 172.21.15.23:0/27009324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 0x7f337c0c4a10 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f337c08ba40 tx=0x7f337400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f066d7241e6d77 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.145+0000 7f337a7fc700 1 -- 172.21.15.23:0/27009324 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337400e070 con 0x7f337c0c45f0 2023-06-22T08:13:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.145+0000 7f337a7fc700 1 -- 172.21.15.23:0/27009324 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337400bb20 con 0x7f337c0c45f0 2023-06-22T08:13:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.145+0000 7f337a7fc700 1 -- 172.21.15.23:0/27009324 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374012430 con 0x7f337c0c45f0 2023-06-22T08:13:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.145+0000 7f3381dff700 1 -- 172.21.15.23:0/27009324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 msgr2=0x7f337c0c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:33.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.146+0000 7f3381dff700 1 --2- 172.21.15.23:0/27009324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 0x7f337c0c4a10 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f337c08ba40 tx=0x7f337400b330 comp rx=0 tx=0).stop 2023-06-22T08:13:33.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.146+0000 7f3381dff700 1 -- 172.21.15.23:0/27009324 shutdown_connections 2023-06-22T08:13:33.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.146+0000 7f3381dff700 1 --2- 172.21.15.23:0/27009324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 0x7f337c0c4a10 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:33.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.146+0000 7f3381dff700 1 --2- 172.21.15.23:0/27009324 >> v2:172.21.15.67:3300/0 conn(0x7f337c0c97d0 0x7f337c159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:33.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.146+0000 7f3381dff700 1 -- 172.21.15.23:0/27009324 >> 172.21.15.23:0/27009324 conn(0x7f337c06ae30 msgr2=0x7f337c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:33.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.146+0000 7f3381dff700 1 -- 172.21.15.23:0/27009324 shutdown_connections 2023-06-22T08:13:33.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.146+0000 7f3381dff700 1 -- 172.21.15.23:0/27009324 wait complete. 2023-06-22T08:13:33.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.147+0000 7f3381dff700 1 Processor -- start 2023-06-22T08:13:33.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.148+0000 7f3381dff700 1 -- start start 2023-06-22T08:13:33.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.148+0000 7f3381dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 0x7f337c1f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:33.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.148+0000 7f3381dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f337c0c97d0 0x7f337c151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:33.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.148+0000 7f3381dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337c152d70 0x7f337c0b2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:33.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.148+0000 7f3381dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c0c7b40 con 0x7f337c0c45f0 2023-06-22T08:13:33.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.148+0000 7f3381dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c0c8410 con 0x7f337c152d70 2023-06-22T08:13:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.148+0000 7f3381dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337c1f27c0 con 0x7f337c0c97d0 2023-06-22T08:13:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 0x7f337c1f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 0x7f337c1f2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56674/0 (socket says 172.21.15.23:56674) 2023-06-22T08:13:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337b7fe700 1 -- 172.21.15.23:0/874534425 learned_addr learned my addr 172.21.15.23:0/874534425 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:33.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337affd700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f337c0c97d0 0x7f337c151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:33.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337bfff700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337c152d70 0x7f337c0b2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:33.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337b7fe700 1 -- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f337c0c97d0 msgr2=0x7f337c151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:33.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337b7fe700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f337c0c97d0 0x7f337c151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:33.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337b7fe700 1 -- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337c152d70 msgr2=0x7f337c0b2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:33.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337b7fe700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337c152d70 0x7f337c0b2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337b7fe700 1 -- 172.21.15.23:0/874534425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3374009a70 con 0x7f337c0c45f0 2023-06-22T08:13:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.149+0000 7f337b7fe700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 0x7f337c1f2280 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f336c00c480 tx=0x7f336c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.150+0000 7f3363fff700 1 -- 172.21.15.23:0/874534425 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c005d70 con 0x7f337c0c45f0 2023-06-22T08:13:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.150+0000 7f3363fff700 1 -- 172.21.15.23:0/874534425 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c0070d0 con 0x7f337c0c45f0 2023-06-22T08:13:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.150+0000 7f3363fff700 1 -- 172.21.15.23:0/874534425 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c007440 con 0x7f337c0c45f0 2023-06-22T08:13:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.150+0000 7f3381dff700 1 -- 172.21.15.23:0/874534425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337c0c5ab0 con 0x7f337c0c45f0 2023-06-22T08:13:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.150+0000 7f3381dff700 1 -- 172.21.15.23:0/874534425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337c0c6bf0 con 0x7f337c0c45f0 2023-06-22T08:13:33.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.151+0000 7f3381dff700 1 -- 172.21.15.23:0/874534425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337c0c6ed0 con 0x7f337c0c45f0 2023-06-22T08:13:33.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.152+0000 7f3363fff700 1 -- 172.21.15.23:0/874534425 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f336c0075a0 con 0x7f337c0c45f0 2023-06-22T08:13:33.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.152+0000 7f3363fff700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f336806ace0 0x7f336806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:33.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.153+0000 7f3363fff700 1 -- 172.21.15.23:0/874534425 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f336c088d00 con 0x7f337c0c45f0 2023-06-22T08:13:33.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.153+0000 7f337affd700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f336806ace0 0x7f336806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:33.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.154+0000 7f337affd700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f336806ace0 0x7f336806d190 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f337c0b2ad0 tx=0x7f3370008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:33.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.157+0000 7f3363fff700 1 -- 172.21.15.23:0/874534425 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f336c053900 con 0x7f337c0c45f0 2023-06-22T08:13:33.415 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.412+0000 7f3381dff700 1 -- 172.21.15.23:0/874534425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f337c04ea20 con 0x7f337c0c45f0 2023-06-22T08:13:33.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.412+0000 7f3363fff700 1 -- 172.21.15.23:0/874534425 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5515 (secure 0 0 0) 0x7f336c056f20 con 0x7f337c0c45f0 2023-06-22T08:13:33.416 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:33.416 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:28.891263+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:33.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 -- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f336806ace0 msgr2=0x7f336806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:33.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f336806ace0 0x7f336806d190 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f337c0b2ad0 tx=0x7f3370008040 comp rx=0 tx=0).stop 2023-06-22T08:13:33.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 -- 172.21.15.23:0/874534425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 msgr2=0x7f337c1f2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:33.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 0x7f337c1f2280 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f336c00c480 tx=0x7f336c00c840 comp rx=0 tx=0).stop 2023-06-22T08:13:33.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 -- 172.21.15.23:0/874534425 shutdown_connections 2023-06-22T08:13:33.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f336806ace0 0x7f336806d190 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:33.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337c152d70 0x7f337c0b2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:33.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f337c0c45f0 0x7f337c1f2280 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:33.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 --2- 172.21.15.23:0/874534425 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f337c0c97d0 0x7f337c151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:33.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 -- 172.21.15.23:0/874534425 >> 172.21.15.23:0/874534425 conn(0x7f337c06ae30 msgr2=0x7f337c0ccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:33.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 -- 172.21.15.23:0/874534425 shutdown_connections 2023-06-22T08:13:33.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:33.415+0000 7f3381dff700 1 -- 172.21.15.23:0/874534425 wait complete. 2023-06-22T08:13:33.420 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 104 2023-06-22T08:13:33.436 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:13:35.437 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:35.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.644+0000 7f43361ff700 1 Processor -- start 2023-06-22T08:13:35.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.645+0000 7f43361ff700 1 -- start start 2023-06-22T08:13:35.652 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.646+0000 7f43361ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 0x7f43300c9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:35.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.646+0000 7f43361ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f433015cb00 0x7f43300c4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:35.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.646+0000 7f43361ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43300c9540 con 0x7f4330071850 2023-06-22T08:13:35.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.646+0000 7f43361ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4330158c10 con 0x7f433015cb00 2023-06-22T08:13:35.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.646+0000 7f43361ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4330158ef0 con 0x7f4330071c20 2023-06-22T08:13:35.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.646+0000 7f432ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f433015cb00 0x7f43300c4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:35.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.646+0000 7f432effd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 0x7f43300c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:35.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.646+0000 7f432ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f433015cb00 0x7f43300c4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56690/0 (socket says 172.21.15.23:56690) 2023-06-22T08:13:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.646+0000 7f432ffff700 1 -- 172.21.15.23:0/1853922437 learned_addr learned my addr 172.21.15.23:0/1853922437 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.646+0000 7f432f7fe700 1 --1- 172.21.15.23:0/1853922437 >> v1:172.21.15.67:6789/0 conn(0x7f4330071850 0x7f433015c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:43066/0 (socket says 172.21.15.23:43066) 2023-06-22T08:13:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.647+0000 7f432ffff700 1 -- 172.21.15.23:0/1853922437 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 msgr2=0x7f43300c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.647+0000 7f432ffff700 1 --2- 172.21.15.23:0/1853922437 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 0x7f43300c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.647+0000 7f432ffff700 1 -- 172.21.15.23:0/1853922437 >> v1:172.21.15.67:6789/0 conn(0x7f4330071850 legacy=0x7f433015c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.647+0000 7f432ffff700 1 -- 172.21.15.23:0/1853922437 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43301591d0 con 0x7f433015cb00 2023-06-22T08:13:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.647+0000 7f432e7fc700 1 -- 172.21.15.23:0/1853922437 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303106263 0 0) 0x7f43300c9540 con 0x7f4330071850 2023-06-22T08:13:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.647+0000 7f432ffff700 1 --2- 172.21.15.23:0/1853922437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433015cb00 0x7f43300c4a40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4328009e50 tx=0x7f432800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc125df8c3686315 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.647+0000 7f432e7fc700 1 -- 172.21.15.23:0/1853922437 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432800e070 con 0x7f433015cb00 2023-06-22T08:13:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.647+0000 7f432e7fc700 1 -- 172.21.15.23:0/1853922437 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432800b910 con 0x7f433015cb00 2023-06-22T08:13:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.647+0000 7f432e7fc700 1 -- 172.21.15.23:0/1853922437 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432800bcc0 con 0x7f433015cb00 2023-06-22T08:13:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.648+0000 7f43361ff700 1 -- 172.21.15.23:0/1853922437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433015cb00 msgr2=0x7f43300c4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.648+0000 7f43361ff700 1 --2- 172.21.15.23:0/1853922437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433015cb00 0x7f43300c4a40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4328009e50 tx=0x7f432800b330 comp rx=0 tx=0).stop 2023-06-22T08:13:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.648+0000 7f43361ff700 1 -- 172.21.15.23:0/1853922437 shutdown_connections 2023-06-22T08:13:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.648+0000 7f43361ff700 1 --2- 172.21.15.23:0/1853922437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433015cb00 0x7f43300c4a40 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:35.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.648+0000 7f43361ff700 1 --2- 172.21.15.23:0/1853922437 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 0x7f43300c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:35.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.648+0000 7f43361ff700 1 -- 172.21.15.23:0/1853922437 >> 172.21.15.23:0/1853922437 conn(0x7f433006b890 msgr2=0x7f433006bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:35.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.648+0000 7f43361ff700 1 -- 172.21.15.23:0/1853922437 shutdown_connections 2023-06-22T08:13:35.656 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.648+0000 7f43361ff700 1 -- 172.21.15.23:0/1853922437 wait complete. 2023-06-22T08:13:35.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.650+0000 7f43361ff700 1 Processor -- start 2023-06-22T08:13:35.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.651+0000 7f43361ff700 1 -- start start 2023-06-22T08:13:35.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f43361ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4330071850 0x7f433014e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:35.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f43361ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 0x7f43300ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:35.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f43361ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433015cb00 0x7f43300caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f43361ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43300cef50 con 0x7f433015cb00 2023-06-22T08:13:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f43361ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43300cf3e0 con 0x7f4330071850 2023-06-22T08:13:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f43361ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43300b25f0 con 0x7f4330071c20 2023-06-22T08:13:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f432ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433015cb00 0x7f43300caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f432f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4330071850 0x7f433014e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f432effd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 0x7f43300ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f432effd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 0x7f43300ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:42016/0 (socket says 172.21.15.23:42016) 2023-06-22T08:13:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f432effd700 1 -- 172.21.15.23:0/96097615 learned_addr learned my addr 172.21.15.23:0/96097615 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.652+0000 7f432ffff700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433015cb00 0x7f43300caa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56696/0 (socket says 172.21.15.23:56696) 2023-06-22T08:13:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f432effd700 1 -- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4330071850 msgr2=0x7f433014e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f432effd700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4330071850 0x7f433014e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f432effd700 1 -- 172.21.15.23:0/96097615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433015cb00 msgr2=0x7f43300caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f432effd700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433015cb00 0x7f43300caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f432effd700 1 -- 172.21.15.23:0/96097615 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4328009b30 con 0x7f4330071c20 2023-06-22T08:13:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f432effd700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 0x7f43300ca510 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f432400e9b0 tx=0x7f432400ecc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:35.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f4317fff700 1 -- 172.21.15.23:0/96097615 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324009710 con 0x7f4330071c20 2023-06-22T08:13:35.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f4317fff700 1 -- 172.21.15.23:0/96097615 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432400ee50 con 0x7f4330071c20 2023-06-22T08:13:35.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f43361ff700 1 -- 172.21.15.23:0/96097615 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43300b2890 con 0x7f4330071c20 2023-06-22T08:13:35.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f4317fff700 1 -- 172.21.15.23:0/96097615 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432400ca50 con 0x7f4330071c20 2023-06-22T08:13:35.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.653+0000 7f43361ff700 1 -- 172.21.15.23:0/96097615 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43300ce900 con 0x7f4330071c20 2023-06-22T08:13:35.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.654+0000 7f43361ff700 1 -- 172.21.15.23:0/96097615 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43300ce740 con 0x7f4330071c20 2023-06-22T08:13:35.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.655+0000 7f4317fff700 1 -- 172.21.15.23:0/96097615 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f432400cbb0 con 0x7f4330071c20 2023-06-22T08:13:35.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.656+0000 7f4317fff700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431c06ace0 0x7f431c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:35.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.656+0000 7f4317fff700 1 -- 172.21.15.23:0/96097615 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f4324089a00 con 0x7f4330071c20 2023-06-22T08:13:35.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.656+0000 7f432f7fe700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431c06ace0 0x7f431c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:35.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.657+0000 7f432f7fe700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431c06ace0 0x7f431c06d190 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4320000f60 tx=0x7f4320007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:35.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.661+0000 7f4317fff700 1 -- 172.21.15.23:0/96097615 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4324057fe0 con 0x7f4330071c20 2023-06-22T08:13:35.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.917+0000 7f43361ff700 1 -- 172.21.15.23:0/96097615 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f433004ea20 con 0x7f4330071c20 2023-06-22T08:13:35.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.918+0000 7f4317fff700 1 -- 172.21.15.23:0/96097615 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5515 (secure 0 0 0) 0x7f432400ce60 con 0x7f4330071c20 2023-06-22T08:13:35.921 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:35.922 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:34.929040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:35.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 -- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431c06ace0 msgr2=0x7f431c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:35.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431c06ace0 0x7f431c06d190 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4320000f60 tx=0x7f4320007e70 comp rx=0 tx=0).stop 2023-06-22T08:13:35.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 -- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 msgr2=0x7f43300ca510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:35.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 0x7f43300ca510 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f432400e9b0 tx=0x7f432400ecc0 comp rx=0 tx=0).stop 2023-06-22T08:13:35.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 -- 172.21.15.23:0/96097615 shutdown_connections 2023-06-22T08:13:35.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433015cb00 0x7f43300caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:35.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431c06ace0 0x7f431c06d190 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:35.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4330071850 0x7f433014e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:35.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 --2- 172.21.15.23:0/96097615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4330071c20 0x7f43300ca510 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:35.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 -- 172.21.15.23:0/96097615 >> 172.21.15.23:0/96097615 conn(0x7f433006b890 msgr2=0x7f4330152430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:35.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 -- 172.21.15.23:0/96097615 shutdown_connections 2023-06-22T08:13:35.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:35.920+0000 7f43361ff700 1 -- 172.21.15.23:0/96097615 wait complete. 2023-06-22T08:13:35.926 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 105 2023-06-22T08:13:35.942 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=3, replacing gid=5060 2023-06-22T08:13:35.942 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.d 2023-06-22T08:13:39.319 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:13:39.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:13:41.943 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-06-22T08:13:41.943 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-22T08:13:41.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-22T08:13:41.946 INFO:tasks.ceph.mds.d:Started 2023-06-22T08:13:41.946 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:41.952 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 1% (3/246) Updating files: 2% (5/246) Updating files: 3% (8/246) Updating files: 4% (10/246) Updating files: 5% (13/246) Updating files: 5% (14/246) Updating files: 6% (15/246) Updating files: 7% (18/246) Updating files: 8% (20/246) Updating files: 9% (23/246) Updating files: 10% (25/246) Updating files: 11% (28/246) Updating files: 12% (30/246) Updating files: 13% (32/246) Updating files: 14% (35/246) Updating files: 14% (36/246) Updating files: 15% (37/246) Updating files: 16% (40/246) Updating files: 17% (42/246) Updating files: 18% (45/246) Updating files: 19% (47/246) Updating files: 20% (50/246) Updating files: 21% (52/246) Updating files: 22% (55/246) Updating files: 23% (57/246) Updating files: 24% (60/246) Updating files: 25% (62/246) Updating files: 26% (64/246) Updating files: 27% (67/246) Updating files: 28% (69/246) Updating files: 29% (72/246) Updating files: 30% (74/246) Updating files: 31% (77/246) Updating files: 32% (79/246) Updating files: 33% (82/246) Updating files: 34% (84/246) Updating files: 35% (87/246) Updating files: 36% (89/246) Updating files: 37% (92/246) Updating files: 38% (94/246) Updating files: 39% (96/246) Updating files: 40% (99/246) Updating files: 41% (101/246) Updating files: 42% (104/246) Updating files: 43% (106/246) Updating files: 44% (109/246) Updating files: 45% (111/246) Updating files: 46% (114/246) Updating files: 47% (116/246) Updating files: 48% (119/246) Updating files: 49% (121/246) Updating files: 50% (123/246) Updating files: 51% (126/246) Updating files: 52% (128/246) Updating files: 53% (131/246) Updating files: 54% (133/246) Updating files: 55% (136/246) Updating files: 56% (138/246) Updating files: 57% (141/246) Updating files: 58% (143/246) Updating files: 59% (146/246) Updating files: 60% (148/246) Updating files: 61% (151/246) Updating files: 62% (153/246) Updating files: 63% (155/246) Updating files: 64% (158/246) Updating files: 65% (160/246) Updating files: 66% (163/246) Updating files: 67% (165/246) Updating files: 68% (168/246) Updating files: 69% (170/246) Updating files: 70% (173/246) Updating files: 71% (175/246) Updating files: 72% (178/246) Updating files: 73% (180/246) Updating files: 74% (183/246) Updating files: 75% (185/246) Updating files: 76% (187/246) Updating files: 77% (190/246) Updating files: 78% (192/246) Updating files: 79% (195/246) Updating files: 80% (197/246) Updating files: 81% (200/246) Updating files: 82% (202/246) Updating files: 83% (205/246) Updating files: 84% (207/246) Updating files: 85% (210/246) Updating files: 86% (212/246) Updating files: 87% (215/246) Updating files: 88% (217/246) Updating files: 89% (219/246) Updating files: 90% (222/246) Updating files: 91% (224/246) Updating files: 92% (227/246) Updating files: 93% (229/246) Updating files: 94% (232/246) Updating files: 95% (234/246) Updating files: 96% (237/246) Updating files: 97% (239/246) Updating files: 98% (242/246) Updating files: 99% (244/246) Updating files: 100% (246/246) Updating files: 100% (246/246), done. 2023-06-22T08:13:42.025 INFO:tasks.ceph.mds.d.smithi067.stdout:starting mds.d at 2023-06-22T08:13:42.064 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 1a32f0a0b42 v0.56 2023-06-22T08:13:42.072 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.56 2023-06-22T08:13:42.132 INFO:tasks.workunit.client.0.smithi023.stdout:v0.57 2023-06-22T08:13:42.132 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:13:42.132 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 57 -eq 48 ']' 2023-06-22T08:13:42.133 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.57 2023-06-22T08:13:42.133 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.57 2023-06-22T08:13:42.133 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.57 2023-06-22T08:13:42.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.217+0000 7f00223ff700 1 Processor -- start 2023-06-22T08:13:42.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.218+0000 7f00223ff700 1 -- start start 2023-06-22T08:13:42.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.219+0000 7f00223ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f001c0c3a40 0x7f001c0c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.219+0000 7f00223ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c0c6a60 con 0x7f001c06d050 2023-06-22T08:13:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.219+0000 7f00223ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c14d690 con 0x7f001c15a320 2023-06-22T08:13:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.219+0000 7f00223ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001c157570 con 0x7f001c0c3a40 2023-06-22T08:13:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.219+0000 7f001b7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f001c06d050 0x7f001c159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:43136/0 (socket says 172.21.15.23:43136) 2023-06-22T08:13:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.219+0000 7f001b7fe700 1 -- 172.21.15.23:0/2702631501 learned_addr learned my addr 172.21.15.23:0/2702631501 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.219+0000 7f001bfff700 1 --2- 172.21.15.23:0/2702631501 >> v2:172.21.15.67:3300/0 conn(0x7f001c0c3a40 0x7f001c0c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:42.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.220+0000 7f001bfff700 1 -- 172.21.15.23:0/2702631501 >> v1:172.21.15.67:6789/0 conn(0x7f001c15a320 legacy=0x7f001c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:42.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.220+0000 7f001bfff700 1 -- 172.21.15.23:0/2702631501 >> v1:172.21.15.23:6789/0 conn(0x7f001c06d050 legacy=0x7f001c159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:42.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.220+0000 7f001affd700 1 -- 172.21.15.23:0/2702631501 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662202971 0 0) 0x7f001c0c6a60 con 0x7f001c06d050 2023-06-22T08:13:42.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.220+0000 7f001bfff700 1 -- 172.21.15.23:0/2702631501 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c0c6430 con 0x7f001c0c3a40 2023-06-22T08:13:42.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.220+0000 7f001bfff700 1 --2- 172.21.15.23:0/2702631501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c0c3a40 0x7f001c0c3e10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f000c009840 tx=0x7f000c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=676b6046574f588f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.220+0000 7f001affd700 1 -- 172.21.15.23:0/2702631501 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000c011830 con 0x7f001c0c3a40 2023-06-22T08:13:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.221+0000 7f001affd700 1 -- 172.21.15.23:0/2702631501 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000c011990 con 0x7f001c0c3a40 2023-06-22T08:13:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.221+0000 7f001affd700 1 -- 172.21.15.23:0/2702631501 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000c011d20 con 0x7f001c0c3a40 2023-06-22T08:13:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.221+0000 7f00223ff700 1 -- 172.21.15.23:0/2702631501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c0c3a40 msgr2=0x7f001c0c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.221+0000 7f00223ff700 1 --2- 172.21.15.23:0/2702631501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c0c3a40 0x7f001c0c3e10 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f000c009840 tx=0x7f000c009b50 comp rx=0 tx=0).stop 2023-06-22T08:13:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.221+0000 7f00223ff700 1 -- 172.21.15.23:0/2702631501 shutdown_connections 2023-06-22T08:13:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.221+0000 7f00223ff700 1 --2- 172.21.15.23:0/2702631501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c0c3a40 0x7f001c0c3e10 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.221+0000 7f00223ff700 1 -- 172.21.15.23:0/2702631501 >> 172.21.15.23:0/2702631501 conn(0x7f001c0ca3f0 msgr2=0x7f001c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.221+0000 7f00223ff700 1 -- 172.21.15.23:0/2702631501 shutdown_connections 2023-06-22T08:13:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.221+0000 7f00223ff700 1 -- 172.21.15.23:0/2702631501 wait complete. 2023-06-22T08:13:42.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.223+0000 7f00223ff700 1 Processor -- start 2023-06-22T08:13:42.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.223+0000 7f00223ff700 1 -- start start 2023-06-22T08:13:42.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f00223ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c06d050 0x7f001c143440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:42.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f00223ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f001c0c3a40 0x7f001c1ee100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:42.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f00223ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f001c15a320 0x7f001c155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:42.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f00223ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c14ec30 con 0x7f001c15a320 2023-06-22T08:13:42.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f00223ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c14cc40 con 0x7f001c06d050 2023-06-22T08:13:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f00223ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001c143980 con 0x7f001c0c3a40 2023-06-22T08:13:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f001bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c06d050 0x7f001c143440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f001bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c06d050 0x7f001c143440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:46454/0 (socket says 172.21.15.23:46454) 2023-06-22T08:13:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f001bfff700 1 -- 172.21.15.23:0/3765551229 learned_addr learned my addr 172.21.15.23:0/3765551229 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f001b7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f001c0c3a40 0x7f001c1ee100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.224+0000 7f002099c700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f001c15a320 0x7f001c155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f001bfff700 1 -- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f001c0c3a40 msgr2=0x7f001c1ee100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f001bfff700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f001c0c3a40 0x7f001c1ee100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f001bfff700 1 -- 172.21.15.23:0/3765551229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f001c15a320 msgr2=0x7f001c155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f001bfff700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f001c15a320 0x7f001c155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f001bfff700 1 -- 172.21.15.23:0/3765551229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c0094a0 con 0x7f001c06d050 2023-06-22T08:13:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f002099c700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f001c15a320 0x7f001c155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:13:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f001bfff700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c06d050 0x7f001c143440 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f001c06b050 tx=0x7f000c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f0018ff9700 1 -- 172.21.15.23:0/3765551229 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000c01a810 con 0x7f001c06d050 2023-06-22T08:13:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f0018ff9700 1 -- 172.21.15.23:0/3765551229 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000c01a970 con 0x7f001c06d050 2023-06-22T08:13:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f00223ff700 1 -- 172.21.15.23:0/3765551229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001c143ed0 con 0x7f001c06d050 2023-06-22T08:13:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f0018ff9700 1 -- 172.21.15.23:0/3765551229 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000c01ace0 con 0x7f001c06d050 2023-06-22T08:13:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.225+0000 7f00223ff700 1 -- 172.21.15.23:0/3765551229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001c14ffb0 con 0x7f001c06d050 2023-06-22T08:13:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.226+0000 7f00223ff700 1 -- 172.21.15.23:0/3765551229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001c04ea20 con 0x7f001c06d050 2023-06-22T08:13:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.227+0000 7f0018ff9700 1 -- 172.21.15.23:0/3765551229 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f000c022070 con 0x7f001c06d050 2023-06-22T08:13:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.229+0000 7f0018ff9700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f000806ace0 0x7f000806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.229+0000 7f0018ff9700 1 -- 172.21.15.23:0/3765551229 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f000c08a9f0 con 0x7f001c06d050 2023-06-22T08:13:42.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.229+0000 7f001b7fe700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f000806ace0 0x7f000806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:42.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.230+0000 7f001b7fe700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f000806ace0 0x7f000806d190 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0010007b10 tx=0x7f0010007a60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:42.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.233+0000 7f0018ff9700 1 -- 172.21.15.23:0/3765551229 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f000c055680 con 0x7f001c06d050 2023-06-22T08:13:42.504 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.501+0000 7f00223ff700 1 -- 172.21.15.23:0/3765551229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f001c143ed0 con 0x7f001c06d050 2023-06-22T08:13:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.502+0000 7f0018ff9700 1 -- 172.21.15.23:0/3765551229 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5515 (secure 0 0 0) 0x7f001c143ed0 con 0x7f001c06d050 2023-06-22T08:13:42.505 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:42.506 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:34.929040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.504+0000 7f00223ff700 1 -- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f000806ace0 msgr2=0x7f000806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.504+0000 7f00223ff700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f000806ace0 0x7f000806d190 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0010007b10 tx=0x7f0010007a60 comp rx=0 tx=0).stop 2023-06-22T08:13:42.508 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.504+0000 7f00223ff700 1 -- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c06d050 msgr2=0x7f001c143440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.504+0000 7f00223ff700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c06d050 0x7f001c143440 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f001c06b050 tx=0x7f000c004750 comp rx=0 tx=0).stop 2023-06-22T08:13:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.504+0000 7f00223ff700 1 -- 172.21.15.23:0/3765551229 shutdown_connections 2023-06-22T08:13:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.505+0000 7f00223ff700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f001c15a320 0x7f001c155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.505+0000 7f00223ff700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f000806ace0 0x7f000806d190 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.505+0000 7f00223ff700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f001c06d050 0x7f001c143440 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.505+0000 7f00223ff700 1 --2- 172.21.15.23:0/3765551229 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f001c0c3a40 0x7f001c1ee100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.505+0000 7f00223ff700 1 -- 172.21.15.23:0/3765551229 >> 172.21.15.23:0/3765551229 conn(0x7f001c0ca3f0 msgr2=0x7f001c0ca7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.505+0000 7f00223ff700 1 -- 172.21.15.23:0/3765551229 shutdown_connections 2023-06-22T08:13:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:42.505+0000 7f00223ff700 1 -- 172.21.15.23:0/3765551229 wait complete. 2023-06-22T08:13:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 105 2023-06-22T08:13:42.526 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-06-22T08:13:44.527 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:44.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.740+0000 7fb3d5b9d700 1 Processor -- start 2023-06-22T08:13:44.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.740+0000 7fb3d5b9d700 1 -- start start 2023-06-22T08:13:44.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.741+0000 7fb3d5b9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb3d00c9890 0x7fb3d006bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:44.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.741+0000 7fb3d5b9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d00c9c60 0x7fb3d0158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:44.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.741+0000 7fb3d5b9d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00c8f80 con 0x7fb3d006ae60 2023-06-22T08:13:44.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.741+0000 7fb3d5b9d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d00c94b0 con 0x7fb3d00c9890 2023-06-22T08:13:44.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.741+0000 7fb3d5b9d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d00b4620 con 0x7fb3d00c9c60 2023-06-22T08:13:44.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.742+0000 7fb3cffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d00c9c60 0x7fb3d0158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:44.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.742+0000 7fb3d539c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb3d006ae60 0x7fb3d015b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:43148/0 (socket says 172.21.15.23:43148) 2023-06-22T08:13:44.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.742+0000 7fb3d4b9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb3d00c9890 0x7fb3d006bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:44.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.742+0000 7fb3d539c700 1 -- 172.21.15.23:0/1418749487 learned_addr learned my addr 172.21.15.23:0/1418749487 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:44.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.742+0000 7fb3d4b9b700 1 -- 172.21.15.23:0/1418749487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d00c9c60 msgr2=0x7fb3d0158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:44.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.742+0000 7fb3d4b9b700 1 --2- 172.21.15.23:0/1418749487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d00c9c60 0x7fb3d0158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:44.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.742+0000 7fb3d4b9b700 1 -- 172.21.15.23:0/1418749487 >> v1:172.21.15.23:6789/0 conn(0x7fb3d006ae60 legacy=0x7fb3d015b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:44.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.742+0000 7fb3d4b9b700 1 -- 172.21.15.23:0/1418749487 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d01593f0 con 0x7fb3d00c9890 2023-06-22T08:13:44.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.742+0000 7fb3cffff700 1 --2- 172.21.15.23:0/1418749487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d00c9c60 0x7fb3d0158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:13:44.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.742+0000 7fb3d4b9b700 1 --2- 172.21.15.23:0/1418749487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d00c9890 0x7fb3d006bc40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009840 tx=0x7fb3c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e247fe08438863 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:44.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3cf7fe700 1 -- 172.21.15.23:0/1418749487 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4011830 con 0x7fb3d00c9890 2023-06-22T08:13:44.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3cf7fe700 1 -- 172.21.15.23:0/1418749487 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c4011990 con 0x7fb3d00c9890 2023-06-22T08:13:44.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3cf7fe700 1 -- 172.21.15.23:0/1418749487 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4011d20 con 0x7fb3d00c9890 2023-06-22T08:13:44.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/1418749487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d00c9890 msgr2=0x7fb3d006bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:44.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3d5b9d700 1 --2- 172.21.15.23:0/1418749487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d00c9890 0x7fb3d006bc40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009840 tx=0x7fb3c4009b50 comp rx=0 tx=0).stop 2023-06-22T08:13:44.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/1418749487 shutdown_connections 2023-06-22T08:13:44.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3d5b9d700 1 --2- 172.21.15.23:0/1418749487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d00c9890 0x7fb3d006bc40 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:44.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3d5b9d700 1 --2- 172.21.15.23:0/1418749487 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d00c9c60 0x7fb3d0158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:44.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/1418749487 >> 172.21.15.23:0/1418749487 conn(0x7fb3d0065b90 msgr2=0x7fb3d0065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:44.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/1418749487 shutdown_connections 2023-06-22T08:13:44.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.743+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/1418749487 wait complete. 2023-06-22T08:13:44.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.745+0000 7fb3d5b9d700 1 Processor -- start 2023-06-22T08:13:44.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.745+0000 7fb3d5b9d700 1 -- start start 2023-06-22T08:13:44.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3d5b9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d006ae60 0x7fb3d0153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:44.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3d5b9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d00c9890 0x7fb3d01ef8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:44.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3d5b9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d00c9c60 0x7fb3d01efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:44.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3d5b9d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d01ef4a0 con 0x7fb3d00c9890 2023-06-22T08:13:44.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3d5b9d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d00a9e60 con 0x7fb3d00c9c60 2023-06-22T08:13:44.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3d5b9d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d01f1ed0 con 0x7fb3d006ae60 2023-06-22T08:13:44.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3cffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d00c9890 0x7fb3d01ef8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:44.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3d4b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d006ae60 0x7fb3d0153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:44.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3d539c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d00c9c60 0x7fb3d01efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:44.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3d4b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d006ae60 0x7fb3d0153410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49662/0 (socket says 172.21.15.23:49662) 2023-06-22T08:13:44.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.746+0000 7fb3d4b9b700 1 -- 172.21.15.23:0/2087047319 learned_addr learned my addr 172.21.15.23:0/2087047319 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:44.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3d539c700 1 -- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d006ae60 msgr2=0x7fb3d0153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:44.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3d539c700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d006ae60 0x7fb3d0153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:44.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3d539c700 1 -- 172.21.15.23:0/2087047319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d00c9890 msgr2=0x7fb3d01ef8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:44.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3d539c700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d00c9890 0x7fb3d01ef8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:44.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3d539c700 1 -- 172.21.15.23:0/2087047319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc0094a0 con 0x7fb3d00c9c60 2023-06-22T08:13:44.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3d4b9b700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d006ae60 0x7fb3d0153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:13:44.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3cffff700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d00c9890 0x7fb3d01ef8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:13:44.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3d539c700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d00c9c60 0x7fb3d01efe00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb3c800caa0 tx=0x7fb3c800cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:44.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3cd7fa700 1 -- 172.21.15.23:0/2087047319 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c800a9a0 con 0x7fb3d00c9c60 2023-06-22T08:13:44.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/2087047319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c40094a0 con 0x7fb3d00c9c60 2023-06-22T08:13:44.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3cd7fa700 1 -- 172.21.15.23:0/2087047319 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c800ab00 con 0x7fb3d00c9c60 2023-06-22T08:13:44.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3cd7fa700 1 -- 172.21.15.23:0/2087047319 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c800ae70 con 0x7fb3d00c9c60 2023-06-22T08:13:44.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.747+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/2087047319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d00cacc0 con 0x7fb3d00c9c60 2023-06-22T08:13:44.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.749+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/2087047319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d01ef5e0 con 0x7fb3d00c9c60 2023-06-22T08:13:44.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.751+0000 7fb3cd7fa700 1 -- 172.21.15.23:0/2087047319 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb3c8012400 con 0x7fb3d00c9c60 2023-06-22T08:13:44.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.752+0000 7fb3cd7fa700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb3b406ace0 0x7fb3b406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:44.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.752+0000 7fb3cd7fa700 1 -- 172.21.15.23:0/2087047319 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7fb3c80894b0 con 0x7fb3d00c9c60 2023-06-22T08:13:44.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.752+0000 7fb3d4b9b700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb3b406ace0 0x7fb3b406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:44.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.753+0000 7fb3d4b9b700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb3b406ace0 0x7fb3b406d190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009b90 tx=0x7fb3c4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:44.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:44.756+0000 7fb3cd7fa700 1 -- 172.21.15.23:0/2087047319 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb3d01ef5e0 con 0x7fb3d00c9c60 2023-06-22T08:13:45.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.093+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/2087047319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3d01ef5e0 con 0x7fb3d00c9c60 2023-06-22T08:13:45.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.094+0000 7fb3cd7fa700 1 -- 172.21.15.23:0/2087047319 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+6272 (secure 0 0 0) 0x7fb3d01ef5e0 con 0x7fb3d00c9c60 2023-06-22T08:13:45.098 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:45.098 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:34.929040+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:45.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb3b406ace0 msgr2=0x7fb3b406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:45.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb3b406ace0 0x7fb3b406d190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009b90 tx=0x7fb3c4008030 comp rx=0 tx=0).stop 2023-06-22T08:13:45.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d00c9c60 msgr2=0x7fb3d01efe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:45.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d00c9c60 0x7fb3d01efe00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb3c800caa0 tx=0x7fb3c800cdb0 comp rx=0 tx=0).stop 2023-06-22T08:13:45.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/2087047319 shutdown_connections 2023-06-22T08:13:45.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb3b406ace0 0x7fb3b406d190 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:45.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d00c9c60 0x7fb3d01efe00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:45.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb3d006ae60 0x7fb3d0153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:45.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 --2- 172.21.15.23:0/2087047319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d00c9890 0x7fb3d01ef8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:45.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/2087047319 >> 172.21.15.23:0/2087047319 conn(0x7fb3d0065b90 msgr2=0x7fb3d0157180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:45.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.096+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/2087047319 shutdown_connections 2023-06-22T08:13:45.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:45.097+0000 7fb3d5b9d700 1 -- 172.21.15.23:0/2087047319 wait complete. 2023-06-22T08:13:45.104 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 106 2023-06-22T08:13:45.106 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 56% (171/304) Updating files: 57% (174/304) Updating files: 58% (177/304) Updating files: 59% (180/304) Updating files: 60% (183/304) Updating files: 61% (186/304) Updating files: 62% (189/304) Updating files: 63% (192/304) Updating files: 64% (195/304) Updating files: 65% (198/304) Updating files: 66% (201/304) Updating files: 67% (204/304) Updating files: 68% (207/304) Updating files: 69% (210/304) Updating files: 70% (213/304) Updating files: 71% (216/304) Updating files: 72% (219/304) Updating files: 73% (222/304) Updating files: 74% (225/304) Updating files: 75% (228/304) Updating files: 76% (232/304) Updating files: 77% (235/304) Updating files: 78% (238/304) Updating files: 79% (241/304) Updating files: 80% (244/304) Updating files: 81% (247/304) Updating files: 82% (250/304) Updating files: 83% (253/304) Updating files: 84% (256/304) Updating files: 85% (259/304) Updating files: 86% (262/304) Updating files: 87% (265/304) Updating files: 88% (268/304) Updating files: 89% (271/304) Updating files: 90% (274/304) Updating files: 91% (277/304) Updating files: 92% (280/304) Updating files: 93% (283/304) Updating files: 94% (286/304) Updating files: 95% (289/304) Updating files: 96% (292/304) Updating files: 97% (295/304) Updating files: 98% (298/304) Updating files: 99% (301/304) Updating files: 100% (304/304) Updating files: 100% (304/304), done. 2023-06-22T08:13:45.122 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-06-22T08:13:45.123 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before thrashing 2023-06-22T08:13:45.274 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 9a7a9d06c06 v0.57 2023-06-22T08:13:45.283 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.57 2023-06-22T08:13:45.387 INFO:tasks.workunit.client.0.smithi023.stdout:v0.58 2023-06-22T08:13:45.388 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:13:45.388 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 58 -eq 48 ']' 2023-06-22T08:13:45.389 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.58 2023-06-22T08:13:45.389 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.58 2023-06-22T08:13:45.389 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.58 2023-06-22T08:13:46.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:46.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.360+0000 7f618f9ff700 1 Processor -- start 2023-06-22T08:13:46.364 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.360+0000 7f618f9ff700 1 -- start start 2023-06-22T08:13:46.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.361+0000 7f618f9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6188152d70 0x7f61880c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:46.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.361+0000 7f618f9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188159330 0x7f61880cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:46.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618f9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618815bea0 con 0x7f61880cb800 2023-06-22T08:13:46.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618f9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61880c95e0 con 0x7f6188152d70 2023-06-22T08:13:46.365 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618f9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618815c160 con 0x7f6188159330 2023-06-22T08:13:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618d79b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6188152d70 0x7f61880c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618cf9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188159330 0x7f61880cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618d79b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6188152d70 0x7f61880c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:60274/0 (socket says 172.21.15.23:60274) 2023-06-22T08:13:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618d79b700 1 -- 172.21.15.23:0/3946138120 learned_addr learned my addr 172.21.15.23:0/3946138120 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618d79b700 1 -- 172.21.15.23:0/3946138120 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188159330 msgr2=0x7f61880cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618d79b700 1 --2- 172.21.15.23:0/3946138120 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188159330 0x7f61880cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:46.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618d79b700 1 -- 172.21.15.23:0/3946138120 >> v1:172.21.15.23:6789/0 conn(0x7f61880cb800 legacy=0x7f61880c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:46.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.362+0000 7f618d79b700 1 -- 172.21.15.23:0/3946138120 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618815c440 con 0x7f6188152d70 2023-06-22T08:13:46.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.363+0000 7f618d79b700 1 --2- 172.21.15.23:0/3946138120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188152d70 0x7f61880c97d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f61840098f0 tx=0x7f6184009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b15327c24f64c8ce server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:46.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.363+0000 7f617ffff700 1 -- 172.21.15.23:0/3946138120 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184011830 con 0x7f6188152d70 2023-06-22T08:13:46.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.363+0000 7f617ffff700 1 -- 172.21.15.23:0/3946138120 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6184011990 con 0x7f6188152d70 2023-06-22T08:13:46.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.363+0000 7f617ffff700 1 -- 172.21.15.23:0/3946138120 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184011d20 con 0x7f6188152d70 2023-06-22T08:13:46.368 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.363+0000 7f618f9ff700 1 -- 172.21.15.23:0/3946138120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188152d70 msgr2=0x7f61880c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:46.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.363+0000 7f618f9ff700 1 --2- 172.21.15.23:0/3946138120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188152d70 0x7f61880c97d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f61840098f0 tx=0x7f6184009c00 comp rx=0 tx=0).stop 2023-06-22T08:13:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.364+0000 7f618f9ff700 1 -- 172.21.15.23:0/3946138120 shutdown_connections 2023-06-22T08:13:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.364+0000 7f618f9ff700 1 --2- 172.21.15.23:0/3946138120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6188152d70 0x7f61880c97d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.364+0000 7f618f9ff700 1 --2- 172.21.15.23:0/3946138120 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188159330 0x7f61880cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.364+0000 7f618f9ff700 1 -- 172.21.15.23:0/3946138120 >> 172.21.15.23:0/3946138120 conn(0x7f618806ae30 msgr2=0x7f618806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.364+0000 7f618f9ff700 1 -- 172.21.15.23:0/3946138120 shutdown_connections 2023-06-22T08:13:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.364+0000 7f618f9ff700 1 -- 172.21.15.23:0/3946138120 wait complete. 2023-06-22T08:13:46.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.366+0000 7f618f9ff700 1 Processor -- start 2023-06-22T08:13:46.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.366+0000 7f618f9ff700 1 -- start start 2023-06-22T08:13:46.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.367+0000 7f618f9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61880cb800 0x7f61880ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:46.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.367+0000 7f618f9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188152d70 0x7f6188158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.367+0000 7f618f9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6188159330 0x7f6188148760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.367+0000 7f618f9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6188150fc0 con 0x7f6188159330 2023-06-22T08:13:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.367+0000 7f618f9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61880c89f0 con 0x7f61880cb800 2023-06-22T08:13:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.367+0000 7f618f9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61881eeef0 con 0x7f6188152d70 2023-06-22T08:13:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618df9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6188159330 0x7f6188148760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618cf9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188152d70 0x7f6188158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618d79b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61880cb800 0x7f61880ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618cf9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188152d70 0x7f6188158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49100/0 (socket says 172.21.15.23:49100) 2023-06-22T08:13:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618cf9a700 1 -- 172.21.15.23:0/594947788 learned_addr learned my addr 172.21.15.23:0/594947788 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618cf9a700 1 -- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61880cb800 msgr2=0x7f61880ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618cf9a700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61880cb800 0x7f61880ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618cf9a700 1 -- 172.21.15.23:0/594947788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6188159330 msgr2=0x7f6188148760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618cf9a700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6188159330 0x7f6188148760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618cf9a700 1 -- 172.21.15.23:0/594947788 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6184009550 con 0x7f6188152d70 2023-06-22T08:13:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.368+0000 7f618d79b700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61880cb800 0x7f61880ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:13:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.369+0000 7f618cf9a700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188152d70 0x7f6188158460 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f617800e870 tx=0x7f617800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.369+0000 7f617dffb700 1 -- 172.21.15.23:0/594947788 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178009590 con 0x7f6188152d70 2023-06-22T08:13:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.369+0000 7f617dffb700 1 -- 172.21.15.23:0/594947788 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6178013680 con 0x7f6188152d70 2023-06-22T08:13:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.369+0000 7f617dffb700 1 -- 172.21.15.23:0/594947788 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61780139f0 con 0x7f6188152d70 2023-06-22T08:13:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.369+0000 7f618f9ff700 1 -- 172.21.15.23:0/594947788 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61881ef1d0 con 0x7f6188152d70 2023-06-22T08:13:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.369+0000 7f618f9ff700 1 -- 172.21.15.23:0/594947788 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61881514f0 con 0x7f6188152d70 2023-06-22T08:13:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.370+0000 7f618f9ff700 1 -- 172.21.15.23:0/594947788 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61880cbd50 con 0x7f6188152d70 2023-06-22T08:13:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.372+0000 7f617dffb700 1 -- 172.21.15.23:0/594947788 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6178013b50 con 0x7f6188152d70 2023-06-22T08:13:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.373+0000 7f617dffb700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f616c06ad10 0x7f616c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.373+0000 7f617dffb700 1 -- 172.21.15.23:0/594947788 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f6178089800 con 0x7f6188152d70 2023-06-22T08:13:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.373+0000 7f618d79b700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f616c06ad10 0x7f616c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.374+0000 7f618d79b700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f616c06ad10 0x7f616c06d1c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6184009c40 tx=0x7f6184008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.378+0000 7f617dffb700 1 -- 172.21.15.23:0/594947788 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6178054360 con 0x7f6188152d70 2023-06-22T08:13:46.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.707+0000 7f618f9ff700 1 -- 172.21.15.23:0/594947788 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f618804ea20 con 0x7f6188152d70 2023-06-22T08:13:46.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.708+0000 7f617dffb700 1 -- 172.21.15.23:0/594947788 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6273 (secure 0 0 0) 0x7f6178057980 con 0x7f6188152d70 2023-06-22T08:13:46.712 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:46.712 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:45.483544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:46.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.711+0000 7f618f9ff700 1 -- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f616c06ad10 msgr2=0x7f616c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:46.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.711+0000 7f618f9ff700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f616c06ad10 0x7f616c06d1c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6184009c40 tx=0x7f6184008030 comp rx=0 tx=0).stop 2023-06-22T08:13:46.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.711+0000 7f618f9ff700 1 -- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188152d70 msgr2=0x7f6188158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:46.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.711+0000 7f618f9ff700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188152d70 0x7f6188158460 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f617800e870 tx=0x7f617800eb80 comp rx=0 tx=0).stop 2023-06-22T08:13:46.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.711+0000 7f618f9ff700 1 -- 172.21.15.23:0/594947788 shutdown_connections 2023-06-22T08:13:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.711+0000 7f618f9ff700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6188159330 0x7f6188148760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.711+0000 7f618f9ff700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f616c06ad10 0x7f616c06d1c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.711+0000 7f618f9ff700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61880cb800 0x7f61880ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.711+0000 7f618f9ff700 1 --2- 172.21.15.23:0/594947788 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6188152d70 0x7f6188158460 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.711+0000 7f618f9ff700 1 -- 172.21.15.23:0/594947788 >> 172.21.15.23:0/594947788 conn(0x7f618806ae30 msgr2=0x7f618814e650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.712+0000 7f618f9ff700 1 -- 172.21.15.23:0/594947788 shutdown_connections 2023-06-22T08:13:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:46.712+0000 7f618f9ff700 1 -- 172.21.15.23:0/594947788 wait complete. 2023-06-22T08:13:46.717 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 107 2023-06-22T08:13:46.738 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=4) 2023-06-22T08:13:46.739 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-06-22T08:13:46.739 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:13:46.765 INFO:tasks.ceph.mds.e:Stopped 2023-06-22T08:13:46.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.030+0000 7f6fe37ff700 1 Processor -- start 2023-06-22T08:13:47.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.031+0000 7f6fe37ff700 1 -- start start 2023-06-22T08:13:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.032+0000 7f6fe37ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fdc152ce0 0x7f6fdc0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.032+0000 7f6fe37ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6fdc0c9fd0 0x7f6fdc15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.032+0000 7f6fe37ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fdc15c370 con 0x7f6fdc158880 2023-06-22T08:13:47.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.032+0000 7f6fe37ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fdc1530b0 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.032+0000 7f6fe37ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc0cf6f0 con 0x7f6fdc152ce0 2023-06-22T08:13:47.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.032+0000 7f6fe0d9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6fdc0c9fd0 0x7f6fdc15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:47.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.032+0000 7f6fe1d9c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6fdc158880 0x7f6fdc158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:54660/0 (socket says 172.21.15.23:54660) 2023-06-22T08:13:47.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.032+0000 7f6fe159b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fdc152ce0 0x7f6fdc0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:47.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.032+0000 7f6fe1d9c700 1 -- 172.21.15.23:0/785560266 learned_addr learned my addr 172.21.15.23:0/785560266 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:47.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.032+0000 7f6fe0d9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6fdc0c9fd0 0x7f6fdc15c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36058/0 (socket says 172.21.15.23:36058) 2023-06-22T08:13:47.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.033+0000 7f6fe0d9a700 1 -- 172.21.15.23:0/785560266 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fdc152ce0 msgr2=0x7f6fdc0cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:47.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.033+0000 7f6fe0d9a700 1 --2- 172.21.15.23:0/785560266 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fdc152ce0 0x7f6fdc0cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:47.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.033+0000 7f6fe0d9a700 1 -- 172.21.15.23:0/785560266 >> v1:172.21.15.67:6789/0 conn(0x7f6fdc158880 legacy=0x7f6fdc158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:47.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.033+0000 7f6fe0d9a700 1 -- 172.21.15.23:0/785560266 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc15bda0 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.033+0000 7f6fd3fff700 1 -- 172.21.15.23:0/785560266 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507164085 0 0) 0x7f6fdc15c370 con 0x7f6fdc158880 2023-06-22T08:13:47.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.033+0000 7f6fe0d9a700 1 --2- 172.21.15.23:0/785560266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 0x7f6fdc15c800 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009840 tx=0x7f6fcc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e34e121cab810b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:47.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.033+0000 7f6fd3fff700 1 -- 172.21.15.23:0/785560266 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc011830 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.033+0000 7f6fd3fff700 1 -- 172.21.15.23:0/785560266 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc011990 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.033+0000 7f6fd3fff700 1 -- 172.21.15.23:0/785560266 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc011d20 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.034+0000 7f6fe37ff700 1 -- 172.21.15.23:0/785560266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 msgr2=0x7f6fdc15c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:47.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.034+0000 7f6fe37ff700 1 --2- 172.21.15.23:0/785560266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 0x7f6fdc15c800 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009840 tx=0x7f6fcc009b50 comp rx=0 tx=0).stop 2023-06-22T08:13:47.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.034+0000 7f6fe37ff700 1 -- 172.21.15.23:0/785560266 shutdown_connections 2023-06-22T08:13:47.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.034+0000 7f6fe37ff700 1 --2- 172.21.15.23:0/785560266 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fdc152ce0 0x7f6fdc0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:47.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.034+0000 7f6fe37ff700 1 --2- 172.21.15.23:0/785560266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 0x7f6fdc15c800 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:47.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.034+0000 7f6fe37ff700 1 -- 172.21.15.23:0/785560266 >> 172.21.15.23:0/785560266 conn(0x7f6fdc06ae30 msgr2=0x7f6fdc06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:47.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.034+0000 7f6fe37ff700 1 -- 172.21.15.23:0/785560266 shutdown_connections 2023-06-22T08:13:47.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.034+0000 7f6fe37ff700 1 -- 172.21.15.23:0/785560266 wait complete. 2023-06-22T08:13:47.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.035+0000 7f6fe37ff700 1 Processor -- start 2023-06-22T08:13:47.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.036+0000 7f6fe37ff700 1 -- start start 2023-06-22T08:13:47.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.036+0000 7f6fe37ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 0x7f6fdc1e20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:47.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.036+0000 7f6fe37ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fdc152ce0 0x7f6fdc1ef6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:47.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe37ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fdc158880 0x7f6fdc1efc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:47.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe37ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc1e4470 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe37ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc1e37f0 con 0x7f6fdc158880 2023-06-22T08:13:47.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe37ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc1e3ad0 con 0x7f6fdc152ce0 2023-06-22T08:13:47.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe159b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 0x7f6fdc1e20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:47.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe159b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 0x7f6fdc1e20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36064/0 (socket says 172.21.15.23:36064) 2023-06-22T08:13:47.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe159b700 1 -- 172.21.15.23:0/2397707144 learned_addr learned my addr 172.21.15.23:0/2397707144 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:47.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe0d9a700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fdc152ce0 0x7f6fdc1ef6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:47.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe1d9c700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fdc158880 0x7f6fdc1efc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe159b700 1 -- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fdc152ce0 msgr2=0x7f6fdc1ef6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe159b700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fdc152ce0 0x7f6fdc1ef6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe159b700 1 -- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fdc158880 msgr2=0x7f6fdc1efc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe159b700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fdc158880 0x7f6fdc1efc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe159b700 1 -- 172.21.15.23:0/2397707144 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc0094a0 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.037+0000 7f6fe159b700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 0x7f6fdc1e20b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800e9b0 tx=0x7f6fd800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.038+0000 7f6fd1ffb700 1 -- 172.21.15.23:0/2397707144 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8009710 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.038+0000 7f6fd1ffb700 1 -- 172.21.15.23:0/2397707144 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd800ee50 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.038+0000 7f6fe37ff700 1 -- 172.21.15.23:0/2397707144 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc1e3db0 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.038+0000 7f6fd1ffb700 1 -- 172.21.15.23:0/2397707144 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd800ca50 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.038+0000 7f6fe37ff700 1 -- 172.21.15.23:0/2397707144 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc1e34e0 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.039+0000 7f6fe37ff700 1 -- 172.21.15.23:0/2397707144 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fdc1586a0 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.040+0000 7f6fd1ffb700 1 -- 172.21.15.23:0/2397707144 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6fd800cbb0 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.041+0000 7f6fd1ffb700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fc006ad10 0x7f6fc006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:47.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.041+0000 7f6fd1ffb700 1 -- 172.21.15.23:0/2397707144 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f6fd80899f0 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.041+0000 7f6fe0d9a700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fc006ad10 0x7f6fc006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:47.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.042+0000 7f6fe0d9a700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fc006ad10 0x7f6fc006d1c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0090b0 tx=0x7f6fcc01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:47.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.046+0000 7f6fd1ffb700 1 -- 172.21.15.23:0/2397707144 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6fd8057f50 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.380+0000 7f6fe37ff700 1 -- 172.21.15.23:0/2397707144 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fdc04ea20 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.380+0000 7f6fd1ffb700 1 -- 172.21.15.23:0/2397707144 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6273 (secure 0 0 0) 0x7f6fd8017090 con 0x7f6fdc0c9fd0 2023-06-22T08:13:47.384 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:47.384 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:45.483544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.382+0000 7f6fe37ff700 1 -- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fc006ad10 msgr2=0x7f6fc006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.382+0000 7f6fe37ff700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fc006ad10 0x7f6fc006d1c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0090b0 tx=0x7f6fcc01c040 comp rx=0 tx=0).stop 2023-06-22T08:13:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.382+0000 7f6fe37ff700 1 -- 172.21.15.23:0/2397707144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 msgr2=0x7f6fdc1e20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.382+0000 7f6fe37ff700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 0x7f6fdc1e20b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800e9b0 tx=0x7f6fd800ecc0 comp rx=0 tx=0).stop 2023-06-22T08:13:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.382+0000 7f6fe37ff700 1 -- 172.21.15.23:0/2397707144 shutdown_connections 2023-06-22T08:13:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.382+0000 7f6fe37ff700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fc006ad10 0x7f6fc006d1c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.382+0000 7f6fe37ff700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fdc158880 0x7f6fdc1efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.382+0000 7f6fe37ff700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fdc0c9fd0 0x7f6fdc1e20b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.382+0000 7f6fe37ff700 1 --2- 172.21.15.23:0/2397707144 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fdc152ce0 0x7f6fdc1ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.382+0000 7f6fe37ff700 1 -- 172.21.15.23:0/2397707144 >> 172.21.15.23:0/2397707144 conn(0x7f6fdc06ae30 msgr2=0x7f6fdc1564e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.383+0000 7f6fe37ff700 1 -- 172.21.15.23:0/2397707144 shutdown_connections 2023-06-22T08:13:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:47.383+0000 7f6fe37ff700 1 -- 172.21.15.23:0/2397707144 wait complete. 2023-06-22T08:13:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 107 2023-06-22T08:13:47.404 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:13:49.405 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:49.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.613+0000 7f41735ff700 1 Processor -- start 2023-06-22T08:13:49.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.613+0000 7f41735ff700 1 -- start start 2023-06-22T08:13:49.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f41735ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f416c0c9fd0 0x7f416c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:49.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f41735ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f416c0c40b0 0x7f416c0c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:49.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f41735ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f416c15c370 con 0x7f416c152ce0 2023-06-22T08:13:49.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f41735ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416c1530b0 con 0x7f416c0c40b0 2023-06-22T08:13:49.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f41735ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416c0cf6f0 con 0x7f416c0c9fd0 2023-06-22T08:13:49.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f4171b9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f416c0c40b0 0x7f416c0c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:49.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f417139b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f416c152ce0 0x7f416c15c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57864/0 (socket says 172.21.15.23:57864) 2023-06-22T08:13:49.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f4171b9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f416c0c40b0 0x7f416c0c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36068/0 (socket says 172.21.15.23:36068) 2023-06-22T08:13:49.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f417139b700 1 -- 172.21.15.23:0/657570334 learned_addr learned my addr 172.21.15.23:0/657570334 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:49.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f4170b9a700 1 --2- 172.21.15.23:0/657570334 >> v2:172.21.15.67:3300/0 conn(0x7f416c0c9fd0 0x7f416c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:49.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f4171b9c700 1 -- 172.21.15.23:0/657570334 >> v2:172.21.15.67:3300/0 conn(0x7f416c0c9fd0 msgr2=0x7f416c0cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:49.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f4171b9c700 1 --2- 172.21.15.23:0/657570334 >> v2:172.21.15.67:3300/0 conn(0x7f416c0c9fd0 0x7f416c0cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:49.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f4171b9c700 1 -- 172.21.15.23:0/657570334 >> v1:172.21.15.23:6789/0 conn(0x7f416c152ce0 legacy=0x7f416c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:49.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f4171b9c700 1 -- 172.21.15.23:0/657570334 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c15bda0 con 0x7f416c0c40b0 2023-06-22T08:13:49.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.615+0000 7f4163fff700 1 -- 172.21.15.23:0/657570334 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551078114 0 0) 0x7f416c15c370 con 0x7f416c152ce0 2023-06-22T08:13:49.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.616+0000 7f4171b9c700 1 --2- 172.21.15.23:0/657570334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 0x7f416c0c4530 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f416c0cc380 tx=0x7f416400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca527590ae3c003 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:49.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.616+0000 7f4163fff700 1 -- 172.21.15.23:0/657570334 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416400e070 con 0x7f416c0c40b0 2023-06-22T08:13:49.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.616+0000 7f4163fff700 1 -- 172.21.15.23:0/657570334 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416400bb20 con 0x7f416c0c40b0 2023-06-22T08:13:49.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.616+0000 7f4163fff700 1 -- 172.21.15.23:0/657570334 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4164012430 con 0x7f416c0c40b0 2023-06-22T08:13:49.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.616+0000 7f41735ff700 1 -- 172.21.15.23:0/657570334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 msgr2=0x7f416c0c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:49.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.616+0000 7f41735ff700 1 --2- 172.21.15.23:0/657570334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 0x7f416c0c4530 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f416c0cc380 tx=0x7f416400b330 comp rx=0 tx=0).stop 2023-06-22T08:13:49.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.617+0000 7f41735ff700 1 -- 172.21.15.23:0/657570334 shutdown_connections 2023-06-22T08:13:49.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.617+0000 7f41735ff700 1 --2- 172.21.15.23:0/657570334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 0x7f416c0c4530 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:49.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.617+0000 7f41735ff700 1 --2- 172.21.15.23:0/657570334 >> v2:172.21.15.67:3300/0 conn(0x7f416c0c9fd0 0x7f416c0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:49.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.617+0000 7f41735ff700 1 -- 172.21.15.23:0/657570334 >> 172.21.15.23:0/657570334 conn(0x7f416c06ae30 msgr2=0x7f416c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:49.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.617+0000 7f41735ff700 1 -- 172.21.15.23:0/657570334 shutdown_connections 2023-06-22T08:13:49.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.617+0000 7f41735ff700 1 -- 172.21.15.23:0/657570334 wait complete. 2023-06-22T08:13:49.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.618+0000 7f41735ff700 1 Processor -- start 2023-06-22T08:13:49.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.619+0000 7f41735ff700 1 -- start start 2023-06-22T08:13:49.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.619+0000 7f41735ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 0x7f416c1ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:49.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.619+0000 7f41735ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f416c0c9fd0 0x7f416c155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:49.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.619+0000 7f41735ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f416c152ce0 0x7f416c0b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:49.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.619+0000 7f41735ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c1eddb0 con 0x7f416c0c40b0 2023-06-22T08:13:49.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.619+0000 7f41735ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c1f1120 con 0x7f416c152ce0 2023-06-22T08:13:49.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.619+0000 7f41735ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f416c0ccef0 con 0x7f416c0c9fd0 2023-06-22T08:13:49.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f417139b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 0x7f416c1ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:49.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f417139b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 0x7f416c1ee7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36072/0 (socket says 172.21.15.23:36072) 2023-06-22T08:13:49.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f417139b700 1 -- 172.21.15.23:0/2813072587 learned_addr learned my addr 172.21.15.23:0/2813072587 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:49.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f4171b9c700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f416c152ce0 0x7f416c0b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:49.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f4170b9a700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f416c0c9fd0 0x7f416c155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:49.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f417139b700 1 -- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f416c0c9fd0 msgr2=0x7f416c155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:49.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f417139b700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f416c0c9fd0 0x7f416c155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:49.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f417139b700 1 -- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f416c152ce0 msgr2=0x7f416c0b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:49.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f417139b700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f416c152ce0 0x7f416c0b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:49.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f417139b700 1 -- 172.21.15.23:0/2813072587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4164009a70 con 0x7f416c0c40b0 2023-06-22T08:13:49.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.620+0000 7f417139b700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 0x7f416c1ee7c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f416800c550 tx=0x7f416800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:49.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.621+0000 7f4161ffb700 1 -- 172.21.15.23:0/2813072587 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4168004c80 con 0x7f416c0c40b0 2023-06-22T08:13:49.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.621+0000 7f4161ffb700 1 -- 172.21.15.23:0/2813072587 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4168004e00 con 0x7f416c0c40b0 2023-06-22T08:13:49.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.621+0000 7f4161ffb700 1 -- 172.21.15.23:0/2813072587 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4168007310 con 0x7f416c0c40b0 2023-06-22T08:13:49.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.621+0000 7f41735ff700 1 -- 172.21.15.23:0/2813072587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c0cc9c0 con 0x7f416c0c40b0 2023-06-22T08:13:49.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.621+0000 7f41735ff700 1 -- 172.21.15.23:0/2813072587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c156030 con 0x7f416c0c40b0 2023-06-22T08:13:49.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.622+0000 7f41735ff700 1 -- 172.21.15.23:0/2813072587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c04ea20 con 0x7f416c0c40b0 2023-06-22T08:13:49.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.623+0000 7f4161ffb700 1 -- 172.21.15.23:0/2813072587 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4168007470 con 0x7f416c0c40b0 2023-06-22T08:13:49.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.624+0000 7f4161ffb700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f415006ace0 0x7f415006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:49.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.624+0000 7f4161ffb700 1 -- 172.21.15.23:0/2813072587 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f4168089680 con 0x7f416c0c40b0 2023-06-22T08:13:49.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.625+0000 7f4170b9a700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f415006ace0 0x7f415006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:49.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.625+0000 7f4170b9a700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f415006ace0 0x7f415006d190 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f416c0c5020 tx=0x7f415c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.629+0000 7f4161ffb700 1 -- 172.21.15.23:0/2813072587 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4168054140 con 0x7f416c0c40b0 2023-06-22T08:13:49.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.883+0000 7f41735ff700 1 -- 172.21.15.23:0/2813072587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f416c0b13d0 con 0x7f416c0c40b0 2023-06-22T08:13:49.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.883+0000 7f4161ffb700 1 -- 172.21.15.23:0/2813072587 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+6273 (secure 0 0 0) 0x7f4168057760 con 0x7f416c0c40b0 2023-06-22T08:13:49.889 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:49.889 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:45.483544+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:49.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.885+0000 7f41735ff700 1 -- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f415006ace0 msgr2=0x7f415006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:49.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.885+0000 7f41735ff700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f415006ace0 0x7f415006d190 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f416c0c5020 tx=0x7f415c009210 comp rx=0 tx=0).stop 2023-06-22T08:13:49.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.885+0000 7f41735ff700 1 -- 172.21.15.23:0/2813072587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 msgr2=0x7f416c1ee7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:49.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.885+0000 7f41735ff700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 0x7f416c1ee7c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f416800c550 tx=0x7f416800c860 comp rx=0 tx=0).stop 2023-06-22T08:13:49.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.886+0000 7f41735ff700 1 -- 172.21.15.23:0/2813072587 shutdown_connections 2023-06-22T08:13:49.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.886+0000 7f41735ff700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f415006ace0 0x7f415006d190 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:49.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.886+0000 7f41735ff700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f416c152ce0 0x7f416c0b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:49.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.886+0000 7f41735ff700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f416c0c40b0 0x7f416c1ee7c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:49.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.886+0000 7f41735ff700 1 --2- 172.21.15.23:0/2813072587 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f416c0c9fd0 0x7f416c155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:49.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.886+0000 7f41735ff700 1 -- 172.21.15.23:0/2813072587 >> 172.21.15.23:0/2813072587 conn(0x7f416c06ae30 msgr2=0x7f416c151ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:49.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.886+0000 7f41735ff700 1 -- 172.21.15.23:0/2813072587 shutdown_connections 2023-06-22T08:13:49.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:49.886+0000 7f41735ff700 1 -- 172.21.15.23:0/2813072587 wait complete. 2023-06-22T08:13:49.892 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 107 2023-06-22T08:13:49.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:13:51.908 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:52.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.127+0000 7f4d3599d700 1 Processor -- start 2023-06-22T08:13:52.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.127+0000 7f4d3599d700 1 -- start start 2023-06-22T08:13:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.128+0000 7f4d3599d700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d300c3b30 0x7f4d30156910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.128+0000 7f4d3599d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d3006c300 con 0x7f4d300c3f00 2023-06-22T08:13:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.128+0000 7f4d3599d700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d3015a9d0 con 0x7f4d3006cfb0 2023-06-22T08:13:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.128+0000 7f4d3599d700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3006bd40 con 0x7f4d300c3b30 2023-06-22T08:13:52.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.129+0000 7f4d2ffff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4d300c3f00 0x7f4d30150420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57866/0 (socket says 172.21.15.23:57866) 2023-06-22T08:13:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.129+0000 7f4d2ffff700 1 -- 172.21.15.23:0/2191998240 learned_addr learned my addr 172.21.15.23:0/2191998240 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:52.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.129+0000 7f4d3499b700 1 --2- 172.21.15.23:0/2191998240 >> v2:172.21.15.67:3300/0 conn(0x7f4d300c3b30 0x7f4d30156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.129+0000 7f4d2f7fe700 1 -- 172.21.15.23:0/2191998240 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004339433 0 0) 0x7f4d3006c300 con 0x7f4d300c3f00 2023-06-22T08:13:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.129+0000 7f4d2f7fe700 1 -- 172.21.15.23:0/2191998240 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d200035b0 con 0x7f4d300c3f00 2023-06-22T08:13:52.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.129+0000 7f4d3499b700 1 -- 172.21.15.23:0/2191998240 >> v1:172.21.15.67:6789/0 conn(0x7f4d3006cfb0 legacy=0x7f4d30160f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.129+0000 7f4d3499b700 1 -- 172.21.15.23:0/2191998240 >> v1:172.21.15.23:6789/0 conn(0x7f4d300c3f00 legacy=0x7f4d30150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.129+0000 7f4d3499b700 1 -- 172.21.15.23:0/2191998240 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3006c020 con 0x7f4d300c3b30 2023-06-22T08:13:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.129+0000 7f4d3499b700 1 --2- 172.21.15.23:0/2191998240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3b30 0x7f4d30156910 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009840 tx=0x7f4d24009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d309e1730f4cd6c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.130+0000 7f4d2f7fe700 1 -- 172.21.15.23:0/2191998240 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24011830 con 0x7f4d300c3b30 2023-06-22T08:13:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.130+0000 7f4d2f7fe700 1 -- 172.21.15.23:0/2191998240 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d24011990 con 0x7f4d300c3b30 2023-06-22T08:13:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.130+0000 7f4d2f7fe700 1 -- 172.21.15.23:0/2191998240 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24011d00 con 0x7f4d300c3b30 2023-06-22T08:13:52.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.130+0000 7f4d3599d700 1 -- 172.21.15.23:0/2191998240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3b30 msgr2=0x7f4d30156910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:52.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.130+0000 7f4d3599d700 1 --2- 172.21.15.23:0/2191998240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3b30 0x7f4d30156910 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009840 tx=0x7f4d24009b50 comp rx=0 tx=0).stop 2023-06-22T08:13:52.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.130+0000 7f4d3599d700 1 -- 172.21.15.23:0/2191998240 shutdown_connections 2023-06-22T08:13:52.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.130+0000 7f4d3599d700 1 --2- 172.21.15.23:0/2191998240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3b30 0x7f4d30156910 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:52.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.130+0000 7f4d3599d700 1 -- 172.21.15.23:0/2191998240 >> 172.21.15.23:0/2191998240 conn(0x7f4d3006e3b0 msgr2=0x7f4d30069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:52.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.130+0000 7f4d3599d700 1 -- 172.21.15.23:0/2191998240 shutdown_connections 2023-06-22T08:13:52.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.130+0000 7f4d3599d700 1 -- 172.21.15.23:0/2191998240 wait complete. 2023-06-22T08:13:52.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.132+0000 7f4d3599d700 1 Processor -- start 2023-06-22T08:13:52.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.132+0000 7f4d3599d700 1 -- start start 2023-06-22T08:13:52.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d3599d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d300c3b30 0x7f4d300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:52.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d3599d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3f00 0x7f4d301f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:52.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d3599d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4d301ed950 0x7f4d300c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:52.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d3599d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d300c1d10 con 0x7f4d300c3b30 2023-06-22T08:13:52.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d3599d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d301eddf0 con 0x7f4d300c3f00 2023-06-22T08:13:52.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d3599d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d300c1750 con 0x7f4d301ed950 2023-06-22T08:13:52.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d3499b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d300c3b30 0x7f4d300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:52.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d2ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3f00 0x7f4d301f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:52.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d3499b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d300c3b30 0x7f4d300702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36088/0 (socket says 172.21.15.23:36088) 2023-06-22T08:13:52.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d2ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3f00 0x7f4d301f1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:60322/0 (socket says 172.21.15.23:60322) 2023-06-22T08:13:52.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.133+0000 7f4d3499b700 1 -- 172.21.15.23:0/1734460828 learned_addr learned my addr 172.21.15.23:0/1734460828 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:52.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d3519c700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4d301ed950 0x7f4d300c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:52.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d2ffff700 1 -- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4d301ed950 msgr2=0x7f4d300c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:52.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d2ffff700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4d301ed950 0x7f4d300c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:52.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d2ffff700 1 -- 172.21.15.23:0/1734460828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d300c3b30 msgr2=0x7f4d300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:52.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d2ffff700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d300c3b30 0x7f4d300702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:52.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d2ffff700 1 -- 172.21.15.23:0/1734460828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d240094a0 con 0x7f4d300c3f00 2023-06-22T08:13:52.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d3499b700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d300c3b30 0x7f4d300702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:13:52.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d2ffff700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3f00 0x7f4d301f1a40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00c400 tx=0x7f4d1c00c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:52.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d2d7fa700 1 -- 172.21.15.23:0/1734460828 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c005d70 con 0x7f4d300c3f00 2023-06-22T08:13:52.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d2d7fa700 1 -- 172.21.15.23:0/1734460828 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c00ce90 con 0x7f4d300c3f00 2023-06-22T08:13:52.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d3599d700 1 -- 172.21.15.23:0/1734460828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d300c1a30 con 0x7f4d300c3f00 2023-06-22T08:13:52.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.134+0000 7f4d2d7fa700 1 -- 172.21.15.23:0/1734460828 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c0072f0 con 0x7f4d300c3f00 2023-06-22T08:13:52.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.135+0000 7f4d3599d700 1 -- 172.21.15.23:0/1734460828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d301f1240 con 0x7f4d300c3f00 2023-06-22T08:13:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.135+0000 7f4d3599d700 1 -- 172.21.15.23:0/1734460828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d30061b00 con 0x7f4d300c3f00 2023-06-22T08:13:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.136+0000 7f4d2d7fa700 1 -- 172.21.15.23:0/1734460828 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4d1c007450 con 0x7f4d300c3f00 2023-06-22T08:13:52.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.138+0000 7f4d2d7fa700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4d1406ad10 0x7f4d1406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.138+0000 7f4d2d7fa700 1 -- 172.21.15.23:0/1734460828 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f4d1c088ce0 con 0x7f4d300c3f00 2023-06-22T08:13:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.138+0000 7f4d3499b700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4d1406ad10 0x7f4d1406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:52.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.139+0000 7f4d3499b700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4d1406ad10 0x7f4d1406d1c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009b90 tx=0x7f4d24008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:52.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.142+0000 7f4d2d7fa700 1 -- 172.21.15.23:0/1734460828 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4d1c053710 con 0x7f4d300c3f00 2023-06-22T08:13:52.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.394+0000 7f4d3599d700 1 -- 172.21.15.23:0/1734460828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d3006bb20 con 0x7f4d300c3f00 2023-06-22T08:13:52.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.395+0000 7f4d2d7fa700 1 -- 172.21.15.23:0/1734460828 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6277 (secure 0 0 0) 0x7f4d1c056d30 con 0x7f4d300c3f00 2023-06-22T08:13:52.398 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:52.398 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:50.860209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:52.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 -- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4d1406ad10 msgr2=0x7f4d1406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:52.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4d1406ad10 0x7f4d1406d1c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009b90 tx=0x7f4d24008030 comp rx=0 tx=0).stop 2023-06-22T08:13:52.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 -- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3f00 msgr2=0x7f4d301f1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:52.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3f00 0x7f4d301f1a40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00c400 tx=0x7f4d1c00c710 comp rx=0 tx=0).stop 2023-06-22T08:13:52.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 -- 172.21.15.23:0/1734460828 shutdown_connections 2023-06-22T08:13:52.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4d301ed950 0x7f4d300c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:52.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d300c3b30 0x7f4d300702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:52.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4d1406ad10 0x7f4d1406d1c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:52.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 --2- 172.21.15.23:0/1734460828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d300c3f00 0x7f4d301f1a40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:52.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 -- 172.21.15.23:0/1734460828 >> 172.21.15.23:0/1734460828 conn(0x7f4d3006cfb0 msgr2=0x7f4d301501f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:52.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 -- 172.21.15.23:0/1734460828 shutdown_connections 2023-06-22T08:13:52.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:52.397+0000 7f4d3599d700 1 -- 172.21.15.23:0/1734460828 wait complete. 2023-06-22T08:13:52.403 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 108 2023-06-22T08:13:52.421 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:13:54.422 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:54.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.630+0000 7f621f1ff700 1 Processor -- start 2023-06-22T08:13:54.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.631+0000 7f621f1ff700 1 -- start start 2023-06-22T08:13:54.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f621f1ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6218152d70 0x7f62180c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:54.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f621f1ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6218159330 0x7f62180cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:54.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f621f1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f62180cb800 0x7f62180c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:54.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f621f1ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62180c95e0 con 0x7f6218159330 2023-06-22T08:13:54.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f621f1ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6218159790 con 0x7f6218152d70 2023-06-22T08:13:54.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f621f1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621815bea0 con 0x7f62180cb800 2023-06-22T08:13:54.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f6217fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6218159330 0x7f62180cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:54.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f6217fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6218159330 0x7f62180cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36096/0 (socket says 172.21.15.23:36096) 2023-06-22T08:13:54.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f6217fff700 1 -- 172.21.15.23:0/621198000 learned_addr learned my addr 172.21.15.23:0/621198000 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:54.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f621cf9b700 1 --2- 172.21.15.23:0/621198000 >> v2:172.21.15.67:3300/0 conn(0x7f6218152d70 0x7f62180c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:54.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.632+0000 7f621d79c700 1 --2- 172.21.15.23:0/621198000 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f62180cb800 0x7f62180c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:54.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.633+0000 7f6217fff700 1 -- 172.21.15.23:0/621198000 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f62180cb800 msgr2=0x7f62180c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:54.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.633+0000 7f6217fff700 1 --2- 172.21.15.23:0/621198000 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f62180cb800 0x7f62180c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.633+0000 7f6217fff700 1 -- 172.21.15.23:0/621198000 >> v2:172.21.15.67:3300/0 conn(0x7f6218152d70 msgr2=0x7f62180c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:13:54.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.633+0000 7f6217fff700 1 --2- 172.21.15.23:0/621198000 >> v2:172.21.15.67:3300/0 conn(0x7f6218152d70 0x7f62180c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.633+0000 7f6217fff700 1 -- 172.21.15.23:0/621198000 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621815c160 con 0x7f6218159330 2023-06-22T08:13:54.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.633+0000 7f6217fff700 1 --2- 172.21.15.23:0/621198000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6218159330 0x7f62180cb2c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f62180c8ef0 tx=0x7f6204009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5dae78635de025 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:54.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.633+0000 7f62177fe700 1 -- 172.21.15.23:0/621198000 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6204011830 con 0x7f6218159330 2023-06-22T08:13:54.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.633+0000 7f62177fe700 1 -- 172.21.15.23:0/621198000 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6204011990 con 0x7f6218159330 2023-06-22T08:13:54.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.633+0000 7f62177fe700 1 -- 172.21.15.23:0/621198000 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6204011d20 con 0x7f6218159330 2023-06-22T08:13:54.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.634+0000 7f621f1ff700 1 -- 172.21.15.23:0/621198000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6218159330 msgr2=0x7f62180cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:54.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.634+0000 7f621f1ff700 1 --2- 172.21.15.23:0/621198000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6218159330 0x7f62180cb2c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f62180c8ef0 tx=0x7f6204009bd0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.634+0000 7f621f1ff700 1 -- 172.21.15.23:0/621198000 shutdown_connections 2023-06-22T08:13:54.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.634+0000 7f621f1ff700 1 --2- 172.21.15.23:0/621198000 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f62180cb800 0x7f62180c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.634+0000 7f621f1ff700 1 --2- 172.21.15.23:0/621198000 >> v2:172.21.15.67:3300/0 conn(0x7f6218152d70 0x7f62180c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.634+0000 7f621f1ff700 1 --2- 172.21.15.23:0/621198000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6218159330 0x7f62180cb2c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.634+0000 7f621f1ff700 1 -- 172.21.15.23:0/621198000 >> 172.21.15.23:0/621198000 conn(0x7f621806ae30 msgr2=0x7f621806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:54.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.634+0000 7f621f1ff700 1 -- 172.21.15.23:0/621198000 shutdown_connections 2023-06-22T08:13:54.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.634+0000 7f621f1ff700 1 -- 172.21.15.23:0/621198000 wait complete. 2023-06-22T08:13:54.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.636+0000 7f621f1ff700 1 Processor -- start 2023-06-22T08:13:54.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.636+0000 7f621f1ff700 1 -- start start 2023-06-22T08:13:54.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621f1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62180cb800 0x7f62180ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:54.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621f1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218152d70 0x7f62180b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:54.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621f1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6218159330 0x7f62180ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:54.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621f1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180b4880 con 0x7f62180cb800 2023-06-22T08:13:54.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621f1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180cc790 con 0x7f6218152d70 2023-06-22T08:13:54.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621f1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62180cca70 con 0x7f6218159330 2023-06-22T08:13:54.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621cf9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62180cb800 0x7f62180ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:54.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621cf9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62180cb800 0x7f62180ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36108/0 (socket says 172.21.15.23:36108) 2023-06-22T08:13:54.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621cf9b700 1 -- 172.21.15.23:0/3013517247 learned_addr learned my addr 172.21.15.23:0/3013517247 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:54.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f6217fff700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218152d70 0x7f62180b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:54.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621d79c700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6218159330 0x7f62180ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:54.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621cf9b700 1 -- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6218159330 msgr2=0x7f62180ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:54.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621cf9b700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6218159330 0x7f62180ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621cf9b700 1 -- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218152d70 msgr2=0x7f62180b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:54.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621cf9b700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218152d70 0x7f62180b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.637+0000 7f621cf9b700 1 -- 172.21.15.23:0/3013517247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6204009550 con 0x7f62180cb800 2023-06-22T08:13:54.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.638+0000 7f621cf9b700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62180cb800 0x7f62180ce310 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f620c00e9b0 tx=0x7f620c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:54.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.638+0000 7f62157fa700 1 -- 172.21.15.23:0/3013517247 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620c009710 con 0x7f62180cb800 2023-06-22T08:13:54.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.638+0000 7f62157fa700 1 -- 172.21.15.23:0/3013517247 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620c00ee50 con 0x7f62180cb800 2023-06-22T08:13:54.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.638+0000 7f621f1ff700 1 -- 172.21.15.23:0/3013517247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62180ccd50 con 0x7f62180cb800 2023-06-22T08:13:54.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.638+0000 7f62157fa700 1 -- 172.21.15.23:0/3013517247 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620c00ca50 con 0x7f62180cb800 2023-06-22T08:13:54.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.638+0000 7f621f1ff700 1 -- 172.21.15.23:0/3013517247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62180cd310 con 0x7f62180cb800 2023-06-22T08:13:54.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.639+0000 7f621f1ff700 1 -- 172.21.15.23:0/3013517247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621804ea20 con 0x7f62180cb800 2023-06-22T08:13:54.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.640+0000 7f62157fa700 1 -- 172.21.15.23:0/3013517247 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f620c00cbb0 con 0x7f62180cb800 2023-06-22T08:13:54.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.642+0000 7f62157fa700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f61fc06ad10 0x7f61fc06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:54.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.642+0000 7f62157fa700 1 -- 172.21.15.23:0/3013517247 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f620c089a90 con 0x7f62180cb800 2023-06-22T08:13:54.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.642+0000 7f6217fff700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f61fc06ad10 0x7f61fc06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:54.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.643+0000 7f6217fff700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f61fc06ad10 0x7f61fc06d1c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6204009c10 tx=0x7f6204008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:54.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.647+0000 7f62157fa700 1 -- 172.21.15.23:0/3013517247 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f620c057f50 con 0x7f62180cb800 2023-06-22T08:13:54.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.902+0000 7f621f1ff700 1 -- 172.21.15.23:0/3013517247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62180c8cd0 con 0x7f62180cb800 2023-06-22T08:13:54.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.903+0000 7f62157fa700 1 -- 172.21.15.23:0/3013517247 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6277 (secure 0 0 0) 0x7f620c017090 con 0x7f62180cb800 2023-06-22T08:13:54.906 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:54.907 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:50.860209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.905+0000 7f621f1ff700 1 -- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f61fc06ad10 msgr2=0x7f61fc06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.905+0000 7f621f1ff700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f61fc06ad10 0x7f61fc06d1c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6204009c10 tx=0x7f6204008030 comp rx=0 tx=0).stop 2023-06-22T08:13:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.905+0000 7f621f1ff700 1 -- 172.21.15.23:0/3013517247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62180cb800 msgr2=0x7f62180ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.905+0000 7f621f1ff700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62180cb800 0x7f62180ce310 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f620c00e9b0 tx=0x7f620c00ecc0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.905+0000 7f621f1ff700 1 -- 172.21.15.23:0/3013517247 shutdown_connections 2023-06-22T08:13:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.905+0000 7f621f1ff700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6218159330 0x7f62180ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.905+0000 7f621f1ff700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62180cb800 0x7f62180ce310 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.905+0000 7f621f1ff700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f61fc06ad10 0x7f61fc06d1c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.905+0000 7f621f1ff700 1 --2- 172.21.15.23:0/3013517247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6218152d70 0x7f62180b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.905+0000 7f621f1ff700 1 -- 172.21.15.23:0/3013517247 >> 172.21.15.23:0/3013517247 conn(0x7f621806ae30 msgr2=0x7f62180caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.906+0000 7f621f1ff700 1 -- 172.21.15.23:0/3013517247 shutdown_connections 2023-06-22T08:13:54.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:54.906+0000 7f621f1ff700 1 -- 172.21.15.23:0/3013517247 wait complete. 2023-06-22T08:13:54.911 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 108 2023-06-22T08:13:54.927 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:13:56.928 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:57.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.139+0000 7fb9b4bff700 1 Processor -- start 2023-06-22T08:13:57.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.139+0000 7fb9b4bff700 1 -- start start 2023-06-22T08:13:57.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.140+0000 7fb9b4bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 0x7fb9b00cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:57.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.140+0000 7fb9b4bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b015c370 con 0x7fb9b00c40b0 2023-06-22T08:13:57.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.140+0000 7fb9b4bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b015bda0 con 0x7fb9b00c9fd0 2023-06-22T08:13:57.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.140+0000 7fb9b4bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b01530b0 con 0x7fb9b0152ce0 2023-06-22T08:13:57.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.140+0000 7fb9aed9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb9b00c40b0 0x7fb9b00c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:43850/0 (socket says 172.21.15.23:43850) 2023-06-22T08:13:57.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.140+0000 7fb9aed9d700 1 -- 172.21.15.23:0/4156386553 learned_addr learned my addr 172.21.15.23:0/4156386553 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:57.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.140+0000 7fb9ae59c700 1 --2- 172.21.15.23:0/4156386553 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 0x7fb9b00cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:57.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.141+0000 7fb9ad59a700 1 -- 172.21.15.23:0/4156386553 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301692087 0 0) 0x7fb9b015c370 con 0x7fb9b00c40b0 2023-06-22T08:13:57.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.141+0000 7fb9ad59a700 1 -- 172.21.15.23:0/4156386553 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9980035b0 con 0x7fb9b00c40b0 2023-06-22T08:13:57.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.141+0000 7fb9ad59a700 1 -- 172.21.15.23:0/4156386553 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331224284 0 0) 0x7fb9b015bda0 con 0x7fb9b00c9fd0 2023-06-22T08:13:57.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.141+0000 7fb9ad59a700 1 -- 172.21.15.23:0/4156386553 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b015c370 con 0x7fb9b00c9fd0 2023-06-22T08:13:57.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.141+0000 7fb9ae59c700 1 -- 172.21.15.23:0/4156386553 >> v1:172.21.15.67:6789/0 conn(0x7fb9b00c9fd0 legacy=0x7fb9b015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:57.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.141+0000 7fb9ae59c700 1 -- 172.21.15.23:0/4156386553 >> v1:172.21.15.23:6789/0 conn(0x7fb9b00c40b0 legacy=0x7fb9b00c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:57.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.141+0000 7fb9ae59c700 1 -- 172.21.15.23:0/4156386553 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b00c9460 con 0x7fb9b0152ce0 2023-06-22T08:13:57.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.141+0000 7fb9ae59c700 1 --2- 172.21.15.23:0/4156386553 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 0x7fb9b00cf1b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009870 tx=0x7fb9a0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa06911a0228fb1c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:57.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.141+0000 7fb9ad59a700 1 -- 172.21.15.23:0/4156386553 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0011830 con 0x7fb9b0152ce0 2023-06-22T08:13:57.147 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.142+0000 7fb9ad59a700 1 -- 172.21.15.23:0/4156386553 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a0011990 con 0x7fb9b0152ce0 2023-06-22T08:13:57.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.142+0000 7fb9ad59a700 1 -- 172.21.15.23:0/4156386553 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0011d20 con 0x7fb9b0152ce0 2023-06-22T08:13:57.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.142+0000 7fb9b4bff700 1 -- 172.21.15.23:0/4156386553 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 msgr2=0x7fb9b00cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:57.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.142+0000 7fb9b4bff700 1 --2- 172.21.15.23:0/4156386553 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 0x7fb9b00cf1b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009870 tx=0x7fb9a0009b80 comp rx=0 tx=0).stop 2023-06-22T08:13:57.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.142+0000 7fb9b4bff700 1 -- 172.21.15.23:0/4156386553 shutdown_connections 2023-06-22T08:13:57.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.142+0000 7fb9b4bff700 1 --2- 172.21.15.23:0/4156386553 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 0x7fb9b00cf1b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:57.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.142+0000 7fb9b4bff700 1 -- 172.21.15.23:0/4156386553 >> 172.21.15.23:0/4156386553 conn(0x7fb9b006ae30 msgr2=0x7fb9b006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:57.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.142+0000 7fb9b4bff700 1 -- 172.21.15.23:0/4156386553 shutdown_connections 2023-06-22T08:13:57.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.142+0000 7fb9b4bff700 1 -- 172.21.15.23:0/4156386553 wait complete. 2023-06-22T08:13:57.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.145+0000 7fb9b4bff700 1 Processor -- start 2023-06-22T08:13:57.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.145+0000 7fb9b4bff700 1 -- start start 2023-06-22T08:13:57.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9b4bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b00c40b0 0x7fb9b01e2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:57.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9b4bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b00c9fd0 0x7fb9b015ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:57.152 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9b4bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 0x7fb9b015b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:57.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9b4bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b0158200 con 0x7fb9b00c9fd0 2023-06-22T08:13:57.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9b4bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b01e3810 con 0x7fb9b00c40b0 2023-06-22T08:13:57.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9b4bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b01eeb40 con 0x7fb9b0152ce0 2023-06-22T08:13:57.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9ae59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b00c40b0 0x7fb9b01e2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:57.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9add9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b00c9fd0 0x7fb9b015ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:57.153 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9ae59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b00c40b0 0x7fb9b01e2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:53306/0 (socket says 172.21.15.23:53306) 2023-06-22T08:13:57.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9ae59c700 1 -- 172.21.15.23:0/422540538 learned_addr learned my addr 172.21.15.23:0/422540538 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:57.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.146+0000 7fb9aed9d700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 0x7fb9b015b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:57.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.147+0000 7fb9ae59c700 1 -- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 msgr2=0x7fb9b015b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:57.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.147+0000 7fb9ae59c700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 0x7fb9b015b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:57.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.147+0000 7fb9ae59c700 1 -- 172.21.15.23:0/422540538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b00c9fd0 msgr2=0x7fb9b015ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:57.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.147+0000 7fb9ae59c700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b00c9fd0 0x7fb9b015ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:57.154 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.147+0000 7fb9ae59c700 1 -- 172.21.15.23:0/422540538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a00094a0 con 0x7fb9b00c40b0 2023-06-22T08:13:57.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.147+0000 7fb9ae59c700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b00c40b0 0x7fb9b01e2750 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb9a00097c0 tx=0x7fb9a0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:57.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.147+0000 7fb99effd700 1 -- 172.21.15.23:0/422540538 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a001a8f0 con 0x7fb9b00c40b0 2023-06-22T08:13:57.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.147+0000 7fb99effd700 1 -- 172.21.15.23:0/422540538 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a001aa70 con 0x7fb9b00c40b0 2023-06-22T08:13:57.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.148+0000 7fb9b4bff700 1 -- 172.21.15.23:0/422540538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b0155db0 con 0x7fb9b00c40b0 2023-06-22T08:13:57.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.148+0000 7fb99effd700 1 -- 172.21.15.23:0/422540538 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a001ae20 con 0x7fb9b00c40b0 2023-06-22T08:13:57.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.148+0000 7fb9b4bff700 1 -- 172.21.15.23:0/422540538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b01e31c0 con 0x7fb9b00c40b0 2023-06-22T08:13:57.155 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.149+0000 7fb9b4bff700 1 -- 172.21.15.23:0/422540538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b004ea20 con 0x7fb9b00c40b0 2023-06-22T08:13:57.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.150+0000 7fb99effd700 1 -- 172.21.15.23:0/422540538 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb9a002e450 con 0x7fb9b00c40b0 2023-06-22T08:13:57.156 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.150+0000 7fb99effd700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb99006ace0 0x7fb99006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:57.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.151+0000 7fb99effd700 1 -- 172.21.15.23:0/422540538 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7fb9a008ad70 con 0x7fb9b00c40b0 2023-06-22T08:13:57.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.151+0000 7fb9add9b700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb99006ace0 0x7fb99006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:57.157 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.151+0000 7fb9add9b700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb99006ace0 0x7fb99006d190 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb9b00b4760 tx=0x7fb9a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:57.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.155+0000 7fb99effd700 1 -- 172.21.15.23:0/422540538 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb9a00558c0 con 0x7fb9b00c40b0 2023-06-22T08:13:57.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.413+0000 7fb9b4bff700 1 -- 172.21.15.23:0/422540538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9b00c9db0 con 0x7fb9b00c40b0 2023-06-22T08:13:57.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.414+0000 7fb99effd700 1 -- 172.21.15.23:0/422540538 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6277 (secure 0 0 0) 0x7fb9a0058ee0 con 0x7fb9b00c40b0 2023-06-22T08:13:57.418 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:57.418 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:50.860209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:57.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.417+0000 7fb9b4bff700 1 -- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb99006ace0 msgr2=0x7fb99006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:57.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.417+0000 7fb9b4bff700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb99006ace0 0x7fb99006d190 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb9b00b4760 tx=0x7fb9a4009040 comp rx=0 tx=0).stop 2023-06-22T08:13:57.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.417+0000 7fb9b4bff700 1 -- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b00c40b0 msgr2=0x7fb9b01e2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:57.422 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.417+0000 7fb9b4bff700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b00c40b0 0x7fb9b01e2750 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb9a00097c0 tx=0x7fb9a0004750 comp rx=0 tx=0).stop 2023-06-22T08:13:57.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.418+0000 7fb9b4bff700 1 -- 172.21.15.23:0/422540538 shutdown_connections 2023-06-22T08:13:57.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.418+0000 7fb9b4bff700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b0152ce0 0x7fb9b015b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:57.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.418+0000 7fb9b4bff700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb99006ace0 0x7fb99006d190 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:57.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.418+0000 7fb9b4bff700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b00c40b0 0x7fb9b01e2750 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:57.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.418+0000 7fb9b4bff700 1 --2- 172.21.15.23:0/422540538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b00c9fd0 0x7fb9b015ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:57.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.418+0000 7fb9b4bff700 1 -- 172.21.15.23:0/422540538 >> 172.21.15.23:0/422540538 conn(0x7fb9b006ae30 msgr2=0x7fb9b014c630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:57.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.418+0000 7fb9b4bff700 1 -- 172.21.15.23:0/422540538 shutdown_connections 2023-06-22T08:13:57.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:57.419+0000 7fb9b4bff700 1 -- 172.21.15.23:0/422540538 wait complete. 2023-06-22T08:13:57.425 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 108 2023-06-22T08:13:57.446 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:13:59.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:13:59.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.655+0000 7f1cf23ff700 1 Processor -- start 2023-06-22T08:13:59.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.655+0000 7f1cf23ff700 1 -- start start 2023-06-22T08:13:59.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.656+0000 7f1cf23ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec0c97d0 0x7f1cec159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:59.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.656+0000 7f1cf23ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec0c9c00 con 0x7f1cec0c45f0 2023-06-22T08:13:59.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.656+0000 7f1cf23ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec15bea0 con 0x7f1cec152d70 2023-06-22T08:13:59.660 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.656+0000 7f1cf23ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec0c95e0 con 0x7f1cec0c97d0 2023-06-22T08:13:59.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.657+0000 7f1cf099c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1cec0c45f0 0x7f1cec0c4a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:43862/0 (socket says 172.21.15.23:43862) 2023-06-22T08:13:59.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.657+0000 7f1cf099c700 1 -- 172.21.15.23:0/98828214 learned_addr learned my addr 172.21.15.23:0/98828214 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:13:59.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.657+0000 7f1ceb7fe700 1 --2- 172.21.15.23:0/98828214 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec0c97d0 0x7f1cec159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:59.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.657+0000 7f1ceaffd700 1 -- 172.21.15.23:0/98828214 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171881011 0 0) 0x7f1cec0c9c00 con 0x7f1cec0c45f0 2023-06-22T08:13:59.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.657+0000 7f1ceaffd700 1 -- 172.21.15.23:0/98828214 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd40035b0 con 0x7f1cec0c45f0 2023-06-22T08:13:59.662 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.657+0000 7f1ceaffd700 1 -- 172.21.15.23:0/98828214 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975101054 0 0) 0x7f1cec15bea0 con 0x7f1cec152d70 2023-06-22T08:13:59.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.657+0000 7f1ceaffd700 1 -- 172.21.15.23:0/98828214 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cec0c9c00 con 0x7f1cec152d70 2023-06-22T08:13:59.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.657+0000 7f1ceb7fe700 1 -- 172.21.15.23:0/98828214 >> v1:172.21.15.67:6789/0 conn(0x7f1cec152d70 legacy=0x7f1cec0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:59.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.657+0000 7f1ceb7fe700 1 -- 172.21.15.23:0/98828214 >> v1:172.21.15.23:6789/0 conn(0x7f1cec0c45f0 legacy=0x7f1cec0c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:59.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.657+0000 7f1ceb7fe700 1 -- 172.21.15.23:0/98828214 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec15c160 con 0x7f1cec0c97d0 2023-06-22T08:13:59.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.658+0000 7f1ceb7fe700 1 --2- 172.21.15.23:0/98828214 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec0c97d0 0x7f1cec159330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009870 tx=0x7f1ce0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8624444f557280f9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:59.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.658+0000 7f1ceaffd700 1 -- 172.21.15.23:0/98828214 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0011830 con 0x7f1cec0c97d0 2023-06-22T08:13:59.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.658+0000 7f1ceaffd700 1 -- 172.21.15.23:0/98828214 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce0011990 con 0x7f1cec0c97d0 2023-06-22T08:13:59.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.658+0000 7f1ceaffd700 1 -- 172.21.15.23:0/98828214 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0011d20 con 0x7f1cec0c97d0 2023-06-22T08:13:59.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.658+0000 7f1cf23ff700 1 -- 172.21.15.23:0/98828214 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec0c97d0 msgr2=0x7f1cec159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:59.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.658+0000 7f1cf23ff700 1 --2- 172.21.15.23:0/98828214 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec0c97d0 0x7f1cec159330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009870 tx=0x7f1ce0009b80 comp rx=0 tx=0).stop 2023-06-22T08:13:59.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.659+0000 7f1cf23ff700 1 -- 172.21.15.23:0/98828214 shutdown_connections 2023-06-22T08:13:59.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.659+0000 7f1cf23ff700 1 --2- 172.21.15.23:0/98828214 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec0c97d0 0x7f1cec159330 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:59.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.659+0000 7f1cf23ff700 1 -- 172.21.15.23:0/98828214 >> 172.21.15.23:0/98828214 conn(0x7f1cec06ae30 msgr2=0x7f1cec06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:59.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.659+0000 7f1cf23ff700 1 -- 172.21.15.23:0/98828214 shutdown_connections 2023-06-22T08:13:59.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.659+0000 7f1cf23ff700 1 -- 172.21.15.23:0/98828214 wait complete. 2023-06-22T08:13:59.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.660+0000 7f1cf23ff700 1 Processor -- start 2023-06-22T08:13:59.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.661+0000 7f1cf23ff700 1 -- start start 2023-06-22T08:13:59.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1cf23ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cec0c45f0 0x7f1cec14be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:59.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1cf23ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cec0c97d0 0x7f1cec14b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:59.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1cf23ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec152d70 0x7f1cec14b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:59.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1cf23ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec1ef160 con 0x7f1cec0c97d0 2023-06-22T08:13:59.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1cf23ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec159b10 con 0x7f1cec0c45f0 2023-06-22T08:13:59.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1cf23ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec1eee80 con 0x7f1cec152d70 2023-06-22T08:13:59.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1ceb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cec0c97d0 0x7f1cec14b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:59.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1ceb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cec0c97d0 0x7f1cec14b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49618/0 (socket says 172.21.15.23:49618) 2023-06-22T08:13:59.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1ceb7fe700 1 -- 172.21.15.23:0/4052994163 learned_addr learned my addr 172.21.15.23:0/4052994163 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:13:59.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1cebfff700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cec0c45f0 0x7f1cec14be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:59.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1cf099c700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec152d70 0x7f1cec14b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:59.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1ceb7fe700 1 -- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec152d70 msgr2=0x7f1cec14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:59.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1ceb7fe700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec152d70 0x7f1cec14b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:59.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1ceb7fe700 1 -- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cec0c45f0 msgr2=0x7f1cec14be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:59.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1ceb7fe700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cec0c45f0 0x7f1cec14be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:59.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1ceb7fe700 1 -- 172.21.15.23:0/4052994163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce00094a0 con 0x7f1cec0c97d0 2023-06-22T08:13:59.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.662+0000 7f1cebfff700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cec0c45f0 0x7f1cec14be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:13:59.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.663+0000 7f1ceb7fe700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cec0c97d0 0x7f1cec14b290 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00090b0 tx=0x7f1ce0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:59.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.663+0000 7f1ce8ff9700 1 -- 172.21.15.23:0/4052994163 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce001a860 con 0x7f1cec0c97d0 2023-06-22T08:13:59.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.663+0000 7f1ce8ff9700 1 -- 172.21.15.23:0/4052994163 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce001a9c0 con 0x7f1cec0c97d0 2023-06-22T08:13:59.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.663+0000 7f1cf23ff700 1 -- 172.21.15.23:0/4052994163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec0b43e0 con 0x7f1cec0c97d0 2023-06-22T08:13:59.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.663+0000 7f1ce8ff9700 1 -- 172.21.15.23:0/4052994163 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce001ad30 con 0x7f1cec0c97d0 2023-06-22T08:13:59.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.663+0000 7f1cf23ff700 1 -- 172.21.15.23:0/4052994163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec1eeb90 con 0x7f1cec0c97d0 2023-06-22T08:13:59.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.664+0000 7f1cf23ff700 1 -- 172.21.15.23:0/4052994163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec04ea20 con 0x7f1cec0c97d0 2023-06-22T08:13:59.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.665+0000 7f1ce8ff9700 1 -- 172.21.15.23:0/4052994163 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1ce0022070 con 0x7f1cec0c97d0 2023-06-22T08:13:59.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.666+0000 7f1ce8ff9700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1cd806ace0 0x7f1cd806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:13:59.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.666+0000 7f1ce8ff9700 1 -- 172.21.15.23:0/4052994163 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7f1ce008aaf0 con 0x7f1cec0c97d0 2023-06-22T08:13:59.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.666+0000 7f1cebfff700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1cd806ace0 0x7f1cd806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:13:59.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.667+0000 7f1cebfff700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1cd806ace0 0x7f1cd806d190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0011e0 tx=0x7f1cdc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:13:59.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.670+0000 7f1ce8ff9700 1 -- 172.21.15.23:0/4052994163 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1ce0055780 con 0x7f1cec0c97d0 2023-06-22T08:13:59.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.927+0000 7f1cf23ff700 1 -- 172.21.15.23:0/4052994163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cec0c9e00 con 0x7f1cec0c97d0 2023-06-22T08:13:59.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.928+0000 7f1ce8ff9700 1 -- 172.21.15.23:0/4052994163 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6277 (secure 0 0 0) 0x7f1ce0015090 con 0x7f1cec0c97d0 2023-06-22T08:13:59.931 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:13:59.931 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:50.860209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:13:59.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 -- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1cd806ace0 msgr2=0x7f1cd806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:59.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1cd806ace0 0x7f1cd806d190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0011e0 tx=0x7f1cdc009040 comp rx=0 tx=0).stop 2023-06-22T08:13:59.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 -- 172.21.15.23:0/4052994163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cec0c97d0 msgr2=0x7f1cec14b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:13:59.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cec0c97d0 0x7f1cec14b290 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00090b0 tx=0x7f1ce0004750 comp rx=0 tx=0).stop 2023-06-22T08:13:59.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 -- 172.21.15.23:0/4052994163 shutdown_connections 2023-06-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1cec152d70 0x7f1cec14b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1cd806ace0 0x7f1cd806d190 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cec0c45f0 0x7f1cec14be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 --2- 172.21.15.23:0/4052994163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cec0c97d0 0x7f1cec14b290 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 -- 172.21.15.23:0/4052994163 >> 172.21.15.23:0/4052994163 conn(0x7f1cec06ae30 msgr2=0x7f1cec0c3850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 -- 172.21.15.23:0/4052994163 shutdown_connections 2023-06-22T08:13:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:13:59.930+0000 7f1cf23ff700 1 -- 172.21.15.23:0/4052994163 wait complete. 2023-06-22T08:13:59.936 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 108 2023-06-22T08:13:59.952 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:14:01.953 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:02.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.157+0000 7fed0f7ff700 1 Processor -- start 2023-06-22T08:14:02.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.157+0000 7fed0f7ff700 1 -- start start 2023-06-22T08:14:02.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.159+0000 7fed0f7ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fed080c4a30 0x7fed080cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:02.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.159+0000 7fed0f7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed080cb7a0 0x7fed0815c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:02.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.159+0000 7fed0f7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed08159730 con 0x7fed08152a10 2023-06-22T08:14:02.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.159+0000 7fed0f7ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed080c5470 con 0x7fed080c4a30 2023-06-22T08:14:02.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.159+0000 7fed0f7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed080cbbd0 con 0x7fed080cb7a0 2023-06-22T08:14:02.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.159+0000 7fed0cd9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fed080c4a30 0x7fed080cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:02.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.159+0000 7fed0cd9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fed080c4a30 0x7fed080cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49622/0 (socket says 172.21.15.23:49622) 2023-06-22T08:14:02.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.159+0000 7fed0cd9a700 1 -- 172.21.15.23:0/4118891014 learned_addr learned my addr 172.21.15.23:0/4118891014 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:02.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.159+0000 7fed0dd9c700 1 --2- 172.21.15.23:0/4118891014 >> v2:172.21.15.67:3300/0 conn(0x7fed080cb7a0 0x7fed0815c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:02.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.160+0000 7fed0cd9a700 1 -- 172.21.15.23:0/4118891014 >> v2:172.21.15.67:3300/0 conn(0x7fed080cb7a0 msgr2=0x7fed0815c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:14:02.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.160+0000 7fed0cd9a700 1 --2- 172.21.15.23:0/4118891014 >> v2:172.21.15.67:3300/0 conn(0x7fed080cb7a0 0x7fed0815c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:02.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.160+0000 7fed0cd9a700 1 -- 172.21.15.23:0/4118891014 >> v1:172.21.15.67:6789/0 conn(0x7fed08152a10 legacy=0x7fed080c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:02.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.160+0000 7fed0cd9a700 1 -- 172.21.15.23:0/4118891014 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08158910 con 0x7fed080c4a30 2023-06-22T08:14:02.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.160+0000 7fed0cd9a700 1 --2- 172.21.15.23:0/4118891014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 0x7fed080cb260 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fed0806c300 tx=0x7fecf8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4bc001c528505f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:02.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.160+0000 7fecfffff700 1 -- 172.21.15.23:0/4118891014 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8011830 con 0x7fed080c4a30 2023-06-22T08:14:02.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.160+0000 7fecfffff700 1 -- 172.21.15.23:0/4118891014 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf8011990 con 0x7fed080c4a30 2023-06-22T08:14:02.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.160+0000 7fecfffff700 1 -- 172.21.15.23:0/4118891014 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8011d20 con 0x7fed080c4a30 2023-06-22T08:14:02.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.161+0000 7fed0f7ff700 1 -- 172.21.15.23:0/4118891014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 msgr2=0x7fed080cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:02.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.161+0000 7fed0f7ff700 1 --2- 172.21.15.23:0/4118891014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 0x7fed080cb260 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fed0806c300 tx=0x7fecf8009bd0 comp rx=0 tx=0).stop 2023-06-22T08:14:02.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.161+0000 7fed0f7ff700 1 -- 172.21.15.23:0/4118891014 shutdown_connections 2023-06-22T08:14:02.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.161+0000 7fed0f7ff700 1 --2- 172.21.15.23:0/4118891014 >> v2:172.21.15.67:3300/0 conn(0x7fed080cb7a0 0x7fed0815c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:02.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.161+0000 7fed0f7ff700 1 --2- 172.21.15.23:0/4118891014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 0x7fed080cb260 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:02.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.161+0000 7fed0f7ff700 1 -- 172.21.15.23:0/4118891014 >> 172.21.15.23:0/4118891014 conn(0x7fed08065b90 msgr2=0x7fed08065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:02.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.161+0000 7fed0f7ff700 1 -- 172.21.15.23:0/4118891014 shutdown_connections 2023-06-22T08:14:02.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.161+0000 7fed0f7ff700 1 -- 172.21.15.23:0/4118891014 wait complete. 2023-06-22T08:14:02.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.162+0000 7fed0f7ff700 1 Processor -- start 2023-06-22T08:14:02.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.163+0000 7fed0f7ff700 1 -- start start 2023-06-22T08:14:02.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.163+0000 7fed0f7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 0x7fed080c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:02.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.163+0000 7fed0f7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed080cb7a0 0x7fed08151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:02.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0f7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed08152a10 0x7fed08151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:02.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0f7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed08152280 con 0x7fed080c4a30 2023-06-22T08:14:02.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0f7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed08155d10 con 0x7fed08152a10 2023-06-22T08:14:02.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0f7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed08155ff0 con 0x7fed080cb7a0 2023-06-22T08:14:02.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0d59b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 0x7fed080c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:02.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0d59b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 0x7fed080c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49634/0 (socket says 172.21.15.23:49634) 2023-06-22T08:14:02.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0d59b700 1 -- 172.21.15.23:0/2731270838 learned_addr learned my addr 172.21.15.23:0/2731270838 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:02.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0dd9c700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed08152a10 0x7fed08151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:02.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0cd9a700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed080cb7a0 0x7fed08151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:02.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0d59b700 1 -- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed080cb7a0 msgr2=0x7fed08151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:02.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0d59b700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed080cb7a0 0x7fed08151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:02.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0d59b700 1 -- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed08152a10 msgr2=0x7fed08151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:02.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0d59b700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed08152a10 0x7fed08151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:02.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0d59b700 1 -- 172.21.15.23:0/2731270838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf8009550 con 0x7fed080c4a30 2023-06-22T08:14:02.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.164+0000 7fed0d59b700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 0x7fed080c7380 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c450 tx=0x7fed0400c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:02.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.165+0000 7fecfdffb700 1 -- 172.21.15.23:0/2731270838 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040052d0 con 0x7fed080c4a30 2023-06-22T08:14:02.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.165+0000 7fecfdffb700 1 -- 172.21.15.23:0/2731270838 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed04005430 con 0x7fed080c4a30 2023-06-22T08:14:02.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.165+0000 7fed0f7ff700 1 -- 172.21.15.23:0/2731270838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed081ee750 con 0x7fed080c4a30 2023-06-22T08:14:02.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.165+0000 7fecfdffb700 1 -- 172.21.15.23:0/2731270838 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed040038c0 con 0x7fed080c4a30 2023-06-22T08:14:02.172 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.165+0000 7fed0f7ff700 1 -- 172.21.15.23:0/2731270838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed081eed10 con 0x7fed080c4a30 2023-06-22T08:14:02.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.166+0000 7fed0f7ff700 1 -- 172.21.15.23:0/2731270838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0804ea20 con 0x7fed080c4a30 2023-06-22T08:14:02.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.167+0000 7fecfdffb700 1 -- 172.21.15.23:0/2731270838 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fed04003a20 con 0x7fed080c4a30 2023-06-22T08:14:02.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.168+0000 7fecfdffb700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fecec06ad10 0x7fecec06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:02.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.168+0000 7fecfdffb700 1 -- 172.21.15.23:0/2731270838 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 7415+0+0 (secure 0 0 0) 0x7fed04088e20 con 0x7fed080c4a30 2023-06-22T08:14:02.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.168+0000 7fed0cd9a700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fecec06ad10 0x7fecec06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:02.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.169+0000 7fed0cd9a700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fecec06ad10 0x7fecec06d1c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009c10 tx=0x7fecf8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.172+0000 7fecfdffb700 1 -- 172.21.15.23:0/2731270838 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fed04053850 con 0x7fed080c4a30 2023-06-22T08:14:02.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.424+0000 7fed0f7ff700 1 -- 172.21.15.23:0/2731270838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed080cbd20 con 0x7fed080c4a30 2023-06-22T08:14:02.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.425+0000 7fecfdffb700 1 -- 172.21.15.23:0/2731270838 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6277 (secure 0 0 0) 0x7fed04017020 con 0x7fed080c4a30 2023-06-22T08:14:02.429 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:02.429 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5576,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:13:50.860209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5211},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5211":{"gid":5211,"name":"e","rank":4,"incarnation":35,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/2058668008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":2058668008},{"type":"v1","addr":"172.21.15.23:6837","nonce":2058668008}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:02.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 -- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fecec06ad10 msgr2=0x7fecec06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:02.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fecec06ad10 0x7fecec06d1c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009c10 tx=0x7fecf8008030 comp rx=0 tx=0).stop 2023-06-22T08:14:02.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 -- 172.21.15.23:0/2731270838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 msgr2=0x7fed080c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:02.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 0x7fed080c7380 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c450 tx=0x7fed0400c760 comp rx=0 tx=0).stop 2023-06-22T08:14:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 -- 172.21.15.23:0/2731270838 shutdown_connections 2023-06-22T08:14:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fecec06ad10 0x7fecec06d1c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed08152a10 0x7fed08151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed080c4a30 0x7fed080c7380 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 --2- 172.21.15.23:0/2731270838 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed080cb7a0 0x7fed08151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 -- 172.21.15.23:0/2731270838 >> 172.21.15.23:0/2731270838 conn(0x7fed08065b90 msgr2=0x7fed080ccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 -- 172.21.15.23:0/2731270838 shutdown_connections 2023-06-22T08:14:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:02.427+0000 7fed0f7ff700 1 -- 172.21.15.23:0/2731270838 wait complete. 2023-06-22T08:14:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 108 2023-06-22T08:14:02.449 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:14:04.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.671+0000 7fb80e7ff700 1 Processor -- start 2023-06-22T08:14:04.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.671+0000 7fb80e7ff700 1 -- start start 2023-06-22T08:14:04.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.673+0000 7fb80e7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c97d0 0x7fb808159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:04.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.673+0000 7fb80e7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8080c9c00 con 0x7fb808152d70 2023-06-22T08:14:04.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.673+0000 7fb80e7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb80815bea0 con 0x7fb8080c45f0 2023-06-22T08:14:04.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.673+0000 7fb80e7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8080c95e0 con 0x7fb8080c97d0 2023-06-22T08:14:04.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.673+0000 7fb807fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb808152d70 0x7fb8080cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:43874/0 (socket says 172.21.15.23:43874) 2023-06-22T08:14:04.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.673+0000 7fb807fff700 1 -- 172.21.15.23:0/4016540524 learned_addr learned my addr 172.21.15.23:0/4016540524 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:14:04.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.674+0000 7fb8077fe700 1 --2- 172.21.15.23:0/4016540524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c97d0 0x7fb808159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:04.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.674+0000 7fb806ffd700 1 -- 172.21.15.23:0/4016540524 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998619233 0 0) 0x7fb8080c9c00 con 0x7fb808152d70 2023-06-22T08:14:04.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.674+0000 7fb806ffd700 1 -- 172.21.15.23:0/4016540524 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f00035b0 con 0x7fb808152d70 2023-06-22T08:14:04.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.674+0000 7fb806ffd700 1 -- 172.21.15.23:0/4016540524 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414144953 0 0) 0x7fb80815bea0 con 0x7fb8080c45f0 2023-06-22T08:14:04.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.674+0000 7fb806ffd700 1 -- 172.21.15.23:0/4016540524 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8080c9c00 con 0x7fb8080c45f0 2023-06-22T08:14:04.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.674+0000 7fb8077fe700 1 -- 172.21.15.23:0/4016540524 >> v1:172.21.15.67:6789/0 conn(0x7fb8080c45f0 legacy=0x7fb8080c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.674+0000 7fb8077fe700 1 -- 172.21.15.23:0/4016540524 >> v1:172.21.15.23:6789/0 conn(0x7fb808152d70 legacy=0x7fb8080cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.674+0000 7fb8077fe700 1 -- 172.21.15.23:0/4016540524 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80815c160 con 0x7fb8080c97d0 2023-06-22T08:14:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.674+0000 7fb8077fe700 1 --2- 172.21.15.23:0/4016540524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c97d0 0x7fb808159330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009870 tx=0x7fb7fc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ff1f076c9fd2c4a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.675+0000 7fb806ffd700 1 -- 172.21.15.23:0/4016540524 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc011830 con 0x7fb8080c97d0 2023-06-22T08:14:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.675+0000 7fb806ffd700 1 -- 172.21.15.23:0/4016540524 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc011990 con 0x7fb8080c97d0 2023-06-22T08:14:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.675+0000 7fb806ffd700 1 -- 172.21.15.23:0/4016540524 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc011d20 con 0x7fb8080c97d0 2023-06-22T08:14:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.675+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4016540524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c97d0 msgr2=0x7fb808159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:04.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.675+0000 7fb80e7ff700 1 --2- 172.21.15.23:0/4016540524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c97d0 0x7fb808159330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009870 tx=0x7fb7fc009b80 comp rx=0 tx=0).stop 2023-06-22T08:14:04.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.675+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4016540524 shutdown_connections 2023-06-22T08:14:04.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.675+0000 7fb80e7ff700 1 --2- 172.21.15.23:0/4016540524 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c97d0 0x7fb808159330 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:04.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.675+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4016540524 >> 172.21.15.23:0/4016540524 conn(0x7fb80806ae30 msgr2=0x7fb80806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:04.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.675+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4016540524 shutdown_connections 2023-06-22T08:14:04.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.675+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4016540524 wait complete. 2023-06-22T08:14:04.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.677+0000 7fb80e7ff700 1 Processor -- start 2023-06-22T08:14:04.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.677+0000 7fb80e7ff700 1 -- start start 2023-06-22T08:14:04.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.678+0000 7fb80e7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c45f0 0x7fb8080cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:04.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.678+0000 7fb80e7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8080c97d0 0x7fb80814c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:04.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.678+0000 7fb80e7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808152d70 0x7fb80814c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:04.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.678+0000 7fb80e7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8081ef500 con 0x7fb8080c97d0 2023-06-22T08:14:04.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.678+0000 7fb80e7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8080cf0b0 con 0x7fb808152d70 2023-06-22T08:14:04.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.678+0000 7fb80e7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8081eef60 con 0x7fb8080c45f0 2023-06-22T08:14:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.679+0000 7fb8077fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8080c97d0 0x7fb80814c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.679+0000 7fb8077fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8080c97d0 0x7fb80814c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49644/0 (socket says 172.21.15.23:49644) 2023-06-22T08:14:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.679+0000 7fb8077fe700 1 -- 172.21.15.23:0/4153229091 learned_addr learned my addr 172.21.15.23:0/4153229091 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.679+0000 7fb807fff700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c45f0 0x7fb8080cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.679+0000 7fb8077fe700 1 -- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c45f0 msgr2=0x7fb8080cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.679+0000 7fb8077fe700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c45f0 0x7fb8080cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.679+0000 7fb8077fe700 1 -- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808152d70 msgr2=0x7fb80814c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.679+0000 7fb8077fe700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808152d70 0x7fb80814c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.679+0000 7fb8077fe700 1 -- 172.21.15.23:0/4153229091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc0094a0 con 0x7fb8080c97d0 2023-06-22T08:14:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.679+0000 7fb8077fe700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8080c97d0 0x7fb80814c200 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0090b0 tx=0x7fb7fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.680+0000 7fb804ff9700 1 -- 172.21.15.23:0/4153229091 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc01a760 con 0x7fb8080c97d0 2023-06-22T08:14:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.680+0000 7fb804ff9700 1 -- 172.21.15.23:0/4153229091 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc01a8c0 con 0x7fb8080c97d0 2023-06-22T08:14:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.680+0000 7fb804ff9700 1 -- 172.21.15.23:0/4153229091 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc01ac30 con 0x7fb8080c97d0 2023-06-22T08:14:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.680+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4153229091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb808153980 con 0x7fb8080c97d0 2023-06-22T08:14:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.680+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4153229091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8081ef220 con 0x7fb8080c97d0 2023-06-22T08:14:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.681+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4153229091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8080cef10 con 0x7fb8080c97d0 2023-06-22T08:14:04.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.682+0000 7fb804ff9700 1 -- 172.21.15.23:0/4153229091 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb7fc022070 con 0x7fb8080c97d0 2023-06-22T08:14:04.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.683+0000 7fb804ff9700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb7f406ace0 0x7fb7f406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:04.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.683+0000 7fb804ff9700 1 -- 172.21.15.23:0/4153229091 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7fb7fc08a7e0 con 0x7fb8080c97d0 2023-06-22T08:14:04.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.683+0000 7fb807fff700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb7f406ace0 0x7fb7f406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:04.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.684+0000 7fb807fff700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb7f406ace0 0x7fb7f406d190 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8000f60 tx=0x7fb7f8007e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:04.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.687+0000 7fb804ff9700 1 -- 172.21.15.23:0/4153229091 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb8080cef10 con 0x7fb8080c97d0 2023-06-22T08:14:04.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.954+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4153229091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8080cef10 con 0x7fb8080c97d0 2023-06-22T08:14:04.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.954+0000 7fb804ff9700 1 -- 172.21.15.23:0/4153229091 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5515 (secure 0 0 0) 0x7fb8080cef10 con 0x7fb8080c97d0 2023-06-22T08:14:04.958 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:04.959 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:04.482489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:04.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb7f406ace0 msgr2=0x7fb7f406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:04.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb7f406ace0 0x7fb7f406d190 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8000f60 tx=0x7fb7f8007e00 comp rx=0 tx=0).stop 2023-06-22T08:14:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4153229091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8080c97d0 msgr2=0x7fb80814c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8080c97d0 0x7fb80814c200 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0090b0 tx=0x7fb7fc004750 comp rx=0 tx=0).stop 2023-06-22T08:14:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4153229091 shutdown_connections 2023-06-22T08:14:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb7f406ace0 0x7fb7f406d190 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb808152d70 0x7fb80814c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb8080c45f0 0x7fb8080cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 --2- 172.21.15.23:0/4153229091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8080c97d0 0x7fb80814c200 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:04.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4153229091 >> 172.21.15.23:0/4153229091 conn(0x7fb80806ae30 msgr2=0x7fb8080ca710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:04.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4153229091 shutdown_connections 2023-06-22T08:14:04.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:04.957+0000 7fb80e7ff700 1 -- 172.21.15.23:0/4153229091 wait complete. 2023-06-22T08:14:04.962 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 109 2023-06-22T08:14:04.982 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-06-22T08:14:04.982 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:14:04.983 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:05.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.197+0000 7fbcfe1ff700 1 Processor -- start 2023-06-22T08:14:05.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.197+0000 7fbcfe1ff700 1 -- start start 2023-06-22T08:14:05.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.198+0000 7fbcfe1ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcf80c9fd0 0x7fbcf80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:05.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.198+0000 7fbcfe1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c40b0 0x7fbcf80c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:05.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.198+0000 7fbcfe1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf815c370 con 0x7fbcf8152ce0 2023-06-22T08:14:05.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.198+0000 7fbcfe1ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf81530b0 con 0x7fbcf80c9fd0 2023-06-22T08:14:05.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.198+0000 7fbcfe1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf80cf6f0 con 0x7fbcf80c40b0 2023-06-22T08:14:05.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf6ffd700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcf80c9fd0 0x7fbcf80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:05.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf77fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbcf8152ce0 0x7fbcf815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:41142/0 (socket says 172.21.15.23:41142) 2023-06-22T08:14:05.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf77fe700 1 -- 172.21.15.23:0/631125554 learned_addr learned my addr 172.21.15.23:0/631125554 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:14:05.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf7fff700 1 --2- 172.21.15.23:0/631125554 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c40b0 0x7fbcf80c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:05.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf6ffd700 1 -- 172.21.15.23:0/631125554 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c40b0 msgr2=0x7fbcf80c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:14:05.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf6ffd700 1 --2- 172.21.15.23:0/631125554 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c40b0 0x7fbcf80c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:05.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf6ffd700 1 -- 172.21.15.23:0/631125554 >> v1:172.21.15.67:6789/0 conn(0x7fbcf8152ce0 legacy=0x7fbcf815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:05.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf6ffd700 1 -- 172.21.15.23:0/631125554 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf815bda0 con 0x7fbcf80c9fd0 2023-06-22T08:14:05.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf7fff700 1 --2- 172.21.15.23:0/631125554 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c40b0 0x7fbcf80c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:14:05.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf67fc700 1 -- 172.21.15.23:0/631125554 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951840865 0 0) 0x7fbcf815c370 con 0x7fbcf8152ce0 2023-06-22T08:14:05.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.199+0000 7fbcf6ffd700 1 --2- 172.21.15.23:0/631125554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf80c9fd0 0x7fbcf80cf1b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbcec009840 tx=0x7fbcec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd65b329ca17179b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:05.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcf67fc700 1 -- 172.21.15.23:0/631125554 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec011830 con 0x7fbcf80c9fd0 2023-06-22T08:14:05.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcf67fc700 1 -- 172.21.15.23:0/631125554 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcec011990 con 0x7fbcf80c9fd0 2023-06-22T08:14:05.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcf67fc700 1 -- 172.21.15.23:0/631125554 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec011d20 con 0x7fbcf80c9fd0 2023-06-22T08:14:05.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/631125554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf80c9fd0 msgr2=0x7fbcf80cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:05.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcfe1ff700 1 --2- 172.21.15.23:0/631125554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf80c9fd0 0x7fbcf80cf1b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbcec009840 tx=0x7fbcec009b50 comp rx=0 tx=0).stop 2023-06-22T08:14:05.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/631125554 shutdown_connections 2023-06-22T08:14:05.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcfe1ff700 1 --2- 172.21.15.23:0/631125554 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c40b0 0x7fbcf80c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:05.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcfe1ff700 1 --2- 172.21.15.23:0/631125554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf80c9fd0 0x7fbcf80cf1b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:05.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/631125554 >> 172.21.15.23:0/631125554 conn(0x7fbcf806ae30 msgr2=0x7fbcf806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:05.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/631125554 shutdown_connections 2023-06-22T08:14:05.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.200+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/631125554 wait complete. 2023-06-22T08:14:05.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.202+0000 7fbcfe1ff700 1 Processor -- start 2023-06-22T08:14:05.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.202+0000 7fbcfe1ff700 1 -- start start 2023-06-22T08:14:05.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcfe1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf80c40b0 0x7fbcf81e1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:05.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcfe1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c9fd0 0x7fbcf814da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:05.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcfe1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcf8152ce0 0x7fbcf814a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:05.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcfe1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf8159a60 con 0x7fbcf8152ce0 2023-06-22T08:14:05.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcfe1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf81e42a0 con 0x7fbcf80c40b0 2023-06-22T08:14:05.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcfe1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf81e3f10 con 0x7fbcf80c9fd0 2023-06-22T08:14:05.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcf7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcf8152ce0 0x7fbcf814a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:05.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcf77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf80c40b0 0x7fbcf81e1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:05.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcf7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcf8152ce0 0x7fbcf814a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49660/0 (socket says 172.21.15.23:49660) 2023-06-22T08:14:05.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcf7fff700 1 -- 172.21.15.23:0/1845772792 learned_addr learned my addr 172.21.15.23:0/1845772792 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:05.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.203+0000 7fbcf6ffd700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c9fd0 0x7fbcf814da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:05.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.204+0000 7fbcf77fe700 1 -- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c9fd0 msgr2=0x7fbcf814da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:05.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.204+0000 7fbcf77fe700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c9fd0 0x7fbcf814da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:05.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.204+0000 7fbcf77fe700 1 -- 172.21.15.23:0/1845772792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcf8152ce0 msgr2=0x7fbcf814a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:05.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.204+0000 7fbcf77fe700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcf8152ce0 0x7fbcf814a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:05.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.204+0000 7fbcf77fe700 1 -- 172.21.15.23:0/1845772792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf0009a70 con 0x7fbcf80c40b0 2023-06-22T08:14:05.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.204+0000 7fbcf77fe700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf80c40b0 0x7fbcf81e1510 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbce800c410 tx=0x7fbce800c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:05.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.204+0000 7fbcdffff700 1 -- 172.21.15.23:0/1845772792 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce8004c80 con 0x7fbcf80c40b0 2023-06-22T08:14:05.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.204+0000 7fbcdffff700 1 -- 172.21.15.23:0/1845772792 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce8004de0 con 0x7fbcf80c40b0 2023-06-22T08:14:05.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.204+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/1845772792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcec0094a0 con 0x7fbcf80c40b0 2023-06-22T08:14:05.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.204+0000 7fbcdffff700 1 -- 172.21.15.23:0/1845772792 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce8007310 con 0x7fbcf80c40b0 2023-06-22T08:14:05.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.205+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/1845772792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf80ce020 con 0x7fbcf80c40b0 2023-06-22T08:14:05.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.205+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/1845772792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf804ea20 con 0x7fbcf80c40b0 2023-06-22T08:14:05.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.206+0000 7fbcdffff700 1 -- 172.21.15.23:0/1845772792 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbce8007470 con 0x7fbcf80c40b0 2023-06-22T08:14:05.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.207+0000 7fbcdffff700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbce406ace0 0x7fbce406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:05.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.207+0000 7fbcdffff700 1 -- 172.21.15.23:0/1845772792 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7fbce8089580 con 0x7fbcf80c40b0 2023-06-22T08:14:05.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.208+0000 7fbcf6ffd700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbce406ace0 0x7fbce406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:05.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.208+0000 7fbcf6ffd700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbce406ace0 0x7fbce406d190 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0090b0 tx=0x7fbcec008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:05.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.212+0000 7fbcdffff700 1 -- 172.21.15.23:0/1845772792 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbce8054080 con 0x7fbcf80c40b0 2023-06-22T08:14:05.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.466+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/1845772792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcf815bff0 con 0x7fbcf80c40b0 2023-06-22T08:14:05.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.467+0000 7fbcdffff700 1 -- 172.21.15.23:0/1845772792 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5515 (secure 0 0 0) 0x7fbce80576a0 con 0x7fbcf80c40b0 2023-06-22T08:14:05.470 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:05.470 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:04.482489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:05.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbce406ace0 msgr2=0x7fbce406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbce406ace0 0x7fbce406d190 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0090b0 tx=0x7fbcec008030 comp rx=0 tx=0).stop 2023-06-22T08:14:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf80c40b0 msgr2=0x7fbcf81e1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf80c40b0 0x7fbcf81e1510 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbce800c410 tx=0x7fbce800c720 comp rx=0 tx=0).stop 2023-06-22T08:14:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/1845772792 shutdown_connections 2023-06-22T08:14:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcf8152ce0 0x7fbcf814a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:05.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbce406ace0 0x7fbce406d190 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf80c40b0 0x7fbcf81e1510 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 --2- 172.21.15.23:0/1845772792 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf80c9fd0 0x7fbcf814da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/1845772792 >> 172.21.15.23:0/1845772792 conn(0x7fbcf806ae30 msgr2=0x7fbcf806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/1845772792 shutdown_connections 2023-06-22T08:14:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:05.469+0000 7fbcfe1ff700 1 -- 172.21.15.23:0/1845772792 wait complete. 2023-06-22T08:14:05.474 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 109 2023-06-22T08:14:05.491 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 109, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 109, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 26 }, "gid_5576": { "addr": "172.21.15.67:6837/2064462140", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 2064462140, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 2064462140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5576, "incarnation": 109, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:14:04.482489+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 5551, "mds_3": 5995, "mds_4": 5576 } } } ], "standbys": [] } 2023-06-22T08:14:07.493 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:07.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.692+0000 7ff03b5ff700 1 Processor -- start 2023-06-22T08:14:07.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.692+0000 7ff03b5ff700 1 -- start start 2023-06-22T08:14:07.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.693+0000 7ff03b5ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff03415a320 0x7ff0340c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:07.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.693+0000 7ff03b5ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0340c6a60 con 0x7ff03406d050 2023-06-22T08:14:07.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.693+0000 7ff03b5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff03414d690 con 0x7ff0340c3a40 2023-06-22T08:14:07.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.693+0000 7ff03b5ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff034157570 con 0x7ff03415a320 2023-06-22T08:14:07.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.693+0000 7ff038b9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff03415a320 0x7ff0340c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:07.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.693+0000 7ff038b9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff03415a320 0x7ff0340c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53898/0 (socket says 172.21.15.23:53898) 2023-06-22T08:14:07.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.693+0000 7ff039b9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff03406d050 0x7ff034160ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:47258/0 (socket says 172.21.15.23:47258) 2023-06-22T08:14:07.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.693+0000 7ff038b9a700 1 -- 172.21.15.23:0/616426984 learned_addr learned my addr 172.21.15.23:0/616426984 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:07.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.694+0000 7ff038b9a700 1 -- 172.21.15.23:0/616426984 >> v1:172.21.15.67:6789/0 conn(0x7ff0340c3a40 legacy=0x7ff034159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:07.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.694+0000 7ff038b9a700 1 -- 172.21.15.23:0/616426984 >> v1:172.21.15.23:6789/0 conn(0x7ff03406d050 legacy=0x7ff034160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:07.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.694+0000 7ff038b9a700 1 -- 172.21.15.23:0/616426984 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0340c6430 con 0x7ff03415a320 2023-06-22T08:14:07.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.694+0000 7ff038b9a700 1 --2- 172.21.15.23:0/616426984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff03415a320 0x7ff0340c3e10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff0240098f0 tx=0x7ff024009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87b73337fa67867 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:07.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.694+0000 7ff02bfff700 1 -- 172.21.15.23:0/616426984 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02400b8a0 con 0x7ff03415a320 2023-06-22T08:14:07.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.694+0000 7ff02bfff700 1 -- 172.21.15.23:0/616426984 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02400ba00 con 0x7ff03415a320 2023-06-22T08:14:07.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.694+0000 7ff02bfff700 1 -- 172.21.15.23:0/616426984 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02400bd90 con 0x7ff03415a320 2023-06-22T08:14:07.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.695+0000 7ff03b5ff700 1 -- 172.21.15.23:0/616426984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff03415a320 msgr2=0x7ff0340c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:07.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.695+0000 7ff03b5ff700 1 --2- 172.21.15.23:0/616426984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff03415a320 0x7ff0340c3e10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff0240098f0 tx=0x7ff024009c00 comp rx=0 tx=0).stop 2023-06-22T08:14:07.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.695+0000 7ff03b5ff700 1 -- 172.21.15.23:0/616426984 shutdown_connections 2023-06-22T08:14:07.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.695+0000 7ff03b5ff700 1 --2- 172.21.15.23:0/616426984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff03415a320 0x7ff0340c3e10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:07.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.695+0000 7ff03b5ff700 1 -- 172.21.15.23:0/616426984 >> 172.21.15.23:0/616426984 conn(0x7ff0340ca3f0 msgr2=0x7ff0340ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:07.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.695+0000 7ff03b5ff700 1 -- 172.21.15.23:0/616426984 shutdown_connections 2023-06-22T08:14:07.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.695+0000 7ff03b5ff700 1 -- 172.21.15.23:0/616426984 wait complete. 2023-06-22T08:14:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.697+0000 7ff03b5ff700 1 Processor -- start 2023-06-22T08:14:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.697+0000 7ff03b5ff700 1 -- start start 2023-06-22T08:14:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff03b5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03406d050 0x7ff0340c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff03b5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff0340c3a40 0x7ff0341edd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff03b5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff03415a320 0x7ff0341f0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:07.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff03b5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0340c9060 con 0x7ff03415a320 2023-06-22T08:14:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff03b5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0340c9900 con 0x7ff03406d050 2023-06-22T08:14:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff03b5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03414efc0 con 0x7ff0340c3a40 2023-06-22T08:14:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff038b9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff0340c3a40 0x7ff0341edd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff03939b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03406d050 0x7ff0340c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff039b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff03415a320 0x7ff0341f0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff038b9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff0340c3a40 0x7ff0341edd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:47444/0 (socket says 172.21.15.23:47444) 2023-06-22T08:14:07.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff03939b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03406d050 0x7ff0340c9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50294/0 (socket says 172.21.15.23:50294) 2023-06-22T08:14:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.698+0000 7ff038b9a700 1 -- 172.21.15.23:0/2885835659 learned_addr learned my addr 172.21.15.23:0/2885835659 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff039b9c700 1 -- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff0340c3a40 msgr2=0x7ff0341edd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff039b9c700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff0340c3a40 0x7ff0341edd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff039b9c700 1 -- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03406d050 msgr2=0x7ff0340c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff039b9c700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03406d050 0x7ff0340c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff039b9c700 1 -- 172.21.15.23:0/2885835659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff024009550 con 0x7ff03415a320 2023-06-22T08:14:07.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff039b9c700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff03415a320 0x7ff0341f0080 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00eeb0 tx=0x7ff02c010330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff029ffb700 1 -- 172.21.15.23:0/2885835659 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c013070 con 0x7ff03415a320 2023-06-22T08:14:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff029ffb700 1 -- 172.21.15.23:0/2885835659 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c00e2e0 con 0x7ff03415a320 2023-06-22T08:14:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff029ffb700 1 -- 172.21.15.23:0/2885835659 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c010b10 con 0x7ff03415a320 2023-06-22T08:14:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff03b5ff700 1 -- 172.21.15.23:0/2885835659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0341405f0 con 0x7ff03415a320 2023-06-22T08:14:07.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.699+0000 7ff03b5ff700 1 -- 172.21.15.23:0/2885835659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0340c9240 con 0x7ff03415a320 2023-06-22T08:14:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.700+0000 7ff03b5ff700 1 -- 172.21.15.23:0/2885835659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0340c9a40 con 0x7ff03415a320 2023-06-22T08:14:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.701+0000 7ff029ffb700 1 -- 172.21.15.23:0/2885835659 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff02c010c70 con 0x7ff03415a320 2023-06-22T08:14:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.702+0000 7ff029ffb700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff01806ace0 0x7ff01806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.702+0000 7ff029ffb700 1 -- 172.21.15.23:0/2885835659 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7ff02c08da60 con 0x7ff03415a320 2023-06-22T08:14:07.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.702+0000 7ff03939b700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff01806ace0 0x7ff01806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:07.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.703+0000 7ff03939b700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff01806ace0 0x7ff01806d190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff030007fd0 tx=0x7ff030009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:07.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.706+0000 7ff029ffb700 1 -- 172.21.15.23:0/2885835659 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff02c0584d0 con 0x7ff03415a320 2023-06-22T08:14:07.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.962+0000 7ff03b5ff700 1 -- 172.21.15.23:0/2885835659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff03406cc00 con 0x7ff03415a320 2023-06-22T08:14:07.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.963+0000 7ff029ffb700 1 -- 172.21.15.23:0/2885835659 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5515 (secure 0 0 0) 0x7ff02c05baf0 con 0x7ff03415a320 2023-06-22T08:14:07.966 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:07.967 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:04.482489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:07.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.965+0000 7ff03b5ff700 1 -- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff01806ace0 msgr2=0x7ff01806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:07.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.965+0000 7ff03b5ff700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff01806ace0 0x7ff01806d190 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff030007fd0 tx=0x7ff030009040 comp rx=0 tx=0).stop 2023-06-22T08:14:07.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.965+0000 7ff03b5ff700 1 -- 172.21.15.23:0/2885835659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff03415a320 msgr2=0x7ff0341f0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:07.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.965+0000 7ff03b5ff700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff03415a320 0x7ff0341f0080 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00eeb0 tx=0x7ff02c010330 comp rx=0 tx=0).stop 2023-06-22T08:14:07.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.965+0000 7ff03b5ff700 1 -- 172.21.15.23:0/2885835659 shutdown_connections 2023-06-22T08:14:07.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.965+0000 7ff03b5ff700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff03415a320 0x7ff0341f0080 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:07.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.965+0000 7ff03b5ff700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff01806ace0 0x7ff01806d190 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:07.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.965+0000 7ff03b5ff700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff03406d050 0x7ff0340c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:07.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.965+0000 7ff03b5ff700 1 --2- 172.21.15.23:0/2885835659 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff0340c3a40 0x7ff0341edd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:07.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.965+0000 7ff03b5ff700 1 -- 172.21.15.23:0/2885835659 >> 172.21.15.23:0/2885835659 conn(0x7ff0340ca3f0 msgr2=0x7ff0340c76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:07.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.966+0000 7ff03b5ff700 1 -- 172.21.15.23:0/2885835659 shutdown_connections 2023-06-22T08:14:07.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:07.966+0000 7ff03b5ff700 1 -- 172.21.15.23:0/2885835659 wait complete. 2023-06-22T08:14:07.971 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 109 2023-06-22T08:14:07.987 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:14:09.444 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:14:09.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:14:09.988 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.201+0000 7f286bbff700 1 Processor -- start 2023-06-22T08:14:10.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.201+0000 7f286bbff700 1 -- start start 2023-06-22T08:14:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.202+0000 7f286bbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28640c4a30 0x7f28640cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.202+0000 7f286bbff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28640cb7a0 0x7f286415c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.202+0000 7f286bbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2864159730 con 0x7f2864152a10 2023-06-22T08:14:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.202+0000 7f286bbff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28640c5470 con 0x7f28640cb7a0 2023-06-22T08:14:10.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.202+0000 7f286bbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2864158910 con 0x7f28640c4a30 2023-06-22T08:14:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.202+0000 7f286919a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28640c4a30 0x7f28640cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f286a19c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f28640cb7a0 0x7f286415c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f286999b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2864152a10 0x7f28640c9200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:51652/0 (socket says 172.21.15.23:51652) 2023-06-22T08:14:10.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f286999b700 1 -- 172.21.15.23:0/4293599655 learned_addr learned my addr 172.21.15.23:0/4293599655 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:14:10.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f2868999700 1 -- 172.21.15.23:0/4293599655 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522102101 0 0) 0x7f2864159730 con 0x7f2864152a10 2023-06-22T08:14:10.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f2868999700 1 -- 172.21.15.23:0/4293599655 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28500035b0 con 0x7f2864152a10 2023-06-22T08:14:10.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f286a19c700 1 -- 172.21.15.23:0/4293599655 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28640c4a30 msgr2=0x7f28640cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:14:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f286a19c700 1 --2- 172.21.15.23:0/4293599655 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28640c4a30 0x7f28640cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f286a19c700 1 -- 172.21.15.23:0/4293599655 >> v1:172.21.15.67:6789/0 conn(0x7f2864152a10 legacy=0x7f28640c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:10.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f286a19c700 1 -- 172.21.15.23:0/4293599655 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2864158bb0 con 0x7f28640cb7a0 2023-06-22T08:14:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f286919a700 1 --2- 172.21.15.23:0/4293599655 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28640c4a30 0x7f28640cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:14:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f286a19c700 1 --2- 172.21.15.23:0/4293599655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 0x7f286415c0b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f285c009d90 tx=0x7f285c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8bb632fc9f00d26 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.203+0000 7f2868999700 1 -- 172.21.15.23:0/4293599655 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c00e070 con 0x7f28640cb7a0 2023-06-22T08:14:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.204+0000 7f2868999700 1 -- 172.21.15.23:0/4293599655 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c00bb20 con 0x7f28640cb7a0 2023-06-22T08:14:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.204+0000 7f2868999700 1 -- 172.21.15.23:0/4293599655 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c012430 con 0x7f28640cb7a0 2023-06-22T08:14:10.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.204+0000 7f286bbff700 1 -- 172.21.15.23:0/4293599655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 msgr2=0x7f286415c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:10.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.204+0000 7f286bbff700 1 --2- 172.21.15.23:0/4293599655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 0x7f286415c0b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f285c009d90 tx=0x7f285c00b330 comp rx=0 tx=0).stop 2023-06-22T08:14:10.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.204+0000 7f286bbff700 1 -- 172.21.15.23:0/4293599655 shutdown_connections 2023-06-22T08:14:10.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.204+0000 7f286bbff700 1 --2- 172.21.15.23:0/4293599655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 0x7f286415c0b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:10.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.204+0000 7f286bbff700 1 --2- 172.21.15.23:0/4293599655 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f28640c4a30 0x7f28640cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:10.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.204+0000 7f286bbff700 1 -- 172.21.15.23:0/4293599655 >> 172.21.15.23:0/4293599655 conn(0x7f2864065b90 msgr2=0x7f2864065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:10.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.204+0000 7f286bbff700 1 -- 172.21.15.23:0/4293599655 shutdown_connections 2023-06-22T08:14:10.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.204+0000 7f286bbff700 1 -- 172.21.15.23:0/4293599655 wait complete. 2023-06-22T08:14:10.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.207+0000 7f286bbff700 1 Processor -- start 2023-06-22T08:14:10.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.207+0000 7f286bbff700 1 -- start start 2023-06-22T08:14:10.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286bbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28640c4a30 0x7f2864153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:10.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286bbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 0x7f28640b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:10.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286bbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2864152a10 0x7f2864157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:10.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286bbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2864153fa0 con 0x7f28640c4a30 2023-06-22T08:14:10.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286bbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28640b32b0 con 0x7f28640cb7a0 2023-06-22T08:14:10.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286bbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28640b4350 con 0x7f2864152a10 2023-06-22T08:14:10.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286919a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 0x7f28640b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:10.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286999b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28640c4a30 0x7f2864153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:10.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286919a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 0x7f28640b3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50310/0 (socket says 172.21.15.23:50310) 2023-06-22T08:14:10.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286919a700 1 -- 172.21.15.23:0/770256600 learned_addr learned my addr 172.21.15.23:0/770256600 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:10.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.208+0000 7f286a19c700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2864152a10 0x7f2864157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:10.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f286919a700 1 -- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2864152a10 msgr2=0x7f2864157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:10.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f286919a700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2864152a10 0x7f2864157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:10.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f286919a700 1 -- 172.21.15.23:0/770256600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28640c4a30 msgr2=0x7f2864153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:10.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f286919a700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28640c4a30 0x7f2864153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:10.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f286919a700 1 -- 172.21.15.23:0/770256600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28540094a0 con 0x7f28640cb7a0 2023-06-22T08:14:10.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f286919a700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 0x7f28640b3a70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f285400ec10 tx=0x7f285400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:10.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f285a7fc700 1 -- 172.21.15.23:0/770256600 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285400c850 con 0x7f28640cb7a0 2023-06-22T08:14:10.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f285a7fc700 1 -- 172.21.15.23:0/770256600 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285400c9b0 con 0x7f28640cb7a0 2023-06-22T08:14:10.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f286bbff700 1 -- 172.21.15.23:0/770256600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c009a70 con 0x7f28640cb7a0 2023-06-22T08:14:10.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f285a7fc700 1 -- 172.21.15.23:0/770256600 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285400cd20 con 0x7f28640cb7a0 2023-06-22T08:14:10.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.209+0000 7f286bbff700 1 -- 172.21.15.23:0/770256600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28640b3550 con 0x7f28640cb7a0 2023-06-22T08:14:10.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.210+0000 7f286bbff700 1 -- 172.21.15.23:0/770256600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28640b4870 con 0x7f28640cb7a0 2023-06-22T08:14:10.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.211+0000 7f285a7fc700 1 -- 172.21.15.23:0/770256600 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2854013470 con 0x7f28640cb7a0 2023-06-22T08:14:10.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.212+0000 7f285a7fc700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284806ace0 0x7f284806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:10.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.212+0000 7f285a7fc700 1 -- 172.21.15.23:0/770256600 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7f2854089f90 con 0x7f28640cb7a0 2023-06-22T08:14:10.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.212+0000 7f286999b700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284806ace0 0x7f284806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:10.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.213+0000 7f286999b700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284806ace0 0x7f284806d190 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2860007fd0 tx=0x7f2860007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:10.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.217+0000 7f285a7fc700 1 -- 172.21.15.23:0/770256600 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f28640b4870 con 0x7f28640cb7a0 2023-06-22T08:14:10.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.475+0000 7f286bbff700 1 -- 172.21.15.23:0/770256600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28640b4870 con 0x7f28640cb7a0 2023-06-22T08:14:10.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.476+0000 7f285a7fc700 1 -- 172.21.15.23:0/770256600 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5515 (secure 0 0 0) 0x7f28640b4870 con 0x7f28640cb7a0 2023-06-22T08:14:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:10.479 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:04.482489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:10.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 -- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284806ace0 msgr2=0x7f284806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:10.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284806ace0 0x7f284806d190 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2860007fd0 tx=0x7f2860007dc0 comp rx=0 tx=0).stop 2023-06-22T08:14:10.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 -- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 msgr2=0x7f28640b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:10.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 0x7f28640b3a70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f285400ec10 tx=0x7f285400ef20 comp rx=0 tx=0).stop 2023-06-22T08:14:10.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 -- 172.21.15.23:0/770256600 shutdown_connections 2023-06-22T08:14:10.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2864152a10 0x7f2864157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28640c4a30 0x7f2864153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284806ace0 0x7f284806d190 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 --2- 172.21.15.23:0/770256600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28640cb7a0 0x7f28640b3a70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 -- 172.21.15.23:0/770256600 >> 172.21.15.23:0/770256600 conn(0x7f2864065b90 msgr2=0x7f2864065f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 -- 172.21.15.23:0/770256600 shutdown_connections 2023-06-22T08:14:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:10.478+0000 7f286bbff700 1 -- 172.21.15.23:0/770256600 wait complete. 2023-06-22T08:14:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 109 2023-06-22T08:14:10.499 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:14:12.500 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:12.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.706+0000 7f6fcffff700 1 Processor -- start 2023-06-22T08:14:12.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.707+0000 7f6fcffff700 1 -- start start 2023-06-22T08:14:12.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.708+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8152ce0 0x7f6fc80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:12.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.708+0000 7f6fcffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6fc80c9fd0 0x7f6fc815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:12.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.708+0000 7f6fcffff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc815c370 con 0x7f6fc8158880 2023-06-22T08:14:12.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.708+0000 7f6fcffff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc81530b0 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.708+0000 7f6fcffff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc80cf6f0 con 0x7f6fc8152ce0 2023-06-22T08:14:12.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.708+0000 7f6fcd59a700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6fc80c9fd0 0x7f6fc815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:12.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.708+0000 7f6fce59c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6fc8158880 0x7f6fc8158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:51658/0 (socket says 172.21.15.23:51658) 2023-06-22T08:14:12.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.708+0000 7f6fcd59a700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6fc80c9fd0 0x7f6fc815c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50322/0 (socket says 172.21.15.23:50322) 2023-06-22T08:14:12.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.708+0000 7f6fcdd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8152ce0 0x7f6fc80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:12.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.708+0000 7f6fce59c700 1 -- 172.21.15.23:0/817496291 learned_addr learned my addr 172.21.15.23:0/817496291 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:14:12.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.709+0000 7f6fcd59a700 1 -- 172.21.15.23:0/817496291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8152ce0 msgr2=0x7f6fc80cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:14:12.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.709+0000 7f6fcd59a700 1 --2- 172.21.15.23:0/817496291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8152ce0 0x7f6fc80cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:12.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.709+0000 7f6fcd59a700 1 -- 172.21.15.23:0/817496291 >> v1:172.21.15.67:6789/0 conn(0x7f6fc8158880 legacy=0x7f6fc8158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:12.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.709+0000 7f6fcd59a700 1 -- 172.21.15.23:0/817496291 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc815bda0 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.709+0000 7f6fcdd9b700 1 --2- 172.21.15.23:0/817496291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8152ce0 0x7f6fc80cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:14:12.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.709+0000 7f6fccd99700 1 -- 172.21.15.23:0/817496291 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623328223 0 0) 0x7f6fc815c370 con 0x7f6fc8158880 2023-06-22T08:14:12.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.709+0000 7f6fcd59a700 1 --2- 172.21.15.23:0/817496291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fc80c9fd0 0x7f6fc815c800 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009840 tx=0x7f6fb8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=274f4697ee0adb8f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:12.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.709+0000 7f6fccd99700 1 -- 172.21.15.23:0/817496291 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8011830 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.709+0000 7f6fccd99700 1 -- 172.21.15.23:0/817496291 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb8011990 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.709+0000 7f6fccd99700 1 -- 172.21.15.23:0/817496291 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8011d20 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.710+0000 7f6fcffff700 1 -- 172.21.15.23:0/817496291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fc80c9fd0 msgr2=0x7f6fc815c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:12.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.710+0000 7f6fcffff700 1 --2- 172.21.15.23:0/817496291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fc80c9fd0 0x7f6fc815c800 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009840 tx=0x7f6fb8009b50 comp rx=0 tx=0).stop 2023-06-22T08:14:12.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.710+0000 7f6fcffff700 1 -- 172.21.15.23:0/817496291 shutdown_connections 2023-06-22T08:14:12.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.710+0000 7f6fcffff700 1 --2- 172.21.15.23:0/817496291 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8152ce0 0x7f6fc80cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:12.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.710+0000 7f6fcffff700 1 --2- 172.21.15.23:0/817496291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fc80c9fd0 0x7f6fc815c800 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:12.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.710+0000 7f6fcffff700 1 -- 172.21.15.23:0/817496291 >> 172.21.15.23:0/817496291 conn(0x7f6fc806ae30 msgr2=0x7f6fc806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:12.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.710+0000 7f6fcffff700 1 -- 172.21.15.23:0/817496291 shutdown_connections 2023-06-22T08:14:12.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.710+0000 7f6fcffff700 1 -- 172.21.15.23:0/817496291 wait complete. 2023-06-22T08:14:12.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.712+0000 7f6fcffff700 1 Processor -- start 2023-06-22T08:14:12.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.712+0000 7f6fcffff700 1 -- start start 2023-06-22T08:14:12.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fc80c9fd0 0x7f6fc81e20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:12.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fc8152ce0 0x7f6fc81ef6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:12.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8158880 0x7f6fc81efc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:12.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81e4470 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcffff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81e37f0 con 0x7f6fc8152ce0 2023-06-22T08:14:12.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcffff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc81e3ad0 con 0x7f6fc8158880 2023-06-22T08:14:12.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcdd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fc80c9fd0 0x7f6fc81e20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:12.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcd59a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fc8152ce0 0x7f6fc81ef6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:12.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcdd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fc80c9fd0 0x7f6fc81e20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53928/0 (socket says 172.21.15.23:53928) 2023-06-22T08:14:12.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcdd9b700 1 -- 172.21.15.23:0/3288721863 learned_addr learned my addr 172.21.15.23:0/3288721863 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:12.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fce59c700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8158880 0x7f6fc81efc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:12.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcdd9b700 1 -- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8158880 msgr2=0x7f6fc81efc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:12.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.713+0000 7f6fcdd9b700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8158880 0x7f6fc81efc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:12.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.714+0000 7f6fcdd9b700 1 -- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fc8152ce0 msgr2=0x7f6fc81ef6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:12.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.714+0000 7f6fcdd9b700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fc8152ce0 0x7f6fc81ef6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:12.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.714+0000 7f6fcdd9b700 1 -- 172.21.15.23:0/3288721863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc40094a0 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.714+0000 7f6fcd59a700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fc8152ce0 0x7f6fc81ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:14:12.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.714+0000 7f6fcdd9b700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fc80c9fd0 0x7f6fc81e20b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400ebe0 tx=0x7f6fc400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:12.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.714+0000 7f6fbe7fc700 1 -- 172.21.15.23:0/3288721863 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400c850 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.714+0000 7f6fbe7fc700 1 -- 172.21.15.23:0/3288721863 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc400c9b0 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.714+0000 7f6fbe7fc700 1 -- 172.21.15.23:0/3288721863 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc400cd40 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.714+0000 7f6fcffff700 1 -- 172.21.15.23:0/3288721863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb80094a0 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.714+0000 7f6fcffff700 1 -- 172.21.15.23:0/3288721863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc81e4090 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.715+0000 7f6fcffff700 1 -- 172.21.15.23:0/3288721863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc81586a0 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.716+0000 7f6fbe7fc700 1 -- 172.21.15.23:0/3288721863 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6fc4013470 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.717+0000 7f6fbe7fc700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fac06ad10 0x7f6fac06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:12.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.717+0000 7f6fbe7fc700 1 -- 172.21.15.23:0/3288721863 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7f6fc4089be0 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.718+0000 7f6fcd59a700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fac06ad10 0x7f6fac06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:12.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.718+0000 7f6fcd59a700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fac06ad10 0x7f6fac06d1c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80090b0 tx=0x7f6fb801c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:12.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.722+0000 7f6fbe7fc700 1 -- 172.21.15.23:0/3288721863 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6fc40545c0 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.976+0000 7f6fcffff700 1 -- 172.21.15.23:0/3288721863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fc804ea20 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.976+0000 7f6fbe7fc700 1 -- 172.21.15.23:0/3288721863 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+5516 (secure 0 0 0) 0x7f6fc4017090 con 0x7f6fc80c9fd0 2023-06-22T08:14:12.980 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:12.980 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:11.919491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:resolve","state_seq":9,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:12.982 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 -- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fac06ad10 msgr2=0x7f6fac06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:12.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fac06ad10 0x7f6fac06d1c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80090b0 tx=0x7f6fb801c040 comp rx=0 tx=0).stop 2023-06-22T08:14:12.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 -- 172.21.15.23:0/3288721863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fc80c9fd0 msgr2=0x7f6fc81e20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:12.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fc80c9fd0 0x7f6fc81e20b0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6fc400ebe0 tx=0x7f6fc400eef0 comp rx=0 tx=0).stop 2023-06-22T08:14:12.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 -- 172.21.15.23:0/3288721863 shutdown_connections 2023-06-22T08:14:12.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6fc8158880 0x7f6fc81efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:12.983 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fc80c9fd0 0x7f6fc81e20b0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:12.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fac06ad10 0x7f6fac06d1c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:12.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 --2- 172.21.15.23:0/3288721863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fc8152ce0 0x7f6fc81ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:12.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 -- 172.21.15.23:0/3288721863 >> 172.21.15.23:0/3288721863 conn(0x7f6fc806ae30 msgr2=0x7f6fc81564e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:12.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 -- 172.21.15.23:0/3288721863 shutdown_connections 2023-06-22T08:14:12.984 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:12.979+0000 7f6fcffff700 1 -- 172.21.15.23:0/3288721863 wait complete. 2023-06-22T08:14:12.985 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 110 2023-06-22T08:14:13.001 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:14:15.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:15.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.208+0000 7f2866bff700 1 Processor -- start 2023-06-22T08:14:15.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.208+0000 7f2866bff700 1 -- start start 2023-06-22T08:14:15.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.209+0000 7f2866bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2860152ce0 0x7f28600cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:15.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.209+0000 7f2866bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286015c370 con 0x7f28600c9fd0 2023-06-22T08:14:15.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.209+0000 7f2866bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286015bda0 con 0x7f28600c40b0 2023-06-22T08:14:15.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.209+0000 7f2866bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28601530b0 con 0x7f2860152ce0 2023-06-22T08:14:15.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f286499b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2860152ce0 0x7f28600cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:15.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f286499b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2860152ce0 0x7f28600cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53936/0 (socket says 172.21.15.23:53936) 2023-06-22T08:14:15.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f286499b700 1 -- 172.21.15.23:0/1910180169 learned_addr learned my addr 172.21.15.23:0/1910180169 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:15.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f286519c700 1 --1- 172.21.15.23:0/1910180169 >> v1:172.21.15.67:6789/0 conn(0x7f28600c40b0 0x7f28600c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:51660/0 (socket says 172.21.15.23:51660) 2023-06-22T08:14:15.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f285f7fe700 1 -- 172.21.15.23:0/1910180169 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401224038 0 0) 0x7f286015bda0 con 0x7f28600c40b0 2023-06-22T08:14:15.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f285f7fe700 1 -- 172.21.15.23:0/1910180169 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28500035b0 con 0x7f28600c40b0 2023-06-22T08:14:15.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f285f7fe700 1 -- 172.21.15.23:0/1910180169 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471415522 0 0) 0x7f286015c370 con 0x7f28600c9fd0 2023-06-22T08:14:15.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f286499b700 1 -- 172.21.15.23:0/1910180169 >> v1:172.21.15.67:6789/0 conn(0x7f28600c40b0 legacy=0x7f28600c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:15.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f286499b700 1 -- 172.21.15.23:0/1910180169 >> v1:172.21.15.23:6789/0 conn(0x7f28600c9fd0 legacy=0x7f286015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:15.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f286499b700 1 -- 172.21.15.23:0/1910180169 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600c9460 con 0x7f2860152ce0 2023-06-22T08:14:15.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.210+0000 7f286499b700 1 --2- 172.21.15.23:0/1910180169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2860152ce0 0x7f28600cf1b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f28540098f0 tx=0x7f2854009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381e291927c0aabc server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:15.216 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.211+0000 7f285f7fe700 1 -- 172.21.15.23:0/1910180169 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854011830 con 0x7f2860152ce0 2023-06-22T08:14:15.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.211+0000 7f285f7fe700 1 -- 172.21.15.23:0/1910180169 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2854011990 con 0x7f2860152ce0 2023-06-22T08:14:15.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.211+0000 7f285f7fe700 1 -- 172.21.15.23:0/1910180169 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854011d20 con 0x7f2860152ce0 2023-06-22T08:14:15.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.211+0000 7f2866bff700 1 -- 172.21.15.23:0/1910180169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2860152ce0 msgr2=0x7f28600cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:15.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.211+0000 7f2866bff700 1 --2- 172.21.15.23:0/1910180169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2860152ce0 0x7f28600cf1b0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f28540098f0 tx=0x7f2854009c00 comp rx=0 tx=0).stop 2023-06-22T08:14:15.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.211+0000 7f2866bff700 1 -- 172.21.15.23:0/1910180169 shutdown_connections 2023-06-22T08:14:15.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.211+0000 7f2866bff700 1 --2- 172.21.15.23:0/1910180169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2860152ce0 0x7f28600cf1b0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:15.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.211+0000 7f2866bff700 1 -- 172.21.15.23:0/1910180169 >> 172.21.15.23:0/1910180169 conn(0x7f286006ae30 msgr2=0x7f286006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:15.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.211+0000 7f2866bff700 1 -- 172.21.15.23:0/1910180169 shutdown_connections 2023-06-22T08:14:15.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.211+0000 7f2866bff700 1 -- 172.21.15.23:0/1910180169 wait complete. 2023-06-22T08:14:15.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.213+0000 7f2866bff700 1 Processor -- start 2023-06-22T08:14:15.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.213+0000 7f2866bff700 1 -- start start 2023-06-22T08:14:15.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.214+0000 7f2866bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c40b0 0x7f2860153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:15.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.214+0000 7f2866bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28600c9fd0 0x7f28601edf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:15.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.214+0000 7f2866bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2860152ce0 0x7f28601f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:15.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.214+0000 7f2866bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28601f1470 con 0x7f28600c9fd0 2023-06-22T08:14:15.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.214+0000 7f2866bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28601f1a30 con 0x7f28600c40b0 2023-06-22T08:14:15.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.214+0000 7f2866bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2860153290 con 0x7f2860152ce0 2023-06-22T08:14:15.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.214+0000 7f285ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28600c9fd0 0x7f28601edf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:15.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.214+0000 7f285ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28600c9fd0 0x7f28601edf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53948/0 (socket says 172.21.15.23:53948) 2023-06-22T08:14:15.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f285ffff700 1 -- 172.21.15.23:0/2678685472 learned_addr learned my addr 172.21.15.23:0/2678685472 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:15.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f286499b700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c40b0 0x7f2860153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:15.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f286519c700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2860152ce0 0x7f28601f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:15.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f285ffff700 1 -- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2860152ce0 msgr2=0x7f28601f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:15.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f285ffff700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2860152ce0 0x7f28601f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:15.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f285ffff700 1 -- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c40b0 msgr2=0x7f2860153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:15.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f285ffff700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c40b0 0x7f2860153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:15.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f285ffff700 1 -- 172.21.15.23:0/2678685472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2854009550 con 0x7f28600c9fd0 2023-06-22T08:14:15.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f285ffff700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28600c9fd0 0x7f28601edf00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f284c00c750 tx=0x7f284c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:15.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f285d7fa700 1 -- 172.21.15.23:0/2678685472 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c0054e0 con 0x7f28600c9fd0 2023-06-22T08:14:15.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f285d7fa700 1 -- 172.21.15.23:0/2678685472 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c003680 con 0x7f28600c9fd0 2023-06-22T08:14:15.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f2866bff700 1 -- 172.21.15.23:0/2678685472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2860153530 con 0x7f28600c9fd0 2023-06-22T08:14:15.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.215+0000 7f285d7fa700 1 -- 172.21.15.23:0/2678685472 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c003a30 con 0x7f28600c9fd0 2023-06-22T08:14:15.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.216+0000 7f2866bff700 1 -- 172.21.15.23:0/2678685472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28601f16c0 con 0x7f28600c9fd0 2023-06-22T08:14:15.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.217+0000 7f2866bff700 1 -- 172.21.15.23:0/2678685472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286004ea20 con 0x7f28600c9fd0 2023-06-22T08:14:15.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.218+0000 7f285d7fa700 1 -- 172.21.15.23:0/2678685472 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f284c003b90 con 0x7f28600c9fd0 2023-06-22T08:14:15.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.219+0000 7f285d7fa700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284406acb0 0x7f284406d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:15.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.219+0000 7f285d7fa700 1 -- 172.21.15.23:0/2678685472 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7f284c088df0 con 0x7f28600c9fd0 2023-06-22T08:14:15.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.219+0000 7f286499b700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284406acb0 0x7f284406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:15.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.220+0000 7f286499b700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284406acb0 0x7f284406d160 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2854009130 tx=0x7f285401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.223+0000 7f285d7fa700 1 -- 172.21.15.23:0/2678685472 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f284c053a50 con 0x7f28600c9fd0 2023-06-22T08:14:15.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.484+0000 7f2866bff700 1 -- 172.21.15.23:0/2678685472 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28600cf680 con 0x7f28600c9fd0 2023-06-22T08:14:15.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.484+0000 7f285d7fa700 1 -- 172.21.15.23:0/2678685472 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+5514 (secure 0 0 0) 0x7f284c017090 con 0x7f28600c9fd0 2023-06-22T08:14:15.488 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:15.488 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:13.928472+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:resolve","state_seq":9,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 -- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284406acb0 msgr2=0x7f284406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284406acb0 0x7f284406d160 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2854009130 tx=0x7f285401c040 comp rx=0 tx=0).stop 2023-06-22T08:14:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 -- 172.21.15.23:0/2678685472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28600c9fd0 msgr2=0x7f28601edf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28600c9fd0 0x7f28601edf00 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f284c00c750 tx=0x7f284c00ca60 comp rx=0 tx=0).stop 2023-06-22T08:14:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 -- 172.21.15.23:0/2678685472 shutdown_connections 2023-06-22T08:14:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2860152ce0 0x7f28601f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f284406acb0 0x7f284406d160 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28600c40b0 0x7f2860153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 --2- 172.21.15.23:0/2678685472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28600c9fd0 0x7f28601edf00 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 -- 172.21.15.23:0/2678685472 >> 172.21.15.23:0/2678685472 conn(0x7f286006ae30 msgr2=0x7f2860150bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 -- 172.21.15.23:0/2678685472 shutdown_connections 2023-06-22T08:14:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:15.487+0000 7f2866bff700 1 -- 172.21.15.23:0/2678685472 wait complete. 2023-06-22T08:14:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 111 2023-06-22T08:14:15.509 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:14:17.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:17.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.718+0000 7fa26f7ff700 1 Processor -- start 2023-06-22T08:14:17.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.718+0000 7fa26f7ff700 1 -- start start 2023-06-22T08:14:17.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.719+0000 7fa26f7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa268152ce0 0x7fa2680cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:17.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.719+0000 7fa26f7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa26815c370 con 0x7fa2680c40b0 2023-06-22T08:14:17.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.719+0000 7fa26f7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa26815bda0 con 0x7fa2680c9fd0 2023-06-22T08:14:17.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.719+0000 7fa26f7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681530b0 con 0x7fa268152ce0 2023-06-22T08:14:17.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.719+0000 7fa26dd9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa2680c40b0 0x7fa2680c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:53710/0 (socket says 172.21.15.23:53710) 2023-06-22T08:14:17.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.719+0000 7fa26dd9c700 1 -- 172.21.15.23:0/2175121934 learned_addr learned my addr 172.21.15.23:0/2175121934 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:14:17.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.719+0000 7fa26d59b700 1 --2- 172.21.15.23:0/2175121934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa268152ce0 0x7fa2680cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:17.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa25ffff700 1 -- 172.21.15.23:0/2175121934 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524467553 0 0) 0x7fa26815c370 con 0x7fa2680c40b0 2023-06-22T08:14:17.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa25ffff700 1 -- 172.21.15.23:0/2175121934 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2540035b0 con 0x7fa2680c40b0 2023-06-22T08:14:17.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa25ffff700 1 -- 172.21.15.23:0/2175121934 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440044631 0 0) 0x7fa26815bda0 con 0x7fa2680c9fd0 2023-06-22T08:14:17.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa25ffff700 1 -- 172.21.15.23:0/2175121934 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa26815c370 con 0x7fa2680c9fd0 2023-06-22T08:14:17.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa26d59b700 1 -- 172.21.15.23:0/2175121934 >> v1:172.21.15.67:6789/0 conn(0x7fa2680c9fd0 legacy=0x7fa26815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:17.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa26d59b700 1 -- 172.21.15.23:0/2175121934 >> v1:172.21.15.23:6789/0 conn(0x7fa2680c40b0 legacy=0x7fa2680c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:17.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa26d59b700 1 -- 172.21.15.23:0/2175121934 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2680c9460 con 0x7fa268152ce0 2023-06-22T08:14:17.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa26d59b700 1 --2- 172.21.15.23:0/2175121934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa268152ce0 0x7fa2680cf1b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa264009870 tx=0x7fa264009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5480ff236fda480a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:17.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa25ffff700 1 -- 172.21.15.23:0/2175121934 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa264011830 con 0x7fa268152ce0 2023-06-22T08:14:17.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa25ffff700 1 -- 172.21.15.23:0/2175121934 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa264011990 con 0x7fa268152ce0 2023-06-22T08:14:17.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.720+0000 7fa25ffff700 1 -- 172.21.15.23:0/2175121934 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa264011d20 con 0x7fa268152ce0 2023-06-22T08:14:17.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.721+0000 7fa26f7ff700 1 -- 172.21.15.23:0/2175121934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa268152ce0 msgr2=0x7fa2680cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:17.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.721+0000 7fa26f7ff700 1 --2- 172.21.15.23:0/2175121934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa268152ce0 0x7fa2680cf1b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa264009870 tx=0x7fa264009b80 comp rx=0 tx=0).stop 2023-06-22T08:14:17.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.721+0000 7fa26f7ff700 1 -- 172.21.15.23:0/2175121934 shutdown_connections 2023-06-22T08:14:17.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.721+0000 7fa26f7ff700 1 --2- 172.21.15.23:0/2175121934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa268152ce0 0x7fa2680cf1b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:17.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.721+0000 7fa26f7ff700 1 -- 172.21.15.23:0/2175121934 >> 172.21.15.23:0/2175121934 conn(0x7fa26806ae30 msgr2=0x7fa26806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:17.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.721+0000 7fa26f7ff700 1 -- 172.21.15.23:0/2175121934 shutdown_connections 2023-06-22T08:14:17.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.721+0000 7fa26f7ff700 1 -- 172.21.15.23:0/2175121934 wait complete. 2023-06-22T08:14:17.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.723+0000 7fa26f7ff700 1 Processor -- start 2023-06-22T08:14:17.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.723+0000 7fa26f7ff700 1 -- start start 2023-06-22T08:14:17.729 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26f7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa2680c40b0 0x7fa2681e2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:17.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26f7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2680c9fd0 0x7fa26815ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:17.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26f7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa268152ce0 0x7fa26815b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:17.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26f7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa268158200 con 0x7fa268152ce0 2023-06-22T08:14:17.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26f7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681e3810 con 0x7fa2680c9fd0 2023-06-22T08:14:17.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26f7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2681eeb40 con 0x7fa2680c40b0 2023-06-22T08:14:17.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26dd9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa268152ce0 0x7fa26815b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26dd9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa268152ce0 0x7fa26815b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41188/0 (socket says 172.21.15.23:41188) 2023-06-22T08:14:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26dd9c700 1 -- 172.21.15.23:0/3096446495 learned_addr learned my addr 172.21.15.23:0/3096446495 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26d59b700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa2680c40b0 0x7fa2681e2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.724+0000 7fa26cd9a700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2680c9fd0 0x7fa26815ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa26dd9c700 1 -- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa2680c40b0 msgr2=0x7fa2681e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa26dd9c700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa2680c40b0 0x7fa2681e2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa26dd9c700 1 -- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2680c9fd0 msgr2=0x7fa26815ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:17.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa26dd9c700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2680c9fd0 0x7fa26815ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:17.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa26dd9c700 1 -- 172.21.15.23:0/3096446495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2640094a0 con 0x7fa268152ce0 2023-06-22T08:14:17.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa26dd9c700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa268152ce0 0x7fa26815b180 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa26000caf0 tx=0x7fa26000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:17.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa25dffb700 1 -- 172.21.15.23:0/3096446495 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26000a9e0 con 0x7fa268152ce0 2023-06-22T08:14:17.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa25dffb700 1 -- 172.21.15.23:0/3096446495 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26000ab40 con 0x7fa268152ce0 2023-06-22T08:14:17.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa26f7ff700 1 -- 172.21.15.23:0/3096446495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa268155db0 con 0x7fa268152ce0 2023-06-22T08:14:17.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa25dffb700 1 -- 172.21.15.23:0/3096446495 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa260012400 con 0x7fa268152ce0 2023-06-22T08:14:17.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.725+0000 7fa26f7ff700 1 -- 172.21.15.23:0/3096446495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2681e31c0 con 0x7fa268152ce0 2023-06-22T08:14:17.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.726+0000 7fa26f7ff700 1 -- 172.21.15.23:0/3096446495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2681566b0 con 0x7fa268152ce0 2023-06-22T08:14:17.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.727+0000 7fa25dffb700 1 -- 172.21.15.23:0/3096446495 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa26001b3b0 con 0x7fa268152ce0 2023-06-22T08:14:17.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.728+0000 7fa25dffb700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa24c06ace0 0x7fa24c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:17.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.728+0000 7fa25dffb700 1 -- 172.21.15.23:0/3096446495 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7fa260089720 con 0x7fa268152ce0 2023-06-22T08:14:17.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.729+0000 7fa26d59b700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa24c06ace0 0x7fa24c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:17.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.729+0000 7fa26d59b700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa24c06ace0 0x7fa24c06d190 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa2640090b0 tx=0x7fa26401d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:17.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:17.733+0000 7fa25dffb700 1 -- 172.21.15.23:0/3096446495 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa2681566b0 con 0x7fa268152ce0 2023-06-22T08:14:18.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.011+0000 7fa26f7ff700 1 -- 172.21.15.23:0/3096446495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2681566b0 con 0x7fa268152ce0 2023-06-22T08:14:18.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.011+0000 7fa25dffb700 1 -- 172.21.15.23:0/3096446495 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5517 (secure 0 0 0) 0x7fa2681566b0 con 0x7fa268152ce0 2023-06-22T08:14:18.015 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:18.016 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:17.948785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:18.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 -- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa24c06ace0 msgr2=0x7fa24c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:18.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa24c06ace0 0x7fa24c06d190 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa2640090b0 tx=0x7fa26401d040 comp rx=0 tx=0).stop 2023-06-22T08:14:18.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 -- 172.21.15.23:0/3096446495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa268152ce0 msgr2=0x7fa26815b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:18.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa268152ce0 0x7fa26815b180 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa26000caf0 tx=0x7fa26000ce00 comp rx=0 tx=0).stop 2023-06-22T08:14:18.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 -- 172.21.15.23:0/3096446495 shutdown_connections 2023-06-22T08:14:18.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa268152ce0 0x7fa26815b180 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:18.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa2680c40b0 0x7fa2681e2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:18.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa24c06ace0 0x7fa24c06d190 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:18.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 --2- 172.21.15.23:0/3096446495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2680c9fd0 0x7fa26815ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:18.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 -- 172.21.15.23:0/3096446495 >> 172.21.15.23:0/3096446495 conn(0x7fa26806ae30 msgr2=0x7fa26814c630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:18.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 -- 172.21.15.23:0/3096446495 shutdown_connections 2023-06-22T08:14:18.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:18.014+0000 7fa26f7ff700 1 -- 172.21.15.23:0/3096446495 wait complete. 2023-06-22T08:14:18.019 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 112 2023-06-22T08:14:18.040 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 112, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 112, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5551": { "addr": "172.21.15.23:6834/1707022233", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 1707022233, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 1707022233, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5551, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 26 }, "gid_5576": { "addr": "172.21.15.67:6837/2064462140", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 2064462140, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 2064462140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5576, "incarnation": 109, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:reconnect", "state_seq": 11 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:14:17.948785+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 5551, "mds_3": 5995, "mds_4": 5576 } } } ], "standbys": [] } 2023-06-22T08:14:20.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:20.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.248+0000 7f1f50dff700 1 Processor -- start 2023-06-22T08:14:20.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.248+0000 7f1f50dff700 1 -- start start 2023-06-22T08:14:20.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f50dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1f4c0c4d00 0x7f1f4c159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:20.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f50dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c0c9c30 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f50dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4c15bea0 con 0x7f1f4c152d70 2023-06-22T08:14:20.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f50dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4c0c95e0 con 0x7f1f4c0c4d00 2023-06-22T08:14:20.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f4ad9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1f4c0c4d00 0x7f1f4c159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:20.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f4ad9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1f4c0c4d00 0x7f1f4c159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41196/0 (socket says 172.21.15.23:41196) 2023-06-22T08:14:20.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f4ad9d700 1 -- 172.21.15.23:0/1157280073 learned_addr learned my addr 172.21.15.23:0/1157280073 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:20.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f43fff700 1 --1- 172.21.15.23:0/1157280073 >> v1:172.21.15.23:6789/0 conn(0x7f1f4c0c97d0 0x7f1f4c0c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:53722/0 (socket says 172.21.15.23:53722) 2023-06-22T08:14:20.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f4ad9d700 1 -- 172.21.15.23:0/1157280073 >> v1:172.21.15.67:6789/0 conn(0x7f1f4c152d70 legacy=0x7f1f4c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:20.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f4ad9d700 1 -- 172.21.15.23:0/1157280073 >> v1:172.21.15.23:6789/0 conn(0x7f1f4c0c97d0 legacy=0x7f1f4c0c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:20.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f4ad9d700 1 -- 172.21.15.23:0/1157280073 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c15c160 con 0x7f1f4c0c4d00 2023-06-22T08:14:20.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.250+0000 7f1f49d9b700 1 -- 172.21.15.23:0/1157280073 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898585030 0 0) 0x7f1f4c15bea0 con 0x7f1f4c152d70 2023-06-22T08:14:20.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.251+0000 7f1f4ad9d700 1 --2- 172.21.15.23:0/1157280073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c4d00 0x7f1f4c159330 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009e50 tx=0x7f1f4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf65b6d85ffc413 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:20.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.251+0000 7f1f49d9b700 1 -- 172.21.15.23:0/1157280073 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4400e070 con 0x7f1f4c0c4d00 2023-06-22T08:14:20.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.251+0000 7f1f49d9b700 1 -- 172.21.15.23:0/1157280073 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f44003a20 con 0x7f1f4c0c4d00 2023-06-22T08:14:20.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.251+0000 7f1f49d9b700 1 -- 172.21.15.23:0/1157280073 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44003dd0 con 0x7f1f4c0c4d00 2023-06-22T08:14:20.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.251+0000 7f1f50dff700 1 -- 172.21.15.23:0/1157280073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c4d00 msgr2=0x7f1f4c159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:20.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.251+0000 7f1f50dff700 1 --2- 172.21.15.23:0/1157280073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c4d00 0x7f1f4c159330 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009e50 tx=0x7f1f4400b330 comp rx=0 tx=0).stop 2023-06-22T08:14:20.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.252+0000 7f1f50dff700 1 -- 172.21.15.23:0/1157280073 shutdown_connections 2023-06-22T08:14:20.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.252+0000 7f1f50dff700 1 --2- 172.21.15.23:0/1157280073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c4d00 0x7f1f4c159330 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:20.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.252+0000 7f1f50dff700 1 -- 172.21.15.23:0/1157280073 >> 172.21.15.23:0/1157280073 conn(0x7f1f4c06ae30 msgr2=0x7f1f4c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:20.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.252+0000 7f1f50dff700 1 -- 172.21.15.23:0/1157280073 shutdown_connections 2023-06-22T08:14:20.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.252+0000 7f1f50dff700 1 -- 172.21.15.23:0/1157280073 wait complete. 2023-06-22T08:14:20.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.253+0000 7f1f50dff700 1 Processor -- start 2023-06-22T08:14:20.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.253+0000 7f1f50dff700 1 -- start start 2023-06-22T08:14:20.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.254+0000 7f1f50dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4c0c4d00 0x7f1f4c0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:20.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.254+0000 7f1f50dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c97d0 0x7f1f4c150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:20.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.254+0000 7f1f50dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1f4c152d70 0x7f1f4c1558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:20.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.254+0000 7f1f50dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c151690 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.254+0000 7f1f50dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c1eec20 con 0x7f1f4c0c4d00 2023-06-22T08:14:20.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.254+0000 7f1f50dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4c0b2490 con 0x7f1f4c152d70 2023-06-22T08:14:20.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c97d0 0x7f1f4c150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:20.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c97d0 0x7f1f4c150480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41206/0 (socket says 172.21.15.23:41206) 2023-06-22T08:14:20.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f43fff700 1 -- 172.21.15.23:0/2495892712 learned_addr learned my addr 172.21.15.23:0/2495892712 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:20.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f4a59c700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4c0c4d00 0x7f1f4c0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:20.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f4ad9d700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1f4c152d70 0x7f1f4c1558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:20.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f43fff700 1 -- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1f4c152d70 msgr2=0x7f1f4c1558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:20.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f43fff700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1f4c152d70 0x7f1f4c1558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:20.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f43fff700 1 -- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4c0c4d00 msgr2=0x7f1f4c0b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:20.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f43fff700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4c0c4d00 0x7f1f4c0b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:20.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f43fff700 1 -- 172.21.15.23:0/2495892712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f44009b30 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f43fff700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c97d0 0x7f1f4c150480 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400c7b0 tx=0x7f1f3400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:20.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.255+0000 7f1f42ffd700 1 -- 172.21.15.23:0/2495892712 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f34004c80 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.256+0000 7f1f42ffd700 1 -- 172.21.15.23:0/2495892712 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f34004e00 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.256+0000 7f1f42ffd700 1 -- 172.21.15.23:0/2495892712 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f34007310 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.256+0000 7f1f50dff700 1 -- 172.21.15.23:0/2495892712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4c0b2c50 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.256+0000 7f1f50dff700 1 -- 172.21.15.23:0/2495892712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4c1eee10 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.257+0000 7f1f50dff700 1 -- 172.21.15.23:0/2495892712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f4c04ea20 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.258+0000 7f1f42ffd700 1 -- 172.21.15.23:0/2495892712 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1f34007470 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.259+0000 7f1f42ffd700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1f2c06ace0 0x7f1f2c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:20.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.259+0000 7f1f42ffd700 1 -- 172.21.15.23:0/2495892712 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7f1f34089800 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.259+0000 7f1f4a59c700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1f2c06ace0 0x7f1f2c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:20.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.260+0000 7f1f4a59c700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1f2c06ace0 0x7f1f2c06d190 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c000f60 tx=0x7f1f3c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:20.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.264+0000 7f1f42ffd700 1 -- 172.21.15.23:0/2495892712 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1f340541e0 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.519+0000 7f1f50dff700 1 -- 172.21.15.23:0/2495892712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f4c0c9e00 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.520+0000 7f1f42ffd700 1 -- 172.21.15.23:0/2495892712 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5517 (secure 0 0 0) 0x7f1f34057800 con 0x7f1f4c0c97d0 2023-06-22T08:14:20.524 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:20.524 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:17.948785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:20.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.522+0000 7f1f50dff700 1 -- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1f2c06ace0 msgr2=0x7f1f2c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:20.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.522+0000 7f1f50dff700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1f2c06ace0 0x7f1f2c06d190 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c000f60 tx=0x7f1f3c009040 comp rx=0 tx=0).stop 2023-06-22T08:14:20.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.522+0000 7f1f50dff700 1 -- 172.21.15.23:0/2495892712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c97d0 msgr2=0x7f1f4c150480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:20.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.522+0000 7f1f50dff700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c97d0 0x7f1f4c150480 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400c7b0 tx=0x7f1f3400cac0 comp rx=0 tx=0).stop 2023-06-22T08:14:20.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.522+0000 7f1f50dff700 1 -- 172.21.15.23:0/2495892712 shutdown_connections 2023-06-22T08:14:20.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.522+0000 7f1f50dff700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1f4c152d70 0x7f1f4c1558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:20.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.522+0000 7f1f50dff700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1f2c06ace0 0x7f1f2c06d190 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:20.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.522+0000 7f1f50dff700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f4c0c4d00 0x7f1f4c0b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:20.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.522+0000 7f1f50dff700 1 --2- 172.21.15.23:0/2495892712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f4c0c97d0 0x7f1f4c150480 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:20.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.522+0000 7f1f50dff700 1 -- 172.21.15.23:0/2495892712 >> 172.21.15.23:0/2495892712 conn(0x7f1f4c06ae30 msgr2=0x7f1f4c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:20.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.523+0000 7f1f50dff700 1 -- 172.21.15.23:0/2495892712 shutdown_connections 2023-06-22T08:14:20.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:20.523+0000 7f1f50dff700 1 -- 172.21.15.23:0/2495892712 wait complete. 2023-06-22T08:14:20.528 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 112 2023-06-22T08:14:20.545 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:14:22.547 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:22.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.753+0000 7f75d69ff700 1 Processor -- start 2023-06-22T08:14:22.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.753+0000 7f75d69ff700 1 -- start start 2023-06-22T08:14:22.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75d69ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f75d00cb260 0x7f75d00c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:22.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75d69ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75d00c9200 0x7f75d00cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:22.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75d69ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 0x7f75d015be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:22.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75d69ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d00cbb70 con 0x7f75d00cb260 2023-06-22T08:14:22.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75d69ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d00cbd60 con 0x7f75d00c9200 2023-06-22T08:14:22.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75d69ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d0159550 con 0x7f75d00b0fc0 2023-06-22T08:14:22.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75cffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f75d00cb260 0x7f75d00c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:22.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75cffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f75d00cb260 0x7f75d00c4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41220/0 (socket says 172.21.15.23:41220) 2023-06-22T08:14:22.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75cffff700 1 -- 172.21.15.23:0/2807100111 learned_addr learned my addr 172.21.15.23:0/2807100111 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:22.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75d4f9c700 1 --2- 172.21.15.23:0/2807100111 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 0x7f75d015be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:22.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.755+0000 7f75cf7fe700 1 --2- 172.21.15.23:0/2807100111 >> v2:172.21.15.67:3300/0 conn(0x7f75d00c9200 0x7f75d00cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:22.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.756+0000 7f75cffff700 1 -- 172.21.15.23:0/2807100111 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 msgr2=0x7f75d015be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:14:22.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.756+0000 7f75cffff700 1 --2- 172.21.15.23:0/2807100111 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 0x7f75d015be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:22.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.756+0000 7f75cffff700 1 -- 172.21.15.23:0/2807100111 >> v2:172.21.15.67:3300/0 conn(0x7f75d00c9200 msgr2=0x7f75d00cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:14:22.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.756+0000 7f75cffff700 1 --2- 172.21.15.23:0/2807100111 >> v2:172.21.15.67:3300/0 conn(0x7f75d00c9200 0x7f75d00cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:22.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.756+0000 7f75cffff700 1 -- 172.21.15.23:0/2807100111 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d0158980 con 0x7f75d00cb260 2023-06-22T08:14:22.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.756+0000 7f75d4f9c700 1 --2- 172.21.15.23:0/2807100111 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 0x7f75d015be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:14:22.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.756+0000 7f75cffff700 1 --2- 172.21.15.23:0/2807100111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d00cb260 0x7f75d00c4a50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f75d008c760 tx=0x7f75c0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8811833e4b6657a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.756+0000 7f75ceffd700 1 -- 172.21.15.23:0/2807100111 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0011830 con 0x7f75d00cb260 2023-06-22T08:14:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.756+0000 7f75ceffd700 1 -- 172.21.15.23:0/2807100111 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c0011990 con 0x7f75d00cb260 2023-06-22T08:14:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.756+0000 7f75ceffd700 1 -- 172.21.15.23:0/2807100111 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0011d20 con 0x7f75d00cb260 2023-06-22T08:14:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.757+0000 7f75d69ff700 1 -- 172.21.15.23:0/2807100111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d00cb260 msgr2=0x7f75d00c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:22.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.757+0000 7f75d69ff700 1 --2- 172.21.15.23:0/2807100111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d00cb260 0x7f75d00c4a50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f75d008c760 tx=0x7f75c0009c30 comp rx=0 tx=0).stop 2023-06-22T08:14:22.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.757+0000 7f75d69ff700 1 -- 172.21.15.23:0/2807100111 shutdown_connections 2023-06-22T08:14:22.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.757+0000 7f75d69ff700 1 --2- 172.21.15.23:0/2807100111 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 0x7f75d015be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:22.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.757+0000 7f75d69ff700 1 --2- 172.21.15.23:0/2807100111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d00cb260 0x7f75d00c4a50 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.757+0000 7f75d69ff700 1 --2- 172.21.15.23:0/2807100111 >> v2:172.21.15.67:3300/0 conn(0x7f75d00c9200 0x7f75d00cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.757+0000 7f75d69ff700 1 -- 172.21.15.23:0/2807100111 >> 172.21.15.23:0/2807100111 conn(0x7f75d0065b90 msgr2=0x7f75d0065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.757+0000 7f75d69ff700 1 -- 172.21.15.23:0/2807100111 shutdown_connections 2023-06-22T08:14:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.757+0000 7f75d69ff700 1 -- 172.21.15.23:0/2807100111 wait complete. 2023-06-22T08:14:22.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.759+0000 7f75d69ff700 1 Processor -- start 2023-06-22T08:14:22.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.759+0000 7f75d69ff700 1 -- start start 2023-06-22T08:14:22.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75d69ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 0x7f75d01e4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:22.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75d69ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d00c9200 0x7f75d01ee880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:22.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75d69ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d00cb260 0x7f75d01f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:22.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75d69ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d0157850 con 0x7f75d00cb260 2023-06-22T08:14:22.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75d69ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d0157b10 con 0x7f75d00c9200 2023-06-22T08:14:22.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75d69ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d0157df0 con 0x7f75d00b0fc0 2023-06-22T08:14:22.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75cffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 0x7f75d01e4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:22.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75cf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d00c9200 0x7f75d01ee880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:22.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75d4f9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d00cb260 0x7f75d01f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:22.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75cf7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d00c9200 0x7f75d01ee880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:44092/0 (socket says 172.21.15.23:44092) 2023-06-22T08:14:22.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75cffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 0x7f75d01e4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:57278/0 (socket says 172.21.15.23:57278) 2023-06-22T08:14:22.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75cf7fe700 1 -- 172.21.15.23:0/1157606587 learned_addr learned my addr 172.21.15.23:0/1157606587 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:22.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75cf7fe700 1 -- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 msgr2=0x7f75d01e4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:22.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75cf7fe700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 0x7f75d01e4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:22.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75cf7fe700 1 -- 172.21.15.23:0/1157606587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d00cb260 msgr2=0x7f75d01f1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:22.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.760+0000 7f75cf7fe700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d00cb260 0x7f75d01f1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:22.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.761+0000 7f75cf7fe700 1 -- 172.21.15.23:0/1157606587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c8009a70 con 0x7f75d00c9200 2023-06-22T08:14:22.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.761+0000 7f75d4f9c700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d00cb260 0x7f75d01f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:14:22.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.761+0000 7f75cf7fe700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d00c9200 0x7f75d01ee880 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f75c400e9b0 tx=0x7f75c400ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:22.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.761+0000 7f75ccff9700 1 -- 172.21.15.23:0/1157606587 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4009710 con 0x7f75d00c9200 2023-06-22T08:14:22.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.761+0000 7f75ccff9700 1 -- 172.21.15.23:0/1157606587 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c400ee40 con 0x7f75d00c9200 2023-06-22T08:14:22.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.761+0000 7f75d69ff700 1 -- 172.21.15.23:0/1157606587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c0009550 con 0x7f75d00c9200 2023-06-22T08:14:22.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.761+0000 7f75ccff9700 1 -- 172.21.15.23:0/1157606587 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c400ca50 con 0x7f75d00c9200 2023-06-22T08:14:22.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.761+0000 7f75d69ff700 1 -- 172.21.15.23:0/1157606587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d0147440 con 0x7f75d00c9200 2023-06-22T08:14:22.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.762+0000 7f75d69ff700 1 -- 172.21.15.23:0/1157606587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d004ea20 con 0x7f75d00c9200 2023-06-22T08:14:22.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.763+0000 7f75ccff9700 1 -- 172.21.15.23:0/1157606587 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f75c400cbb0 con 0x7f75d00c9200 2023-06-22T08:14:22.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.764+0000 7f75ccff9700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f75bc06ace0 0x7f75bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:22.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.764+0000 7f75ccff9700 1 -- 172.21.15.23:0/1157606587 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7f75c4089b70 con 0x7f75d00c9200 2023-06-22T08:14:22.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.764+0000 7f75cffff700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f75bc06ace0 0x7f75bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:22.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.765+0000 7f75cffff700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f75bc06ace0 0x7f75bc06d190 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f75c0014040 tx=0x7f75c0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:22.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:22.770+0000 7f75ccff9700 1 -- 172.21.15.23:0/1157606587 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f75c4057fe0 con 0x7f75d00c9200 2023-06-22T08:14:23.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.024+0000 7f75d69ff700 1 -- 172.21.15.23:0/1157606587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75d01524c0 con 0x7f75d00c9200 2023-06-22T08:14:23.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.025+0000 7f75ccff9700 1 -- 172.21.15.23:0/1157606587 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5517 (secure 0 0 0) 0x7f75c4021070 con 0x7f75d00c9200 2023-06-22T08:14:23.029 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:23.029 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:17.948785+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:23.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 -- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f75bc06ace0 msgr2=0x7f75bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:23.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f75bc06ace0 0x7f75bc06d190 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f75c0014040 tx=0x7f75c0008030 comp rx=0 tx=0).stop 2023-06-22T08:14:23.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 -- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d00c9200 msgr2=0x7f75d01ee880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:23.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d00c9200 0x7f75d01ee880 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f75c400e9b0 tx=0x7f75c400ecc0 comp rx=0 tx=0).stop 2023-06-22T08:14:23.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 -- 172.21.15.23:0/1157606587 shutdown_connections 2023-06-22T08:14:23.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75d00cb260 0x7f75d01f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:23.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f75d00b0fc0 0x7f75d01e4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:23.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f75bc06ace0 0x7f75bc06d190 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:23.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 --2- 172.21.15.23:0/1157606587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d00c9200 0x7f75d01ee880 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:23.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 -- 172.21.15.23:0/1157606587 >> 172.21.15.23:0/1157606587 conn(0x7f75d0065b90 msgr2=0x7f75d014fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:23.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 -- 172.21.15.23:0/1157606587 shutdown_connections 2023-06-22T08:14:23.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:23.028+0000 7f75d69ff700 1 -- 172.21.15.23:0/1157606587 wait complete. 2023-06-22T08:14:23.033 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 112 2023-06-22T08:14:23.049 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:14:25.050 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:25.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.257+0000 7f2e69fff700 1 Processor -- start 2023-06-22T08:14:25.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.257+0000 7f2e69fff700 1 -- start start 2023-06-22T08:14:25.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.258+0000 7f2e69fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2e6415a320 0x7f2e640c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:25.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.258+0000 7f2e69fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e6406d050 0x7f2e64160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:25.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.258+0000 7f2e69fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e640c6a60 con 0x7f2e640c3a40 2023-06-22T08:14:25.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.258+0000 7f2e69fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e64157570 con 0x7f2e6415a320 2023-06-22T08:14:25.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.258+0000 7f2e69fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e640c4350 con 0x7f2e6406d050 2023-06-22T08:14:25.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.258+0000 7f2e5bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2e6415a320 0x7f2e640c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:25.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.259+0000 7f2e637fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2e640c3a40 0x7f2e64159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:53734/0 (socket says 172.21.15.23:53734) 2023-06-22T08:14:25.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.259+0000 7f2e637fe700 1 -- 172.21.15.23:0/306101551 learned_addr learned my addr 172.21.15.23:0/306101551 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:14:25.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.259+0000 7f2e63fff700 1 --2- 172.21.15.23:0/306101551 >> v2:172.21.15.67:3300/0 conn(0x7f2e6406d050 0x7f2e64160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:25.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.259+0000 7f2e5bfff700 1 -- 172.21.15.23:0/306101551 >> v2:172.21.15.67:3300/0 conn(0x7f2e6406d050 msgr2=0x7f2e64160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:14:25.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.259+0000 7f2e5bfff700 1 --2- 172.21.15.23:0/306101551 >> v2:172.21.15.67:3300/0 conn(0x7f2e6406d050 0x7f2e64160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:25.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.259+0000 7f2e5bfff700 1 -- 172.21.15.23:0/306101551 >> v1:172.21.15.23:6789/0 conn(0x7f2e640c3a40 legacy=0x7f2e64159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:25.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.259+0000 7f2e5bfff700 1 -- 172.21.15.23:0/306101551 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6414d690 con 0x7f2e6415a320 2023-06-22T08:14:25.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.259+0000 7f2e62ffd700 1 -- 172.21.15.23:0/306101551 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151540224 0 0) 0x7f2e640c6a60 con 0x7f2e640c3a40 2023-06-22T08:14:25.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.259+0000 7f2e5bfff700 1 --2- 172.21.15.23:0/306101551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e6415a320 0x7f2e640c3e10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2e64072440 tx=0x7f2e4c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99c89c941c4bc54 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:25.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e62ffd700 1 -- 172.21.15.23:0/306101551 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c011830 con 0x7f2e6415a320 2023-06-22T08:14:25.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e62ffd700 1 -- 172.21.15.23:0/306101551 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c011990 con 0x7f2e6415a320 2023-06-22T08:14:25.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e62ffd700 1 -- 172.21.15.23:0/306101551 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c011d20 con 0x7f2e6415a320 2023-06-22T08:14:25.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e69fff700 1 -- 172.21.15.23:0/306101551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e6415a320 msgr2=0x7f2e640c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:25.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e69fff700 1 --2- 172.21.15.23:0/306101551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e6415a320 0x7f2e640c3e10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2e64072440 tx=0x7f2e4c009b20 comp rx=0 tx=0).stop 2023-06-22T08:14:25.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e69fff700 1 -- 172.21.15.23:0/306101551 shutdown_connections 2023-06-22T08:14:25.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e69fff700 1 --2- 172.21.15.23:0/306101551 >> v2:172.21.15.67:3300/0 conn(0x7f2e6406d050 0x7f2e64160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:25.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e69fff700 1 --2- 172.21.15.23:0/306101551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e6415a320 0x7f2e640c3e10 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:25.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e69fff700 1 -- 172.21.15.23:0/306101551 >> 172.21.15.23:0/306101551 conn(0x7f2e640ca3f0 msgr2=0x7f2e640ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:25.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e69fff700 1 -- 172.21.15.23:0/306101551 shutdown_connections 2023-06-22T08:14:25.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.260+0000 7f2e69fff700 1 -- 172.21.15.23:0/306101551 wait complete. 2023-06-22T08:14:25.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.262+0000 7f2e69fff700 1 Processor -- start 2023-06-22T08:14:25.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.262+0000 7f2e69fff700 1 -- start start 2023-06-22T08:14:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e69fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6406d050 0x7f2e64071330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e69fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e640c3a40 0x7f2e640c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e69fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e6415a320 0x7f2e641e2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e69fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6414fff0 con 0x7f2e6415a320 2023-06-22T08:14:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e69fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e640c98b0 con 0x7f2e6406d050 2023-06-22T08:14:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e69fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e640c8270 con 0x7f2e640c3a40 2023-06-22T08:14:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e637fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6406d050 0x7f2e64071330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e5bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e640c3a40 0x7f2e640c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e637fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6406d050 0x7f2e64071330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:44110/0 (socket says 172.21.15.23:44110) 2023-06-22T08:14:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e637fe700 1 -- 172.21.15.23:0/622185681 learned_addr learned my addr 172.21.15.23:0/622185681 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.263+0000 7f2e63fff700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e6415a320 0x7f2e641e2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e637fe700 1 -- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e640c3a40 msgr2=0x7f2e640c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e637fe700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e640c3a40 0x7f2e640c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e637fe700 1 -- 172.21.15.23:0/622185681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e6415a320 msgr2=0x7f2e641e2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e637fe700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e6415a320 0x7f2e641e2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e637fe700 1 -- 172.21.15.23:0/622185681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c0094a0 con 0x7f2e6406d050 2023-06-22T08:14:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e637fe700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6406d050 0x7f2e64071330 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2e540011e0 tx=0x7f2e5400ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e60ff9700 1 -- 172.21.15.23:0/622185681 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54004c80 con 0x7f2e6406d050 2023-06-22T08:14:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e60ff9700 1 -- 172.21.15.23:0/622185681 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e54004e00 con 0x7f2e6406d050 2023-06-22T08:14:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e69fff700 1 -- 172.21.15.23:0/622185681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e640c8550 con 0x7f2e6406d050 2023-06-22T08:14:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e60ff9700 1 -- 172.21.15.23:0/622185681 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e54007310 con 0x7f2e6406d050 2023-06-22T08:14:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.264+0000 7f2e69fff700 1 -- 172.21.15.23:0/622185681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e640c8b10 con 0x7f2e6406d050 2023-06-22T08:14:25.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.265+0000 7f2e69fff700 1 -- 172.21.15.23:0/622185681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6404ea20 con 0x7f2e6406d050 2023-06-22T08:14:25.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.266+0000 7f2e60ff9700 1 -- 172.21.15.23:0/622185681 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2e54007470 con 0x7f2e6406d050 2023-06-22T08:14:25.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.267+0000 7f2e60ff9700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e4406ace0 0x7f2e4406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:25.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.267+0000 7f2e60ff9700 1 -- 172.21.15.23:0/622185681 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7f2e54089770 con 0x7f2e6406d050 2023-06-22T08:14:25.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.267+0000 7f2e5bfff700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e4406ace0 0x7f2e4406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:25.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.268+0000 7f2e5bfff700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e4406ace0 0x7f2e4406d190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c000c00 tx=0x7f2e4c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:25.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.272+0000 7f2e60ff9700 1 -- 172.21.15.23:0/622185681 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2e540541e0 con 0x7f2e6406d050 2023-06-22T08:14:25.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.525+0000 7f2e69fff700 1 -- 172.21.15.23:0/622185681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e640c5f30 con 0x7f2e6406d050 2023-06-22T08:14:25.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.526+0000 7f2e60ff9700 1 -- 172.21.15.23:0/622185681 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5514 (secure 0 0 0) 0x7f2e54057800 con 0x7f2e6406d050 2023-06-22T08:14:25.530 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:25.530 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:23.869001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:25.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 -- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e4406ace0 msgr2=0x7f2e4406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:25.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e4406ace0 0x7f2e4406d190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c000c00 tx=0x7f2e4c008030 comp rx=0 tx=0).stop 2023-06-22T08:14:25.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 -- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6406d050 msgr2=0x7f2e64071330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:25.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6406d050 0x7f2e64071330 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2e540011e0 tx=0x7f2e5400ca30 comp rx=0 tx=0).stop 2023-06-22T08:14:25.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 -- 172.21.15.23:0/622185681 shutdown_connections 2023-06-22T08:14:25.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e6415a320 0x7f2e641e2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:25.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2e4406ace0 0x7f2e4406d190 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:25.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e6406d050 0x7f2e64071330 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:25.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 --2- 172.21.15.23:0/622185681 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e640c3a40 0x7f2e640c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:25.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 -- 172.21.15.23:0/622185681 >> 172.21.15.23:0/622185681 conn(0x7f2e640ca3f0 msgr2=0x7f2e6414b090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:25.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 -- 172.21.15.23:0/622185681 shutdown_connections 2023-06-22T08:14:25.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:25.529+0000 7f2e69fff700 1 -- 172.21.15.23:0/622185681 wait complete. 2023-06-22T08:14:25.535 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 113 2023-06-22T08:14:25.551 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:14:27.552 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:27.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.759+0000 7fa678bff700 1 Processor -- start 2023-06-22T08:14:27.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.759+0000 7fa678bff700 1 -- start start 2023-06-22T08:14:27.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.760+0000 7fa678bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa674150670 0x7fa6740c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:27.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.760+0000 7fa678bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa6740c8190 0x7fa6740c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:27.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.760+0000 7fa678bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6740704d0 con 0x7fa6740c71c0 2023-06-22T08:14:27.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.760+0000 7fa678bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6740c6e40 con 0x7fa6740c8190 2023-06-22T08:14:27.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.760+0000 7fa678bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa674070230 con 0x7fa674150670 2023-06-22T08:14:27.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.760+0000 7fa66bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa6740c8190 0x7fa6740c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:27.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.760+0000 7fa66bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa6740c8190 0x7fa6740c3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35458/0 (socket says 172.21.15.23:35458) 2023-06-22T08:14:27.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.760+0000 7fa66bfff700 1 -- 172.21.15.23:0/1079814079 learned_addr learned my addr 172.21.15.23:0/1079814079 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:27.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.760+0000 7fa67259c700 1 --2- 172.21.15.23:0/1079814079 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa674150670 0x7fa6740c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:27.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.761+0000 7fa66bfff700 1 -- 172.21.15.23:0/1079814079 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa674150670 msgr2=0x7fa6740c3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:14:27.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.761+0000 7fa66bfff700 1 --2- 172.21.15.23:0/1079814079 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa674150670 0x7fa6740c3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:27.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.761+0000 7fa66bfff700 1 -- 172.21.15.23:0/1079814079 >> v1:172.21.15.67:6789/0 conn(0x7fa6740c71c0 legacy=0x7fa674070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:27.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.761+0000 7fa66bfff700 1 -- 172.21.15.23:0/1079814079 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6740c7d00 con 0x7fa6740c8190 2023-06-22T08:14:27.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.761+0000 7fa671d9b700 1 -- 172.21.15.23:0/1079814079 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465813125 0 0) 0x7fa6740704d0 con 0x7fa6740c71c0 2023-06-22T08:14:27.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.761+0000 7fa66bfff700 1 --2- 172.21.15.23:0/1079814079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 0x7fa6740c3f80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0098f0 tx=0x7fa65c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e45ab26e6c49ad server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:27.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.761+0000 7fa671d9b700 1 -- 172.21.15.23:0/1079814079 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c011830 con 0x7fa6740c8190 2023-06-22T08:14:27.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.761+0000 7fa671d9b700 1 -- 172.21.15.23:0/1079814079 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c011990 con 0x7fa6740c8190 2023-06-22T08:14:27.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.761+0000 7fa671d9b700 1 -- 172.21.15.23:0/1079814079 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c011d20 con 0x7fa6740c8190 2023-06-22T08:14:27.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.762+0000 7fa678bff700 1 -- 172.21.15.23:0/1079814079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 msgr2=0x7fa6740c3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:27.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.762+0000 7fa678bff700 1 --2- 172.21.15.23:0/1079814079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 0x7fa6740c3f80 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0098f0 tx=0x7fa65c009c00 comp rx=0 tx=0).stop 2023-06-22T08:14:27.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.762+0000 7fa678bff700 1 -- 172.21.15.23:0/1079814079 shutdown_connections 2023-06-22T08:14:27.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.762+0000 7fa678bff700 1 --2- 172.21.15.23:0/1079814079 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa674150670 0x7fa6740c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:27.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.762+0000 7fa678bff700 1 --2- 172.21.15.23:0/1079814079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 0x7fa6740c3f80 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:27.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.762+0000 7fa678bff700 1 -- 172.21.15.23:0/1079814079 >> 172.21.15.23:0/1079814079 conn(0x7fa67406c910 msgr2=0x7fa67406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:27.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.762+0000 7fa678bff700 1 -- 172.21.15.23:0/1079814079 shutdown_connections 2023-06-22T08:14:27.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.762+0000 7fa678bff700 1 -- 172.21.15.23:0/1079814079 wait complete. 2023-06-22T08:14:27.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.764+0000 7fa678bff700 1 Processor -- start 2023-06-22T08:14:27.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.764+0000 7fa678bff700 1 -- start start 2023-06-22T08:14:27.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa678bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6740c71c0 0x7fa67414e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:27.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa678bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 0x7fa674152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:27.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa678bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa674150670 0x7fa674152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:27.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa678bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6741edde0 con 0x7fa6740c8190 2023-06-22T08:14:27.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa678bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6741edfd0 con 0x7fa6740c71c0 2023-06-22T08:14:27.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa678bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6741ee270 con 0x7fa674150670 2023-06-22T08:14:27.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa66bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 0x7fa674152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:27.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa66bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 0x7fa674152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35464/0 (socket says 172.21.15.23:35464) 2023-06-22T08:14:27.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa66bfff700 1 -- 172.21.15.23:0/1684251688 learned_addr learned my addr 172.21.15.23:0/1684251688 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:27.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa67259c700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6740c71c0 0x7fa67414e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:27.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa672d9d700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa674150670 0x7fa674152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:27.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa66bfff700 1 -- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa674150670 msgr2=0x7fa674152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:27.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa66bfff700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa674150670 0x7fa674152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:27.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa66bfff700 1 -- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6740c71c0 msgr2=0x7fa67414e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:27.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa66bfff700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6740c71c0 0x7fa67414e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:27.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.765+0000 7fa66bfff700 1 -- 172.21.15.23:0/1684251688 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c009550 con 0x7fa6740c8190 2023-06-22T08:14:27.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.766+0000 7fa66bfff700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 0x7fa674152230 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa65c014040 tx=0x7fa65c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:27.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.766+0000 7fa66affd700 1 -- 172.21.15.23:0/1684251688 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c004590 con 0x7fa6740c8190 2023-06-22T08:14:27.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.766+0000 7fa66affd700 1 -- 172.21.15.23:0/1684251688 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c01e070 con 0x7fa6740c8190 2023-06-22T08:14:27.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.766+0000 7fa66affd700 1 -- 172.21.15.23:0/1684251688 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0038c0 con 0x7fa6740c8190 2023-06-22T08:14:27.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.766+0000 7fa678bff700 1 -- 172.21.15.23:0/1684251688 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6741ee550 con 0x7fa6740c8190 2023-06-22T08:14:27.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.766+0000 7fa678bff700 1 -- 172.21.15.23:0/1684251688 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6741eeb10 con 0x7fa6740c8190 2023-06-22T08:14:27.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.767+0000 7fa678bff700 1 -- 172.21.15.23:0/1684251688 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67404ea20 con 0x7fa6740c8190 2023-06-22T08:14:27.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.768+0000 7fa66affd700 1 -- 172.21.15.23:0/1684251688 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa65c003a20 con 0x7fa6740c8190 2023-06-22T08:14:27.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.769+0000 7fa66affd700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa65406ace0 0x7fa65406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:27.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.770+0000 7fa66affd700 1 -- 172.21.15.23:0/1684251688 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7fa65c08a930 con 0x7fa6740c8190 2023-06-22T08:14:27.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.770+0000 7fa67259c700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa65406ace0 0x7fa65406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:27.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.770+0000 7fa67259c700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa65406ace0 0x7fa65406d190 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa664009be0 tx=0x7fa664009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:27.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:27.774+0000 7fa66affd700 1 -- 172.21.15.23:0/1684251688 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa65c055430 con 0x7fa6740c8190 2023-06-22T08:14:28.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.029+0000 7fa678bff700 1 -- 172.21.15.23:0/1684251688 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6740c6fa0 con 0x7fa6740c8190 2023-06-22T08:14:28.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.029+0000 7fa66affd700 1 -- 172.21.15.23:0/1684251688 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5514 (secure 0 0 0) 0x7fa65c020090 con 0x7fa6740c8190 2023-06-22T08:14:28.033 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:28.033 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:23.869001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:28.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.031+0000 7fa678bff700 1 -- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa65406ace0 msgr2=0x7fa65406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:28.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.031+0000 7fa678bff700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa65406ace0 0x7fa65406d190 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa664009be0 tx=0x7fa664009210 comp rx=0 tx=0).stop 2023-06-22T08:14:28.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.031+0000 7fa678bff700 1 -- 172.21.15.23:0/1684251688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 msgr2=0x7fa674152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:28.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.031+0000 7fa678bff700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 0x7fa674152230 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa65c014040 tx=0x7fa65c004750 comp rx=0 tx=0).stop 2023-06-22T08:14:28.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.031+0000 7fa678bff700 1 -- 172.21.15.23:0/1684251688 shutdown_connections 2023-06-22T08:14:28.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.031+0000 7fa678bff700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa674150670 0x7fa674152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:28.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.031+0000 7fa678bff700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa65406ace0 0x7fa65406d190 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:28.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.031+0000 7fa678bff700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6740c71c0 0x7fa67414e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:28.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.031+0000 7fa678bff700 1 --2- 172.21.15.23:0/1684251688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6740c8190 0x7fa674152230 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:28.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.032+0000 7fa678bff700 1 -- 172.21.15.23:0/1684251688 >> 172.21.15.23:0/1684251688 conn(0x7fa67406c910 msgr2=0x7fa67414ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:28.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.032+0000 7fa678bff700 1 -- 172.21.15.23:0/1684251688 shutdown_connections 2023-06-22T08:14:28.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:28.032+0000 7fa678bff700 1 -- 172.21.15.23:0/1684251688 wait complete. 2023-06-22T08:14:28.037 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 113 2023-06-22T08:14:28.053 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:14:30.053 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:30.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.262+0000 7fa9f55ff700 1 Processor -- start 2023-06-22T08:14:30.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.262+0000 7fa9f55ff700 1 -- start start 2023-06-22T08:14:30.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.263+0000 7fa9f55ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9f0152ce0 0x7fa9f00cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:30.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.263+0000 7fa9f55ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa9f00c9fd0 0x7fa9f015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:30.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.263+0000 7fa9f55ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f015c370 con 0x7fa9f0158880 2023-06-22T08:14:30.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.263+0000 7fa9f55ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f01530b0 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.263+0000 7fa9f55ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f00cf6f0 con 0x7fa9f0152ce0 2023-06-22T08:14:30.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.264+0000 7fa9ee7fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa9f00c9fd0 0x7fa9f015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:30.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.264+0000 7fa9ee7fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa9f00c9fd0 0x7fa9f015c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35466/0 (socket says 172.21.15.23:35466) 2023-06-22T08:14:30.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.264+0000 7fa9ee7fc700 1 -- 172.21.15.23:0/1241494216 learned_addr learned my addr 172.21.15.23:0/1241494216 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:30.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.264+0000 7fa9eeffd700 1 --2- 172.21.15.23:0/1241494216 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9f0152ce0 0x7fa9f00cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:30.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.264+0000 7fa9ee7fc700 1 -- 172.21.15.23:0/1241494216 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9f0152ce0 msgr2=0x7fa9f00cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:14:30.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.264+0000 7fa9ee7fc700 1 --2- 172.21.15.23:0/1241494216 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9f0152ce0 0x7fa9f00cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:30.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.264+0000 7fa9ee7fc700 1 -- 172.21.15.23:0/1241494216 >> v1:172.21.15.67:6789/0 conn(0x7fa9f0158880 legacy=0x7fa9f0158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:30.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.264+0000 7fa9ee7fc700 1 -- 172.21.15.23:0/1241494216 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f015bda0 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.264+0000 7fa9ee7fc700 1 --2- 172.21.15.23:0/1241494216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 0x7fa9f015c800 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40098f0 tx=0x7fa9e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364edc108d68c946 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:30.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9edffb700 1 -- 172.21.15.23:0/1241494216 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e400b8a0 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9edffb700 1 -- 172.21.15.23:0/1241494216 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e400ba00 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9edffb700 1 -- 172.21.15.23:0/1241494216 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e400bd90 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1241494216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 msgr2=0x7fa9f015c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:30.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9f55ff700 1 --2- 172.21.15.23:0/1241494216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 0x7fa9f015c800 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa9e40098f0 tx=0x7fa9e4009c00 comp rx=0 tx=0).stop 2023-06-22T08:14:30.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1241494216 shutdown_connections 2023-06-22T08:14:30.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9f55ff700 1 --2- 172.21.15.23:0/1241494216 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9f0152ce0 0x7fa9f00cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:30.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9f55ff700 1 --2- 172.21.15.23:0/1241494216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 0x7fa9f015c800 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:30.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1241494216 >> 172.21.15.23:0/1241494216 conn(0x7fa9f006ae30 msgr2=0x7fa9f006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:30.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1241494216 shutdown_connections 2023-06-22T08:14:30.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.265+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1241494216 wait complete. 2023-06-22T08:14:30.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.267+0000 7fa9f55ff700 1 Processor -- start 2023-06-22T08:14:30.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.267+0000 7fa9f55ff700 1 -- start start 2023-06-22T08:14:30.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9f55ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 0x7fa9f0151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:30.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9f55ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f0152ce0 0x7fa9f014a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:30.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9f55ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9f0158880 0x7fa9f01509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:30.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9f55ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f01ee440 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9f55ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f0150540 con 0x7fa9f0152ce0 2023-06-22T08:14:30.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9f55ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f01ee650 con 0x7fa9f0158880 2023-06-22T08:14:30.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9eeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 0x7fa9f0151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:30.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9eeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 0x7fa9f0151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35478/0 (socket says 172.21.15.23:35478) 2023-06-22T08:14:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9eeffd700 1 -- 172.21.15.23:0/1181799441 learned_addr learned my addr 172.21.15.23:0/1181799441 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9ee7fc700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f0152ce0 0x7fa9f014a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.268+0000 7fa9ef7fe700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9f0158880 0x7fa9f01509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.269+0000 7fa9eeffd700 1 -- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9f0158880 msgr2=0x7fa9f01509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.269+0000 7fa9eeffd700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9f0158880 0x7fa9f01509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.269+0000 7fa9eeffd700 1 -- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f0152ce0 msgr2=0x7fa9f014a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.269+0000 7fa9eeffd700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f0152ce0 0x7fa9f014a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:30.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.269+0000 7fa9eeffd700 1 -- 172.21.15.23:0/1181799441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e4009550 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.269+0000 7fa9eeffd700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 0x7fa9f0151d90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000e810 tx=0x7fa9e000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.270+0000 7fa9d77fe700 1 -- 172.21.15.23:0/1181799441 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000c070 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.270+0000 7fa9d77fe700 1 -- 172.21.15.23:0/1181799441 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e0013610 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.270+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1181799441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f01ee930 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.270+0000 7fa9d77fe700 1 -- 172.21.15.23:0/1181799441 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e00139c0 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.270+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1181799441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f01eeef0 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.271+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1181799441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f004ea20 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.272+0000 7fa9d77fe700 1 -- 172.21.15.23:0/1181799441 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa9e0013b20 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.273+0000 7fa9d77fe700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa9dc06ad30 0x7fa9dc06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:30.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.273+0000 7fa9d77fe700 1 -- 172.21.15.23:0/1181799441 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7fa9e00897c0 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.273+0000 7fa9ee7fc700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa9dc06ad30 0x7fa9dc06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:30.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.274+0000 7fa9ee7fc700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa9dc06ad30 0x7fa9dc06d1e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4014040 tx=0x7fa9e4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:30.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.277+0000 7fa9d77fe700 1 -- 172.21.15.23:0/1181799441 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa9e00542c0 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.533+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1181799441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9f0156320 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.534+0000 7fa9d77fe700 1 -- 172.21.15.23:0/1181799441 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5514 (secure 0 0 0) 0x7fa9e00578e0 con 0x7fa9f00c9fd0 2023-06-22T08:14:30.537 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:30.538 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:29.902075+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.536+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa9dc06ad30 msgr2=0x7fa9dc06d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.536+0000 7fa9f55ff700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa9dc06ad30 0x7fa9dc06d1e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4014040 tx=0x7fa9e4008030 comp rx=0 tx=0).stop 2023-06-22T08:14:30.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.536+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1181799441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 msgr2=0x7fa9f0151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.536+0000 7fa9f55ff700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 0x7fa9f0151d90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000e810 tx=0x7fa9e000eb20 comp rx=0 tx=0).stop 2023-06-22T08:14:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.536+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1181799441 shutdown_connections 2023-06-22T08:14:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.536+0000 7fa9f55ff700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9f0158880 0x7fa9f01509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.536+0000 7fa9f55ff700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9f00c9fd0 0x7fa9f0151d90 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.536+0000 7fa9f55ff700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa9dc06ad30 0x7fa9dc06d1e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.536+0000 7fa9f55ff700 1 --2- 172.21.15.23:0/1181799441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9f0152ce0 0x7fa9f014a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:30.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.536+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1181799441 >> 172.21.15.23:0/1181799441 conn(0x7fa9f006ae30 msgr2=0x7fa9f00cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.537+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1181799441 shutdown_connections 2023-06-22T08:14:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:30.537+0000 7fa9f55ff700 1 -- 172.21.15.23:0/1181799441 wait complete. 2023-06-22T08:14:30.542 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 114 2023-06-22T08:14:30.558 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=4, replacing gid=5211 2023-06-22T08:14:30.558 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.e 2023-06-22T08:14:35.824 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 49% (132/268) Updating files: 50% (134/268) Updating files: 51% (137/268) Updating files: 52% (140/268) Updating files: 53% (143/268) Updating files: 54% (145/268) Updating files: 54% (146/268) Updating files: 55% (148/268) Updating files: 56% (151/268) Updating files: 57% (153/268) Updating files: 58% (156/268) Updating files: 59% (159/268) Updating files: 60% (161/268) Updating files: 61% (164/268) Updating files: 62% (167/268) Updating files: 63% (169/268) Updating files: 64% (172/268) Updating files: 65% (175/268) Updating files: 66% (177/268) Updating files: 67% (180/268) Updating files: 68% (183/268) Updating files: 69% (185/268) Updating files: 70% (188/268) Updating files: 71% (191/268) Updating files: 72% (193/268) Updating files: 73% (196/268) Updating files: 74% (199/268) Updating files: 75% (201/268) Updating files: 76% (204/268) Updating files: 77% (207/268) Updating files: 78% (210/268) Updating files: 79% (212/268) Updating files: 80% (215/268) Updating files: 81% (218/268) Updating files: 82% (220/268) Updating files: 83% (223/268) Updating files: 84% (226/268) Updating files: 85% (228/268) Updating files: 86% (231/268) Updating files: 87% (234/268) Updating files: 88% (236/268) Updating files: 89% (239/268) Updating files: 90% (242/268) Updating files: 91% (244/268) Updating files: 92% (247/268) Updating files: 93% (250/268) Updating files: 94% (252/268) Updating files: 95% (255/268) Updating files: 96% (258/268) Updating files: 97% (260/268) Updating files: 98% (263/268) Updating files: 99% (266/268) Updating files: 100% (268/268) Updating files: 100% (268/268), done. 2023-06-22T08:14:36.045 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at ba3f91e7504 v0.58 2023-06-22T08:14:36.054 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.58 2023-06-22T08:14:36.129 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:14:36.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 59 -eq 48 ']' 2023-06-22T08:14:36.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.59 2023-06-22T08:14:36.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.59 2023-06-22T08:14:36.130 INFO:tasks.workunit.client.0.smithi023.stdout:v0.59 2023-06-22T08:14:36.131 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.59 2023-06-22T08:14:37.560 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-06-22T08:14:37.560 INFO:tasks.ceph.mds.e:Restarting daemon 2023-06-22T08:14:37.560 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-06-22T08:14:37.565 INFO:tasks.ceph.mds.e:Started 2023-06-22T08:14:37.565 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:14:37.720 INFO:tasks.ceph.mds.e.smithi023.stdout:starting mds.e at 2023-06-22T08:14:37.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.788+0000 7f9f8f5ff700 1 Processor -- start 2023-06-22T08:14:37.792 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.788+0000 7f9f8f5ff700 1 -- start start 2023-06-22T08:14:37.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.789+0000 7f9f8f5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f880c4d00 0x7f9f88159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:37.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.789+0000 7f9f8f5ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f880c9c30 con 0x7f9f880c97d0 2023-06-22T08:14:37.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f8f5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8815bea0 con 0x7f9f88152d70 2023-06-22T08:14:37.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f8f5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f880c95e0 con 0x7f9f880c4d00 2023-06-22T08:14:37.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f8d39b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9f88152d70 0x7f9f880cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:35572/0 (socket says 172.21.15.23:35572) 2023-06-22T08:14:37.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f8d39b700 1 -- 172.21.15.23:0/668822941 learned_addr learned my addr 172.21.15.23:0/668822941 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:14:37.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f8db9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f880c4d00 0x7f9f88159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:37.795 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f7ffff700 1 -- 172.21.15.23:0/668822941 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285861807 0 0) 0x7f9f8815bea0 con 0x7f9f88152d70 2023-06-22T08:14:37.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f7ffff700 1 -- 172.21.15.23:0/668822941 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f740035b0 con 0x7f9f88152d70 2023-06-22T08:14:37.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f7ffff700 1 -- 172.21.15.23:0/668822941 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181965446 0 0) 0x7f9f880c9c30 con 0x7f9f880c97d0 2023-06-22T08:14:37.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f7ffff700 1 -- 172.21.15.23:0/668822941 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f8815bea0 con 0x7f9f880c97d0 2023-06-22T08:14:37.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f8db9c700 1 -- 172.21.15.23:0/668822941 >> v1:172.21.15.67:6789/0 conn(0x7f9f88152d70 legacy=0x7f9f880cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:37.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f8db9c700 1 -- 172.21.15.23:0/668822941 >> v1:172.21.15.23:6789/0 conn(0x7f9f880c97d0 legacy=0x7f9f880c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:37.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.790+0000 7f9f8db9c700 1 -- 172.21.15.23:0/668822941 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8815c160 con 0x7f9f880c4d00 2023-06-22T08:14:37.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.791+0000 7f9f8db9c700 1 --2- 172.21.15.23:0/668822941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f880c4d00 0x7f9f88159330 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009d90 tx=0x7f9f8000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d74f9635f6f67b9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.791+0000 7f9f7ffff700 1 -- 172.21.15.23:0/668822941 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8000e070 con 0x7f9f880c4d00 2023-06-22T08:14:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.791+0000 7f9f7ffff700 1 -- 172.21.15.23:0/668822941 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8000bb20 con 0x7f9f880c4d00 2023-06-22T08:14:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.791+0000 7f9f7ffff700 1 -- 172.21.15.23:0/668822941 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80012430 con 0x7f9f880c4d00 2023-06-22T08:14:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.791+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/668822941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f880c4d00 msgr2=0x7f9f88159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.791+0000 7f9f8f5ff700 1 --2- 172.21.15.23:0/668822941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f880c4d00 0x7f9f88159330 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009d90 tx=0x7f9f8000b330 comp rx=0 tx=0).stop 2023-06-22T08:14:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.791+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/668822941 shutdown_connections 2023-06-22T08:14:37.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.791+0000 7f9f8f5ff700 1 --2- 172.21.15.23:0/668822941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f880c4d00 0x7f9f88159330 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:37.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.791+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/668822941 >> 172.21.15.23:0/668822941 conn(0x7f9f8806ae30 msgr2=0x7f9f8806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:37.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.792+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/668822941 shutdown_connections 2023-06-22T08:14:37.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.792+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/668822941 wait complete. 2023-06-22T08:14:37.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.794+0000 7f9f8f5ff700 1 Processor -- start 2023-06-22T08:14:37.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.794+0000 7f9f8f5ff700 1 -- start start 2023-06-22T08:14:37.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8f5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f880c4d00 0x7f9f88147f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:37.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8f5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f880c97d0 0x7f9f88159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:37.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8f5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f88152d70 0x7f9f880c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8f5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f880c7420 con 0x7f9f880c97d0 2023-06-22T08:14:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8f5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f88150830 con 0x7f9f88152d70 2023-06-22T08:14:37.800 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8f5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f881520d0 con 0x7f9f880c4d00 2023-06-22T08:14:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8cb9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f880c97d0 0x7f9f88159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8cb9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f880c97d0 0x7f9f88159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44466/0 (socket says 172.21.15.23:44466) 2023-06-22T08:14:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8db9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f88152d70 0x7f9f880c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8cb9a700 1 -- 172.21.15.23:0/2822918920 learned_addr learned my addr 172.21.15.23:0/2822918920 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:14:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.796+0000 7f9f8d39b700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f880c4d00 0x7f9f88147f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.797+0000 7f9f8db9c700 1 -- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f880c4d00 msgr2=0x7f9f88147f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.797+0000 7f9f8db9c700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f880c4d00 0x7f9f88147f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.797+0000 7f9f8db9c700 1 -- 172.21.15.23:0/2822918920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f880c97d0 msgr2=0x7f9f88159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.797+0000 7f9f8db9c700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f880c97d0 0x7f9f88159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.797+0000 7f9f8db9c700 1 -- 172.21.15.23:0/2822918920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f80009a70 con 0x7f9f88152d70 2023-06-22T08:14:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.797+0000 7f9f8cb9a700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f880c97d0 0x7f9f88159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:14:37.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.797+0000 7f9f8db9c700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f88152d70 0x7f9f880c6290 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9f8808ba40 tx=0x7f9f80004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:37.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.797+0000 7f9f7dffb700 1 -- 172.21.15.23:0/2822918920 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f80012430 con 0x7f9f88152d70 2023-06-22T08:14:37.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.798+0000 7f9f7dffb700 1 -- 172.21.15.23:0/2822918920 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f80020070 con 0x7f9f88152d70 2023-06-22T08:14:37.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.798+0000 7f9f7dffb700 1 -- 172.21.15.23:0/2822918920 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8001ca50 con 0x7f9f88152d70 2023-06-22T08:14:37.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.798+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/2822918920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f88150300 con 0x7f9f88152d70 2023-06-22T08:14:37.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.798+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/2822918920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f880c84d0 con 0x7f9f88152d70 2023-06-22T08:14:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.799+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/2822918920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f880c86d0 con 0x7f9f88152d70 2023-06-22T08:14:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.800+0000 7f9f7dffb700 1 -- 172.21.15.23:0/2822918920 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9f8001cbb0 con 0x7f9f88152d70 2023-06-22T08:14:37.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.801+0000 7f9f7dffb700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6c06ace0 0x7f9f6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:14:37.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.802+0000 7f9f7dffb700 1 -- 172.21.15.23:0/2822918920 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7f9f80022070 con 0x7f9f88152d70 2023-06-22T08:14:37.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.802+0000 7f9f8d39b700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6c06ace0 0x7f9f6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:14:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.802+0000 7f9f8d39b700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6c06ace0 0x7f9f6c06d190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9f84007fd0 tx=0x7f9f84007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:14:37.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:37.806+0000 7f9f7dffb700 1 -- 172.21.15.23:0/2822918920 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9f880c86d0 con 0x7f9f88152d70 2023-06-22T08:14:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.104+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/2822918920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f880c86d0 con 0x7f9f88152d70 2023-06-22T08:14:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.105+0000 7f9f7dffb700 1 -- 172.21.15.23:0/2822918920 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+6271 (secure 0 0 0) 0x7f9f880c86d0 con 0x7f9f88152d70 2023-06-22T08:14:38.108 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:14:38.109 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:14:29.902075+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:14:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6c06ace0 msgr2=0x7f9f6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6c06ace0 0x7f9f6c06d190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9f84007fd0 tx=0x7f9f84007de0 comp rx=0 tx=0).stop 2023-06-22T08:14:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f88152d70 msgr2=0x7f9f880c6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:14:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f88152d70 0x7f9f880c6290 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9f8808ba40 tx=0x7f9f80004ad0 comp rx=0 tx=0).stop 2023-06-22T08:14:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/2822918920 shutdown_connections 2023-06-22T08:14:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9f6c06ace0 0x7f9f6c06d190 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f88152d70 0x7f9f880c6290 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9f880c4d00 0x7f9f88147f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 --2- 172.21.15.23:0/2822918920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f880c97d0 0x7f9f88159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:14:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/2822918920 >> 172.21.15.23:0/2822918920 conn(0x7f9f8806ae30 msgr2=0x7f9f8806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:14:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.107+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/2822918920 shutdown_connections 2023-06-22T08:14:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:14:38.108+0000 7f9f8f5ff700 1 -- 172.21.15.23:0/2822918920 wait complete. 2023-06-22T08:14:38.113 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 115 2023-06-22T08:14:38.129 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-06-22T08:14:38.129 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2023-06-22T08:14:38.721 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 75% (159/211) Updating files: 76% (161/211) Updating files: 77% (163/211) Updating files: 78% (165/211) Updating files: 79% (167/211) Updating files: 80% (169/211) Updating files: 81% (171/211) Updating files: 82% (174/211) Updating files: 83% (176/211) Updating files: 84% (178/211) Updating files: 85% (180/211) Updating files: 86% (182/211) Updating files: 87% (184/211) Updating files: 88% (186/211) Updating files: 89% (188/211) Updating files: 90% (190/211) Updating files: 91% (193/211) Updating files: 92% (195/211) Updating files: 93% (197/211) Updating files: 94% (199/211) Updating files: 95% (201/211) Updating files: 96% (203/211) Updating files: 97% (205/211) Updating files: 98% (207/211) Updating files: 99% (209/211) Updating files: 100% (211/211) Updating files: 100% (211/211), done. 2023-06-22T08:14:38.803 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at cbae6a435c6 v0.59 2023-06-22T08:14:38.808 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.59 2023-06-22T08:14:38.903 INFO:tasks.workunit.client.0.smithi023.stdout:v0.60 2023-06-22T08:14:38.903 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:14:38.903 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 60 -eq 48 ']' 2023-06-22T08:14:38.904 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.60 2023-06-22T08:14:38.904 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.60 2023-06-22T08:14:38.904 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.60 2023-06-22T08:14:39.569 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:14:39.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:14:40.971 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 71% (184/256) Updating files: 72% (185/256) Updating files: 73% (187/256) Updating files: 74% (190/256) Updating files: 75% (192/256) Updating files: 76% (195/256) Updating files: 77% (198/256) Updating files: 78% (200/256) Updating files: 79% (203/256) Updating files: 80% (205/256) Updating files: 81% (208/256) Updating files: 82% (210/256) Updating files: 83% (213/256) Updating files: 84% (216/256) Updating files: 85% (218/256) Updating files: 86% (221/256) Updating files: 87% (223/256) Updating files: 88% (226/256) Updating files: 89% (228/256) Updating files: 90% (231/256) Updating files: 91% (233/256) Updating files: 92% (236/256) Updating files: 93% (239/256) Updating files: 94% (241/256) Updating files: 95% (244/256) Updating files: 96% (246/256) Updating files: 97% (249/256) Updating files: 98% (251/256) Updating files: 99% (254/256) Updating files: 100% (256/256) Updating files: 100% (256/256), done. 2023-06-22T08:14:41.122 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at f26f7a39021 v0.60 2023-06-22T08:14:41.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.60 2023-06-22T08:14:41.181 INFO:tasks.workunit.client.0.smithi023.stdout:v0.61 2023-06-22T08:14:41.181 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:14:41.182 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 61 -eq 48 ']' 2023-06-22T08:14:41.182 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.61 2023-06-22T08:14:41.182 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.61 2023-06-22T08:14:41.182 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.61 2023-06-22T08:14:43.335 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 59% (225/376) Updating files: 60% (226/376) Updating files: 61% (230/376) Updating files: 62% (234/376) Updating files: 63% (237/376) Updating files: 64% (241/376) Updating files: 65% (245/376) Updating files: 66% (249/376) Updating files: 67% (252/376) Updating files: 68% (256/376) Updating files: 69% (260/376) Updating files: 70% (264/376) Updating files: 71% (267/376) Updating files: 72% (271/376) Updating files: 73% (275/376) Updating files: 74% (279/376) Updating files: 75% (282/376) Updating files: 76% (286/376) Updating files: 77% (290/376) Updating files: 78% (294/376) Updating files: 79% (298/376) Updating files: 80% (301/376) Updating files: 81% (305/376) Updating files: 82% (309/376) Updating files: 83% (313/376) Updating files: 84% (316/376) Updating files: 85% (320/376) Updating files: 86% (324/376) Updating files: 87% (328/376) Updating files: 88% (331/376) Updating files: 89% (335/376) Updating files: 90% (339/376) Updating files: 91% (343/376) Updating files: 92% (346/376) Updating files: 93% (350/376) Updating files: 94% (354/376) Updating files: 95% (358/376) Updating files: 96% (361/376) Updating files: 97% (365/376) Updating files: 98% (369/376) Updating files: 99% (373/376) Updating files: 100% (376/376) Updating files: 100% (376/376), done. 2023-06-22T08:14:43.428 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 237f3f1e8d8 v0.61 2023-06-22T08:14:43.436 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.61 2023-06-22T08:14:43.514 INFO:tasks.workunit.client.0.smithi023.stdout:v0.62 2023-06-22T08:14:43.515 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:14:43.515 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 62 -eq 48 ']' 2023-06-22T08:14:43.515 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.62 2023-06-22T08:14:43.515 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.62 2023-06-22T08:14:43.516 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.62 2023-06-22T08:14:44.552 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 9382379cc6b v0.62 2023-06-22T08:14:44.555 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.62 2023-06-22T08:14:44.645 INFO:tasks.workunit.client.0.smithi023.stdout:v0.63 2023-06-22T08:14:44.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:14:44.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 63 -eq 48 ']' 2023-06-22T08:14:44.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.63 2023-06-22T08:14:44.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.63 2023-06-22T08:14:44.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.63 2023-06-22T08:14:46.218 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 054e96cf79e v0.63 2023-06-22T08:14:46.224 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.63 2023-06-22T08:14:46.278 INFO:tasks.workunit.client.0.smithi023.stdout:v0.64 2023-06-22T08:14:46.279 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:14:46.279 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 64 -eq 48 ']' 2023-06-22T08:14:46.279 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.64 2023-06-22T08:14:46.279 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.64 2023-06-22T08:14:46.280 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.64 2023-06-22T08:14:47.493 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 42e06c12db6 v0.64 2023-06-22T08:14:47.499 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.64 2023-06-22T08:14:47.576 INFO:tasks.workunit.client.0.smithi023.stdout:v0.65 2023-06-22T08:14:47.577 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:14:47.577 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 65 -eq 48 ']' 2023-06-22T08:14:47.577 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.65 2023-06-22T08:14:47.577 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.65 2023-06-22T08:14:47.577 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.65 2023-06-22T08:14:48.970 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 94% (192/203) Updating files: 95% (193/203) Updating files: 96% (195/203) Updating files: 97% (197/203) Updating files: 98% (199/203) Updating files: 99% (201/203) Updating files: 100% (203/203) Updating files: 100% (203/203), done. 2023-06-22T08:14:49.077 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at c2d517ef966 v0.65 2023-06-22T08:14:49.082 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.65 2023-06-22T08:14:49.162 INFO:tasks.workunit.client.0.smithi023.stdout:v0.66 2023-06-22T08:14:49.163 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:14:49.163 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 66 -eq 48 ']' 2023-06-22T08:14:49.163 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.66 2023-06-22T08:14:49.163 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.66 2023-06-22T08:14:49.163 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.66 2023-06-22T08:14:50.941 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 74% (113/152) Updating files: 75% (114/152) Updating files: 76% (116/152) Updating files: 77% (118/152) Updating files: 78% (119/152) Updating files: 79% (121/152) Updating files: 80% (122/152) Updating files: 81% (124/152) Updating files: 82% (125/152) Updating files: 83% (127/152) Updating files: 84% (128/152) Updating files: 85% (130/152) Updating files: 86% (131/152) Updating files: 87% (133/152) Updating files: 88% (134/152) Updating files: 89% (136/152) Updating files: 90% (137/152) Updating files: 91% (139/152) Updating files: 92% (140/152) Updating files: 93% (142/152) Updating files: 94% (143/152) Updating files: 95% (145/152) Updating files: 96% (146/152) Updating files: 97% (148/152) Updating files: 98% (149/152) Updating files: 99% (151/152) Updating files: 100% (152/152) Updating files: 100% (152/152), done. 2023-06-22T08:14:51.035 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at b6b48dbefad v0.66 2023-06-22T08:14:51.039 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.66 2023-06-22T08:14:51.116 INFO:tasks.workunit.client.0.smithi023.stdout:v0.67 2023-06-22T08:14:51.117 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:14:51.117 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 67 -eq 48 ']' 2023-06-22T08:14:51.118 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.67 2023-06-22T08:14:51.118 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.67 2023-06-22T08:14:51.118 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.67 2023-06-22T08:15:03.648 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 58% (208/355) Updating files: 59% (210/355) Updating files: 60% (213/355) Updating files: 61% (217/355) Updating files: 62% (221/355) Updating files: 63% (224/355) Updating files: 64% (228/355) Updating files: 65% (231/355) Updating files: 66% (235/355) Updating files: 67% (238/355) Updating files: 68% (242/355) Updating files: 68% (244/355) Updating files: 69% (245/355) Updating files: 70% (249/355) Updating files: 71% (253/355) Updating files: 72% (256/355) Updating files: 73% (260/355) Updating files: 74% (263/355) Updating files: 75% (267/355) Updating files: 76% (270/355) Updating files: 77% (274/355) Updating files: 78% (277/355) Updating files: 79% (281/355) Updating files: 80% (284/355) Updating files: 81% (288/355) Updating files: 82% (292/355) Updating files: 83% (295/355) Updating files: 84% (299/355) Updating files: 85% (302/355) Updating files: 86% (306/355) Updating files: 87% (309/355) Updating files: 88% (313/355) Updating files: 89% (316/355) Updating files: 90% (320/355) Updating files: 91% (324/355) Updating files: 91% (326/355) Updating files: 92% (327/355) Updating files: 93% (331/355) Updating files: 94% (334/355) Updating files: 95% (338/355) Updating files: 96% (341/355) Updating files: 97% (345/355) Updating files: 98% (348/355) Updating files: 99% (352/355) Updating files: 100% (355/355) Updating files: 100% (355/355), done. 2023-06-22T08:15:03.749 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at e3b7bc5bce8 v0.67 2023-06-22T08:15:03.756 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.67 2023-06-22T08:15:03.787 INFO:tasks.workunit.client.0.smithi023.stdout:v0.68 2023-06-22T08:15:03.787 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:03.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 68 -eq 48 ']' 2023-06-22T08:15:03.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.68 2023-06-22T08:15:03.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.68 2023-06-22T08:15:03.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.68 2023-06-22T08:15:07.459 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 31% (77/241) Updating files: 32% (78/241) Updating files: 33% (80/241) Updating files: 34% (82/241) Updating files: 35% (85/241) Updating files: 36% (87/241) Updating files: 37% (90/241) Updating files: 38% (92/241) Updating files: 39% (94/241) Updating files: 40% (97/241) Updating files: 41% (99/241) Updating files: 42% (102/241) Updating files: 43% (104/241) Updating files: 44% (107/241) Updating files: 45% (109/241) Updating files: 46% (111/241) Updating files: 47% (114/241) Updating files: 48% (116/241) Updating files: 49% (119/241) Updating files: 50% (121/241) Updating files: 51% (123/241) Updating files: 52% (126/241) Updating files: 53% (128/241) Updating files: 54% (131/241) Updating files: 55% (133/241) Updating files: 56% (135/241) Updating files: 57% (138/241) Updating files: 58% (140/241) Updating files: 59% (143/241) Updating files: 60% (145/241) Updating files: 61% (148/241) Updating files: 62% (150/241) Updating files: 63% (152/241) Updating files: 64% (155/241) Updating files: 65% (157/241) Updating files: 66% (160/241) Updating files: 66% (161/241) Updating files: 67% (162/241) Updating files: 68% (164/241) Updating files: 69% (167/241) Updating files: 70% (169/241) Updating files: 71% (172/241) Updating files: 72% (174/241) Updating files: 73% (176/241) Updating files: 74% (179/241) Updating files: 75% (181/241) Updating files: 76% (184/241) Updating files: 77% (186/241) Updating files: 78% (188/241) Updating files: 79% (191/241) Updating files: 80% (193/241) Updating files: 81% (196/241) Updating files: 82% (198/241) Updating files: 83% (201/241) Updating files: 84% (203/241) Updating files: 85% (205/241) Updating files: 86% (208/241) Updating files: 87% (210/241) Updating files: 88% (213/241) Updating files: 89% (215/241) Updating files: 90% (217/241) Updating files: 91% (220/241) Updating files: 92% (222/241) Updating files: 93% (225/241) Updating files: 94% (227/241) Updating files: 95% (229/241) Updating files: 96% (232/241) Updating files: 97% (234/241) Updating files: 98% (237/241) Updating files: 99% (239/241) Updating files: 100% (241/241) Updating files: 100% (241/241), done. 2023-06-22T08:15:07.598 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at b4cf0f2574b v0.68 2023-06-22T08:15:07.603 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.68 2023-06-22T08:15:07.720 INFO:tasks.workunit.client.0.smithi023.stdout:v0.69 2023-06-22T08:15:07.721 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:07.721 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 69 -eq 48 ']' 2023-06-22T08:15:07.721 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.69 2023-06-22T08:15:07.721 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.69 2023-06-22T08:15:07.721 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.69 2023-06-22T08:15:09.280 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 6ca6f2f9f75 v0.69 2023-06-22T08:15:09.286 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.69 2023-06-22T08:15:09.405 INFO:tasks.workunit.client.0.smithi023.stdout:v0.70 2023-06-22T08:15:09.405 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:09.405 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 70 -eq 48 ']' 2023-06-22T08:15:09.406 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.70 2023-06-22T08:15:09.406 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.70 2023-06-22T08:15:09.406 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.70 2023-06-22T08:15:09.685 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:15:09.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:15:10.269 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at e3bb0656d92 v0.70 2023-06-22T08:15:10.274 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.70 2023-06-22T08:15:10.333 INFO:tasks.workunit.client.0.smithi023.stdout:v0.71 2023-06-22T08:15:10.334 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:10.334 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 71 -eq 48 ']' 2023-06-22T08:15:10.334 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.71 2023-06-22T08:15:10.334 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.71 2023-06-22T08:15:10.335 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.71 2023-06-22T08:15:21.300 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 20% (85/405) Updating files: 21% (86/405) Updating files: 22% (90/405) Updating files: 23% (94/405) Updating files: 23% (95/405) Updating files: 24% (98/405) Updating files: 24% (100/405) Updating files: 25% (102/405) Updating files: 26% (106/405) Updating files: 27% (110/405) Updating files: 28% (114/405) Updating files: 29% (118/405) Updating files: 30% (122/405) Updating files: 31% (126/405) Updating files: 32% (130/405) Updating files: 33% (134/405) Updating files: 34% (138/405) Updating files: 35% (142/405) Updating files: 36% (146/405) Updating files: 37% (150/405) Updating files: 38% (154/405) Updating files: 39% (158/405) Updating files: 40% (162/405) Updating files: 41% (167/405) Updating files: 42% (171/405) Updating files: 43% (175/405) Updating files: 44% (179/405) Updating files: 45% (183/405) Updating files: 46% (187/405) Updating files: 47% (191/405) Updating files: 48% (195/405) Updating files: 49% (199/405) Updating files: 50% (203/405) Updating files: 51% (207/405) Updating files: 52% (211/405) Updating files: 53% (215/405) Updating files: 54% (219/405) Updating files: 55% (223/405) Updating files: 56% (227/405) Updating files: 56% (228/405) Updating files: 57% (231/405) Updating files: 58% (235/405) Updating files: 59% (239/405) Updating files: 60% (243/405) Updating files: 61% (248/405) Updating files: 62% (252/405) Updating files: 63% (256/405) Updating files: 64% (260/405) Updating files: 65% (264/405) Updating files: 66% (268/405) Updating files: 67% (272/405) Updating files: 68% (276/405) Updating files: 69% (280/405) Updating files: 70% (284/405) Updating files: 71% (288/405) Updating files: 72% (292/405) Updating files: 73% (296/405) Updating files: 74% (300/405) Updating files: 75% (304/405) Updating files: 76% (308/405) Updating files: 77% (312/405) Updating files: 78% (316/405) Updating files: 79% (320/405) Updating files: 80% (324/405) Updating files: 81% (329/405) Updating files: 82% (333/405) Updating files: 83% (337/405) Updating files: 84% (341/405) Updating files: 85% (345/405) Updating files: 86% (349/405) Updating files: 87% (353/405) Updating files: 88% (357/405) Updating files: 89% (361/405) Updating files: 90% (365/405) Updating files: 91% (369/405) Updating files: 92% (373/405) Updating files: 93% (377/405) Updating files: 94% (381/405) Updating files: 95% (385/405) Updating files: 96% (389/405) Updating files: 97% (393/405) Updating files: 98% (397/405) Updating files: 99% (401/405) Updating files: 100% (405/405) Updating files: 100% (405/405), done. 2023-06-22T08:15:21.405 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at e509cb1e69c v0.71 2023-06-22T08:15:21.412 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.71 2023-06-22T08:15:21.488 INFO:tasks.workunit.client.0.smithi023.stdout:v0.72 2023-06-22T08:15:21.488 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:21.488 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 72 -eq 48 ']' 2023-06-22T08:15:21.489 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.72 2023-06-22T08:15:21.489 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.72 2023-06-22T08:15:21.489 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.72 2023-06-22T08:15:22.718 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 100% (191/191) Updating files: 100% (191/191), done. 2023-06-22T08:15:22.843 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 5832e2603c7 v0.72 2023-06-22T08:15:22.850 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.72 2023-06-22T08:15:22.960 INFO:tasks.workunit.client.0.smithi023.stdout:v0.73 2023-06-22T08:15:22.960 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:22.961 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 73 -eq 48 ']' 2023-06-22T08:15:22.961 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.73 2023-06-22T08:15:22.962 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.73 2023-06-22T08:15:22.962 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.73 2023-06-22T08:15:24.549 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 97% (145/148) Updating files: 98% (146/148) Updating files: 99% (147/148) Updating files: 100% (148/148) Updating files: 100% (148/148), done. 2023-06-22T08:15:24.661 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at d8ad51ee8ab v0.73 2023-06-22T08:15:24.665 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.73 2023-06-22T08:15:24.780 INFO:tasks.workunit.client.0.smithi023.stdout:v0.74 2023-06-22T08:15:24.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:24.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 74 -eq 48 ']' 2023-06-22T08:15:24.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.74 2023-06-22T08:15:24.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.74 2023-06-22T08:15:24.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.74 2023-06-22T08:15:26.127 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 95% (183/192) Updating files: 96% (185/192) Updating files: 97% (187/192) Updating files: 98% (189/192) Updating files: 99% (191/192) Updating files: 100% (192/192) Updating files: 100% (192/192), done. 2023-06-22T08:15:26.250 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at c165483bc72 v0.74 2023-06-22T08:15:26.256 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.74 2023-06-22T08:15:26.351 INFO:tasks.workunit.client.0.smithi023.stdout:v0.75 2023-06-22T08:15:26.351 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:26.352 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 75 -eq 48 ']' 2023-06-22T08:15:26.352 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.75 2023-06-22T08:15:26.352 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.75 2023-06-22T08:15:26.352 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.75 2023-06-22T08:15:39.794 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:15:39.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:15:41.859 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 53% (156/291) Updating files: 54% (158/291) Updating files: 55% (161/291) Updating files: 56% (163/291) Updating files: 57% (166/291) Updating files: 58% (169/291) Updating files: 59% (172/291) Updating files: 60% (175/291) Updating files: 61% (178/291) Updating files: 62% (181/291) Updating files: 63% (184/291) Updating files: 64% (187/291) Updating files: 65% (190/291) Updating files: 66% (193/291) Updating files: 67% (195/291) Updating files: 68% (198/291) Updating files: 69% (201/291) Updating files: 70% (204/291) Updating files: 71% (207/291) Updating files: 72% (210/291) Updating files: 72% (212/291) Updating files: 73% (213/291) Updating files: 74% (216/291) Updating files: 75% (219/291) Updating files: 76% (222/291) Updating files: 77% (225/291) Updating files: 78% (227/291) Updating files: 79% (230/291) Updating files: 80% (233/291) Updating files: 81% (236/291) Updating files: 81% (238/291) Updating files: 82% (239/291) Updating files: 83% (242/291) Updating files: 84% (245/291) Updating files: 85% (248/291) Updating files: 86% (251/291) Updating files: 87% (254/291) Updating files: 88% (257/291) Updating files: 89% (259/291) Updating files: 90% (262/291) Updating files: 91% (265/291) Updating files: 92% (268/291) Updating files: 93% (271/291) Updating files: 94% (274/291) Updating files: 95% (277/291) Updating files: 96% (280/291) Updating files: 97% (283/291) Updating files: 98% (286/291) Updating files: 99% (289/291) Updating files: 100% (291/291) Updating files: 100% (291/291), done. 2023-06-22T08:15:41.947 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 946d6036958 v0.75 2023-06-22T08:15:41.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.75 2023-06-22T08:15:42.032 INFO:tasks.workunit.client.0.smithi023.stdout:v0.76 2023-06-22T08:15:42.033 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:42.033 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 76 -eq 48 ']' 2023-06-22T08:15:42.033 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.76 2023-06-22T08:15:42.033 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.76 2023-06-22T08:15:42.033 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.76 2023-06-22T08:15:43.488 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 3b990136bfa v0.76 2023-06-22T08:15:43.492 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.76 2023-06-22T08:15:43.583 INFO:tasks.workunit.client.0.smithi023.stdout:v0.77 2023-06-22T08:15:43.583 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:43.584 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 77 -eq 48 ']' 2023-06-22T08:15:43.584 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.77 2023-06-22T08:15:43.584 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.77 2023-06-22T08:15:43.584 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.77 2023-06-22T08:15:45.739 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 53% (158/298) Updating files: 54% (161/298) Updating files: 55% (164/298) Updating files: 56% (167/298) Updating files: 57% (170/298) Updating files: 58% (173/298) Updating files: 59% (176/298) Updating files: 60% (179/298) Updating files: 61% (182/298) Updating files: 62% (185/298) Updating files: 63% (188/298) Updating files: 64% (191/298) Updating files: 65% (194/298) Updating files: 66% (197/298) Updating files: 67% (200/298) Updating files: 68% (203/298) Updating files: 69% (206/298) Updating files: 70% (209/298) Updating files: 71% (212/298) Updating files: 72% (215/298) Updating files: 73% (218/298) Updating files: 74% (221/298) Updating files: 75% (224/298) Updating files: 76% (227/298) Updating files: 77% (230/298) Updating files: 78% (233/298) Updating files: 79% (236/298) Updating files: 80% (239/298) Updating files: 81% (242/298) Updating files: 82% (245/298) Updating files: 83% (248/298) Updating files: 84% (251/298) Updating files: 85% (254/298) Updating files: 86% (257/298) Updating files: 87% (260/298) Updating files: 88% (263/298) Updating files: 89% (266/298) Updating files: 90% (269/298) Updating files: 91% (272/298) Updating files: 92% (275/298) Updating files: 93% (278/298) Updating files: 94% (281/298) Updating files: 95% (284/298) Updating files: 96% (287/298) Updating files: 97% (290/298) Updating files: 98% (293/298) Updating files: 99% (296/298) Updating files: 100% (298/298) Updating files: 100% (298/298), done. 2023-06-22T08:15:45.824 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 1bca9c5c412 v0.77 2023-06-22T08:15:45.830 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.77 2023-06-22T08:15:45.951 INFO:tasks.workunit.client.0.smithi023.stdout:v0.78 2023-06-22T08:15:45.951 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:45.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 78 -eq 48 ']' 2023-06-22T08:15:45.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.78 2023-06-22T08:15:45.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.78 2023-06-22T08:15:45.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.78 2023-06-22T08:15:58.656 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 49% (246/496) Updating files: 50% (248/496) Updating files: 51% (253/496) Updating files: 52% (258/496) Updating files: 53% (263/496) Updating files: 54% (268/496) Updating files: 55% (273/496) Updating files: 56% (278/496) Updating files: 57% (283/496) Updating files: 58% (288/496) Updating files: 59% (293/496) Updating files: 60% (298/496) Updating files: 61% (303/496) Updating files: 62% (308/496) Updating files: 63% (313/496) Updating files: 64% (318/496) Updating files: 65% (323/496) Updating files: 66% (328/496) Updating files: 67% (333/496) Updating files: 68% (338/496) Updating files: 69% (343/496) Updating files: 69% (346/496) Updating files: 70% (348/496) Updating files: 71% (353/496) Updating files: 72% (358/496) Updating files: 73% (363/496) Updating files: 74% (368/496) Updating files: 75% (372/496) Updating files: 75% (373/496) Updating files: 76% (377/496) Updating files: 77% (382/496) Updating files: 78% (387/496) Updating files: 79% (392/496) Updating files: 80% (397/496) Updating files: 81% (402/496) Updating files: 82% (407/496) Updating files: 83% (412/496) Updating files: 84% (417/496) Updating files: 85% (422/496) Updating files: 86% (427/496) Updating files: 87% (432/496) Updating files: 88% (437/496) Updating files: 89% (442/496) Updating files: 90% (447/496) Updating files: 91% (452/496) Updating files: 92% (457/496) Updating files: 93% (462/496) Updating files: 94% (467/496) Updating files: 95% (472/496) Updating files: 96% (477/496) Updating files: 97% (482/496) Updating files: 98% (487/496) Updating files: 99% (492/496) Updating files: 100% (496/496) Updating files: 100% (496/496), done. 2023-06-22T08:15:58.782 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at f6c746c314d 0.78 2023-06-22T08:15:58.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.78 2023-06-22T08:15:58.906 INFO:tasks.workunit.client.0.smithi023.stdout:v0.79 2023-06-22T08:15:58.906 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:15:58.906 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 79 -eq 48 ']' 2023-06-22T08:15:58.907 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.79 2023-06-22T08:15:58.907 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.79 2023-06-22T08:15:58.907 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.79 2023-06-22T08:16:01.186 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 48% (158/329) Updating files: 49% (162/329) Updating files: 50% (165/329) Updating files: 51% (168/329) Updating files: 52% (172/329) Updating files: 53% (175/329) Updating files: 54% (178/329) Updating files: 55% (181/329) Updating files: 56% (185/329) Updating files: 57% (188/329) Updating files: 58% (191/329) Updating files: 59% (195/329) Updating files: 60% (198/329) Updating files: 61% (201/329) Updating files: 62% (204/329) Updating files: 63% (208/329) Updating files: 64% (211/329) Updating files: 65% (214/329) Updating files: 66% (218/329) Updating files: 67% (221/329) Updating files: 68% (224/329) Updating files: 69% (228/329) Updating files: 70% (231/329) Updating files: 71% (234/329) Updating files: 72% (237/329) Updating files: 73% (241/329) Updating files: 74% (244/329) Updating files: 75% (247/329) Updating files: 76% (251/329) Updating files: 77% (254/329) Updating files: 78% (257/329) Updating files: 79% (260/329) Updating files: 80% (264/329) Updating files: 81% (267/329) Updating files: 82% (270/329) Updating files: 83% (274/329) Updating files: 84% (277/329) Updating files: 85% (280/329) Updating files: 86% (283/329) Updating files: 87% (287/329) Updating files: 88% (290/329) Updating files: 89% (293/329) Updating files: 90% (297/329) Updating files: 91% (300/329) Updating files: 92% (303/329) Updating files: 93% (306/329) Updating files: 94% (310/329) Updating files: 95% (313/329) Updating files: 96% (316/329) Updating files: 97% (320/329) Updating files: 98% (323/329) Updating files: 99% (326/329) Updating files: 100% (329/329) Updating files: 100% (329/329), done. 2023-06-22T08:16:01.328 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 4c2d73a5095 0.79 2023-06-22T08:16:01.338 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.79 2023-06-22T08:16:01.517 INFO:tasks.workunit.client.0.smithi023.stdout:v0.80 2023-06-22T08:16:01.518 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:16:01.518 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 80 -eq 48 ']' 2023-06-22T08:16:01.518 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.80 2023-06-22T08:16:01.518 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.80 2023-06-22T08:16:01.518 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.80 2023-06-22T08:16:03.317 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 77% (136/176) Updating files: 78% (138/176) Updating files: 79% (140/176) Updating files: 80% (141/176) Updating files: 81% (143/176) Updating files: 82% (145/176) Updating files: 83% (147/176) Updating files: 84% (148/176) Updating files: 85% (150/176) Updating files: 86% (152/176) Updating files: 87% (154/176) Updating files: 88% (155/176) Updating files: 89% (157/176) Updating files: 90% (159/176) Updating files: 91% (161/176) Updating files: 92% (162/176) Updating files: 93% (164/176) Updating files: 94% (166/176) Updating files: 95% (168/176) Updating files: 96% (169/176) Updating files: 97% (171/176) Updating files: 98% (173/176) Updating files: 99% (175/176) Updating files: 100% (176/176) Updating files: 100% (176/176), done. 2023-06-22T08:16:03.459 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at b78644e7dee 0.80 2023-06-22T08:16:03.464 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.80 2023-06-22T08:16:03.530 INFO:tasks.workunit.client.0.smithi023.stdout:v0.81 2023-06-22T08:16:03.530 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:16:03.530 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 81 -eq 48 ']' 2023-06-22T08:16:03.530 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.81 2023-06-22T08:16:03.531 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.81 2023-06-22T08:16:03.531 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.81 2023-06-22T08:16:07.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:07.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.514+0000 7f83c03ff700 1 Processor -- start 2023-06-22T08:16:07.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.514+0000 7f83c03ff700 1 -- start start 2023-06-22T08:16:07.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.516+0000 7f83c03ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83b8069eb0 0x7f83b806d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:07.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.516+0000 7f83c03ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83b815d9b0 0x7f83b806a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:07.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.517+0000 7f83c03ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b80c4d70 con 0x7f83b80c5aa0 2023-06-22T08:16:07.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.517+0000 7f83c03ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b80703c0 con 0x7f83b8069eb0 2023-06-22T08:16:07.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.517+0000 7f83c03ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b806a7c0 con 0x7f83b815d9b0 2023-06-22T08:16:07.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.519+0000 7f83be19b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83b8069eb0 0x7f83b806d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:07.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.519+0000 7f83be19b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83b8069eb0 0x7f83b806d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:58928/0 (socket says 172.21.15.23:58928) 2023-06-22T08:16:07.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.519+0000 7f83be19b700 1 -- 172.21.15.23:0/1277400043 learned_addr learned my addr 172.21.15.23:0/1277400043 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:07.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.520+0000 7f83bd99a700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83b815d9b0 0x7f83b806a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:07.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.524+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755343116 0 0) 0x7f83b80c4d70 con 0x7f83b80c5aa0 2023-06-22T08:16:07.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.524+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a80035b0 con 0x7f83b80c5aa0 2023-06-22T08:16:07.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.524+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3002947187 0 0) 0x7f83a80035b0 con 0x7f83b80c5aa0 2023-06-22T08:16:07.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.524+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83b815d9b0 msgr2=0x7f83b806a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:07.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.524+0000 7f83bd199700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83b815d9b0 0x7f83b806a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:07.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.524+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 >> v2:172.21.15.67:3300/0 conn(0x7f83b8069eb0 msgr2=0x7f83b806d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:07.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.524+0000 7f83bd199700 1 --2- 172.21.15.23:0/1277400043 >> v2:172.21.15.67:3300/0 conn(0x7f83b8069eb0 0x7f83b806d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:07.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.525+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80c5610 con 0x7f83b80c5aa0 2023-06-22T08:16:07.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.525+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7f83b0001670 con 0x7f83b80c5aa0 2023-06-22T08:16:07.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.525+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 >> v1:172.21.15.23:6789/0 conn(0x7f83b80c5aa0 legacy=0x7f83b806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:07.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.525+0000 7f83bd199700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a8003a30 0x7f83a8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:07.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.525+0000 7f83bd199700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83a8004850 0x7f83a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:07.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.525+0000 7f83bd199700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83a8005200 0x7f83a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:07.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.525+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b80703c0 con 0x7f83a8004850 2023-06-22T08:16:07.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.525+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b806a7c0 con 0x7f83a8003a30 2023-06-22T08:16:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.525+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a801cf60 con 0x7f83a8005200 2023-06-22T08:16:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.526+0000 7f83be19b700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a8003a30 0x7f83a8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.526+0000 7f83be99c700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83a8005200 0x7f83a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.528+0000 7f83bd99a700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83a8004850 0x7f83a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:07.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.529+0000 7f83be19b700 1 -- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83a8005200 msgr2=0x7f83a800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:07.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.529+0000 7f83be19b700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83a8005200 0x7f83a800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:07.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.529+0000 7f83be19b700 1 -- 172.21.15.23:0/1277400043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83a8004850 msgr2=0x7f83a8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:07.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.529+0000 7f83be19b700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83a8004850 0x7f83a8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:07.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.529+0000 7f83be19b700 1 -- 172.21.15.23:0/1277400043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80c5610 con 0x7f83a8003a30 2023-06-22T08:16:07.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.532+0000 7f83be19b700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a8003a30 0x7f83a8003e40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00e1c0 tx=0x7f83a8005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c1c4abbba6d43d6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:07.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.533+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0092d0 con 0x7f83a8003a30 2023-06-22T08:16:07.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.533+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac00f040 con 0x7f83a8003a30 2023-06-22T08:16:07.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.533+0000 7f83bd199700 1 -- 172.21.15.23:0/1277400043 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac00ece0 con 0x7f83a8003a30 2023-06-22T08:16:07.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.534+0000 7f83c03ff700 1 -- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a8003a30 msgr2=0x7f83a8003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:07.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.534+0000 7f83c03ff700 1 --2- 172.21.15.23:0/1277400043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83a8003a30 0x7f83a8003e40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00e1c0 tx=0x7f83a8005880 comp rx=0 tx=0).stop 2023-06-22T08:16:07.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.534+0000 7f83be99c700 1 -- 172.21.15.23:0/1277400043 reap_dead start 2023-06-22T08:16:07.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.535+0000 7f83c03ff700 1 -- 172.21.15.23:0/1277400043 shutdown_connections 2023-06-22T08:16:07.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.535+0000 7f83c03ff700 1 -- 172.21.15.23:0/1277400043 >> 172.21.15.23:0/1277400043 conn(0x7f83b80ca3f0 msgr2=0x7f83b80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:07.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.536+0000 7f83c03ff700 1 -- 172.21.15.23:0/1277400043 shutdown_connections 2023-06-22T08:16:07.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.536+0000 7f83c03ff700 1 -- 172.21.15.23:0/1277400043 wait complete. 2023-06-22T08:16:07.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.538+0000 7f83c03ff700 1 Processor -- start 2023-06-22T08:16:07.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.539+0000 7f83c03ff700 1 -- start start 2023-06-22T08:16:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.540+0000 7f83c03ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83a8003a30 0x7f83b80c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.540+0000 7f83c03ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b81585b0 0x7f83b8145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.540+0000 7f83c03ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83b8145ba0 0x7f83b80c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:07.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.540+0000 7f83c03ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b81ee270 con 0x7f83b8145ba0 2023-06-22T08:16:07.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.540+0000 7f83c03ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b8144970 con 0x7f83b81585b0 2023-06-22T08:16:07.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.540+0000 7f83c03ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b8144ed0 con 0x7f83a8003a30 2023-06-22T08:16:07.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.541+0000 7f83bd99a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b81585b0 0x7f83b8145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:07.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.541+0000 7f83bd99a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b81585b0 0x7f83b8145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:58954/0 (socket says 172.21.15.23:58954) 2023-06-22T08:16:07.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.541+0000 7f83bd99a700 1 -- 172.21.15.23:0/3265366675 learned_addr learned my addr 172.21.15.23:0/3265366675 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.541+0000 7f83bd99a700 1 -- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83a8003a30 msgr2=0x7f83b80c95c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T08:16:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.541+0000 7f83bd99a700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83a8003a30 0x7f83b80c95c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.541+0000 7f83bd99a700 1 -- 172.21.15.23:0/3265366675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83b8145ba0 msgr2=0x7f83b80c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.541+0000 7f83bd99a700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83b8145ba0 0x7f83b80c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.541+0000 7f83bd99a700 1 -- 172.21.15.23:0/3265366675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b81edcd0 con 0x7f83b81585b0 2023-06-22T08:16:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.542+0000 7f83bd99a700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b81585b0 0x7f83b8145660 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f83a4011660 tx=0x7f83a4011a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.542+0000 7f83b6ffd700 1 -- 172.21.15.23:0/3265366675 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a4015870 con 0x7f83b81585b0 2023-06-22T08:16:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.543+0000 7f83b6ffd700 1 -- 172.21.15.23:0/3265366675 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a40159d0 con 0x7f83b81585b0 2023-06-22T08:16:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.543+0000 7f83b6ffd700 1 -- 172.21.15.23:0/3265366675 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a4015d80 con 0x7f83b81585b0 2023-06-22T08:16:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.543+0000 7f83c03ff700 1 -- 172.21.15.23:0/3265366675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b8158bd0 con 0x7f83b81585b0 2023-06-22T08:16:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.543+0000 7f83c03ff700 1 -- 172.21.15.23:0/3265366675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b8146bb0 con 0x7f83b81585b0 2023-06-22T08:16:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.545+0000 7f83c03ff700 1 -- 172.21.15.23:0/3265366675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b804ea20 con 0x7f83b81585b0 2023-06-22T08:16:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.547+0000 7f83b6ffd700 1 -- 172.21.15.23:0/3265366675 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f83a4013400 con 0x7f83b81585b0 2023-06-22T08:16:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.548+0000 7f83b6ffd700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f839c06adb0 0x7f839c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.548+0000 7f83b6ffd700 1 -- 172.21.15.23:0/3265366675 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7f83a4089e70 con 0x7f83b81585b0 2023-06-22T08:16:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.548+0000 7f83be19b700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f839c06adb0 0x7f839c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.550+0000 7f83be19b700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f839c06adb0 0x7f839c06d260 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00e1c0 tx=0x7f83ac013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:07.559 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.556+0000 7f83b6ffd700 1 -- 172.21.15.23:0/3265366675 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f83a40548d0 con 0x7f83b81585b0 2023-06-22T08:16:07.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.812+0000 7f83c03ff700 1 -- 172.21.15.23:0/3265366675 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83b80cb300 con 0x7f83b81585b0 2023-06-22T08:16:07.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.813+0000 7f83b6ffd700 1 -- 172.21.15.23:0/3265366675 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6274 (secure 0 0 0) 0x7f83a4023070 con 0x7f83b81585b0 2023-06-22T08:16:07.817 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:07.817 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:00.928887+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:07.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.815+0000 7f83c03ff700 1 -- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f839c06adb0 msgr2=0x7f839c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:07.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.815+0000 7f83c03ff700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f839c06adb0 0x7f839c06d260 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00e1c0 tx=0x7f83ac013000 comp rx=0 tx=0).stop 2023-06-22T08:16:07.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.816+0000 7f83c03ff700 1 -- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b81585b0 msgr2=0x7f83b8145660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:07.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.816+0000 7f83c03ff700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b81585b0 0x7f83b8145660 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f83a4011660 tx=0x7f83a4011a20 comp rx=0 tx=0).stop 2023-06-22T08:16:07.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.816+0000 7f83c03ff700 1 -- 172.21.15.23:0/3265366675 shutdown_connections 2023-06-22T08:16:07.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.816+0000 7f83c03ff700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f83b8145ba0 0x7f83b80c2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:07.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.816+0000 7f83c03ff700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f83a8003a30 0x7f83b80c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:07.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.816+0000 7f83c03ff700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f839c06adb0 0x7f839c06d260 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:07.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.816+0000 7f83c03ff700 1 --2- 172.21.15.23:0/3265366675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b81585b0 0x7f83b8145660 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:07.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.816+0000 7f83c03ff700 1 -- 172.21.15.23:0/3265366675 >> 172.21.15.23:0/3265366675 conn(0x7f83b80ca3f0 msgr2=0x7f83b8151ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:07.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.816+0000 7f83c03ff700 1 -- 172.21.15.23:0/3265366675 shutdown_connections 2023-06-22T08:16:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:07.816+0000 7f83c03ff700 1 -- 172.21.15.23:0/3265366675 wait complete. 2023-06-22T08:16:07.824 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 130 2023-06-22T08:16:07.838 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2023-06-22T08:16:07.838 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-22T08:16:07.839 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:16:07.868 INFO:tasks.ceph.mds.c:Stopped 2023-06-22T08:16:07.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:08.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.108+0000 7fdb353ff700 1 Processor -- start 2023-06-22T08:16:08.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.108+0000 7fdb353ff700 1 -- start start 2023-06-22T08:16:08.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.109+0000 7fdb353ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb30152ce0 0x7fdb300cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:08.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.109+0000 7fdb353ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb300c9fd0 0x7fdb3015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:08.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.109+0000 7fdb353ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3015c370 con 0x7fdb30158880 2023-06-22T08:16:08.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.109+0000 7fdb353ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb301530b0 con 0x7fdb30152ce0 2023-06-22T08:16:08.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.109+0000 7fdb353ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300cf6f0 con 0x7fdb300c9fd0 2023-06-22T08:16:08.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2effd700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb30152ce0 0x7fdb300cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:08.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2f7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fdb30158880 0x7fdb30158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52674/0 (socket says 172.21.15.23:52674) 2023-06-22T08:16:08.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2f7fe700 1 -- 172.21.15.23:0/171125735 learned_addr learned my addr 172.21.15.23:0/171125735 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:08.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2e7fc700 1 --2- 172.21.15.23:0/171125735 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb300c9fd0 0x7fdb3015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:08.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2effd700 1 -- 172.21.15.23:0/171125735 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb300c9fd0 msgr2=0x7fdb3015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:08.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2effd700 1 --2- 172.21.15.23:0/171125735 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb300c9fd0 0x7fdb3015c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:08.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2effd700 1 -- 172.21.15.23:0/171125735 >> v1:172.21.15.23:6789/0 conn(0x7fdb30158880 legacy=0x7fdb30158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:08.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2effd700 1 -- 172.21.15.23:0/171125735 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3015bda0 con 0x7fdb30152ce0 2023-06-22T08:16:08.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2e7fc700 1 --2- 172.21.15.23:0/171125735 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb300c9fd0 0x7fdb3015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:16:08.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2dffb700 1 -- 172.21.15.23:0/171125735 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508351889 0 0) 0x7fdb3015c370 con 0x7fdb30158880 2023-06-22T08:16:08.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.110+0000 7fdb2effd700 1 --2- 172.21.15.23:0/171125735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb30152ce0 0x7fdb300cf1b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdb20009840 tx=0x7fdb20009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=819c7fed0cb59200 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:08.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb2dffb700 1 -- 172.21.15.23:0/171125735 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20011830 con 0x7fdb30152ce0 2023-06-22T08:16:08.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb2dffb700 1 -- 172.21.15.23:0/171125735 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb20011990 con 0x7fdb30152ce0 2023-06-22T08:16:08.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb2dffb700 1 -- 172.21.15.23:0/171125735 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20011d20 con 0x7fdb30152ce0 2023-06-22T08:16:08.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb353ff700 1 -- 172.21.15.23:0/171125735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb30152ce0 msgr2=0x7fdb300cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:08.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb353ff700 1 --2- 172.21.15.23:0/171125735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb30152ce0 0x7fdb300cf1b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdb20009840 tx=0x7fdb20009b50 comp rx=0 tx=0).stop 2023-06-22T08:16:08.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb353ff700 1 -- 172.21.15.23:0/171125735 shutdown_connections 2023-06-22T08:16:08.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb353ff700 1 --2- 172.21.15.23:0/171125735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb30152ce0 0x7fdb300cf1b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:08.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb353ff700 1 --2- 172.21.15.23:0/171125735 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb300c9fd0 0x7fdb3015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:08.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb353ff700 1 -- 172.21.15.23:0/171125735 >> 172.21.15.23:0/171125735 conn(0x7fdb3006ae30 msgr2=0x7fdb3006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:08.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb353ff700 1 -- 172.21.15.23:0/171125735 shutdown_connections 2023-06-22T08:16:08.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.111+0000 7fdb353ff700 1 -- 172.21.15.23:0/171125735 wait complete. 2023-06-22T08:16:08.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.113+0000 7fdb353ff700 1 Processor -- start 2023-06-22T08:16:08.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.113+0000 7fdb353ff700 1 -- start start 2023-06-22T08:16:08.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.114+0000 7fdb353ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb300c9fd0 0x7fdb3015af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:08.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.114+0000 7fdb353ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb30152ce0 0x7fdb300cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:08.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.114+0000 7fdb353ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb30158880 0x7fdb300b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:08.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.114+0000 7fdb353ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb300b4770 con 0x7fdb300c9fd0 2023-06-22T08:16:08.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.114+0000 7fdb353ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3014db90 con 0x7fdb30158880 2023-06-22T08:16:08.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.114+0000 7fdb353ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3015b8a0 con 0x7fdb30152ce0 2023-06-22T08:16:08.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb300c9fd0 0x7fdb3015af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:08.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2e7fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb30152ce0 0x7fdb300cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:08.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb300c9fd0 0x7fdb3015af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41998/0 (socket says 172.21.15.23:41998) 2023-06-22T08:16:08.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2effd700 1 -- 172.21.15.23:0/660711338 learned_addr learned my addr 172.21.15.23:0/660711338 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:08.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2f7fe700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb30158880 0x7fdb300b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:08.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2effd700 1 -- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb30152ce0 msgr2=0x7fdb300cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:08.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2effd700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb30152ce0 0x7fdb300cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:08.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2effd700 1 -- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb30158880 msgr2=0x7fdb300b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:08.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2effd700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb30158880 0x7fdb300b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:08.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2effd700 1 -- 172.21.15.23:0/660711338 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb240094a0 con 0x7fdb300c9fd0 2023-06-22T08:16:08.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb2effd700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb300c9fd0 0x7fdb3015af10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdb20009fd0 tx=0x7fdb20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:08.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb177fe700 1 -- 172.21.15.23:0/660711338 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb200036a0 con 0x7fdb300c9fd0 2023-06-22T08:16:08.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.115+0000 7fdb177fe700 1 -- 172.21.15.23:0/660711338 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2001f070 con 0x7fdb300c9fd0 2023-06-22T08:16:08.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.116+0000 7fdb177fe700 1 -- 172.21.15.23:0/660711338 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20003a60 con 0x7fdb300c9fd0 2023-06-22T08:16:08.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.116+0000 7fdb353ff700 1 -- 172.21.15.23:0/660711338 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb200094a0 con 0x7fdb300c9fd0 2023-06-22T08:16:08.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.116+0000 7fdb353ff700 1 -- 172.21.15.23:0/660711338 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb300b3ed0 con 0x7fdb300c9fd0 2023-06-22T08:16:08.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.117+0000 7fdb353ff700 1 -- 172.21.15.23:0/660711338 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb300742e0 con 0x7fdb300c9fd0 2023-06-22T08:16:08.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.117+0000 7fdb177fe700 1 -- 172.21.15.23:0/660711338 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdb20003bc0 con 0x7fdb300c9fd0 2023-06-22T08:16:08.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.118+0000 7fdb177fe700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdb1c06ace0 0x7fdb1c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:08.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.118+0000 7fdb177fe700 1 -- 172.21.15.23:0/660711338 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7fdb20021070 con 0x7fdb300c9fd0 2023-06-22T08:16:08.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.119+0000 7fdb2e7fc700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdb1c06ace0 0x7fdb1c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:08.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.119+0000 7fdb2e7fc700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdb1c06ace0 0x7fdb1c06d190 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdb300c4d70 tx=0x7fdb24009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:08.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.123+0000 7fdb177fe700 1 -- 172.21.15.23:0/660711338 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdb20055d10 con 0x7fdb300c9fd0 2023-06-22T08:16:08.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.401+0000 7fdb353ff700 1 -- 172.21.15.23:0/660711338 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb3006b760 con 0x7fdb300c9fd0 2023-06-22T08:16:08.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.402+0000 7fdb177fe700 1 -- 172.21.15.23:0/660711338 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6274 (secure 0 0 0) 0x7fdb20059330 con 0x7fdb300c9fd0 2023-06-22T08:16:08.405 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:08.405 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:00.928887+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:08.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 -- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdb1c06ace0 msgr2=0x7fdb1c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:08.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdb1c06ace0 0x7fdb1c06d190 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdb300c4d70 tx=0x7fdb24009210 comp rx=0 tx=0).stop 2023-06-22T08:16:08.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 -- 172.21.15.23:0/660711338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb300c9fd0 msgr2=0x7fdb3015af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:08.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb300c9fd0 0x7fdb3015af10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdb20009fd0 tx=0x7fdb20004750 comp rx=0 tx=0).stop 2023-06-22T08:16:08.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 -- 172.21.15.23:0/660711338 shutdown_connections 2023-06-22T08:16:08.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdb1c06ace0 0x7fdb1c06d190 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:08.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb30158880 0x7fdb300b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:08.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb300c9fd0 0x7fdb3015af10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:08.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 --2- 172.21.15.23:0/660711338 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdb30152ce0 0x7fdb300cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:08.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 -- 172.21.15.23:0/660711338 >> 172.21.15.23:0/660711338 conn(0x7fdb3006ae30 msgr2=0x7fdb30156b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:08.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 -- 172.21.15.23:0/660711338 shutdown_connections 2023-06-22T08:16:08.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:08.404+0000 7fdb353ff700 1 -- 172.21.15.23:0/660711338 wait complete. 2023-06-22T08:16:08.410 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 130 2023-06-22T08:16:08.426 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:16:08.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2023-06-22T08:16:09.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:16:09.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:16:10.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:10.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.637+0000 7fc9849ff700 1 Processor -- start 2023-06-22T08:16:10.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.637+0000 7fc9849ff700 1 -- start start 2023-06-22T08:16:10.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.639+0000 7fc9849ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc9800ccc80 0x7fc9801504d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:10.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.639+0000 7fc9849ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9800c1670 con 0x7fc9800cc8b0 2023-06-22T08:16:10.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.639+0000 7fc9849ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc980157030 con 0x7fc98006e150 2023-06-22T08:16:10.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.639+0000 7fc9849ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc980157310 con 0x7fc9800ccc80 2023-06-22T08:16:10.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.639+0000 7fc97ed9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc9800ccc80 0x7fc9801504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:10.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.639+0000 7fc97e59c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc9800cc8b0 0x7fc9800c20c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52690/0 (socket says 172.21.15.23:52690) 2023-06-22T08:16:10.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.639+0000 7fc97e59c700 1 -- 172.21.15.23:0/503894586 learned_addr learned my addr 172.21.15.23:0/503894586 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:10.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.639+0000 7fc97ed9d700 1 -- 172.21.15.23:0/503894586 >> v1:172.21.15.67:6789/0 conn(0x7fc98006e150 legacy=0x7fc980156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:10.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.639+0000 7fc97ed9d700 1 -- 172.21.15.23:0/503894586 >> v1:172.21.15.23:6789/0 conn(0x7fc9800cc8b0 legacy=0x7fc9800c20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:10.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.639+0000 7fc97ed9d700 1 -- 172.21.15.23:0/503894586 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9801575f0 con 0x7fc9800ccc80 2023-06-22T08:16:10.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.640+0000 7fc97ed9d700 1 --2- 172.21.15.23:0/503894586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9800ccc80 0x7fc9801504d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc978009d90 tx=0x7fc97800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7582c32161a84cf7 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:10.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.640+0000 7fc97d59a700 1 -- 172.21.15.23:0/503894586 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97800e070 con 0x7fc9800ccc80 2023-06-22T08:16:10.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.640+0000 7fc97d59a700 1 -- 172.21.15.23:0/503894586 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97800bbd0 con 0x7fc9800ccc80 2023-06-22T08:16:10.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.640+0000 7fc97d59a700 1 -- 172.21.15.23:0/503894586 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc978012430 con 0x7fc9800ccc80 2023-06-22T08:16:10.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.640+0000 7fc9849ff700 1 -- 172.21.15.23:0/503894586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9800ccc80 msgr2=0x7fc9801504d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:10.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.641+0000 7fc9849ff700 1 --2- 172.21.15.23:0/503894586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9800ccc80 0x7fc9801504d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc978009d90 tx=0x7fc97800b330 comp rx=0 tx=0).stop 2023-06-22T08:16:10.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.641+0000 7fc9849ff700 1 -- 172.21.15.23:0/503894586 shutdown_connections 2023-06-22T08:16:10.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.641+0000 7fc9849ff700 1 --2- 172.21.15.23:0/503894586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9800ccc80 0x7fc9801504d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:10.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.641+0000 7fc9849ff700 1 -- 172.21.15.23:0/503894586 >> 172.21.15.23:0/503894586 conn(0x7fc98006fb70 msgr2=0x7fc98006ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:10.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.641+0000 7fc9849ff700 1 -- 172.21.15.23:0/503894586 shutdown_connections 2023-06-22T08:16:10.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.641+0000 7fc9849ff700 1 -- 172.21.15.23:0/503894586 wait complete. 2023-06-22T08:16:10.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.642+0000 7fc9849ff700 1 Processor -- start 2023-06-22T08:16:10.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.643+0000 7fc9849ff700 1 -- start start 2023-06-22T08:16:10.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc9849ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc98006e150 0x7fc980154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:10.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc9849ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9800cc8b0 0x7fc9801f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:10.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc9849ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc9800ccc80 0x7fc98014b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:10.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc9849ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9800c6390 con 0x7fc98006e150 2023-06-22T08:16:10.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc9849ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9801569f0 con 0x7fc9800cc8b0 2023-06-22T08:16:10.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc9849ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9800c6630 con 0x7fc9800ccc80 2023-06-22T08:16:10.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc97e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc98006e150 0x7fc980154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:10.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc97e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc98006e150 0x7fc980154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42028/0 (socket says 172.21.15.23:42028) 2023-06-22T08:16:10.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc97e59c700 1 -- 172.21.15.23:0/3163739376 learned_addr learned my addr 172.21.15.23:0/3163739376 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:10.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc97dd9b700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9800cc8b0 0x7fc9801f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:10.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc97ed9d700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc9800ccc80 0x7fc98014b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:10.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc97e59c700 1 -- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc9800ccc80 msgr2=0x7fc98014b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:10.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc97e59c700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc9800ccc80 0x7fc98014b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:10.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc97e59c700 1 -- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9800cc8b0 msgr2=0x7fc9801f2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:10.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc97e59c700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9800cc8b0 0x7fc9801f2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:10.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.644+0000 7fc97e59c700 1 -- 172.21.15.23:0/3163739376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc978009a70 con 0x7fc98006e150 2023-06-22T08:16:10.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.645+0000 7fc97e59c700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc98006e150 0x7fc980154f60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc97000c700 tx=0x7fc97000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:10.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.645+0000 7fc96effd700 1 -- 172.21.15.23:0/3163739376 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9700072e0 con 0x7fc98006e150 2023-06-22T08:16:10.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.645+0000 7fc96effd700 1 -- 172.21.15.23:0/3163739376 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc970007440 con 0x7fc98006e150 2023-06-22T08:16:10.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.645+0000 7fc9849ff700 1 -- 172.21.15.23:0/3163739376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9800c6910 con 0x7fc98006e150 2023-06-22T08:16:10.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.645+0000 7fc96effd700 1 -- 172.21.15.23:0/3163739376 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970005830 con 0x7fc98006e150 2023-06-22T08:16:10.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.645+0000 7fc9849ff700 1 -- 172.21.15.23:0/3163739376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc980156cd0 con 0x7fc98006e150 2023-06-22T08:16:10.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.646+0000 7fc9849ff700 1 -- 172.21.15.23:0/3163739376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98004ea20 con 0x7fc98006e150 2023-06-22T08:16:10.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.647+0000 7fc96effd700 1 -- 172.21.15.23:0/3163739376 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc970005990 con 0x7fc98006e150 2023-06-22T08:16:10.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.648+0000 7fc96effd700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc96006ace0 0x7fc96006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:10.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.648+0000 7fc96effd700 1 -- 172.21.15.23:0/3163739376 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7fc9700aac90 con 0x7fc98006e150 2023-06-22T08:16:10.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.648+0000 7fc97dd9b700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc96006ace0 0x7fc96006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:10.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.649+0000 7fc97dd9b700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc96006ace0 0x7fc96006d190 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc98006f1c0 tx=0x7fc974009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:10.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.653+0000 7fc96effd700 1 -- 172.21.15.23:0/3163739376 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc970075720 con 0x7fc98006e150 2023-06-22T08:16:10.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.908+0000 7fc9849ff700 1 -- 172.21.15.23:0/3163739376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9801541a0 con 0x7fc98006e150 2023-06-22T08:16:10.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.908+0000 7fc96effd700 1 -- 172.21.15.23:0/3163739376 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6274 (secure 0 0 0) 0x7fc970039020 con 0x7fc98006e150 2023-06-22T08:16:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:10.912 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:00.928887+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:10.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.911+0000 7fc9849ff700 1 -- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc96006ace0 msgr2=0x7fc96006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:10.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.911+0000 7fc9849ff700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc96006ace0 0x7fc96006d190 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc98006f1c0 tx=0x7fc974009040 comp rx=0 tx=0).stop 2023-06-22T08:16:10.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.911+0000 7fc9849ff700 1 -- 172.21.15.23:0/3163739376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc98006e150 msgr2=0x7fc980154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:10.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.911+0000 7fc9849ff700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc98006e150 0x7fc980154f60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc97000c700 tx=0x7fc97000ca10 comp rx=0 tx=0).stop 2023-06-22T08:16:10.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.912+0000 7fc9849ff700 1 -- 172.21.15.23:0/3163739376 shutdown_connections 2023-06-22T08:16:10.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.912+0000 7fc9849ff700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc9800ccc80 0x7fc98014b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:10.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.912+0000 7fc9849ff700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc98006e150 0x7fc980154f60 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:10.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.912+0000 7fc9849ff700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc96006ace0 0x7fc96006d190 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:10.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.912+0000 7fc9849ff700 1 --2- 172.21.15.23:0/3163739376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9800cc8b0 0x7fc9801f2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:10.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.912+0000 7fc9849ff700 1 -- 172.21.15.23:0/3163739376 >> 172.21.15.23:0/3163739376 conn(0x7fc98006fb70 msgr2=0x7fc98014ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:10.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.912+0000 7fc9849ff700 1 -- 172.21.15.23:0/3163739376 shutdown_connections 2023-06-22T08:16:10.917 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:10.912+0000 7fc9849ff700 1 -- 172.21.15.23:0/3163739376 wait complete. 2023-06-22T08:16:10.917 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 130 2023-06-22T08:16:10.934 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:16:12.935 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:13.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.170+0000 7ff0529ff700 1 Processor -- start 2023-06-22T08:16:13.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.170+0000 7ff0529ff700 1 -- start start 2023-06-22T08:16:13.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.172+0000 7ff0529ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff04c14f0c0 0x7ff04c1558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:13.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.173+0000 7ff0529ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c0c4ea0 con 0x7ff04c0c6610 2023-06-22T08:16:13.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.173+0000 7ff0529ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04c0c5140 con 0x7ff04c0c61e0 2023-06-22T08:16:13.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.173+0000 7ff0529ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04c0c1f70 con 0x7ff04c14f0c0 2023-06-22T08:16:13.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.173+0000 7ff04bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff04c14f0c0 0x7ff04c1558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:13.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.173+0000 7ff050f9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff04c0c6610 0x7ff04c06d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52694/0 (socket says 172.21.15.23:52694) 2023-06-22T08:16:13.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.173+0000 7ff050f9c700 1 -- 172.21.15.23:0/3287469862 learned_addr learned my addr 172.21.15.23:0/3287469862 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:13.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.173+0000 7ff04bfff700 1 -- 172.21.15.23:0/3287469862 >> v1:172.21.15.67:6789/0 conn(0x7ff04c0c61e0 legacy=0x7ff04c06df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:13.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.173+0000 7ff04bfff700 1 -- 172.21.15.23:0/3287469862 >> v1:172.21.15.23:6789/0 conn(0x7ff04c0c6610 legacy=0x7ff04c06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:13.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.173+0000 7ff04bfff700 1 -- 172.21.15.23:0/3287469862 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04c0c5420 con 0x7ff04c14f0c0 2023-06-22T08:16:13.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.173+0000 7ff04affd700 1 -- 172.21.15.23:0/3287469862 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673472320 0 0) 0x7ff04c0c4ea0 con 0x7ff04c0c6610 2023-06-22T08:16:13.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.174+0000 7ff04bfff700 1 --2- 172.21.15.23:0/3287469862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 0x7ff04c1558b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00a9e0 tx=0x7ff03c00acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee4bf631c11bc34 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:13.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.174+0000 7ff04affd700 1 -- 172.21.15.23:0/3287469862 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c00cb70 con 0x7ff04c14f0c0 2023-06-22T08:16:13.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.174+0000 7ff04affd700 1 -- 172.21.15.23:0/3287469862 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c00ccd0 con 0x7ff04c14f0c0 2023-06-22T08:16:13.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.174+0000 7ff04affd700 1 -- 172.21.15.23:0/3287469862 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c018620 con 0x7ff04c14f0c0 2023-06-22T08:16:13.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.174+0000 7ff0529ff700 1 -- 172.21.15.23:0/3287469862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 msgr2=0x7ff04c1558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:13.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.175+0000 7ff0529ff700 1 --2- 172.21.15.23:0/3287469862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 0x7ff04c1558b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00a9e0 tx=0x7ff03c00acf0 comp rx=0 tx=0).stop 2023-06-22T08:16:13.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.175+0000 7ff0529ff700 1 -- 172.21.15.23:0/3287469862 shutdown_connections 2023-06-22T08:16:13.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.175+0000 7ff0529ff700 1 --2- 172.21.15.23:0/3287469862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 0x7ff04c1558b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:13.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.175+0000 7ff0529ff700 1 -- 172.21.15.23:0/3287469862 >> 172.21.15.23:0/3287469862 conn(0x7ff04c0ca3f0 msgr2=0x7ff04c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:13.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.175+0000 7ff0529ff700 1 -- 172.21.15.23:0/3287469862 shutdown_connections 2023-06-22T08:16:13.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.175+0000 7ff0529ff700 1 -- 172.21.15.23:0/3287469862 wait complete. 2023-06-22T08:16:13.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.176+0000 7ff0529ff700 1 Processor -- start 2023-06-22T08:16:13.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.177+0000 7ff0529ff700 1 -- start start 2023-06-22T08:16:13.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff0529ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff04c0c61e0 0x7ff04c1e04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:13.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff0529ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff04c0c6610 0x7ff04c13d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:13.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff0529ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 0x7ff04c1ee500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:13.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff0529ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c154dc0 con 0x7ff04c14f0c0 2023-06-22T08:16:13.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff0529ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c13d600 con 0x7ff04c0c61e0 2023-06-22T08:16:13.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff0529ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04c0c2e80 con 0x7ff04c0c6610 2023-06-22T08:16:13.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff050f9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 0x7ff04c1ee500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:13.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff04bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff04c0c61e0 0x7ff04c1e04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:13.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff050f9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 0x7ff04c1ee500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42042/0 (socket says 172.21.15.23:42042) 2023-06-22T08:16:13.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff050f9c700 1 -- 172.21.15.23:0/2562562743 learned_addr learned my addr 172.21.15.23:0/2562562743 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:13.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff04b7fe700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff04c0c6610 0x7ff04c13d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:13.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff050f9c700 1 -- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff04c0c6610 msgr2=0x7ff04c13d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:13.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff050f9c700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff04c0c6610 0x7ff04c13d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:13.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff050f9c700 1 -- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff04c0c61e0 msgr2=0x7ff04c1e04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:13.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff050f9c700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff04c0c61e0 0x7ff04c1e04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:13.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff050f9c700 1 -- 172.21.15.23:0/2562562743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c00a640 con 0x7ff04c14f0c0 2023-06-22T08:16:13.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.178+0000 7ff04bfff700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff04c0c61e0 0x7ff04c1e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:16:13.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.179+0000 7ff050f9c700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 0x7ff04c1ee500 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff04400cc70 tx=0x7ff04400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:13.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.179+0000 7ff048ff9700 1 -- 172.21.15.23:0/2562562743 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04400aa40 con 0x7ff04c14f0c0 2023-06-22T08:16:13.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.179+0000 7ff048ff9700 1 -- 172.21.15.23:0/2562562743 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04400abc0 con 0x7ff04c14f0c0 2023-06-22T08:16:13.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.179+0000 7ff048ff9700 1 -- 172.21.15.23:0/2562562743 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0440070d0 con 0x7ff04c14f0c0 2023-06-22T08:16:13.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.179+0000 7ff0529ff700 1 -- 172.21.15.23:0/2562562743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff04c154840 con 0x7ff04c14f0c0 2023-06-22T08:16:13.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.179+0000 7ff0529ff700 1 -- 172.21.15.23:0/2562562743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04c154ae0 con 0x7ff04c14f0c0 2023-06-22T08:16:13.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.180+0000 7ff0529ff700 1 -- 172.21.15.23:0/2562562743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04c04ea20 con 0x7ff04c14f0c0 2023-06-22T08:16:13.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.181+0000 7ff048ff9700 1 -- 172.21.15.23:0/2562562743 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff044007230 con 0x7ff04c14f0c0 2023-06-22T08:16:13.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.182+0000 7ff048ff9700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff03806ace0 0x7ff03806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:13.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.182+0000 7ff048ff9700 1 -- 172.21.15.23:0/2562562743 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7ff044089450 con 0x7ff04c14f0c0 2023-06-22T08:16:13.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.182+0000 7ff04bfff700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff03806ace0 0x7ff03806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:13.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.183+0000 7ff04bfff700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff03806ace0 0x7ff03806d190 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00a1c0 tx=0x7ff03c01b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:13.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.186+0000 7ff048ff9700 1 -- 172.21.15.23:0/2562562743 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff044053ec0 con 0x7ff04c14f0c0 2023-06-22T08:16:13.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.455+0000 7ff0529ff700 1 -- 172.21.15.23:0/2562562743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff04c0c2fc0 con 0x7ff04c14f0c0 2023-06-22T08:16:13.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.456+0000 7ff048ff9700 1 -- 172.21.15.23:0/2562562743 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+6274 (secure 0 0 0) 0x7ff0440574e0 con 0x7ff04c14f0c0 2023-06-22T08:16:13.459 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:13.459 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:00.928887+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:13.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 -- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff03806ace0 msgr2=0x7ff03806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:13.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff03806ace0 0x7ff03806d190 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00a1c0 tx=0x7ff03c01b040 comp rx=0 tx=0).stop 2023-06-22T08:16:13.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 -- 172.21.15.23:0/2562562743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 msgr2=0x7ff04c1ee500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:13.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 0x7ff04c1ee500 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff04400cc70 tx=0x7ff04400cf80 comp rx=0 tx=0).stop 2023-06-22T08:16:13.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 -- 172.21.15.23:0/2562562743 shutdown_connections 2023-06-22T08:16:13.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff04c14f0c0 0x7ff04c1ee500 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:13.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff03806ace0 0x7ff03806d190 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:13.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff04c0c61e0 0x7ff04c1e04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:13.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 --2- 172.21.15.23:0/2562562743 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff04c0c6610 0x7ff04c13d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:13.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 -- 172.21.15.23:0/2562562743 >> 172.21.15.23:0/2562562743 conn(0x7ff04c0ca3f0 msgr2=0x7ff04c1e0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:13.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 -- 172.21.15.23:0/2562562743 shutdown_connections 2023-06-22T08:16:13.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:13.458+0000 7ff0529ff700 1 -- 172.21.15.23:0/2562562743 wait complete. 2023-06-22T08:16:13.466 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 130 2023-06-22T08:16:13.481 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:16:15.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2023-06-22T08:16:15.482 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:15.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.681+0000 7f4209dff700 1 Processor -- start 2023-06-22T08:16:15.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.682+0000 7f4209dff700 1 -- start start 2023-06-22T08:16:15.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.683+0000 7f4209dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f420414f850 0x7f42040c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:15.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.683+0000 7f4209dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f42040c69e0 0x7f42040c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:15.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.683+0000 7f4209dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42040c51a0 con 0x7f4204071070 2023-06-22T08:16:15.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.683+0000 7f4209dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42040c44e0 con 0x7f42040c69e0 2023-06-22T08:16:15.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.683+0000 7f4209dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42040c5440 con 0x7f420414f850 2023-06-22T08:16:15.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.683+0000 7f4202ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f42040c69e0 0x7f42040c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:15.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.683+0000 7f4202ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f42040c69e0 0x7f42040c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42056/0 (socket says 172.21.15.23:42056) 2023-06-22T08:16:15.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.683+0000 7f4202ffd700 1 -- 172.21.15.23:0/534697230 learned_addr learned my addr 172.21.15.23:0/534697230 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:15.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.683+0000 7f42037fe700 1 --2- 172.21.15.23:0/534697230 >> v2:172.21.15.67:3300/0 conn(0x7f420414f850 0x7f42040c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:15.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.684+0000 7f4202ffd700 1 -- 172.21.15.23:0/534697230 >> v2:172.21.15.67:3300/0 conn(0x7f420414f850 msgr2=0x7f42040c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:15.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.684+0000 7f4202ffd700 1 --2- 172.21.15.23:0/534697230 >> v2:172.21.15.67:3300/0 conn(0x7f420414f850 0x7f42040c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.684+0000 7f4202ffd700 1 -- 172.21.15.23:0/534697230 >> v1:172.21.15.67:6789/0 conn(0x7f4204071070 legacy=0x7f4204155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:15.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.684+0000 7f4202ffd700 1 -- 172.21.15.23:0/534697230 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42040c5720 con 0x7f42040c69e0 2023-06-22T08:16:15.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.684+0000 7f4202ffd700 1 --2- 172.21.15.23:0/534697230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42040c69e0 0x7f42040c35c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f420406f240 tx=0x7f41f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ad6bed585f3635 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:15.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.684+0000 7f42027fc700 1 -- 172.21.15.23:0/534697230 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8011830 con 0x7f42040c69e0 2023-06-22T08:16:15.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.684+0000 7f42027fc700 1 -- 172.21.15.23:0/534697230 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f8011990 con 0x7f42040c69e0 2023-06-22T08:16:15.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.684+0000 7f42027fc700 1 -- 172.21.15.23:0/534697230 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8011d20 con 0x7f42040c69e0 2023-06-22T08:16:15.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.685+0000 7f4209dff700 1 -- 172.21.15.23:0/534697230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42040c69e0 msgr2=0x7f42040c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:15.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.685+0000 7f4209dff700 1 --2- 172.21.15.23:0/534697230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42040c69e0 0x7f42040c35c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f420406f240 tx=0x7f41f8009bd0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.685+0000 7f4209dff700 1 -- 172.21.15.23:0/534697230 shutdown_connections 2023-06-22T08:16:15.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.685+0000 7f4209dff700 1 --2- 172.21.15.23:0/534697230 >> v2:172.21.15.67:3300/0 conn(0x7f420414f850 0x7f42040c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.685+0000 7f4209dff700 1 --2- 172.21.15.23:0/534697230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42040c69e0 0x7f42040c35c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.685+0000 7f4209dff700 1 -- 172.21.15.23:0/534697230 >> 172.21.15.23:0/534697230 conn(0x7f42040ca3f0 msgr2=0x7f42040ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:15.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.685+0000 7f4209dff700 1 -- 172.21.15.23:0/534697230 shutdown_connections 2023-06-22T08:16:15.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.685+0000 7f4209dff700 1 -- 172.21.15.23:0/534697230 wait complete. 2023-06-22T08:16:15.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.687+0000 7f4209dff700 1 Processor -- start 2023-06-22T08:16:15.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.688+0000 7f4209dff700 1 -- start start 2023-06-22T08:16:15.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f4209dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4204071070 0x7f42041efd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:15.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f4209dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040c69e0 0x7f42041ef2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:15.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f4209dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f420414f850 0x7f420414ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:15.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f4209dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42041564f0 con 0x7f420414f850 2023-06-22T08:16:15.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f4209dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42041507d0 con 0x7f42040c69e0 2023-06-22T08:16:15.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f4209dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42041ef7e0 con 0x7f4204071070 2023-06-22T08:16:15.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f4203fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f420414f850 0x7f420414ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:15.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f42037fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4204071070 0x7f42041efd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:15.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f4203fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f420414f850 0x7f420414ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42058/0 (socket says 172.21.15.23:42058) 2023-06-22T08:16:15.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f4203fff700 1 -- 172.21.15.23:0/4188653732 learned_addr learned my addr 172.21.15.23:0/4188653732 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:15.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f4202ffd700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040c69e0 0x7f42041ef2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:15.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f42037fe700 1 -- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040c69e0 msgr2=0x7f42041ef2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:15.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f42037fe700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040c69e0 0x7f42041ef2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f42037fe700 1 -- 172.21.15.23:0/4188653732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f420414f850 msgr2=0x7f420414ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:15.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.689+0000 7f42037fe700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f420414f850 0x7f420414ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.690+0000 7f42037fe700 1 -- 172.21.15.23:0/4188653732 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f8009550 con 0x7f4204071070 2023-06-22T08:16:15.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.690+0000 7f4203fff700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f420414f850 0x7f420414ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:16:15.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.690+0000 7f42037fe700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4204071070 0x7f42041efd10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f41f400e8a0 tx=0x7f41f400ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:15.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.690+0000 7f41ebfff700 1 -- 172.21.15.23:0/4188653732 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4013680 con 0x7f4204071070 2023-06-22T08:16:15.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.690+0000 7f41ebfff700 1 -- 172.21.15.23:0/4188653732 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f40137e0 con 0x7f4204071070 2023-06-22T08:16:15.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.690+0000 7f4209dff700 1 -- 172.21.15.23:0/4188653732 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42041520c0 con 0x7f4204071070 2023-06-22T08:16:15.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.690+0000 7f41ebfff700 1 -- 172.21.15.23:0/4188653732 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4013b70 con 0x7f4204071070 2023-06-22T08:16:15.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.690+0000 7f4209dff700 1 -- 172.21.15.23:0/4188653732 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4204150110 con 0x7f4204071070 2023-06-22T08:16:15.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.691+0000 7f4209dff700 1 -- 172.21.15.23:0/4188653732 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42040c2950 con 0x7f4204071070 2023-06-22T08:16:15.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.692+0000 7f41ebfff700 1 -- 172.21.15.23:0/4188653732 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f41f4013cd0 con 0x7f4204071070 2023-06-22T08:16:15.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.693+0000 7f41ebfff700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f41f006ace0 0x7f41f006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:15.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.693+0000 7f41ebfff700 1 -- 172.21.15.23:0/4188653732 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7f41f40898d0 con 0x7f4204071070 2023-06-22T08:16:15.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.693+0000 7f4202ffd700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f41f006ace0 0x7f41f006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:15.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.694+0000 7f4202ffd700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f41f006ace0 0x7f41f006d190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009c10 tx=0x7f41f8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:15.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.697+0000 7f41ebfff700 1 -- 172.21.15.23:0/4188653732 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f41f4057d40 con 0x7f4204071070 2023-06-22T08:16:15.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.949+0000 7f4209dff700 1 -- 172.21.15.23:0/4188653732 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f420404ea20 con 0x7f4204071070 2023-06-22T08:16:15.953 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.950+0000 7f41ebfff700 1 -- 172.21.15.23:0/4188653732 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f41f4017090 con 0x7f4204071070 2023-06-22T08:16:15.954 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:15.954 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:15.517753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:15.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.952+0000 7f4209dff700 1 -- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f41f006ace0 msgr2=0x7f41f006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:15.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.952+0000 7f4209dff700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f41f006ace0 0x7f41f006d190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009c10 tx=0x7f41f8008030 comp rx=0 tx=0).stop 2023-06-22T08:16:15.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.952+0000 7f4209dff700 1 -- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4204071070 msgr2=0x7f42041efd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:15.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.952+0000 7f4209dff700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4204071070 0x7f42041efd10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f41f400e8a0 tx=0x7f41f400ebb0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.953+0000 7f4209dff700 1 -- 172.21.15.23:0/4188653732 shutdown_connections 2023-06-22T08:16:15.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.953+0000 7f4209dff700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f420414f850 0x7f420414ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.953+0000 7f4209dff700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4204071070 0x7f42041efd10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.953+0000 7f4209dff700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f41f006ace0 0x7f41f006d190 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.953+0000 7f4209dff700 1 --2- 172.21.15.23:0/4188653732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040c69e0 0x7f42041ef2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:15.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.953+0000 7f4209dff700 1 -- 172.21.15.23:0/4188653732 >> 172.21.15.23:0/4188653732 conn(0x7f42040ca3f0 msgr2=0x7f420414c490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:15.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.953+0000 7f4209dff700 1 -- 172.21.15.23:0/4188653732 shutdown_connections 2023-06-22T08:16:15.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:15.953+0000 7f4209dff700 1 -- 172.21.15.23:0/4188653732 wait complete. 2023-06-22T08:16:15.958 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 131 2023-06-22T08:16:15.974 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:16:17.975 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:18.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.176+0000 7ff853fff700 1 Processor -- start 2023-06-22T08:16:18.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.176+0000 7ff853fff700 1 -- start start 2023-06-22T08:16:18.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.177+0000 7ff853fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff84c0cb260 0x7ff84c0c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:18.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.177+0000 7ff853fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff84c0c9200 0x7ff84c0cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:18.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.177+0000 7ff853fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84c0cbd60 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.177+0000 7ff853fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84c0cbb70 con 0x7ff84c0cb260 2023-06-22T08:16:18.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.177+0000 7ff853fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84c158980 con 0x7ff84c0c9200 2023-06-22T08:16:18.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.178+0000 7ff851d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff84c0cb260 0x7ff84c0c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:18.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.178+0000 7ff85159a700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff84c0c9200 0x7ff84c0cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:18.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.178+0000 7ff851d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff84c0cb260 0x7ff84c0c4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36878/0 (socket says 172.21.15.23:36878) 2023-06-22T08:16:18.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.178+0000 7ff851d9b700 1 -- 172.21.15.23:0/3845820980 learned_addr learned my addr 172.21.15.23:0/3845820980 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:18.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.178+0000 7ff850d99700 1 -- 172.21.15.23:0/3845820980 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364304887 0 0) 0x7ff84c0cbd60 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.178+0000 7ff851d9b700 1 -- 172.21.15.23:0/3845820980 >> v2:172.21.15.67:3300/0 conn(0x7ff84c0c9200 msgr2=0x7ff84c0cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:18.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.178+0000 7ff851d9b700 1 --2- 172.21.15.23:0/3845820980 >> v2:172.21.15.67:3300/0 conn(0x7ff84c0c9200 0x7ff84c0cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:18.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.178+0000 7ff851d9b700 1 -- 172.21.15.23:0/3845820980 >> v1:172.21.15.67:6789/0 conn(0x7ff84c0b0fc0 legacy=0x7ff84c15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:18.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.178+0000 7ff851d9b700 1 -- 172.21.15.23:0/3845820980 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c159550 con 0x7ff84c0cb260 2023-06-22T08:16:18.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.178+0000 7ff851d9b700 1 --2- 172.21.15.23:0/3845820980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff84c0cb260 0x7ff84c0c4a50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff84c08c760 tx=0x7ff848009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d0c6c74611e11f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:18.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff850d99700 1 -- 172.21.15.23:0/3845820980 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848011830 con 0x7ff84c0cb260 2023-06-22T08:16:18.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff850d99700 1 -- 172.21.15.23:0/3845820980 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff848011990 con 0x7ff84c0cb260 2023-06-22T08:16:18.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff850d99700 1 -- 172.21.15.23:0/3845820980 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848011d20 con 0x7ff84c0cb260 2023-06-22T08:16:18.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff853fff700 1 -- 172.21.15.23:0/3845820980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff84c0cb260 msgr2=0x7ff84c0c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:18.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff853fff700 1 --2- 172.21.15.23:0/3845820980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff84c0cb260 0x7ff84c0c4a50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff84c08c760 tx=0x7ff848009bd0 comp rx=0 tx=0).stop 2023-06-22T08:16:18.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff853fff700 1 -- 172.21.15.23:0/3845820980 shutdown_connections 2023-06-22T08:16:18.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff853fff700 1 --2- 172.21.15.23:0/3845820980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff84c0cb260 0x7ff84c0c4a50 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:18.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff853fff700 1 --2- 172.21.15.23:0/3845820980 >> v2:172.21.15.67:3300/0 conn(0x7ff84c0c9200 0x7ff84c0cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:18.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff853fff700 1 -- 172.21.15.23:0/3845820980 >> 172.21.15.23:0/3845820980 conn(0x7ff84c065b90 msgr2=0x7ff84c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:18.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff853fff700 1 -- 172.21.15.23:0/3845820980 shutdown_connections 2023-06-22T08:16:18.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.179+0000 7ff853fff700 1 -- 172.21.15.23:0/3845820980 wait complete. 2023-06-22T08:16:18.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.181+0000 7ff853fff700 1 Processor -- start 2023-06-22T08:16:18.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.181+0000 7ff853fff700 1 -- start start 2023-06-22T08:16:18.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff853fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff84c0b0fc0 0x7ff84c151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:18.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff853fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff84c0c9200 0x7ff84c14e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:18.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff853fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff84c0cb260 0x7ff84c14ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:18.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff853fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c146e60 con 0x7ff84c0cb260 2023-06-22T08:16:18.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff853fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c158660 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff853fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c157b00 con 0x7ff84c0c9200 2023-06-22T08:16:18.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff851d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff84c0b0fc0 0x7ff84c151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:18.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff851d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff84c0b0fc0 0x7ff84c151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:54496/0 (socket says 172.21.15.23:54496) 2023-06-22T08:16:18.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff851d9b700 1 -- 172.21.15.23:0/3452366186 learned_addr learned my addr 172.21.15.23:0/3452366186 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:18.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff85159a700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff84c0c9200 0x7ff84c14e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:18.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.182+0000 7ff85259c700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff84c0cb260 0x7ff84c14ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:18.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.183+0000 7ff851d9b700 1 -- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff84c0c9200 msgr2=0x7ff84c14e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:18.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.183+0000 7ff851d9b700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff84c0c9200 0x7ff84c14e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:18.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.183+0000 7ff851d9b700 1 -- 172.21.15.23:0/3452366186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff84c0cb260 msgr2=0x7ff84c14ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:18.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.183+0000 7ff851d9b700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff84c0cb260 0x7ff84c14ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:18.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.183+0000 7ff851d9b700 1 -- 172.21.15.23:0/3452366186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff848009550 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.183+0000 7ff851d9b700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff84c0b0fc0 0x7ff84c151fa0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff848014040 tx=0x7ff848004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.183+0000 7ff8427fc700 1 -- 172.21.15.23:0/3452366186 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848004590 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.183+0000 7ff8427fc700 1 -- 172.21.15.23:0/3452366186 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84801e070 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.183+0000 7ff8427fc700 1 -- 172.21.15.23:0/3452366186 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8480038c0 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.184+0000 7ff853fff700 1 -- 172.21.15.23:0/3452366186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c157dc0 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.184+0000 7ff853fff700 1 -- 172.21.15.23:0/3452366186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84c158380 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.185+0000 7ff853fff700 1 -- 172.21.15.23:0/3452366186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c04ea20 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.186+0000 7ff8427fc700 1 -- 172.21.15.23:0/3452366186 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff848003a20 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.187+0000 7ff8427fc700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff83006ace0 0x7ff83006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:18.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.187+0000 7ff8427fc700 1 -- 172.21.15.23:0/3452366186 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7ff84808a8d0 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.187+0000 7ff85159a700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff83006ace0 0x7ff83006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:18.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.187+0000 7ff85159a700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff83006ace0 0x7ff83006d190 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff84c152840 tx=0x7ff83c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:18.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.192+0000 7ff8427fc700 1 -- 172.21.15.23:0/3452366186 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff8480553d0 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.457+0000 7ff853fff700 1 -- 172.21.15.23:0/3452366186 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff84c0c3c10 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.458+0000 7ff8427fc700 1 -- 172.21.15.23:0/3452366186 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6272 (secure 0 0 0) 0x7ff848020090 con 0x7ff84c0b0fc0 2023-06-22T08:16:18.462 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:18.462 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:15.517753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 -- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff83006ace0 msgr2=0x7ff83006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff83006ace0 0x7ff83006d190 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff84c152840 tx=0x7ff83c009210 comp rx=0 tx=0).stop 2023-06-22T08:16:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 -- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff84c0b0fc0 msgr2=0x7ff84c151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff84c0b0fc0 0x7ff84c151fa0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff848014040 tx=0x7ff848004750 comp rx=0 tx=0).stop 2023-06-22T08:16:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 -- 172.21.15.23:0/3452366186 shutdown_connections 2023-06-22T08:16:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff84c0cb260 0x7ff84c14ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff83006ace0 0x7ff83006d190 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff84c0b0fc0 0x7ff84c151fa0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 --2- 172.21.15.23:0/3452366186 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff84c0c9200 0x7ff84c14e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 -- 172.21.15.23:0/3452366186 >> 172.21.15.23:0/3452366186 conn(0x7ff84c065b90 msgr2=0x7ff84c0b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 -- 172.21.15.23:0/3452366186 shutdown_connections 2023-06-22T08:16:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:18.461+0000 7ff853fff700 1 -- 172.21.15.23:0/3452366186 wait complete. 2023-06-22T08:16:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 131 2023-06-22T08:16:18.482 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:16:20.483 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:20.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.687+0000 7f7001bff700 1 Processor -- start 2023-06-22T08:16:20.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.687+0000 7f7001bff700 1 -- start start 2023-06-22T08:16:20.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.688+0000 7f7001bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ffc152d70 0x7f6ffc0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:20.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f7001bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc159790 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f7001bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc15bea0 con 0x7f6ffc159330 2023-06-22T08:16:20.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f7001bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc0c95e0 con 0x7f6ffc152d70 2023-06-22T08:16:20.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f6ffb7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ffc152d70 0x7f6ffc0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:20.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f6ffaffd700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6ffc159330 0x7f6ffc0cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:33178/0 (socket says 172.21.15.23:33178) 2023-06-22T08:16:20.694 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f6ffaffd700 1 -- 172.21.15.23:0/3277461786 learned_addr learned my addr 172.21.15.23:0/3277461786 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:20.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f6ffbfff700 1 --1- 172.21.15.23:0/3277461786 >> v1:172.21.15.23:6789/0 conn(0x7f6ffc0c45f0 0x7f6ffc0c4a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:47096/0 (socket says 172.21.15.23:47096) 2023-06-22T08:16:20.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f6ffb7fe700 1 -- 172.21.15.23:0/3277461786 >> v1:172.21.15.67:6789/0 conn(0x7f6ffc159330 legacy=0x7f6ffc0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:20.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f6ffb7fe700 1 -- 172.21.15.23:0/3277461786 >> v1:172.21.15.23:6789/0 conn(0x7f6ffc0c45f0 legacy=0x7f6ffc0c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:20.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f6ffb7fe700 1 -- 172.21.15.23:0/3277461786 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc15c160 con 0x7f6ffc152d70 2023-06-22T08:16:20.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.689+0000 7f6ffa7fc700 1 -- 172.21.15.23:0/3277461786 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257195115 0 0) 0x7f6ffc159790 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.690+0000 7f6ffb7fe700 1 --2- 172.21.15.23:0/3277461786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ffc152d70 0x7f6ffc0c97d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009840 tx=0x7f6fec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4305d9c15e200022 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:20.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.690+0000 7f6ffa7fc700 1 -- 172.21.15.23:0/3277461786 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec011830 con 0x7f6ffc152d70 2023-06-22T08:16:20.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.690+0000 7f6ffa7fc700 1 -- 172.21.15.23:0/3277461786 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec011990 con 0x7f6ffc152d70 2023-06-22T08:16:20.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.690+0000 7f6ffa7fc700 1 -- 172.21.15.23:0/3277461786 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec011d20 con 0x7f6ffc152d70 2023-06-22T08:16:20.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.690+0000 7f7001bff700 1 -- 172.21.15.23:0/3277461786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ffc152d70 msgr2=0x7f6ffc0c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:20.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.690+0000 7f7001bff700 1 --2- 172.21.15.23:0/3277461786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ffc152d70 0x7f6ffc0c97d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009840 tx=0x7f6fec009b50 comp rx=0 tx=0).stop 2023-06-22T08:16:20.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.690+0000 7f7001bff700 1 -- 172.21.15.23:0/3277461786 shutdown_connections 2023-06-22T08:16:20.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.690+0000 7f7001bff700 1 --2- 172.21.15.23:0/3277461786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ffc152d70 0x7f6ffc0c97d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:20.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.690+0000 7f7001bff700 1 -- 172.21.15.23:0/3277461786 >> 172.21.15.23:0/3277461786 conn(0x7f6ffc06ae30 msgr2=0x7f6ffc06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:20.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.690+0000 7f7001bff700 1 -- 172.21.15.23:0/3277461786 shutdown_connections 2023-06-22T08:16:20.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.691+0000 7f7001bff700 1 -- 172.21.15.23:0/3277461786 wait complete. 2023-06-22T08:16:20.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.692+0000 7f7001bff700 1 Processor -- start 2023-06-22T08:16:20.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.692+0000 7f7001bff700 1 -- start start 2023-06-22T08:16:20.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.693+0000 7f7001bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ffc0c45f0 0x7f6ffc0cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:20.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.693+0000 7f7001bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6ffc152d70 0x7f6ffc14abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:20.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.693+0000 7f7001bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ffc159330 0x7f6ffc0b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:20.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.693+0000 7f7001bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc14bcd0 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.693+0000 7f7001bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc151de0 con 0x7f6ffc159330 2023-06-22T08:16:20.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.693+0000 7f7001bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc152000 con 0x7f6ffc152d70 2023-06-22T08:16:20.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.693+0000 7f6ffb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ffc0c45f0 0x7f6ffc0cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:20.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.693+0000 7f6ffb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ffc0c45f0 0x7f6ffc0cef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36894/0 (socket says 172.21.15.23:36894) 2023-06-22T08:16:20.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.693+0000 7f6ffb7fe700 1 -- 172.21.15.23:0/3179951918 learned_addr learned my addr 172.21.15.23:0/3179951918 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:20.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.694+0000 7f6ffaffd700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6ffc152d70 0x7f6ffc14abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:20.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.694+0000 7f6ffbfff700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ffc159330 0x7f6ffc0b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:20.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.694+0000 7f6ffb7fe700 1 -- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6ffc152d70 msgr2=0x7f6ffc14abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:20.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.694+0000 7f6ffb7fe700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6ffc152d70 0x7f6ffc14abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:20.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.694+0000 7f6ffb7fe700 1 -- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ffc159330 msgr2=0x7f6ffc0b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:20.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.694+0000 7f6ffb7fe700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ffc159330 0x7f6ffc0b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:20.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.694+0000 7f6ffb7fe700 1 -- 172.21.15.23:0/3179951918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec0094a0 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.694+0000 7f6ffb7fe700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ffc0c45f0 0x7f6ffc0cef10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc08bcd0 tx=0x7f6fec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:20.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.694+0000 7f6fe3fff700 1 -- 172.21.15.23:0/3179951918 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec01a760 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.695+0000 7f7001bff700 1 -- 172.21.15.23:0/3179951918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc1522e0 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.695+0000 7f6fe3fff700 1 -- 172.21.15.23:0/3179951918 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec01a8c0 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.695+0000 7f6fe3fff700 1 -- 172.21.15.23:0/3179951918 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec01ac30 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.695+0000 7f7001bff700 1 -- 172.21.15.23:0/3179951918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc1ef4b0 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.696+0000 7f7001bff700 1 -- 172.21.15.23:0/3179951918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc1524f0 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.697+0000 7f6fe3fff700 1 -- 172.21.15.23:0/3179951918 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6fec022070 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.697+0000 7f6fe3fff700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fe806ace0 0x7f6fe806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:20.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.698+0000 7f6fe3fff700 1 -- 172.21.15.23:0/3179951918 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7f6fec08abc0 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.698+0000 7f6ffaffd700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fe806ace0 0x7f6fe806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:20.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.698+0000 7f6ffaffd700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fe806ace0 0x7f6fe806d190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc14fb40 tx=0x7f6ff0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:20.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.702+0000 7f6fe3fff700 1 -- 172.21.15.23:0/3179951918 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6fec055810 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.960+0000 7f7001bff700 1 -- 172.21.15.23:0/3179951918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ffc0742e0 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.960+0000 7f6fe3fff700 1 -- 172.21.15.23:0/3179951918 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f6fec058e30 con 0x7f6ffc0c45f0 2023-06-22T08:16:20.964 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:20.964 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:15.517753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:20.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 -- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fe806ace0 msgr2=0x7f6fe806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:20.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fe806ace0 0x7f6fe806d190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc14fb40 tx=0x7f6ff0009040 comp rx=0 tx=0).stop 2023-06-22T08:16:20.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 -- 172.21.15.23:0/3179951918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ffc0c45f0 msgr2=0x7f6ffc0cef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:20.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ffc0c45f0 0x7f6ffc0cef10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc08bcd0 tx=0x7f6fec004750 comp rx=0 tx=0).stop 2023-06-22T08:16:20.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 -- 172.21.15.23:0/3179951918 shutdown_connections 2023-06-22T08:16:20.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6fe806ace0 0x7f6fe806d190 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:20.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ffc159330 0x7f6ffc0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:20.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ffc0c45f0 0x7f6ffc0cef10 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:20.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 --2- 172.21.15.23:0/3179951918 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6ffc152d70 0x7f6ffc14abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:20.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 -- 172.21.15.23:0/3179951918 >> 172.21.15.23:0/3179951918 conn(0x7f6ffc06ae30 msgr2=0x7f6ffc0cdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:20.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 -- 172.21.15.23:0/3179951918 shutdown_connections 2023-06-22T08:16:20.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:20.963+0000 7f7001bff700 1 -- 172.21.15.23:0/3179951918 wait complete. 2023-06-22T08:16:20.970 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 131 2023-06-22T08:16:20.984 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:16:22.985 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:23.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.192+0000 7fb76c1ff700 1 Processor -- start 2023-06-22T08:16:23.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.192+0000 7fb76c1ff700 1 -- start start 2023-06-22T08:16:23.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.193+0000 7fb76c1ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb764152ce0 0x7fb7640cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:23.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.193+0000 7fb76c1ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb7640c40b0 0x7fb7640c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:23.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.193+0000 7fb76c1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76415c370 con 0x7fb7640c9fd0 2023-06-22T08:16:23.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.193+0000 7fb76c1ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7641530b0 con 0x7fb7640c40b0 2023-06-22T08:16:23.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.193+0000 7fb76c1ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7640cf6f0 con 0x7fb764152ce0 2023-06-22T08:16:23.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.194+0000 7fb76a79c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb7640c40b0 0x7fb7640c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:23.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.194+0000 7fb769f9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb764152ce0 0x7fb7640cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:23.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.194+0000 7fb76a79c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb7640c40b0 0x7fb7640c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36898/0 (socket says 172.21.15.23:36898) 2023-06-22T08:16:23.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.194+0000 7fb76a79c700 1 -- 172.21.15.23:0/2422815323 learned_addr learned my addr 172.21.15.23:0/2422815323 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:23.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.194+0000 7fb76a79c700 1 -- 172.21.15.23:0/2422815323 >> v2:172.21.15.67:3300/0 conn(0x7fb764152ce0 msgr2=0x7fb7640cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:23.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.194+0000 7fb76a79c700 1 --2- 172.21.15.23:0/2422815323 >> v2:172.21.15.67:3300/0 conn(0x7fb764152ce0 0x7fb7640cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:23.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.194+0000 7fb76a79c700 1 -- 172.21.15.23:0/2422815323 >> v1:172.21.15.67:6789/0 conn(0x7fb7640c9fd0 legacy=0x7fb76415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:23.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.194+0000 7fb76a79c700 1 -- 172.21.15.23:0/2422815323 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76415bda0 con 0x7fb7640c40b0 2023-06-22T08:16:23.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb76a79c700 1 --2- 172.21.15.23:0/2422815323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 0x7fb7640c4530 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb76408ba10 tx=0x7fb75c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19ca2e87ea6d415 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:23.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb768f99700 1 -- 172.21.15.23:0/2422815323 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00e070 con 0x7fb7640c40b0 2023-06-22T08:16:23.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb768f99700 1 -- 172.21.15.23:0/2422815323 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c00b910 con 0x7fb7640c40b0 2023-06-22T08:16:23.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb768f99700 1 -- 172.21.15.23:0/2422815323 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00bcc0 con 0x7fb7640c40b0 2023-06-22T08:16:23.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2422815323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 msgr2=0x7fb7640c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb76c1ff700 1 --2- 172.21.15.23:0/2422815323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 0x7fb7640c4530 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb76408ba10 tx=0x7fb75c00b330 comp rx=0 tx=0).stop 2023-06-22T08:16:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2422815323 shutdown_connections 2023-06-22T08:16:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb76c1ff700 1 --2- 172.21.15.23:0/2422815323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 0x7fb7640c4530 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb76c1ff700 1 --2- 172.21.15.23:0/2422815323 >> v2:172.21.15.67:3300/0 conn(0x7fb764152ce0 0x7fb7640cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2422815323 >> 172.21.15.23:0/2422815323 conn(0x7fb76406ae30 msgr2=0x7fb76406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.195+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2422815323 shutdown_connections 2023-06-22T08:16:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.196+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2422815323 wait complete. 2023-06-22T08:16:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.197+0000 7fb76c1ff700 1 Processor -- start 2023-06-22T08:16:23.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.197+0000 7fb76c1ff700 1 -- start start 2023-06-22T08:16:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.198+0000 7fb76c1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 0x7fb764154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.198+0000 7fb76c1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb7640c9fd0 0x7fb7641f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.198+0000 7fb76c1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb764152ce0 0x7fb7641f09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:23.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.198+0000 7fb76c1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640cf5b0 con 0x7fb7640c40b0 2023-06-22T08:16:23.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.198+0000 7fb76c1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb764153c00 con 0x7fb764152ce0 2023-06-22T08:16:23.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.198+0000 7fb76c1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7640cf2d0 con 0x7fb7640c9fd0 2023-06-22T08:16:23.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb769f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 0x7fb764154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:23.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb769f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 0x7fb764154650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36912/0 (socket says 172.21.15.23:36912) 2023-06-22T08:16:23.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb769f9b700 1 -- 172.21.15.23:0/2525449342 learned_addr learned my addr 172.21.15.23:0/2525449342 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:23.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb76979a700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb7640c9fd0 0x7fb7641f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:23.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb76a79c700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb764152ce0 0x7fb7641f09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:23.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb769f9b700 1 -- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb7640c9fd0 msgr2=0x7fb7641f16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:23.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb769f9b700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb7640c9fd0 0x7fb7641f16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:23.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb769f9b700 1 -- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb764152ce0 msgr2=0x7fb7641f09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:23.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb769f9b700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb764152ce0 0x7fb7641f09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:23.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb769f9b700 1 -- 172.21.15.23:0/2525449342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c009b30 con 0x7fb7640c40b0 2023-06-22T08:16:23.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.199+0000 7fb769f9b700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 0x7fb764154650 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb76000e8a0 tx=0x7fb76000ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:23.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.200+0000 7fb75a7fc700 1 -- 172.21.15.23:0/2525449342 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760009590 con 0x7fb7640c40b0 2023-06-22T08:16:23.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.200+0000 7fb75a7fc700 1 -- 172.21.15.23:0/2525449342 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb760013680 con 0x7fb7640c40b0 2023-06-22T08:16:23.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.200+0000 7fb75a7fc700 1 -- 172.21.15.23:0/2525449342 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7600139f0 con 0x7fb7640c40b0 2023-06-22T08:16:23.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.200+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2525449342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7641f20f0 con 0x7fb7640c40b0 2023-06-22T08:16:23.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.200+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2525449342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb764154130 con 0x7fb7640c40b0 2023-06-22T08:16:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.201+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2525449342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76404ea20 con 0x7fb7640c40b0 2023-06-22T08:16:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.202+0000 7fb75a7fc700 1 -- 172.21.15.23:0/2525449342 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb760013b50 con 0x7fb7640c40b0 2023-06-22T08:16:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.203+0000 7fb75a7fc700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb74806ace0 0x7fb74806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.203+0000 7fb75a7fc700 1 -- 172.21.15.23:0/2525449342 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 7501+0+0 (secure 0 0 0) 0x7fb7600898e0 con 0x7fb7640c40b0 2023-06-22T08:16:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.203+0000 7fb76979a700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb74806ace0 0x7fb74806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.204+0000 7fb76979a700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb74806ace0 0x7fb74806d190 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb764159710 tx=0x7fb754009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.208+0000 7fb75a7fc700 1 -- 172.21.15.23:0/2525449342 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb7600543e0 con 0x7fb7640c40b0 2023-06-22T08:16:23.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.464+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2525449342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb76406b700 con 0x7fb7640c40b0 2023-06-22T08:16:23.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.465+0000 7fb75a7fc700 1 -- 172.21.15.23:0/2525449342 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+6272 (secure 0 0 0) 0x7fb760057a00 con 0x7fb7640c40b0 2023-06-22T08:16:23.468 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:23.469 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:15.517753+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":5551,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5551":{"gid":5551,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":26,"addr":"172.21.15.23:6834/1707022233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":1707022233},{"type":"v1","addr":"172.21.15.23:6834","nonce":1707022233}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:23.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb74806ace0 msgr2=0x7fb74806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:23.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb74806ace0 0x7fb74806d190 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb764159710 tx=0x7fb754009040 comp rx=0 tx=0).stop 2023-06-22T08:16:23.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2525449342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 msgr2=0x7fb764154650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:23.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 0x7fb764154650 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb76000e8a0 tx=0x7fb76000ebb0 comp rx=0 tx=0).stop 2023-06-22T08:16:23.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2525449342 shutdown_connections 2023-06-22T08:16:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb74806ace0 0x7fb74806d190 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb764152ce0 0x7fb7641f09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7640c40b0 0x7fb764154650 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 --2- 172.21.15.23:0/2525449342 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb7640c9fd0 0x7fb7641f16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2525449342 >> 172.21.15.23:0/2525449342 conn(0x7fb76406ae30 msgr2=0x7fb76414f040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.467+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2525449342 shutdown_connections 2023-06-22T08:16:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:23.468+0000 7fb76c1ff700 1 -- 172.21.15.23:0/2525449342 wait complete. 2023-06-22T08:16:23.473 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 131 2023-06-22T08:16:23.489 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-06-22T08:16:25.490 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:25.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.699+0000 7f072e1ff700 1 Processor -- start 2023-06-22T08:16:25.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.699+0000 7f072e1ff700 1 -- start start 2023-06-22T08:16:25.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.700+0000 7f072e1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0728152ce0 0x7f07280cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:25.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.700+0000 7f072e1ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07280c40b0 0x7f07280c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:25.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.700+0000 7f072e1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072815c370 con 0x7f07280c9fd0 2023-06-22T08:16:25.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.700+0000 7f072e1ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07281530b0 con 0x7f07280c40b0 2023-06-22T08:16:25.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.700+0000 7f072e1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07280cf6f0 con 0x7f0728152ce0 2023-06-22T08:16:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.700+0000 7f07277fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0728152ce0 0x7f07280cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.701+0000 7f0727fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07280c40b0 0x7f07280c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.701+0000 7f07277fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0728152ce0 0x7f07280cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:47834/0 (socket says 172.21.15.23:47834) 2023-06-22T08:16:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.701+0000 7f07277fe700 1 -- 172.21.15.23:0/1605051575 learned_addr learned my addr 172.21.15.23:0/1605051575 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.701+0000 7f0726ffd700 1 --1- 172.21.15.23:0/1605051575 >> v1:172.21.15.67:6789/0 conn(0x7f07280c9fd0 0x7f072815c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:33202/0 (socket says 172.21.15.23:33202) 2023-06-22T08:16:25.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.701+0000 7f07277fe700 1 -- 172.21.15.23:0/1605051575 >> v2:172.21.15.67:3300/0 conn(0x7f07280c40b0 msgr2=0x7f07280c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.701+0000 7f07277fe700 1 --2- 172.21.15.23:0/1605051575 >> v2:172.21.15.67:3300/0 conn(0x7f07280c40b0 0x7f07280c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.701+0000 7f07277fe700 1 -- 172.21.15.23:0/1605051575 >> v1:172.21.15.67:6789/0 conn(0x7f07280c9fd0 legacy=0x7f072815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.701+0000 7f07277fe700 1 -- 172.21.15.23:0/1605051575 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072815bda0 con 0x7f0728152ce0 2023-06-22T08:16:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.701+0000 7f07277fe700 1 --2- 172.21.15.23:0/1605051575 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0728152ce0 0x7f07280cf1b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f07180098f0 tx=0x7f0718009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d399e665eb663506 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f07267fc700 1 -- 172.21.15.23:0/1605051575 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718011830 con 0x7f0728152ce0 2023-06-22T08:16:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f07267fc700 1 -- 172.21.15.23:0/1605051575 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0718011990 con 0x7f0728152ce0 2023-06-22T08:16:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f07267fc700 1 -- 172.21.15.23:0/1605051575 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718011d20 con 0x7f0728152ce0 2023-06-22T08:16:25.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f072e1ff700 1 -- 172.21.15.23:0/1605051575 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0728152ce0 msgr2=0x7f07280cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f072e1ff700 1 --2- 172.21.15.23:0/1605051575 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0728152ce0 0x7f07280cf1b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f07180098f0 tx=0x7f0718009c00 comp rx=0 tx=0).stop 2023-06-22T08:16:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f072e1ff700 1 -- 172.21.15.23:0/1605051575 shutdown_connections 2023-06-22T08:16:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f072e1ff700 1 --2- 172.21.15.23:0/1605051575 >> v2:172.21.15.67:3300/0 conn(0x7f07280c40b0 0x7f07280c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f072e1ff700 1 --2- 172.21.15.23:0/1605051575 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0728152ce0 0x7f07280cf1b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f072e1ff700 1 -- 172.21.15.23:0/1605051575 >> 172.21.15.23:0/1605051575 conn(0x7f072806ae30 msgr2=0x7f072806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f072e1ff700 1 -- 172.21.15.23:0/1605051575 shutdown_connections 2023-06-22T08:16:25.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.702+0000 7f072e1ff700 1 -- 172.21.15.23:0/1605051575 wait complete. 2023-06-22T08:16:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.705+0000 7f072e1ff700 1 Processor -- start 2023-06-22T08:16:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.705+0000 7f072e1ff700 1 -- start start 2023-06-22T08:16:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.706+0000 7f072e1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07280c40b0 0x7f07281e3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.706+0000 7f072e1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07280c9fd0 0x7f07280b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.706+0000 7f072e1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0728152ce0 0x7f07281559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.706+0000 7f072e1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07281e3ce0 con 0x7f07280c40b0 2023-06-22T08:16:25.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.706+0000 7f072e1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07280b34e0 con 0x7f0728152ce0 2023-06-22T08:16:25.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.706+0000 7f072e1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07281e27f0 con 0x7f07280c9fd0 2023-06-22T08:16:25.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.706+0000 7f07277fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07280c40b0 0x7f07281e3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:25.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.706+0000 7f07277fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07280c40b0 0x7f07281e3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36924/0 (socket says 172.21.15.23:36924) 2023-06-22T08:16:25.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.707+0000 7f07277fe700 1 -- 172.21.15.23:0/687619346 learned_addr learned my addr 172.21.15.23:0/687619346 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:25.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.707+0000 7f0726ffd700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07280c9fd0 0x7f07280b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:25.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.707+0000 7f0727fff700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0728152ce0 0x7f07281559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:25.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.707+0000 7f07277fe700 1 -- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07280c9fd0 msgr2=0x7f07280b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:25.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.707+0000 7f07277fe700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07280c9fd0 0x7f07280b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:25.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.707+0000 7f07277fe700 1 -- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0728152ce0 msgr2=0x7f07281559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:25.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.707+0000 7f07277fe700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0728152ce0 0x7f07281559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:25.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.707+0000 7f07277fe700 1 -- 172.21.15.23:0/687619346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0718009550 con 0x7f07280c40b0 2023-06-22T08:16:25.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.707+0000 7f07277fe700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07280c40b0 0x7f07281e3110 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0718009c40 tx=0x7f0718004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:25.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.708+0000 7f070ffff700 1 -- 172.21.15.23:0/687619346 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718004510 con 0x7f07280c40b0 2023-06-22T08:16:25.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.708+0000 7f072e1ff700 1 -- 172.21.15.23:0/687619346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07281e3650 con 0x7f07280c40b0 2023-06-22T08:16:25.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.708+0000 7f070ffff700 1 -- 172.21.15.23:0/687619346 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07180036a0 con 0x7f07280c40b0 2023-06-22T08:16:25.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.708+0000 7f070ffff700 1 -- 172.21.15.23:0/687619346 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718003a50 con 0x7f07280c40b0 2023-06-22T08:16:25.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.708+0000 7f072e1ff700 1 -- 172.21.15.23:0/687619346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07281e3f10 con 0x7f07280c40b0 2023-06-22T08:16:25.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.709+0000 7f072e1ff700 1 -- 172.21.15.23:0/687619346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072804ea20 con 0x7f07280c40b0 2023-06-22T08:16:25.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.710+0000 7f070ffff700 1 -- 172.21.15.23:0/687619346 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0718003bb0 con 0x7f07280c40b0 2023-06-22T08:16:25.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.711+0000 7f070ffff700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f071406ace0 0x7f071406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:25.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.711+0000 7f070ffff700 1 -- 172.21.15.23:0/687619346 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f071808ad50 con 0x7f07280c40b0 2023-06-22T08:16:25.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.711+0000 7f0726ffd700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f071406ace0 0x7f071406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:25.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.712+0000 7f0726ffd700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f071406ace0 0x7f071406d190 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f07280b4490 tx=0x7f071c007ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:25.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.715+0000 7f070ffff700 1 -- 172.21.15.23:0/687619346 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0718055770 con 0x7f07280c40b0 2023-06-22T08:16:25.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.970+0000 7f072e1ff700 1 -- 172.21.15.23:0/687619346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07280b37c0 con 0x7f07280c40b0 2023-06-22T08:16:25.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.971+0000 7f070ffff700 1 -- 172.21.15.23:0/687619346 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5508 (secure 0 0 0) 0x7f0718003e60 con 0x7f07280c40b0 2023-06-22T08:16:25.975 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:25.975 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:24.518604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:25.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.973+0000 7f072e1ff700 1 -- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f071406ace0 msgr2=0x7f071406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:25.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.973+0000 7f072e1ff700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f071406ace0 0x7f071406d190 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f07280b4490 tx=0x7f071c007ed0 comp rx=0 tx=0).stop 2023-06-22T08:16:25.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.973+0000 7f072e1ff700 1 -- 172.21.15.23:0/687619346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07280c40b0 msgr2=0x7f07281e3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:25.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.973+0000 7f072e1ff700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07280c40b0 0x7f07281e3110 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0718009c40 tx=0x7f0718004750 comp rx=0 tx=0).stop 2023-06-22T08:16:25.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.973+0000 7f072e1ff700 1 -- 172.21.15.23:0/687619346 shutdown_connections 2023-06-22T08:16:25.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.973+0000 7f072e1ff700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f071406ace0 0x7f071406d190 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:25.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.973+0000 7f072e1ff700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0728152ce0 0x7f07281559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:25.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.973+0000 7f072e1ff700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07280c40b0 0x7f07281e3110 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:25.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.973+0000 7f072e1ff700 1 --2- 172.21.15.23:0/687619346 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07280c9fd0 0x7f07280b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:25.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.973+0000 7f072e1ff700 1 -- 172.21.15.23:0/687619346 >> 172.21.15.23:0/687619346 conn(0x7f072806ae30 msgr2=0x7f07280ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:25.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.974+0000 7f072e1ff700 1 -- 172.21.15.23:0/687619346 shutdown_connections 2023-06-22T08:16:25.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:25.974+0000 7f072e1ff700 1 -- 172.21.15.23:0/687619346 wait complete. 2023-06-22T08:16:25.979 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 132 2023-06-22T08:16:25.995 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-06-22T08:16:25.995 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:16:25.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:26.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.192+0000 7fb987fff700 1 Processor -- start 2023-06-22T08:16:26.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.192+0000 7fb987fff700 1 -- start start 2023-06-22T08:16:26.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.193+0000 7fb987fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9800c69e0 0x7fb980155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:26.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.193+0000 7fb987fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800c51a0 con 0x7fb980071070 2023-06-22T08:16:26.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.193+0000 7fb987fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800c44e0 con 0x7fb98014f850 2023-06-22T08:16:26.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.193+0000 7fb987fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800c5440 con 0x7fb9800c69e0 2023-06-22T08:16:26.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.193+0000 7fb98659c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb980071070 0x7fb98015fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:43820/0 (socket says 172.21.15.23:43820) 2023-06-22T08:16:26.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.193+0000 7fb98659c700 1 -- 172.21.15.23:0/2199250494 learned_addr learned my addr 172.21.15.23:0/2199250494 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:26.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.193+0000 7fb98559a700 1 --2- 172.21.15.23:0/2199250494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9800c69e0 0x7fb980155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:26.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.194+0000 7fb984d99700 1 -- 172.21.15.23:0/2199250494 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242015516 0 0) 0x7fb9800c51a0 con 0x7fb980071070 2023-06-22T08:16:26.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.194+0000 7fb984d99700 1 -- 172.21.15.23:0/2199250494 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb96c0035b0 con 0x7fb980071070 2023-06-22T08:16:26.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.194+0000 7fb984d99700 1 -- 172.21.15.23:0/2199250494 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967159622 0 0) 0x7fb9800c44e0 con 0x7fb98014f850 2023-06-22T08:16:26.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.194+0000 7fb984d99700 1 -- 172.21.15.23:0/2199250494 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9800c51a0 con 0x7fb98014f850 2023-06-22T08:16:26.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.194+0000 7fb98559a700 1 -- 172.21.15.23:0/2199250494 >> v1:172.21.15.67:6789/0 conn(0x7fb98014f850 legacy=0x7fb9800c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:26.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.194+0000 7fb98559a700 1 -- 172.21.15.23:0/2199250494 >> v1:172.21.15.23:6789/0 conn(0x7fb980071070 legacy=0x7fb98015fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:26.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.194+0000 7fb98559a700 1 -- 172.21.15.23:0/2199250494 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800c5720 con 0x7fb9800c69e0 2023-06-22T08:16:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.194+0000 7fb98559a700 1 --2- 172.21.15.23:0/2199250494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9800c69e0 0x7fb980155600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb970009870 tx=0x7fb970009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e692368b7846f34 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.195+0000 7fb984d99700 1 -- 172.21.15.23:0/2199250494 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970011830 con 0x7fb9800c69e0 2023-06-22T08:16:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.195+0000 7fb984d99700 1 -- 172.21.15.23:0/2199250494 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb970011990 con 0x7fb9800c69e0 2023-06-22T08:16:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.195+0000 7fb984d99700 1 -- 172.21.15.23:0/2199250494 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970011d20 con 0x7fb9800c69e0 2023-06-22T08:16:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.195+0000 7fb987fff700 1 -- 172.21.15.23:0/2199250494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9800c69e0 msgr2=0x7fb980155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.195+0000 7fb987fff700 1 --2- 172.21.15.23:0/2199250494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9800c69e0 0x7fb980155600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb970009870 tx=0x7fb970009b80 comp rx=0 tx=0).stop 2023-06-22T08:16:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.195+0000 7fb987fff700 1 -- 172.21.15.23:0/2199250494 shutdown_connections 2023-06-22T08:16:26.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.195+0000 7fb987fff700 1 --2- 172.21.15.23:0/2199250494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9800c69e0 0x7fb980155600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:26.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.195+0000 7fb987fff700 1 -- 172.21.15.23:0/2199250494 >> 172.21.15.23:0/2199250494 conn(0x7fb9800ca3f0 msgr2=0x7fb9800ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:26.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.195+0000 7fb987fff700 1 -- 172.21.15.23:0/2199250494 shutdown_connections 2023-06-22T08:16:26.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.195+0000 7fb987fff700 1 -- 172.21.15.23:0/2199250494 wait complete. 2023-06-22T08:16:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.197+0000 7fb987fff700 1 Processor -- start 2023-06-22T08:16:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.197+0000 7fb987fff700 1 -- start start 2023-06-22T08:16:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb987fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb980071070 0x7fb98006fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb987fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9800c69e0 0x7fb9801503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb987fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98014f850 0x7fb980150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb987fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb980150f00 con 0x7fb9800c69e0 2023-06-22T08:16:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb987fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb980155330 con 0x7fb98014f850 2023-06-22T08:16:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb987fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9801557b0 con 0x7fb980071070 2023-06-22T08:16:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb98559a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9800c69e0 0x7fb9801503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb98559a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9800c69e0 0x7fb9801503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51062/0 (socket says 172.21.15.23:51062) 2023-06-22T08:16:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb98559a700 1 -- 172.21.15.23:0/1230130307 learned_addr learned my addr 172.21.15.23:0/1230130307 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb985d9b700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb980071070 0x7fb98006fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.198+0000 7fb98659c700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98014f850 0x7fb980150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb98559a700 1 -- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb980071070 msgr2=0x7fb98006fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb98559a700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb980071070 0x7fb98006fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb98559a700 1 -- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98014f850 msgr2=0x7fb980150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb98559a700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98014f850 0x7fb980150920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb98559a700 1 -- 172.21.15.23:0/1230130307 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9700094a0 con 0x7fb9800c69e0 2023-06-22T08:16:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb98559a700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9800c69e0 0x7fb9801503e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb9700090b0 tx=0x7fb970004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb9767fc700 1 -- 172.21.15.23:0/1230130307 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97001a6e0 con 0x7fb9800c69e0 2023-06-22T08:16:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb9767fc700 1 -- 172.21.15.23:0/1230130307 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97001a840 con 0x7fb9800c69e0 2023-06-22T08:16:26.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb9767fc700 1 -- 172.21.15.23:0/1230130307 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97001abb0 con 0x7fb9800c69e0 2023-06-22T08:16:26.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb987fff700 1 -- 172.21.15.23:0/1230130307 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb980155a90 con 0x7fb9800c69e0 2023-06-22T08:16:26.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.199+0000 7fb987fff700 1 -- 172.21.15.23:0/1230130307 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb980156050 con 0x7fb9800c69e0 2023-06-22T08:16:26.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.200+0000 7fb987fff700 1 -- 172.21.15.23:0/1230130307 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98004ea20 con 0x7fb9800c69e0 2023-06-22T08:16:26.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.202+0000 7fb9767fc700 1 -- 172.21.15.23:0/1230130307 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb970022070 con 0x7fb9800c69e0 2023-06-22T08:16:26.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.202+0000 7fb9767fc700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb96406ace0 0x7fb96406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:26.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.203+0000 7fb9767fc700 1 -- 172.21.15.23:0/1230130307 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7fb97008a6c0 con 0x7fb9800c69e0 2023-06-22T08:16:26.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.203+0000 7fb985d9b700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb96406ace0 0x7fb96406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:26.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.204+0000 7fb985d9b700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb96406ace0 0x7fb96406d190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0011e0 tx=0x7fb97c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:26.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.207+0000 7fb9767fc700 1 -- 172.21.15.23:0/1230130307 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb970055180 con 0x7fb9800c69e0 2023-06-22T08:16:26.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.457+0000 7fb987fff700 1 -- 172.21.15.23:0/1230130307 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9800cb300 con 0x7fb9800c69e0 2023-06-22T08:16:26.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.458+0000 7fb9767fc700 1 -- 172.21.15.23:0/1230130307 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5508 (secure 0 0 0) 0x7fb9700587a0 con 0x7fb9800c69e0 2023-06-22T08:16:26.461 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:26.461 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:24.518604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:26.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 -- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb96406ace0 msgr2=0x7fb96406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:26.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb96406ace0 0x7fb96406d190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0011e0 tx=0x7fb97c009040 comp rx=0 tx=0).stop 2023-06-22T08:16:26.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 -- 172.21.15.23:0/1230130307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9800c69e0 msgr2=0x7fb9801503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:26.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9800c69e0 0x7fb9801503e0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb9700090b0 tx=0x7fb970004750 comp rx=0 tx=0).stop 2023-06-22T08:16:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 -- 172.21.15.23:0/1230130307 shutdown_connections 2023-06-22T08:16:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb96406ace0 0x7fb96406d190 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb98014f850 0x7fb980150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb980071070 0x7fb98006fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 --2- 172.21.15.23:0/1230130307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9800c69e0 0x7fb9801503e0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 -- 172.21.15.23:0/1230130307 >> 172.21.15.23:0/1230130307 conn(0x7fb9800ca3f0 msgr2=0x7fb9800c17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:26.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 -- 172.21.15.23:0/1230130307 shutdown_connections 2023-06-22T08:16:26.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:26.460+0000 7fb987fff700 1 -- 172.21.15.23:0/1230130307 wait complete. 2023-06-22T08:16:26.465 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 132 2023-06-22T08:16:26.482 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 132, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 132, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5576": { "addr": "172.21.15.67:6837/2064462140", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 2064462140, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 2064462140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5576, "incarnation": 109, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 }, "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:16:24.518604+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 6313, "mds_3": 5995, "mds_4": 5576 } } } ], "standbys": [] } 2023-06-22T08:16:28.484 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.693+0000 7efda23ff700 1 Processor -- start 2023-06-22T08:16:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.693+0000 7efda23ff700 1 -- start start 2023-06-22T08:16:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.695+0000 7efda23ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd9c152d70 0x7efd9c0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.695+0000 7efda23ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c0c45f0 0x7efd9c0c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.695+0000 7efda23ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c0c4f90 con 0x7efd9c159330 2023-06-22T08:16:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.695+0000 7efda23ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9c0c95e0 con 0x7efd9c152d70 2023-06-22T08:16:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.695+0000 7efda23ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c159790 con 0x7efd9c0c45f0 2023-06-22T08:16:28.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.695+0000 7efda099c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c0c45f0 0x7efd9c0c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:28.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.695+0000 7efd9bfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd9c152d70 0x7efd9c0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:28.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.695+0000 7efda099c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c0c45f0 0x7efd9c0c4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:41428/0 (socket says 172.21.15.23:41428) 2023-06-22T08:16:28.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.695+0000 7efda099c700 1 -- 172.21.15.23:0/3296549044 learned_addr learned my addr 172.21.15.23:0/3296549044 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:28.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.696+0000 7efda099c700 1 -- 172.21.15.23:0/3296549044 >> v2:172.21.15.67:3300/0 conn(0x7efd9c152d70 msgr2=0x7efd9c0c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.696+0000 7efda099c700 1 --2- 172.21.15.23:0/3296549044 >> v2:172.21.15.67:3300/0 conn(0x7efd9c152d70 0x7efd9c0c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.696+0000 7efda099c700 1 -- 172.21.15.23:0/3296549044 >> v1:172.21.15.67:6789/0 conn(0x7efd9c159330 legacy=0x7efd9c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.696+0000 7efda099c700 1 -- 172.21.15.23:0/3296549044 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c15bea0 con 0x7efd9c0c45f0 2023-06-22T08:16:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.696+0000 7efda099c700 1 --2- 172.21.15.23:0/3296549044 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c0c45f0 0x7efd9c0c4a50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd94009e50 tx=0x7efd9400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a446542df96a94fe server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:28.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.696+0000 7efd9affd700 1 -- 172.21.15.23:0/3296549044 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9400e070 con 0x7efd9c0c45f0 2023-06-22T08:16:28.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.696+0000 7efd9affd700 1 -- 172.21.15.23:0/3296549044 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9400b910 con 0x7efd9c0c45f0 2023-06-22T08:16:28.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.696+0000 7efd9affd700 1 -- 172.21.15.23:0/3296549044 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9400bcc0 con 0x7efd9c0c45f0 2023-06-22T08:16:28.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.697+0000 7efda23ff700 1 -- 172.21.15.23:0/3296549044 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c0c45f0 msgr2=0x7efd9c0c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:28.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.697+0000 7efda23ff700 1 --2- 172.21.15.23:0/3296549044 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c0c45f0 0x7efd9c0c4a50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efd94009e50 tx=0x7efd9400b330 comp rx=0 tx=0).stop 2023-06-22T08:16:28.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.697+0000 7efda23ff700 1 -- 172.21.15.23:0/3296549044 shutdown_connections 2023-06-22T08:16:28.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.697+0000 7efda23ff700 1 --2- 172.21.15.23:0/3296549044 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c0c45f0 0x7efd9c0c4a50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:28.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.697+0000 7efda23ff700 1 --2- 172.21.15.23:0/3296549044 >> v2:172.21.15.67:3300/0 conn(0x7efd9c152d70 0x7efd9c0c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.697+0000 7efda23ff700 1 -- 172.21.15.23:0/3296549044 >> 172.21.15.23:0/3296549044 conn(0x7efd9c06ae30 msgr2=0x7efd9c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.697+0000 7efda23ff700 1 -- 172.21.15.23:0/3296549044 shutdown_connections 2023-06-22T08:16:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.697+0000 7efda23ff700 1 -- 172.21.15.23:0/3296549044 wait complete. 2023-06-22T08:16:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.699+0000 7efda23ff700 1 Processor -- start 2023-06-22T08:16:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.699+0000 7efda23ff700 1 -- start start 2023-06-22T08:16:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efda23ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd9c0c45f0 0x7efd9c15b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efda23ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c152d70 0x7efd9c156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efda23ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c159330 0x7efd9c15b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efda23ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c15ab70 con 0x7efd9c0c45f0 2023-06-22T08:16:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efda23ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c1e5d70 con 0x7efd9c159330 2023-06-22T08:16:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efda23ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9c0b2490 con 0x7efd9c152d70 2023-06-22T08:16:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd9c0c45f0 0x7efd9c15b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efd9bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd9c0c45f0 0x7efd9c15b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51074/0 (socket says 172.21.15.23:51074) 2023-06-22T08:16:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efd9bfff700 1 -- 172.21.15.23:0/2553935598 learned_addr learned my addr 172.21.15.23:0/2553935598 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efda099c700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c159330 0x7efd9c15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efd9b7fe700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c152d70 0x7efd9c156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efd9bfff700 1 -- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c152d70 msgr2=0x7efd9c156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:28.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efd9bfff700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c152d70 0x7efd9c156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efd9bfff700 1 -- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c159330 msgr2=0x7efd9c15b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.700+0000 7efd9bfff700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c159330 0x7efd9c15b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.701+0000 7efd9bfff700 1 -- 172.21.15.23:0/2553935598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd94009b30 con 0x7efd9c0c45f0 2023-06-22T08:16:28.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.701+0000 7efda099c700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c159330 0x7efd9c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:16:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.701+0000 7efd9bfff700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd9c0c45f0 0x7efd9c15b770 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efd8c00e9b0 tx=0x7efd8c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.701+0000 7efd98ff9700 1 -- 172.21.15.23:0/2553935598 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c009710 con 0x7efd9c0c45f0 2023-06-22T08:16:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.701+0000 7efd98ff9700 1 -- 172.21.15.23:0/2553935598 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8c00ee50 con 0x7efd9c0c45f0 2023-06-22T08:16:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.701+0000 7efd98ff9700 1 -- 172.21.15.23:0/2553935598 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c00ca50 con 0x7efd9c0c45f0 2023-06-22T08:16:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.701+0000 7efda23ff700 1 -- 172.21.15.23:0/2553935598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9c0b2770 con 0x7efd9c0c45f0 2023-06-22T08:16:28.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.701+0000 7efda23ff700 1 -- 172.21.15.23:0/2553935598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9c15a800 con 0x7efd9c0c45f0 2023-06-22T08:16:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.702+0000 7efda23ff700 1 -- 172.21.15.23:0/2553935598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9c151c90 con 0x7efd9c0c45f0 2023-06-22T08:16:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.703+0000 7efd98ff9700 1 -- 172.21.15.23:0/2553935598 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efd8c00cbb0 con 0x7efd9c0c45f0 2023-06-22T08:16:28.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.704+0000 7efd98ff9700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd8806ace0 0x7efd8806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.704+0000 7efd98ff9700 1 -- 172.21.15.23:0/2553935598 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7efd8c089ae0 con 0x7efd9c0c45f0 2023-06-22T08:16:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.705+0000 7efd9b7fe700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd8806ace0 0x7efd8806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:28.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.705+0000 7efd9b7fe700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd8806ace0 0x7efd8806d190 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0cb090 tx=0x7efd90009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:28.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.709+0000 7efd98ff9700 1 -- 172.21.15.23:0/2553935598 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efd8c057f90 con 0x7efd9c0c45f0 2023-06-22T08:16:28.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.959+0000 7efda23ff700 1 -- 172.21.15.23:0/2553935598 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd9c151f70 con 0x7efd9c0c45f0 2023-06-22T08:16:28.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.959+0000 7efd98ff9700 1 -- 172.21.15.23:0/2553935598 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5508 (secure 0 0 0) 0x7efd8c021070 con 0x7efd9c0c45f0 2023-06-22T08:16:28.963 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:28.964 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:24.518604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:28.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 -- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd8806ace0 msgr2=0x7efd8806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:28.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd8806ace0 0x7efd8806d190 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0cb090 tx=0x7efd90009040 comp rx=0 tx=0).stop 2023-06-22T08:16:28.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 -- 172.21.15.23:0/2553935598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd9c0c45f0 msgr2=0x7efd9c15b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:28.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd9c0c45f0 0x7efd9c15b770 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efd8c00e9b0 tx=0x7efd8c00ecc0 comp rx=0 tx=0).stop 2023-06-22T08:16:28.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 -- 172.21.15.23:0/2553935598 shutdown_connections 2023-06-22T08:16:28.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efd8806ace0 0x7efd8806d190 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:28.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd9c159330 0x7efd9c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:28.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd9c0c45f0 0x7efd9c15b770 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:28.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 --2- 172.21.15.23:0/2553935598 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efd9c152d70 0x7efd9c156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:28.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 -- 172.21.15.23:0/2553935598 >> 172.21.15.23:0/2553935598 conn(0x7efd9c06ae30 msgr2=0x7efd9c150e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:28.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 -- 172.21.15.23:0/2553935598 shutdown_connections 2023-06-22T08:16:28.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:28.962+0000 7efda23ff700 1 -- 172.21.15.23:0/2553935598 wait complete. 2023-06-22T08:16:28.967 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 132 2023-06-22T08:16:28.983 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:16:30.984 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:31.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.193+0000 7f7324dff700 1 Processor -- start 2023-06-22T08:16:31.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.193+0000 7f7324dff700 1 -- start start 2023-06-22T08:16:31.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.195+0000 7f7324dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7320152d70 0x7f73200c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:31.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.195+0000 7f7324dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7320159330 0x7f73200cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:31.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.195+0000 7f7324dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f73200cb800 0x7f73200c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:31.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.195+0000 7f7324dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73200c95e0 con 0x7f7320159330 2023-06-22T08:16:31.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.195+0000 7f7324dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7320159790 con 0x7f7320152d70 2023-06-22T08:16:31.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.195+0000 7f7324dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732015bea0 con 0x7f73200cb800 2023-06-22T08:16:31.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.196+0000 7f731dd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7320159330 0x7f73200cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:31.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.196+0000 7f731dd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7320159330 0x7f73200cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51090/0 (socket says 172.21.15.23:51090) 2023-06-22T08:16:31.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.196+0000 7f731dd9b700 1 -- 172.21.15.23:0/3582257093 learned_addr learned my addr 172.21.15.23:0/3582257093 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:31.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.196+0000 7f731e59c700 1 --2- 172.21.15.23:0/3582257093 >> v2:172.21.15.67:3300/0 conn(0x7f7320152d70 0x7f73200c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:31.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.196+0000 7f731ed9d700 1 --2- 172.21.15.23:0/3582257093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f73200cb800 0x7f73200c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:31.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.196+0000 7f731dd9b700 1 -- 172.21.15.23:0/3582257093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f73200cb800 msgr2=0x7f73200c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:31.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.196+0000 7f731dd9b700 1 --2- 172.21.15.23:0/3582257093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f73200cb800 0x7f73200c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.196+0000 7f731dd9b700 1 -- 172.21.15.23:0/3582257093 >> v2:172.21.15.67:3300/0 conn(0x7f7320152d70 msgr2=0x7f73200c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:31.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.196+0000 7f731dd9b700 1 --2- 172.21.15.23:0/3582257093 >> v2:172.21.15.67:3300/0 conn(0x7f7320152d70 0x7f73200c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.196+0000 7f731dd9b700 1 -- 172.21.15.23:0/3582257093 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732015c160 con 0x7f7320159330 2023-06-22T08:16:31.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.197+0000 7f731dd9b700 1 --2- 172.21.15.23:0/3582257093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320159330 0x7f73200cb2c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f73200c8ef0 tx=0x7f7314009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ad62e0ddbc19f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:31.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.197+0000 7f731d59a700 1 -- 172.21.15.23:0/3582257093 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314011830 con 0x7f7320159330 2023-06-22T08:16:31.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.197+0000 7f731d59a700 1 -- 172.21.15.23:0/3582257093 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7314011990 con 0x7f7320159330 2023-06-22T08:16:31.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.197+0000 7f731d59a700 1 -- 172.21.15.23:0/3582257093 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314011d20 con 0x7f7320159330 2023-06-22T08:16:31.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.198+0000 7f7324dff700 1 -- 172.21.15.23:0/3582257093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320159330 msgr2=0x7f73200cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:31.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.198+0000 7f7324dff700 1 --2- 172.21.15.23:0/3582257093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320159330 0x7f73200cb2c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f73200c8ef0 tx=0x7f7314009bd0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.198+0000 7f7324dff700 1 -- 172.21.15.23:0/3582257093 shutdown_connections 2023-06-22T08:16:31.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.198+0000 7f7324dff700 1 --2- 172.21.15.23:0/3582257093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f73200cb800 0x7f73200c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.198+0000 7f7324dff700 1 --2- 172.21.15.23:0/3582257093 >> v2:172.21.15.67:3300/0 conn(0x7f7320152d70 0x7f73200c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.198+0000 7f7324dff700 1 --2- 172.21.15.23:0/3582257093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320159330 0x7f73200cb2c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.198+0000 7f7324dff700 1 -- 172.21.15.23:0/3582257093 >> 172.21.15.23:0/3582257093 conn(0x7f732006ae30 msgr2=0x7f732006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:31.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.198+0000 7f7324dff700 1 -- 172.21.15.23:0/3582257093 shutdown_connections 2023-06-22T08:16:31.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.198+0000 7f7324dff700 1 -- 172.21.15.23:0/3582257093 wait complete. 2023-06-22T08:16:31.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.200+0000 7f7324dff700 1 Processor -- start 2023-06-22T08:16:31.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.200+0000 7f7324dff700 1 -- start start 2023-06-22T08:16:31.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.201+0000 7f7324dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73200cb800 0x7f73200ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:31.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.201+0000 7f7324dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320152d70 0x7f73200b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:31.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.201+0000 7f7324dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7320159330 0x7f73200ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.201+0000 7f7324dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73200b4880 con 0x7f7320152d70 2023-06-22T08:16:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.201+0000 7f7324dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73200cc790 con 0x7f73200cb800 2023-06-22T08:16:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.201+0000 7f7324dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73200cca70 con 0x7f7320159330 2023-06-22T08:16:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320152d70 0x7f73200b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320152d70 0x7f73200b4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51104/0 (socket says 172.21.15.23:51104) 2023-06-22T08:16:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731dd9b700 1 -- 172.21.15.23:0/1353021141 learned_addr learned my addr 172.21.15.23:0/1353021141 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731e59c700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73200cb800 0x7f73200ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:31.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731ed9d700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7320159330 0x7f73200ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731dd9b700 1 -- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7320159330 msgr2=0x7f73200ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:31.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731dd9b700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7320159330 0x7f73200ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731dd9b700 1 -- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73200cb800 msgr2=0x7f73200ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:31.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731dd9b700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73200cb800 0x7f73200ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731dd9b700 1 -- 172.21.15.23:0/1353021141 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7314009550 con 0x7f7320152d70 2023-06-22T08:16:31.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.202+0000 7f731dd9b700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320152d70 0x7f73200b4340 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7314000c00 tx=0x7f7314004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:31.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.203+0000 7f730effd700 1 -- 172.21.15.23:0/1353021141 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314004240 con 0x7f7320152d70 2023-06-22T08:16:31.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.203+0000 7f730effd700 1 -- 172.21.15.23:0/1353021141 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f731401f070 con 0x7f7320152d70 2023-06-22T08:16:31.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.203+0000 7f7324dff700 1 -- 172.21.15.23:0/1353021141 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73200ccd50 con 0x7f7320152d70 2023-06-22T08:16:31.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.203+0000 7f730effd700 1 -- 172.21.15.23:0/1353021141 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73140036a0 con 0x7f7320152d70 2023-06-22T08:16:31.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.203+0000 7f7324dff700 1 -- 172.21.15.23:0/1353021141 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73200cd310 con 0x7f7320152d70 2023-06-22T08:16:31.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.204+0000 7f7324dff700 1 -- 172.21.15.23:0/1353021141 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732004ea20 con 0x7f7320152d70 2023-06-22T08:16:31.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.205+0000 7f730effd700 1 -- 172.21.15.23:0/1353021141 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7314021030 con 0x7f7320152d70 2023-06-22T08:16:31.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.206+0000 7f730effd700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f730006ad10 0x7f730006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:31.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.206+0000 7f730effd700 1 -- 172.21.15.23:0/1353021141 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f731408a7f0 con 0x7f7320152d70 2023-06-22T08:16:31.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.206+0000 7f731e59c700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f730006ad10 0x7f730006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:31.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.208+0000 7f731e59c700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f730006ad10 0x7f730006d1c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7310009be0 tx=0x7f7310009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:31.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.218+0000 7f730effd700 1 -- 172.21.15.23:0/1353021141 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7314055230 con 0x7f7320152d70 2023-06-22T08:16:31.511 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.508+0000 7f7324dff700 1 -- 172.21.15.23:0/1353021141 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73200c8cd0 con 0x7f7320152d70 2023-06-22T08:16:31.512 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.508+0000 7f730effd700 1 -- 172.21.15.23:0/1353021141 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5508 (secure 0 0 0) 0x7f731402a090 con 0x7f7320152d70 2023-06-22T08:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:24.518604+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:31.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 -- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f730006ad10 msgr2=0x7f730006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:31.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f730006ad10 0x7f730006d1c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7310009be0 tx=0x7f7310009210 comp rx=0 tx=0).stop 2023-06-22T08:16:31.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 -- 172.21.15.23:0/1353021141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320152d70 msgr2=0x7f73200b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:31.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320152d70 0x7f73200b4340 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7314000c00 tx=0x7f7314004750 comp rx=0 tx=0).stop 2023-06-22T08:16:31.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 -- 172.21.15.23:0/1353021141 shutdown_connections 2023-06-22T08:16:31.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7320159330 0x7f73200ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f730006ad10 0x7f730006d1c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73200cb800 0x7f73200ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 --2- 172.21.15.23:0/1353021141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7320152d70 0x7f73200b4340 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:31.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 -- 172.21.15.23:0/1353021141 >> 172.21.15.23:0/1353021141 conn(0x7f732006ae30 msgr2=0x7f73200caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:31.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 -- 172.21.15.23:0/1353021141 shutdown_connections 2023-06-22T08:16:31.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:31.511+0000 7f7324dff700 1 -- 172.21.15.23:0/1353021141 wait complete. 2023-06-22T08:16:31.517 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 132 2023-06-22T08:16:31.535 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:16:33.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.743+0000 7f825dfff700 1 Processor -- start 2023-06-22T08:16:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.743+0000 7f825dfff700 1 -- start start 2023-06-22T08:16:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.744+0000 7f825dfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8258152ce0 0x7f82580cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.744+0000 7f825dfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82580c9fd0 0x7f825815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.744+0000 7f825dfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8258158880 0x7f8258158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.744+0000 7f825dfff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82581530b0 con 0x7f8258152ce0 2023-06-22T08:16:33.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.744+0000 7f825dfff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82580cf6f0 con 0x7f8258158880 2023-06-22T08:16:33.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.744+0000 7f825dfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825815c370 con 0x7f82580c9fd0 2023-06-22T08:16:33.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.745+0000 7f82577fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8258152ce0 0x7f82580cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:33.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.745+0000 7f82577fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8258152ce0 0x7f82580cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51106/0 (socket says 172.21.15.23:51106) 2023-06-22T08:16:33.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.745+0000 7f82577fe700 1 -- 172.21.15.23:0/4143818734 learned_addr learned my addr 172.21.15.23:0/4143818734 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:33.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.745+0000 7f8256ffd700 1 --2- 172.21.15.23:0/4143818734 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82580c9fd0 0x7f825815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:33.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.745+0000 7f8257fff700 1 --2- 172.21.15.23:0/4143818734 >> v2:172.21.15.67:3300/0 conn(0x7f8258158880 0x7f8258158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:33.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.745+0000 7f82577fe700 1 -- 172.21.15.23:0/4143818734 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82580c9fd0 msgr2=0x7f825815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:33.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.745+0000 7f82577fe700 1 --2- 172.21.15.23:0/4143818734 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82580c9fd0 0x7f825815c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:33.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.745+0000 7f82577fe700 1 -- 172.21.15.23:0/4143818734 >> v2:172.21.15.67:3300/0 conn(0x7f8258158880 msgr2=0x7f8258158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:33.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.745+0000 7f82577fe700 1 --2- 172.21.15.23:0/4143818734 >> v2:172.21.15.67:3300/0 conn(0x7f8258158880 0x7f8258158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:33.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.745+0000 7f82577fe700 1 -- 172.21.15.23:0/4143818734 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825815bda0 con 0x7f8258152ce0 2023-06-22T08:16:33.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.746+0000 7f82577fe700 1 --2- 172.21.15.23:0/4143818734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 0x7f82580cf1b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f825808b9e0 tx=0x7f8248009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf51ca54c8357b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:33.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.746+0000 7f82567fc700 1 -- 172.21.15.23:0/4143818734 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8248011830 con 0x7f8258152ce0 2023-06-22T08:16:33.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.746+0000 7f82567fc700 1 -- 172.21.15.23:0/4143818734 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8248011990 con 0x7f8258152ce0 2023-06-22T08:16:33.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.746+0000 7f82567fc700 1 -- 172.21.15.23:0/4143818734 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8248011d20 con 0x7f8258152ce0 2023-06-22T08:16:33.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.746+0000 7f825dfff700 1 -- 172.21.15.23:0/4143818734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 msgr2=0x7f82580cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.746+0000 7f825dfff700 1 --2- 172.21.15.23:0/4143818734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 0x7f82580cf1b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f825808b9e0 tx=0x7f8248009bd0 comp rx=0 tx=0).stop 2023-06-22T08:16:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.746+0000 7f825dfff700 1 -- 172.21.15.23:0/4143818734 shutdown_connections 2023-06-22T08:16:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.747+0000 7f825dfff700 1 --2- 172.21.15.23:0/4143818734 >> v2:172.21.15.67:3300/0 conn(0x7f8258158880 0x7f8258158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.747+0000 7f825dfff700 1 --2- 172.21.15.23:0/4143818734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 0x7f82580cf1b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.747+0000 7f825dfff700 1 --2- 172.21.15.23:0/4143818734 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82580c9fd0 0x7f825815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.747+0000 7f825dfff700 1 -- 172.21.15.23:0/4143818734 >> 172.21.15.23:0/4143818734 conn(0x7f825806ae30 msgr2=0x7f825806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.747+0000 7f825dfff700 1 -- 172.21.15.23:0/4143818734 shutdown_connections 2023-06-22T08:16:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.747+0000 7f825dfff700 1 -- 172.21.15.23:0/4143818734 wait complete. 2023-06-22T08:16:33.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.748+0000 7f825dfff700 1 Processor -- start 2023-06-22T08:16:33.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.748+0000 7f825dfff700 1 -- start start 2023-06-22T08:16:33.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.749+0000 7f825dfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82580c9fd0 0x7f82581514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.749+0000 7f825dfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 0x7f82581ef740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.749+0000 7f825dfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8258158880 0x7f82581efc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.749+0000 7f825dfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82580ce950 con 0x7f8258152ce0 2023-06-22T08:16:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.749+0000 7f825dfff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82581547b0 con 0x7f8258158880 2023-06-22T08:16:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.749+0000 7f825dfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82581ef2b0 con 0x7f82580c9fd0 2023-06-22T08:16:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f8256ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 0x7f82581ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:33.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f8256ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 0x7f82581ef740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51110/0 (socket says 172.21.15.23:51110) 2023-06-22T08:16:33.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f8256ffd700 1 -- 172.21.15.23:0/1239188507 learned_addr learned my addr 172.21.15.23:0/1239188507 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:33.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f8257fff700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8258158880 0x7f82581efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:33.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f82577fe700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82580c9fd0 0x7f82581514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:33.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f8256ffd700 1 -- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82580c9fd0 msgr2=0x7f82581514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:33.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f8256ffd700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82580c9fd0 0x7f82581514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:33.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f8256ffd700 1 -- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8258158880 msgr2=0x7f82581efc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:33.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f8256ffd700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8258158880 0x7f82581efc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:33.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f8256ffd700 1 -- 172.21.15.23:0/1239188507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8248009550 con 0x7f8258152ce0 2023-06-22T08:16:33.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f8256ffd700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 0x7f82581ef740 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f824c00e9b0 tx=0x7f824c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:33.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f823ffff700 1 -- 172.21.15.23:0/1239188507 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c009710 con 0x7f8258152ce0 2023-06-22T08:16:33.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f823ffff700 1 -- 172.21.15.23:0/1239188507 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824c00ee50 con 0x7f8258152ce0 2023-06-22T08:16:33.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.750+0000 7f825dfff700 1 -- 172.21.15.23:0/1239188507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82581549e0 con 0x7f8258152ce0 2023-06-22T08:16:33.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.751+0000 7f823ffff700 1 -- 172.21.15.23:0/1239188507 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824c00ca50 con 0x7f8258152ce0 2023-06-22T08:16:33.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.751+0000 7f825dfff700 1 -- 172.21.15.23:0/1239188507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8258154fa0 con 0x7f8258152ce0 2023-06-22T08:16:33.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.752+0000 7f825dfff700 1 -- 172.21.15.23:0/1239188507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825804ea20 con 0x7f8258152ce0 2023-06-22T08:16:33.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.753+0000 7f823ffff700 1 -- 172.21.15.23:0/1239188507 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f824c00cbb0 con 0x7f8258152ce0 2023-06-22T08:16:33.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.754+0000 7f823ffff700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f824406ace0 0x7f824406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:33.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.754+0000 7f823ffff700 1 -- 172.21.15.23:0/1239188507 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f824c089a40 con 0x7f8258152ce0 2023-06-22T08:16:33.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.754+0000 7f82577fe700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f824406ace0 0x7f824406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.755+0000 7f82577fe700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f824406ace0 0x7f824406d190 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8248009c10 tx=0x7f8248008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:33.758+0000 7f823ffff700 1 -- 172.21.15.23:0/1239188507 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f824c08be60 con 0x7f8258152ce0 2023-06-22T08:16:34.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.013+0000 7f825dfff700 1 -- 172.21.15.23:0/1239188507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82580c9db0 con 0x7f8258152ce0 2023-06-22T08:16:34.016 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.013+0000 7f823ffff700 1 -- 172.21.15.23:0/1239188507 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+5510 (secure 0 0 0) 0x7f824c021070 con 0x7f8258152ce0 2023-06-22T08:16:34.017 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:34.017 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:32.933102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:resolve","state_seq":30,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:34.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 -- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f824406ace0 msgr2=0x7f824406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:34.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f824406ace0 0x7f824406d190 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8248009c10 tx=0x7f8248008030 comp rx=0 tx=0).stop 2023-06-22T08:16:34.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 -- 172.21.15.23:0/1239188507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 msgr2=0x7f82581ef740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:34.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 0x7f82581ef740 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f824c00e9b0 tx=0x7f824c00ecc0 comp rx=0 tx=0).stop 2023-06-22T08:16:34.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 -- 172.21.15.23:0/1239188507 shutdown_connections 2023-06-22T08:16:34.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f824406ace0 0x7f824406d190 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:34.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8258158880 0x7f82581efc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:34.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f82580c9fd0 0x7f82581514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:34.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 --2- 172.21.15.23:0/1239188507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8258152ce0 0x7f82581ef740 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:34.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 -- 172.21.15.23:0/1239188507 >> 172.21.15.23:0/1239188507 conn(0x7f825806ae30 msgr2=0x7f82580c49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:34.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 -- 172.21.15.23:0/1239188507 shutdown_connections 2023-06-22T08:16:34.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:34.016+0000 7f825dfff700 1 -- 172.21.15.23:0/1239188507 wait complete. 2023-06-22T08:16:34.021 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 133 2023-06-22T08:16:34.037 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:16:36.038 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:36.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.247+0000 7f47225ff700 1 Processor -- start 2023-06-22T08:16:36.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.248+0000 7f47225ff700 1 -- start start 2023-06-22T08:16:36.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.249+0000 7f47225ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f471c0c4a30 0x7f471c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:36.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.249+0000 7f47225ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f471c0cb7a0 0x7f471c15c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:36.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.249+0000 7f47225ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c159730 con 0x7f471c152a10 2023-06-22T08:16:36.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.249+0000 7f47225ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471c0c5470 con 0x7f471c0cb7a0 2023-06-22T08:16:36.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.249+0000 7f47225ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c158910 con 0x7f471c0c4a30 2023-06-22T08:16:36.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.249+0000 7f4720b9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f471c0cb7a0 0x7f471c15c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:36.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.249+0000 7f471b7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f471c0c4a30 0x7f471c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:36.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.249+0000 7f471bfff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f471c152a10 0x7f471c0c9200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:55464/0 (socket says 172.21.15.23:55464) 2023-06-22T08:16:36.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.249+0000 7f4720b9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f471c0cb7a0 0x7f471c15c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34760/0 (socket says 172.21.15.23:34760) 2023-06-22T08:16:36.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.249+0000 7f471bfff700 1 -- 172.21.15.23:0/1243653502 learned_addr learned my addr 172.21.15.23:0/1243653502 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:36.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.250+0000 7f4720b9c700 1 -- 172.21.15.23:0/1243653502 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f471c0c4a30 msgr2=0x7f471c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:36.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.250+0000 7f4720b9c700 1 --2- 172.21.15.23:0/1243653502 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f471c0c4a30 0x7f471c0cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:36.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.250+0000 7f4720b9c700 1 -- 172.21.15.23:0/1243653502 >> v1:172.21.15.67:6789/0 conn(0x7f471c152a10 legacy=0x7f471c0c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:36.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.250+0000 7f4720b9c700 1 -- 172.21.15.23:0/1243653502 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c158bb0 con 0x7f471c0cb7a0 2023-06-22T08:16:36.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.250+0000 7f4720b9c700 1 --2- 172.21.15.23:0/1243653502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0cb7a0 0x7f471c15c0b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4714009d90 tx=0x7f471400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ecb99091665f8f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:36.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.250+0000 7f471affd700 1 -- 172.21.15.23:0/1243653502 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471400e070 con 0x7f471c0cb7a0 2023-06-22T08:16:36.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.250+0000 7f471affd700 1 -- 172.21.15.23:0/1243653502 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471400bb20 con 0x7f471c0cb7a0 2023-06-22T08:16:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.250+0000 7f471affd700 1 -- 172.21.15.23:0/1243653502 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4714012430 con 0x7f471c0cb7a0 2023-06-22T08:16:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.251+0000 7f47225ff700 1 -- 172.21.15.23:0/1243653502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0cb7a0 msgr2=0x7f471c15c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.251+0000 7f47225ff700 1 --2- 172.21.15.23:0/1243653502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0cb7a0 0x7f471c15c0b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4714009d90 tx=0x7f471400b330 comp rx=0 tx=0).stop 2023-06-22T08:16:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.251+0000 7f47225ff700 1 -- 172.21.15.23:0/1243653502 shutdown_connections 2023-06-22T08:16:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.251+0000 7f47225ff700 1 --2- 172.21.15.23:0/1243653502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0cb7a0 0x7f471c15c0b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.251+0000 7f47225ff700 1 --2- 172.21.15.23:0/1243653502 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f471c0c4a30 0x7f471c0cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.251+0000 7f47225ff700 1 -- 172.21.15.23:0/1243653502 >> 172.21.15.23:0/1243653502 conn(0x7f471c065b90 msgr2=0x7f471c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.251+0000 7f47225ff700 1 -- 172.21.15.23:0/1243653502 shutdown_connections 2023-06-22T08:16:36.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.251+0000 7f47225ff700 1 -- 172.21.15.23:0/1243653502 wait complete. 2023-06-22T08:16:36.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.253+0000 7f47225ff700 1 Processor -- start 2023-06-22T08:16:36.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.253+0000 7f47225ff700 1 -- start start 2023-06-22T08:16:36.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f47225ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0c4a30 0x7f471c1495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:36.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f47225ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f471c0cb7a0 0x7f471c15aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f47225ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c152a10 0x7f471c1f04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f47225ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c1520f0 con 0x7f471c0c4a30 2023-06-22T08:16:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f47225ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c1599b0 con 0x7f471c152a10 2023-06-22T08:16:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f47225ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c159c70 con 0x7f471c0cb7a0 2023-06-22T08:16:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f471bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0c4a30 0x7f471c1495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f471bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0c4a30 0x7f471c1495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34772/0 (socket says 172.21.15.23:34772) 2023-06-22T08:16:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f471bfff700 1 -- 172.21.15.23:0/1741341736 learned_addr learned my addr 172.21.15.23:0/1741341736 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f471b7fe700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f471c0cb7a0 0x7f471c15aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f4720b9c700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c152a10 0x7f471c1f04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f471bfff700 1 -- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f471c0cb7a0 msgr2=0x7f471c15aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f471bfff700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f471c0cb7a0 0x7f471c15aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f471bfff700 1 -- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c152a10 msgr2=0x7f471c1f04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f471bfff700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c152a10 0x7f471c1f04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.254+0000 7f471bfff700 1 -- 172.21.15.23:0/1741341736 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4714009a70 con 0x7f471c0c4a30 2023-06-22T08:16:36.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.255+0000 7f471bfff700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0c4a30 0x7f471c1495d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f470c00c480 tx=0x7f470c00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:36.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.255+0000 7f4718ff9700 1 -- 172.21.15.23:0/1741341736 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c005d70 con 0x7f471c0c4a30 2023-06-22T08:16:36.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.255+0000 7f4718ff9700 1 -- 172.21.15.23:0/1741341736 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c00ce90 con 0x7f471c0c4a30 2023-06-22T08:16:36.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.255+0000 7f47225ff700 1 -- 172.21.15.23:0/1741341736 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c1efa00 con 0x7f471c0c4a30 2023-06-22T08:16:36.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.255+0000 7f4718ff9700 1 -- 172.21.15.23:0/1741341736 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c0072d0 con 0x7f471c0c4a30 2023-06-22T08:16:36.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.255+0000 7f47225ff700 1 -- 172.21.15.23:0/1741341736 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c1effc0 con 0x7f471c0c4a30 2023-06-22T08:16:36.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.256+0000 7f47225ff700 1 -- 172.21.15.23:0/1741341736 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c1ef860 con 0x7f471c0c4a30 2023-06-22T08:16:36.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.257+0000 7f4718ff9700 1 -- 172.21.15.23:0/1741341736 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f470c007430 con 0x7f471c0c4a30 2023-06-22T08:16:36.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.258+0000 7f4718ff9700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f470806ace0 0x7f470806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:36.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.258+0000 7f4718ff9700 1 -- 172.21.15.23:0/1741341736 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f470c088c00 con 0x7f471c0c4a30 2023-06-22T08:16:36.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.258+0000 7f471b7fe700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f470806ace0 0x7f470806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:36.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.259+0000 7f471b7fe700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f470806ace0 0x7f470806d190 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f471c0ce450 tx=0x7f4710009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:36.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.262+0000 7f4718ff9700 1 -- 172.21.15.23:0/1741341736 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f471c1ef860 con 0x7f471c0c4a30 2023-06-22T08:16:36.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.515+0000 7f47225ff700 1 -- 172.21.15.23:0/1741341736 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f471c1ef860 con 0x7f471c0c4a30 2023-06-22T08:16:36.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.516+0000 7f4718ff9700 1 -- 172.21.15.23:0/1741341736 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+5510 (secure 0 0 0) 0x7f471c1ef860 con 0x7f471c0c4a30 2023-06-22T08:16:36.519 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:36.519 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:32.933102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:resolve","state_seq":30,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:36.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 -- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f470806ace0 msgr2=0x7f470806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:36.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f470806ace0 0x7f470806d190 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f471c0ce450 tx=0x7f4710009210 comp rx=0 tx=0).stop 2023-06-22T08:16:36.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 -- 172.21.15.23:0/1741341736 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0c4a30 msgr2=0x7f471c1495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:36.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0c4a30 0x7f471c1495d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f470c00c480 tx=0x7f470c00c790 comp rx=0 tx=0).stop 2023-06-22T08:16:36.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 -- 172.21.15.23:0/1741341736 shutdown_connections 2023-06-22T08:16:36.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f470806ace0 0x7f470806d190 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:36.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f471c152a10 0x7f471c1f04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:36.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f471c0c4a30 0x7f471c1495d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:36.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 --2- 172.21.15.23:0/1741341736 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f471c0cb7a0 0x7f471c15aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:36.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 -- 172.21.15.23:0/1741341736 >> 172.21.15.23:0/1741341736 conn(0x7f471c065b90 msgr2=0x7f471c06b740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:36.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 -- 172.21.15.23:0/1741341736 shutdown_connections 2023-06-22T08:16:36.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:36.518+0000 7f47225ff700 1 -- 172.21.15.23:0/1741341736 wait complete. 2023-06-22T08:16:36.523 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 133 2023-06-22T08:16:36.540 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:16:38.541 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:38.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.746+0000 7fc3fe1ff700 1 Processor -- start 2023-06-22T08:16:38.750 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.747+0000 7fc3fe1ff700 1 -- start start 2023-06-22T08:16:38.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.748+0000 7fc3fe1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8069eb0 0x7fc3f806a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:38.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.748+0000 7fc3fe1ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f80703c0 con 0x7fc3f80c4e00 2023-06-22T08:16:38.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.748+0000 7fc3fe1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f8070680 con 0x7fc3f8071350 2023-06-22T08:16:38.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.748+0000 7fc3fe1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f806a7c0 con 0x7fc3f8069eb0 2023-06-22T08:16:38.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.748+0000 7fc3f7fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc3f80c4e00 0x7fc3f8163120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52854/0 (socket says 172.21.15.23:52854) 2023-06-22T08:16:38.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.748+0000 7fc3f7fff700 1 -- 172.21.15.23:0/86924752 learned_addr learned my addr 172.21.15.23:0/86924752 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:38.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.748+0000 7fc3f77fe700 1 --2- 172.21.15.23:0/86924752 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8069eb0 0x7fc3f806a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f67fc700 1 -- 172.21.15.23:0/86924752 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809095686 0 0) 0x7fc3f80703c0 con 0x7fc3f80c4e00 2023-06-22T08:16:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f67fc700 1 -- 172.21.15.23:0/86924752 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e00035b0 con 0x7fc3f80c4e00 2023-06-22T08:16:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f67fc700 1 -- 172.21.15.23:0/86924752 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370303406 0 0) 0x7fc3f8070680 con 0x7fc3f8071350 2023-06-22T08:16:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f67fc700 1 -- 172.21.15.23:0/86924752 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f80703c0 con 0x7fc3f8071350 2023-06-22T08:16:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f77fe700 1 -- 172.21.15.23:0/86924752 >> v1:172.21.15.67:6789/0 conn(0x7fc3f8071350 legacy=0x7fc3f806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f77fe700 1 -- 172.21.15.23:0/86924752 >> v1:172.21.15.23:6789/0 conn(0x7fc3f80c4e00 legacy=0x7fc3f8163120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f77fe700 1 -- 172.21.15.23:0/86924752 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f8158990 con 0x7fc3f8069eb0 2023-06-22T08:16:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f77fe700 1 --2- 172.21.15.23:0/86924752 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8069eb0 0x7fc3f806a280 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8009870 tx=0x7fc3e8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=215eb36e1b31cd53 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f67fc700 1 -- 172.21.15.23:0/86924752 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8011830 con 0x7fc3f8069eb0 2023-06-22T08:16:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f67fc700 1 -- 172.21.15.23:0/86924752 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e8011990 con 0x7fc3f8069eb0 2023-06-22T08:16:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.749+0000 7fc3f67fc700 1 -- 172.21.15.23:0/86924752 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8011d20 con 0x7fc3f8069eb0 2023-06-22T08:16:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.750+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/86924752 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8069eb0 msgr2=0x7fc3f806a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.750+0000 7fc3fe1ff700 1 --2- 172.21.15.23:0/86924752 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8069eb0 0x7fc3f806a280 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8009870 tx=0x7fc3e8009b80 comp rx=0 tx=0).stop 2023-06-22T08:16:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.750+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/86924752 shutdown_connections 2023-06-22T08:16:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.750+0000 7fc3fe1ff700 1 --2- 172.21.15.23:0/86924752 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8069eb0 0x7fc3f806a280 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.750+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/86924752 >> 172.21.15.23:0/86924752 conn(0x7fc3f80ca3f0 msgr2=0x7fc3f80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.750+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/86924752 shutdown_connections 2023-06-22T08:16:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.750+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/86924752 wait complete. 2023-06-22T08:16:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.752+0000 7fc3fe1ff700 1 Processor -- start 2023-06-22T08:16:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.752+0000 7fc3fe1ff700 1 -- start start 2023-06-22T08:16:38.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3fe1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f8069eb0 0x7fc3f814ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:38.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3fe1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8071350 0x7fc3f80c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:38.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3fe1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3f80c4e00 0x7fc3f80c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:38.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3fe1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f8156220 con 0x7fc3f80c4e00 2023-06-22T08:16:38.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3fe1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f81ee900 con 0x7fc3f8069eb0 2023-06-22T08:16:38.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3fe1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f81eebe0 con 0x7fc3f8071350 2023-06-22T08:16:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3f7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3f80c4e00 0x7fc3f80c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3f7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3f80c4e00 0x7fc3f80c9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34776/0 (socket says 172.21.15.23:34776) 2023-06-22T08:16:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3f7fff700 1 -- 172.21.15.23:0/3095029583 learned_addr learned my addr 172.21.15.23:0/3095029583 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3f77fe700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f8069eb0 0x7fc3f814ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3f6ffd700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8071350 0x7fc3f80c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3f7fff700 1 -- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8071350 msgr2=0x7fc3f80c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:38.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3f7fff700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8071350 0x7fc3f80c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:38.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3f7fff700 1 -- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f8069eb0 msgr2=0x7fc3f814ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:38.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3f7fff700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f8069eb0 0x7fc3f814ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:38.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.753+0000 7fc3f7fff700 1 -- 172.21.15.23:0/3095029583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80094a0 con 0x7fc3f80c4e00 2023-06-22T08:16:38.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.754+0000 7fc3f7fff700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3f80c4e00 0x7fc3f80c9890 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000cb40 tx=0x7fc3f000ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:38.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.754+0000 7fc3dffff700 1 -- 172.21.15.23:0/3095029583 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f000a9e0 con 0x7fc3f80c4e00 2023-06-22T08:16:38.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.754+0000 7fc3dffff700 1 -- 172.21.15.23:0/3095029583 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f000ab40 con 0x7fc3f80c4e00 2023-06-22T08:16:38.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.754+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/3095029583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f81eeec0 con 0x7fc3f80c4e00 2023-06-22T08:16:38.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.754+0000 7fc3dffff700 1 -- 172.21.15.23:0/3095029583 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0012400 con 0x7fc3f80c4e00 2023-06-22T08:16:38.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.754+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/3095029583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f81ef480 con 0x7fc3f80c4e00 2023-06-22T08:16:38.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.755+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/3095029583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f804ea20 con 0x7fc3f80c4e00 2023-06-22T08:16:38.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.756+0000 7fc3dffff700 1 -- 172.21.15.23:0/3095029583 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc3f001b3b0 con 0x7fc3f80c4e00 2023-06-22T08:16:38.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.757+0000 7fc3dffff700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc3e406ace0 0x7fc3e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:38.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.757+0000 7fc3dffff700 1 -- 172.21.15.23:0/3095029583 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7fc3f0089890 con 0x7fc3f80c4e00 2023-06-22T08:16:38.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.757+0000 7fc3f77fe700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc3e406ace0 0x7fc3e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:38.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.758+0000 7fc3f77fe700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc3e406ace0 0x7fc3e406d190 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80097c0 tx=0x7fc3e801c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:38.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:38.761+0000 7fc3dffff700 1 -- 172.21.15.23:0/3095029583 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc3f0054210 con 0x7fc3f80c4e00 2023-06-22T08:16:39.014 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.011+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/3095029583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3f80c3c70 con 0x7fc3f80c4e00 2023-06-22T08:16:39.015 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.012+0000 7fc3dffff700 1 -- 172.21.15.23:0/3095029583 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5512 (secure 0 0 0) 0x7fc3f0017090 con 0x7fc3f80c4e00 2023-06-22T08:16:39.015 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:39.016 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:38.541232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc3e406ace0 msgr2=0x7fc3e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc3e406ace0 0x7fc3e406d190 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80097c0 tx=0x7fc3e801c040 comp rx=0 tx=0).stop 2023-06-22T08:16:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/3095029583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3f80c4e00 msgr2=0x7fc3f80c9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3f80c4e00 0x7fc3f80c9890 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc3f000cb40 tx=0x7fc3f000ce50 comp rx=0 tx=0).stop 2023-06-22T08:16:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/3095029583 shutdown_connections 2023-06-22T08:16:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3f80c4e00 0x7fc3f80c9890 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc3e406ace0 0x7fc3e406d190 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3f8069eb0 0x7fc3f814ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 --2- 172.21.15.23:0/3095029583 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3f8071350 0x7fc3f80c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/3095029583 >> 172.21.15.23:0/3095029583 conn(0x7fc3f80ca3f0 msgr2=0x7fc3f806d530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/3095029583 shutdown_connections 2023-06-22T08:16:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:39.014+0000 7fc3fe1ff700 1 -- 172.21.15.23:0/3095029583 wait complete. 2023-06-22T08:16:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 134 2023-06-22T08:16:39.036 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 134, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 134, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5576": { "addr": "172.21.15.67:6837/2064462140", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 2064462140, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 2064462140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5576, "incarnation": 109, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 15 }, "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 }, "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:reconnect", "state_seq": 32 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:16:38.541232+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 6313, "mds_3": 5995, "mds_4": 5576 } } } ], "standbys": [] } 2023-06-22T08:16:41.037 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:41.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.244+0000 7f3f29bff700 1 Processor -- start 2023-06-22T08:16:41.247 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.244+0000 7f3f29bff700 1 -- start start 2023-06-22T08:16:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.245+0000 7f3f29bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3f2415a320 0x7f3f240c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f29bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f240c6a60 con 0x7f3f240c3a40 2023-06-22T08:16:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f29bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2414d690 con 0x7f3f2406d050 2023-06-22T08:16:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f29bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f24157570 con 0x7f3f2415a320 2023-06-22T08:16:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f22ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3f2415a320 0x7f3f240c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f237fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f3f240c3a40 0x7f3f24159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52868/0 (socket says 172.21.15.23:52868) 2023-06-22T08:16:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f22ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3f2415a320 0x7f3f240c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34778/0 (socket says 172.21.15.23:34778) 2023-06-22T08:16:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f237fe700 1 -- 172.21.15.23:0/4237514846 learned_addr learned my addr 172.21.15.23:0/4237514846 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f22ffd700 1 -- 172.21.15.23:0/4237514846 >> v1:172.21.15.67:6789/0 conn(0x7f3f2406d050 legacy=0x7f3f24160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f22ffd700 1 -- 172.21.15.23:0/4237514846 >> v1:172.21.15.23:6789/0 conn(0x7f3f240c3a40 legacy=0x7f3f24159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f22ffd700 1 -- 172.21.15.23:0/4237514846 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f240c6430 con 0x7f3f2415a320 2023-06-22T08:16:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.246+0000 7f3f227fc700 1 -- 172.21.15.23:0/4237514846 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063604458 0 0) 0x7f3f240c6a60 con 0x7f3f240c3a40 2023-06-22T08:16:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.247+0000 7f3f22ffd700 1 --2- 172.21.15.23:0/4237514846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 0x7f3f240c3e10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3f18009840 tx=0x7f3f18009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84b52ba911ec7c3 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.247+0000 7f3f227fc700 1 -- 172.21.15.23:0/4237514846 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18011830 con 0x7f3f2415a320 2023-06-22T08:16:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.247+0000 7f3f227fc700 1 -- 172.21.15.23:0/4237514846 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f18011990 con 0x7f3f2415a320 2023-06-22T08:16:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.247+0000 7f3f227fc700 1 -- 172.21.15.23:0/4237514846 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18011d20 con 0x7f3f2415a320 2023-06-22T08:16:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.247+0000 7f3f29bff700 1 -- 172.21.15.23:0/4237514846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 msgr2=0x7f3f240c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.247+0000 7f3f29bff700 1 --2- 172.21.15.23:0/4237514846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 0x7f3f240c3e10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3f18009840 tx=0x7f3f18009b50 comp rx=0 tx=0).stop 2023-06-22T08:16:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.248+0000 7f3f29bff700 1 -- 172.21.15.23:0/4237514846 shutdown_connections 2023-06-22T08:16:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.248+0000 7f3f29bff700 1 --2- 172.21.15.23:0/4237514846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 0x7f3f240c3e10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.248+0000 7f3f29bff700 1 -- 172.21.15.23:0/4237514846 >> 172.21.15.23:0/4237514846 conn(0x7f3f240ca3f0 msgr2=0x7f3f240ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.248+0000 7f3f29bff700 1 -- 172.21.15.23:0/4237514846 shutdown_connections 2023-06-22T08:16:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.248+0000 7f3f29bff700 1 -- 172.21.15.23:0/4237514846 wait complete. 2023-06-22T08:16:41.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.250+0000 7f3f29bff700 1 Processor -- start 2023-06-22T08:16:41.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.250+0000 7f3f29bff700 1 -- start start 2023-06-22T08:16:41.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.251+0000 7f3f29bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f2406d050 0x7f3f241583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:41.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.251+0000 7f3f29bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f240c3a40 0x7f3f241588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:41.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.251+0000 7f3f29bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 0x7f3f240c9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:41.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.251+0000 7f3f29bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f241e1e50 con 0x7f3f2415a320 2023-06-22T08:16:41.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.251+0000 7f3f29bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f240ca010 con 0x7f3f2406d050 2023-06-22T08:16:41.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.251+0000 7f3f29bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f241f25f0 con 0x7f3f240c3a40 2023-06-22T08:16:41.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f23fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 0x7f3f240c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:41.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f22ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f240c3a40 0x7f3f241588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:41.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f23fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 0x7f3f240c9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34794/0 (socket says 172.21.15.23:34794) 2023-06-22T08:16:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f23fff700 1 -- 172.21.15.23:0/1016240877 learned_addr learned my addr 172.21.15.23:0/1016240877 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f237fe700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f2406d050 0x7f3f241583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f23fff700 1 -- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f240c3a40 msgr2=0x7f3f241588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f23fff700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f240c3a40 0x7f3f241588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f23fff700 1 -- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f2406d050 msgr2=0x7f3f241583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f23fff700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f2406d050 0x7f3f241583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f23fff700 1 -- 172.21.15.23:0/1016240877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180094a0 con 0x7f3f2415a320 2023-06-22T08:16:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.252+0000 7f3f23fff700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 0x7f3f240c9030 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00cc80 tx=0x7f3f1c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:41.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:16:41.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.253+0000 7f3f0bfff700 1 -- 172.21.15.23:0/1016240877 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c003c00 con 0x7f3f2415a320 2023-06-22T08:16:41.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.253+0000 7f3f0bfff700 1 -- 172.21.15.23:0/1016240877 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1c003d60 con 0x7f3f2415a320 2023-06-22T08:16:41.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.253+0000 7f3f29bff700 1 -- 172.21.15.23:0/1016240877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f241f28d0 con 0x7f3f2415a320 2023-06-22T08:16:41.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.253+0000 7f3f0bfff700 1 -- 172.21.15.23:0/1016240877 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c00aaa0 con 0x7f3f2415a320 2023-06-22T08:16:41.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.253+0000 7f3f29bff700 1 -- 172.21.15.23:0/1016240877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f240c8550 con 0x7f3f2415a320 2023-06-22T08:16:41.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.254+0000 7f3f29bff700 1 -- 172.21.15.23:0/1016240877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2414ba20 con 0x7f3f2415a320 2023-06-22T08:16:41.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.255+0000 7f3f0bfff700 1 -- 172.21.15.23:0/1016240877 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3f1c00ac00 con 0x7f3f2415a320 2023-06-22T08:16:41.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.256+0000 7f3f0bfff700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f1006ad10 0x7f3f1006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:41.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.256+0000 7f3f0bfff700 1 -- 172.21.15.23:0/1016240877 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f3f1c089ab0 con 0x7f3f2415a320 2023-06-22T08:16:41.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.256+0000 7f3f237fe700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f1006ad10 0x7f3f1006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:41.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.257+0000 7f3f237fe700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f1006ad10 0x7f3f1006d1c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3f14007b10 tx=0x7f3f14007a60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:41.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:16:41.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.260+0000 7f3f0bfff700 1 -- 172.21.15.23:0/1016240877 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3f1c0544c0 con 0x7f3f2415a320 2023-06-22T08:16:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.515+0000 7f3f29bff700 1 -- 172.21.15.23:0/1016240877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f2404ea20 con 0x7f3f2415a320 2023-06-22T08:16:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.516+0000 7f3f0bfff700 1 -- 172.21.15.23:0/1016240877 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5512 (secure 0 0 0) 0x7f3f1c057ae0 con 0x7f3f2415a320 2023-06-22T08:16:41.520 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:41.520 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:38.541232+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 -- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f1006ad10 msgr2=0x7f3f1006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f1006ad10 0x7f3f1006d1c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3f14007b10 tx=0x7f3f14007a60 comp rx=0 tx=0).stop 2023-06-22T08:16:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 -- 172.21.15.23:0/1016240877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 msgr2=0x7f3f240c9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 0x7f3f240c9030 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00cc80 tx=0x7f3f1c00cf90 comp rx=0 tx=0).stop 2023-06-22T08:16:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 -- 172.21.15.23:0/1016240877 shutdown_connections 2023-06-22T08:16:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2415a320 0x7f3f240c9030 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f1006ad10 0x7f3f1006d1c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f2406d050 0x7f3f241583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 --2- 172.21.15.23:0/1016240877 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3f240c3a40 0x7f3f241588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 -- 172.21.15.23:0/1016240877 >> 172.21.15.23:0/1016240877 conn(0x7f3f240ca3f0 msgr2=0x7f3f241498c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 -- 172.21.15.23:0/1016240877 shutdown_connections 2023-06-22T08:16:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:41.518+0000 7f3f29bff700 1 -- 172.21.15.23:0/1016240877 wait complete. 2023-06-22T08:16:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 134 2023-06-22T08:16:41.539 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:16:43.541 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:43.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.750+0000 7f36831ff700 1 Processor -- start 2023-06-22T08:16:43.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.750+0000 7f36831ff700 1 -- start start 2023-06-22T08:16:43.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.751+0000 7f36831ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f367c0c3a40 0x7f367c0c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:43.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f36831ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c15a320 0x7f367c160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:43.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f36831ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367c0c6a60 con 0x7f367c06d050 2023-06-22T08:16:43.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f36831ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367c157570 con 0x7f367c0c3a40 2023-06-22T08:16:43.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f36831ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0c4350 con 0x7f367c15a320 2023-06-22T08:16:43.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f3680f9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f367c0c3a40 0x7f367c0c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:43.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f3680f9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f367c0c3a40 0x7f367c0c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34810/0 (socket says 172.21.15.23:34810) 2023-06-22T08:16:43.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f3680f9b700 1 -- 172.21.15.23:0/2914668067 learned_addr learned my addr 172.21.15.23:0/2914668067 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:43.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f368179c700 1 --2- 172.21.15.23:0/2914668067 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c15a320 0x7f367c160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:43.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f3680f9b700 1 -- 172.21.15.23:0/2914668067 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c15a320 msgr2=0x7f367c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:43.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f3680f9b700 1 --2- 172.21.15.23:0/2914668067 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c15a320 0x7f367c160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:43.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f3680f9b700 1 -- 172.21.15.23:0/2914668067 >> v1:172.21.15.67:6789/0 conn(0x7f367c06d050 legacy=0x7f367c159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:43.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.752+0000 7f3680f9b700 1 -- 172.21.15.23:0/2914668067 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c14d690 con 0x7f367c0c3a40 2023-06-22T08:16:43.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.753+0000 7f3680f9b700 1 --2- 172.21.15.23:0/2914668067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c0c3a40 0x7f367c0c3e10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f36700098f0 tx=0x7f3670009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4943dce167a68f6f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:43.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.753+0000 7f367b7fe700 1 -- 172.21.15.23:0/2914668067 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670011830 con 0x7f367c0c3a40 2023-06-22T08:16:43.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.753+0000 7f367b7fe700 1 -- 172.21.15.23:0/2914668067 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3670011990 con 0x7f367c0c3a40 2023-06-22T08:16:43.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.753+0000 7f367b7fe700 1 -- 172.21.15.23:0/2914668067 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670011d20 con 0x7f367c0c3a40 2023-06-22T08:16:43.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.753+0000 7f36831ff700 1 -- 172.21.15.23:0/2914668067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c0c3a40 msgr2=0x7f367c0c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:43.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.753+0000 7f36831ff700 1 --2- 172.21.15.23:0/2914668067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c0c3a40 0x7f367c0c3e10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f36700098f0 tx=0x7f3670009c00 comp rx=0 tx=0).stop 2023-06-22T08:16:43.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.753+0000 7f36831ff700 1 -- 172.21.15.23:0/2914668067 shutdown_connections 2023-06-22T08:16:43.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.753+0000 7f36831ff700 1 --2- 172.21.15.23:0/2914668067 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c15a320 0x7f367c160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:43.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.753+0000 7f36831ff700 1 --2- 172.21.15.23:0/2914668067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c0c3a40 0x7f367c0c3e10 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:43.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.753+0000 7f36831ff700 1 -- 172.21.15.23:0/2914668067 >> 172.21.15.23:0/2914668067 conn(0x7f367c0ca3f0 msgr2=0x7f367c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:43.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.754+0000 7f36831ff700 1 -- 172.21.15.23:0/2914668067 shutdown_connections 2023-06-22T08:16:43.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.754+0000 7f36831ff700 1 -- 172.21.15.23:0/2914668067 wait complete. 2023-06-22T08:16:43.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.755+0000 7f36831ff700 1 Processor -- start 2023-06-22T08:16:43.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.755+0000 7f36831ff700 1 -- start start 2023-06-22T08:16:43.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.756+0000 7f36831ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c06d050 0x7f367c1eefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:43.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.756+0000 7f36831ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c0c3a40 0x7f367c0c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:43.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.756+0000 7f36831ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f367c15a320 0x7f367c0c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:43.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.756+0000 7f36831ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c1ef4f0 con 0x7f367c06d050 2023-06-22T08:16:43.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.756+0000 7f36831ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c14ea00 con 0x7f367c15a320 2023-06-22T08:16:43.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.756+0000 7f36831ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c14ee90 con 0x7f367c0c3a40 2023-06-22T08:16:43.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f3680f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c06d050 0x7f367c1eefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:43.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f3680f9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c06d050 0x7f367c1eefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34822/0 (socket says 172.21.15.23:34822) 2023-06-22T08:16:43.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f3680f9b700 1 -- 172.21.15.23:0/1123198083 learned_addr learned my addr 172.21.15.23:0/1123198083 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:43.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f367bfff700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c0c3a40 0x7f367c0c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:43.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f368179c700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f367c15a320 0x7f367c0c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:43.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f3680f9b700 1 -- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c0c3a40 msgr2=0x7f367c0c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:43.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f3680f9b700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c0c3a40 0x7f367c0c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:43.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f3680f9b700 1 -- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f367c15a320 msgr2=0x7f367c0c81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:43.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f3680f9b700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f367c15a320 0x7f367c0c81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:43.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f3680f9b700 1 -- 172.21.15.23:0/1123198083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670009550 con 0x7f367c06d050 2023-06-22T08:16:43.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f367bfff700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c0c3a40 0x7f367c0c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:16:43.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f3680f9b700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c06d050 0x7f367c1eefb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f3670009c40 tx=0x7f3670004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:43.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.757+0000 7f36797fa700 1 -- 172.21.15.23:0/1123198083 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670004570 con 0x7f367c06d050 2023-06-22T08:16:43.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.758+0000 7f36797fa700 1 -- 172.21.15.23:0/1123198083 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367001f070 con 0x7f367c06d050 2023-06-22T08:16:43.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.758+0000 7f36797fa700 1 -- 172.21.15.23:0/1123198083 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36700038e0 con 0x7f367c06d050 2023-06-22T08:16:43.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.758+0000 7f36831ff700 1 -- 172.21.15.23:0/1123198083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367c150980 con 0x7f367c06d050 2023-06-22T08:16:43.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.758+0000 7f36831ff700 1 -- 172.21.15.23:0/1123198083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367c14e3b0 con 0x7f367c06d050 2023-06-22T08:16:43.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.759+0000 7f36831ff700 1 -- 172.21.15.23:0/1123198083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367c04ea20 con 0x7f367c06d050 2023-06-22T08:16:43.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.760+0000 7f36797fa700 1 -- 172.21.15.23:0/1123198083 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3670021030 con 0x7f367c06d050 2023-06-22T08:16:43.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.761+0000 7f36797fa700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f366006ad10 0x7f366006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:43.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.761+0000 7f36797fa700 1 -- 172.21.15.23:0/1123198083 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f367008af10 con 0x7f367c06d050 2023-06-22T08:16:43.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.761+0000 7f367bfff700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f366006ad10 0x7f366006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:43.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.762+0000 7f367bfff700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f366006ad10 0x7f366006d1c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f367c1e4400 tx=0x7f3668009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:43.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:43.765+0000 7f36797fa700 1 -- 172.21.15.23:0/1123198083 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3670055920 con 0x7f367c06d050 2023-06-22T08:16:44.022 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.019+0000 7f36831ff700 1 -- 172.21.15.23:0/1123198083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f367c0c5370 con 0x7f367c06d050 2023-06-22T08:16:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.020+0000 7f36797fa700 1 -- 172.21.15.23:0/1123198083 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5509 (secure 0 0 0) 0x7f3670015090 con 0x7f367c06d050 2023-06-22T08:16:44.023 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:44.023 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:43.914799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:44.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 -- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f366006ad10 msgr2=0x7f366006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:44.025 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f366006ad10 0x7f366006d1c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f367c1e4400 tx=0x7f3668009040 comp rx=0 tx=0).stop 2023-06-22T08:16:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 -- 172.21.15.23:0/1123198083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c06d050 msgr2=0x7f367c1eefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c06d050 0x7f367c1eefb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f3670009c40 tx=0x7f3670004750 comp rx=0 tx=0).stop 2023-06-22T08:16:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 -- 172.21.15.23:0/1123198083 shutdown_connections 2023-06-22T08:16:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f366006ad10 0x7f366006d1c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f367c15a320 0x7f367c0c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f367c06d050 0x7f367c1eefb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 --2- 172.21.15.23:0/1123198083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f367c0c3a40 0x7f367c0c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 -- 172.21.15.23:0/1123198083 >> 172.21.15.23:0/1123198083 conn(0x7f367c0ca3f0 msgr2=0x7f367c070100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 -- 172.21.15.23:0/1123198083 shutdown_connections 2023-06-22T08:16:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:44.022+0000 7f36831ff700 1 -- 172.21.15.23:0/1123198083 wait complete. 2023-06-22T08:16:44.027 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 135 2023-06-22T08:16:44.044 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:16:46.045 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:46.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.251+0000 7fc9511ff700 1 Processor -- start 2023-06-22T08:16:46.254 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.251+0000 7fc9511ff700 1 -- start start 2023-06-22T08:16:46.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.252+0000 7fc9511ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc94c0c97d0 0x7fc94c159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:46.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.252+0000 7fc9511ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94c0c9c30 con 0x7fc94c0c45f0 2023-06-22T08:16:46.255 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.252+0000 7fc9511ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc94c15bea0 con 0x7fc94c152d70 2023-06-22T08:16:46.256 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.252+0000 7fc9511ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc94c0c95e0 con 0x7fc94c0c97d0 2023-06-22T08:16:46.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.252+0000 7fc94a59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc94c0c97d0 0x7fc94c159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:46.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.253+0000 7fc94b59e700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc94c0c45f0 0x7fc94c0c4a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:49266/0 (socket says 172.21.15.23:49266) 2023-06-22T08:16:46.257 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.253+0000 7fc94ad9d700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc94c152d70 0x7fc94c0cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:32924/0 (socket says 172.21.15.23:32924) 2023-06-22T08:16:46.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.253+0000 7fc94b59e700 1 -- 172.21.15.23:0/2794011435 learned_addr learned my addr 172.21.15.23:0/2794011435 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:46.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.253+0000 7fc94a59c700 1 --2- 172.21.15.23:0/2794011435 >> v2:172.21.15.23:3300/0 conn(0x7fc94c0c97d0 0x7fc94c159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54992/0 (socket says 172.21.15.23:54992) 2023-06-22T08:16:46.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.253+0000 7fc94a59c700 1 -- 172.21.15.23:0/2794011435 >> v1:172.21.15.67:6789/0 conn(0x7fc94c152d70 legacy=0x7fc94c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:46.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.253+0000 7fc94a59c700 1 -- 172.21.15.23:0/2794011435 >> v1:172.21.15.23:6789/0 conn(0x7fc94c0c45f0 legacy=0x7fc94c0c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:46.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.253+0000 7fc949d9b700 1 -- 172.21.15.23:0/2794011435 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317042288 0 0) 0x7fc94c15bea0 con 0x7fc94c152d70 2023-06-22T08:16:46.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.253+0000 7fc94a59c700 1 -- 172.21.15.23:0/2794011435 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c15c160 con 0x7fc94c0c97d0 2023-06-22T08:16:46.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.253+0000 7fc94a59c700 1 --2- 172.21.15.23:0/2794011435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c97d0 0x7fc94c159330 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc940009840 tx=0x7fc940009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a6ad74aef429bd server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:46.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.253+0000 7fc949d9b700 1 -- 172.21.15.23:0/2794011435 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940004230 con 0x7fc94c0c97d0 2023-06-22T08:16:46.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.254+0000 7fc949d9b700 1 -- 172.21.15.23:0/2794011435 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc940004390 con 0x7fc94c0c97d0 2023-06-22T08:16:46.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.254+0000 7fc949d9b700 1 -- 172.21.15.23:0/2794011435 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9400036a0 con 0x7fc94c0c97d0 2023-06-22T08:16:46.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.254+0000 7fc9511ff700 1 -- 172.21.15.23:0/2794011435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c97d0 msgr2=0x7fc94c159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:46.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.254+0000 7fc9511ff700 1 --2- 172.21.15.23:0/2794011435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c97d0 0x7fc94c159330 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc940009840 tx=0x7fc940009b50 comp rx=0 tx=0).stop 2023-06-22T08:16:46.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.254+0000 7fc9511ff700 1 -- 172.21.15.23:0/2794011435 shutdown_connections 2023-06-22T08:16:46.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.254+0000 7fc9511ff700 1 --2- 172.21.15.23:0/2794011435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c97d0 0x7fc94c159330 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:46.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.254+0000 7fc9511ff700 1 -- 172.21.15.23:0/2794011435 >> 172.21.15.23:0/2794011435 conn(0x7fc94c06ae30 msgr2=0x7fc94c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:46.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.254+0000 7fc9511ff700 1 -- 172.21.15.23:0/2794011435 shutdown_connections 2023-06-22T08:16:46.260 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.254+0000 7fc9511ff700 1 -- 172.21.15.23:0/2794011435 wait complete. 2023-06-22T08:16:46.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.256+0000 7fc9511ff700 1 Processor -- start 2023-06-22T08:16:46.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.257+0000 7fc9511ff700 1 -- start start 2023-06-22T08:16:46.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc9511ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c45f0 0x7fc94c1f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:46.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc9511ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc94c0c97d0 0x7fc94c0b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:46.261 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc9511ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc94c152d70 0x7fc94c0cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:46.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc9511ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c0b42b0 con 0x7fc94c0c45f0 2023-06-22T08:16:46.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc9511ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c1f2850 con 0x7fc94c152d70 2023-06-22T08:16:46.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc9511ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94c0b2590 con 0x7fc94c0c97d0 2023-06-22T08:16:46.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc94ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c45f0 0x7fc94c1f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:46.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc94ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c45f0 0x7fc94c1f2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54994/0 (socket says 172.21.15.23:54994) 2023-06-22T08:16:46.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc94ad9d700 1 -- 172.21.15.23:0/629140063 learned_addr learned my addr 172.21.15.23:0/629140063 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:46.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc94a59c700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc94c0c97d0 0x7fc94c0b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:46.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.258+0000 7fc94b59e700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc94c152d70 0x7fc94c0cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:46.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc94ad9d700 1 -- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc94c0c97d0 msgr2=0x7fc94c0b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:46.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc94ad9d700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc94c0c97d0 0x7fc94c0b2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:46.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc94ad9d700 1 -- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc94c152d70 msgr2=0x7fc94c0cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:46.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc94ad9d700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc94c152d70 0x7fc94c0cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:46.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc94ad9d700 1 -- 172.21.15.23:0/629140063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9400094a0 con 0x7fc94c0c45f0 2023-06-22T08:16:46.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc94ad9d700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c45f0 0x7fc94c1f2310 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00c550 tx=0x7fc93c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:46.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc9337fe700 1 -- 172.21.15.23:0/629140063 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c004c80 con 0x7fc94c0c45f0 2023-06-22T08:16:46.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc9337fe700 1 -- 172.21.15.23:0/629140063 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c004e00 con 0x7fc94c0c45f0 2023-06-22T08:16:46.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc9511ff700 1 -- 172.21.15.23:0/629140063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94c1ee390 con 0x7fc94c0c45f0 2023-06-22T08:16:46.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc9337fe700 1 -- 172.21.15.23:0/629140063 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c007310 con 0x7fc94c0c45f0 2023-06-22T08:16:46.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.259+0000 7fc9511ff700 1 -- 172.21.15.23:0/629140063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94c1ee650 con 0x7fc94c0c45f0 2023-06-22T08:16:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.260+0000 7fc9511ff700 1 -- 172.21.15.23:0/629140063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94c04ea20 con 0x7fc94c0c45f0 2023-06-22T08:16:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.262+0000 7fc9337fe700 1 -- 172.21.15.23:0/629140063 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc93c007470 con 0x7fc94c0c45f0 2023-06-22T08:16:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.262+0000 7fc9337fe700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc93806ace0 0x7fc93806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.263+0000 7fc9337fe700 1 -- 172.21.15.23:0/629140063 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7fc93c0897c0 con 0x7fc94c0c45f0 2023-06-22T08:16:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.263+0000 7fc94a59c700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc93806ace0 0x7fc93806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.264+0000 7fc94a59c700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc93806ace0 0x7fc93806d190 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc9400090b0 tx=0x7fc940008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:46.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.268+0000 7fc9337fe700 1 -- 172.21.15.23:0/629140063 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc93c054140 con 0x7fc94c0c45f0 2023-06-22T08:16:46.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.519+0000 7fc9511ff700 1 -- 172.21.15.23:0/629140063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc94c06ac10 con 0x7fc94c0c45f0 2023-06-22T08:16:46.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.520+0000 7fc9337fe700 1 -- 172.21.15.23:0/629140063 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5509 (secure 0 0 0) 0x7fc93c057760 con 0x7fc94c0c45f0 2023-06-22T08:16:46.523 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:46.523 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:43.914799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 -- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc93806ace0 msgr2=0x7fc93806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:46.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc93806ace0 0x7fc93806d190 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc9400090b0 tx=0x7fc940008030 comp rx=0 tx=0).stop 2023-06-22T08:16:46.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 -- 172.21.15.23:0/629140063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c45f0 msgr2=0x7fc94c1f2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:46.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c45f0 0x7fc94c1f2310 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00c550 tx=0x7fc93c00c860 comp rx=0 tx=0).stop 2023-06-22T08:16:46.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 -- 172.21.15.23:0/629140063 shutdown_connections 2023-06-22T08:16:46.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc93806ace0 0x7fc93806d190 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:46.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc94c152d70 0x7fc94c0cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:46.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc94c0c45f0 0x7fc94c1f2310 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:46.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 --2- 172.21.15.23:0/629140063 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc94c0c97d0 0x7fc94c0b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:46.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 -- 172.21.15.23:0/629140063 >> 172.21.15.23:0/629140063 conn(0x7fc94c06ae30 msgr2=0x7fc94c0cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:46.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.522+0000 7fc9511ff700 1 -- 172.21.15.23:0/629140063 shutdown_connections 2023-06-22T08:16:46.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:46.523+0000 7fc9511ff700 1 -- 172.21.15.23:0/629140063 wait complete. 2023-06-22T08:16:46.528 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 135 2023-06-22T08:16:46.544 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:16:48.545 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.752+0000 7fb528fff700 1 Processor -- start 2023-06-22T08:16:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.752+0000 7fb528fff700 1 -- start start 2023-06-22T08:16:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.753+0000 7fb528fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5240c3a40 0x7fb5240c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:48.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.753+0000 7fb528fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb52406d050 0x7fb524159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:48.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.753+0000 7fb528fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5240c6a60 con 0x7fb524156fc0 2023-06-22T08:16:48.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.753+0000 7fb528fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb524157570 con 0x7fb5240c3a40 2023-06-22T08:16:48.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.753+0000 7fb528fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5240c4350 con 0x7fb52406d050 2023-06-22T08:16:48.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.753+0000 7fb522d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb524156fc0 0x7fb524069eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:49270/0 (socket says 172.21.15.23:49270) 2023-06-22T08:16:48.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.753+0000 7fb522d9d700 1 -- 172.21.15.23:0/196179334 learned_addr learned my addr 172.21.15.23:0/196179334 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:48.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.753+0000 7fb52259c700 1 --2- 172.21.15.23:0/196179334 >> v2:172.21.15.67:3300/0 conn(0x7fb5240c3a40 0x7fb5240c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:48.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.753+0000 7fb51bfff700 1 --2- 172.21.15.23:0/196179334 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb52406d050 0x7fb524159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:48.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.754+0000 7fb521d9b700 1 -- 172.21.15.23:0/196179334 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647983666 0 0) 0x7fb5240c6a60 con 0x7fb524156fc0 2023-06-22T08:16:48.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.754+0000 7fb521d9b700 1 -- 172.21.15.23:0/196179334 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5100035b0 con 0x7fb524156fc0 2023-06-22T08:16:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.754+0000 7fb52259c700 1 -- 172.21.15.23:0/196179334 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb52406d050 msgr2=0x7fb524159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.754+0000 7fb52259c700 1 --2- 172.21.15.23:0/196179334 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb52406d050 0x7fb524159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.754+0000 7fb52259c700 1 -- 172.21.15.23:0/196179334 >> v1:172.21.15.23:6789/0 conn(0x7fb524156fc0 legacy=0x7fb524069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.754+0000 7fb52259c700 1 -- 172.21.15.23:0/196179334 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52414d690 con 0x7fb5240c3a40 2023-06-22T08:16:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.754+0000 7fb51bfff700 1 --2- 172.21.15.23:0/196179334 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb52406d050 0x7fb524159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:16:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.754+0000 7fb52259c700 1 --2- 172.21.15.23:0/196179334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5240c3a40 0x7fb5240c3e10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb514009840 tx=0x7fb514009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0d2f0ed6e459da9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb521d9b700 1 -- 172.21.15.23:0/196179334 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb514011830 con 0x7fb5240c3a40 2023-06-22T08:16:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb521d9b700 1 -- 172.21.15.23:0/196179334 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb514011990 con 0x7fb5240c3a40 2023-06-22T08:16:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb521d9b700 1 -- 172.21.15.23:0/196179334 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb514011d20 con 0x7fb5240c3a40 2023-06-22T08:16:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb528fff700 1 -- 172.21.15.23:0/196179334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5240c3a40 msgr2=0x7fb5240c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb528fff700 1 --2- 172.21.15.23:0/196179334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5240c3a40 0x7fb5240c3e10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb514009840 tx=0x7fb514009b50 comp rx=0 tx=0).stop 2023-06-22T08:16:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb528fff700 1 -- 172.21.15.23:0/196179334 shutdown_connections 2023-06-22T08:16:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb528fff700 1 --2- 172.21.15.23:0/196179334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5240c3a40 0x7fb5240c3e10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb528fff700 1 --2- 172.21.15.23:0/196179334 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb52406d050 0x7fb524159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:48.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb528fff700 1 -- 172.21.15.23:0/196179334 >> 172.21.15.23:0/196179334 conn(0x7fb5240ca3f0 msgr2=0x7fb5240ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:48.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb528fff700 1 -- 172.21.15.23:0/196179334 shutdown_connections 2023-06-22T08:16:48.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.755+0000 7fb528fff700 1 -- 172.21.15.23:0/196179334 wait complete. 2023-06-22T08:16:48.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.757+0000 7fb528fff700 1 Processor -- start 2023-06-22T08:16:48.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.757+0000 7fb528fff700 1 -- start start 2023-06-22T08:16:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.758+0000 7fb528fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52406d050 0x7fb5241eecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.758+0000 7fb528fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5240c3a40 0x7fb5240c3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.758+0000 7fb528fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb524156fc0 0x7fb524155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.758+0000 7fb528fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5240b9cc0 con 0x7fb5240c3a40 2023-06-22T08:16:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.758+0000 7fb528fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5241497a0 con 0x7fb52406d050 2023-06-22T08:16:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.758+0000 7fb528fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52414f2b0 con 0x7fb524156fc0 2023-06-22T08:16:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.758+0000 7fb52259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52406d050 0x7fb5241eecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb51bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5240c3a40 0x7fb5240c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb522d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb524156fc0 0x7fb524155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb52259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52406d050 0x7fb5241eecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:44936/0 (socket says 172.21.15.23:44936) 2023-06-22T08:16:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb52259c700 1 -- 172.21.15.23:0/3751664178 learned_addr learned my addr 172.21.15.23:0/3751664178 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb51bfff700 1 -- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb524156fc0 msgr2=0x7fb524155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb51bfff700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb524156fc0 0x7fb524155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb51bfff700 1 -- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52406d050 msgr2=0x7fb5241eecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb51bfff700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52406d050 0x7fb5241eecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:48.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb51bfff700 1 -- 172.21.15.23:0/3751664178 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50c0094a0 con 0x7fb5240c3a40 2023-06-22T08:16:48.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb51bfff700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5240c3a40 0x7fb5240c3020 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00e930 tx=0x7fb50c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:48.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb51affd700 1 -- 172.21.15.23:0/3751664178 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c013680 con 0x7fb5240c3a40 2023-06-22T08:16:48.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb51affd700 1 -- 172.21.15.23:0/3751664178 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50c0137e0 con 0x7fb5240c3a40 2023-06-22T08:16:48.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb528fff700 1 -- 172.21.15.23:0/3751664178 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5140094a0 con 0x7fb5240c3a40 2023-06-22T08:16:48.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.759+0000 7fb51affd700 1 -- 172.21.15.23:0/3751664178 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c013b90 con 0x7fb5240c3a40 2023-06-22T08:16:48.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.760+0000 7fb528fff700 1 -- 172.21.15.23:0/3751664178 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5241491e0 con 0x7fb5240c3a40 2023-06-22T08:16:48.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.761+0000 7fb528fff700 1 -- 172.21.15.23:0/3751664178 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52414f440 con 0x7fb5240c3a40 2023-06-22T08:16:48.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.762+0000 7fb51affd700 1 -- 172.21.15.23:0/3751664178 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb50c013cf0 con 0x7fb5240c3a40 2023-06-22T08:16:48.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.762+0000 7fb51affd700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb50406ace0 0x7fb50406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:48.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.763+0000 7fb51affd700 1 -- 172.21.15.23:0/3751664178 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7fb50c089980 con 0x7fb5240c3a40 2023-06-22T08:16:48.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.763+0000 7fb52259c700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb50406ace0 0x7fb50406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:48.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.763+0000 7fb52259c700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb50406ace0 0x7fb50406d190 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb514009b90 tx=0x7fb514008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:48.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:48.767+0000 7fb51affd700 1 -- 172.21.15.23:0/3751664178 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb50c054420 con 0x7fb5240c3a40 2023-06-22T08:16:49.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.053+0000 7fb528fff700 1 -- 172.21.15.23:0/3751664178 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb52404ea20 con 0x7fb5240c3a40 2023-06-22T08:16:49.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.054+0000 7fb51affd700 1 -- 172.21.15.23:0/3751664178 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5509 (secure 0 0 0) 0x7fb50c057a40 con 0x7fb5240c3a40 2023-06-22T08:16:49.057 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:49.057 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:43.914799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:49.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.056+0000 7fb528fff700 1 -- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb50406ace0 msgr2=0x7fb50406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:49.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.056+0000 7fb528fff700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb50406ace0 0x7fb50406d190 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb514009b90 tx=0x7fb514008030 comp rx=0 tx=0).stop 2023-06-22T08:16:49.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.056+0000 7fb528fff700 1 -- 172.21.15.23:0/3751664178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5240c3a40 msgr2=0x7fb5240c3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:49.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.056+0000 7fb528fff700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5240c3a40 0x7fb5240c3020 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00e930 tx=0x7fb50c00ec40 comp rx=0 tx=0).stop 2023-06-22T08:16:49.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.056+0000 7fb528fff700 1 -- 172.21.15.23:0/3751664178 shutdown_connections 2023-06-22T08:16:49.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.056+0000 7fb528fff700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb524156fc0 0x7fb524155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:49.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.056+0000 7fb528fff700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb50406ace0 0x7fb50406d190 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:49.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.056+0000 7fb528fff700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52406d050 0x7fb5241eecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:49.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.056+0000 7fb528fff700 1 --2- 172.21.15.23:0/3751664178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5240c3a40 0x7fb5240c3020 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:49.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.056+0000 7fb528fff700 1 -- 172.21.15.23:0/3751664178 >> 172.21.15.23:0/3751664178 conn(0x7fb5240ca3f0 msgr2=0x7fb5240c81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:49.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.057+0000 7fb528fff700 1 -- 172.21.15.23:0/3751664178 shutdown_connections 2023-06-22T08:16:49.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:49.057+0000 7fb528fff700 1 -- 172.21.15.23:0/3751664178 wait complete. 2023-06-22T08:16:49.062 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 135 2023-06-22T08:16:49.083 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:16:51.084 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:51.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.315+0000 7f6be87ff700 1 Processor -- start 2023-06-22T08:16:51.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.315+0000 7f6be87ff700 1 -- start start 2023-06-22T08:16:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.316+0000 7f6be87ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6be0152d70 0x7f6be00c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.316+0000 7f6be87ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6be00c45f0 0x7f6be00c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.316+0000 7f6be87ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be0159790 con 0x7f6be0159330 2023-06-22T08:16:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.316+0000 7f6be87ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be00c95e0 con 0x7f6be00c45f0 2023-06-22T08:16:51.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.316+0000 7f6be87ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be015bea0 con 0x7f6be0152d70 2023-06-22T08:16:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.317+0000 7f6be6d9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6be00c45f0 0x7f6be00c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.317+0000 7f6be5d9a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f6be0159330 0x7f6be00cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:49278/0 (socket says 172.21.15.23:49278) 2023-06-22T08:16:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.317+0000 7f6be5d9a700 1 -- 172.21.15.23:0/1444591962 learned_addr learned my addr 172.21.15.23:0/1444591962 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:51.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.317+0000 7f6be659b700 1 --2- 172.21.15.23:0/1444591962 >> v2:172.21.15.67:3300/0 conn(0x7f6be0152d70 0x7f6be00c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:51.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.317+0000 7f6be6d9c700 1 -- 172.21.15.23:0/1444591962 >> v2:172.21.15.67:3300/0 conn(0x7f6be0152d70 msgr2=0x7f6be00c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:51.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.317+0000 7f6be6d9c700 1 --2- 172.21.15.23:0/1444591962 >> v2:172.21.15.67:3300/0 conn(0x7f6be0152d70 0x7f6be00c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:51.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.317+0000 7f6be6d9c700 1 -- 172.21.15.23:0/1444591962 >> v1:172.21.15.23:6789/0 conn(0x7f6be0159330 legacy=0x7f6be00cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:51.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.317+0000 7f6be6d9c700 1 -- 172.21.15.23:0/1444591962 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be015c160 con 0x7f6be00c45f0 2023-06-22T08:16:51.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.317+0000 7f6be6d9c700 1 --2- 172.21.15.23:0/1444591962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be00c45f0 0x7f6be00c4a10 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6be00c8ef0 tx=0x7f6bd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee96e43c2e9cebd server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:51.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be5599700 1 -- 172.21.15.23:0/1444591962 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd800e070 con 0x7f6be00c45f0 2023-06-22T08:16:51.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be5599700 1 -- 172.21.15.23:0/1444591962 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd800bb20 con 0x7f6be00c45f0 2023-06-22T08:16:51.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be5599700 1 -- 172.21.15.23:0/1444591962 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8012430 con 0x7f6be00c45f0 2023-06-22T08:16:51.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be87ff700 1 -- 172.21.15.23:0/1444591962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be00c45f0 msgr2=0x7f6be00c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:51.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be87ff700 1 --2- 172.21.15.23:0/1444591962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be00c45f0 0x7f6be00c4a10 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6be00c8ef0 tx=0x7f6bd800b330 comp rx=0 tx=0).stop 2023-06-22T08:16:51.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be87ff700 1 -- 172.21.15.23:0/1444591962 shutdown_connections 2023-06-22T08:16:51.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be87ff700 1 --2- 172.21.15.23:0/1444591962 >> v2:172.21.15.67:3300/0 conn(0x7f6be0152d70 0x7f6be00c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:51.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be87ff700 1 --2- 172.21.15.23:0/1444591962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be00c45f0 0x7f6be00c4a10 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:51.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be87ff700 1 -- 172.21.15.23:0/1444591962 >> 172.21.15.23:0/1444591962 conn(0x7f6be006ae30 msgr2=0x7f6be006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:51.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be87ff700 1 -- 172.21.15.23:0/1444591962 shutdown_connections 2023-06-22T08:16:51.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.318+0000 7f6be87ff700 1 -- 172.21.15.23:0/1444591962 wait complete. 2023-06-22T08:16:51.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.320+0000 7f6be87ff700 1 Processor -- start 2023-06-22T08:16:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.320+0000 7f6be87ff700 1 -- start start 2023-06-22T08:16:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.321+0000 7f6be87ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be00c45f0 0x7f6be01e4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.321+0000 7f6be87ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be0152d70 0x7f6be01e1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.321+0000 7f6be87ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be0159330 0x7f6be01e2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.321+0000 7f6be87ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be01e29c0 con 0x7f6be0159330 2023-06-22T08:16:51.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.321+0000 7f6be87ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be00cd610 con 0x7f6be0152d70 2023-06-22T08:16:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.321+0000 7f6be87ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be01e42f0 con 0x7f6be00c45f0 2023-06-22T08:16:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.321+0000 7f6be5d9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be0152d70 0x7f6be01e1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.321+0000 7f6be659b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be00c45f0 0x7f6be01e4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.321+0000 7f6be6d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be0159330 0x7f6be01e2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6be5d9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be0152d70 0x7f6be01e1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:44952/0 (socket says 172.21.15.23:44952) 2023-06-22T08:16:51.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6be5d9a700 1 -- 172.21.15.23:0/807785756 learned_addr learned my addr 172.21.15.23:0/807785756 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6be5d9a700 1 -- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be00c45f0 msgr2=0x7f6be01e4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6be5d9a700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be00c45f0 0x7f6be01e4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6be5d9a700 1 -- 172.21.15.23:0/807785756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be0159330 msgr2=0x7f6be01e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6be5d9a700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be0159330 0x7f6be01e2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:51.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6be5d9a700 1 -- 172.21.15.23:0/807785756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd8009a70 con 0x7f6be0152d70 2023-06-22T08:16:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6be6d9c700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be0159330 0x7f6be01e2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:16:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6be5d9a700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be0152d70 0x7f6be01e1f40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000c480 tx=0x7f6bd000c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:51.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6bd6ffd700 1 -- 172.21.15.23:0/807785756 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0005d70 con 0x7f6be0152d70 2023-06-22T08:16:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6bd6ffd700 1 -- 172.21.15.23:0/807785756 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd000ce90 con 0x7f6be0152d70 2023-06-22T08:16:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6bd6ffd700 1 -- 172.21.15.23:0/807785756 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd00072d0 con 0x7f6be0152d70 2023-06-22T08:16:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.322+0000 7f6be87ff700 1 -- 172.21.15.23:0/807785756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be0159b50 con 0x7f6be0152d70 2023-06-22T08:16:51.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.323+0000 7f6be87ff700 1 -- 172.21.15.23:0/807785756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be00cce70 con 0x7f6be0152d70 2023-06-22T08:16:51.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.324+0000 7f6be87ff700 1 -- 172.21.15.23:0/807785756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be01e4460 con 0x7f6be0152d70 2023-06-22T08:16:51.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.325+0000 7f6bd6ffd700 1 -- 172.21.15.23:0/807785756 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6bd0007430 con 0x7f6be0152d70 2023-06-22T08:16:51.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.326+0000 7f6bd6ffd700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bc406ace0 0x7f6bc406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:51.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.326+0000 7f6bd6ffd700 1 -- 172.21.15.23:0/807785756 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f6bd0088ca0 con 0x7f6be0152d70 2023-06-22T08:16:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.326+0000 7f6be659b700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bc406ace0 0x7f6bc406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.327+0000 7f6be659b700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bc406ace0 0x7f6bc406d190 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009be0 tx=0x7f6bdc009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:51.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.330+0000 7f6bd6ffd700 1 -- 172.21.15.23:0/807785756 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6bd00537f0 con 0x7f6be0152d70 2023-06-22T08:16:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.590+0000 7f6be87ff700 1 -- 172.21.15.23:0/807785756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6be004ea20 con 0x7f6be0152d70 2023-06-22T08:16:51.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.591+0000 7f6bd6ffd700 1 -- 172.21.15.23:0/807785756 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+5509 (secure 0 0 0) 0x7f6bd0056e10 con 0x7f6be0152d70 2023-06-22T08:16:51.594 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:51.594 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:49.937453+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:51.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 -- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bc406ace0 msgr2=0x7f6bc406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:51.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bc406ace0 0x7f6bc406d190 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009be0 tx=0x7f6bdc009210 comp rx=0 tx=0).stop 2023-06-22T08:16:51.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 -- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be0152d70 msgr2=0x7f6be01e1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:51.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be0152d70 0x7f6be01e1f40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000c480 tx=0x7f6bd000c790 comp rx=0 tx=0).stop 2023-06-22T08:16:51.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 -- 172.21.15.23:0/807785756 shutdown_connections 2023-06-22T08:16:51.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6be0159330 0x7f6be01e2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:51.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6be00c45f0 0x7f6be01e4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:51.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f6bc406ace0 0x7f6bc406d190 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:51.598 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 --2- 172.21.15.23:0/807785756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6be0152d70 0x7f6be01e1f40 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:51.598 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 -- 172.21.15.23:0/807785756 >> 172.21.15.23:0/807785756 conn(0x7f6be006ae30 msgr2=0x7f6be00c3850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:51.598 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 -- 172.21.15.23:0/807785756 shutdown_connections 2023-06-22T08:16:51.598 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:51.593+0000 7f6be87ff700 1 -- 172.21.15.23:0/807785756 wait complete. 2023-06-22T08:16:51.598 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 136 2023-06-22T08:16:51.614 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=2, replacing gid=5551 2023-06-22T08:16:51.615 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.c 2023-06-22T08:16:55.615 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-06-22T08:16:55.616 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-22T08:16:55.616 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-22T08:16:55.621 INFO:tasks.ceph.mds.c:Started 2023-06-22T08:16:55.622 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:55.790 INFO:tasks.ceph.mds.c.smithi023.stdout:starting mds.c at 2023-06-22T08:16:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.839+0000 7f8a87bff700 1 Processor -- start 2023-06-22T08:16:55.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.839+0000 7f8a87bff700 1 -- start start 2023-06-22T08:16:55.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.840+0000 7f8a87bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a80152dd0 0x7f8a800aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:55.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.840+0000 7f8a87bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8a800c8bd0 0x7f8a8006afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:55.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.840+0000 7f8a87bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a800cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:55.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.840+0000 7f8a87bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a800c7400 con 0x7f8a800c8bd0 2023-06-22T08:16:55.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.840+0000 7f8a87bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a800c81c0 con 0x7f8a80152dd0 2023-06-22T08:16:55.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.840+0000 7f8a87bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800c84a0 con 0x7f8a8006b500 2023-06-22T08:16:55.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.840+0000 7f8a8599b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a80152dd0 0x7f8a800aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:55.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.840+0000 7f8a8519a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8a800c8bd0 0x7f8a8006afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.840+0000 7f8a8619c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a800cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.840+0000 7f8a8599b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8a80152dd0 0x7f8a800aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:44970/0 (socket says 172.21.15.23:44970) 2023-06-22T08:16:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a8619c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a800cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:42414/0 (socket says 172.21.15.23:42414) 2023-06-22T08:16:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a8519a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8a800c8bd0 0x7f8a8006afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55038/0 (socket says 172.21.15.23:55038) 2023-06-22T08:16:55.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a8599b700 1 -- 172.21.15.23:0/3788758919 learned_addr learned my addr 172.21.15.23:0/3788758919 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a8599b700 1 -- 172.21.15.23:0/3788758919 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 msgr2=0x7f8a800cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:55.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a8599b700 1 --2- 172.21.15.23:0/3788758919 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a800cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:55.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a8599b700 1 -- 172.21.15.23:0/3788758919 >> v2:172.21.15.23:3300/0 conn(0x7f8a800c8bd0 msgr2=0x7f8a8006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:55.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a8599b700 1 --2- 172.21.15.23:0/3788758919 >> v2:172.21.15.23:3300/0 conn(0x7f8a800c8bd0 0x7f8a8006afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:55.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a8599b700 1 -- 172.21.15.23:0/3788758919 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a800cd370 con 0x7f8a80152dd0 2023-06-22T08:16:55.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a8599b700 1 --2- 172.21.15.23:0/3788758919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80152dd0 0x7f8a800aedc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8a80070880 tx=0x7f8a7c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28dd3a3d04c0b9d9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:55.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a84999700 1 -- 172.21.15.23:0/3788758919 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c005e30 con 0x7f8a80152dd0 2023-06-22T08:16:55.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a84999700 1 -- 172.21.15.23:0/3788758919 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7c004030 con 0x7f8a80152dd0 2023-06-22T08:16:55.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.841+0000 7f8a84999700 1 -- 172.21.15.23:0/3788758919 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c0043e0 con 0x7f8a80152dd0 2023-06-22T08:16:55.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.842+0000 7f8a87bff700 1 -- 172.21.15.23:0/3788758919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80152dd0 msgr2=0x7f8a800aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:55.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.842+0000 7f8a87bff700 1 --2- 172.21.15.23:0/3788758919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80152dd0 0x7f8a800aedc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8a80070880 tx=0x7f8a7c009bd0 comp rx=0 tx=0).stop 2023-06-22T08:16:55.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.842+0000 7f8a87bff700 1 -- 172.21.15.23:0/3788758919 shutdown_connections 2023-06-22T08:16:55.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.842+0000 7f8a87bff700 1 --2- 172.21.15.23:0/3788758919 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a800cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:55.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.842+0000 7f8a87bff700 1 --2- 172.21.15.23:0/3788758919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80152dd0 0x7f8a800aedc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:55.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.842+0000 7f8a87bff700 1 --2- 172.21.15.23:0/3788758919 >> v2:172.21.15.23:3300/0 conn(0x7f8a800c8bd0 0x7f8a8006afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:55.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.842+0000 7f8a87bff700 1 -- 172.21.15.23:0/3788758919 >> 172.21.15.23:0/3788758919 conn(0x7f8a8006fb70 msgr2=0x7f8a8006ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:55.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.842+0000 7f8a87bff700 1 -- 172.21.15.23:0/3788758919 shutdown_connections 2023-06-22T08:16:55.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.842+0000 7f8a87bff700 1 -- 172.21.15.23:0/3788758919 wait complete. 2023-06-22T08:16:55.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.844+0000 7f8a87bff700 1 Processor -- start 2023-06-22T08:16:55.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.844+0000 7f8a87bff700 1 -- start start 2023-06-22T08:16:55.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a87bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a8014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:55.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a87bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a800c8bd0 0x7f8a8006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:55.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a87bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80152dd0 0x7f8a8014f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:55.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a87bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a801f1930 con 0x7f8a800c8bd0 2023-06-22T08:16:55.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a87bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80153490 con 0x7f8a80152dd0 2023-06-22T08:16:55.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a87bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a801f1b40 con 0x7f8a8006b500 2023-06-22T08:16:55.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a8599b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a8014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:55.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a8599b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a8014fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:42430/0 (socket says 172.21.15.23:42430) 2023-06-22T08:16:55.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a8599b700 1 -- 172.21.15.23:0/2955412017 learned_addr learned my addr 172.21.15.23:0/2955412017 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:55.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a8519a700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a800c8bd0 0x7f8a8006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:55.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.845+0000 7f8a8619c700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80152dd0 0x7f8a8014f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:55.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.846+0000 7f8a8599b700 1 -- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80152dd0 msgr2=0x7f8a8014f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:55.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.846+0000 7f8a8599b700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80152dd0 0x7f8a8014f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:55.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.846+0000 7f8a8599b700 1 -- 172.21.15.23:0/2955412017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a800c8bd0 msgr2=0x7f8a8006e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:55.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.846+0000 7f8a8599b700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a800c8bd0 0x7f8a8006e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:55.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.846+0000 7f8a8599b700 1 -- 172.21.15.23:0/2955412017 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7c009500 con 0x7f8a8006b500 2023-06-22T08:16:55.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.846+0000 7f8a8599b700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a8014fff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c000c00 tx=0x7f8a7c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:55.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.846+0000 7f8a767fc700 1 -- 172.21.15.23:0/2955412017 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c003bb0 con 0x7f8a8006b500 2023-06-22T08:16:55.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.846+0000 7f8a767fc700 1 -- 172.21.15.23:0/2955412017 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7c003d10 con 0x7f8a8006b500 2023-06-22T08:16:55.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.846+0000 7f8a767fc700 1 -- 172.21.15.23:0/2955412017 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c01c660 con 0x7f8a8006b500 2023-06-22T08:16:55.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.846+0000 7f8a87bff700 1 -- 172.21.15.23:0/2955412017 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a801f1e20 con 0x7f8a8006b500 2023-06-22T08:16:55.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.847+0000 7f8a87bff700 1 -- 172.21.15.23:0/2955412017 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a801f23e0 con 0x7f8a8006b500 2023-06-22T08:16:55.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.848+0000 7f8a87bff700 1 -- 172.21.15.23:0/2955412017 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a800c7270 con 0x7f8a8006b500 2023-06-22T08:16:55.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.849+0000 7f8a767fc700 1 -- 172.21.15.23:0/2955412017 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8a7c01c7c0 con 0x7f8a8006b500 2023-06-22T08:16:55.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.850+0000 7f8a767fc700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8a6406ace0 0x7f8a6406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:55.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.850+0000 7f8a767fc700 1 -- 172.21.15.23:0/2955412017 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f8a7c08a980 con 0x7f8a8006b500 2023-06-22T08:16:55.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.850+0000 7f8a8519a700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8a6406ace0 0x7f8a6406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:55.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.850+0000 7f8a8519a700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8a6406ace0 0x7f8a6406d190 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8a8015c390 tx=0x7f8a70009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:55.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:55.854+0000 7f8a767fc700 1 -- 172.21.15.23:0/2955412017 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8a7c055390 con 0x7f8a8006b500 2023-06-22T08:16:55.934 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 56% (175/309) Updating files: 57% (177/309) Updating files: 58% (180/309) Updating files: 59% (183/309) Updating files: 60% (186/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 86% (267/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 99% (308/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2023-06-22T08:16:56.089 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 8de9501df27 0.81 2023-06-22T08:16:56.102 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.81 2023-06-22T08:16:56.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.172+0000 7f8a87bff700 1 -- 172.21.15.23:0/2955412017 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a800c3570 con 0x7f8a8006b500 2023-06-22T08:16:56.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.173+0000 7f8a767fc700 1 -- 172.21.15.23:0/2955412017 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5508 (secure 0 0 0) 0x7f8a7c0589b0 con 0x7f8a8006b500 2023-06-22T08:16:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:56.176 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:55.529725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:56.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 -- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8a6406ace0 msgr2=0x7f8a6406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:56.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8a6406ace0 0x7f8a6406d190 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8a8015c390 tx=0x7f8a70009210 comp rx=0 tx=0).stop 2023-06-22T08:16:56.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 -- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 msgr2=0x7f8a8014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:56.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a8014fff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c000c00 tx=0x7f8a7c004750 comp rx=0 tx=0).stop 2023-06-22T08:16:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 -- 172.21.15.23:0/2955412017 shutdown_connections 2023-06-22T08:16:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8a6406ace0 0x7f8a6406d190 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a80152dd0 0x7f8a8014f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8a8006b500 0x7f8a8014fff0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 --2- 172.21.15.23:0/2955412017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a800c8bd0 0x7f8a8006e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 -- 172.21.15.23:0/2955412017 >> 172.21.15.23:0/2955412017 conn(0x7f8a8006fb70 msgr2=0x7f8a8015b620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 -- 172.21.15.23:0/2955412017 shutdown_connections 2023-06-22T08:16:56.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:56.176+0000 7f8a87bff700 1 -- 172.21.15.23:0/2955412017 wait complete. 2023-06-22T08:16:56.181 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 137 2023-06-22T08:16:56.200 INFO:tasks.workunit.client.0.smithi023.stdout:v0.82 2023-06-22T08:16:56.201 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:16:56.201 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 82 -eq 48 ']' 2023-06-22T08:16:56.201 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.82 2023-06-22T08:16:56.201 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.82 2023-06-22T08:16:56.202 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.82 2023-06-22T08:16:56.202 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-06-22T08:16:58.203 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:16:58.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.416+0000 7f206f3ff700 1 Processor -- start 2023-06-22T08:16:58.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.416+0000 7f206f3ff700 1 -- start start 2023-06-22T08:16:58.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206f3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2068152ce0 0x7f20680cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:58.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206f3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f20680c9fd0 0x7f206815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:58.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206f3ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206815c370 con 0x7f2068158880 2023-06-22T08:16:58.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206f3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20681530b0 con 0x7f20680c9fd0 2023-06-22T08:16:58.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206f3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20680cf6f0 con 0x7f2068152ce0 2023-06-22T08:16:58.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206d19b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2068152ce0 0x7f20680cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:58.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206c99a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f20680c9fd0 0x7f206815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:58.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206d99c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2068158880 0x7f2068158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57068/0 (socket says 172.21.15.23:57068) 2023-06-22T08:16:58.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206d19b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2068152ce0 0x7f20680cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:50076/0 (socket says 172.21.15.23:50076) 2023-06-22T08:16:58.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206d99c700 1 -- 172.21.15.23:0/486807568 learned_addr learned my addr 172.21.15.23:0/486807568 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:16:58.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.417+0000 7f206c99a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f20680c9fd0 0x7f206815c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51352/0 (socket says 172.21.15.23:51352) 2023-06-22T08:16:58.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.418+0000 7f206d19b700 1 -- 172.21.15.23:0/486807568 >> v2:172.21.15.23:3300/0 conn(0x7f20680c9fd0 msgr2=0x7f206815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:16:58.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.418+0000 7f206d19b700 1 --2- 172.21.15.23:0/486807568 >> v2:172.21.15.23:3300/0 conn(0x7f20680c9fd0 0x7f206815c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:58.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.418+0000 7f206d19b700 1 -- 172.21.15.23:0/486807568 >> v1:172.21.15.23:6789/0 conn(0x7f2068158880 legacy=0x7f2068158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:58.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.418+0000 7f206d19b700 1 -- 172.21.15.23:0/486807568 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206815bda0 con 0x7f2068152ce0 2023-06-22T08:16:58.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.418+0000 7f206d19b700 1 --2- 172.21.15.23:0/486807568 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2068152ce0 0x7f20680cf1b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2064009840 tx=0x7f2064009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6db527826996ec00 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:58.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.418+0000 7f205ffff700 1 -- 172.21.15.23:0/486807568 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2064011830 con 0x7f2068152ce0 2023-06-22T08:16:58.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.418+0000 7f205ffff700 1 -- 172.21.15.23:0/486807568 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2064011990 con 0x7f2068152ce0 2023-06-22T08:16:58.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.418+0000 7f205ffff700 1 -- 172.21.15.23:0/486807568 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2064011d20 con 0x7f2068152ce0 2023-06-22T08:16:58.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.419+0000 7f206f3ff700 1 -- 172.21.15.23:0/486807568 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2068152ce0 msgr2=0x7f20680cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:58.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.419+0000 7f206f3ff700 1 --2- 172.21.15.23:0/486807568 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2068152ce0 0x7f20680cf1b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2064009840 tx=0x7f2064009b50 comp rx=0 tx=0).stop 2023-06-22T08:16:58.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.419+0000 7f206f3ff700 1 -- 172.21.15.23:0/486807568 shutdown_connections 2023-06-22T08:16:58.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.419+0000 7f206f3ff700 1 --2- 172.21.15.23:0/486807568 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2068152ce0 0x7f20680cf1b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:58.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.419+0000 7f206f3ff700 1 --2- 172.21.15.23:0/486807568 >> v2:172.21.15.23:3300/0 conn(0x7f20680c9fd0 0x7f206815c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:58.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.419+0000 7f206f3ff700 1 -- 172.21.15.23:0/486807568 >> 172.21.15.23:0/486807568 conn(0x7f206806ae30 msgr2=0x7f206806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:58.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.419+0000 7f206f3ff700 1 -- 172.21.15.23:0/486807568 shutdown_connections 2023-06-22T08:16:58.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.419+0000 7f206f3ff700 1 -- 172.21.15.23:0/486807568 wait complete. 2023-06-22T08:16:58.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.421+0000 7f206f3ff700 1 Processor -- start 2023-06-22T08:16:58.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.421+0000 7f206f3ff700 1 -- start start 2023-06-22T08:16:58.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206f3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20680c9fd0 0x7f20681eec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:58.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206f3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2068152ce0 0x7f206814caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:58.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206f3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2068158880 0x7f20681485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:58.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206f3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20680cf130 con 0x7f2068158880 2023-06-22T08:16:58.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206f3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2068148b10 con 0x7f2068152ce0 2023-06-22T08:16:58.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206f3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20680ccc10 con 0x7f20680c9fd0 2023-06-22T08:16:58.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206c99a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2068152ce0 0x7f206814caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:58.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206d99c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2068158880 0x7f20681485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:58.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206c99a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2068152ce0 0x7f206814caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:57470/0 (socket says 172.21.15.23:57470) 2023-06-22T08:16:58.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206c99a700 1 -- 172.21.15.23:0/3194583654 learned_addr learned my addr 172.21.15.23:0/3194583654 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:16:58.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.422+0000 7f206d19b700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20680c9fd0 0x7f20681eec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:58.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f206c99a700 1 -- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20680c9fd0 msgr2=0x7f20681eec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:58.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f206c99a700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20680c9fd0 0x7f20681eec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:58.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f206c99a700 1 -- 172.21.15.23:0/3194583654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2068158880 msgr2=0x7f20681485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:58.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f206c99a700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2068158880 0x7f20681485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:58.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f206c99a700 1 -- 172.21.15.23:0/3194583654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20640094a0 con 0x7f2068152ce0 2023-06-22T08:16:58.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f206d99c700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2068158880 0x7f20681485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:16:58.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f206c99a700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2068152ce0 0x7f206814caf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f205800e8a0 tx=0x7f205800ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:58.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f205dffb700 1 -- 172.21.15.23:0/3194583654 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2058009590 con 0x7f2068152ce0 2023-06-22T08:16:58.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f205dffb700 1 -- 172.21.15.23:0/3194583654 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2058013680 con 0x7f2068152ce0 2023-06-22T08:16:58.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f206f3ff700 1 -- 172.21.15.23:0/3194583654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20680cced0 con 0x7f2068152ce0 2023-06-22T08:16:58.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f205dffb700 1 -- 172.21.15.23:0/3194583654 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20580139f0 con 0x7f2068152ce0 2023-06-22T08:16:58.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.423+0000 7f206f3ff700 1 -- 172.21.15.23:0/3194583654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20680cd490 con 0x7f2068152ce0 2023-06-22T08:16:58.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.424+0000 7f206f3ff700 1 -- 172.21.15.23:0/3194583654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206804ea20 con 0x7f2068152ce0 2023-06-22T08:16:58.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.425+0000 7f205dffb700 1 -- 172.21.15.23:0/3194583654 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2058013b50 con 0x7f2068152ce0 2023-06-22T08:16:58.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.426+0000 7f205dffb700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f204c06ace0 0x7f204c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:16:58.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.426+0000 7f205dffb700 1 -- 172.21.15.23:0/3194583654 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f20580898f0 con 0x7f2068152ce0 2023-06-22T08:16:58.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.426+0000 7f206d19b700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f204c06ace0 0x7f204c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:16:58.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.427+0000 7f206d19b700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f204c06ace0 0x7f204c06d190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2064000c00 tx=0x7f2064008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:16:58.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.431+0000 7f205dffb700 1 -- 172.21.15.23:0/3194583654 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2058057e20 con 0x7f2068152ce0 2023-06-22T08:16:58.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.701+0000 7f206f3ff700 1 -- 172.21.15.23:0/3194583654 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20681562c0 con 0x7f2068152ce0 2023-06-22T08:16:58.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.702+0000 7f205dffb700 1 -- 172.21.15.23:0/3194583654 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+6262 (secure 0 0 0) 0x7f20580579b0 con 0x7f2068152ce0 2023-06-22T08:16:58.705 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:16:58.705 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:16:55.529725+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:16:58.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.704+0000 7f206f3ff700 1 -- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f204c06ace0 msgr2=0x7f204c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:58.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.704+0000 7f206f3ff700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f204c06ace0 0x7f204c06d190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2064000c00 tx=0x7f2064008030 comp rx=0 tx=0).stop 2023-06-22T08:16:58.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.704+0000 7f206f3ff700 1 -- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2068152ce0 msgr2=0x7f206814caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:16:58.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.704+0000 7f206f3ff700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2068152ce0 0x7f206814caf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f205800e8a0 tx=0x7f205800ebb0 comp rx=0 tx=0).stop 2023-06-22T08:16:58.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.705+0000 7f206f3ff700 1 -- 172.21.15.23:0/3194583654 shutdown_connections 2023-06-22T08:16:58.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.705+0000 7f206f3ff700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2068158880 0x7f20681485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:58.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.705+0000 7f206f3ff700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20680c9fd0 0x7f20681eec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:58.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.705+0000 7f206f3ff700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f204c06ace0 0x7f204c06d190 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:58.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.705+0000 7f206f3ff700 1 --2- 172.21.15.23:0/3194583654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2068152ce0 0x7f206814caf0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:16:58.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.705+0000 7f206f3ff700 1 -- 172.21.15.23:0/3194583654 >> 172.21.15.23:0/3194583654 conn(0x7f206806ae30 msgr2=0x7f206814f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:16:58.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.705+0000 7f206f3ff700 1 -- 172.21.15.23:0/3194583654 shutdown_connections 2023-06-22T08:16:58.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:16:58.705+0000 7f206f3ff700 1 -- 172.21.15.23:0/3194583654 wait complete. 2023-06-22T08:16:58.710 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 138 2023-06-22T08:16:58.730 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-06-22T08:16:58.731 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 99 secs before thrashing 2023-06-22T08:16:59.716 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 69% (142/204) Updating files: 70% (143/204) Updating files: 71% (145/204) Updating files: 72% (147/204) Updating files: 73% (149/204) Updating files: 74% (151/204) Updating files: 75% (153/204) Updating files: 76% (156/204) Updating files: 77% (158/204) Updating files: 78% (160/204) Updating files: 79% (162/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 85% (174/204) Updating files: 86% (176/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2023-06-22T08:16:59.792 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 14085f42ddd 0.82 2023-06-22T08:16:59.798 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.82 2023-06-22T08:16:59.862 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:16:59.862 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 83 -eq 48 ']' 2023-06-22T08:16:59.862 INFO:tasks.workunit.client.0.smithi023.stdout:v0.83 2023-06-22T08:16:59.863 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.83 2023-06-22T08:16:59.863 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.83 2023-06-22T08:16:59.863 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.83 2023-06-22T08:17:02.572 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 27% (59/218) Updating files: 28% (62/218) Updating files: 29% (64/218) Updating files: 30% (66/218) Updating files: 31% (68/218) Updating files: 32% (70/218) Updating files: 33% (72/218) Updating files: 34% (75/218) Updating files: 35% (77/218) Updating files: 36% (79/218) Updating files: 37% (81/218) Updating files: 38% (83/218) Updating files: 39% (86/218) Updating files: 40% (88/218) Updating files: 41% (90/218) Updating files: 42% (92/218) Updating files: 43% (94/218) Updating files: 44% (96/218) Updating files: 45% (99/218) Updating files: 46% (101/218) Updating files: 47% (103/218) Updating files: 48% (105/218) Updating files: 49% (107/218) Updating files: 50% (109/218) Updating files: 51% (112/218) Updating files: 52% (114/218) Updating files: 53% (116/218) Updating files: 54% (118/218) Updating files: 55% (120/218) Updating files: 56% (123/218) Updating files: 57% (125/218) Updating files: 58% (127/218) Updating files: 59% (129/218) Updating files: 60% (131/218) Updating files: 61% (133/218) Updating files: 62% (136/218) Updating files: 63% (138/218) Updating files: 64% (140/218) Updating files: 65% (142/218) Updating files: 66% (144/218) Updating files: 67% (147/218) Updating files: 68% (149/218) Updating files: 69% (151/218) Updating files: 70% (153/218) Updating files: 71% (155/218) Updating files: 72% (157/218) Updating files: 73% (160/218) Updating files: 74% (162/218) Updating files: 75% (164/218) Updating files: 76% (166/218) Updating files: 77% (168/218) Updating files: 78% (171/218) Updating files: 79% (173/218) Updating files: 80% (175/218) Updating files: 81% (177/218) Updating files: 82% (179/218) Updating files: 83% (181/218) Updating files: 84% (184/218) Updating files: 85% (186/218) Updating files: 86% (188/218) Updating files: 87% (190/218) Updating files: 88% (192/218) Updating files: 89% (195/218) Updating files: 90% (197/218) Updating files: 91% (199/218) Updating files: 91% (200/218) Updating files: 92% (201/218) Updating files: 93% (203/218) Updating files: 94% (205/218) Updating files: 95% (208/218) Updating files: 96% (210/218) Updating files: 97% (212/218) Updating files: 98% (214/218) Updating files: 99% (216/218) Updating files: 100% (218/218) Updating files: 100% (218/218), done. 2023-06-22T08:17:02.679 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 78ff1f0a5df 0.83 2023-06-22T08:17:02.684 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.83 2023-06-22T08:17:02.779 INFO:tasks.workunit.client.0.smithi023.stdout:v0.84 2023-06-22T08:17:02.779 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:02.779 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 84 -eq 48 ']' 2023-06-22T08:17:02.780 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.84 2023-06-22T08:17:02.780 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.84 2023-06-22T08:17:02.780 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.84 2023-06-22T08:17:05.005 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 48% (174/356) Updating files: 49% (175/356) Updating files: 50% (178/356) Updating files: 51% (182/356) Updating files: 52% (186/356) Updating files: 53% (189/356) Updating files: 54% (193/356) Updating files: 55% (196/356) Updating files: 56% (200/356) Updating files: 57% (203/356) Updating files: 58% (207/356) Updating files: 59% (211/356) Updating files: 60% (214/356) Updating files: 61% (218/356) Updating files: 62% (221/356) Updating files: 63% (225/356) Updating files: 64% (228/356) Updating files: 65% (232/356) Updating files: 66% (235/356) Updating files: 67% (239/356) Updating files: 68% (243/356) Updating files: 69% (246/356) Updating files: 70% (250/356) Updating files: 71% (253/356) Updating files: 72% (257/356) Updating files: 73% (260/356) Updating files: 74% (264/356) Updating files: 75% (267/356) Updating files: 76% (271/356) Updating files: 77% (275/356) Updating files: 78% (278/356) Updating files: 79% (282/356) Updating files: 80% (285/356) Updating files: 81% (289/356) Updating files: 82% (292/356) Updating files: 83% (296/356) Updating files: 84% (300/356) Updating files: 85% (303/356) Updating files: 86% (307/356) Updating files: 87% (310/356) Updating files: 88% (314/356) Updating files: 89% (317/356) Updating files: 90% (321/356) Updating files: 91% (324/356) Updating files: 92% (328/356) Updating files: 93% (332/356) Updating files: 94% (335/356) Updating files: 95% (339/356) Updating files: 96% (342/356) Updating files: 97% (346/356) Updating files: 98% (349/356) Updating files: 99% (353/356) Updating files: 100% (356/356) Updating files: 100% (356/356), done. 2023-06-22T08:17:05.108 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 8336f81c5c7 0.84 2023-06-22T08:17:05.116 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.84 2023-06-22T08:17:05.363 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:05.363 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 85 -eq 48 ']' 2023-06-22T08:17:05.363 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.85 2023-06-22T08:17:05.364 INFO:tasks.workunit.client.0.smithi023.stdout:v0.85 2023-06-22T08:17:05.364 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.85 2023-06-22T08:17:05.364 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.85 2023-06-22T08:17:07.326 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 61% (150/243) Updating files: 62% (151/243) Updating files: 63% (154/243) Updating files: 64% (156/243) Updating files: 65% (158/243) Updating files: 66% (161/243) Updating files: 67% (163/243) Updating files: 68% (166/243) Updating files: 69% (168/243) Updating files: 70% (171/243) Updating files: 71% (173/243) Updating files: 72% (175/243) Updating files: 73% (178/243) Updating files: 74% (180/243) Updating files: 75% (183/243) Updating files: 76% (185/243) Updating files: 77% (188/243) Updating files: 78% (190/243) Updating files: 79% (192/243) Updating files: 80% (195/243) Updating files: 81% (197/243) Updating files: 82% (200/243) Updating files: 83% (202/243) Updating files: 84% (205/243) Updating files: 85% (207/243) Updating files: 86% (209/243) Updating files: 87% (212/243) Updating files: 88% (214/243) Updating files: 89% (217/243) Updating files: 90% (219/243) Updating files: 91% (222/243) Updating files: 92% (224/243) Updating files: 93% (226/243) Updating files: 94% (229/243) Updating files: 95% (231/243) Updating files: 96% (234/243) Updating files: 97% (236/243) Updating files: 98% (239/243) Updating files: 99% (241/243) Updating files: 100% (243/243) Updating files: 100% (243/243), done. 2023-06-22T08:17:07.442 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at a0c22842db9 0.85 2023-06-22T08:17:07.448 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.85 2023-06-22T08:17:07.566 INFO:tasks.workunit.client.0.smithi023.stdout:v0.86 2023-06-22T08:17:07.566 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:07.566 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 86 -eq 48 ']' 2023-06-22T08:17:07.567 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.86 2023-06-22T08:17:07.567 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.86 2023-06-22T08:17:07.567 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.86 2023-06-22T08:17:10.818 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 30% (137/447) Updating files: 31% (139/447) Updating files: 32% (144/447) Updating files: 33% (148/447) Updating files: 34% (152/447) Updating files: 35% (157/447) Updating files: 36% (161/447) Updating files: 37% (166/447) Updating files: 38% (170/447) Updating files: 39% (175/447) Updating files: 40% (179/447) Updating files: 41% (184/447) Updating files: 42% (188/447) Updating files: 43% (193/447) Updating files: 44% (197/447) Updating files: 45% (202/447) Updating files: 46% (206/447) Updating files: 47% (211/447) Updating files: 48% (215/447) Updating files: 49% (220/447) Updating files: 50% (224/447) Updating files: 51% (228/447) Updating files: 52% (233/447) Updating files: 53% (237/447) Updating files: 54% (242/447) Updating files: 55% (246/447) Updating files: 56% (251/447) Updating files: 57% (255/447) Updating files: 58% (260/447) Updating files: 59% (264/447) Updating files: 60% (269/447) Updating files: 61% (273/447) Updating files: 62% (278/447) Updating files: 63% (282/447) Updating files: 64% (287/447) Updating files: 65% (291/447) Updating files: 66% (296/447) Updating files: 67% (300/447) Updating files: 68% (304/447) Updating files: 68% (305/447) Updating files: 69% (309/447) Updating files: 70% (313/447) Updating files: 71% (318/447) Updating files: 72% (322/447) Updating files: 73% (327/447) Updating files: 74% (331/447) Updating files: 75% (336/447) Updating files: 76% (340/447) Updating files: 77% (345/447) Updating files: 78% (349/447) Updating files: 79% (354/447) Updating files: 80% (358/447) Updating files: 81% (363/447) Updating files: 82% (367/447) Updating files: 83% (372/447) Updating files: 84% (376/447) Updating files: 85% (380/447) Updating files: 86% (385/447) Updating files: 87% (389/447) Updating files: 88% (394/447) Updating files: 89% (398/447) Updating files: 90% (403/447) Updating files: 91% (407/447) Updating files: 92% (412/447) Updating files: 93% (416/447) Updating files: 94% (421/447) Updating files: 95% (425/447) Updating files: 96% (430/447) Updating files: 97% (434/447) Updating files: 98% (439/447) Updating files: 99% (443/447) Updating files: 100% (447/447) Updating files: 100% (447/447), done. 2023-06-22T08:17:10.908 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 97dcc0539df 0.86 2023-06-22T08:17:10.915 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.86 2023-06-22T08:17:11.034 INFO:tasks.workunit.client.0.smithi023.stdout:v0.87 2023-06-22T08:17:11.034 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:11.034 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 87 -eq 48 ']' 2023-06-22T08:17:11.035 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.87 2023-06-22T08:17:11.035 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.87 2023-06-22T08:17:11.035 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.87 2023-06-22T08:17:11.372 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:17:11.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:17:30.138 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2023-06-22T08:17:30.276 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at c51c8f9d80f 0.87 2023-06-22T08:17:30.279 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.87 2023-06-22T08:17:30.307 INFO:tasks.workunit.client.0.smithi023.stdout:v0.88 2023-06-22T08:17:30.307 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:30.308 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 88 -eq 48 ']' 2023-06-22T08:17:30.308 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.88 2023-06-22T08:17:30.308 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.88 2023-06-22T08:17:30.309 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.88 2023-06-22T08:17:32.858 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 51% (172/335) Updating files: 52% (175/335) Updating files: 53% (178/335) Updating files: 54% (181/335) Updating files: 55% (185/335) Updating files: 56% (188/335) Updating files: 57% (191/335) Updating files: 58% (195/335) Updating files: 59% (198/335) Updating files: 60% (201/335) Updating files: 61% (205/335) Updating files: 62% (208/335) Updating files: 63% (212/335) Updating files: 64% (215/335) Updating files: 65% (218/335) Updating files: 66% (222/335) Updating files: 67% (225/335) Updating files: 68% (228/335) Updating files: 69% (232/335) Updating files: 70% (235/335) Updating files: 71% (238/335) Updating files: 72% (242/335) Updating files: 73% (245/335) Updating files: 74% (248/335) Updating files: 75% (252/335) Updating files: 76% (255/335) Updating files: 77% (258/335) Updating files: 78% (262/335) Updating files: 79% (265/335) Updating files: 80% (268/335) Updating files: 81% (272/335) Updating files: 82% (275/335) Updating files: 83% (279/335) Updating files: 84% (282/335) Updating files: 85% (285/335) Updating files: 86% (289/335) Updating files: 87% (292/335) Updating files: 88% (295/335) Updating files: 89% (299/335) Updating files: 90% (302/335) Updating files: 91% (305/335) Updating files: 92% (309/335) Updating files: 93% (312/335) Updating files: 94% (315/335) Updating files: 95% (319/335) Updating files: 96% (322/335) Updating files: 97% (325/335) Updating files: 98% (329/335) Updating files: 99% (332/335) Updating files: 100% (335/335) Updating files: 100% (335/335), done. 2023-06-22T08:17:32.960 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 4be687bf448 0.88 2023-06-22T08:17:32.967 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.88 2023-06-22T08:17:33.073 INFO:tasks.workunit.client.0.smithi023.stdout:v0.89 2023-06-22T08:17:33.073 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:33.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 89 -eq 48 ']' 2023-06-22T08:17:33.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.89 2023-06-22T08:17:33.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.89 2023-06-22T08:17:33.074 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.89 2023-06-22T08:17:34.816 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 73% (145/197) Updating files: 74% (146/197) Updating files: 75% (148/197) Updating files: 76% (150/197) Updating files: 77% (152/197) Updating files: 78% (154/197) Updating files: 79% (156/197) Updating files: 80% (158/197) Updating files: 81% (160/197) Updating files: 82% (162/197) Updating files: 83% (164/197) Updating files: 84% (166/197) Updating files: 85% (168/197) Updating files: 86% (170/197) Updating files: 87% (172/197) Updating files: 88% (174/197) Updating files: 89% (176/197) Updating files: 90% (178/197) Updating files: 91% (180/197) Updating files: 92% (182/197) Updating files: 93% (184/197) Updating files: 94% (186/197) Updating files: 95% (188/197) Updating files: 96% (190/197) Updating files: 97% (192/197) Updating files: 98% (194/197) Updating files: 99% (196/197) Updating files: 100% (197/197) Updating files: 100% (197/197), done. 2023-06-22T08:17:34.930 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 68fdc0f68e6 0.89 2023-06-22T08:17:34.936 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.89 2023-06-22T08:17:35.315 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:35.315 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 90 -eq 48 ']' 2023-06-22T08:17:35.316 INFO:tasks.workunit.client.0.smithi023.stdout:v0.90 2023-06-22T08:17:35.316 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.90 2023-06-22T08:17:35.317 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo v0.90 2023-06-22T08:17:35.317 INFO:tasks.workunit.client.0.smithi023.stderr:+ git reset --hard v0.90 2023-06-22T08:17:44.380 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:17:44.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:17:45.325 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 63% (167/265) Updating files: 64% (170/265) Updating files: 65% (173/265) Updating files: 66% (175/265) Updating files: 67% (178/265) Updating files: 68% (181/265) Updating files: 69% (183/265) Updating files: 70% (186/265) Updating files: 71% (189/265) Updating files: 72% (191/265) Updating files: 73% (194/265) Updating files: 74% (197/265) Updating files: 75% (199/265) Updating files: 76% (202/265) Updating files: 77% (205/265) Updating files: 78% (207/265) Updating files: 79% (210/265) Updating files: 80% (212/265) Updating files: 81% (215/265) Updating files: 82% (218/265) Updating files: 83% (220/265) Updating files: 84% (223/265) Updating files: 85% (226/265) Updating files: 86% (228/265) Updating files: 87% (231/265) Updating files: 88% (234/265) Updating files: 89% (236/265) Updating files: 89% (238/265) Updating files: 90% (239/265) Updating files: 91% (242/265) Updating files: 92% (244/265) Updating files: 93% (247/265) Updating files: 94% (250/265) Updating files: 95% (252/265) Updating files: 96% (255/265) Updating files: 97% (258/265) Updating files: 98% (260/265) Updating files: 99% (263/265) Updating files: 100% (265/265) Updating files: 100% (265/265), done. 2023-06-22T08:17:45.440 INFO:tasks.workunit.client.0.smithi023.stdout:HEAD is now at 08bd1e1eee3 0.90 2023-06-22T08:17:45.445 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/v0.90 2023-06-22T08:17:45.509 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.1 2023-06-22T08:17:45.509 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:45.509 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 1 -eq 48 ']' 2023-06-22T08:17:45.510 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.1 2023-06-22T08:17:45.510 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.1 2023-06-22T08:17:45.510 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.1 2023-06-22T08:17:45.602 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:49.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:49.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:49.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -eq 48 ']' 2023-06-22T08:17:49.005 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.2 2023-06-22T08:17:49.006 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.2 2023-06-22T08:17:49.006 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.2 2023-06-22T08:17:49.006 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.2 2023-06-22T08:17:49.006 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:49.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:49.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:49.663 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 3 -eq 48 ']' 2023-06-22T08:17:49.663 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.3 2023-06-22T08:17:49.663 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.3 2023-06-22T08:17:49.663 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.3 2023-06-22T08:17:49.664 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.3 2023-06-22T08:17:49.664 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:50.567 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:50.567 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:50.567 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 4 -eq 48 ']' 2023-06-22T08:17:50.568 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.4 2023-06-22T08:17:50.568 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.4 2023-06-22T08:17:50.569 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.4 2023-06-22T08:17:50.569 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.4 2023-06-22T08:17:50.569 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:51.310 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:51.316 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:51.316 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 5 -eq 48 ']' 2023-06-22T08:17:51.316 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.5 2023-06-22T08:17:51.316 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.5 2023-06-22T08:17:51.317 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.5 2023-06-22T08:17:51.317 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.5 2023-06-22T08:17:51.317 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:51.900 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:51.900 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:51.900 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 6 -eq 48 ']' 2023-06-22T08:17:51.901 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.6 2023-06-22T08:17:51.901 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.6 2023-06-22T08:17:51.901 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.6 2023-06-22T08:17:51.902 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.6 2023-06-22T08:17:51.902 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:52.460 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:52.462 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:52.462 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 7 -eq 48 ']' 2023-06-22T08:17:52.462 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.7 2023-06-22T08:17:52.462 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.7 2023-06-22T08:17:52.463 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.7 2023-06-22T08:17:52.463 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.7 2023-06-22T08:17:52.463 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:53.128 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:53.128 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:53.128 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 8 -eq 48 ']' 2023-06-22T08:17:53.129 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.8 2023-06-22T08:17:53.129 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.8 2023-06-22T08:17:53.129 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.8 2023-06-22T08:17:53.129 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.8 2023-06-22T08:17:53.129 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:53.866 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:53.867 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:53.867 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 9 -eq 48 ']' 2023-06-22T08:17:53.867 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.9 2023-06-22T08:17:53.867 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.9 2023-06-22T08:17:53.867 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.9 2023-06-22T08:17:53.868 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.9 2023-06-22T08:17:53.868 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:54.492 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:54.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:54.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 10 -eq 48 ']' 2023-06-22T08:17:54.493 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.10 2023-06-22T08:17:54.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.10 2023-06-22T08:17:54.494 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.10 2023-06-22T08:17:54.494 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.10 2023-06-22T08:17:54.494 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:55.328 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:55.329 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:55.329 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 11 -eq 48 ']' 2023-06-22T08:17:55.329 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.11 2023-06-22T08:17:55.330 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.11 2023-06-22T08:17:55.330 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.11 2023-06-22T08:17:55.330 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.11 2023-06-22T08:17:55.330 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:55.945 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:55.945 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:55.946 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 12 -eq 48 ']' 2023-06-22T08:17:55.946 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.12 2023-06-22T08:17:55.946 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.12 2023-06-22T08:17:55.947 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.12 2023-06-22T08:17:55.947 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.12 2023-06-22T08:17:55.947 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:56.562 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:56.562 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:56.563 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 13 -eq 48 ']' 2023-06-22T08:17:56.563 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.13 2023-06-22T08:17:56.563 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.13 2023-06-22T08:17:56.564 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.13 2023-06-22T08:17:56.564 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.13 2023-06-22T08:17:56.564 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:57.098 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:57.098 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:57.099 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 14 -eq 48 ']' 2023-06-22T08:17:57.099 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.14 2023-06-22T08:17:57.099 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.14 2023-06-22T08:17:57.100 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.14 2023-06-22T08:17:57.100 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.14 2023-06-22T08:17:57.100 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:57.653 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:57.654 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:57.654 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 15 -eq 48 ']' 2023-06-22T08:17:57.654 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.15 2023-06-22T08:17:57.655 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.15 2023-06-22T08:17:57.655 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.15 2023-06-22T08:17:57.655 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.15 2023-06-22T08:17:57.655 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:58.383 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:58.383 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:58.383 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 16 -eq 48 ']' 2023-06-22T08:17:58.383 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.16 2023-06-22T08:17:58.384 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.16 2023-06-22T08:17:58.384 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.16 2023-06-22T08:17:58.384 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.16 2023-06-22T08:17:58.385 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:58.940 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:58.941 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:58.941 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 17 -eq 48 ']' 2023-06-22T08:17:58.941 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.17 2023-06-22T08:17:58.942 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.17 2023-06-22T08:17:58.942 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.17 2023-06-22T08:17:58.943 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.17 2023-06-22T08:17:58.943 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:17:59.477 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:17:59.478 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:17:59.478 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 18 -eq 48 ']' 2023-06-22T08:17:59.478 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.18 2023-06-22T08:17:59.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.18 2023-06-22T08:17:59.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.18 2023-06-22T08:17:59.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.18 2023-06-22T08:17:59.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:00.297 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:00.297 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:00.298 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 19 -eq 48 ']' 2023-06-22T08:18:00.298 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.19 2023-06-22T08:18:00.299 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.19 2023-06-22T08:18:00.299 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.19 2023-06-22T08:18:00.300 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.19 2023-06-22T08:18:00.300 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:01.122 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:01.123 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:01.123 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 20 -eq 48 ']' 2023-06-22T08:18:01.123 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.20 2023-06-22T08:18:01.124 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.20 2023-06-22T08:18:01.124 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.20 2023-06-22T08:18:01.124 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.20 2023-06-22T08:18:01.124 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:02.104 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:02.104 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:02.104 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 21 -eq 48 ']' 2023-06-22T08:18:02.105 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.21 2023-06-22T08:18:02.106 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.21 2023-06-22T08:18:02.106 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.21 2023-06-22T08:18:02.106 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.21 2023-06-22T08:18:02.106 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:03.054 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:03.055 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:03.055 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 22 -eq 48 ']' 2023-06-22T08:18:03.055 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.22 2023-06-22T08:18:03.055 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.22 2023-06-22T08:18:03.056 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.22 2023-06-22T08:18:03.056 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.22 2023-06-22T08:18:03.056 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:04.077 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:04.078 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:04.078 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 23 -eq 48 ']' 2023-06-22T08:18:04.078 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.23 2023-06-22T08:18:04.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.23 2023-06-22T08:18:04.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.23 2023-06-22T08:18:04.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.23 2023-06-22T08:18:04.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:05.031 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:05.032 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:05.032 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 24 -eq 48 ']' 2023-06-22T08:18:05.032 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.24 2023-06-22T08:18:05.032 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.24 2023-06-22T08:18:05.033 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.24 2023-06-22T08:18:05.033 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.24 2023-06-22T08:18:05.033 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:06.292 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:06.292 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:06.293 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 25 -eq 48 ']' 2023-06-22T08:18:06.293 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.25 2023-06-22T08:18:06.293 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.25 2023-06-22T08:18:06.293 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.25 2023-06-22T08:18:06.294 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.25 2023-06-22T08:18:06.294 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:07.024 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:07.024 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:07.024 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 26 -eq 48 ']' 2023-06-22T08:18:07.025 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.26 2023-06-22T08:18:07.026 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.26 2023-06-22T08:18:07.026 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.26 2023-06-22T08:18:07.026 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.26 2023-06-22T08:18:07.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:07.821 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:07.822 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:07.822 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 27 -eq 48 ']' 2023-06-22T08:18:07.822 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.27 2023-06-22T08:18:07.823 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.27 2023-06-22T08:18:07.823 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.27 2023-06-22T08:18:07.823 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.27 2023-06-22T08:18:07.823 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:08.618 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:08.619 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:08.619 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 28 -eq 48 ']' 2023-06-22T08:18:08.619 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.28 2023-06-22T08:18:08.620 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.28 2023-06-22T08:18:08.620 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.28 2023-06-22T08:18:08.620 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.28 2023-06-22T08:18:08.620 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:09.173 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:09.174 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:09.174 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 29 -eq 48 ']' 2023-06-22T08:18:09.174 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.29 2023-06-22T08:18:09.175 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.29 2023-06-22T08:18:09.175 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.29 2023-06-22T08:18:09.175 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.29 2023-06-22T08:18:09.175 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:09.828 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:09.828 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:09.828 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 30 -eq 48 ']' 2023-06-22T08:18:09.829 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.30 2023-06-22T08:18:09.829 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.30 2023-06-22T08:18:09.829 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.30 2023-06-22T08:18:09.829 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.30 2023-06-22T08:18:09.829 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:10.710 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:10.711 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:10.711 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 31 -eq 48 ']' 2023-06-22T08:18:10.711 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.31 2023-06-22T08:18:10.711 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.31 2023-06-22T08:18:10.712 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.31 2023-06-22T08:18:10.712 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.31 2023-06-22T08:18:10.713 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:11.555 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:11.556 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:11.556 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 32 -eq 48 ']' 2023-06-22T08:18:11.556 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.32 2023-06-22T08:18:11.557 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.32 2023-06-22T08:18:11.557 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.32 2023-06-22T08:18:11.557 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.32 2023-06-22T08:18:11.557 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:12.390 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:12.391 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:12.391 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 33 -eq 48 ']' 2023-06-22T08:18:12.391 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.33 2023-06-22T08:18:12.391 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.33 2023-06-22T08:18:12.392 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.33 2023-06-22T08:18:12.392 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.33 2023-06-22T08:18:12.392 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:13.221 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:13.221 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:13.221 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 34 -eq 48 ']' 2023-06-22T08:18:13.222 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.34 2023-06-22T08:18:13.223 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.34 2023-06-22T08:18:13.223 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.34 2023-06-22T08:18:13.223 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.34 2023-06-22T08:18:13.223 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:13.998 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:13.998 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:13.998 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 35 -eq 48 ']' 2023-06-22T08:18:13.999 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.35 2023-06-22T08:18:13.999 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.35 2023-06-22T08:18:13.999 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.35 2023-06-22T08:18:13.999 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.35 2023-06-22T08:18:14.000 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:14.484 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:18:14.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:18:14.822 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:14.823 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:14.823 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 36 -eq 48 ']' 2023-06-22T08:18:14.823 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.36 2023-06-22T08:18:14.823 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.36 2023-06-22T08:18:14.824 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.36 2023-06-22T08:18:14.824 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.36 2023-06-22T08:18:14.824 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:15.772 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:15.772 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:15.772 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 37 -eq 48 ']' 2023-06-22T08:18:15.773 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.37 2023-06-22T08:18:15.773 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.37 2023-06-22T08:18:15.773 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.37 2023-06-22T08:18:15.774 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.37 2023-06-22T08:18:15.774 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:16.423 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:16.424 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:16.424 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 38 -eq 48 ']' 2023-06-22T08:18:16.424 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.38 2023-06-22T08:18:16.425 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.38 2023-06-22T08:18:16.425 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.38 2023-06-22T08:18:16.425 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.38 2023-06-22T08:18:16.425 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:17.329 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:17.330 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:17.330 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 39 -eq 48 ']' 2023-06-22T08:18:17.330 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.39 2023-06-22T08:18:17.331 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.39 2023-06-22T08:18:17.331 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.39 2023-06-22T08:18:17.331 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.39 2023-06-22T08:18:17.331 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:18.196 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:18.197 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:18.197 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 40 -eq 48 ']' 2023-06-22T08:18:18.197 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.40 2023-06-22T08:18:18.198 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.40 2023-06-22T08:18:18.198 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.40 2023-06-22T08:18:18.198 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.40 2023-06-22T08:18:18.198 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:19.057 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:19.057 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:19.058 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 41 -eq 48 ']' 2023-06-22T08:18:19.058 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.41 2023-06-22T08:18:19.058 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.41 2023-06-22T08:18:19.059 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.41 2023-06-22T08:18:19.059 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.41 2023-06-22T08:18:19.059 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:20.141 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:20.142 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:20.142 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 42 -eq 48 ']' 2023-06-22T08:18:20.142 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.42 2023-06-22T08:18:20.143 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.42 2023-06-22T08:18:20.143 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.42 2023-06-22T08:18:20.143 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.42 2023-06-22T08:18:20.143 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:20.920 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:20.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:20.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 43 -eq 48 ']' 2023-06-22T08:18:20.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.43 2023-06-22T08:18:20.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.43 2023-06-22T08:18:20.922 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.43 2023-06-22T08:18:20.923 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.43 2023-06-22T08:18:20.923 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:22.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:22.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:22.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 44 -eq 48 ']' 2023-06-22T08:18:22.008 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.44 2023-06-22T08:18:22.009 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.44 2023-06-22T08:18:22.009 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.44 2023-06-22T08:18:22.009 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.44 2023-06-22T08:18:22.010 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:22.933 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:22.933 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:22.933 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 45 -eq 48 ']' 2023-06-22T08:18:22.933 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.45 2023-06-22T08:18:22.934 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.45 2023-06-22T08:18:22.934 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.45 2023-06-22T08:18:22.934 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.45 2023-06-22T08:18:22.934 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:23.775 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:23.775 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:23.775 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 46 -eq 48 ']' 2023-06-22T08:18:23.776 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.46 2023-06-22T08:18:23.776 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.46 2023-06-22T08:18:23.776 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.46 2023-06-22T08:18:23.777 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.46 2023-06-22T08:18:23.777 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:24.593 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:24.594 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:24.594 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 47 -eq 48 ']' 2023-06-22T08:18:24.594 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.47 2023-06-22T08:18:24.594 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.47 2023-06-22T08:18:24.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.47 2023-06-22T08:18:24.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.47 2023-06-22T08:18:24.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:25.380 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:25.381 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:25.381 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 48 -eq 48 ']' 2023-06-22T08:18:25.381 INFO:tasks.workunit.client.0.smithi023.stderr:+ continue 2023-06-22T08:18:25.381 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:25.381 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 49 -eq 48 ']' 2023-06-22T08:18:25.381 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.49 2023-06-22T08:18:25.382 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.49 2023-06-22T08:18:25.382 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.49 2023-06-22T08:18:25.382 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.49 2023-06-22T08:18:25.383 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:26.113 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:26.113 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:26.113 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 50 -eq 48 ']' 2023-06-22T08:18:26.114 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.50 2023-06-22T08:18:26.114 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.50 2023-06-22T08:18:26.114 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.50 2023-06-22T08:18:26.115 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.50 2023-06-22T08:18:26.115 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:26.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:26.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:26.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 51 -eq 48 ']' 2023-06-22T08:18:26.782 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.51 2023-06-22T08:18:26.783 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.51 2023-06-22T08:18:26.783 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.51 2023-06-22T08:18:26.783 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.51 2023-06-22T08:18:26.783 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:27.576 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:27.577 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:27.577 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 52 -eq 48 ']' 2023-06-22T08:18:27.577 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.52 2023-06-22T08:18:27.577 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.52 2023-06-22T08:18:27.578 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.52 2023-06-22T08:18:27.579 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.52 2023-06-22T08:18:27.579 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:28.473 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:28.474 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:28.474 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 53 -eq 48 ']' 2023-06-22T08:18:28.474 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.53 2023-06-22T08:18:28.475 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.53 2023-06-22T08:18:28.475 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.53 2023-06-22T08:18:28.475 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.53 2023-06-22T08:18:28.475 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:29.427 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:29.428 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:29.428 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 54 -eq 48 ']' 2023-06-22T08:18:29.428 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.54 2023-06-22T08:18:29.429 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.54 2023-06-22T08:18:29.429 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.54 2023-06-22T08:18:29.429 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.54 2023-06-22T08:18:29.429 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:30.252 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:30.253 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:30.253 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 55 -eq 48 ']' 2023-06-22T08:18:30.253 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.55 2023-06-22T08:18:30.254 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.55 2023-06-22T08:18:30.254 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.55 2023-06-22T08:18:30.254 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.55 2023-06-22T08:18:30.254 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:31.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:31.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:31.003 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 56 -eq 48 ']' 2023-06-22T08:18:31.003 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.56 2023-06-22T08:18:31.003 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.56 2023-06-22T08:18:31.004 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.56 2023-06-22T08:18:31.004 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.56 2023-06-22T08:18:31.004 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:32.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:32.002 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:32.003 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 57 -eq 48 ']' 2023-06-22T08:18:32.003 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.57 2023-06-22T08:18:32.003 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.57 2023-06-22T08:18:32.004 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.57 2023-06-22T08:18:32.004 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.57 2023-06-22T08:18:32.004 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:32.797 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:32.797 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:32.798 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 58 -eq 48 ']' 2023-06-22T08:18:32.798 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.58 2023-06-22T08:18:32.798 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.58 2023-06-22T08:18:32.799 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.58 2023-06-22T08:18:32.799 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.58 2023-06-22T08:18:32.799 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:33.764 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:33.764 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:33.765 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 59 -eq 48 ']' 2023-06-22T08:18:33.765 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.59 2023-06-22T08:18:33.765 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.59 2023-06-22T08:18:33.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.59 2023-06-22T08:18:33.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.59 2023-06-22T08:18:33.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:34.780 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:34.780 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:34.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 60 -eq 48 ']' 2023-06-22T08:18:34.781 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.60 2023-06-22T08:18:34.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.60 2023-06-22T08:18:34.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.60 2023-06-22T08:18:34.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.60 2023-06-22T08:18:34.782 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:35.919 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:35.920 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:35.920 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 61 -eq 48 ']' 2023-06-22T08:18:35.920 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.61 2023-06-22T08:18:35.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.61 2023-06-22T08:18:35.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.61 2023-06-22T08:18:35.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.61 2023-06-22T08:18:35.921 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:36.831 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:36.831 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:36.831 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 62 -eq 48 ']' 2023-06-22T08:18:36.832 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.62 2023-06-22T08:18:36.833 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.62 2023-06-22T08:18:36.833 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.62 2023-06-22T08:18:36.833 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.62 2023-06-22T08:18:36.834 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:37.676 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:18:37.677 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:18:37.677 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 63 -eq 48 ']' 2023-06-22T08:18:37.677 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.63 2023-06-22T08:18:37.678 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.63 2023-06-22T08:18:37.678 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.63 2023-06-22T08:18:37.678 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.63 2023-06-22T08:18:37.678 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:18:37.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.964+0000 7fd4e4dff700 1 Processor -- start 2023-06-22T08:18:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.964+0000 7fd4e4dff700 1 -- start start 2023-06-22T08:18:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.965+0000 7fd4e4dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e00cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.965+0000 7fd4e4dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd4e00c9fd0 0x7fd4e015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.965+0000 7fd4e4dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e015c370 con 0x7fd4e0158880 2023-06-22T08:18:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.965+0000 7fd4e4dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e01530b0 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.965+0000 7fd4e4dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e00cf6f0 con 0x7fd4e0152ce0 2023-06-22T08:18:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.966+0000 7fd4de59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e00cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.966+0000 7fd4de59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e00cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35896/0 (socket says 172.21.15.23:35896) 2023-06-22T08:18:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.966+0000 7fd4de59c700 1 -- 172.21.15.23:0/1027110922 learned_addr learned my addr 172.21.15.23:0/1027110922 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:37.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.966+0000 7fd4ddd9b700 1 --2- 172.21.15.23:0/1027110922 >> v2:172.21.15.23:3300/0 conn(0x7fd4e00c9fd0 0x7fd4e015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:37.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.966+0000 7fd4de59c700 1 -- 172.21.15.23:0/1027110922 >> v2:172.21.15.23:3300/0 conn(0x7fd4e00c9fd0 msgr2=0x7fd4e015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:18:37.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.966+0000 7fd4de59c700 1 --2- 172.21.15.23:0/1027110922 >> v2:172.21.15.23:3300/0 conn(0x7fd4e00c9fd0 0x7fd4e015c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:37.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.966+0000 7fd4de59c700 1 -- 172.21.15.23:0/1027110922 >> v1:172.21.15.67:6789/0 conn(0x7fd4e0158880 legacy=0x7fd4e0158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:37.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.966+0000 7fd4de59c700 1 -- 172.21.15.23:0/1027110922 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e015bda0 con 0x7fd4e0152ce0 2023-06-22T08:18:37.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.967+0000 7fd4de59c700 1 --2- 172.21.15.23:0/1027110922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e00cf1b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00098f0 tx=0x7fd4d0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5534707676b683c0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:37.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.967+0000 7fd4dd59a700 1 -- 172.21.15.23:0/1027110922 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0011830 con 0x7fd4e0152ce0 2023-06-22T08:18:37.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.967+0000 7fd4dd59a700 1 -- 172.21.15.23:0/1027110922 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d0011990 con 0x7fd4e0152ce0 2023-06-22T08:18:37.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.967+0000 7fd4dd59a700 1 -- 172.21.15.23:0/1027110922 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0011d20 con 0x7fd4e0152ce0 2023-06-22T08:18:37.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.967+0000 7fd4e4dff700 1 -- 172.21.15.23:0/1027110922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 msgr2=0x7fd4e00cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:37.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.967+0000 7fd4e4dff700 1 --2- 172.21.15.23:0/1027110922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e00cf1b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00098f0 tx=0x7fd4d0009c00 comp rx=0 tx=0).stop 2023-06-22T08:18:37.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.967+0000 7fd4e4dff700 1 -- 172.21.15.23:0/1027110922 shutdown_connections 2023-06-22T08:18:37.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.967+0000 7fd4e4dff700 1 --2- 172.21.15.23:0/1027110922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e00cf1b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:37.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.967+0000 7fd4e4dff700 1 --2- 172.21.15.23:0/1027110922 >> v2:172.21.15.23:3300/0 conn(0x7fd4e00c9fd0 0x7fd4e015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.967+0000 7fd4e4dff700 1 -- 172.21.15.23:0/1027110922 >> 172.21.15.23:0/1027110922 conn(0x7fd4e006ae30 msgr2=0x7fd4e006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.968+0000 7fd4e4dff700 1 -- 172.21.15.23:0/1027110922 shutdown_connections 2023-06-22T08:18:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.968+0000 7fd4e4dff700 1 -- 172.21.15.23:0/1027110922 wait complete. 2023-06-22T08:18:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.970+0000 7fd4e4dff700 1 Processor -- start 2023-06-22T08:18:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.970+0000 7fd4e4dff700 1 -- start start 2023-06-22T08:18:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4e4dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e00c9fd0 0x7fd4e01e4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4e4dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e0147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4e4dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e0158880 0x7fd4e01ee2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4e4dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e00b49b0 con 0x7fd4e0158880 2023-06-22T08:18:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4e4dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e00b4150 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4e4dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e01e4ac0 con 0x7fd4e0152ce0 2023-06-22T08:18:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4ddd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e0147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4de59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e00c9fd0 0x7fd4e01e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4ded9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e0158880 0x7fd4e01ee2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4de59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e00c9fd0 0x7fd4e01e4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:35504/0 (socket says 172.21.15.23:35504) 2023-06-22T08:18:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4ddd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e0147ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35908/0 (socket says 172.21.15.23:35908) 2023-06-22T08:18:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.971+0000 7fd4de59c700 1 -- 172.21.15.23:0/3370188710 learned_addr learned my addr 172.21.15.23:0/3370188710 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4de59c700 1 -- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 msgr2=0x7fd4e0147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4de59c700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e0147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4de59c700 1 -- 172.21.15.23:0/3370188710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e0158880 msgr2=0x7fd4e01ee2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4de59c700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e0158880 0x7fd4e01ee2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4de59c700 1 -- 172.21.15.23:0/3370188710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d0009550 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4ddd9b700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e0147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4de59c700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e00c9fd0 0x7fd4e01e4580 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0009c40 tx=0x7fd4d0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4ceffd700 1 -- 172.21.15.23:0/3370188710 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0004400 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4ceffd700 1 -- 172.21.15.23:0/3370188710 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d001f070 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4ceffd700 1 -- 172.21.15.23:0/3370188710 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00038e0 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.972+0000 7fd4e4dff700 1 -- 172.21.15.23:0/3370188710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e01e40e0 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.973+0000 7fd4e4dff700 1 -- 172.21.15.23:0/3370188710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e00b46d0 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.974+0000 7fd4e4dff700 1 -- 172.21.15.23:0/3370188710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e004ea20 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.975+0000 7fd4ceffd700 1 -- 172.21.15.23:0/3370188710 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd4d0021030 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.976+0000 7fd4ceffd700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c006ace0 0x7fd4c006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.976+0000 7fd4ceffd700 1 -- 172.21.15.23:0/3370188710 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7fd4d008ae30 con 0x7fd4e00c9fd0 2023-06-22T08:18:37.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.976+0000 7fd4ddd9b700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c006ace0 0x7fd4c006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:37.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.977+0000 7fd4ddd9b700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c006ace0 0x7fd4c006d190 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00b3bf0 tx=0x7fd4d4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:37.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:37.983+0000 7fd4ceffd700 1 -- 172.21.15.23:0/3370188710 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd4d0055840 con 0x7fd4e00c9fd0 2023-06-22T08:18:38.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.342+0000 7fd4e4dff700 1 -- 172.21.15.23:0/3370188710 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4e0152500 con 0x7fd4e00c9fd0 2023-06-22T08:18:38.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.343+0000 7fd4ceffd700 1 -- 172.21.15.23:0/3370188710 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6272 (secure 0 0 0) 0x7fd4d0058e60 con 0x7fd4e00c9fd0 2023-06-22T08:18:38.346 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:38.346 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:30.949335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:38.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.346+0000 7fd4e4dff700 1 -- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c006ace0 msgr2=0x7fd4c006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.346+0000 7fd4e4dff700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c006ace0 0x7fd4c006d190 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00b3bf0 tx=0x7fd4d4009210 comp rx=0 tx=0).stop 2023-06-22T08:18:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.346+0000 7fd4e4dff700 1 -- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e00c9fd0 msgr2=0x7fd4e01e4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.346+0000 7fd4e4dff700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e00c9fd0 0x7fd4e01e4580 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0009c40 tx=0x7fd4d0004750 comp rx=0 tx=0).stop 2023-06-22T08:18:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.346+0000 7fd4e4dff700 1 -- 172.21.15.23:0/3370188710 shutdown_connections 2023-06-22T08:18:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.346+0000 7fd4e4dff700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e0158880 0x7fd4e01ee2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.346+0000 7fd4e4dff700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd4c006ace0 0x7fd4c006d190 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.346+0000 7fd4e4dff700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4e00c9fd0 0x7fd4e01e4580 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.346+0000 7fd4e4dff700 1 --2- 172.21.15.23:0/3370188710 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd4e0152ce0 0x7fd4e0147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.346+0000 7fd4e4dff700 1 -- 172.21.15.23:0/3370188710 >> 172.21.15.23:0/3370188710 conn(0x7fd4e006ae30 msgr2=0x7fd4e00c4410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:38.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.347+0000 7fd4e4dff700 1 -- 172.21.15.23:0/3370188710 shutdown_connections 2023-06-22T08:18:38.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.347+0000 7fd4e4dff700 1 -- 172.21.15.23:0/3370188710 wait complete. 2023-06-22T08:18:38.351 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 150 2023-06-22T08:18:38.369 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=4) 2023-06-22T08:18:38.370 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-22T08:18:38.370 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:18:38.384 INFO:tasks.ceph.mds.d:Stopped 2023-06-22T08:18:38.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:38.584 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.581+0000 7fd8963ff700 1 Processor -- start 2023-06-22T08:18:38.585 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.581+0000 7fd8963ff700 1 -- start start 2023-06-22T08:18:38.586 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd8963ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890152d70 0x7fd8900c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:38.586 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd8963ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd8900c45f0 0x7fd8900c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:38.586 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd8963ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8900c4fb0 con 0x7fd890159330 2023-06-22T08:18:38.586 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd8963ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8900c95e0 con 0x7fd8900c45f0 2023-06-22T08:18:38.587 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd8963ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89015bea0 con 0x7fd890152d70 2023-06-22T08:18:38.587 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd88ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890152d70 0x7fd8900c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:38.587 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd88f7fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd890159330 0x7fd8900cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:44822/0 (socket says 172.21.15.23:44822) 2023-06-22T08:18:38.587 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd88f7fe700 1 -- 172.21.15.23:0/3222624810 learned_addr learned my addr 172.21.15.23:0/3222624810 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:18:38.587 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd89499c700 1 --2- 172.21.15.23:0/3222624810 >> v2:172.21.15.23:3300/0 conn(0x7fd8900c45f0 0x7fd8900c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:38.588 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd88effd700 1 -- 172.21.15.23:0/3222624810 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464484037 0 0) 0x7fd8900c4fb0 con 0x7fd890159330 2023-06-22T08:18:38.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd88effd700 1 -- 172.21.15.23:0/3222624810 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8780035b0 con 0x7fd890159330 2023-06-22T08:18:38.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.583+0000 7fd88ffff700 1 -- 172.21.15.23:0/3222624810 >> v2:172.21.15.23:3300/0 conn(0x7fd8900c45f0 msgr2=0x7fd8900c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:18:38.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.584+0000 7fd88ffff700 1 --2- 172.21.15.23:0/3222624810 >> v2:172.21.15.23:3300/0 conn(0x7fd8900c45f0 0x7fd8900c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.584+0000 7fd88ffff700 1 -- 172.21.15.23:0/3222624810 >> v1:172.21.15.67:6789/0 conn(0x7fd890159330 legacy=0x7fd8900cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:38.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.584+0000 7fd88ffff700 1 -- 172.21.15.23:0/3222624810 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89015c160 con 0x7fd890152d70 2023-06-22T08:18:38.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.584+0000 7fd89499c700 1 --2- 172.21.15.23:0/3222624810 >> v2:172.21.15.23:3300/0 conn(0x7fd8900c45f0 0x7fd8900c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:38.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.584+0000 7fd88ffff700 1 --2- 172.21.15.23:0/3222624810 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890152d70 0x7fd8900c97d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd880009840 tx=0x7fd880009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c499c145025b8719 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:38.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.584+0000 7fd88effd700 1 -- 172.21.15.23:0/3222624810 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880011830 con 0x7fd890152d70 2023-06-22T08:18:38.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.584+0000 7fd88effd700 1 -- 172.21.15.23:0/3222624810 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd880011990 con 0x7fd890152d70 2023-06-22T08:18:38.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.584+0000 7fd88effd700 1 -- 172.21.15.23:0/3222624810 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880011d20 con 0x7fd890152d70 2023-06-22T08:18:38.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.585+0000 7fd8963ff700 1 -- 172.21.15.23:0/3222624810 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890152d70 msgr2=0x7fd8900c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:38.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.585+0000 7fd8963ff700 1 --2- 172.21.15.23:0/3222624810 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890152d70 0x7fd8900c97d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd880009840 tx=0x7fd880009b50 comp rx=0 tx=0).stop 2023-06-22T08:18:38.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.585+0000 7fd8963ff700 1 -- 172.21.15.23:0/3222624810 shutdown_connections 2023-06-22T08:18:38.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.585+0000 7fd8963ff700 1 --2- 172.21.15.23:0/3222624810 >> v2:172.21.15.23:3300/0 conn(0x7fd8900c45f0 0x7fd8900c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.585+0000 7fd8963ff700 1 --2- 172.21.15.23:0/3222624810 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890152d70 0x7fd8900c97d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.585+0000 7fd8963ff700 1 -- 172.21.15.23:0/3222624810 >> 172.21.15.23:0/3222624810 conn(0x7fd89006ae30 msgr2=0x7fd89006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:38.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.585+0000 7fd8963ff700 1 -- 172.21.15.23:0/3222624810 shutdown_connections 2023-06-22T08:18:38.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.585+0000 7fd8963ff700 1 -- 172.21.15.23:0/3222624810 wait complete. 2023-06-22T08:18:38.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.586+0000 7fd8963ff700 1 Processor -- start 2023-06-22T08:18:38.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.587+0000 7fd8963ff700 1 -- start start 2023-06-22T08:18:38.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.587+0000 7fd8963ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900c45f0 0x7fd8901f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:38.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.587+0000 7fd8963ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd890152d70 0x7fd89014eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:38.593 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.587+0000 7fd8963ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890159330 0x7fd8900c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:38.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.587+0000 7fd8963ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89014b0b0 con 0x7fd890152d70 2023-06-22T08:18:38.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.587+0000 7fd8963ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd890158460 con 0x7fd8900c45f0 2023-06-22T08:18:38.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd8963ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8901f1c50 con 0x7fd890159330 2023-06-22T08:18:38.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900c45f0 0x7fd8901f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:38.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd890152d70 0x7fd89014eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:38.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd89499c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890159330 0x7fd8900c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:38.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900c45f0 0x7fd8901f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:35506/0 (socket says 172.21.15.23:35506) 2023-06-22T08:18:38.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88ffff700 1 -- 172.21.15.23:0/634115905 learned_addr learned my addr 172.21.15.23:0/634115905 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:38.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88ffff700 1 -- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890159330 msgr2=0x7fd8900c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:38.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88ffff700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890159330 0x7fd8900c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88ffff700 1 -- 172.21.15.23:0/634115905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd890152d70 msgr2=0x7fd89014eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:38.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88ffff700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd890152d70 0x7fd89014eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88ffff700 1 -- 172.21.15.23:0/634115905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd888009a70 con 0x7fd8900c45f0 2023-06-22T08:18:38.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88f7fe700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd890152d70 0x7fd89014eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:38.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.588+0000 7fd88ffff700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900c45f0 0x7fd8901f20e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd89015bfe0 tx=0x7fd880004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:38.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.589+0000 7fd88cff9700 1 -- 172.21.15.23:0/634115905 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880004550 con 0x7fd8900c45f0 2023-06-22T08:18:38.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.589+0000 7fd88cff9700 1 -- 172.21.15.23:0/634115905 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88001f070 con 0x7fd8900c45f0 2023-06-22T08:18:38.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.589+0000 7fd88cff9700 1 -- 172.21.15.23:0/634115905 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8800038e0 con 0x7fd8900c45f0 2023-06-22T08:18:38.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.589+0000 7fd8963ff700 1 -- 172.21.15.23:0/634115905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8800094a0 con 0x7fd8900c45f0 2023-06-22T08:18:38.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.589+0000 7fd8963ff700 1 -- 172.21.15.23:0/634115905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd890158990 con 0x7fd8900c45f0 2023-06-22T08:18:38.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.590+0000 7fd8963ff700 1 -- 172.21.15.23:0/634115905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89004ea20 con 0x7fd8900c45f0 2023-06-22T08:18:38.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.591+0000 7fd88cff9700 1 -- 172.21.15.23:0/634115905 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd880003a40 con 0x7fd8900c45f0 2023-06-22T08:18:38.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.592+0000 7fd88cff9700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd87c06ace0 0x7fd87c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:38.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.592+0000 7fd88cff9700 1 -- 172.21.15.23:0/634115905 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7fd880021070 con 0x7fd8900c45f0 2023-06-22T08:18:38.598 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.592+0000 7fd88f7fe700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd87c06ace0 0x7fd87c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:38.598 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.592+0000 7fd88f7fe700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd87c06ace0 0x7fd87c06d190 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd8900ce080 tx=0x7fd884007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:38.599 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.596+0000 7fd88cff9700 1 -- 172.21.15.23:0/634115905 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd880055cb0 con 0x7fd8900c45f0 2023-06-22T08:18:38.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.853+0000 7fd8963ff700 1 -- 172.21.15.23:0/634115905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8900c9e00 con 0x7fd8900c45f0 2023-06-22T08:18:38.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.854+0000 7fd88cff9700 1 -- 172.21.15.23:0/634115905 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6272 (secure 0 0 0) 0x7fd8800592d0 con 0x7fd8900c45f0 2023-06-22T08:18:38.857 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:38.858 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:30.949335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:38.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.856+0000 7fd8963ff700 1 -- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd87c06ace0 msgr2=0x7fd87c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:38.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.856+0000 7fd8963ff700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd87c06ace0 0x7fd87c06d190 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fd8900ce080 tx=0x7fd884007dc0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.856+0000 7fd8963ff700 1 -- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900c45f0 msgr2=0x7fd8901f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:38.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.856+0000 7fd8963ff700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900c45f0 0x7fd8901f20e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd89015bfe0 tx=0x7fd880004750 comp rx=0 tx=0).stop 2023-06-22T08:18:38.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.857+0000 7fd8963ff700 1 -- 172.21.15.23:0/634115905 shutdown_connections 2023-06-22T08:18:38.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.857+0000 7fd8963ff700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd890159330 0x7fd8900c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.857+0000 7fd8963ff700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd87c06ace0 0x7fd87c06d190 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.857+0000 7fd8963ff700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8900c45f0 0x7fd8901f20e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.857+0000 7fd8963ff700 1 --2- 172.21.15.23:0/634115905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd890152d70 0x7fd89014eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:38.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.857+0000 7fd8963ff700 1 -- 172.21.15.23:0/634115905 >> 172.21.15.23:0/634115905 conn(0x7fd89006ae30 msgr2=0x7fd89006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:38.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.857+0000 7fd8963ff700 1 -- 172.21.15.23:0/634115905 shutdown_connections 2023-06-22T08:18:38.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:38.857+0000 7fd8963ff700 1 -- 172.21.15.23:0/634115905 wait complete. 2023-06-22T08:18:38.861 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 150 2023-06-22T08:18:38.877 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:18:40.878 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.104+0000 7f05507ff700 1 Processor -- start 2023-06-22T08:18:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.104+0000 7f05507ff700 1 -- start start 2023-06-22T08:18:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.105+0000 7f05507ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0548152d70 0x7f05480c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.105+0000 7f05507ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05480c45f0 0x7f05480c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.105+0000 7f05507ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480c4fb0 con 0x7f0548159330 2023-06-22T08:18:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.105+0000 7f05507ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05480c95e0 con 0x7f05480c45f0 2023-06-22T08:18:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.105+0000 7f05507ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054815bea0 con 0x7f0548152d70 2023-06-22T08:18:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054dd9a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f0548159330 0x7f05480cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59312/0 (socket says 172.21.15.23:59312) 2023-06-22T08:18:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054dd9a700 1 -- 172.21.15.23:0/2614218585 learned_addr learned my addr 172.21.15.23:0/2614218585 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:18:41.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054ed9c700 1 --2- 172.21.15.23:0/2614218585 >> v2:172.21.15.67:3300/0 conn(0x7f05480c45f0 0x7f05480c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:41.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054e59b700 1 --2- 172.21.15.23:0/2614218585 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0548152d70 0x7f05480c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:41.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054d599700 1 -- 172.21.15.23:0/2614218585 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190280252 0 0) 0x7f05480c4fb0 con 0x7f0548159330 2023-06-22T08:18:41.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054d599700 1 -- 172.21.15.23:0/2614218585 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05340035b0 con 0x7f0548159330 2023-06-22T08:18:41.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054ed9c700 1 -- 172.21.15.23:0/2614218585 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0548152d70 msgr2=0x7f05480c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:18:41.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054ed9c700 1 --2- 172.21.15.23:0/2614218585 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0548152d70 0x7f05480c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:41.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054ed9c700 1 -- 172.21.15.23:0/2614218585 >> v1:172.21.15.23:6789/0 conn(0x7f0548159330 legacy=0x7f05480cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:41.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054ed9c700 1 -- 172.21.15.23:0/2614218585 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054815c160 con 0x7f05480c45f0 2023-06-22T08:18:41.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.106+0000 7f054e59b700 1 --2- 172.21.15.23:0/2614218585 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0548152d70 0x7f05480c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:41.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.107+0000 7f054ed9c700 1 --2- 172.21.15.23:0/2614218585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05480c45f0 0x7f05480c4a70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0540009d90 tx=0x7f054000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48b4e37d94b44bf4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:41.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.107+0000 7f054d599700 1 -- 172.21.15.23:0/2614218585 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054000e070 con 0x7f05480c45f0 2023-06-22T08:18:41.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.107+0000 7f054d599700 1 -- 172.21.15.23:0/2614218585 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054000bb20 con 0x7f05480c45f0 2023-06-22T08:18:41.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.107+0000 7f054d599700 1 -- 172.21.15.23:0/2614218585 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540012430 con 0x7f05480c45f0 2023-06-22T08:18:41.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.107+0000 7f05507ff700 1 -- 172.21.15.23:0/2614218585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05480c45f0 msgr2=0x7f05480c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:41.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.107+0000 7f05507ff700 1 --2- 172.21.15.23:0/2614218585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05480c45f0 0x7f05480c4a70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0540009d90 tx=0x7f054000b330 comp rx=0 tx=0).stop 2023-06-22T08:18:41.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.108+0000 7f05507ff700 1 -- 172.21.15.23:0/2614218585 shutdown_connections 2023-06-22T08:18:41.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.108+0000 7f05507ff700 1 --2- 172.21.15.23:0/2614218585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05480c45f0 0x7f05480c4a70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:41.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.108+0000 7f05507ff700 1 --2- 172.21.15.23:0/2614218585 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0548152d70 0x7f05480c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:41.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.108+0000 7f05507ff700 1 -- 172.21.15.23:0/2614218585 >> 172.21.15.23:0/2614218585 conn(0x7f054806ae30 msgr2=0x7f054806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:41.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.108+0000 7f05507ff700 1 -- 172.21.15.23:0/2614218585 shutdown_connections 2023-06-22T08:18:41.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.108+0000 7f05507ff700 1 -- 172.21.15.23:0/2614218585 wait complete. 2023-06-22T08:18:41.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.109+0000 7f05507ff700 1 Processor -- start 2023-06-22T08:18:41.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.109+0000 7f05507ff700 1 -- start start 2023-06-22T08:18:41.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.110+0000 7f05507ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05480c45f0 0x7f0548153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:41.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.110+0000 7f05507ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0548152d70 0x7f0548153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:41.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.110+0000 7f05507ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0548159330 0x7f0548155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:41.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.110+0000 7f05507ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05480cf740 con 0x7f0548152d70 2023-06-22T08:18:41.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.110+0000 7f05507ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05481ef6e0 con 0x7f0548159330 2023-06-22T08:18:41.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.110+0000 7f05507ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05481ef180 con 0x7f05480c45f0 2023-06-22T08:18:41.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054dd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0548152d70 0x7f0548153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:41.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054dd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0548152d70 0x7f0548153d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40196/0 (socket says 172.21.15.23:40196) 2023-06-22T08:18:41.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054dd9a700 1 -- 172.21.15.23:0/396742383 learned_addr learned my addr 172.21.15.23:0/396742383 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:41.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054ed9c700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0548159330 0x7f0548155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:41.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054e59b700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05480c45f0 0x7f0548153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:41.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054dd9a700 1 -- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05480c45f0 msgr2=0x7f0548153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:41.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054dd9a700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05480c45f0 0x7f0548153850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:41.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054dd9a700 1 -- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0548159330 msgr2=0x7f0548155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:41.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054dd9a700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0548159330 0x7f0548155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:41.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054dd9a700 1 -- 172.21.15.23:0/396742383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05440094a0 con 0x7f0548152d70 2023-06-22T08:18:41.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f054dd9a700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0548152d70 0x7f0548153d90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f053800c4d0 tx=0x7f053800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:41.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.111+0000 7f053effd700 1 -- 172.21.15.23:0/396742383 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053800cea0 con 0x7f0548152d70 2023-06-22T08:18:41.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.112+0000 7f053effd700 1 -- 172.21.15.23:0/396742383 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0538005d70 con 0x7f0548152d70 2023-06-22T08:18:41.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.112+0000 7f05507ff700 1 -- 172.21.15.23:0/396742383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0540009a70 con 0x7f0548152d70 2023-06-22T08:18:41.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.112+0000 7f053effd700 1 -- 172.21.15.23:0/396742383 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05380072f0 con 0x7f0548152d70 2023-06-22T08:18:41.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.112+0000 7f05507ff700 1 -- 172.21.15.23:0/396742383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05480cd110 con 0x7f0548152d70 2023-06-22T08:18:41.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.113+0000 7f05507ff700 1 -- 172.21.15.23:0/396742383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0548159050 con 0x7f0548152d70 2023-06-22T08:18:41.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.114+0000 7f053effd700 1 -- 172.21.15.23:0/396742383 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0538007450 con 0x7f0548152d70 2023-06-22T08:18:41.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.115+0000 7f053effd700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f052c06ace0 0x7f052c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:41.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.115+0000 7f053effd700 1 -- 172.21.15.23:0/396742383 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f0538088da0 con 0x7f0548152d70 2023-06-22T08:18:41.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.115+0000 7f054e59b700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f052c06ace0 0x7f052c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:41.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.116+0000 7f054e59b700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f052c06ace0 0x7f052c06d190 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0544009d70 tx=0x7f0544009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:41.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.120+0000 7f053effd700 1 -- 172.21.15.23:0/396742383 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f05380538e0 con 0x7f0548152d70 2023-06-22T08:18:41.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.372+0000 7f05507ff700 1 -- 172.21.15.23:0/396742383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f054804ea20 con 0x7f0548152d70 2023-06-22T08:18:41.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.372+0000 7f053effd700 1 -- 172.21.15.23:0/396742383 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f0538056f00 con 0x7f0548152d70 2023-06-22T08:18:41.376 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:41.376 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:30.949335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 -- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f052c06ace0 msgr2=0x7f052c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f052c06ace0 0x7f052c06d190 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0544009d70 tx=0x7f0544009210 comp rx=0 tx=0).stop 2023-06-22T08:18:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 -- 172.21.15.23:0/396742383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0548152d70 msgr2=0x7f0548153d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0548152d70 0x7f0548153d90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f053800c4d0 tx=0x7f053800c7e0 comp rx=0 tx=0).stop 2023-06-22T08:18:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 -- 172.21.15.23:0/396742383 shutdown_connections 2023-06-22T08:18:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f052c06ace0 0x7f052c06d190 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:41.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0548159330 0x7f0548155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:41.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05480c45f0 0x7f0548153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:41.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 --2- 172.21.15.23:0/396742383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0548152d70 0x7f0548153d90 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:41.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 -- 172.21.15.23:0/396742383 >> 172.21.15.23:0/396742383 conn(0x7f054806ae30 msgr2=0x7f054814a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:41.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 -- 172.21.15.23:0/396742383 shutdown_connections 2023-06-22T08:18:41.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:41.375+0000 7f05507ff700 1 -- 172.21.15.23:0/396742383 wait complete. 2023-06-22T08:18:41.380 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 150 2023-06-22T08:18:41.396 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:18:43.396 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:43.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.604+0000 7f8b39bff700 1 Processor -- start 2023-06-22T08:18:43.607 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.604+0000 7f8b39bff700 1 -- start start 2023-06-22T08:18:43.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.605+0000 7f8b39bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b34159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:43.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.605+0000 7f8b39bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b340c9c30 con 0x7f8b340c97d0 2023-06-22T08:18:43.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.605+0000 7f8b39bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3415bea0 con 0x7f8b34152d70 2023-06-22T08:18:43.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.605+0000 7f8b39bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340c95e0 con 0x7f8b340c4d00 2023-06-22T08:18:43.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.606+0000 7f8b32ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f8b340c97d0 0x7f8b340c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59322/0 (socket says 172.21.15.23:59322) 2023-06-22T08:18:43.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.606+0000 7f8b32ffd700 1 -- 172.21.15.23:0/3074277300 learned_addr learned my addr 172.21.15.23:0/3074277300 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:18:43.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.606+0000 7f8b33fff700 1 --2- 172.21.15.23:0/3074277300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b34159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:43.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.606+0000 7f8b327fc700 1 -- 172.21.15.23:0/3074277300 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537992197 0 0) 0x7f8b340c9c30 con 0x7f8b340c97d0 2023-06-22T08:18:43.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.606+0000 7f8b327fc700 1 -- 172.21.15.23:0/3074277300 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b1c0035b0 con 0x7f8b340c97d0 2023-06-22T08:18:43.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.606+0000 7f8b327fc700 1 -- 172.21.15.23:0/3074277300 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321871787 0 0) 0x7f8b3415bea0 con 0x7f8b34152d70 2023-06-22T08:18:43.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.606+0000 7f8b327fc700 1 -- 172.21.15.23:0/3074277300 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b340c9c30 con 0x7f8b34152d70 2023-06-22T08:18:43.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.606+0000 7f8b33fff700 1 -- 172.21.15.23:0/3074277300 >> v1:172.21.15.67:6789/0 conn(0x7f8b34152d70 legacy=0x7f8b340cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:43.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.606+0000 7f8b33fff700 1 -- 172.21.15.23:0/3074277300 >> v1:172.21.15.23:6789/0 conn(0x7f8b340c97d0 legacy=0x7f8b340c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:43.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.606+0000 7f8b33fff700 1 -- 172.21.15.23:0/3074277300 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3415c160 con 0x7f8b340c4d00 2023-06-22T08:18:43.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.607+0000 7f8b33fff700 1 --2- 172.21.15.23:0/3074277300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b34159330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009dc0 tx=0x7f8b2c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b25bd1a70ad5d36a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:43.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.607+0000 7f8b327fc700 1 -- 172.21.15.23:0/3074277300 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c00e070 con 0x7f8b340c4d00 2023-06-22T08:18:43.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.607+0000 7f8b327fc700 1 -- 172.21.15.23:0/3074277300 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c00bb20 con 0x7f8b340c4d00 2023-06-22T08:18:43.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.607+0000 7f8b327fc700 1 -- 172.21.15.23:0/3074277300 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c00be90 con 0x7f8b340c4d00 2023-06-22T08:18:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.607+0000 7f8b39bff700 1 -- 172.21.15.23:0/3074277300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 msgr2=0x7f8b34159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.607+0000 7f8b39bff700 1 --2- 172.21.15.23:0/3074277300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b34159330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009dc0 tx=0x7f8b2c00b330 comp rx=0 tx=0).stop 2023-06-22T08:18:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.607+0000 7f8b39bff700 1 -- 172.21.15.23:0/3074277300 shutdown_connections 2023-06-22T08:18:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.607+0000 7f8b39bff700 1 --2- 172.21.15.23:0/3074277300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b34159330 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.607+0000 7f8b39bff700 1 -- 172.21.15.23:0/3074277300 >> 172.21.15.23:0/3074277300 conn(0x7f8b3406ae30 msgr2=0x7f8b3406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.607+0000 7f8b39bff700 1 -- 172.21.15.23:0/3074277300 shutdown_connections 2023-06-22T08:18:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.608+0000 7f8b39bff700 1 -- 172.21.15.23:0/3074277300 wait complete. 2023-06-22T08:18:43.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.609+0000 7f8b39bff700 1 Processor -- start 2023-06-22T08:18:43.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.609+0000 7f8b39bff700 1 -- start start 2023-06-22T08:18:43.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.610+0000 7f8b39bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b340b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:43.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.610+0000 7f8b39bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340c97d0 0x7f8b340cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:43.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.610+0000 7f8b39bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b34152d70 0x7f8b3414e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:43.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.610+0000 7f8b39bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340ccaa0 con 0x7f8b34152d70 2023-06-22T08:18:43.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.610+0000 7f8b39bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340b2d20 con 0x7f8b340c97d0 2023-06-22T08:18:43.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.610+0000 7f8b39bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b341ef020 con 0x7f8b340c4d00 2023-06-22T08:18:43.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.610+0000 7f8b337fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b340b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:43.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b32ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340c97d0 0x7f8b340cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:43.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b33fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b34152d70 0x7f8b3414e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:43.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b337fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b340b31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:35966/0 (socket says 172.21.15.23:35966) 2023-06-22T08:18:43.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b337fe700 1 -- 172.21.15.23:0/3160490943 learned_addr learned my addr 172.21.15.23:0/3160490943 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:43.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b337fe700 1 -- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340c97d0 msgr2=0x7f8b340cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:43.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b337fe700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340c97d0 0x7f8b340cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:43.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b337fe700 1 -- 172.21.15.23:0/3160490943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b34152d70 msgr2=0x7f8b3414e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:43.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b337fe700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b34152d70 0x7f8b3414e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:43.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b337fe700 1 -- 172.21.15.23:0/3160490943 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c009a70 con 0x7f8b340c4d00 2023-06-22T08:18:43.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b33fff700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b34152d70 0x7f8b3414e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:43.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.611+0000 7f8b337fe700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b340b31b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400c500 tx=0x7f8b2400c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:43.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.612+0000 7f8b1bfff700 1 -- 172.21.15.23:0/3160490943 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b240052d0 con 0x7f8b340c4d00 2023-06-22T08:18:43.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.612+0000 7f8b39bff700 1 -- 172.21.15.23:0/3160490943 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b341ef2c0 con 0x7f8b340c4d00 2023-06-22T08:18:43.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.612+0000 7f8b1bfff700 1 -- 172.21.15.23:0/3160490943 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b24005450 con 0x7f8b340c4d00 2023-06-22T08:18:43.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.612+0000 7f8b1bfff700 1 -- 172.21.15.23:0/3160490943 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b240038c0 con 0x7f8b340c4d00 2023-06-22T08:18:43.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.612+0000 7f8b39bff700 1 -- 172.21.15.23:0/3160490943 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b340b26d0 con 0x7f8b340c4d00 2023-06-22T08:18:43.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.613+0000 7f8b39bff700 1 -- 172.21.15.23:0/3160490943 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b340b3000 con 0x7f8b340c4d00 2023-06-22T08:18:43.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.614+0000 7f8b1bfff700 1 -- 172.21.15.23:0/3160490943 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8b24003a20 con 0x7f8b340c4d00 2023-06-22T08:18:43.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.615+0000 7f8b1bfff700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b2006ace0 0x7f8b2006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:43.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.615+0000 7f8b1bfff700 1 -- 172.21.15.23:0/3160490943 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f8b240890e0 con 0x7f8b340c4d00 2023-06-22T08:18:43.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.615+0000 7f8b32ffd700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b2006ace0 0x7f8b2006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:43.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.616+0000 7f8b32ffd700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b2006ace0 0x7f8b2006d190 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8b34073380 tx=0x7f8b28007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:43.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.618+0000 7f8b1bfff700 1 -- 172.21.15.23:0/3160490943 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8b24053b80 con 0x7f8b340c4d00 2023-06-22T08:18:43.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.868+0000 7f8b39bff700 1 -- 172.21.15.23:0/3160490943 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b340b1c20 con 0x7f8b340c4d00 2023-06-22T08:18:43.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.869+0000 7f8b1bfff700 1 -- 172.21.15.23:0/3160490943 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f8b240571a0 con 0x7f8b340c4d00 2023-06-22T08:18:43.872 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:43.872 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:30.949335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:43.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.871+0000 7f8b39bff700 1 -- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b2006ace0 msgr2=0x7f8b2006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:43.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.871+0000 7f8b39bff700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b2006ace0 0x7f8b2006d190 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8b34073380 tx=0x7f8b28007de0 comp rx=0 tx=0).stop 2023-06-22T08:18:43.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.871+0000 7f8b39bff700 1 -- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 msgr2=0x7f8b340b31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:43.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.871+0000 7f8b39bff700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b340b31b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400c500 tx=0x7f8b2400c810 comp rx=0 tx=0).stop 2023-06-22T08:18:43.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.872+0000 7f8b39bff700 1 -- 172.21.15.23:0/3160490943 shutdown_connections 2023-06-22T08:18:43.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.872+0000 7f8b39bff700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b34152d70 0x7f8b3414e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:43.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.872+0000 7f8b39bff700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8b340c4d00 0x7f8b340b31b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:43.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.872+0000 7f8b39bff700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8b2006ace0 0x7f8b2006d190 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:43.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.872+0000 7f8b39bff700 1 --2- 172.21.15.23:0/3160490943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b340c97d0 0x7f8b340cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:43.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.872+0000 7f8b39bff700 1 -- 172.21.15.23:0/3160490943 >> 172.21.15.23:0/3160490943 conn(0x7f8b3406ae30 msgr2=0x7f8b340cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:43.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.872+0000 7f8b39bff700 1 -- 172.21.15.23:0/3160490943 shutdown_connections 2023-06-22T08:18:43.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:43.872+0000 7f8b39bff700 1 -- 172.21.15.23:0/3160490943 wait complete. 2023-06-22T08:18:43.879 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 150 2023-06-22T08:18:43.892 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:18:44.599 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:18:44.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:18:45.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:46.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.100+0000 7fd138fff700 1 Processor -- start 2023-06-22T08:18:46.104 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.100+0000 7fd138fff700 1 -- start start 2023-06-22T08:18:46.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd138fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd13414f850 0x7fd1340c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:46.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd138fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 0x7fd1340c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:46.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd138fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1340c51a0 con 0x7fd1340c69e0 2023-06-22T08:18:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd138fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1340c44e0 con 0x7fd13414f850 2023-06-22T08:18:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd138fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1340c5440 con 0x7fd134071070 2023-06-22T08:18:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd13259c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd13414f850 0x7fd1340c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd132d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 0x7fd1340c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd13259c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd13414f850 0x7fd1340c3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:59038/0 (socket says 172.21.15.23:59038) 2023-06-22T08:18:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd13259c700 1 -- 172.21.15.23:0/2917760643 learned_addr learned my addr 172.21.15.23:0/2917760643 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd132d9d700 1 -- 172.21.15.23:0/2917760643 >> v2:172.21.15.67:3300/0 conn(0x7fd13414f850 msgr2=0x7fd1340c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:18:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd132d9d700 1 --2- 172.21.15.23:0/2917760643 >> v2:172.21.15.67:3300/0 conn(0x7fd13414f850 0x7fd1340c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd132d9d700 1 -- 172.21.15.23:0/2917760643 >> v1:172.21.15.67:6789/0 conn(0x7fd1340c69e0 legacy=0x7fd134155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd132d9d700 1 -- 172.21.15.23:0/2917760643 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1340c5720 con 0x7fd134071070 2023-06-22T08:18:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.102+0000 7fd13259c700 1 --2- 172.21.15.23:0/2917760643 >> v2:172.21.15.67:3300/0 conn(0x7fd13414f850 0x7fd1340c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:46.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.103+0000 7fd132d9d700 1 --2- 172.21.15.23:0/2917760643 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 0x7fd1340c35c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009d90 tx=0x7fd12c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=820e8f140b6ee8bb server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:46.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.103+0000 7fd131d9b700 1 -- 172.21.15.23:0/2917760643 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c00e070 con 0x7fd134071070 2023-06-22T08:18:46.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.103+0000 7fd131d9b700 1 -- 172.21.15.23:0/2917760643 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12c00bb20 con 0x7fd134071070 2023-06-22T08:18:46.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.103+0000 7fd131d9b700 1 -- 172.21.15.23:0/2917760643 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c012430 con 0x7fd134071070 2023-06-22T08:18:46.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.103+0000 7fd138fff700 1 -- 172.21.15.23:0/2917760643 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 msgr2=0x7fd1340c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:46.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.103+0000 7fd138fff700 1 --2- 172.21.15.23:0/2917760643 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 0x7fd1340c35c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009d90 tx=0x7fd12c00b330 comp rx=0 tx=0).stop 2023-06-22T08:18:46.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.104+0000 7fd138fff700 1 -- 172.21.15.23:0/2917760643 shutdown_connections 2023-06-22T08:18:46.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.104+0000 7fd138fff700 1 --2- 172.21.15.23:0/2917760643 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 0x7fd1340c35c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.104+0000 7fd138fff700 1 --2- 172.21.15.23:0/2917760643 >> v2:172.21.15.67:3300/0 conn(0x7fd13414f850 0x7fd1340c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.104+0000 7fd138fff700 1 -- 172.21.15.23:0/2917760643 >> 172.21.15.23:0/2917760643 conn(0x7fd1340ca3f0 msgr2=0x7fd1340ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:46.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.104+0000 7fd138fff700 1 -- 172.21.15.23:0/2917760643 shutdown_connections 2023-06-22T08:18:46.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.104+0000 7fd138fff700 1 -- 172.21.15.23:0/2917760643 wait complete. 2023-06-22T08:18:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.105+0000 7fd138fff700 1 Processor -- start 2023-06-22T08:18:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.106+0000 7fd138fff700 1 -- start start 2023-06-22T08:18:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.106+0000 7fd138fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 0x7fd1341e14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd138fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1340c69e0 0x7fd1341f0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd138fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd13414f850 0x7fd134143040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd138fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1341f0890 con 0x7fd13414f850 2023-06-22T08:18:46.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd138fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1341e1b30 con 0x7fd1340c69e0 2023-06-22T08:18:46.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd138fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1341ec4e0 con 0x7fd134071070 2023-06-22T08:18:46.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd132d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd13414f850 0x7fd134143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:46.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd132d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd13414f850 0x7fd134143040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58736/0 (socket says 172.21.15.23:58736) 2023-06-22T08:18:46.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd132d9d700 1 -- 172.21.15.23:0/2546806302 learned_addr learned my addr 172.21.15.23:0/2546806302 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:46.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd13259c700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 0x7fd1341e14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd12bfff700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1340c69e0 0x7fd1341f0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd132d9d700 1 -- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 msgr2=0x7fd1341e14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd132d9d700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 0x7fd1341e14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd132d9d700 1 -- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1340c69e0 msgr2=0x7fd1341f0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd132d9d700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1340c69e0 0x7fd1341f0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.107+0000 7fd132d9d700 1 -- 172.21.15.23:0/2546806302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd124009550 con 0x7fd13414f850 2023-06-22T08:18:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.108+0000 7fd132d9d700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd13414f850 0x7fd134143040 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd12c014040 tx=0x7fd12c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.108+0000 7fd12affd700 1 -- 172.21.15.23:0/2546806302 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c012430 con 0x7fd13414f850 2023-06-22T08:18:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.108+0000 7fd12affd700 1 -- 172.21.15.23:0/2546806302 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12c020070 con 0x7fd13414f850 2023-06-22T08:18:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.108+0000 7fd12affd700 1 -- 172.21.15.23:0/2546806302 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c003ea0 con 0x7fd13414f850 2023-06-22T08:18:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.108+0000 7fd138fff700 1 -- 172.21.15.23:0/2546806302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c009a70 con 0x7fd13414f850 2023-06-22T08:18:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.108+0000 7fd138fff700 1 -- 172.21.15.23:0/2546806302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1341ecaa0 con 0x7fd13414f850 2023-06-22T08:18:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.109+0000 7fd138fff700 1 -- 172.21.15.23:0/2546806302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1340c2950 con 0x7fd13414f850 2023-06-22T08:18:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.110+0000 7fd12affd700 1 -- 172.21.15.23:0/2546806302 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd12c004040 con 0x7fd13414f850 2023-06-22T08:18:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.111+0000 7fd12affd700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd11406ad10 0x7fd11406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:46.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.111+0000 7fd12affd700 1 -- 172.21.15.23:0/2546806302 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7fd12c022070 con 0x7fd13414f850 2023-06-22T08:18:46.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.111+0000 7fd13259c700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd11406ad10 0x7fd11406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:46.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.112+0000 7fd13259c700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd11406ad10 0x7fd11406d1c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd124006d30 tx=0x7fd1240092c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:46.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.116+0000 7fd12affd700 1 -- 172.21.15.23:0/2546806302 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd12c056180 con 0x7fd13414f850 2023-06-22T08:18:46.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.368+0000 7fd138fff700 1 -- 172.21.15.23:0/2546806302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd13404ea20 con 0x7fd13414f850 2023-06-22T08:18:46.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.369+0000 7fd12affd700 1 -- 172.21.15.23:0/2546806302 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6272 (secure 0 0 0) 0x7fd12c0597a0 con 0x7fd13414f850 2023-06-22T08:18:46.373 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:46.373 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:30.949335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 -- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd11406ad10 msgr2=0x7fd11406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd11406ad10 0x7fd11406d1c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd124006d30 tx=0x7fd1240092c0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 -- 172.21.15.23:0/2546806302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd13414f850 msgr2=0x7fd134143040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd13414f850 0x7fd134143040 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd12c014040 tx=0x7fd12c004ad0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 -- 172.21.15.23:0/2546806302 shutdown_connections 2023-06-22T08:18:46.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd13414f850 0x7fd134143040 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd134071070 0x7fd1341e14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd11406ad10 0x7fd11406d1c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 --2- 172.21.15.23:0/2546806302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1340c69e0 0x7fd1341f0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 -- 172.21.15.23:0/2546806302 >> 172.21.15.23:0/2546806302 conn(0x7fd1340ca3f0 msgr2=0x7fd134069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 -- 172.21.15.23:0/2546806302 shutdown_connections 2023-06-22T08:18:46.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:46.371+0000 7fd138fff700 1 -- 172.21.15.23:0/2546806302 wait complete. 2023-06-22T08:18:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 150 2023-06-22T08:18:46.393 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:18:48.394 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.602+0000 7f63e2fff700 1 Processor -- start 2023-06-22T08:18:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.602+0000 7f63e2fff700 1 -- start start 2023-06-22T08:18:48.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.603+0000 7f63e2fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f63dc0c9fd0 0x7f63dc0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:48.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.603+0000 7f63e2fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc0c40b0 0x7f63dc0c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:48.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.603+0000 7f63e2fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63dc15c370 con 0x7f63dc152ce0 2023-06-22T08:18:48.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.603+0000 7f63e2fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63dc1530b0 con 0x7f63dc0c9fd0 2023-06-22T08:18:48.606 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.603+0000 7f63e2fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0cf6f0 con 0x7f63dc0c40b0 2023-06-22T08:18:48.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.603+0000 7f63dbfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f63dc0c9fd0 0x7f63dc0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:48.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.603+0000 7f63e0d9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f63dc152ce0 0x7f63dc15c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:44564/0 (socket says 172.21.15.23:44564) 2023-06-22T08:18:48.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.603+0000 7f63e159c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc0c40b0 0x7f63dc0c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:48.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.603+0000 7f63e0d9b700 1 -- 172.21.15.23:0/191836617 learned_addr learned my addr 172.21.15.23:0/191836617 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:18:48.608 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.604+0000 7f63e159c700 1 -- 172.21.15.23:0/191836617 >> v2:172.21.15.23:3300/0 conn(0x7f63dc0c9fd0 msgr2=0x7f63dc0cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:18:48.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.604+0000 7f63db7fe700 1 -- 172.21.15.23:0/191836617 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024774629 0 0) 0x7f63dc15c370 con 0x7f63dc152ce0 2023-06-22T08:18:48.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.604+0000 7f63e159c700 1 --2- 172.21.15.23:0/191836617 >> v2:172.21.15.23:3300/0 conn(0x7f63dc0c9fd0 0x7f63dc0cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:48.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.604+0000 7f63e159c700 1 -- 172.21.15.23:0/191836617 >> v1:172.21.15.23:6789/0 conn(0x7f63dc152ce0 legacy=0x7f63dc15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:48.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.604+0000 7f63e159c700 1 -- 172.21.15.23:0/191836617 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc15bda0 con 0x7f63dc0c40b0 2023-06-22T08:18:48.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.604+0000 7f63dbfff700 1 --2- 172.21.15.23:0/191836617 >> v2:172.21.15.23:3300/0 conn(0x7f63dc0c9fd0 0x7f63dc0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:48.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.604+0000 7f63e159c700 1 --2- 172.21.15.23:0/191836617 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc0c40b0 0x7f63dc0c4530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009d90 tx=0x7f63d400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d342c4c43c063a2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:48.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.604+0000 7f63db7fe700 1 -- 172.21.15.23:0/191836617 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d400e070 con 0x7f63dc0c40b0 2023-06-22T08:18:48.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.605+0000 7f63db7fe700 1 -- 172.21.15.23:0/191836617 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d400bb20 con 0x7f63dc0c40b0 2023-06-22T08:18:48.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.605+0000 7f63db7fe700 1 -- 172.21.15.23:0/191836617 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4012430 con 0x7f63dc0c40b0 2023-06-22T08:18:48.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.605+0000 7f63e2fff700 1 -- 172.21.15.23:0/191836617 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc0c40b0 msgr2=0x7f63dc0c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:48.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.605+0000 7f63e2fff700 1 --2- 172.21.15.23:0/191836617 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc0c40b0 0x7f63dc0c4530 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009d90 tx=0x7f63d400b330 comp rx=0 tx=0).stop 2023-06-22T08:18:48.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.605+0000 7f63e2fff700 1 -- 172.21.15.23:0/191836617 shutdown_connections 2023-06-22T08:18:48.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.605+0000 7f63e2fff700 1 --2- 172.21.15.23:0/191836617 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc0c40b0 0x7f63dc0c4530 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:48.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.605+0000 7f63e2fff700 1 --2- 172.21.15.23:0/191836617 >> v2:172.21.15.23:3300/0 conn(0x7f63dc0c9fd0 0x7f63dc0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:48.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.605+0000 7f63e2fff700 1 -- 172.21.15.23:0/191836617 >> 172.21.15.23:0/191836617 conn(0x7f63dc06ae30 msgr2=0x7f63dc06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:48.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.605+0000 7f63e2fff700 1 -- 172.21.15.23:0/191836617 shutdown_connections 2023-06-22T08:18:48.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.605+0000 7f63e2fff700 1 -- 172.21.15.23:0/191836617 wait complete. 2023-06-22T08:18:48.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.607+0000 7f63e2fff700 1 Processor -- start 2023-06-22T08:18:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.607+0000 7f63e2fff700 1 -- start start 2023-06-22T08:18:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63e2fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63dc0c40b0 0x7f63dc1e2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63e2fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0c9fd0 0x7f63dc1ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63e2fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc152ce0 0x7f63dc150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63e2fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc1ee910 con 0x7f63dc0c40b0 2023-06-22T08:18:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63e2fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0cf450 con 0x7f63dc0c9fd0 2023-06-22T08:18:48.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63e2fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0cee70 con 0x7f63dc152ce0 2023-06-22T08:18:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63e0d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63dc0c40b0 0x7f63dc1e2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63e0d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63dc0c40b0 0x7f63dc1e2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58758/0 (socket says 172.21.15.23:58758) 2023-06-22T08:18:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63e0d9b700 1 -- 172.21.15.23:0/2761405300 learned_addr learned my addr 172.21.15.23:0/2761405300 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63e159c700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc152ce0 0x7f63dc150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.608+0000 7f63dbfff700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0c9fd0 0x7f63dc1ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63e0d9b700 1 -- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc152ce0 msgr2=0x7f63dc150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:48.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63e0d9b700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc152ce0 0x7f63dc150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:48.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63e0d9b700 1 -- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0c9fd0 msgr2=0x7f63dc1ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:48.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63e0d9b700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0c9fd0 0x7f63dc1ee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:48.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63e0d9b700 1 -- 172.21.15.23:0/2761405300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c80094a0 con 0x7f63dc0c40b0 2023-06-22T08:18:48.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63e0d9b700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63dc0c40b0 0x7f63dc1e2860 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f63d000c580 tx=0x7f63d000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:48.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63d97fa700 1 -- 172.21.15.23:0/2761405300 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0004c80 con 0x7f63dc0c40b0 2023-06-22T08:18:48.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63d97fa700 1 -- 172.21.15.23:0/2761405300 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d0004e00 con 0x7f63dc0c40b0 2023-06-22T08:18:48.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63e2fff700 1 -- 172.21.15.23:0/2761405300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d4009a70 con 0x7f63dc0c40b0 2023-06-22T08:18:48.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63d97fa700 1 -- 172.21.15.23:0/2761405300 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0007310 con 0x7f63dc0c40b0 2023-06-22T08:18:48.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.609+0000 7f63e2fff700 1 -- 172.21.15.23:0/2761405300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63dc0ce990 con 0x7f63dc0c40b0 2023-06-22T08:18:48.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.610+0000 7f63e2fff700 1 -- 172.21.15.23:0/2761405300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63dc04ea20 con 0x7f63dc0c40b0 2023-06-22T08:18:48.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.611+0000 7f63d97fa700 1 -- 172.21.15.23:0/2761405300 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f63d0007470 con 0x7f63dc0c40b0 2023-06-22T08:18:48.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.612+0000 7f63d97fa700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f63c006ad10 0x7f63c006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:48.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.613+0000 7f63d97fa700 1 -- 172.21.15.23:0/2761405300 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7f63d00896b0 con 0x7f63dc0c40b0 2023-06-22T08:18:48.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.613+0000 7f63dbfff700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f63c006ad10 0x7f63c006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:48.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.613+0000 7f63dbfff700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f63c006ad10 0x7f63c006d1c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0b1230 tx=0x7f63c8009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:48.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.617+0000 7f63d97fa700 1 -- 172.21.15.23:0/2761405300 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f63d0054150 con 0x7f63dc0c40b0 2023-06-22T08:18:48.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.869+0000 7f63e2fff700 1 -- 172.21.15.23:0/2761405300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63dc0c9db0 con 0x7f63dc0c40b0 2023-06-22T08:18:48.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.870+0000 7f63d97fa700 1 -- 172.21.15.23:0/2761405300 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6272 (secure 0 0 0) 0x7f63d0057770 con 0x7f63dc0c40b0 2023-06-22T08:18:48.873 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:48.873 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:30.949335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 -- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f63c006ad10 msgr2=0x7f63c006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f63c006ad10 0x7f63c006d1c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0b1230 tx=0x7f63c8009210 comp rx=0 tx=0).stop 2023-06-22T08:18:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 -- 172.21.15.23:0/2761405300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63dc0c40b0 msgr2=0x7f63dc1e2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63dc0c40b0 0x7f63dc1e2860 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f63d000c580 tx=0x7f63d000c890 comp rx=0 tx=0).stop 2023-06-22T08:18:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 -- 172.21.15.23:0/2761405300 shutdown_connections 2023-06-22T08:18:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f63dc152ce0 0x7f63dc150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63dc0c40b0 0x7f63dc1e2860 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f63c006ad10 0x7f63c006d1c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 --2- 172.21.15.23:0/2761405300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0c9fd0 0x7f63dc1ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 -- 172.21.15.23:0/2761405300 >> 172.21.15.23:0/2761405300 conn(0x7f63dc06ae30 msgr2=0x7f63dc0b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 -- 172.21.15.23:0/2761405300 shutdown_connections 2023-06-22T08:18:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:48.872+0000 7f63e2fff700 1 -- 172.21.15.23:0/2761405300 wait complete. 2023-06-22T08:18:48.877 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 150 2023-06-22T08:18:48.893 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:18:50.893 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:51.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.102+0000 7fa51ffff700 1 Processor -- start 2023-06-22T08:18:51.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.102+0000 7fa51ffff700 1 -- start start 2023-06-22T08:18:51.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.103+0000 7fa51ffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa518152d70 0x7fa5180c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:51.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.103+0000 7fa51ffff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa518159790 con 0x7fa518159330 2023-06-22T08:18:51.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.103+0000 7fa51ffff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51815bea0 con 0x7fa5180c45f0 2023-06-22T08:18:51.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.103+0000 7fa51ffff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5180c95e0 con 0x7fa518152d70 2023-06-22T08:18:51.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.104+0000 7fa51d59a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa518159330 0x7fa5180cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:44566/0 (socket says 172.21.15.23:44566) 2023-06-22T08:18:51.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.104+0000 7fa51d59a700 1 -- 172.21.15.23:0/4070792101 learned_addr learned my addr 172.21.15.23:0/4070792101 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:18:51.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.104+0000 7fa51dd9b700 1 --2- 172.21.15.23:0/4070792101 >> v2:172.21.15.67:3300/0 conn(0x7fa518152d70 0x7fa5180c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:51.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.104+0000 7fa51cd99700 1 -- 172.21.15.23:0/4070792101 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685897799 0 0) 0x7fa518159790 con 0x7fa518159330 2023-06-22T08:18:51.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.104+0000 7fa51cd99700 1 -- 172.21.15.23:0/4070792101 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5040035b0 con 0x7fa518159330 2023-06-22T08:18:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.104+0000 7fa51dd9b700 1 -- 172.21.15.23:0/4070792101 >> v1:172.21.15.67:6789/0 conn(0x7fa5180c45f0 legacy=0x7fa5180c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.104+0000 7fa51cd99700 1 -- 172.21.15.23:0/4070792101 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404184624 0 0) 0x7fa51815bea0 con 0x7fa5180c45f0 2023-06-22T08:18:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.104+0000 7fa51dd9b700 1 -- 172.21.15.23:0/4070792101 >> v1:172.21.15.23:6789/0 conn(0x7fa518159330 legacy=0x7fa5180cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.104+0000 7fa51dd9b700 1 -- 172.21.15.23:0/4070792101 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51815c160 con 0x7fa518152d70 2023-06-22T08:18:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.104+0000 7fa51dd9b700 1 --2- 172.21.15.23:0/4070792101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518152d70 0x7fa5180c97d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa514009840 tx=0x7fa514009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9467bfe67d32cfe6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.105+0000 7fa51cd99700 1 -- 172.21.15.23:0/4070792101 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514011830 con 0x7fa518152d70 2023-06-22T08:18:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.105+0000 7fa51cd99700 1 -- 172.21.15.23:0/4070792101 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa514011990 con 0x7fa518152d70 2023-06-22T08:18:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.105+0000 7fa51cd99700 1 -- 172.21.15.23:0/4070792101 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514011d20 con 0x7fa518152d70 2023-06-22T08:18:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.105+0000 7fa51ffff700 1 -- 172.21.15.23:0/4070792101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518152d70 msgr2=0x7fa5180c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.105+0000 7fa51ffff700 1 --2- 172.21.15.23:0/4070792101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518152d70 0x7fa5180c97d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa514009840 tx=0x7fa514009b50 comp rx=0 tx=0).stop 2023-06-22T08:18:51.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.105+0000 7fa51ffff700 1 -- 172.21.15.23:0/4070792101 shutdown_connections 2023-06-22T08:18:51.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.105+0000 7fa51ffff700 1 --2- 172.21.15.23:0/4070792101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518152d70 0x7fa5180c97d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:51.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.105+0000 7fa51ffff700 1 -- 172.21.15.23:0/4070792101 >> 172.21.15.23:0/4070792101 conn(0x7fa51806ae30 msgr2=0x7fa51806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:51.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.105+0000 7fa51ffff700 1 -- 172.21.15.23:0/4070792101 shutdown_connections 2023-06-22T08:18:51.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.106+0000 7fa51ffff700 1 -- 172.21.15.23:0/4070792101 wait complete. 2023-06-22T08:18:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.107+0000 7fa51ffff700 1 Processor -- start 2023-06-22T08:18:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.107+0000 7fa51ffff700 1 -- start start 2023-06-22T08:18:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5180c45f0 0x7fa51814d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa518152d70 0x7fa5180ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518159330 0x7fa5181efb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51ffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181ef6a0 con 0x7fa5180c45f0 2023-06-22T08:18:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51ffff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518157ea0 con 0x7fa518159330 2023-06-22T08:18:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51ffff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa518158160 con 0x7fa518152d70 2023-06-22T08:18:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5180c45f0 0x7fa51814d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:51.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5180c45f0 0x7fa51814d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58774/0 (socket says 172.21.15.23:58774) 2023-06-22T08:18:51.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51dd9b700 1 -- 172.21.15.23:0/2190988809 learned_addr learned my addr 172.21.15.23:0/2190988809 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:51.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51d59a700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa518152d70 0x7fa5180ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:51.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.108+0000 7fa51e59c700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518159330 0x7fa5181efb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:51.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.109+0000 7fa51dd9b700 1 -- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa518152d70 msgr2=0x7fa5180ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:51.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.109+0000 7fa51dd9b700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa518152d70 0x7fa5180ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:51.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.109+0000 7fa51dd9b700 1 -- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518159330 msgr2=0x7fa5181efb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:51.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.109+0000 7fa51dd9b700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518159330 0x7fa5181efb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.109+0000 7fa51dd9b700 1 -- 172.21.15.23:0/2190988809 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5140094a0 con 0x7fa5180c45f0 2023-06-22T08:18:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.109+0000 7fa51dd9b700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5180c45f0 0x7fa51814d710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa5140090b0 tx=0x7fa514004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:51.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.109+0000 7fa50e7fc700 1 -- 172.21.15.23:0/2190988809 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51401a7e0 con 0x7fa5180c45f0 2023-06-22T08:18:51.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.109+0000 7fa50e7fc700 1 -- 172.21.15.23:0/2190988809 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51401a940 con 0x7fa5180c45f0 2023-06-22T08:18:51.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.109+0000 7fa51ffff700 1 -- 172.21.15.23:0/2190988809 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa518158440 con 0x7fa5180c45f0 2023-06-22T08:18:51.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.109+0000 7fa50e7fc700 1 -- 172.21.15.23:0/2190988809 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51401acd0 con 0x7fa5180c45f0 2023-06-22T08:18:51.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.110+0000 7fa51ffff700 1 -- 172.21.15.23:0/2190988809 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5181ef330 con 0x7fa5180c45f0 2023-06-22T08:18:51.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.111+0000 7fa51ffff700 1 -- 172.21.15.23:0/2190988809 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5181ef980 con 0x7fa5180c45f0 2023-06-22T08:18:51.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.111+0000 7fa50e7fc700 1 -- 172.21.15.23:0/2190988809 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa514022070 con 0x7fa5180c45f0 2023-06-22T08:18:51.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.112+0000 7fa50e7fc700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4fc06ace0 0x7fa4fc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:51.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.113+0000 7fa50e7fc700 1 -- 172.21.15.23:0/2190988809 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7fa51408a810 con 0x7fa5180c45f0 2023-06-22T08:18:51.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.113+0000 7fa51d59a700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4fc06ace0 0x7fa4fc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:51.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.113+0000 7fa51d59a700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4fc06ace0 0x7fa4fc06d190 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa51815ce00 tx=0x7fa508009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:51.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.117+0000 7fa50e7fc700 1 -- 172.21.15.23:0/2190988809 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa5140553d0 con 0x7fa5180c45f0 2023-06-22T08:18:51.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.375+0000 7fa51ffff700 1 -- 172.21.15.23:0/2190988809 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5181ef4c0 con 0x7fa5180c45f0 2023-06-22T08:18:51.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.376+0000 7fa50e7fc700 1 -- 172.21.15.23:0/2190988809 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6272 (secure 0 0 0) 0x7fa514015090 con 0x7fa5180c45f0 2023-06-22T08:18:51.379 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:51.379 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:30.949335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:51.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 -- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4fc06ace0 msgr2=0x7fa4fc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:51.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4fc06ace0 0x7fa4fc06d190 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa51815ce00 tx=0x7fa508009040 comp rx=0 tx=0).stop 2023-06-22T08:18:51.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 -- 172.21.15.23:0/2190988809 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5180c45f0 msgr2=0x7fa51814d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:51.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5180c45f0 0x7fa51814d710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa5140090b0 tx=0x7fa514004750 comp rx=0 tx=0).stop 2023-06-22T08:18:51.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 -- 172.21.15.23:0/2190988809 shutdown_connections 2023-06-22T08:18:51.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa4fc06ace0 0x7fa4fc06d190 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:51.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa518159330 0x7fa5181efb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:51.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5180c45f0 0x7fa51814d710 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:51.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 --2- 172.21.15.23:0/2190988809 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa518152d70 0x7fa5180ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:51.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 -- 172.21.15.23:0/2190988809 >> 172.21.15.23:0/2190988809 conn(0x7fa51806ae30 msgr2=0x7fa5180ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:51.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 -- 172.21.15.23:0/2190988809 shutdown_connections 2023-06-22T08:18:51.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:51.378+0000 7fa51ffff700 1 -- 172.21.15.23:0/2190988809 wait complete. 2023-06-22T08:18:51.386 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 150 2023-06-22T08:18:51.400 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:18:53.400 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:53.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.613+0000 7efca9fff700 1 Processor -- start 2023-06-22T08:18:53.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.613+0000 7efca9fff700 1 -- start start 2023-06-22T08:18:53.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.614+0000 7efca9fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efca4152ce0 0x7efca40cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:53.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca9fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca415c370 con 0x7efca40c9fd0 2023-06-22T08:18:53.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca9fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca415bda0 con 0x7efca40c40b0 2023-06-22T08:18:53.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca9fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca41530b0 con 0x7efca4152ce0 2023-06-22T08:18:53.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca2ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7efca40c9fd0 0x7efca415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:44574/0 (socket says 172.21.15.23:44574) 2023-06-22T08:18:53.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca2ffd700 1 -- 172.21.15.23:0/2507285388 learned_addr learned my addr 172.21.15.23:0/2507285388 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:18:53.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca37fe700 1 --2- 172.21.15.23:0/2507285388 >> v2:172.21.15.67:3300/0 conn(0x7efca4152ce0 0x7efca40cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:53.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca27fc700 1 -- 172.21.15.23:0/2507285388 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743650289 0 0) 0x7efca415c370 con 0x7efca40c9fd0 2023-06-22T08:18:53.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca27fc700 1 -- 172.21.15.23:0/2507285388 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc8c0035b0 con 0x7efca40c9fd0 2023-06-22T08:18:53.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca37fe700 1 -- 172.21.15.23:0/2507285388 >> v1:172.21.15.67:6789/0 conn(0x7efca40c40b0 legacy=0x7efca40c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:53.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca37fe700 1 -- 172.21.15.23:0/2507285388 >> v1:172.21.15.23:6789/0 conn(0x7efca40c9fd0 legacy=0x7efca415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:53.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.615+0000 7efca37fe700 1 -- 172.21.15.23:0/2507285388 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca40c9460 con 0x7efca4152ce0 2023-06-22T08:18:53.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.616+0000 7efca37fe700 1 --2- 172.21.15.23:0/2507285388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca4152ce0 0x7efca40cf1b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7efc94009840 tx=0x7efc94009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b36cd00d0bed8ffe server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:53.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.616+0000 7efca27fc700 1 -- 172.21.15.23:0/2507285388 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc94011830 con 0x7efca4152ce0 2023-06-22T08:18:53.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.616+0000 7efca27fc700 1 -- 172.21.15.23:0/2507285388 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc94011990 con 0x7efca4152ce0 2023-06-22T08:18:53.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.616+0000 7efca27fc700 1 -- 172.21.15.23:0/2507285388 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc94011d20 con 0x7efca4152ce0 2023-06-22T08:18:53.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.616+0000 7efca9fff700 1 -- 172.21.15.23:0/2507285388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca4152ce0 msgr2=0x7efca40cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:53.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.616+0000 7efca9fff700 1 --2- 172.21.15.23:0/2507285388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca4152ce0 0x7efca40cf1b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7efc94009840 tx=0x7efc94009b50 comp rx=0 tx=0).stop 2023-06-22T08:18:53.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.617+0000 7efca9fff700 1 -- 172.21.15.23:0/2507285388 shutdown_connections 2023-06-22T08:18:53.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.617+0000 7efca9fff700 1 --2- 172.21.15.23:0/2507285388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca4152ce0 0x7efca40cf1b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:53.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.617+0000 7efca9fff700 1 -- 172.21.15.23:0/2507285388 >> 172.21.15.23:0/2507285388 conn(0x7efca406ae30 msgr2=0x7efca406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:53.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.617+0000 7efca9fff700 1 -- 172.21.15.23:0/2507285388 shutdown_connections 2023-06-22T08:18:53.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.617+0000 7efca9fff700 1 -- 172.21.15.23:0/2507285388 wait complete. 2023-06-22T08:18:53.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.619+0000 7efca9fff700 1 Processor -- start 2023-06-22T08:18:53.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.619+0000 7efca9fff700 1 -- start start 2023-06-22T08:18:53.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.620+0000 7efca9fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca40c40b0 0x7efca41e3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:53.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.620+0000 7efca9fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca40c9fd0 0x7efca41e1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:53.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.620+0000 7efca9fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efca4152ce0 0x7efca415b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:53.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.620+0000 7efca9fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca40ccb40 con 0x7efca40c9fd0 2023-06-22T08:18:53.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.620+0000 7efca9fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca40b38b0 con 0x7efca40c40b0 2023-06-22T08:18:53.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.620+0000 7efca9fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca40cce00 con 0x7efca4152ce0 2023-06-22T08:18:53.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca40c9fd0 0x7efca41e1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:53.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca40c40b0 0x7efca41e3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:53.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca40c9fd0 0x7efca41e1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58778/0 (socket says 172.21.15.23:58778) 2023-06-22T08:18:53.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca40c40b0 0x7efca41e3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:59100/0 (socket says 172.21.15.23:59100) 2023-06-22T08:18:53.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca2ffd700 1 -- 172.21.15.23:0/360396999 learned_addr learned my addr 172.21.15.23:0/360396999 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:53.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca3fff700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efca4152ce0 0x7efca415b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:53.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca2ffd700 1 -- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efca4152ce0 msgr2=0x7efca415b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:53.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca2ffd700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efca4152ce0 0x7efca415b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:53.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca2ffd700 1 -- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca40c40b0 msgr2=0x7efca41e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:53.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca2ffd700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca40c40b0 0x7efca41e3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:53.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca2ffd700 1 -- 172.21.15.23:0/360396999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc940094a0 con 0x7efca40c9fd0 2023-06-22T08:18:53.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca37fe700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca40c40b0 0x7efca41e3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:53.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.621+0000 7efca2ffd700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca40c9fd0 0x7efca41e1e20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7efc9800c730 tx=0x7efc9800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:53.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.622+0000 7efc8bfff700 1 -- 172.21.15.23:0/360396999 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98005d70 con 0x7efca40c9fd0 2023-06-22T08:18:53.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.622+0000 7efc8bfff700 1 -- 172.21.15.23:0/360396999 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc980070d0 con 0x7efca40c9fd0 2023-06-22T08:18:53.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.622+0000 7efc8bfff700 1 -- 172.21.15.23:0/360396999 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98007440 con 0x7efca40c9fd0 2023-06-22T08:18:53.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.622+0000 7efca9fff700 1 -- 172.21.15.23:0/360396999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca41e35b0 con 0x7efca40c9fd0 2023-06-22T08:18:53.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.622+0000 7efca9fff700 1 -- 172.21.15.23:0/360396999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca41e3b00 con 0x7efca40c9fd0 2023-06-22T08:18:53.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.623+0000 7efca9fff700 1 -- 172.21.15.23:0/360396999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca404ea20 con 0x7efca40c9fd0 2023-06-22T08:18:53.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.624+0000 7efc8bfff700 1 -- 172.21.15.23:0/360396999 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7efc980075a0 con 0x7efca40c9fd0 2023-06-22T08:18:53.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.625+0000 7efc8bfff700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc9006ace0 0x7efc9006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:53.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.625+0000 7efc8bfff700 1 -- 172.21.15.23:0/360396999 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 7587+0+0 (secure 0 0 0) 0x7efc98088e20 con 0x7efca40c9fd0 2023-06-22T08:18:53.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.625+0000 7efca37fe700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc9006ace0 0x7efc9006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:53.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.626+0000 7efca37fe700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc9006ace0 0x7efc9006d190 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7efc940090b0 tx=0x7efc9401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:53.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.630+0000 7efc8bfff700 1 -- 172.21.15.23:0/360396999 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7efc980538e0 con 0x7efca40c9fd0 2023-06-22T08:18:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.881+0000 7efca9fff700 1 -- 172.21.15.23:0/360396999 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efca4151b20 con 0x7efca40c9fd0 2023-06-22T08:18:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.882+0000 7efc8bfff700 1 -- 172.21.15.23:0/360396999 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6272 (secure 0 0 0) 0x7efc98056f00 con 0x7efca40c9fd0 2023-06-22T08:18:53.885 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:53.885 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6318,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":138}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:30.949335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":5576},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5576":{"gid":5576,"name":"d","rank":4,"incarnation":109,"state":"up:active","state_seq":15,"addr":"172.21.15.67:6837/2064462140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":2064462140},{"type":"v1","addr":"172.21.15.67:6837","nonce":2064462140}]},"join_fscid":-1,"export_targets":[0,1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 -- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc9006ace0 msgr2=0x7efc9006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc9006ace0 0x7efc9006d190 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7efc940090b0 tx=0x7efc9401c040 comp rx=0 tx=0).stop 2023-06-22T08:18:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 -- 172.21.15.23:0/360396999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca40c9fd0 msgr2=0x7efca41e1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca40c9fd0 0x7efca41e1e20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7efc9800c730 tx=0x7efc9800ca40 comp rx=0 tx=0).stop 2023-06-22T08:18:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 -- 172.21.15.23:0/360396999 shutdown_connections 2023-06-22T08:18:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7efca4152ce0 0x7efca415b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7efc9006ace0 0x7efc9006d190 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efca40c40b0 0x7efca41e3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 --2- 172.21.15.23:0/360396999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efca40c9fd0 0x7efca41e1e20 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 -- 172.21.15.23:0/360396999 >> 172.21.15.23:0/360396999 conn(0x7efca406ae30 msgr2=0x7efca40b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 -- 172.21.15.23:0/360396999 shutdown_connections 2023-06-22T08:18:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:53.884+0000 7efca9fff700 1 -- 172.21.15.23:0/360396999 wait complete. 2023-06-22T08:18:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 150 2023-06-22T08:18:53.905 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-06-22T08:18:55.906 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.114+0000 7f689cbff700 1 Processor -- start 2023-06-22T08:18:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.114+0000 7f689cbff700 1 -- start start 2023-06-22T08:18:56.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.116+0000 7f689cbff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f689814f850 0x7f68980c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.116+0000 7f689cbff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f68980c69e0 0x7f68980c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.116+0000 7f689cbff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68980c51a0 con 0x7f6898071070 2023-06-22T08:18:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.116+0000 7f689cbff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68980c44e0 con 0x7f689814f850 2023-06-22T08:18:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.116+0000 7f689cbff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68980c5440 con 0x7f68980c69e0 2023-06-22T08:18:56.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f6896d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f6898071070 0x7f6898155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33806/0 (socket says 172.21.15.23:33806) 2023-06-22T08:18:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f689659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f689814f850 0x7f68980c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f6896d9d700 1 -- 172.21.15.23:0/1211492230 learned_addr learned my addr 172.21.15.23:0/1211492230 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:18:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f6895d9b700 1 --2- 172.21.15.23:0/1211492230 >> v2:172.21.15.67:3300/0 conn(0x7f68980c69e0 0x7f68980c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f689559a700 1 -- 172.21.15.23:0/1211492230 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637647940 0 0) 0x7f68980c51a0 con 0x7f6898071070 2023-06-22T08:18:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f689559a700 1 -- 172.21.15.23:0/1211492230 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68800035b0 con 0x7f6898071070 2023-06-22T08:18:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f689659c700 1 -- 172.21.15.23:0/1211492230 >> v2:172.21.15.67:3300/0 conn(0x7f68980c69e0 msgr2=0x7f68980c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:18:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f689659c700 1 --2- 172.21.15.23:0/1211492230 >> v2:172.21.15.67:3300/0 conn(0x7f68980c69e0 0x7f68980c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f689659c700 1 -- 172.21.15.23:0/1211492230 >> v1:172.21.15.23:6789/0 conn(0x7f6898071070 legacy=0x7f6898155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f689659c700 1 -- 172.21.15.23:0/1211492230 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68980c5720 con 0x7f689814f850 2023-06-22T08:18:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f689659c700 1 --2- 172.21.15.23:0/1211492230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f689814f850 0x7f68980c3080 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f689806f9a0 tx=0x7f6888009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4fa2f61d6939f28 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.117+0000 7f689559a700 1 -- 172.21.15.23:0/1211492230 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888011830 con 0x7f689814f850 2023-06-22T08:18:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.118+0000 7f689559a700 1 -- 172.21.15.23:0/1211492230 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6888011990 con 0x7f689814f850 2023-06-22T08:18:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.118+0000 7f689559a700 1 -- 172.21.15.23:0/1211492230 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888011d00 con 0x7f689814f850 2023-06-22T08:18:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.118+0000 7f689cbff700 1 -- 172.21.15.23:0/1211492230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f689814f850 msgr2=0x7f68980c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.118+0000 7f689cbff700 1 --2- 172.21.15.23:0/1211492230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f689814f850 0x7f68980c3080 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f689806f9a0 tx=0x7f6888009b20 comp rx=0 tx=0).stop 2023-06-22T08:18:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.118+0000 7f689cbff700 1 -- 172.21.15.23:0/1211492230 shutdown_connections 2023-06-22T08:18:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.118+0000 7f689cbff700 1 --2- 172.21.15.23:0/1211492230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f689814f850 0x7f68980c3080 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.118+0000 7f689cbff700 1 --2- 172.21.15.23:0/1211492230 >> v2:172.21.15.67:3300/0 conn(0x7f68980c69e0 0x7f68980c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.118+0000 7f689cbff700 1 -- 172.21.15.23:0/1211492230 >> 172.21.15.23:0/1211492230 conn(0x7f68980ca3f0 msgr2=0x7f68980ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.118+0000 7f689cbff700 1 -- 172.21.15.23:0/1211492230 shutdown_connections 2023-06-22T08:18:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.118+0000 7f689cbff700 1 -- 172.21.15.23:0/1211492230 wait complete. 2023-06-22T08:18:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.120+0000 7f689cbff700 1 Processor -- start 2023-06-22T08:18:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.120+0000 7f689cbff700 1 -- start start 2023-06-22T08:18:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f689cbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6898071070 0x7f68981ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f689cbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68980c69e0 0x7f6898143c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f689cbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f689814f850 0x7f6898140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f689cbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68980ca250 con 0x7f68980c69e0 2023-06-22T08:18:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f689cbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68981dfe80 con 0x7f6898071070 2023-06-22T08:18:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f689cbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68981df5e0 con 0x7f689814f850 2023-06-22T08:18:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f6895d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68980c69e0 0x7f6898143c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f689659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6898071070 0x7f68981ef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f6895d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68980c69e0 0x7f6898143c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55474/0 (socket says 172.21.15.23:55474) 2023-06-22T08:18:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f6895d9b700 1 -- 172.21.15.23:0/2689288131 learned_addr learned my addr 172.21.15.23:0/2689288131 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.121+0000 7f6896d9d700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f689814f850 0x7f6898140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f6895d9b700 1 -- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f689814f850 msgr2=0x7f6898140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f6895d9b700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f689814f850 0x7f6898140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f6895d9b700 1 -- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6898071070 msgr2=0x7f68981ef8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f6895d9b700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6898071070 0x7f68981ef8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f6895d9b700 1 -- 172.21.15.23:0/2689288131 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68880094a0 con 0x7f68980c69e0 2023-06-22T08:18:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f689659c700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6898071070 0x7f68981ef8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f6895d9b700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68980c69e0 0x7f6898143c70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f688c00e9b0 tx=0x7f688c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f6886ffd700 1 -- 172.21.15.23:0/2689288131 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c009710 con 0x7f68980c69e0 2023-06-22T08:18:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f6886ffd700 1 -- 172.21.15.23:0/2689288131 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688c00ee50 con 0x7f68980c69e0 2023-06-22T08:18:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f6886ffd700 1 -- 172.21.15.23:0/2689288131 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688c00ca50 con 0x7f68980c69e0 2023-06-22T08:18:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f689cbff700 1 -- 172.21.15.23:0/2689288131 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68981efdf0 con 0x7f68980c69e0 2023-06-22T08:18:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.122+0000 7f689cbff700 1 -- 172.21.15.23:0/2689288131 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68980c9c90 con 0x7f68980c69e0 2023-06-22T08:18:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.123+0000 7f689cbff700 1 -- 172.21.15.23:0/2689288131 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68980c2950 con 0x7f68980c69e0 2023-06-22T08:18:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.124+0000 7f6886ffd700 1 -- 172.21.15.23:0/2689288131 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f688c00cbb0 con 0x7f68980c69e0 2023-06-22T08:18:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.125+0000 7f6886ffd700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f687806ace0 0x7f687806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.125+0000 7f6886ffd700 1 -- 172.21.15.23:0/2689288131 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f688c089c20 con 0x7f68980c69e0 2023-06-22T08:18:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.126+0000 7f689659c700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f687806ace0 0x7f687806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.126+0000 7f689659c700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f687806ace0 0x7f687806d190 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6888009b60 tx=0x7f688801c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:56.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.130+0000 7f6886ffd700 1 -- 172.21.15.23:0/2689288131 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f688c057fe0 con 0x7f68980c69e0 2023-06-22T08:18:56.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.385+0000 7f689cbff700 1 -- 172.21.15.23:0/2689288131 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68981dfff0 con 0x7f68980c69e0 2023-06-22T08:18:56.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.385+0000 7f6886ffd700 1 -- 172.21.15.23:0/2689288131 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5509 (secure 0 0 0) 0x7f688c017070 con 0x7f68980c69e0 2023-06-22T08:18:56.389 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:56.390 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:54.559203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:56.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 -- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f687806ace0 msgr2=0x7f687806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f687806ace0 0x7f687806d190 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6888009b60 tx=0x7f688801c040 comp rx=0 tx=0).stop 2023-06-22T08:18:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 -- 172.21.15.23:0/2689288131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68980c69e0 msgr2=0x7f6898143c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68980c69e0 0x7f6898143c70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f688c00e9b0 tx=0x7f688c00ecc0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 -- 172.21.15.23:0/2689288131 shutdown_connections 2023-06-22T08:18:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f689814f850 0x7f6898140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f687806ace0 0x7f687806d190 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6898071070 0x7f68981ef8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 --2- 172.21.15.23:0/2689288131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68980c69e0 0x7f6898143c70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 -- 172.21.15.23:0/2689288131 >> 172.21.15.23:0/2689288131 conn(0x7f68980ca3f0 msgr2=0x7f6898070270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 -- 172.21.15.23:0/2689288131 shutdown_connections 2023-06-22T08:18:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.388+0000 7f689cbff700 1 -- 172.21.15.23:0/2689288131 wait complete. 2023-06-22T08:18:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 151 2023-06-22T08:18:56.409 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-06-22T08:18:56.409 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:18:56.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:56.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.606+0000 7fc8f85ff700 1 Processor -- start 2023-06-22T08:18:56.609 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.606+0000 7fc8f85ff700 1 -- start start 2023-06-22T08:18:56.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.608+0000 7fc8f85ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0152d70 0x7fc8f00c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.610 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.608+0000 7fc8f85ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8f00c45f0 0x7fc8f00c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.608+0000 7fc8f85ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f00c4fb0 con 0x7fc8f0159330 2023-06-22T08:18:56.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.608+0000 7fc8f85ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f00c95e0 con 0x7fc8f00c45f0 2023-06-22T08:18:56.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.608+0000 7fc8f85ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f015bea0 con 0x7fc8f0152d70 2023-06-22T08:18:56.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.608+0000 7fc8f6b9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8f00c45f0 0x7fc8f00c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.608+0000 7fc8f639b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0152d70 0x7fc8f00c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.608+0000 7fc8f6b9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8f00c45f0 0x7fc8f00c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51862/0 (socket says 172.21.15.23:51862) 2023-06-22T08:18:56.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.608+0000 7fc8f6b9c700 1 -- 172.21.15.23:0/1374661842 learned_addr learned my addr 172.21.15.23:0/1374661842 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:56.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.609+0000 7fc8f6b9c700 1 -- 172.21.15.23:0/1374661842 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0152d70 msgr2=0x7fc8f00c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:18:56.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.609+0000 7fc8f6b9c700 1 --2- 172.21.15.23:0/1374661842 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0152d70 0x7fc8f00c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.609+0000 7fc8f6b9c700 1 -- 172.21.15.23:0/1374661842 >> v1:172.21.15.67:6789/0 conn(0x7fc8f0159330 legacy=0x7fc8f00cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.609+0000 7fc8f6b9c700 1 -- 172.21.15.23:0/1374661842 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f015c160 con 0x7fc8f00c45f0 2023-06-22T08:18:56.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.609+0000 7fc8f639b700 1 --2- 172.21.15.23:0/1374661842 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0152d70 0x7fc8f00c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:56.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.609+0000 7fc8f6b9c700 1 --2- 172.21.15.23:0/1374661842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f00c45f0 0x7fc8f00c4a70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009e50 tx=0x7fc8e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a668f6755077d4b4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:56.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.609+0000 7fc8f5399700 1 -- 172.21.15.23:0/1374661842 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e800e070 con 0x7fc8f00c45f0 2023-06-22T08:18:56.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.609+0000 7fc8f5399700 1 -- 172.21.15.23:0/1374661842 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e800b910 con 0x7fc8f00c45f0 2023-06-22T08:18:56.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.609+0000 7fc8f5399700 1 -- 172.21.15.23:0/1374661842 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e800bcc0 con 0x7fc8f00c45f0 2023-06-22T08:18:56.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.610+0000 7fc8f85ff700 1 -- 172.21.15.23:0/1374661842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f00c45f0 msgr2=0x7fc8f00c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.610+0000 7fc8f85ff700 1 --2- 172.21.15.23:0/1374661842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f00c45f0 0x7fc8f00c4a70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009e50 tx=0x7fc8e800b330 comp rx=0 tx=0).stop 2023-06-22T08:18:56.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.610+0000 7fc8f85ff700 1 -- 172.21.15.23:0/1374661842 shutdown_connections 2023-06-22T08:18:56.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.610+0000 7fc8f85ff700 1 --2- 172.21.15.23:0/1374661842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f00c45f0 0x7fc8f00c4a70 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.610+0000 7fc8f85ff700 1 --2- 172.21.15.23:0/1374661842 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0152d70 0x7fc8f00c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.610+0000 7fc8f85ff700 1 -- 172.21.15.23:0/1374661842 >> 172.21.15.23:0/1374661842 conn(0x7fc8f006ae30 msgr2=0x7fc8f006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:56.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.610+0000 7fc8f85ff700 1 -- 172.21.15.23:0/1374661842 shutdown_connections 2023-06-22T08:18:56.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.610+0000 7fc8f85ff700 1 -- 172.21.15.23:0/1374661842 wait complete. 2023-06-22T08:18:56.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.611+0000 7fc8f85ff700 1 Processor -- start 2023-06-22T08:18:56.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.612+0000 7fc8f85ff700 1 -- start start 2023-06-22T08:18:56.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.612+0000 7fc8f85ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f00c45f0 0x7fc8f014dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.612+0000 7fc8f85ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f0152d70 0x7fc8f0157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.612+0000 7fc8f85ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0159330 0x7fc8f0157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.612+0000 7fc8f85ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f01ef0c0 con 0x7fc8f0152d70 2023-06-22T08:18:56.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.612+0000 7fc8f85ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f01eeb20 con 0x7fc8f00c45f0 2023-06-22T08:18:56.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.612+0000 7fc8f85ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f00c8f10 con 0x7fc8f0159330 2023-06-22T08:18:56.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f5b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f0152d70 0x7fc8f0157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f5b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f0152d70 0x7fc8f0157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55476/0 (socket says 172.21.15.23:55476) 2023-06-22T08:18:56.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f5b9a700 1 -- 172.21.15.23:0/3221216725 learned_addr learned my addr 172.21.15.23:0/3221216725 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:56.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f639b700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f00c45f0 0x7fc8f014dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f6b9c700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0159330 0x7fc8f0157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f5b9a700 1 -- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0159330 msgr2=0x7fc8f0157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f5b9a700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0159330 0x7fc8f0157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f5b9a700 1 -- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f00c45f0 msgr2=0x7fc8f014dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f5b9a700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f00c45f0 0x7fc8f014dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f5b9a700 1 -- 172.21.15.23:0/3221216725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec0094a0 con 0x7fc8f0152d70 2023-06-22T08:18:56.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.613+0000 7fc8f5b9a700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f0152d70 0x7fc8f0157240 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000c450 tx=0x7fc8e000c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:56.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.614+0000 7fc8e6ffd700 1 -- 172.21.15.23:0/3221216725 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e00052d0 con 0x7fc8f0152d70 2023-06-22T08:18:56.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.614+0000 7fc8e6ffd700 1 -- 172.21.15.23:0/3221216725 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e0005430 con 0x7fc8f0152d70 2023-06-22T08:18:56.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.614+0000 7fc8e6ffd700 1 -- 172.21.15.23:0/3221216725 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e00038c0 con 0x7fc8f0152d70 2023-06-22T08:18:56.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.614+0000 7fc8f85ff700 1 -- 172.21.15.23:0/3221216725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e8009b30 con 0x7fc8f0152d70 2023-06-22T08:18:56.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.614+0000 7fc8f85ff700 1 -- 172.21.15.23:0/3221216725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f01eede0 con 0x7fc8f0152d70 2023-06-22T08:18:56.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.615+0000 7fc8f85ff700 1 -- 172.21.15.23:0/3221216725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f004e1d0 con 0x7fc8f0152d70 2023-06-22T08:18:56.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.616+0000 7fc8e6ffd700 1 -- 172.21.15.23:0/3221216725 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc8e0003a20 con 0x7fc8f0152d70 2023-06-22T08:18:56.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.617+0000 7fc8e6ffd700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8d406ace0 0x7fc8d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:56.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.617+0000 7fc8e6ffd700 1 -- 172.21.15.23:0/3221216725 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7fc8e0088f20 con 0x7fc8f0152d70 2023-06-22T08:18:56.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.618+0000 7fc8f639b700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8d406ace0 0x7fc8d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:56.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.618+0000 7fc8f639b700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8d406ace0 0x7fc8d406d190 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009d70 tx=0x7fc8ec009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:56.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.622+0000 7fc8e6ffd700 1 -- 172.21.15.23:0/3221216725 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc8e0053850 con 0x7fc8f0152d70 2023-06-22T08:18:56.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.872+0000 7fc8f85ff700 1 -- 172.21.15.23:0/3221216725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8f004ea20 con 0x7fc8f0152d70 2023-06-22T08:18:56.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.873+0000 7fc8e6ffd700 1 -- 172.21.15.23:0/3221216725 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5509 (secure 0 0 0) 0x7fc8e0017020 con 0x7fc8f0152d70 2023-06-22T08:18:56.876 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:56.876 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:54.559203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:56.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 -- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8d406ace0 msgr2=0x7fc8d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8d406ace0 0x7fc8d406d190 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009d70 tx=0x7fc8ec009210 comp rx=0 tx=0).stop 2023-06-22T08:18:56.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 -- 172.21.15.23:0/3221216725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f0152d70 msgr2=0x7fc8f0157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:56.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f0152d70 0x7fc8f0157240 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc8e000c450 tx=0x7fc8e000c760 comp rx=0 tx=0).stop 2023-06-22T08:18:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 -- 172.21.15.23:0/3221216725 shutdown_connections 2023-06-22T08:18:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc8f0159330 0x7fc8f0157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc8d406ace0 0x7fc8d406d190 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8f00c45f0 0x7fc8f014dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 --2- 172.21.15.23:0/3221216725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8f0152d70 0x7fc8f0157240 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 -- 172.21.15.23:0/3221216725 >> 172.21.15.23:0/3221216725 conn(0x7fc8f006ae30 msgr2=0x7fc8f0150420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:56.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 -- 172.21.15.23:0/3221216725 shutdown_connections 2023-06-22T08:18:56.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:56.875+0000 7fc8f85ff700 1 -- 172.21.15.23:0/3221216725 wait complete. 2023-06-22T08:18:56.880 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 151 2023-06-22T08:18:56.897 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 151, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 151, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 }, "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 36 }, "gid_6318": { "addr": "172.21.15.23:6834/904774922", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 904774922, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 904774922, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 151, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:18:54.559203+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 6313, "mds_3": 5995, "mds_4": 6318 } } } ], "standbys": [] } 2023-06-22T08:18:58.898 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:18:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.106+0000 7fbd269ff700 1 Processor -- start 2023-06-22T08:18:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.106+0000 7fbd269ff700 1 -- start start 2023-06-22T08:18:59.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.108+0000 7fbd269ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd20152ce0 0x7fbd200cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:59.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.108+0000 7fbd269ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd200c9fd0 0x7fbd2015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:59.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.108+0000 7fbd269ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2015c370 con 0x7fbd20158880 2023-06-22T08:18:59.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.108+0000 7fbd269ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd201530b0 con 0x7fbd200c9fd0 2023-06-22T08:18:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.108+0000 7fbd269ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd200cf6f0 con 0x7fbd20152ce0 2023-06-22T08:18:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.108+0000 7fbd1ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd20152ce0 0x7fbd200cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.108+0000 7fbd1f7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd200c9fd0 0x7fbd2015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.108+0000 7fbd24f9c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbd20158880 0x7fbd20158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:32952/0 (socket says 172.21.15.23:32952) 2023-06-22T08:18:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.108+0000 7fbd1f7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd200c9fd0 0x7fbd2015c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51882/0 (socket says 172.21.15.23:51882) 2023-06-22T08:18:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.108+0000 7fbd24f9c700 1 -- 172.21.15.23:0/3474182036 learned_addr learned my addr 172.21.15.23:0/3474182036 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:18:59.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.109+0000 7fbd1f7fe700 1 -- 172.21.15.23:0/3474182036 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd20152ce0 msgr2=0x7fbd200cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:18:59.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.109+0000 7fbd1f7fe700 1 --2- 172.21.15.23:0/3474182036 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd20152ce0 0x7fbd200cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:59.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.109+0000 7fbd1f7fe700 1 -- 172.21.15.23:0/3474182036 >> v1:172.21.15.67:6789/0 conn(0x7fbd20158880 legacy=0x7fbd20158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:59.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.109+0000 7fbd1f7fe700 1 -- 172.21.15.23:0/3474182036 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2015bda0 con 0x7fbd200c9fd0 2023-06-22T08:18:59.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.109+0000 7fbd1ffff700 1 --2- 172.21.15.23:0/3474182036 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd20152ce0 0x7fbd200cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:59.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.109+0000 7fbd1effd700 1 -- 172.21.15.23:0/3474182036 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295417102 0 0) 0x7fbd2015c370 con 0x7fbd20158880 2023-06-22T08:18:59.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.109+0000 7fbd1f7fe700 1 --2- 172.21.15.23:0/3474182036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd200c9fd0 0x7fbd2015c800 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009840 tx=0x7fbd14009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeb0aad66680ec7a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:59.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.109+0000 7fbd1effd700 1 -- 172.21.15.23:0/3474182036 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14011830 con 0x7fbd200c9fd0 2023-06-22T08:18:59.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.109+0000 7fbd1effd700 1 -- 172.21.15.23:0/3474182036 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd14011990 con 0x7fbd200c9fd0 2023-06-22T08:18:59.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.109+0000 7fbd1effd700 1 -- 172.21.15.23:0/3474182036 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14011d20 con 0x7fbd200c9fd0 2023-06-22T08:18:59.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.110+0000 7fbd269ff700 1 -- 172.21.15.23:0/3474182036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd200c9fd0 msgr2=0x7fbd2015c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:59.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.110+0000 7fbd269ff700 1 --2- 172.21.15.23:0/3474182036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd200c9fd0 0x7fbd2015c800 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009840 tx=0x7fbd14009b50 comp rx=0 tx=0).stop 2023-06-22T08:18:59.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.110+0000 7fbd269ff700 1 -- 172.21.15.23:0/3474182036 shutdown_connections 2023-06-22T08:18:59.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.110+0000 7fbd269ff700 1 --2- 172.21.15.23:0/3474182036 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd20152ce0 0x7fbd200cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:59.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.110+0000 7fbd269ff700 1 --2- 172.21.15.23:0/3474182036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd200c9fd0 0x7fbd2015c800 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:59.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.110+0000 7fbd269ff700 1 -- 172.21.15.23:0/3474182036 >> 172.21.15.23:0/3474182036 conn(0x7fbd2006ae30 msgr2=0x7fbd2006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:59.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.110+0000 7fbd269ff700 1 -- 172.21.15.23:0/3474182036 shutdown_connections 2023-06-22T08:18:59.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.110+0000 7fbd269ff700 1 -- 172.21.15.23:0/3474182036 wait complete. 2023-06-22T08:18:59.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.112+0000 7fbd269ff700 1 Processor -- start 2023-06-22T08:18:59.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.113+0000 7fbd269ff700 1 -- start start 2023-06-22T08:18:59.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd269ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd200c9fd0 0x7fbd20147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd269ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd20152ce0 0x7fbd200ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd269ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd20158880 0x7fbd200cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd269ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd200ced60 con 0x7fbd20158880 2023-06-22T08:18:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd269ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd201f2430 con 0x7fbd20152ce0 2023-06-22T08:18:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd269ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd200cc690 con 0x7fbd200c9fd0 2023-06-22T08:18:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd24f9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd20158880 0x7fbd200cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd1f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd20152ce0 0x7fbd200ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:59.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd1ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd200c9fd0 0x7fbd20147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:59.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd1ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd200c9fd0 0x7fbd20147690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:46490/0 (socket says 172.21.15.23:46490) 2023-06-22T08:18:59.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd1ffff700 1 -- 172.21.15.23:0/2977999921 learned_addr learned my addr 172.21.15.23:0/2977999921 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:18:59.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd1f7fe700 1 -- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd200c9fd0 msgr2=0x7fbd20147690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:59.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.114+0000 7fbd1f7fe700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd200c9fd0 0x7fbd20147690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:59.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.115+0000 7fbd1f7fe700 1 -- 172.21.15.23:0/2977999921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd20158880 msgr2=0x7fbd200cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:59.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.115+0000 7fbd1f7fe700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd20158880 0x7fbd200cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:59.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.115+0000 7fbd1f7fe700 1 -- 172.21.15.23:0/2977999921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd100094a0 con 0x7fbd20152ce0 2023-06-22T08:18:59.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.115+0000 7fbd24f9c700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd20158880 0x7fbd200cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:18:59.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.115+0000 7fbd1f7fe700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd20152ce0 0x7fbd200ccae0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbd2008b9e0 tx=0x7fbd14004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.115+0000 7fbd1cff9700 1 -- 172.21.15.23:0/2977999921 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd140036a0 con 0x7fbd20152ce0 2023-06-22T08:18:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.115+0000 7fbd1cff9700 1 -- 172.21.15.23:0/2977999921 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1401f070 con 0x7fbd20152ce0 2023-06-22T08:18:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.115+0000 7fbd269ff700 1 -- 172.21.15.23:0/2977999921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd140094a0 con 0x7fbd20152ce0 2023-06-22T08:18:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.115+0000 7fbd1cff9700 1 -- 172.21.15.23:0/2977999921 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14003a20 con 0x7fbd20152ce0 2023-06-22T08:18:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.115+0000 7fbd269ff700 1 -- 172.21.15.23:0/2977999921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd201f1ef0 con 0x7fbd20152ce0 2023-06-22T08:18:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.116+0000 7fbd269ff700 1 -- 172.21.15.23:0/2977999921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd201f21f0 con 0x7fbd20152ce0 2023-06-22T08:18:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.117+0000 7fbd1cff9700 1 -- 172.21.15.23:0/2977999921 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbd14003b80 con 0x7fbd20152ce0 2023-06-22T08:18:59.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.118+0000 7fbd1cff9700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbd0c06ad10 0x7fbd0c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:18:59.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.118+0000 7fbd1cff9700 1 -- 172.21.15.23:0/2977999921 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7fbd14021070 con 0x7fbd20152ce0 2023-06-22T08:18:59.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.118+0000 7fbd1ffff700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbd0c06ad10 0x7fbd0c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:18:59.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.119+0000 7fbd1ffff700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbd0c06ad10 0x7fbd0c06d1c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009e70 tx=0x7fbd10009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:18:59.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.122+0000 7fbd1cff9700 1 -- 172.21.15.23:0/2977999921 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbd140561a0 con 0x7fbd20152ce0 2023-06-22T08:18:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.379+0000 7fbd269ff700 1 -- 172.21.15.23:0/2977999921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd200cc800 con 0x7fbd20152ce0 2023-06-22T08:18:59.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.381+0000 7fbd1cff9700 1 -- 172.21.15.23:0/2977999921 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5509 (secure 0 0 0) 0x7fbd200cc800 con 0x7fbd20152ce0 2023-06-22T08:18:59.383 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:18:59.383 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:54.559203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:18:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.382+0000 7fbd269ff700 1 -- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbd0c06ad10 msgr2=0x7fbd0c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:59.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.382+0000 7fbd269ff700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbd0c06ad10 0x7fbd0c06d1c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009e70 tx=0x7fbd10009210 comp rx=0 tx=0).stop 2023-06-22T08:18:59.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.382+0000 7fbd269ff700 1 -- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd20152ce0 msgr2=0x7fbd200ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:18:59.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.382+0000 7fbd269ff700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd20152ce0 0x7fbd200ccae0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbd2008b9e0 tx=0x7fbd14004750 comp rx=0 tx=0).stop 2023-06-22T08:18:59.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.383+0000 7fbd269ff700 1 -- 172.21.15.23:0/2977999921 shutdown_connections 2023-06-22T08:18:59.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.383+0000 7fbd269ff700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd20158880 0x7fbd200cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:59.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.383+0000 7fbd269ff700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbd200c9fd0 0x7fbd20147690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:59.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.383+0000 7fbd269ff700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbd0c06ad10 0x7fbd0c06d1c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:59.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.383+0000 7fbd269ff700 1 --2- 172.21.15.23:0/2977999921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd20152ce0 0x7fbd200ccae0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:18:59.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.383+0000 7fbd269ff700 1 -- 172.21.15.23:0/2977999921 >> 172.21.15.23:0/2977999921 conn(0x7fbd2006ae30 msgr2=0x7fbd2014b860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:18:59.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.383+0000 7fbd269ff700 1 -- 172.21.15.23:0/2977999921 shutdown_connections 2023-06-22T08:18:59.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:18:59.383+0000 7fbd269ff700 1 -- 172.21.15.23:0/2977999921 wait complete. 2023-06-22T08:18:59.387 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 151 2023-06-22T08:18:59.404 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:19:01.404 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:01.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.629+0000 7f8cf1dff700 1 Processor -- start 2023-06-22T08:19:01.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.629+0000 7f8cf1dff700 1 -- start start 2023-06-22T08:19:01.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8cf1dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cec152dd0 0x7f8cec0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:01.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8cf1dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec0c8bd0 0x7f8cec06afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:01.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8cf1dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8cec06b500 0x7f8cec0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:01.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8cf1dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0c7400 con 0x7f8cec06b500 2023-06-22T08:19:01.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8cf1dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0c81c0 con 0x7f8cec152dd0 2023-06-22T08:19:01.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8cf1dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0c84a0 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8ceb7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cec152dd0 0x7f8cec0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:01.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8ceaffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec0c8bd0 0x7f8cec06afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:01.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8ceb7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cec152dd0 0x7f8cec0aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51890/0 (socket says 172.21.15.23:51890) 2023-06-22T08:19:01.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8ceb7fe700 1 -- 172.21.15.23:0/3818851256 learned_addr learned my addr 172.21.15.23:0/3818851256 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:01.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.631+0000 7f8cebfff700 1 --2- 172.21.15.23:0/3818851256 >> v2:172.21.15.23:3300/0 conn(0x7f8cec06b500 0x7f8cec0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:01.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.632+0000 7f8ceaffd700 1 -- 172.21.15.23:0/3818851256 >> v2:172.21.15.67:3300/0 conn(0x7f8cec152dd0 msgr2=0x7f8cec0aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:19:01.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.632+0000 7f8ceaffd700 1 --2- 172.21.15.23:0/3818851256 >> v2:172.21.15.67:3300/0 conn(0x7f8cec152dd0 0x7f8cec0aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.632+0000 7f8ceaffd700 1 -- 172.21.15.23:0/3818851256 >> v2:172.21.15.23:3300/0 conn(0x7f8cec06b500 msgr2=0x7f8cec0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:19:01.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.632+0000 7f8ceaffd700 1 --2- 172.21.15.23:0/3818851256 >> v2:172.21.15.23:3300/0 conn(0x7f8cec06b500 0x7f8cec0cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.632+0000 7f8ceaffd700 1 -- 172.21.15.23:0/3818851256 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec0cd370 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.632+0000 7f8cebfff700 1 --2- 172.21.15.23:0/3818851256 >> v2:172.21.15.23:3300/0 conn(0x7f8cec06b500 0x7f8cec0cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:19:01.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.632+0000 7f8ceb7fe700 1 --2- 172.21.15.23:0/3818851256 >> v2:172.21.15.67:3300/0 conn(0x7f8cec152dd0 0x7f8cec0aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:19:01.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.632+0000 7f8ceaffd700 1 --2- 172.21.15.23:0/3818851256 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec0c8bd0 0x7f8cec06afc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8cec070880 tx=0x7f8ce0009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d063780fa3340939 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:01.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cea7fc700 1 -- 172.21.15.23:0/3818851256 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0011830 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cea7fc700 1 -- 172.21.15.23:0/3818851256 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce0011990 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cea7fc700 1 -- 172.21.15.23:0/3818851256 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0011d20 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cf1dff700 1 -- 172.21.15.23:0/3818851256 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec0c8bd0 msgr2=0x7f8cec06afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:01.643 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cf1dff700 1 --2- 172.21.15.23:0/3818851256 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec0c8bd0 0x7f8cec06afc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8cec070880 tx=0x7f8ce0009b20 comp rx=0 tx=0).stop 2023-06-22T08:19:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cf1dff700 1 -- 172.21.15.23:0/3818851256 shutdown_connections 2023-06-22T08:19:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cf1dff700 1 --2- 172.21.15.23:0/3818851256 >> v2:172.21.15.23:3300/0 conn(0x7f8cec06b500 0x7f8cec0cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cf1dff700 1 --2- 172.21.15.23:0/3818851256 >> v2:172.21.15.67:3300/0 conn(0x7f8cec152dd0 0x7f8cec0aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cf1dff700 1 --2- 172.21.15.23:0/3818851256 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec0c8bd0 0x7f8cec06afc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cf1dff700 1 -- 172.21.15.23:0/3818851256 >> 172.21.15.23:0/3818851256 conn(0x7f8cec06fb70 msgr2=0x7f8cec06ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.633+0000 7f8cf1dff700 1 -- 172.21.15.23:0/3818851256 shutdown_connections 2023-06-22T08:19:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.634+0000 7f8cf1dff700 1 -- 172.21.15.23:0/3818851256 wait complete. 2023-06-22T08:19:01.645 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.636+0000 7f8cf1dff700 1 Processor -- start 2023-06-22T08:19:01.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.636+0000 7f8cf1dff700 1 -- start start 2023-06-22T08:19:01.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.637+0000 7f8cf1dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec06b500 0x7f8cec14a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:01.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.637+0000 7f8cf1dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cec0c8bd0 0x7f8cec0c5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:01.646 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8cf1dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec152dd0 0x7f8cec0c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:01.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8cf1dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0c4f30 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8cf1dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0ae430 con 0x7f8cec06b500 2023-06-22T08:19:01.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8cf1dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0c4790 con 0x7f8cec152dd0 2023-06-22T08:19:01.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8ceaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cec0c8bd0 0x7f8cec0c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:01.647 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8ceaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cec0c8bd0 0x7f8cec0c5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55488/0 (socket says 172.21.15.23:55488) 2023-06-22T08:19:01.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8ceaffd700 1 -- 172.21.15.23:0/2515037045 learned_addr learned my addr 172.21.15.23:0/2515037045 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:01.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8cebfff700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec152dd0 0x7f8cec0c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:01.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8ceb7fe700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec06b500 0x7f8cec14a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:01.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8ceaffd700 1 -- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec152dd0 msgr2=0x7f8cec0c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:01.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8ceaffd700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec152dd0 0x7f8cec0c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8ceaffd700 1 -- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec06b500 msgr2=0x7f8cec14a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:01.648 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8ceaffd700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec06b500 0x7f8cec14a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.638+0000 7f8ceaffd700 1 -- 172.21.15.23:0/2515037045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce4009a70 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.639+0000 7f8ceaffd700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cec0c8bd0 0x7f8cec0c5350 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009b60 tx=0x7f8ce0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:01.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.639+0000 7f8cd3fff700 1 -- 172.21.15.23:0/2515037045 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0004510 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.639+0000 7f8cd3fff700 1 -- 172.21.15.23:0/2515037045 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce001f070 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.639+0000 7f8cd3fff700 1 -- 172.21.15.23:0/2515037045 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce00038e0 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.639+0000 7f8cf1dff700 1 -- 172.21.15.23:0/2515037045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cdc009500 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.649 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.639+0000 7f8cf1dff700 1 -- 172.21.15.23:0/2515037045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce00094a0 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.650 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.640+0000 7f8cf1dff700 1 -- 172.21.15.23:0/2515037045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec04ea20 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.641+0000 7f8cd3fff700 1 -- 172.21.15.23:0/2515037045 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8ce0003a40 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.642+0000 7f8cd3fff700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cd806ace0 0x7f8cd806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:01.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.642+0000 7f8cd3fff700 1 -- 172.21.15.23:0/2515037045 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f8ce0021070 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.643+0000 7f8ceb7fe700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cd806ace0 0x7f8cd806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:01.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.644+0000 7f8ceb7fe700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cd806ace0 0x7f8cd806d190 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc006fd0 tx=0x7f8cdc009290 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:01.651 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.647+0000 7f8cd3fff700 1 -- 172.21.15.23:0/2515037045 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8ce0055b20 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.928+0000 7f8cf1dff700 1 -- 172.21.15.23:0/2515037045 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cec0c3830 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.929+0000 7f8cd3fff700 1 -- 172.21.15.23:0/2515037045 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5509 (secure 0 0 0) 0x7f8ce0015090 con 0x7f8cec0c8bd0 2023-06-22T08:19:01.932 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:01.932 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:18:54.559203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:01.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.931+0000 7f8cf1dff700 1 -- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cd806ace0 msgr2=0x7f8cd806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:01.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.931+0000 7f8cf1dff700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cd806ace0 0x7f8cd806d190 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc006fd0 tx=0x7f8cdc009290 comp rx=0 tx=0).stop 2023-06-22T08:19:01.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.931+0000 7f8cf1dff700 1 -- 172.21.15.23:0/2515037045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cec0c8bd0 msgr2=0x7f8cec0c5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:01.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.931+0000 7f8cf1dff700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cec0c8bd0 0x7f8cec0c5350 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009b60 tx=0x7f8ce0004750 comp rx=0 tx=0).stop 2023-06-22T08:19:01.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.931+0000 7f8cf1dff700 1 -- 172.21.15.23:0/2515037045 shutdown_connections 2023-06-22T08:19:01.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.931+0000 7f8cf1dff700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8cec152dd0 0x7f8cec0c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.931+0000 7f8cf1dff700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8cd806ace0 0x7f8cd806d190 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.931+0000 7f8cf1dff700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cec06b500 0x7f8cec14a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.931+0000 7f8cf1dff700 1 --2- 172.21.15.23:0/2515037045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cec0c8bd0 0x7f8cec0c5350 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:01.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.931+0000 7f8cf1dff700 1 -- 172.21.15.23:0/2515037045 >> 172.21.15.23:0/2515037045 conn(0x7f8cec06fb70 msgr2=0x7f8cec15b360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:01.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.932+0000 7f8cf1dff700 1 -- 172.21.15.23:0/2515037045 shutdown_connections 2023-06-22T08:19:01.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:01.932+0000 7f8cf1dff700 1 -- 172.21.15.23:0/2515037045 wait complete. 2023-06-22T08:19:01.936 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 151 2023-06-22T08:19:01.962 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:19:03.963 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:04.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.170+0000 7f90e21ff700 1 Processor -- start 2023-06-22T08:19:04.173 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.170+0000 7f90e21ff700 1 -- start start 2023-06-22T08:19:04.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.171+0000 7f90e21ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 0x7f90dc06df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:04.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.171+0000 7f90e21ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f90dc06e450 0x7f90dc06d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.171+0000 7f90e21ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90dc0c4ea0 con 0x7f90dc14f0c0 2023-06-22T08:19:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.171+0000 7f90e21ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90dc0c1f70 con 0x7f90dc06e450 2023-06-22T08:19:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.171+0000 7f90e21ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0c68f0 con 0x7f90dc1558b0 2023-06-22T08:19:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90daffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 0x7f90dc06df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90db7fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f90dc14f0c0 0x7f90dc0c61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:32962/0 (socket says 172.21.15.23:32962) 2023-06-22T08:19:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90db7fe700 1 -- 172.21.15.23:0/634630565 learned_addr learned my addr 172.21.15.23:0/634630565 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:19:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90dbfff700 1 --2- 172.21.15.23:0/634630565 >> v2:172.21.15.23:3300/0 conn(0x7f90dc06e450 0x7f90dc06d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:04.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90daffd700 1 -- 172.21.15.23:0/634630565 >> v2:172.21.15.23:3300/0 conn(0x7f90dc06e450 msgr2=0x7f90dc06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:19:04.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90da7fc700 1 -- 172.21.15.23:0/634630565 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667462188 0 0) 0x7f90dc0c4ea0 con 0x7f90dc14f0c0 2023-06-22T08:19:04.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90daffd700 1 --2- 172.21.15.23:0/634630565 >> v2:172.21.15.23:3300/0 conn(0x7f90dc06e450 0x7f90dc06d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:04.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90daffd700 1 -- 172.21.15.23:0/634630565 >> v1:172.21.15.67:6789/0 conn(0x7f90dc14f0c0 legacy=0x7f90dc0c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:04.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90daffd700 1 -- 172.21.15.23:0/634630565 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc155ce0 con 0x7f90dc1558b0 2023-06-22T08:19:04.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90dbfff700 1 --2- 172.21.15.23:0/634630565 >> v2:172.21.15.23:3300/0 conn(0x7f90dc06e450 0x7f90dc06d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:19:04.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.172+0000 7f90daffd700 1 --2- 172.21.15.23:0/634630565 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 0x7f90dc06df10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009840 tx=0x7f90d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45e53e510328c4b5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:04.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90da7fc700 1 -- 172.21.15.23:0/634630565 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d0011830 con 0x7f90dc1558b0 2023-06-22T08:19:04.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90da7fc700 1 -- 172.21.15.23:0/634630565 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d0011990 con 0x7f90dc1558b0 2023-06-22T08:19:04.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90da7fc700 1 -- 172.21.15.23:0/634630565 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d0011d20 con 0x7f90dc1558b0 2023-06-22T08:19:04.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90e21ff700 1 -- 172.21.15.23:0/634630565 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 msgr2=0x7f90dc06df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:04.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90e21ff700 1 --2- 172.21.15.23:0/634630565 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 0x7f90dc06df10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009840 tx=0x7f90d0009b50 comp rx=0 tx=0).stop 2023-06-22T08:19:04.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90e21ff700 1 -- 172.21.15.23:0/634630565 shutdown_connections 2023-06-22T08:19:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90e21ff700 1 --2- 172.21.15.23:0/634630565 >> v2:172.21.15.23:3300/0 conn(0x7f90dc06e450 0x7f90dc06d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90e21ff700 1 --2- 172.21.15.23:0/634630565 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 0x7f90dc06df10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90e21ff700 1 -- 172.21.15.23:0/634630565 >> 172.21.15.23:0/634630565 conn(0x7f90dc0ca3f0 msgr2=0x7f90dc0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90e21ff700 1 -- 172.21.15.23:0/634630565 shutdown_connections 2023-06-22T08:19:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.173+0000 7f90e21ff700 1 -- 172.21.15.23:0/634630565 wait complete. 2023-06-22T08:19:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.176+0000 7f90e21ff700 1 Processor -- start 2023-06-22T08:19:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.176+0000 7f90e21ff700 1 -- start start 2023-06-22T08:19:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.177+0000 7f90e21ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc06e450 0x7f90dc1ef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:04.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.177+0000 7f90e21ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc14f0c0 0x7f90dc0c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:04.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.177+0000 7f90e21ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 0x7f90dc0c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:04.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.177+0000 7f90e21ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc14bde0 con 0x7f90dc06e450 2023-06-22T08:19:04.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.177+0000 7f90e21ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc14c200 con 0x7f90dc14f0c0 2023-06-22T08:19:04.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.177+0000 7f90e21ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc0ca180 con 0x7f90dc1558b0 2023-06-22T08:19:04.181 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.177+0000 7f90db7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc06e450 0x7f90dc1ef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.177+0000 7f90daffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc14f0c0 0x7f90dc0c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.177+0000 7f90dbfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 0x7f90dc0c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.177+0000 7f90db7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc06e450 0x7f90dc1ef7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55518/0 (socket says 172.21.15.23:55518) 2023-06-22T08:19:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90db7fe700 1 -- 172.21.15.23:0/1775624556 learned_addr learned my addr 172.21.15.23:0/1775624556 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:04.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90db7fe700 1 -- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 msgr2=0x7f90dc0c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:04.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90db7fe700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 0x7f90dc0c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:04.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90db7fe700 1 -- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc14f0c0 msgr2=0x7f90dc0c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:04.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90db7fe700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc14f0c0 0x7f90dc0c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:04.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90db7fe700 1 -- 172.21.15.23:0/1775624556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4009a70 con 0x7f90dc06e450 2023-06-22T08:19:04.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90daffd700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc14f0c0 0x7f90dc0c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:19:04.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90db7fe700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc06e450 0x7f90dc1ef7d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00c750 tx=0x7f90cc00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:04.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90c3fff700 1 -- 172.21.15.23:0/1775624556 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc004c80 con 0x7f90dc06e450 2023-06-22T08:19:04.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90c3fff700 1 -- 172.21.15.23:0/1775624556 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90cc004e00 con 0x7f90dc06e450 2023-06-22T08:19:04.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.178+0000 7f90e21ff700 1 -- 172.21.15.23:0/1775624556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d00094a0 con 0x7f90dc06e450 2023-06-22T08:19:04.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.179+0000 7f90c3fff700 1 -- 172.21.15.23:0/1775624556 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc007310 con 0x7f90dc06e450 2023-06-22T08:19:04.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.179+0000 7f90e21ff700 1 -- 172.21.15.23:0/1775624556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc154a90 con 0x7f90dc06e450 2023-06-22T08:19:04.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.180+0000 7f90e21ff700 1 -- 172.21.15.23:0/1775624556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90dc1536a0 con 0x7f90dc06e450 2023-06-22T08:19:04.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.181+0000 7f90c3fff700 1 -- 172.21.15.23:0/1775624556 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f90cc007470 con 0x7f90dc06e450 2023-06-22T08:19:04.185 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.181+0000 7f90c3fff700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f90c806ad10 0x7f90c806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:04.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.182+0000 7f90c3fff700 1 -- 172.21.15.23:0/1775624556 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f90cc089830 con 0x7f90dc06e450 2023-06-22T08:19:04.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.182+0000 7f90daffd700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f90c806ad10 0x7f90c806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:04.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.182+0000 7f90daffd700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f90c806ad10 0x7f90c806d1c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f90d00090b0 tx=0x7f90d0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:04.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.186+0000 7f90c3fff700 1 -- 172.21.15.23:0/1775624556 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f90cc054200 con 0x7f90dc06e450 2023-06-22T08:19:04.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.439+0000 7f90e21ff700 1 -- 172.21.15.23:0/1775624556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90dc04ea20 con 0x7f90dc06e450 2023-06-22T08:19:04.443 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.439+0000 7f90c3fff700 1 -- 172.21.15.23:0/1775624556 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5511 (secure 0 0 0) 0x7f90cc057820 con 0x7f90dc06e450 2023-06-22T08:19:04.443 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:04.444 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:19:02.957935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:resolve","state_seq":33,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:04.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 -- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f90c806ad10 msgr2=0x7f90c806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:04.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f90c806ad10 0x7f90c806d1c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f90d00090b0 tx=0x7f90d0008030 comp rx=0 tx=0).stop 2023-06-22T08:19:04.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 -- 172.21.15.23:0/1775624556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc06e450 msgr2=0x7f90dc1ef7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:04.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc06e450 0x7f90dc1ef7d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00c750 tx=0x7f90cc00ca60 comp rx=0 tx=0).stop 2023-06-22T08:19:04.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 -- 172.21.15.23:0/1775624556 shutdown_connections 2023-06-22T08:19:04.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90dc1558b0 0x7f90dc0c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:04.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90dc06e450 0x7f90dc1ef7d0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:04.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f90c806ad10 0x7f90c806d1c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:04.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 --2- 172.21.15.23:0/1775624556 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90dc14f0c0 0x7f90dc0c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:04.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 -- 172.21.15.23:0/1775624556 >> 172.21.15.23:0/1775624556 conn(0x7f90dc0ca3f0 msgr2=0x7f90dc0c2840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:04.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 -- 172.21.15.23:0/1775624556 shutdown_connections 2023-06-22T08:19:04.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:04.442+0000 7f90e21ff700 1 -- 172.21.15.23:0/1775624556 wait complete. 2023-06-22T08:19:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 152 2023-06-22T08:19:04.463 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:19:06.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:06.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.695+0000 7fdd61d9d700 1 Processor -- start 2023-06-22T08:19:06.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.695+0000 7fdd61d9d700 1 -- start start 2023-06-22T08:19:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.696+0000 7fdd61d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdd5c0c9890 0x7fdd5c06bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.696+0000 7fdd61d9d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0c8f80 con 0x7fdd5c0c9c60 2023-06-22T08:19:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.696+0000 7fdd61d9d700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5c0c94b0 con 0x7fdd5c06ae60 2023-06-22T08:19:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.696+0000 7fdd61d9d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5c0b4620 con 0x7fdd5c0c9890 2023-06-22T08:19:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.697+0000 7fdd5bfff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fdd5c0c9c60 0x7fdd5c158eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58074/0 (socket says 172.21.15.23:58074) 2023-06-22T08:19:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.697+0000 7fdd5bfff700 1 -- 172.21.15.23:0/1007843647 learned_addr learned my addr 172.21.15.23:0/1007843647 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:19:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.697+0000 7fdd60d9b700 1 --2- 172.21.15.23:0/1007843647 >> v2:172.21.15.23:3300/0 conn(0x7fdd5c0c9890 0x7fdd5c06bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:06.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.697+0000 7fdd60d9b700 1 -- 172.21.15.23:0/1007843647 >> v1:172.21.15.67:6789/0 conn(0x7fdd5c06ae60 legacy=0x7fdd5c06b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:06.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.697+0000 7fdd60d9b700 1 -- 172.21.15.23:0/1007843647 >> v1:172.21.15.23:6789/0 conn(0x7fdd5c0c9c60 legacy=0x7fdd5c158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.697+0000 7fdd60d9b700 1 -- 172.21.15.23:0/1007843647 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0c8060 con 0x7fdd5c0c9890 2023-06-22T08:19:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.697+0000 7fdd60d9b700 1 --2- 172.21.15.23:0/1007843647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd5c0c9890 0x7fdd5c06bc40 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009840 tx=0x7fdd50009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d564331bf52729c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.698+0000 7fdd5b7fe700 1 -- 172.21.15.23:0/1007843647 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50011830 con 0x7fdd5c0c9890 2023-06-22T08:19:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.698+0000 7fdd5b7fe700 1 -- 172.21.15.23:0/1007843647 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd50011990 con 0x7fdd5c0c9890 2023-06-22T08:19:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.698+0000 7fdd5b7fe700 1 -- 172.21.15.23:0/1007843647 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50011d20 con 0x7fdd5c0c9890 2023-06-22T08:19:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.698+0000 7fdd61d9d700 1 -- 172.21.15.23:0/1007843647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd5c0c9890 msgr2=0x7fdd5c06bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.698+0000 7fdd61d9d700 1 --2- 172.21.15.23:0/1007843647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd5c0c9890 0x7fdd5c06bc40 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009840 tx=0x7fdd50009b50 comp rx=0 tx=0).stop 2023-06-22T08:19:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.698+0000 7fdd61d9d700 1 -- 172.21.15.23:0/1007843647 shutdown_connections 2023-06-22T08:19:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.698+0000 7fdd61d9d700 1 --2- 172.21.15.23:0/1007843647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd5c0c9890 0x7fdd5c06bc40 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.698+0000 7fdd61d9d700 1 -- 172.21.15.23:0/1007843647 >> 172.21.15.23:0/1007843647 conn(0x7fdd5c065b90 msgr2=0x7fdd5c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.698+0000 7fdd61d9d700 1 -- 172.21.15.23:0/1007843647 shutdown_connections 2023-06-22T08:19:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.699+0000 7fdd61d9d700 1 -- 172.21.15.23:0/1007843647 wait complete. 2023-06-22T08:19:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.700+0000 7fdd61d9d700 1 Processor -- start 2023-06-22T08:19:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.700+0000 7fdd61d9d700 1 -- start start 2023-06-22T08:19:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.701+0000 7fdd61d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd5c06ae60 0x7fdd5c1ee960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.701+0000 7fdd61d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0c9890 0x7fdd5c150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.701+0000 7fdd61d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdd5c0c9c60 0x7fdd5c14f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.701+0000 7fdd61d9d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c1edf50 con 0x7fdd5c06ae60 2023-06-22T08:19:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.701+0000 7fdd61d9d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0b12c0 con 0x7fdd5c0c9890 2023-06-22T08:19:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.701+0000 7fdd61d9d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c1eeea0 con 0x7fdd5c0c9c60 2023-06-22T08:19:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.701+0000 7fdd60d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd5c06ae60 0x7fdd5c1ee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd5bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0c9890 0x7fdd5c150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd6159c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdd5c0c9c60 0x7fdd5c14f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd5bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0c9890 0x7fdd5c150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:59938/0 (socket says 172.21.15.23:59938) 2023-06-22T08:19:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd5bfff700 1 -- 172.21.15.23:0/3845677522 learned_addr learned my addr 172.21.15.23:0/3845677522 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd60d9b700 1 -- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdd5c0c9c60 msgr2=0x7fdd5c14f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd60d9b700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdd5c0c9c60 0x7fdd5c14f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd60d9b700 1 -- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0c9890 msgr2=0x7fdd5c150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd60d9b700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0c9890 0x7fdd5c150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd60d9b700 1 -- 172.21.15.23:0/3845677522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500094a0 con 0x7fdd5c06ae60 2023-06-22T08:19:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd60d9b700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd5c06ae60 0x7fdd5c1ee960 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c08b5c0 tx=0x7fdd50004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd597fa700 1 -- 172.21.15.23:0/3845677522 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd500036a0 con 0x7fdd5c06ae60 2023-06-22T08:19:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd597fa700 1 -- 172.21.15.23:0/3845677522 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5001f070 con 0x7fdd5c06ae60 2023-06-22T08:19:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd597fa700 1 -- 172.21.15.23:0/3845677522 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd50003a40 con 0x7fdd5c06ae60 2023-06-22T08:19:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.702+0000 7fdd61d9d700 1 -- 172.21.15.23:0/3845677522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5c156e20 con 0x7fdd5c06ae60 2023-06-22T08:19:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.703+0000 7fdd61d9d700 1 -- 172.21.15.23:0/3845677522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5c1573e0 con 0x7fdd5c06ae60 2023-06-22T08:19:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.704+0000 7fdd61d9d700 1 -- 172.21.15.23:0/3845677522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5c06b2a0 con 0x7fdd5c06ae60 2023-06-22T08:19:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.704+0000 7fdd597fa700 1 -- 172.21.15.23:0/3845677522 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fdd50003ba0 con 0x7fdd5c06ae60 2023-06-22T08:19:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.706+0000 7fdd597fa700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd4006ace0 0x7fdd4006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.706+0000 7fdd597fa700 1 -- 172.21.15.23:0/3845677522 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7fdd50021070 con 0x7fdd5c06ae60 2023-06-22T08:19:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.706+0000 7fdd5bfff700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd4006ace0 0x7fdd4006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.707+0000 7fdd5bfff700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd4006ace0 0x7fdd4006d190 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c06ab20 tx=0x7fdd48009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.710+0000 7fdd597fa700 1 -- 172.21.15.23:0/3845677522 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fdd50055e60 con 0x7fdd5c06ae60 2023-06-22T08:19:06.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.964+0000 7fdd61d9d700 1 -- 172.21.15.23:0/3845677522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd5c0cb820 con 0x7fdd5c06ae60 2023-06-22T08:19:06.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.964+0000 7fdd597fa700 1 -- 172.21.15.23:0/3845677522 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5511 (secure 0 0 0) 0x7fdd50059480 con 0x7fdd5c06ae60 2023-06-22T08:19:06.967 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:06.967 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:19:02.957935+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:resolve","state_seq":33,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:06.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 -- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd4006ace0 msgr2=0x7fdd4006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:06.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd4006ace0 0x7fdd4006d190 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c06ab20 tx=0x7fdd48009040 comp rx=0 tx=0).stop 2023-06-22T08:19:06.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 -- 172.21.15.23:0/3845677522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd5c06ae60 msgr2=0x7fdd5c1ee960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:06.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd5c06ae60 0x7fdd5c1ee960 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c08b5c0 tx=0x7fdd50004750 comp rx=0 tx=0).stop 2023-06-22T08:19:06.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 -- 172.21.15.23:0/3845677522 shutdown_connections 2023-06-22T08:19:06.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdd5c0c9c60 0x7fdd5c14f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:06.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd5c06ae60 0x7fdd5c1ee960 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:06.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fdd4006ace0 0x7fdd4006d190 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:06.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 --2- 172.21.15.23:0/3845677522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd5c0c9890 0x7fdd5c150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:06.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 -- 172.21.15.23:0/3845677522 >> 172.21.15.23:0/3845677522 conn(0x7fdd5c065b90 msgr2=0x7fdd5c0b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:06.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 -- 172.21.15.23:0/3845677522 shutdown_connections 2023-06-22T08:19:06.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:06.967+0000 7fdd61d9d700 1 -- 172.21.15.23:0/3845677522 wait complete. 2023-06-22T08:19:06.972 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 152 2023-06-22T08:19:06.988 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:19:08.989 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:09.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.195+0000 7f15ed5ff700 1 Processor -- start 2023-06-22T08:19:09.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.196+0000 7f15ed5ff700 1 -- start start 2023-06-22T08:19:09.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.197+0000 7f15ed5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15e8071350 0x7f15e806a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:09.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.197+0000 7f15ed5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15e80c4e00 0x7f15e80c5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:09.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.197+0000 7f15ed5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e80c57c0 con 0x7f15e8069eb0 2023-06-22T08:19:09.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.197+0000 7f15ed5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e806a7c0 con 0x7f15e80c4e00 2023-06-22T08:19:09.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.197+0000 7f15ed5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80703c0 con 0x7f15e8071350 2023-06-22T08:19:09.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.197+0000 7f15e77fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15e80c4e00 0x7f15e80c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:09.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.197+0000 7f15e67fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15e8071350 0x7f15e806a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:09.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.197+0000 7f15e6ffd700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f15e8069eb0 0x7f15e806fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:48602/0 (socket says 172.21.15.23:48602) 2023-06-22T08:19:09.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.197+0000 7f15e6ffd700 1 -- 172.21.15.23:0/3478843377 learned_addr learned my addr 172.21.15.23:0/3478843377 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:19:09.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.198+0000 7f15e77fe700 1 -- 172.21.15.23:0/3478843377 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15e8071350 msgr2=0x7f15e806a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:19:09.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.198+0000 7f15e77fe700 1 --2- 172.21.15.23:0/3478843377 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15e8071350 0x7f15e806a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:09.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.198+0000 7f15e77fe700 1 -- 172.21.15.23:0/3478843377 >> v1:172.21.15.67:6789/0 conn(0x7f15e8069eb0 legacy=0x7f15e806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:09.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.198+0000 7f15e77fe700 1 -- 172.21.15.23:0/3478843377 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e8070680 con 0x7f15e80c4e00 2023-06-22T08:19:09.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.198+0000 7f15e77fe700 1 --2- 172.21.15.23:0/3478843377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e80c4e00 0x7f15e80c5280 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009d90 tx=0x7f15e000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f76d185d5904b376 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:09.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.198+0000 7f15e5ffb700 1 -- 172.21.15.23:0/3478843377 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e000e070 con 0x7f15e80c4e00 2023-06-22T08:19:09.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.198+0000 7f15e5ffb700 1 -- 172.21.15.23:0/3478843377 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e000bb20 con 0x7f15e80c4e00 2023-06-22T08:19:09.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.198+0000 7f15e5ffb700 1 -- 172.21.15.23:0/3478843377 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e0012430 con 0x7f15e80c4e00 2023-06-22T08:19:09.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.199+0000 7f15ed5ff700 1 -- 172.21.15.23:0/3478843377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e80c4e00 msgr2=0x7f15e80c5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:09.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.199+0000 7f15ed5ff700 1 --2- 172.21.15.23:0/3478843377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e80c4e00 0x7f15e80c5280 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009d90 tx=0x7f15e000b330 comp rx=0 tx=0).stop 2023-06-22T08:19:09.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.199+0000 7f15ed5ff700 1 -- 172.21.15.23:0/3478843377 shutdown_connections 2023-06-22T08:19:09.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.199+0000 7f15ed5ff700 1 --2- 172.21.15.23:0/3478843377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e80c4e00 0x7f15e80c5280 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:09.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.199+0000 7f15ed5ff700 1 --2- 172.21.15.23:0/3478843377 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15e8071350 0x7f15e806a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:09.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.199+0000 7f15ed5ff700 1 -- 172.21.15.23:0/3478843377 >> 172.21.15.23:0/3478843377 conn(0x7f15e80ca3f0 msgr2=0x7f15e80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:09.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.199+0000 7f15ed5ff700 1 -- 172.21.15.23:0/3478843377 shutdown_connections 2023-06-22T08:19:09.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.199+0000 7f15ed5ff700 1 -- 172.21.15.23:0/3478843377 wait complete. 2023-06-22T08:19:09.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.201+0000 7f15ed5ff700 1 Processor -- start 2023-06-22T08:19:09.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.201+0000 7f15ed5ff700 1 -- start start 2023-06-22T08:19:09.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15ed5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15e8069eb0 0x7f15e8151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:09.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15ed5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15e8071350 0x7f15e81e54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:09.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15ed5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e80c4e00 0x7f15e81ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:09.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15ed5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81ec250 con 0x7f15e8071350 2023-06-22T08:19:09.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15ed5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e8150930 con 0x7f15e80c4e00 2023-06-22T08:19:09.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15ed5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81ee510 con 0x7f15e8069eb0 2023-06-22T08:19:09.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15e6ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15e8069eb0 0x7f15e8151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:09.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15e67fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15e8071350 0x7f15e81e54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:09.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15e67fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15e8071350 0x7f15e81e54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57690/0 (socket says 172.21.15.23:57690) 2023-06-22T08:19:09.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15e77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e80c4e00 0x7f15e81ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:09.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15e67fc700 1 -- 172.21.15.23:0/355769188 learned_addr learned my addr 172.21.15.23:0/355769188 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:09.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.202+0000 7f15e67fc700 1 -- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15e8069eb0 msgr2=0x7f15e8151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:09.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.203+0000 7f15e67fc700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15e8069eb0 0x7f15e8151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:09.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.203+0000 7f15e67fc700 1 -- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e80c4e00 msgr2=0x7f15e81ee9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:09.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.203+0000 7f15e67fc700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e80c4e00 0x7f15e81ee9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:09.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.203+0000 7f15e67fc700 1 -- 172.21.15.23:0/355769188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e0009a70 con 0x7f15e8071350 2023-06-22T08:19:09.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.203+0000 7f15e67fc700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15e8071350 0x7f15e81e54f0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00e9b0 tx=0x7f15dc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:09.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.203+0000 7f15cf7fe700 1 -- 172.21.15.23:0/355769188 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc009710 con 0x7f15e8071350 2023-06-22T08:19:09.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.203+0000 7f15cf7fe700 1 -- 172.21.15.23:0/355769188 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc00ee50 con 0x7f15e8071350 2023-06-22T08:19:09.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.203+0000 7f15cf7fe700 1 -- 172.21.15.23:0/355769188 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc00ca50 con 0x7f15e8071350 2023-06-22T08:19:09.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.203+0000 7f15ed5ff700 1 -- 172.21.15.23:0/355769188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e81ee210 con 0x7f15e8071350 2023-06-22T08:19:09.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.203+0000 7f15ed5ff700 1 -- 172.21.15.23:0/355769188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e814cda0 con 0x7f15e8071350 2023-06-22T08:19:09.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.204+0000 7f15ed5ff700 1 -- 172.21.15.23:0/355769188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e81ee6a0 con 0x7f15e8071350 2023-06-22T08:19:09.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.206+0000 7f15cf7fe700 1 -- 172.21.15.23:0/355769188 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f15dc00cbb0 con 0x7f15e8071350 2023-06-22T08:19:09.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.207+0000 7f15cf7fe700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f15d406ace0 0x7f15d406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:09.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.207+0000 7f15cf7fe700 1 -- 172.21.15.23:0/355769188 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f15dc089ac0 con 0x7f15e8071350 2023-06-22T08:19:09.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.208+0000 7f15e6ffd700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f15d406ace0 0x7f15d406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:09.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.208+0000 7f15e6ffd700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f15d406ace0 0x7f15d406d190 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f15d8005910 tx=0x7f15d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:09.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.212+0000 7f15cf7fe700 1 -- 172.21.15.23:0/355769188 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f15dc057fb0 con 0x7f15e8071350 2023-06-22T08:19:09.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.468+0000 7f15ed5ff700 1 -- 172.21.15.23:0/355769188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15e80c3a30 con 0x7f15e8071350 2023-06-22T08:19:09.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.468+0000 7f15cf7fe700 1 -- 172.21.15.23:0/355769188 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5513 (secure 0 0 0) 0x7f15dc00ce60 con 0x7f15e8071350 2023-06-22T08:19:09.471 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:09.471 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:19:08.953676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 -- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f15d406ace0 msgr2=0x7f15d406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f15d406ace0 0x7f15d406d190 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f15d8005910 tx=0x7f15d8009040 comp rx=0 tx=0).stop 2023-06-22T08:19:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 -- 172.21.15.23:0/355769188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15e8071350 msgr2=0x7f15e81e54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:09.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15e8071350 0x7f15e81e54f0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00e9b0 tx=0x7f15dc00ecc0 comp rx=0 tx=0).stop 2023-06-22T08:19:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 -- 172.21.15.23:0/355769188 shutdown_connections 2023-06-22T08:19:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f15d406ace0 0x7f15d406d190 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15e80c4e00 0x7f15e81ee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15e8069eb0 0x7f15e8151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 --2- 172.21.15.23:0/355769188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15e8071350 0x7f15e81e54f0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 -- 172.21.15.23:0/355769188 >> 172.21.15.23:0/355769188 conn(0x7f15e80ca3f0 msgr2=0x7f15e806d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:09.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 -- 172.21.15.23:0/355769188 shutdown_connections 2023-06-22T08:19:09.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:09.471+0000 7f15ed5ff700 1 -- 172.21.15.23:0/355769188 wait complete. 2023-06-22T08:19:09.476 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 153 2023-06-22T08:19:09.494 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 153, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 153, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5845": { "addr": "172.21.15.67:6836/3080866617", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 3080866617, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 3080866617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5845, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 40 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 }, "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 36 }, "gid_6318": { "addr": "172.21.15.23:6834/904774922", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 904774922, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 904774922, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 151, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:reconnect", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:19:08.953676+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5845, "mds_1": 5707, "mds_2": 6313, "mds_3": 5995, "mds_4": 6318 } } } ], "standbys": [] } 2023-06-22T08:19:11.496 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:11.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.704+0000 7f6737bff700 1 Processor -- start 2023-06-22T08:19:11.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.704+0000 7f6737bff700 1 -- start start 2023-06-22T08:19:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.705+0000 7f6737bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6730152ce0 0x7f67300cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.705+0000 7f6737bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f67300c9fd0 0x7f673015c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.705+0000 7f6737bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673015c370 con 0x7f6730158880 2023-06-22T08:19:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.705+0000 7f6737bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67301530b0 con 0x7f67300c9fd0 2023-06-22T08:19:11.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.705+0000 7f6737bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67300cf6f0 con 0x7f6730152ce0 2023-06-22T08:19:11.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.705+0000 7f673519a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f67300c9fd0 0x7f673015c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:11.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.705+0000 7f673599b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6730152ce0 0x7f67300cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:11.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.705+0000 7f673619c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6730158880 0x7f6730158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:48608/0 (socket says 172.21.15.23:48608) 2023-06-22T08:19:11.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.705+0000 7f673619c700 1 -- 172.21.15.23:0/2544646292 learned_addr learned my addr 172.21.15.23:0/2544646292 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:19:11.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.706+0000 7f673599b700 1 -- 172.21.15.23:0/2544646292 >> v2:172.21.15.23:3300/0 conn(0x7f67300c9fd0 msgr2=0x7f673015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:19:11.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.706+0000 7f673599b700 1 --2- 172.21.15.23:0/2544646292 >> v2:172.21.15.23:3300/0 conn(0x7f67300c9fd0 0x7f673015c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:11.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.706+0000 7f6734999700 1 -- 172.21.15.23:0/2544646292 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359988918 0 0) 0x7f673015c370 con 0x7f6730158880 2023-06-22T08:19:11.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.706+0000 7f673599b700 1 -- 172.21.15.23:0/2544646292 >> v1:172.21.15.67:6789/0 conn(0x7f6730158880 legacy=0x7f6730158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:11.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.706+0000 7f673599b700 1 -- 172.21.15.23:0/2544646292 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673015bda0 con 0x7f6730152ce0 2023-06-22T08:19:11.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.706+0000 7f673519a700 1 --2- 172.21.15.23:0/2544646292 >> v2:172.21.15.23:3300/0 conn(0x7f67300c9fd0 0x7f673015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:19:11.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.706+0000 7f673599b700 1 --2- 172.21.15.23:0/2544646292 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6730152ce0 0x7f67300cf1b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f672c009840 tx=0x7f672c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a417acb98ec40c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:11.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.706+0000 7f6734999700 1 -- 172.21.15.23:0/2544646292 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c011830 con 0x7f6730152ce0 2023-06-22T08:19:11.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.706+0000 7f6734999700 1 -- 172.21.15.23:0/2544646292 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c011990 con 0x7f6730152ce0 2023-06-22T08:19:11.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.706+0000 7f6734999700 1 -- 172.21.15.23:0/2544646292 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c011d20 con 0x7f6730152ce0 2023-06-22T08:19:11.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.707+0000 7f6737bff700 1 -- 172.21.15.23:0/2544646292 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6730152ce0 msgr2=0x7f67300cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:11.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.707+0000 7f6737bff700 1 --2- 172.21.15.23:0/2544646292 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6730152ce0 0x7f67300cf1b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f672c009840 tx=0x7f672c009b50 comp rx=0 tx=0).stop 2023-06-22T08:19:11.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.707+0000 7f6737bff700 1 -- 172.21.15.23:0/2544646292 shutdown_connections 2023-06-22T08:19:11.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.707+0000 7f6737bff700 1 --2- 172.21.15.23:0/2544646292 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f6730152ce0 0x7f67300cf1b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:11.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.707+0000 7f6737bff700 1 --2- 172.21.15.23:0/2544646292 >> v2:172.21.15.23:3300/0 conn(0x7f67300c9fd0 0x7f673015c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:11.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.707+0000 7f6737bff700 1 -- 172.21.15.23:0/2544646292 >> 172.21.15.23:0/2544646292 conn(0x7f673006ae30 msgr2=0x7f673006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:11.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.707+0000 7f6737bff700 1 -- 172.21.15.23:0/2544646292 shutdown_connections 2023-06-22T08:19:11.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.707+0000 7f6737bff700 1 -- 172.21.15.23:0/2544646292 wait complete. 2023-06-22T08:19:11.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.710+0000 7f6737bff700 1 Processor -- start 2023-06-22T08:19:11.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.710+0000 7f6737bff700 1 -- start start 2023-06-22T08:19:11.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f6737bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f67300c9fd0 0x7f67300cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:11.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f6737bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6730152ce0 0x7f67301553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:11.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f6737bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730158880 0x7f6730155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:11.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f6737bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67301549e0 con 0x7f6730152ce0 2023-06-22T08:19:11.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f6737bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67300c8760 con 0x7f6730158880 2023-06-22T08:19:11.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f6737bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67300c8e60 con 0x7f67300c9fd0 2023-06-22T08:19:11.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f673519a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6730152ce0 0x7f67301553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:11.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f673519a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6730152ce0 0x7f67301553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57698/0 (socket says 172.21.15.23:57698) 2023-06-22T08:19:11.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f673519a700 1 -- 172.21.15.23:0/3877119961 learned_addr learned my addr 172.21.15.23:0/3877119961 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:11.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f673599b700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f67300c9fd0 0x7f67300cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:11.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.711+0000 7f673619c700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730158880 0x7f6730155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:11.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f673519a700 1 -- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f67300c9fd0 msgr2=0x7f67300cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:11.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f673519a700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f67300c9fd0 0x7f67300cead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:11.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f673519a700 1 -- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730158880 msgr2=0x7f6730155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:11.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f673519a700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730158880 0x7f6730155900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:11.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f673519a700 1 -- 172.21.15.23:0/3877119961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67200094a0 con 0x7f6730152ce0 2023-06-22T08:19:11.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f673619c700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730158880 0x7f6730155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:19:11.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f673519a700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6730152ce0 0x7f67301553c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f672000ebe0 tx=0x7f672000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:11.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f67267fc700 1 -- 172.21.15.23:0/3877119961 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672000c850 con 0x7f6730152ce0 2023-06-22T08:19:11.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f67267fc700 1 -- 172.21.15.23:0/3877119961 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672000c9b0 con 0x7f6730152ce0 2023-06-22T08:19:11.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f67267fc700 1 -- 172.21.15.23:0/3877119961 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672000cd40 con 0x7f6730152ce0 2023-06-22T08:19:11.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f6737bff700 1 -- 172.21.15.23:0/3877119961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c0094a0 con 0x7f6730152ce0 2023-06-22T08:19:11.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.712+0000 7f6737bff700 1 -- 172.21.15.23:0/3877119961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6730153f80 con 0x7f6730152ce0 2023-06-22T08:19:11.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.714+0000 7f6737bff700 1 -- 172.21.15.23:0/3877119961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67301541a0 con 0x7f6730152ce0 2023-06-22T08:19:11.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.715+0000 7f67267fc700 1 -- 172.21.15.23:0/3877119961 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6720013470 con 0x7f6730152ce0 2023-06-22T08:19:11.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.716+0000 7f67267fc700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f671406ace0 0x7f671406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:11.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.716+0000 7f67267fc700 1 -- 172.21.15.23:0/3877119961 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f6720089d10 con 0x7f6730152ce0 2023-06-22T08:19:11.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.716+0000 7f673599b700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f671406ace0 0x7f671406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:11.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.717+0000 7f673599b700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f671406ace0 0x7f671406d190 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f672c0090b0 tx=0x7f672c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:11.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.720+0000 7f67267fc700 1 -- 172.21.15.23:0/3877119961 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f672008f020 con 0x7f6730152ce0 2023-06-22T08:19:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.971+0000 7f6737bff700 1 -- 172.21.15.23:0/3877119961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f673004ea20 con 0x7f6730152ce0 2023-06-22T08:19:11.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.972+0000 7f67267fc700 1 -- 172.21.15.23:0/3877119961 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5513 (secure 0 0 0) 0x7f6720017070 con 0x7f6730152ce0 2023-06-22T08:19:11.975 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:11.975 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:19:08.953676+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:11.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.974+0000 7f6737bff700 1 -- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f671406ace0 msgr2=0x7f671406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:11.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.974+0000 7f6737bff700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f671406ace0 0x7f671406d190 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f672c0090b0 tx=0x7f672c008030 comp rx=0 tx=0).stop 2023-06-22T08:19:11.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.974+0000 7f6737bff700 1 -- 172.21.15.23:0/3877119961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6730152ce0 msgr2=0x7f67301553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:11.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.974+0000 7f6737bff700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6730152ce0 0x7f67301553c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f672000ebe0 tx=0x7f672000eef0 comp rx=0 tx=0).stop 2023-06-22T08:19:11.978 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.975+0000 7f6737bff700 1 -- 172.21.15.23:0/3877119961 shutdown_connections 2023-06-22T08:19:11.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.975+0000 7f6737bff700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f671406ace0 0x7f671406d190 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:11.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.975+0000 7f6737bff700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6730158880 0x7f6730155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:11.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.975+0000 7f6737bff700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f67300c9fd0 0x7f67300cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:11.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.975+0000 7f6737bff700 1 --2- 172.21.15.23:0/3877119961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6730152ce0 0x7f67301553c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:11.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.975+0000 7f6737bff700 1 -- 172.21.15.23:0/3877119961 >> 172.21.15.23:0/3877119961 conn(0x7f673006ae30 msgr2=0x7f67300b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:11.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.975+0000 7f6737bff700 1 -- 172.21.15.23:0/3877119961 shutdown_connections 2023-06-22T08:19:11.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:11.975+0000 7f6737bff700 1 -- 172.21.15.23:0/3877119961 wait complete. 2023-06-22T08:19:11.980 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 153 2023-06-22T08:19:11.996 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:19:13.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:14.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.203+0000 7f9e4edff700 1 Processor -- start 2023-06-22T08:19:14.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.203+0000 7f9e4edff700 1 -- start start 2023-06-22T08:19:14.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.204+0000 7f9e4edff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 0x7f9e480c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:14.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.204+0000 7f9e4edff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9e480c69e0 0x7f9e480c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:14.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.204+0000 7f9e4edff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e480c51a0 con 0x7f9e48071070 2023-06-22T08:19:14.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.204+0000 7f9e4edff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e480c44e0 con 0x7f9e480c69e0 2023-06-22T08:19:14.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.204+0000 7f9e4edff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e480c5440 con 0x7f9e4814f850 2023-06-22T08:19:14.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.205+0000 7f9e47fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9e480c69e0 0x7f9e480c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:14.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.205+0000 7f9e4cb9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 0x7f9e480c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:14.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.205+0000 7f9e4d39c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9e48071070 0x7f9e48155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58076/0 (socket says 172.21.15.23:58076) 2023-06-22T08:19:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.205+0000 7f9e4d39c700 1 -- 172.21.15.23:0/2668953314 learned_addr learned my addr 172.21.15.23:0/2668953314 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:19:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.205+0000 7f9e477fe700 1 -- 172.21.15.23:0/2668953314 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860389506 0 0) 0x7f9e480c51a0 con 0x7f9e48071070 2023-06-22T08:19:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.205+0000 7f9e47fff700 1 -- 172.21.15.23:0/2668953314 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 msgr2=0x7f9e480c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:19:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.205+0000 7f9e47fff700 1 --2- 172.21.15.23:0/2668953314 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 0x7f9e480c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.205+0000 7f9e47fff700 1 -- 172.21.15.23:0/2668953314 >> v1:172.21.15.23:6789/0 conn(0x7f9e48071070 legacy=0x7f9e48155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.205+0000 7f9e47fff700 1 -- 172.21.15.23:0/2668953314 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e480c5720 con 0x7f9e480c69e0 2023-06-22T08:19:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.205+0000 7f9e4cb9b700 1 --2- 172.21.15.23:0/2668953314 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 0x7f9e480c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:19:14.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e47fff700 1 --2- 172.21.15.23:0/2668953314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 0x7f9e480c35c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009840 tx=0x7f9e34009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa01316a60a59add server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:14.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e477fe700 1 -- 172.21.15.23:0/2668953314 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e34011830 con 0x7f9e480c69e0 2023-06-22T08:19:14.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e477fe700 1 -- 172.21.15.23:0/2668953314 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e34011990 con 0x7f9e480c69e0 2023-06-22T08:19:14.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e477fe700 1 -- 172.21.15.23:0/2668953314 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e34011d20 con 0x7f9e480c69e0 2023-06-22T08:19:14.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e4edff700 1 -- 172.21.15.23:0/2668953314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 msgr2=0x7f9e480c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e4edff700 1 --2- 172.21.15.23:0/2668953314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 0x7f9e480c35c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009840 tx=0x7f9e34009b50 comp rx=0 tx=0).stop 2023-06-22T08:19:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e4edff700 1 -- 172.21.15.23:0/2668953314 shutdown_connections 2023-06-22T08:19:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e4edff700 1 --2- 172.21.15.23:0/2668953314 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 0x7f9e480c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e4edff700 1 --2- 172.21.15.23:0/2668953314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 0x7f9e480c35c0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e4edff700 1 -- 172.21.15.23:0/2668953314 >> 172.21.15.23:0/2668953314 conn(0x7f9e480ca3f0 msgr2=0x7f9e480ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.206+0000 7f9e4edff700 1 -- 172.21.15.23:0/2668953314 shutdown_connections 2023-06-22T08:19:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.207+0000 7f9e4edff700 1 -- 172.21.15.23:0/2668953314 wait complete. 2023-06-22T08:19:14.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.208+0000 7f9e4edff700 1 Processor -- start 2023-06-22T08:19:14.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.208+0000 7f9e4edff700 1 -- start start 2023-06-22T08:19:14.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.209+0000 7f9e4edff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e48071070 0x7f9e48149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:14.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.209+0000 7f9e4edff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 0x7f9e48148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:14.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.209+0000 7f9e4edff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 0x7f9e48143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:14.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.209+0000 7f9e4edff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e48144f40 con 0x7f9e480c69e0 2023-06-22T08:19:14.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.209+0000 7f9e4edff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e48146130 con 0x7f9e48071070 2023-06-22T08:19:14.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.209+0000 7f9e4edff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e48145560 con 0x7f9e4814f850 2023-06-22T08:19:14.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.209+0000 7f9e47fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 0x7f9e48148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:14.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.209+0000 7f9e47fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 0x7f9e48148670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57720/0 (socket says 172.21.15.23:57720) 2023-06-22T08:19:14.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e47fff700 1 -- 172.21.15.23:0/938049093 learned_addr learned my addr 172.21.15.23:0/938049093 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:14.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e4d39c700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 0x7f9e48143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:14.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e4cb9b700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e48071070 0x7f9e48149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:14.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e47fff700 1 -- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 msgr2=0x7f9e48143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:14.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e47fff700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 0x7f9e48143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:14.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e47fff700 1 -- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e48071070 msgr2=0x7f9e48149f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:14.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e47fff700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e48071070 0x7f9e48149f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:14.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e47fff700 1 -- 172.21.15.23:0/938049093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c0094a0 con 0x7f9e480c69e0 2023-06-22T08:19:14.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e47fff700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 0x7f9e48148670 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009fd0 tx=0x7f9e34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:14.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e457fa700 1 -- 172.21.15.23:0/938049093 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e340036a0 con 0x7f9e480c69e0 2023-06-22T08:19:14.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.210+0000 7f9e457fa700 1 -- 172.21.15.23:0/938049093 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3401f070 con 0x7f9e480c69e0 2023-06-22T08:19:14.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.211+0000 7f9e457fa700 1 -- 172.21.15.23:0/938049093 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e34003a60 con 0x7f9e480c69e0 2023-06-22T08:19:14.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.211+0000 7f9e4edff700 1 -- 172.21.15.23:0/938049093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e340094a0 con 0x7f9e480c69e0 2023-06-22T08:19:14.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.211+0000 7f9e4edff700 1 -- 172.21.15.23:0/938049093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e48145ae0 con 0x7f9e480c69e0 2023-06-22T08:19:14.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.212+0000 7f9e4edff700 1 -- 172.21.15.23:0/938049093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e480c2950 con 0x7f9e480c69e0 2023-06-22T08:19:14.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.213+0000 7f9e457fa700 1 -- 172.21.15.23:0/938049093 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9e34003bc0 con 0x7f9e480c69e0 2023-06-22T08:19:14.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.214+0000 7f9e457fa700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e2c06ace0 0x7f9e2c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:14.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.214+0000 7f9e457fa700 1 -- 172.21.15.23:0/938049093 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f9e34021070 con 0x7f9e480c69e0 2023-06-22T08:19:14.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.214+0000 7f9e4cb9b700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e2c06ace0 0x7f9e2c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:14.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.215+0000 7f9e4cb9b700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e2c06ace0 0x7f9e2c06d190 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009cd0 tx=0x7f9e3c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:14.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.218+0000 7f9e457fa700 1 -- 172.21.15.23:0/938049093 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9e34055d60 con 0x7f9e480c69e0 2023-06-22T08:19:14.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.473+0000 7f9e4edff700 1 -- 172.21.15.23:0/938049093 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e481462a0 con 0x7f9e480c69e0 2023-06-22T08:19:14.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.474+0000 7f9e457fa700 1 -- 172.21.15.23:0/938049093 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5510 (secure 0 0 0) 0x7f9e34059380 con 0x7f9e480c69e0 2023-06-22T08:19:14.477 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:14.477 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:19:13.986448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:14.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 -- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e2c06ace0 msgr2=0x7f9e2c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:14.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e2c06ace0 0x7f9e2c06d190 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009cd0 tx=0x7f9e3c009210 comp rx=0 tx=0).stop 2023-06-22T08:19:14.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 -- 172.21.15.23:0/938049093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 msgr2=0x7f9e48148670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:14.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 0x7f9e48148670 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9e34009fd0 tx=0x7f9e34004750 comp rx=0 tx=0).stop 2023-06-22T08:19:14.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 -- 172.21.15.23:0/938049093 shutdown_connections 2023-06-22T08:19:14.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e4814f850 0x7f9e48143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:14.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e2c06ace0 0x7f9e2c06d190 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:14.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e48071070 0x7f9e48149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:14.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 --2- 172.21.15.23:0/938049093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e480c69e0 0x7f9e48148670 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:14.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 -- 172.21.15.23:0/938049093 >> 172.21.15.23:0/938049093 conn(0x7f9e480ca3f0 msgr2=0x7f9e480c6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:14.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 -- 172.21.15.23:0/938049093 shutdown_connections 2023-06-22T08:19:14.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:14.476+0000 7f9e4edff700 1 -- 172.21.15.23:0/938049093 wait complete. 2023-06-22T08:19:14.481 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 154 2023-06-22T08:19:14.497 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:19:14.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:19:14.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:19:16.498 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:16.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.706+0000 7f0aed1ff700 1 Processor -- start 2023-06-22T08:19:16.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.706+0000 7f0aed1ff700 1 -- start start 2023-06-22T08:19:16.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.707+0000 7f0aed1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80c5740 0x7f0ae8069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:16.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.707+0000 7f0aed1ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae8158f60 con 0x7f0ae8152e10 2023-06-22T08:19:16.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.707+0000 7f0aed1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae80cc450 con 0x7f0ae80cb6b0 2023-06-22T08:19:16.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.707+0000 7f0aed1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae806bd20 con 0x7f0ae80c5740 2023-06-22T08:19:16.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.708+0000 7f0ae659c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f0ae8152e10 0x7f0ae8161100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:46672/0 (socket says 172.21.15.23:46672) 2023-06-22T08:19:16.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.708+0000 7f0ae659c700 1 -- 172.21.15.23:0/331825277 learned_addr learned my addr 172.21.15.23:0/331825277 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:19:16.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.708+0000 7f0ae759e700 1 --2- 172.21.15.23:0/331825277 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80c5740 0x7f0ae8069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:16.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.708+0000 7f0ae5d9b700 1 -- 172.21.15.23:0/331825277 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516846964 0 0) 0x7f0ae8158f60 con 0x7f0ae8152e10 2023-06-22T08:19:16.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.708+0000 7f0ae5d9b700 1 -- 172.21.15.23:0/331825277 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad00035b0 con 0x7f0ae8152e10 2023-06-22T08:19:16.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.708+0000 7f0ae5d9b700 1 -- 172.21.15.23:0/331825277 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984573320 0 0) 0x7f0ae80cc450 con 0x7f0ae80cb6b0 2023-06-22T08:19:16.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.708+0000 7f0ae5d9b700 1 -- 172.21.15.23:0/331825277 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae8158f60 con 0x7f0ae80cb6b0 2023-06-22T08:19:16.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.708+0000 7f0ae759e700 1 -- 172.21.15.23:0/331825277 >> v1:172.21.15.67:6789/0 conn(0x7f0ae80cb6b0 legacy=0x7f0ae8152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:16.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.708+0000 7f0ae759e700 1 -- 172.21.15.23:0/331825277 >> v1:172.21.15.23:6789/0 conn(0x7f0ae8152e10 legacy=0x7f0ae8161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:16.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.708+0000 7f0ae759e700 1 -- 172.21.15.23:0/331825277 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae8158980 con 0x7f0ae80c5740 2023-06-22T08:19:16.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.709+0000 7f0ae759e700 1 --2- 172.21.15.23:0/331825277 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80c5740 0x7f0ae8069eb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009dc0 tx=0x7f0ae000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18900cdcd03929bb server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.709+0000 7f0ae5d9b700 1 -- 172.21.15.23:0/331825277 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae000e070 con 0x7f0ae80c5740 2023-06-22T08:19:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.709+0000 7f0ae5d9b700 1 -- 172.21.15.23:0/331825277 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae000bb20 con 0x7f0ae80c5740 2023-06-22T08:19:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.709+0000 7f0ae5d9b700 1 -- 172.21.15.23:0/331825277 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0012430 con 0x7f0ae80c5740 2023-06-22T08:19:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.709+0000 7f0aed1ff700 1 -- 172.21.15.23:0/331825277 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80c5740 msgr2=0x7f0ae8069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.709+0000 7f0aed1ff700 1 --2- 172.21.15.23:0/331825277 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80c5740 0x7f0ae8069eb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009dc0 tx=0x7f0ae000b330 comp rx=0 tx=0).stop 2023-06-22T08:19:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.709+0000 7f0aed1ff700 1 -- 172.21.15.23:0/331825277 shutdown_connections 2023-06-22T08:19:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.709+0000 7f0aed1ff700 1 --2- 172.21.15.23:0/331825277 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80c5740 0x7f0ae8069eb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:16.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.709+0000 7f0aed1ff700 1 -- 172.21.15.23:0/331825277 >> 172.21.15.23:0/331825277 conn(0x7f0ae8070d80 msgr2=0x7f0ae8071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:16.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.710+0000 7f0aed1ff700 1 -- 172.21.15.23:0/331825277 shutdown_connections 2023-06-22T08:19:16.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.710+0000 7f0aed1ff700 1 -- 172.21.15.23:0/331825277 wait complete. 2023-06-22T08:19:16.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.711+0000 7f0aed1ff700 1 Processor -- start 2023-06-22T08:19:16.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.711+0000 7f0aed1ff700 1 -- start start 2023-06-22T08:19:16.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.712+0000 7f0aed1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80cb6b0 0x7f0ae815a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:16.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.712+0000 7f0aed1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ae8152e10 0x7f0ae81f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:16.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.712+0000 7f0aed1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae81599d0 0x7f0ae8142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:16.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.712+0000 7f0aed1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae8143b50 con 0x7f0ae81599d0 2023-06-22T08:19:16.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.712+0000 7f0aed1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae81f2160 con 0x7f0ae8152e10 2023-06-22T08:19:16.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.712+0000 7f0aed1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae8143da0 con 0x7f0ae80cb6b0 2023-06-22T08:19:16.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae759e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae81599d0 0x7f0ae8142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:16.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.712+0000 7f0ae6d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80cb6b0 0x7f0ae815a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:16.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ae8152e10 0x7f0ae81f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:16.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae6d9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80cb6b0 0x7f0ae815a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:45438/0 (socket says 172.21.15.23:45438) 2023-06-22T08:19:16.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae6d9d700 1 -- 172.21.15.23:0/1158767684 learned_addr learned my addr 172.21.15.23:0/1158767684 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:16.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae759e700 1 -- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80cb6b0 msgr2=0x7f0ae815a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:16.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae759e700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80cb6b0 0x7f0ae815a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:16.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae759e700 1 -- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ae8152e10 msgr2=0x7f0ae81f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:16.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae759e700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ae8152e10 0x7f0ae81f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:16.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae759e700 1 -- 172.21.15.23:0/1158767684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae0009a70 con 0x7f0ae81599d0 2023-06-22T08:19:16.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae6d9d700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80cb6b0 0x7f0ae815a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:19:16.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.713+0000 7f0ae759e700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae81599d0 0x7f0ae8142e00 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0ae815bd10 tx=0x7f0ae0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:16.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.714+0000 7f0acf7fe700 1 -- 172.21.15.23:0/1158767684 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0012430 con 0x7f0ae81599d0 2023-06-22T08:19:16.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.714+0000 7f0acf7fe700 1 -- 172.21.15.23:0/1158767684 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae0020070 con 0x7f0ae81599d0 2023-06-22T08:19:16.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.714+0000 7f0aed1ff700 1 -- 172.21.15.23:0/1158767684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae815b320 con 0x7f0ae81599d0 2023-06-22T08:19:16.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.714+0000 7f0acf7fe700 1 -- 172.21.15.23:0/1158767684 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae001ca90 con 0x7f0ae81599d0 2023-06-22T08:19:16.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.714+0000 7f0aed1ff700 1 -- 172.21.15.23:0/1158767684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae81565d0 con 0x7f0ae81599d0 2023-06-22T08:19:16.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.715+0000 7f0aed1ff700 1 -- 172.21.15.23:0/1158767684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae81f22a0 con 0x7f0ae81599d0 2023-06-22T08:19:16.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.716+0000 7f0acf7fe700 1 -- 172.21.15.23:0/1158767684 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0ae001cbf0 con 0x7f0ae81599d0 2023-06-22T08:19:16.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.717+0000 7f0acf7fe700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ad406ace0 0x7f0ad406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:16.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.717+0000 7f0acf7fe700 1 -- 172.21.15.23:0/1158767684 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f0ae0022070 con 0x7f0ae81599d0 2023-06-22T08:19:16.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.717+0000 7f0ae6d9d700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ad406ace0 0x7f0ad406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:16.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.718+0000 7f0ae6d9d700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ad406ace0 0x7f0ad406d190 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80011e0 tx=0x7f0ad8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:16.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.722+0000 7f0acf7fe700 1 -- 172.21.15.23:0/1158767684 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0ae81f22a0 con 0x7f0ae81599d0 2023-06-22T08:19:16.976 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.973+0000 7f0aed1ff700 1 -- 172.21.15.23:0/1158767684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ae81f22a0 con 0x7f0ae81599d0 2023-06-22T08:19:16.977 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.974+0000 7f0acf7fe700 1 -- 172.21.15.23:0/1158767684 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5510 (secure 0 0 0) 0x7f0ae81f22a0 con 0x7f0ae81599d0 2023-06-22T08:19:16.978 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:19:13.986448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:16.979 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.976+0000 7f0aed1ff700 1 -- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ad406ace0 msgr2=0x7f0ad406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:16.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.976+0000 7f0aed1ff700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ad406ace0 0x7f0ad406d190 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80011e0 tx=0x7f0ad8009040 comp rx=0 tx=0).stop 2023-06-22T08:19:16.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.976+0000 7f0aed1ff700 1 -- 172.21.15.23:0/1158767684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae81599d0 msgr2=0x7f0ae8142e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:16.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.976+0000 7f0aed1ff700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae81599d0 0x7f0ae8142e00 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0ae815bd10 tx=0x7f0ae0004ad0 comp rx=0 tx=0).stop 2023-06-22T08:19:16.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.976+0000 7f0aed1ff700 1 -- 172.21.15.23:0/1158767684 shutdown_connections 2023-06-22T08:19:16.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.976+0000 7f0aed1ff700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ae81599d0 0x7f0ae8142e00 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:16.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.976+0000 7f0aed1ff700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0ae80cb6b0 0x7f0ae815a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:16.980 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.976+0000 7f0aed1ff700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ad406ace0 0x7f0ad406d190 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:16.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.976+0000 7f0aed1ff700 1 --2- 172.21.15.23:0/1158767684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ae8152e10 0x7f0ae81f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:16.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.976+0000 7f0aed1ff700 1 -- 172.21.15.23:0/1158767684 >> 172.21.15.23:0/1158767684 conn(0x7f0ae8070d80 msgr2=0x7f0ae806a590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:16.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.977+0000 7f0aed1ff700 1 -- 172.21.15.23:0/1158767684 shutdown_connections 2023-06-22T08:19:16.981 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:16.977+0000 7f0aed1ff700 1 -- 172.21.15.23:0/1158767684 wait complete. 2023-06-22T08:19:16.981 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 154 2023-06-22T08:19:16.998 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:19:18.998 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:19.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.265+0000 7f3567fff700 1 Processor -- start 2023-06-22T08:19:19.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.265+0000 7f3567fff700 1 -- start start 2023-06-22T08:19:19.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.266+0000 7f3567fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3568157020 0x7f3568161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:19.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.266+0000 7f3567fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356806c300 con 0x7f35680c3f00 2023-06-22T08:19:19.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.266+0000 7f3567fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356815a9d0 con 0x7f35680c3b30 2023-06-22T08:19:19.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.266+0000 7f3567fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356806bd40 con 0x7f3568157020 2023-06-22T08:19:19.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.267+0000 7f35667fc700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f35680c3f00 0x7f3568150420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:46686/0 (socket says 172.21.15.23:46686) 2023-06-22T08:19:19.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.267+0000 7f35667fc700 1 -- 172.21.15.23:0/1515945658 learned_addr learned my addr 172.21.15.23:0/1515945658 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:19:19.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.267+0000 7f35677fe700 1 --2- 172.21.15.23:0/1515945658 >> v2:172.21.15.23:3300/0 conn(0x7f3568157020 0x7f3568161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:19.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.267+0000 7f3565ffb700 1 -- 172.21.15.23:0/1515945658 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873426123 0 0) 0x7f356806c300 con 0x7f35680c3f00 2023-06-22T08:19:19.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.267+0000 7f3565ffb700 1 -- 172.21.15.23:0/1515945658 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35500035b0 con 0x7f35680c3f00 2023-06-22T08:19:19.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.267+0000 7f35677fe700 1 -- 172.21.15.23:0/1515945658 >> v1:172.21.15.67:6789/0 conn(0x7f35680c3b30 legacy=0x7f3568156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:19.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.267+0000 7f35677fe700 1 -- 172.21.15.23:0/1515945658 >> v1:172.21.15.23:6789/0 conn(0x7f35680c3f00 legacy=0x7f3568150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:19.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.267+0000 7f35677fe700 1 -- 172.21.15.23:0/1515945658 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356806c020 con 0x7f3568157020 2023-06-22T08:19:19.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.267+0000 7f35677fe700 1 --2- 172.21.15.23:0/1515945658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3568157020 0x7f3568161f10 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3560009d90 tx=0x7f356000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4128010c512441 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:19.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.268+0000 7f3565ffb700 1 -- 172.21.15.23:0/1515945658 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356000e070 con 0x7f3568157020 2023-06-22T08:19:19.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.268+0000 7f3565ffb700 1 -- 172.21.15.23:0/1515945658 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356000bb20 con 0x7f3568157020 2023-06-22T08:19:19.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.268+0000 7f3565ffb700 1 -- 172.21.15.23:0/1515945658 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560012430 con 0x7f3568157020 2023-06-22T08:19:19.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.268+0000 7f3567fff700 1 -- 172.21.15.23:0/1515945658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3568157020 msgr2=0x7f3568161f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:19.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.268+0000 7f3567fff700 1 --2- 172.21.15.23:0/1515945658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3568157020 0x7f3568161f10 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3560009d90 tx=0x7f356000b330 comp rx=0 tx=0).stop 2023-06-22T08:19:19.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.268+0000 7f3567fff700 1 -- 172.21.15.23:0/1515945658 shutdown_connections 2023-06-22T08:19:19.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.268+0000 7f3567fff700 1 --2- 172.21.15.23:0/1515945658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3568157020 0x7f3568161f10 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:19.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.268+0000 7f3567fff700 1 -- 172.21.15.23:0/1515945658 >> 172.21.15.23:0/1515945658 conn(0x7f356806e3b0 msgr2=0x7f3568069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:19.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.268+0000 7f3567fff700 1 -- 172.21.15.23:0/1515945658 shutdown_connections 2023-06-22T08:19:19.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.268+0000 7f3567fff700 1 -- 172.21.15.23:0/1515945658 wait complete. 2023-06-22T08:19:19.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.270+0000 7f3567fff700 1 Processor -- start 2023-06-22T08:19:19.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.270+0000 7f3567fff700 1 -- start start 2023-06-22T08:19:19.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.271+0000 7f3567fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35680c3f00 0x7f35681f1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:19.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.271+0000 7f3567fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3568157020 0x7f3568151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:19.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.271+0000 7f3567fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3568151780 0x7f35681577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:19.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.271+0000 7f3567fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35681f11e0 con 0x7f3568151780 2023-06-22T08:19:19.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.271+0000 7f3567fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3568152d30 con 0x7f3568157020 2023-06-22T08:19:19.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.271+0000 7f3567fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35681f0eb0 con 0x7f35680c3f00 2023-06-22T08:19:19.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f35667fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3568157020 0x7f3568151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:19.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f3566ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35680c3f00 0x7f35681f1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:19.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f3566ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35680c3f00 0x7f35681f1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:45452/0 (socket says 172.21.15.23:45452) 2023-06-22T08:19:19.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f3566ffd700 1 -- 172.21.15.23:0/1642346411 learned_addr learned my addr 172.21.15.23:0/1642346411 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:19.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f35667fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3568157020 0x7f3568151240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:46892/0 (socket says 172.21.15.23:46892) 2023-06-22T08:19:19.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f35677fe700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3568151780 0x7f35681577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:19.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f3566ffd700 1 -- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3568157020 msgr2=0x7f3568151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:19.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f3566ffd700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3568157020 0x7f3568151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:19.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f3566ffd700 1 -- 172.21.15.23:0/1642346411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3568151780 msgr2=0x7f35681577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:19.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f3566ffd700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3568151780 0x7f35681577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:19.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f3566ffd700 1 -- 172.21.15.23:0/1642346411 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3560009a70 con 0x7f35680c3f00 2023-06-22T08:19:19.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f35667fc700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3568157020 0x7f3568151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:19:19.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f3566ffd700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35680c3f00 0x7f35681f1670 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f355800c510 tx=0x7f355800c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:19.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f354f7fe700 1 -- 172.21.15.23:0/1642346411 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35580052d0 con 0x7f35680c3f00 2023-06-22T08:19:19.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f354f7fe700 1 -- 172.21.15.23:0/1642346411 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3558005430 con 0x7f35680c3f00 2023-06-22T08:19:19.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.272+0000 7f3567fff700 1 -- 172.21.15.23:0/1642346411 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3568151bf0 con 0x7f35680c3f00 2023-06-22T08:19:19.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.273+0000 7f354f7fe700 1 -- 172.21.15.23:0/1642346411 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35580038c0 con 0x7f35680c3f00 2023-06-22T08:19:19.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.273+0000 7f3567fff700 1 -- 172.21.15.23:0/1642346411 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356814d650 con 0x7f35680c3f00 2023-06-22T08:19:19.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.274+0000 7f3567fff700 1 -- 172.21.15.23:0/1642346411 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35681f14c0 con 0x7f35680c3f00 2023-06-22T08:19:19.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.275+0000 7f354f7fe700 1 -- 172.21.15.23:0/1642346411 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3558003a20 con 0x7f35680c3f00 2023-06-22T08:19:19.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.275+0000 7f354f7fe700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f355406ad10 0x7f355406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:19.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.276+0000 7f354f7fe700 1 -- 172.21.15.23:0/1642346411 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f3558088e60 con 0x7f35680c3f00 2023-06-22T08:19:19.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.276+0000 7f35667fc700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f355406ad10 0x7f355406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:19.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.276+0000 7f35667fc700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f355406ad10 0x7f355406d1c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3568150c50 tx=0x7f355c007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:19.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.280+0000 7f354f7fe700 1 -- 172.21.15.23:0/1642346411 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f35580538b0 con 0x7f35680c3f00 2023-06-22T08:19:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.536+0000 7f3567fff700 1 -- 172.21.15.23:0/1642346411 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35681f0ff0 con 0x7f35680c3f00 2023-06-22T08:19:19.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.537+0000 7f354f7fe700 1 -- 172.21.15.23:0/1642346411 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5510 (secure 0 0 0) 0x7f3558056ed0 con 0x7f35680c3f00 2023-06-22T08:19:19.541 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:19.541 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:19:13.986448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:19.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 -- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f355406ad10 msgr2=0x7f355406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:19.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f355406ad10 0x7f355406d1c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3568150c50 tx=0x7f355c007dc0 comp rx=0 tx=0).stop 2023-06-22T08:19:19.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 -- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35680c3f00 msgr2=0x7f35681f1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:19.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35680c3f00 0x7f35681f1670 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f355800c510 tx=0x7f355800c820 comp rx=0 tx=0).stop 2023-06-22T08:19:19.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 -- 172.21.15.23:0/1642346411 shutdown_connections 2023-06-22T08:19:19.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3568151780 0x7f35681577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:19.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35680c3f00 0x7f35681f1670 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:19.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f355406ad10 0x7f355406d1c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:19.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 --2- 172.21.15.23:0/1642346411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3568157020 0x7f3568151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:19.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 -- 172.21.15.23:0/1642346411 >> 172.21.15.23:0/1642346411 conn(0x7f35680c3b30 msgr2=0x7f356806fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:19.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 -- 172.21.15.23:0/1642346411 shutdown_connections 2023-06-22T08:19:19.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:19.540+0000 7f3567fff700 1 -- 172.21.15.23:0/1642346411 wait complete. 2023-06-22T08:19:19.545 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 154 2023-06-22T08:19:19.561 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:19:21.563 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.772+0000 7f46f1bff700 1 Processor -- start 2023-06-22T08:19:21.775 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.772+0000 7f46f1bff700 1 -- start start 2023-06-22T08:19:21.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.773+0000 7f46f1bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46ec152d70 0x7f46ec0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:21.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.773+0000 7f46f1bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec159330 0x7f46ec0cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:21.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46f1bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec15bea0 con 0x7f46ec0cb800 2023-06-22T08:19:21.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46f1bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec0c95e0 con 0x7f46ec152d70 2023-06-22T08:19:21.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46f1bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec15c160 con 0x7f46ec159330 2023-06-22T08:19:21.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46e3fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec159330 0x7f46ec0cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:21.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46eb7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46ec152d70 0x7f46ec0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:21.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46e3fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec159330 0x7f46ec0cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:45460/0 (socket says 172.21.15.23:45460) 2023-06-22T08:19:21.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46eb7fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46ec152d70 0x7f46ec0c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:46900/0 (socket says 172.21.15.23:46900) 2023-06-22T08:19:21.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46ebfff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f46ec0cb800 0x7f46ec0c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:37740/0 (socket says 172.21.15.23:37740) 2023-06-22T08:19:21.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46e3fff700 1 -- 172.21.15.23:0/3828211054 learned_addr learned my addr 172.21.15.23:0/3828211054 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:21.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46eb7fe700 1 -- 172.21.15.23:0/3828211054 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec159330 msgr2=0x7f46ec0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:19:21.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46eb7fe700 1 --2- 172.21.15.23:0/3828211054 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec159330 0x7f46ec0cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:21.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46eb7fe700 1 -- 172.21.15.23:0/3828211054 >> v1:172.21.15.67:6789/0 conn(0x7f46ec0cb800 legacy=0x7f46ec0c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:21.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46eb7fe700 1 -- 172.21.15.23:0/3828211054 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec15c440 con 0x7f46ec152d70 2023-06-22T08:19:21.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.774+0000 7f46eaffd700 1 -- 172.21.15.23:0/3828211054 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676309030 0 0) 0x7f46ec15bea0 con 0x7f46ec0cb800 2023-06-22T08:19:21.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.775+0000 7f46eb7fe700 1 --2- 172.21.15.23:0/3828211054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46ec152d70 0x7f46ec0c97d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f46dc009840 tx=0x7f46dc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eda003ba122d07a5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:21.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.775+0000 7f46eaffd700 1 -- 172.21.15.23:0/3828211054 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc011830 con 0x7f46ec152d70 2023-06-22T08:19:21.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.775+0000 7f46eaffd700 1 -- 172.21.15.23:0/3828211054 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46dc011990 con 0x7f46ec152d70 2023-06-22T08:19:21.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.775+0000 7f46eaffd700 1 -- 172.21.15.23:0/3828211054 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc011d20 con 0x7f46ec152d70 2023-06-22T08:19:21.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.776+0000 7f46f1bff700 1 -- 172.21.15.23:0/3828211054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46ec152d70 msgr2=0x7f46ec0c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.776+0000 7f46f1bff700 1 --2- 172.21.15.23:0/3828211054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46ec152d70 0x7f46ec0c97d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f46dc009840 tx=0x7f46dc009b50 comp rx=0 tx=0).stop 2023-06-22T08:19:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.776+0000 7f46f1bff700 1 -- 172.21.15.23:0/3828211054 shutdown_connections 2023-06-22T08:19:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.776+0000 7f46f1bff700 1 --2- 172.21.15.23:0/3828211054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46ec152d70 0x7f46ec0c97d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.776+0000 7f46f1bff700 1 --2- 172.21.15.23:0/3828211054 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec159330 0x7f46ec0cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.776+0000 7f46f1bff700 1 -- 172.21.15.23:0/3828211054 >> 172.21.15.23:0/3828211054 conn(0x7f46ec06ae30 msgr2=0x7f46ec06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.776+0000 7f46f1bff700 1 -- 172.21.15.23:0/3828211054 shutdown_connections 2023-06-22T08:19:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.776+0000 7f46f1bff700 1 -- 172.21.15.23:0/3828211054 wait complete. 2023-06-22T08:19:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.778+0000 7f46f1bff700 1 Processor -- start 2023-06-22T08:19:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.778+0000 7f46f1bff700 1 -- start start 2023-06-22T08:19:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.779+0000 7f46f1bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec0cb800 0x7f46ec0ce660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.779+0000 7f46f1bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46ec152d70 0x7f46ec153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:21.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46f1bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46ec159330 0x7f46ec1539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:21.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46f1bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec0ce240 con 0x7f46ec152d70 2023-06-22T08:19:21.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46f1bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec156180 con 0x7f46ec159330 2023-06-22T08:19:21.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46f1bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec153ef0 con 0x7f46ec0cb800 2023-06-22T08:19:21.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46e3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46ec152d70 0x7f46ec153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:21.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46e3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46ec152d70 0x7f46ec153470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57870/0 (socket says 172.21.15.23:57870) 2023-06-22T08:19:21.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46e3fff700 1 -- 172.21.15.23:0/1334536062 learned_addr learned my addr 172.21.15.23:0/1334536062 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:21.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46eb7fe700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec0cb800 0x7f46ec0ce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:21.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46ebfff700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46ec159330 0x7f46ec1539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:21.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46e3fff700 1 -- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec0cb800 msgr2=0x7f46ec0ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:21.788 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46e3fff700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec0cb800 0x7f46ec0ce660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:21.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46e3fff700 1 -- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46ec159330 msgr2=0x7f46ec1539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:21.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46e3fff700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46ec159330 0x7f46ec1539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:21.789 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46e3fff700 1 -- 172.21.15.23:0/1334536062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc0094a0 con 0x7f46ec152d70 2023-06-22T08:19:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.780+0000 7f46e3fff700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46ec152d70 0x7f46ec153470 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f46d400e8b0 tx=0x7f46d400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.781+0000 7f46e8ff9700 1 -- 172.21.15.23:0/1334536062 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4009600 con 0x7f46ec152d70 2023-06-22T08:19:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.781+0000 7f46e8ff9700 1 -- 172.21.15.23:0/1334536062 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d4013680 con 0x7f46ec152d70 2023-06-22T08:19:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.781+0000 7f46e8ff9700 1 -- 172.21.15.23:0/1334536062 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4013a30 con 0x7f46ec152d70 2023-06-22T08:19:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.781+0000 7f46f1bff700 1 -- 172.21.15.23:0/1334536062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec0ceba0 con 0x7f46ec152d70 2023-06-22T08:19:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.781+0000 7f46f1bff700 1 -- 172.21.15.23:0/1334536062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec0cd8a0 con 0x7f46ec152d70 2023-06-22T08:19:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.782+0000 7f46f1bff700 1 -- 172.21.15.23:0/1334536062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ec0ce380 con 0x7f46ec152d70 2023-06-22T08:19:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.783+0000 7f46e8ff9700 1 -- 172.21.15.23:0/1334536062 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f46d4013b90 con 0x7f46ec152d70 2023-06-22T08:19:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.784+0000 7f46e8ff9700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f46cc06ace0 0x7f46cc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.784+0000 7f46e8ff9700 1 -- 172.21.15.23:0/1334536062 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7f46d4089a40 con 0x7f46ec152d70 2023-06-22T08:19:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.784+0000 7f46eb7fe700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f46cc06ace0 0x7f46cc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.785+0000 7f46eb7fe700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f46cc06ace0 0x7f46cc06d190 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0090b0 tx=0x7f46dc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:21.794 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:21.789+0000 7f46e8ff9700 1 -- 172.21.15.23:0/1334536062 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f46ec0ce380 con 0x7f46ec152d70 2023-06-22T08:19:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.044+0000 7f46f1bff700 1 -- 172.21.15.23:0/1334536062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46ec0ce380 con 0x7f46ec152d70 2023-06-22T08:19:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.045+0000 7f46e8ff9700 1 -- 172.21.15.23:0/1334536062 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5510 (secure 0 0 0) 0x7f46ec0ce380 con 0x7f46ec152d70 2023-06-22T08:19:22.048 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:22.048 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:19:19.962809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:active","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:22.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 -- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f46cc06ace0 msgr2=0x7f46cc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:22.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f46cc06ace0 0x7f46cc06d190 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0090b0 tx=0x7f46dc008030 comp rx=0 tx=0).stop 2023-06-22T08:19:22.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 -- 172.21.15.23:0/1334536062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46ec152d70 msgr2=0x7f46ec153470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46ec152d70 0x7f46ec153470 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f46d400e8b0 tx=0x7f46d400ebc0 comp rx=0 tx=0).stop 2023-06-22T08:19:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 -- 172.21.15.23:0/1334536062 shutdown_connections 2023-06-22T08:19:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f46cc06ace0 0x7f46cc06d190 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46ec159330 0x7f46ec1539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f46ec0cb800 0x7f46ec0ce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 --2- 172.21.15.23:0/1334536062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46ec152d70 0x7f46ec153470 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:22.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 -- 172.21.15.23:0/1334536062 >> 172.21.15.23:0/1334536062 conn(0x7f46ec06ae30 msgr2=0x7f46ec0b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:22.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 -- 172.21.15.23:0/1334536062 shutdown_connections 2023-06-22T08:19:22.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:22.047+0000 7f46f1bff700 1 -- 172.21.15.23:0/1334536062 wait complete. 2023-06-22T08:19:22.053 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 155 2023-06-22T08:19:22.069 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=4, replacing gid=5576 2023-06-22T08:19:22.070 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.d 2023-06-22T08:19:25.671 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:25.678 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:25.678 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 64 -eq 48 ']' 2023-06-22T08:19:25.678 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.64 2023-06-22T08:19:25.679 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.64 2023-06-22T08:19:25.679 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.64 2023-06-22T08:19:25.679 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.64 2023-06-22T08:19:25.679 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:26.623 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:26.623 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:26.623 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 65 -eq 48 ']' 2023-06-22T08:19:26.624 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.65 2023-06-22T08:19:26.624 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.65 2023-06-22T08:19:26.624 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.65 2023-06-22T08:19:26.624 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.65 2023-06-22T08:19:26.625 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:27.071 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-06-22T08:19:27.071 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-22T08:19:27.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-22T08:19:27.074 INFO:tasks.ceph.mds.d:Started 2023-06-22T08:19:27.074 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:27.155 INFO:tasks.ceph.mds.d.smithi067.stdout:starting mds.d at 2023-06-22T08:19:27.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.318+0000 7fe03f19d700 1 Processor -- start 2023-06-22T08:19:27.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.318+0000 7fe03f19d700 1 -- start start 2023-06-22T08:19:27.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.320+0000 7fe03f19d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe0380c6f30 0x7fe038070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:27.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.320+0000 7fe03f19d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c7300 0x7fe0381694f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:27.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.320+0000 7fe03f19d700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03806fbb0 con 0x7fe0380c4d10 2023-06-22T08:19:27.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.320+0000 7fe03f19d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe038169a30 con 0x7fe0380c6f30 2023-06-22T08:19:27.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.320+0000 7fe03f19d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe03806a740 con 0x7fe0380c7300 2023-06-22T08:19:27.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.320+0000 7fe03d99a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c7300 0x7fe0381694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:27.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.320+0000 7fe03e99c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe0380c4d10 0x7fe0380c5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:52604/0 (socket says 172.21.15.23:52604) 2023-06-22T08:19:27.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.320+0000 7fe03e99c700 1 -- 172.21.15.23:0/1198816557 learned_addr learned my addr 172.21.15.23:0/1198816557 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:19:27.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.320+0000 7fe03e19b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe0380c6f30 0x7fe038070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:27.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.321+0000 7fe03d199700 1 -- 172.21.15.23:0/1198816557 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963030730 0 0) 0x7fe03806fbb0 con 0x7fe0380c4d10 2023-06-22T08:19:27.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.469+0000 7fe03d199700 1 -- 172.21.15.23:0/1198816557 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe024003590 con 0x7fe0380c4d10 2023-06-22T08:19:27.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.469+0000 7fe03d99a700 1 -- 172.21.15.23:0/1198816557 >> v2:172.21.15.23:3300/0 conn(0x7fe0380c6f30 msgr2=0x7fe038070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:19:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.469+0000 7fe03d99a700 1 --2- 172.21.15.23:0/1198816557 >> v2:172.21.15.23:3300/0 conn(0x7fe0380c6f30 0x7fe038070fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.469+0000 7fe03d99a700 1 -- 172.21.15.23:0/1198816557 >> v1:172.21.15.67:6789/0 conn(0x7fe0380c4d10 legacy=0x7fe0380c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.469+0000 7fe03d199700 1 -- 172.21.15.23:0/1198816557 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1279291867 0 0) 0x7fe024003590 con 0x7fe0380c4d10 2023-06-22T08:19:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.469+0000 7fe03d99a700 1 -- 172.21.15.23:0/1198816557 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03806aa20 con 0x7fe0380c7300 2023-06-22T08:19:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.470+0000 7fe03d99a700 1 --2- 172.21.15.23:0/1198816557 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c7300 0x7fe0381694f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe02800b8f0 tx=0x7fe02800bc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20a1fc7c5307149d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.470+0000 7fe03d199700 1 -- 172.21.15.23:0/1198816557 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0300019f0 con 0x7fe0380c7300 2023-06-22T08:19:27.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.470+0000 7fe03d199700 1 -- 172.21.15.23:0/1198816557 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe028004030 con 0x7fe0380c7300 2023-06-22T08:19:27.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.470+0000 7fe03d199700 1 -- 172.21.15.23:0/1198816557 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0280043e0 con 0x7fe0380c7300 2023-06-22T08:19:27.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.471+0000 7fe03f19d700 1 -- 172.21.15.23:0/1198816557 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c7300 msgr2=0x7fe0381694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:27.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.471+0000 7fe03f19d700 1 --2- 172.21.15.23:0/1198816557 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c7300 0x7fe0381694f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe02800b8f0 tx=0x7fe02800bc00 comp rx=0 tx=0).stop 2023-06-22T08:19:27.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.471+0000 7fe03f19d700 1 -- 172.21.15.23:0/1198816557 shutdown_connections 2023-06-22T08:19:27.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.471+0000 7fe03f19d700 1 --2- 172.21.15.23:0/1198816557 >> v2:172.21.15.23:3300/0 conn(0x7fe0380c6f30 0x7fe038070fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:27.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.471+0000 7fe03f19d700 1 --2- 172.21.15.23:0/1198816557 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c7300 0x7fe0381694f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:27.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.471+0000 7fe03f19d700 1 -- 172.21.15.23:0/1198816557 >> 172.21.15.23:0/1198816557 conn(0x7fe03806e3b0 msgr2=0x7fe038069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:27.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.471+0000 7fe03f19d700 1 -- 172.21.15.23:0/1198816557 shutdown_connections 2023-06-22T08:19:27.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.471+0000 7fe03f19d700 1 -- 172.21.15.23:0/1198816557 wait complete. 2023-06-22T08:19:27.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.473+0000 7fe03f19d700 1 Processor -- start 2023-06-22T08:19:27.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.473+0000 7fe03f19d700 1 -- start start 2023-06-22T08:19:27.484 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.474+0000 7fe03f19d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c6f30 0x7fe0381616e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:27.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.474+0000 7fe03f19d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0380c7300 0x7fe0381fe910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:27.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03f19d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe038166720 0x7fe03815bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:27.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03f19d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0381671b0 con 0x7fe038166720 2023-06-22T08:19:27.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03f19d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe038166bd0 con 0x7fe0380c7300 2023-06-22T08:19:27.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03f19d700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0380ca950 con 0x7fe0380c6f30 2023-06-22T08:19:27.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03e19b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c6f30 0x7fe0381616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:27.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03e19b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c6f30 0x7fe0381616e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:50872/0 (socket says 172.21.15.23:50872) 2023-06-22T08:19:27.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03e19b700 1 -- 172.21.15.23:0/1693488462 learned_addr learned my addr 172.21.15.23:0/1693488462 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:27.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03d99a700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0380c7300 0x7fe0381fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:27.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03e99c700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe038166720 0x7fe03815bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:27.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03e19b700 1 -- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0380c7300 msgr2=0x7fe0381fe910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:27.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03e19b700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0380c7300 0x7fe0381fe910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:27.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03e19b700 1 -- 172.21.15.23:0/1693488462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe038166720 msgr2=0x7fe03815bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:27.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03e19b700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe038166720 0x7fe03815bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:27.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03e19b700 1 -- 172.21.15.23:0/1693488462 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02800b550 con 0x7fe0380c6f30 2023-06-22T08:19:27.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.475+0000 7fe03e19b700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c6f30 0x7fe0381616e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe03400b990 tx=0x7fe03400bca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:27.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.476+0000 7fe02effd700 1 -- 172.21.15.23:0/1693488462 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0340089b0 con 0x7fe0380c6f30 2023-06-22T08:19:27.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.476+0000 7fe03f19d700 1 -- 172.21.15.23:0/1693488462 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0380cac10 con 0x7fe0380c6f30 2023-06-22T08:19:27.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.476+0000 7fe02effd700 1 -- 172.21.15.23:0/1693488462 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe034008b10 con 0x7fe0380c6f30 2023-06-22T08:19:27.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.476+0000 7fe02effd700 1 -- 172.21.15.23:0/1693488462 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe034008e80 con 0x7fe0380c6f30 2023-06-22T08:19:27.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.476+0000 7fe03f19d700 1 -- 172.21.15.23:0/1693488462 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe038167640 con 0x7fe0380c6f30 2023-06-22T08:19:27.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.477+0000 7fe03f19d700 1 -- 172.21.15.23:0/1693488462 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe038167490 con 0x7fe0380c6f30 2023-06-22T08:19:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.478+0000 7fe02effd700 1 -- 172.21.15.23:0/1693488462 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fe03401e400 con 0x7fe0380c6f30 2023-06-22T08:19:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.479+0000 7fe02effd700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe01c06ad10 0x7fe01c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.480+0000 7fe02effd700 1 -- 172.21.15.23:0/1693488462 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7fe034089a20 con 0x7fe0380c6f30 2023-06-22T08:19:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.480+0000 7fe03d99a700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe01c06ad10 0x7fe01c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.480+0000 7fe03d99a700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe01c06ad10 0x7fe01c06d1c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe028020040 tx=0x7fe02801b000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:27.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.484+0000 7fe02effd700 1 -- 172.21.15.23:0/1693488462 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fe034054410 con 0x7fe0380c6f30 2023-06-22T08:19:27.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.771+0000 7fe03f19d700 1 -- 172.21.15.23:0/1693488462 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0380c42c0 con 0x7fe0380c6f30 2023-06-22T08:19:27.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.772+0000 7fe02effd700 1 -- 172.21.15.23:0/1693488462 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5510 (secure 0 0 0) 0x7fe034057a30 con 0x7fe0380c6f30 2023-06-22T08:19:27.774 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:27.775 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:19:19.962809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:active","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:27.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 -- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe01c06ad10 msgr2=0x7fe01c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:27.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe01c06ad10 0x7fe01c06d1c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe028020040 tx=0x7fe02801b000 comp rx=0 tx=0).stop 2023-06-22T08:19:27.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 -- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c6f30 msgr2=0x7fe0381616e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:27.777 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c6f30 0x7fe0381616e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe03400b990 tx=0x7fe03400bca0 comp rx=0 tx=0).stop 2023-06-22T08:19:27.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 -- 172.21.15.23:0/1693488462 shutdown_connections 2023-06-22T08:19:27.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe038166720 0x7fe03815bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:27.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe0380c6f30 0x7fe0381616e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:27.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fe01c06ad10 0x7fe01c06d1c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:27.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 --2- 172.21.15.23:0/1693488462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0380c7300 0x7fe0381fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:27.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 -- 172.21.15.23:0/1693488462 >> 172.21.15.23:0/1693488462 conn(0x7fe0380c4d10 msgr2=0x7fe038165950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:27.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 -- 172.21.15.23:0/1693488462 shutdown_connections 2023-06-22T08:19:27.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:27.774+0000 7fe03f19d700 1 -- 172.21.15.23:0/1693488462 wait complete. 2023-06-22T08:19:27.779 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 155 2023-06-22T08:19:27.798 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-06-22T08:19:27.916 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:27.916 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:27.916 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 66 -eq 48 ']' 2023-06-22T08:19:27.916 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.66 2023-06-22T08:19:27.917 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.66 2023-06-22T08:19:27.917 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.66 2023-06-22T08:19:27.917 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.66 2023-06-22T08:19:27.918 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:28.846 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:28.847 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:28.847 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 67 -eq 48 ']' 2023-06-22T08:19:28.847 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.67 2023-06-22T08:19:28.847 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.67 2023-06-22T08:19:28.848 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.67 2023-06-22T08:19:28.848 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.67 2023-06-22T08:19:28.848 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:29.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:29.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:29.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 68 -eq 48 ']' 2023-06-22T08:19:29.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.68 2023-06-22T08:19:29.740 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.68 2023-06-22T08:19:29.740 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.68 2023-06-22T08:19:29.740 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.68 2023-06-22T08:19:29.741 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:29.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:19:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.045+0000 7fd9227ff700 1 Processor -- start 2023-06-22T08:19:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.045+0000 7fd9227ff700 1 -- start start 2023-06-22T08:19:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.047+0000 7fd9227ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd91c14f0c0 0x7fd91c1558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.047+0000 7fd9227ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd91c0c61e0 0x7fd91c06df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:30.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.047+0000 7fd9227ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0c4ea0 con 0x7fd91c06e450 2023-06-22T08:19:30.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.047+0000 7fd9227ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91c0c1f70 con 0x7fd91c0c61e0 2023-06-22T08:19:30.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.047+0000 7fd9227ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91c0c5140 con 0x7fd91c14f0c0 2023-06-22T08:19:30.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.047+0000 7fd91b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd91c0c61e0 0x7fd91c06df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:30.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.047+0000 7fd920d9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fd91c06e450 0x7fd91c06d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:43734/0 (socket says 172.21.15.23:43734) 2023-06-22T08:19:30.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.047+0000 7fd920d9c700 1 -- 172.21.15.23:0/3890741466 learned_addr learned my addr 172.21.15.23:0/3890741466 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:19:30.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.048+0000 7fd91affd700 1 -- 172.21.15.23:0/3890741466 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908847022 0 0) 0x7fd91c0c4ea0 con 0x7fd91c06e450 2023-06-22T08:19:30.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.048+0000 7fd91affd700 1 -- 172.21.15.23:0/3890741466 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9040035b0 con 0x7fd91c06e450 2023-06-22T08:19:30.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.048+0000 7fd91bfff700 1 --2- 172.21.15.23:0/3890741466 >> v2:172.21.15.67:3300/0 conn(0x7fd91c14f0c0 0x7fd91c1558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:30.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.048+0000 7fd91b7fe700 1 -- 172.21.15.23:0/3890741466 >> v2:172.21.15.67:3300/0 conn(0x7fd91c14f0c0 msgr2=0x7fd91c1558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:19:30.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.048+0000 7fd91b7fe700 1 --2- 172.21.15.23:0/3890741466 >> v2:172.21.15.67:3300/0 conn(0x7fd91c14f0c0 0x7fd91c1558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:30.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.048+0000 7fd91b7fe700 1 -- 172.21.15.23:0/3890741466 >> v1:172.21.15.23:6789/0 conn(0x7fd91c06e450 legacy=0x7fd91c06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:30.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.048+0000 7fd91b7fe700 1 -- 172.21.15.23:0/3890741466 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c0c5420 con 0x7fd91c0c61e0 2023-06-22T08:19:30.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.049+0000 7fd91b7fe700 1 --2- 172.21.15.23:0/3890741466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd91c0c61e0 0x7fd91c06df10 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd91c06fab0 tx=0x7fd910009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49235f463dfbadda server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:30.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.049+0000 7fd91affd700 1 -- 172.21.15.23:0/3890741466 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910011830 con 0x7fd91c0c61e0 2023-06-22T08:19:30.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.049+0000 7fd91affd700 1 -- 172.21.15.23:0/3890741466 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910011990 con 0x7fd91c0c61e0 2023-06-22T08:19:30.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.049+0000 7fd91affd700 1 -- 172.21.15.23:0/3890741466 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910011d20 con 0x7fd91c0c61e0 2023-06-22T08:19:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.050+0000 7fd9227ff700 1 -- 172.21.15.23:0/3890741466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd91c0c61e0 msgr2=0x7fd91c06df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.050+0000 7fd9227ff700 1 --2- 172.21.15.23:0/3890741466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd91c0c61e0 0x7fd91c06df10 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd91c06fab0 tx=0x7fd910009b20 comp rx=0 tx=0).stop 2023-06-22T08:19:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.050+0000 7fd9227ff700 1 -- 172.21.15.23:0/3890741466 shutdown_connections 2023-06-22T08:19:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.050+0000 7fd9227ff700 1 --2- 172.21.15.23:0/3890741466 >> v2:172.21.15.67:3300/0 conn(0x7fd91c14f0c0 0x7fd91c1558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.050+0000 7fd9227ff700 1 --2- 172.21.15.23:0/3890741466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd91c0c61e0 0x7fd91c06df10 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.050+0000 7fd9227ff700 1 -- 172.21.15.23:0/3890741466 >> 172.21.15.23:0/3890741466 conn(0x7fd91c0ca3f0 msgr2=0x7fd91c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.050+0000 7fd9227ff700 1 -- 172.21.15.23:0/3890741466 shutdown_connections 2023-06-22T08:19:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.051+0000 7fd9227ff700 1 -- 172.21.15.23:0/3890741466 wait complete. 2023-06-22T08:19:30.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.053+0000 7fd9227ff700 1 Processor -- start 2023-06-22T08:19:30.056 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.053+0000 7fd9227ff700 1 -- start start 2023-06-22T08:19:30.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd9227ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd91c06e450 0x7fd91c0c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:30.057 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd9227ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd91c0c61e0 0x7fd91c153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:30.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd9227ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd91c14f0c0 0x7fd91c0c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:30.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd9227ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c0ca270 con 0x7fd91c14f0c0 2023-06-22T08:19:30.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd9227ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c152e90 con 0x7fd91c0c61e0 2023-06-22T08:19:30.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd9227ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c14f870 con 0x7fd91c06e450 2023-06-22T08:19:30.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd91bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd91c06e450 0x7fd91c0c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd91bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd91c06e450 0x7fd91c0c40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:50886/0 (socket says 172.21.15.23:50886) 2023-06-22T08:19:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd91bfff700 1 -- 172.21.15.23:0/4186870149 learned_addr learned my addr 172.21.15.23:0/4186870149 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:19:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd91b7fe700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd91c0c61e0 0x7fd91c153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd920d9c700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd91c14f0c0 0x7fd91c0c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd91bfff700 1 -- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd91c0c61e0 msgr2=0x7fd91c153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd91bfff700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd91c0c61e0 0x7fd91c153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd91bfff700 1 -- 172.21.15.23:0/4186870149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd91c14f0c0 msgr2=0x7fd91c0c81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.055+0000 7fd91bfff700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd91c14f0c0 0x7fd91c0c81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.056+0000 7fd91bfff700 1 -- 172.21.15.23:0/4186870149 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9100094a0 con 0x7fd91c06e450 2023-06-22T08:19:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.056+0000 7fd91bfff700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd91c06e450 0x7fd91c0c40a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00c7f0 tx=0x7fd90c00cb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.056+0000 7fd918ff9700 1 -- 172.21.15.23:0/4186870149 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c011800 con 0x7fd91c06e450 2023-06-22T08:19:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.056+0000 7fd918ff9700 1 -- 172.21.15.23:0/4186870149 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c011960 con 0x7fd91c06e450 2023-06-22T08:19:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.056+0000 7fd9227ff700 1 -- 172.21.15.23:0/4186870149 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c153130 con 0x7fd91c06e450 2023-06-22T08:19:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.056+0000 7fd918ff9700 1 -- 172.21.15.23:0/4186870149 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c011cd0 con 0x7fd91c06e450 2023-06-22T08:19:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.056+0000 7fd9227ff700 1 -- 172.21.15.23:0/4186870149 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c14fe10 con 0x7fd91c06e450 2023-06-22T08:19:30.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.058+0000 7fd9227ff700 1 -- 172.21.15.23:0/4186870149 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91c04ea20 con 0x7fd91c06e450 2023-06-22T08:19:30.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.059+0000 7fd918ff9700 1 -- 172.21.15.23:0/4186870149 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd90c010350 con 0x7fd91c06e450 2023-06-22T08:19:30.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.060+0000 7fd918ff9700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd90806ad10 0x7fd90806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:19:30.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.060+0000 7fd918ff9700 1 -- 172.21.15.23:0/4186870149 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7673+0+0 (secure 0 0 0) 0x7fd90c089620 con 0x7fd91c06e450 2023-06-22T08:19:30.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.060+0000 7fd91b7fe700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd90806ad10 0x7fd90806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:19:30.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.061+0000 7fd91b7fe700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd90806ad10 0x7fd90806d1c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd9100090b0 tx=0x7fd910008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:19:30.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.066+0000 7fd918ff9700 1 -- 172.21.15.23:0/4186870149 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd90c053ff0 con 0x7fd91c06e450 2023-06-22T08:19:30.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.417+0000 7fd9227ff700 1 -- 172.21.15.23:0/4186870149 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd91c0cb300 con 0x7fd91c06e450 2023-06-22T08:19:30.420 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.417+0000 7fd918ff9700 1 -- 172.21.15.23:0/4186870149 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+6267 (secure 0 0 0) 0x7fd90c057610 con 0x7fd91c06e450 2023-06-22T08:19:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:19:30.421 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6616,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:19:19.962809+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5845,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5845":{"gid":5845,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":40,"addr":"172.21.15.67:6836/3080866617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3080866617},{"type":"v1","addr":"172.21.15.67:6836","nonce":3080866617}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:active","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:19:30.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 -- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd90806ad10 msgr2=0x7fd90806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:30.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd90806ad10 0x7fd90806d1c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd9100090b0 tx=0x7fd910008030 comp rx=0 tx=0).stop 2023-06-22T08:19:30.423 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 -- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd91c06e450 msgr2=0x7fd91c0c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:19:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd91c06e450 0x7fd91c0c40a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00c7f0 tx=0x7fd90c00cb00 comp rx=0 tx=0).stop 2023-06-22T08:19:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 -- 172.21.15.23:0/4186870149 shutdown_connections 2023-06-22T08:19:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd91c14f0c0 0x7fd91c0c81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd91c06e450 0x7fd91c0c40a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd90806ad10 0x7fd90806d1c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 --2- 172.21.15.23:0/4186870149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd91c0c61e0 0x7fd91c153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:19:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 -- 172.21.15.23:0/4186870149 >> 172.21.15.23:0/4186870149 conn(0x7fd91c0ca3f0 msgr2=0x7fd91c14ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:19:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 -- 172.21.15.23:0/4186870149 shutdown_connections 2023-06-22T08:19:30.425 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:19:30.420+0000 7fd9227ff700 1 -- 172.21.15.23:0/4186870149 wait complete. 2023-06-22T08:19:30.425 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 156 2023-06-22T08:19:30.446 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-06-22T08:19:30.446 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 102 secs before thrashing 2023-06-22T08:19:30.882 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:30.883 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:30.884 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 69 -eq 48 ']' 2023-06-22T08:19:30.884 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.69 2023-06-22T08:19:30.884 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.69 2023-06-22T08:19:30.885 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.69 2023-06-22T08:19:30.885 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.69 2023-06-22T08:19:30.886 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:31.737 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:31.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:31.740 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 70 -eq 48 ']' 2023-06-22T08:19:31.740 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.70 2023-06-22T08:19:31.740 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.70 2023-06-22T08:19:31.740 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.70 2023-06-22T08:19:31.741 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.70 2023-06-22T08:19:31.741 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:32.501 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:32.501 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:32.501 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 71 -eq 48 ']' 2023-06-22T08:19:32.502 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.71 2023-06-22T08:19:32.502 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.71 2023-06-22T08:19:32.502 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.71 2023-06-22T08:19:32.502 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.71 2023-06-22T08:19:32.503 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:33.470 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:33.470 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:33.470 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 72 -eq 48 ']' 2023-06-22T08:19:33.471 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.72 2023-06-22T08:19:33.471 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.72 2023-06-22T08:19:33.472 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.72 2023-06-22T08:19:33.472 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.72 2023-06-22T08:19:33.472 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:34.447 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:34.447 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:34.447 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 73 -eq 48 ']' 2023-06-22T08:19:34.448 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.73 2023-06-22T08:19:34.448 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.73 2023-06-22T08:19:34.449 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.73 2023-06-22T08:19:34.449 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.73 2023-06-22T08:19:34.449 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:35.389 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:35.389 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:35.390 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 74 -eq 48 ']' 2023-06-22T08:19:35.390 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.74 2023-06-22T08:19:35.390 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.74 2023-06-22T08:19:35.390 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.74 2023-06-22T08:19:35.390 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.74 2023-06-22T08:19:35.398 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:36.678 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:36.680 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:36.680 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 75 -eq 48 ']' 2023-06-22T08:19:36.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.75 2023-06-22T08:19:36.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.75 2023-06-22T08:19:36.681 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.75 2023-06-22T08:19:36.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.75 2023-06-22T08:19:36.682 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:37.606 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:37.606 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:37.606 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 76 -eq 48 ']' 2023-06-22T08:19:37.606 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.76 2023-06-22T08:19:37.607 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.76 2023-06-22T08:19:37.607 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.76 2023-06-22T08:19:37.607 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.76 2023-06-22T08:19:37.608 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:38.570 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:38.571 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:38.571 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 77 -eq 48 ']' 2023-06-22T08:19:38.571 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.77 2023-06-22T08:19:38.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.77 2023-06-22T08:19:38.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.77 2023-06-22T08:19:38.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.77 2023-06-22T08:19:38.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:39.611 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:39.611 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:39.611 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 78 -eq 48 ']' 2023-06-22T08:19:39.612 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.78 2023-06-22T08:19:39.612 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.78 2023-06-22T08:19:39.612 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.78 2023-06-22T08:19:39.612 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.78 2023-06-22T08:19:39.613 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:40.724 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:40.725 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:40.725 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 79 -eq 48 ']' 2023-06-22T08:19:40.725 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.79 2023-06-22T08:19:40.726 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.79 2023-06-22T08:19:40.726 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.79 2023-06-22T08:19:40.726 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.79 2023-06-22T08:19:40.726 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:41.661 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:41.661 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:41.661 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 80 -eq 48 ']' 2023-06-22T08:19:41.661 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.80 2023-06-22T08:19:41.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.80 2023-06-22T08:19:41.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.80 2023-06-22T08:19:41.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.80 2023-06-22T08:19:41.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:42.680 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:42.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:42.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 81 -eq 48 ']' 2023-06-22T08:19:42.681 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.81 2023-06-22T08:19:42.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.81 2023-06-22T08:19:42.682 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.81 2023-06-22T08:19:42.682 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.81 2023-06-22T08:19:42.682 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:43.564 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:43.564 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:43.565 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 82 -eq 48 ']' 2023-06-22T08:19:43.565 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.82 2023-06-22T08:19:43.565 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.82 2023-06-22T08:19:43.566 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.82 2023-06-22T08:19:43.566 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.82 2023-06-22T08:19:43.566 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:44.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:44.596 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:44.596 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 83 -eq 48 ']' 2023-06-22T08:19:44.596 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.83 2023-06-22T08:19:44.597 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.83 2023-06-22T08:19:44.597 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.83 2023-06-22T08:19:44.597 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.83 2023-06-22T08:19:44.597 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:44.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:19:44.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:19:45.746 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:45.747 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:45.747 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 84 -eq 48 ']' 2023-06-22T08:19:45.747 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.84 2023-06-22T08:19:45.747 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.84 2023-06-22T08:19:45.748 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.84 2023-06-22T08:19:45.748 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.84 2023-06-22T08:19:45.748 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:46.701 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:46.702 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:46.702 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 85 -eq 48 ']' 2023-06-22T08:19:46.702 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.85 2023-06-22T08:19:46.702 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.85 2023-06-22T08:19:46.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.85 2023-06-22T08:19:46.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.85 2023-06-22T08:19:46.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:47.759 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:47.759 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:47.760 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 86 -eq 48 ']' 2023-06-22T08:19:47.760 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.86 2023-06-22T08:19:47.761 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.86 2023-06-22T08:19:47.761 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.86 2023-06-22T08:19:47.761 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.86 2023-06-22T08:19:47.762 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:48.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:48.952 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:48.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 87 -eq 48 ']' 2023-06-22T08:19:48.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.87 2023-06-22T08:19:48.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.87 2023-06-22T08:19:48.953 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.87 2023-06-22T08:19:48.954 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.87 2023-06-22T08:19:48.954 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:49.963 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:49.964 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:49.964 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 88 -eq 48 ']' 2023-06-22T08:19:49.964 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.88 2023-06-22T08:19:49.965 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.88 2023-06-22T08:19:49.965 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.88 2023-06-22T08:19:49.965 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.88 2023-06-22T08:19:49.965 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:51.168 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:51.168 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:51.168 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 89 -eq 48 ']' 2023-06-22T08:19:51.169 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.89 2023-06-22T08:19:51.170 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.89 2023-06-22T08:19:51.170 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.89 2023-06-22T08:19:51.170 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.89 2023-06-22T08:19:51.171 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:52.355 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:52.356 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:52.356 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 90 -eq 48 ']' 2023-06-22T08:19:52.356 INFO:tasks.workunit.client.0.smithi023.stdout:checking v0.90 2023-06-22T08:19:52.357 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.90 2023-06-22T08:19:52.357 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo checking v0.90 2023-06-22T08:19:52.357 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd .snap/v0.90 2023-06-22T08:19:52.357 INFO:tasks.workunit.client.0.smithi023.stderr:+ git diff --exit-code 2023-06-22T08:19:53.571 INFO:tasks.workunit.client.0.smithi023.stderr:+ cd ../.. 2023-06-22T08:19:53.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 1 -eq 48 ']' 2023-06-22T08:19:53.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.1 2023-06-22T08:19:53.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.1 2023-06-22T08:19:53.584 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.585 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 2 -eq 48 ']' 2023-06-22T08:19:53.585 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.2 2023-06-22T08:19:53.585 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.2 2023-06-22T08:19:53.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 3 -eq 48 ']' 2023-06-22T08:19:53.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.3 2023-06-22T08:19:53.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.3 2023-06-22T08:19:53.605 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.605 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 4 -eq 48 ']' 2023-06-22T08:19:53.606 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.4 2023-06-22T08:19:53.606 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.4 2023-06-22T08:19:53.615 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.615 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 5 -eq 48 ']' 2023-06-22T08:19:53.616 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.5 2023-06-22T08:19:53.616 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.5 2023-06-22T08:19:53.625 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.625 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 6 -eq 48 ']' 2023-06-22T08:19:53.626 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.6 2023-06-22T08:19:53.626 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.6 2023-06-22T08:19:53.634 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.635 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 7 -eq 48 ']' 2023-06-22T08:19:53.635 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.7 2023-06-22T08:19:53.635 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.7 2023-06-22T08:19:53.644 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.644 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 8 -eq 48 ']' 2023-06-22T08:19:53.644 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.8 2023-06-22T08:19:53.645 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.8 2023-06-22T08:19:53.653 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.654 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 9 -eq 48 ']' 2023-06-22T08:19:53.654 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.9 2023-06-22T08:19:53.654 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.9 2023-06-22T08:19:53.663 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.663 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 10 -eq 48 ']' 2023-06-22T08:19:53.663 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.10 2023-06-22T08:19:53.663 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.10 2023-06-22T08:19:53.672 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.672 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 11 -eq 48 ']' 2023-06-22T08:19:53.672 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.11 2023-06-22T08:19:53.672 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.11 2023-06-22T08:19:53.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 12 -eq 48 ']' 2023-06-22T08:19:53.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.12 2023-06-22T08:19:53.681 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.12 2023-06-22T08:19:53.691 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.691 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 13 -eq 48 ']' 2023-06-22T08:19:53.691 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.13 2023-06-22T08:19:53.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.13 2023-06-22T08:19:53.700 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.701 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 14 -eq 48 ']' 2023-06-22T08:19:53.701 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.14 2023-06-22T08:19:53.701 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.14 2023-06-22T08:19:53.712 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.712 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 15 -eq 48 ']' 2023-06-22T08:19:53.712 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.15 2023-06-22T08:19:53.712 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.15 2023-06-22T08:19:53.723 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.723 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 16 -eq 48 ']' 2023-06-22T08:19:53.723 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.16 2023-06-22T08:19:53.723 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.16 2023-06-22T08:19:53.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 17 -eq 48 ']' 2023-06-22T08:19:53.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.17 2023-06-22T08:19:53.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.17 2023-06-22T08:19:53.747 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.747 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 18 -eq 48 ']' 2023-06-22T08:19:53.747 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.18 2023-06-22T08:19:53.748 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.18 2023-06-22T08:19:53.757 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.758 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 19 -eq 48 ']' 2023-06-22T08:19:53.758 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.19 2023-06-22T08:19:53.758 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.19 2023-06-22T08:19:53.768 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.768 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 20 -eq 48 ']' 2023-06-22T08:19:53.768 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.20 2023-06-22T08:19:53.768 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.20 2023-06-22T08:19:53.778 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.778 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 21 -eq 48 ']' 2023-06-22T08:19:53.778 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.21 2023-06-22T08:19:53.779 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.21 2023-06-22T08:19:53.787 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 22 -eq 48 ']' 2023-06-22T08:19:53.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.22 2023-06-22T08:19:53.788 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.22 2023-06-22T08:19:53.797 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.797 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 23 -eq 48 ']' 2023-06-22T08:19:53.797 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.23 2023-06-22T08:19:53.797 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.23 2023-06-22T08:19:53.809 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.809 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 24 -eq 48 ']' 2023-06-22T08:19:53.809 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.24 2023-06-22T08:19:53.810 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.24 2023-06-22T08:19:53.824 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.824 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 25 -eq 48 ']' 2023-06-22T08:19:53.825 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.25 2023-06-22T08:19:53.825 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.25 2023-06-22T08:19:53.837 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.838 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 26 -eq 48 ']' 2023-06-22T08:19:53.838 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.26 2023-06-22T08:19:53.838 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.26 2023-06-22T08:19:53.852 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.852 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 27 -eq 48 ']' 2023-06-22T08:19:53.853 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.27 2023-06-22T08:19:53.853 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.27 2023-06-22T08:19:53.867 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.867 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 28 -eq 48 ']' 2023-06-22T08:19:53.867 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.28 2023-06-22T08:19:53.868 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.28 2023-06-22T08:19:53.882 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.882 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 29 -eq 48 ']' 2023-06-22T08:19:53.883 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.29 2023-06-22T08:19:53.883 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.29 2023-06-22T08:19:53.897 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.897 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 30 -eq 48 ']' 2023-06-22T08:19:53.898 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.30 2023-06-22T08:19:53.898 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.30 2023-06-22T08:19:53.911 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.911 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 31 -eq 48 ']' 2023-06-22T08:19:53.912 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.31 2023-06-22T08:19:53.912 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.31 2023-06-22T08:19:53.926 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.926 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 32 -eq 48 ']' 2023-06-22T08:19:53.927 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.32 2023-06-22T08:19:53.927 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.32 2023-06-22T08:19:53.938 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.939 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 33 -eq 48 ']' 2023-06-22T08:19:53.939 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.33 2023-06-22T08:19:53.939 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.33 2023-06-22T08:19:53.950 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.950 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 34 -eq 48 ']' 2023-06-22T08:19:53.950 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.34 2023-06-22T08:19:53.951 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.34 2023-06-22T08:19:53.967 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.967 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 35 -eq 48 ']' 2023-06-22T08:19:53.967 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.35 2023-06-22T08:19:53.967 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.35 2023-06-22T08:19:53.983 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.983 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 36 -eq 48 ']' 2023-06-22T08:19:53.983 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.36 2023-06-22T08:19:53.983 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.36 2023-06-22T08:19:53.992 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:53.992 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 37 -eq 48 ']' 2023-06-22T08:19:53.992 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.37 2023-06-22T08:19:53.993 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.37 2023-06-22T08:19:54.000 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.001 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 38 -eq 48 ']' 2023-06-22T08:19:54.001 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.38 2023-06-22T08:19:54.001 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.38 2023-06-22T08:19:54.009 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.009 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 39 -eq 48 ']' 2023-06-22T08:19:54.009 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.39 2023-06-22T08:19:54.009 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.39 2023-06-22T08:19:54.017 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.018 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 40 -eq 48 ']' 2023-06-22T08:19:54.018 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.40 2023-06-22T08:19:54.018 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.40 2023-06-22T08:19:54.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 41 -eq 48 ']' 2023-06-22T08:19:54.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.41 2023-06-22T08:19:54.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.41 2023-06-22T08:19:54.035 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.035 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 42 -eq 48 ']' 2023-06-22T08:19:54.035 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.42 2023-06-22T08:19:54.036 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.42 2023-06-22T08:19:54.044 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.044 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 43 -eq 48 ']' 2023-06-22T08:19:54.045 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.43 2023-06-22T08:19:54.045 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.43 2023-06-22T08:19:54.053 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.053 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 44 -eq 48 ']' 2023-06-22T08:19:54.053 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.44 2023-06-22T08:19:54.053 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.44 2023-06-22T08:19:54.062 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.063 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 45 -eq 48 ']' 2023-06-22T08:19:54.063 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.45 2023-06-22T08:19:54.063 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.45 2023-06-22T08:19:54.071 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.072 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 46 -eq 48 ']' 2023-06-22T08:19:54.072 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.46 2023-06-22T08:19:54.072 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.46 2023-06-22T08:19:54.080 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.080 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 47 -eq 48 ']' 2023-06-22T08:19:54.081 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.47 2023-06-22T08:19:54.081 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.47 2023-06-22T08:19:54.091 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.091 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 48 -eq 48 ']' 2023-06-22T08:19:54.091 INFO:tasks.workunit.client.0.smithi023.stderr:+ continue 2023-06-22T08:19:54.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 49 -eq 48 ']' 2023-06-22T08:19:54.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.49 2023-06-22T08:19:54.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.49 2023-06-22T08:19:54.100 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.100 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 50 -eq 48 ']' 2023-06-22T08:19:54.101 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.50 2023-06-22T08:19:54.101 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.50 2023-06-22T08:19:54.110 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.111 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 51 -eq 48 ']' 2023-06-22T08:19:54.111 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.51 2023-06-22T08:19:54.111 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.51 2023-06-22T08:19:54.119 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.119 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 52 -eq 48 ']' 2023-06-22T08:19:54.119 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.52 2023-06-22T08:19:54.119 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.52 2023-06-22T08:19:54.129 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 53 -eq 48 ']' 2023-06-22T08:19:54.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.53 2023-06-22T08:19:54.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.53 2023-06-22T08:19:54.140 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.141 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 54 -eq 48 ']' 2023-06-22T08:19:54.141 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.54 2023-06-22T08:19:54.141 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.54 2023-06-22T08:19:54.149 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.149 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 55 -eq 48 ']' 2023-06-22T08:19:54.150 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.55 2023-06-22T08:19:54.150 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.55 2023-06-22T08:19:54.162 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.162 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 56 -eq 48 ']' 2023-06-22T08:19:54.163 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.56 2023-06-22T08:19:54.163 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.56 2023-06-22T08:19:54.173 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.173 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 57 -eq 48 ']' 2023-06-22T08:19:54.173 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.57 2023-06-22T08:19:54.173 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.57 2023-06-22T08:19:54.181 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.181 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 58 -eq 48 ']' 2023-06-22T08:19:54.181 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.58 2023-06-22T08:19:54.182 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.58 2023-06-22T08:19:54.195 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.195 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 59 -eq 48 ']' 2023-06-22T08:19:54.195 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.59 2023-06-22T08:19:54.195 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.59 2023-06-22T08:19:54.203 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.203 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 60 -eq 48 ']' 2023-06-22T08:19:54.203 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.60 2023-06-22T08:19:54.204 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.60 2023-06-22T08:19:54.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 61 -eq 48 ']' 2023-06-22T08:19:54.214 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.61 2023-06-22T08:19:54.215 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.61 2023-06-22T08:19:54.223 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.223 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 62 -eq 48 ']' 2023-06-22T08:19:54.224 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.62 2023-06-22T08:19:54.224 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.62 2023-06-22T08:19:54.234 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.234 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 63 -eq 48 ']' 2023-06-22T08:19:54.234 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.63 2023-06-22T08:19:54.235 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.63 2023-06-22T08:19:54.247 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.248 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 64 -eq 48 ']' 2023-06-22T08:19:54.248 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.64 2023-06-22T08:19:54.248 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.64 2023-06-22T08:19:54.266 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.266 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 65 -eq 48 ']' 2023-06-22T08:19:54.266 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.65 2023-06-22T08:19:54.266 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.65 2023-06-22T08:19:54.291 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.291 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 66 -eq 48 ']' 2023-06-22T08:19:54.291 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.66 2023-06-22T08:19:54.292 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.66 2023-06-22T08:19:54.313 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.314 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 67 -eq 48 ']' 2023-06-22T08:19:54.314 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.67 2023-06-22T08:19:54.314 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.67 2023-06-22T08:19:54.333 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.334 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 68 -eq 48 ']' 2023-06-22T08:19:54.334 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.68 2023-06-22T08:19:54.334 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.68 2023-06-22T08:19:54.354 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.354 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 69 -eq 48 ']' 2023-06-22T08:19:54.354 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.69 2023-06-22T08:19:54.355 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.69 2023-06-22T08:19:54.373 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.373 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 70 -eq 48 ']' 2023-06-22T08:19:54.373 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.70 2023-06-22T08:19:54.374 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.70 2023-06-22T08:19:54.393 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.394 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 71 -eq 48 ']' 2023-06-22T08:19:54.394 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.71 2023-06-22T08:19:54.394 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.71 2023-06-22T08:19:54.414 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.415 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 72 -eq 48 ']' 2023-06-22T08:19:54.415 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.72 2023-06-22T08:19:54.415 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.72 2023-06-22T08:19:54.436 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.436 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 73 -eq 48 ']' 2023-06-22T08:19:54.437 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.73 2023-06-22T08:19:54.437 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.73 2023-06-22T08:19:54.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 74 -eq 48 ']' 2023-06-22T08:19:54.457 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.74 2023-06-22T08:19:54.458 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.74 2023-06-22T08:19:54.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 75 -eq 48 ']' 2023-06-22T08:19:54.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.75 2023-06-22T08:19:54.479 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.75 2023-06-22T08:19:54.500 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.500 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 76 -eq 48 ']' 2023-06-22T08:19:54.501 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.76 2023-06-22T08:19:54.501 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.76 2023-06-22T08:19:54.521 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.522 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 77 -eq 48 ']' 2023-06-22T08:19:54.522 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.77 2023-06-22T08:19:54.523 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.77 2023-06-22T08:19:54.542 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.543 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 78 -eq 48 ']' 2023-06-22T08:19:54.543 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.78 2023-06-22T08:19:54.543 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.78 2023-06-22T08:19:54.567 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.568 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 79 -eq 48 ']' 2023-06-22T08:19:54.568 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.79 2023-06-22T08:19:54.568 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.79 2023-06-22T08:19:54.589 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.590 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 80 -eq 48 ']' 2023-06-22T08:19:54.590 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.80 2023-06-22T08:19:54.590 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.80 2023-06-22T08:19:54.614 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.614 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 81 -eq 48 ']' 2023-06-22T08:19:54.614 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.81 2023-06-22T08:19:54.615 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.81 2023-06-22T08:19:54.645 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.645 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 82 -eq 48 ']' 2023-06-22T08:19:54.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.82 2023-06-22T08:19:54.646 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.82 2023-06-22T08:19:54.668 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.668 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 83 -eq 48 ']' 2023-06-22T08:19:54.668 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.83 2023-06-22T08:19:54.668 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.83 2023-06-22T08:19:54.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 84 -eq 48 ']' 2023-06-22T08:19:54.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.84 2023-06-22T08:19:54.692 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.84 2023-06-22T08:19:54.717 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.717 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 85 -eq 48 ']' 2023-06-22T08:19:54.717 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.85 2023-06-22T08:19:54.717 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.85 2023-06-22T08:19:54.743 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.744 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 86 -eq 48 ']' 2023-06-22T08:19:54.744 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.86 2023-06-22T08:19:54.744 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.86 2023-06-22T08:19:54.768 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.769 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 87 -eq 48 ']' 2023-06-22T08:19:54.769 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.87 2023-06-22T08:19:54.769 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.87 2023-06-22T08:19:54.791 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.791 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 88 -eq 48 ']' 2023-06-22T08:19:54.791 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.88 2023-06-22T08:19:54.791 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.88 2023-06-22T08:19:54.816 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.817 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 89 -eq 48 ']' 2023-06-22T08:19:54.817 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.89 2023-06-22T08:19:54.817 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.89 2023-06-22T08:19:54.840 INFO:tasks.workunit.client.0.smithi023.stderr:+ for v in $versions 2023-06-22T08:19:54.840 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' 90 -eq 48 ']' 2023-06-22T08:19:54.840 INFO:tasks.workunit.client.0.smithi023.stderr:+ ver=v0.90 2023-06-22T08:19:54.840 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/v0.90 2023-06-22T08:19:54.862 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:19:54.863 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:19:54.863 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:19:54.864 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:20:13.450 INFO:tasks.ceph.mds.b.smithi067.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4479-g8627af3e/rpm/el8/BUILD/ceph-18.0.0-4479-g8627af3e/src/mds/CInode.cc: In function 'void CInode::verify_dirfrags()' thread 7f0656ed4700 time 2023-06-22T08:20:13.448324+0000 2023-06-22T08:20:13.450 INFO:tasks.ceph.mds.b.smithi067.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4479-g8627af3e/rpm/el8/BUILD/ceph-18.0.0-4479-g8627af3e/src/mds/CInode.cc: 751: FAILED ceph_assert(!bad) 2023-06-22T08:20:13.451 INFO:tasks.ceph.mds.b.smithi067.stderr: ceph version 18.0.0-4479-g8627af3e (8627af3e0adcb765a3c249fcc209cba9f4873e1b) reef (dev) 2023-06-22T08:20:13.451 INFO:tasks.ceph.mds.b.smithi067.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f066278cdbf] 2023-06-22T08:20:13.451 INFO:tasks.ceph.mds.b.smithi067.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7f066278cf85] 2023-06-22T08:20:13.451 INFO:tasks.ceph.mds.b.smithi067.stderr: 3: (CInode::verify_dirfrags()+0x4a4) [0x556681248884] 2023-06-22T08:20:13.451 INFO:tasks.ceph.mds.b.smithi067.stderr: 4: (MDCache::handle_fragment_notify(boost::intrusive_ptr const&)+0x28c) [0x5566810c468c] 2023-06-22T08:20:13.452 INFO:tasks.ceph.mds.b.smithi067.stderr: 5: (MDCache::dispatch(boost::intrusive_ptr const&)+0x3a4) [0x556681108064] 2023-06-22T08:20:13.452 INFO:tasks.ceph.mds.b.smithi067.stderr: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x556680f71e33] 2023-06-22T08:20:13.452 INFO:tasks.ceph.mds.b.smithi067.stderr: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556680f7244c] 2023-06-22T08:20:13.452 INFO:tasks.ceph.mds.b.smithi067.stderr: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556680f5bccf] 2023-06-22T08:20:13.452 INFO:tasks.ceph.mds.b.smithi067.stderr: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0662a05798] 2023-06-22T08:20:13.452 INFO:tasks.ceph.mds.b.smithi067.stderr: 10: (DispatchQueue::entry()+0x50f) [0x7f0662a0293f] 2023-06-22T08:20:13.452 INFO:tasks.ceph.mds.b.smithi067.stderr: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0662ac8c31] 2023-06-22T08:20:13.453 INFO:tasks.ceph.mds.b.smithi067.stderr: 12: /lib64/libpthread.so.0(+0x81cf) [0x7f06615351cf] 2023-06-22T08:20:13.453 INFO:tasks.ceph.mds.b.smithi067.stderr: 13: clone() 2023-06-22T08:20:13.453 INFO:tasks.ceph.mds.b.smithi067.stderr:2023-06-22T08:20:13.448+0000 7f0656ed4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4479-g8627af3e/rpm/el8/BUILD/ceph-18.0.0-4479-g8627af3e/src/mds/CInode.cc: In function 'void CInode::verify_dirfrags()' thread 7f0656ed4700 time 2023-06-22T08:20:13.448324+0000 2023-06-22T08:20:13.453 INFO:tasks.ceph.mds.b.smithi067.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4479-g8627af3e/rpm/el8/BUILD/ceph-18.0.0-4479-g8627af3e/src/mds/CInode.cc: 751: FAILED ceph_assert(!bad) 2023-06-22T08:20:13.453 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.453 INFO:tasks.ceph.mds.b.smithi067.stderr: ceph version 18.0.0-4479-g8627af3e (8627af3e0adcb765a3c249fcc209cba9f4873e1b) reef (dev) 2023-06-22T08:20:13.453 INFO:tasks.ceph.mds.b.smithi067.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f066278cdbf] 2023-06-22T08:20:13.454 INFO:tasks.ceph.mds.b.smithi067.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7f066278cf85] 2023-06-22T08:20:13.454 INFO:tasks.ceph.mds.b.smithi067.stderr: 3: (CInode::verify_dirfrags()+0x4a4) [0x556681248884] 2023-06-22T08:20:13.454 INFO:tasks.ceph.mds.b.smithi067.stderr: 4: (MDCache::handle_fragment_notify(boost::intrusive_ptr const&)+0x28c) [0x5566810c468c] 2023-06-22T08:20:13.454 INFO:tasks.ceph.mds.b.smithi067.stderr: 5: (MDCache::dispatch(boost::intrusive_ptr const&)+0x3a4) [0x556681108064] 2023-06-22T08:20:13.454 INFO:tasks.ceph.mds.b.smithi067.stderr: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x556680f71e33] 2023-06-22T08:20:13.454 INFO:tasks.ceph.mds.b.smithi067.stderr: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556680f7244c] 2023-06-22T08:20:13.454 INFO:tasks.ceph.mds.b.smithi067.stderr: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556680f5bccf] 2023-06-22T08:20:13.455 INFO:tasks.ceph.mds.b.smithi067.stderr: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0662a05798] 2023-06-22T08:20:13.455 INFO:tasks.ceph.mds.b.smithi067.stderr: 10: (DispatchQueue::entry()+0x50f) [0x7f0662a0293f] 2023-06-22T08:20:13.455 INFO:tasks.ceph.mds.b.smithi067.stderr: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0662ac8c31] 2023-06-22T08:20:13.455 INFO:tasks.ceph.mds.b.smithi067.stderr: 12: /lib64/libpthread.so.0(+0x81cf) [0x7f06615351cf] 2023-06-22T08:20:13.455 INFO:tasks.ceph.mds.b.smithi067.stderr: 13: clone() 2023-06-22T08:20:13.455 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.455 INFO:tasks.ceph.mds.b.smithi067.stderr:*** Caught signal (Aborted) ** 2023-06-22T08:20:13.455 INFO:tasks.ceph.mds.b.smithi067.stderr: in thread 7f0656ed4700 thread_name:ms_dispatch 2023-06-22T08:20:13.456 INFO:tasks.ceph.mds.b.smithi067.stderr: ceph version 18.0.0-4479-g8627af3e (8627af3e0adcb765a3c249fcc209cba9f4873e1b) reef (dev) 2023-06-22T08:20:13.456 INFO:tasks.ceph.mds.b.smithi067.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f066153fce0] 2023-06-22T08:20:13.456 INFO:tasks.ceph.mds.b.smithi067.stderr: 2: gsignal() 2023-06-22T08:20:13.456 INFO:tasks.ceph.mds.b.smithi067.stderr: 3: abort() 2023-06-22T08:20:13.457 INFO:tasks.ceph.mds.b.smithi067.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f066278ce19] 2023-06-22T08:20:13.457 INFO:tasks.ceph.mds.b.smithi067.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7f066278cf85] 2023-06-22T08:20:13.457 INFO:tasks.ceph.mds.b.smithi067.stderr: 6: (CInode::verify_dirfrags()+0x4a4) [0x556681248884] 2023-06-22T08:20:13.457 INFO:tasks.ceph.mds.b.smithi067.stderr: 7: (MDCache::handle_fragment_notify(boost::intrusive_ptr const&)+0x28c) [0x5566810c468c] 2023-06-22T08:20:13.457 INFO:tasks.ceph.mds.b.smithi067.stderr: 8: (MDCache::dispatch(boost::intrusive_ptr const&)+0x3a4) [0x556681108064] 2023-06-22T08:20:13.457 INFO:tasks.ceph.mds.b.smithi067.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x556680f71e33] 2023-06-22T08:20:13.457 INFO:tasks.ceph.mds.b.smithi067.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556680f7244c] 2023-06-22T08:20:13.457 INFO:tasks.ceph.mds.b.smithi067.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556680f5bccf] 2023-06-22T08:20:13.458 INFO:tasks.ceph.mds.b.smithi067.stderr: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0662a05798] 2023-06-22T08:20:13.458 INFO:tasks.ceph.mds.b.smithi067.stderr: 13: (DispatchQueue::entry()+0x50f) [0x7f0662a0293f] 2023-06-22T08:20:13.458 INFO:tasks.ceph.mds.b.smithi067.stderr: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0662ac8c31] 2023-06-22T08:20:13.458 INFO:tasks.ceph.mds.b.smithi067.stderr: 15: /lib64/libpthread.so.0(+0x81cf) [0x7f06615351cf] 2023-06-22T08:20:13.458 INFO:tasks.ceph.mds.b.smithi067.stderr: 16: clone() 2023-06-22T08:20:13.458 INFO:tasks.ceph.mds.b.smithi067.stderr:2023-06-22T08:20:13.449+0000 7f0656ed4700 -1 *** Caught signal (Aborted) ** 2023-06-22T08:20:13.458 INFO:tasks.ceph.mds.b.smithi067.stderr: in thread 7f0656ed4700 thread_name:ms_dispatch 2023-06-22T08:20:13.459 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.459 INFO:tasks.ceph.mds.b.smithi067.stderr: ceph version 18.0.0-4479-g8627af3e (8627af3e0adcb765a3c249fcc209cba9f4873e1b) reef (dev) 2023-06-22T08:20:13.459 INFO:tasks.ceph.mds.b.smithi067.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f066153fce0] 2023-06-22T08:20:13.459 INFO:tasks.ceph.mds.b.smithi067.stderr: 2: gsignal() 2023-06-22T08:20:13.459 INFO:tasks.ceph.mds.b.smithi067.stderr: 3: abort() 2023-06-22T08:20:13.459 INFO:tasks.ceph.mds.b.smithi067.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f066278ce19] 2023-06-22T08:20:13.459 INFO:tasks.ceph.mds.b.smithi067.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7f066278cf85] 2023-06-22T08:20:13.459 INFO:tasks.ceph.mds.b.smithi067.stderr: 6: (CInode::verify_dirfrags()+0x4a4) [0x556681248884] 2023-06-22T08:20:13.460 INFO:tasks.ceph.mds.b.smithi067.stderr: 7: (MDCache::handle_fragment_notify(boost::intrusive_ptr const&)+0x28c) [0x5566810c468c] 2023-06-22T08:20:13.460 INFO:tasks.ceph.mds.b.smithi067.stderr: 8: (MDCache::dispatch(boost::intrusive_ptr const&)+0x3a4) [0x556681108064] 2023-06-22T08:20:13.460 INFO:tasks.ceph.mds.b.smithi067.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x556680f71e33] 2023-06-22T08:20:13.460 INFO:tasks.ceph.mds.b.smithi067.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556680f7244c] 2023-06-22T08:20:13.460 INFO:tasks.ceph.mds.b.smithi067.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556680f5bccf] 2023-06-22T08:20:13.460 INFO:tasks.ceph.mds.b.smithi067.stderr: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0662a05798] 2023-06-22T08:20:13.460 INFO:tasks.ceph.mds.b.smithi067.stderr: 13: (DispatchQueue::entry()+0x50f) [0x7f0662a0293f] 2023-06-22T08:20:13.461 INFO:tasks.ceph.mds.b.smithi067.stderr: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0662ac8c31] 2023-06-22T08:20:13.461 INFO:tasks.ceph.mds.b.smithi067.stderr: 15: /lib64/libpthread.so.0(+0x81cf) [0x7f06615351cf] 2023-06-22T08:20:13.461 INFO:tasks.ceph.mds.b.smithi067.stderr: 16: clone() 2023-06-22T08:20:13.461 INFO:tasks.ceph.mds.b.smithi067.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-06-22T08:20:13.461 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.476 INFO:tasks.ceph.mds.b.smithi067.stderr: -1> 2023-06-22T08:20:13.448+0000 7f0656ed4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4479-g8627af3e/rpm/el8/BUILD/ceph-18.0.0-4479-g8627af3e/src/mds/CInode.cc: In function 'void CInode::verify_dirfrags()' thread 7f0656ed4700 time 2023-06-22T08:20:13.448324+0000 2023-06-22T08:20:13.476 INFO:tasks.ceph.mds.b.smithi067.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4479-g8627af3e/rpm/el8/BUILD/ceph-18.0.0-4479-g8627af3e/src/mds/CInode.cc: 751: FAILED ceph_assert(!bad) 2023-06-22T08:20:13.476 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.477 INFO:tasks.ceph.mds.b.smithi067.stderr: ceph version 18.0.0-4479-g8627af3e (8627af3e0adcb765a3c249fcc209cba9f4873e1b) reef (dev) 2023-06-22T08:20:13.477 INFO:tasks.ceph.mds.b.smithi067.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f066278cdbf] 2023-06-22T08:20:13.477 INFO:tasks.ceph.mds.b.smithi067.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7f066278cf85] 2023-06-22T08:20:13.477 INFO:tasks.ceph.mds.b.smithi067.stderr: 3: (CInode::verify_dirfrags()+0x4a4) [0x556681248884] 2023-06-22T08:20:13.477 INFO:tasks.ceph.mds.b.smithi067.stderr: 4: (MDCache::handle_fragment_notify(boost::intrusive_ptr const&)+0x28c) [0x5566810c468c] 2023-06-22T08:20:13.477 INFO:tasks.ceph.mds.b.smithi067.stderr: 5: (MDCache::dispatch(boost::intrusive_ptr const&)+0x3a4) [0x556681108064] 2023-06-22T08:20:13.477 INFO:tasks.ceph.mds.b.smithi067.stderr: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x556680f71e33] 2023-06-22T08:20:13.478 INFO:tasks.ceph.mds.b.smithi067.stderr: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556680f7244c] 2023-06-22T08:20:13.478 INFO:tasks.ceph.mds.b.smithi067.stderr: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556680f5bccf] 2023-06-22T08:20:13.478 INFO:tasks.ceph.mds.b.smithi067.stderr: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0662a05798] 2023-06-22T08:20:13.478 INFO:tasks.ceph.mds.b.smithi067.stderr: 10: (DispatchQueue::entry()+0x50f) [0x7f0662a0293f] 2023-06-22T08:20:13.478 INFO:tasks.ceph.mds.b.smithi067.stderr: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0662ac8c31] 2023-06-22T08:20:13.478 INFO:tasks.ceph.mds.b.smithi067.stderr: 12: /lib64/libpthread.so.0(+0x81cf) [0x7f06615351cf] 2023-06-22T08:20:13.478 INFO:tasks.ceph.mds.b.smithi067.stderr: 13: clone() 2023-06-22T08:20:13.479 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.479 INFO:tasks.ceph.mds.b.smithi067.stderr: 0> 2023-06-22T08:20:13.449+0000 7f0656ed4700 -1 *** Caught signal (Aborted) ** 2023-06-22T08:20:13.479 INFO:tasks.ceph.mds.b.smithi067.stderr: in thread 7f0656ed4700 thread_name:ms_dispatch 2023-06-22T08:20:13.479 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.479 INFO:tasks.ceph.mds.b.smithi067.stderr: ceph version 18.0.0-4479-g8627af3e (8627af3e0adcb765a3c249fcc209cba9f4873e1b) reef (dev) 2023-06-22T08:20:13.479 INFO:tasks.ceph.mds.b.smithi067.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f066153fce0] 2023-06-22T08:20:13.479 INFO:tasks.ceph.mds.b.smithi067.stderr: 2: gsignal() 2023-06-22T08:20:13.479 INFO:tasks.ceph.mds.b.smithi067.stderr: 3: abort() 2023-06-22T08:20:13.480 INFO:tasks.ceph.mds.b.smithi067.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f066278ce19] 2023-06-22T08:20:13.480 INFO:tasks.ceph.mds.b.smithi067.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7f066278cf85] 2023-06-22T08:20:13.480 INFO:tasks.ceph.mds.b.smithi067.stderr: 6: (CInode::verify_dirfrags()+0x4a4) [0x556681248884] 2023-06-22T08:20:13.480 INFO:tasks.ceph.mds.b.smithi067.stderr: 7: (MDCache::handle_fragment_notify(boost::intrusive_ptr const&)+0x28c) [0x5566810c468c] 2023-06-22T08:20:13.480 INFO:tasks.ceph.mds.b.smithi067.stderr: 8: (MDCache::dispatch(boost::intrusive_ptr const&)+0x3a4) [0x556681108064] 2023-06-22T08:20:13.480 INFO:tasks.ceph.mds.b.smithi067.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x556680f71e33] 2023-06-22T08:20:13.480 INFO:tasks.ceph.mds.b.smithi067.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556680f7244c] 2023-06-22T08:20:13.481 INFO:tasks.ceph.mds.b.smithi067.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556680f5bccf] 2023-06-22T08:20:13.481 INFO:tasks.ceph.mds.b.smithi067.stderr: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0662a05798] 2023-06-22T08:20:13.481 INFO:tasks.ceph.mds.b.smithi067.stderr: 13: (DispatchQueue::entry()+0x50f) [0x7f0662a0293f] 2023-06-22T08:20:13.481 INFO:tasks.ceph.mds.b.smithi067.stderr: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0662ac8c31] 2023-06-22T08:20:13.481 INFO:tasks.ceph.mds.b.smithi067.stderr: 15: /lib64/libpthread.so.0(+0x81cf) [0x7f06615351cf] 2023-06-22T08:20:13.481 INFO:tasks.ceph.mds.b.smithi067.stderr: 16: clone() 2023-06-22T08:20:13.481 INFO:tasks.ceph.mds.b.smithi067.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-06-22T08:20:13.481 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.496 INFO:tasks.ceph.mds.b.smithi067.stderr: -1> 2023-06-22T08:20:13.448+0000 7f0656ed4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4479-g8627af3e/rpm/el8/BUILD/ceph-18.0.0-4479-g8627af3e/src/mds/CInode.cc: In function 'void CInode::verify_dirfrags()' thread 7f0656ed4700 time 2023-06-22T08:20:13.448324+0000 2023-06-22T08:20:13.497 INFO:tasks.ceph.mds.b.smithi067.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-4479-g8627af3e/rpm/el8/BUILD/ceph-18.0.0-4479-g8627af3e/src/mds/CInode.cc: 751: FAILED ceph_assert(!bad) 2023-06-22T08:20:13.497 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.497 INFO:tasks.ceph.mds.b.smithi067.stderr: ceph version 18.0.0-4479-g8627af3e (8627af3e0adcb765a3c249fcc209cba9f4873e1b) reef (dev) 2023-06-22T08:20:13.497 INFO:tasks.ceph.mds.b.smithi067.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f066278cdbf] 2023-06-22T08:20:13.497 INFO:tasks.ceph.mds.b.smithi067.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7f066278cf85] 2023-06-22T08:20:13.497 INFO:tasks.ceph.mds.b.smithi067.stderr: 3: (CInode::verify_dirfrags()+0x4a4) [0x556681248884] 2023-06-22T08:20:13.497 INFO:tasks.ceph.mds.b.smithi067.stderr: 4: (MDCache::handle_fragment_notify(boost::intrusive_ptr const&)+0x28c) [0x5566810c468c] 2023-06-22T08:20:13.498 INFO:tasks.ceph.mds.b.smithi067.stderr: 5: (MDCache::dispatch(boost::intrusive_ptr const&)+0x3a4) [0x556681108064] 2023-06-22T08:20:13.498 INFO:tasks.ceph.mds.b.smithi067.stderr: 6: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x556680f71e33] 2023-06-22T08:20:13.498 INFO:tasks.ceph.mds.b.smithi067.stderr: 7: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556680f7244c] 2023-06-22T08:20:13.498 INFO:tasks.ceph.mds.b.smithi067.stderr: 8: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556680f5bccf] 2023-06-22T08:20:13.498 INFO:tasks.ceph.mds.b.smithi067.stderr: 9: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0662a05798] 2023-06-22T08:20:13.498 INFO:tasks.ceph.mds.b.smithi067.stderr: 10: (DispatchQueue::entry()+0x50f) [0x7f0662a0293f] 2023-06-22T08:20:13.498 INFO:tasks.ceph.mds.b.smithi067.stderr: 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0662ac8c31] 2023-06-22T08:20:13.499 INFO:tasks.ceph.mds.b.smithi067.stderr: 12: /lib64/libpthread.so.0(+0x81cf) [0x7f06615351cf] 2023-06-22T08:20:13.499 INFO:tasks.ceph.mds.b.smithi067.stderr: 13: clone() 2023-06-22T08:20:13.499 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.499 INFO:tasks.ceph.mds.b.smithi067.stderr: 0> 2023-06-22T08:20:13.449+0000 7f0656ed4700 -1 *** Caught signal (Aborted) ** 2023-06-22T08:20:13.499 INFO:tasks.ceph.mds.b.smithi067.stderr: in thread 7f0656ed4700 thread_name:ms_dispatch 2023-06-22T08:20:13.499 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:13.499 INFO:tasks.ceph.mds.b.smithi067.stderr: ceph version 18.0.0-4479-g8627af3e (8627af3e0adcb765a3c249fcc209cba9f4873e1b) reef (dev) 2023-06-22T08:20:13.499 INFO:tasks.ceph.mds.b.smithi067.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f066153fce0] 2023-06-22T08:20:13.500 INFO:tasks.ceph.mds.b.smithi067.stderr: 2: gsignal() 2023-06-22T08:20:13.500 INFO:tasks.ceph.mds.b.smithi067.stderr: 3: abort() 2023-06-22T08:20:13.500 INFO:tasks.ceph.mds.b.smithi067.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f066278ce19] 2023-06-22T08:20:13.500 INFO:tasks.ceph.mds.b.smithi067.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7f066278cf85] 2023-06-22T08:20:13.500 INFO:tasks.ceph.mds.b.smithi067.stderr: 6: (CInode::verify_dirfrags()+0x4a4) [0x556681248884] 2023-06-22T08:20:13.500 INFO:tasks.ceph.mds.b.smithi067.stderr: 7: (MDCache::handle_fragment_notify(boost::intrusive_ptr const&)+0x28c) [0x5566810c468c] 2023-06-22T08:20:13.500 INFO:tasks.ceph.mds.b.smithi067.stderr: 8: (MDCache::dispatch(boost::intrusive_ptr const&)+0x3a4) [0x556681108064] 2023-06-22T08:20:13.501 INFO:tasks.ceph.mds.b.smithi067.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x556680f71e33] 2023-06-22T08:20:13.501 INFO:tasks.ceph.mds.b.smithi067.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556680f7244c] 2023-06-22T08:20:13.501 INFO:tasks.ceph.mds.b.smithi067.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556680f5bccf] 2023-06-22T08:20:13.501 INFO:tasks.ceph.mds.b.smithi067.stderr: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f0662a05798] 2023-06-22T08:20:13.501 INFO:tasks.ceph.mds.b.smithi067.stderr: 13: (DispatchQueue::entry()+0x50f) [0x7f0662a0293f] 2023-06-22T08:20:13.501 INFO:tasks.ceph.mds.b.smithi067.stderr: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0662ac8c31] 2023-06-22T08:20:13.501 INFO:tasks.ceph.mds.b.smithi067.stderr: 15: /lib64/libpthread.so.0(+0x81cf) [0x7f06615351cf] 2023-06-22T08:20:13.501 INFO:tasks.ceph.mds.b.smithi067.stderr: 16: clone() 2023-06-22T08:20:13.502 INFO:tasks.ceph.mds.b.smithi067.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-06-22T08:20:13.502 INFO:tasks.ceph.mds.b.smithi067.stderr: 2023-06-22T08:20:14.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:20:14.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:20:18.247 INFO:tasks.ceph.mds.b.smithi067.stderr:daemon-helper: command crashed with signal 6 2023-06-22T08:20:22.737 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-06-22T08:20:29.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~7s 2023-06-22T08:20:36.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~14s 2023-06-22T08:20:43.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~20s 2023-06-22T08:20:45.676 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:20:45.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:20:49.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~27s 2023-06-22T08:20:56.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~34s 2023-06-22T08:21:03.616 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~41s 2023-06-22T08:21:07.972 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T08:21:07.970+0000 7f6838c81700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-22T08:21:07.972 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T08:21:07.970+0000 7f6838c81700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-06-22T08:21:07.972 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T08:21:07.970+0000 7f6838c81700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-06-22T08:21:07.973 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T08:21:07.970+0000 7f6838c81700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-06-22T08:21:07.973 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T08:21:07.970+0000 7f6838c81700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-06-22T08:21:10.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~48s 2023-06-22T08:21:12.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:12.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.759+0000 7f2c7d9ff700 1 Processor -- start 2023-06-22T08:21:12.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.760+0000 7f2c7d9ff700 1 -- start start 2023-06-22T08:21:12.765 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.762+0000 7f2c7d9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2c780cc8b0 0x7f2c780c20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:12.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.762+0000 7f2c7d9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c780c1670 con 0x7f2c7806e150 2023-06-22T08:21:12.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.762+0000 7f2c7d9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c780c2600 con 0x7f2c780ccc80 2023-06-22T08:21:12.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.762+0000 7f2c7d9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c78157030 con 0x7f2c780cc8b0 2023-06-22T08:21:12.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.763+0000 7f2c767fc700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2c7806e150 0x7f2c78156780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58446/0 (socket says 172.21.15.23:58446) 2023-06-22T08:21:12.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.763+0000 7f2c767fc700 1 -- 172.21.15.23:0/3555815083 learned_addr learned my addr 172.21.15.23:0/3555815083 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:12.766 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.763+0000 7f2c777fe700 1 --1- 172.21.15.23:0/3555815083 >> v1:172.21.15.67:6789/0 conn(0x7f2c780ccc80 0x7f2c781504d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:59638/0 (socket says 172.21.15.23:59638) 2023-06-22T08:21:12.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885654062 0 0) 0x7f2c780c2600 con 0x7f2c780ccc80 2023-06-22T08:21:12.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c640035b0 con 0x7f2c780ccc80 2023-06-22T08:21:12.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c76ffd700 1 --2- 172.21.15.23:0/3555815083 >> v2:172.21.15.23:3300/0 conn(0x7f2c780cc8b0 0x7f2c780c20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:12.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012177067 0 0) 0x7f2c780c1670 con 0x7f2c7806e150 2023-06-22T08:21:12.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c780c2600 con 0x7f2c7806e150 2023-06-22T08:21:12.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897771899 0 0) 0x7f2c640035b0 con 0x7f2c780ccc80 2023-06-22T08:21:12.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 >> v2:172.21.15.23:3300/0 conn(0x7f2c780cc8b0 msgr2=0x7f2c780c20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:12.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c75ffb700 1 --2- 172.21.15.23:0/3555815083 >> v2:172.21.15.23:3300/0 conn(0x7f2c780cc8b0 0x7f2c780c20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:12.768 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 >> v1:172.21.15.23:6789/0 conn(0x7f2c7806e150 legacy=0x7f2c78156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:12.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c78157310 con 0x7f2c780ccc80 2023-06-22T08:21:12.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.764+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7f2c700019f0 con 0x7f2c780ccc80 2023-06-22T08:21:12.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.765+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 >> v1:172.21.15.67:6789/0 conn(0x7f2c780ccc80 legacy=0x7f2c781504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:12.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.765+0000 7f2c75ffb700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64003b90 0x7f2c64003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:12.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.765+0000 7f2c75ffb700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c64004850 0x7f2c64004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:12.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.765+0000 7f2c75ffb700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c64005200 0x7f2c6400fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:12.769 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.765+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78157030 con 0x7f2c64004850 2023-06-22T08:21:12.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.765+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c64010470 con 0x7f2c64005200 2023-06-22T08:21:12.770 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.765+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c64010630 con 0x7f2c64003b90 2023-06-22T08:21:12.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.765+0000 7f2c777fe700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c64005200 0x7f2c6400fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:12.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.765+0000 7f2c76ffd700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64003b90 0x7f2c64003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:12.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.766+0000 7f2c767fc700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c64004850 0x7f2c64004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:12.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.766+0000 7f2c777fe700 1 -- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64003b90 msgr2=0x7f2c64003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:12.771 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.766+0000 7f2c777fe700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64003b90 0x7f2c64003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.766+0000 7f2c777fe700 1 -- 172.21.15.23:0/3555815083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c64004850 msgr2=0x7f2c64004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.766+0000 7f2c777fe700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c64004850 0x7f2c64004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.766+0000 7f2c777fe700 1 -- 172.21.15.23:0/3555815083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c640035b0 con 0x7f2c64005200 2023-06-22T08:21:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.766+0000 7f2c777fe700 1 -- 172.21.15.23:0/3555815083 reap_dead start 2023-06-22T08:21:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.766+0000 7f2c76ffd700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64003b90 0x7f2c64003f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.766+0000 7f2c777fe700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c64005200 0x7f2c6400fe20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2c70005930 tx=0x7f2c64005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34e58bf4c0f98d61 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.767+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c700053d0 con 0x7f2c64005200 2023-06-22T08:21:12.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.767+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7000dab0 con 0x7f2c64005200 2023-06-22T08:21:12.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.767+0000 7f2c75ffb700 1 -- 172.21.15.23:0/3555815083 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7000de40 con 0x7f2c64005200 2023-06-22T08:21:12.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.768+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c64005200 msgr2=0x7f2c6400fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:12.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.768+0000 7f2c7d9ff700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c64005200 0x7f2c6400fe20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2c70005930 tx=0x7f2c64005880 comp rx=0 tx=0).stop 2023-06-22T08:21:12.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.768+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/3555815083 shutdown_connections 2023-06-22T08:21:12.773 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.768+0000 7f2c7d9ff700 1 --2- 172.21.15.23:0/3555815083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c64005200 0x7f2c6400fe20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:12.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.768+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/3555815083 >> 172.21.15.23:0/3555815083 conn(0x7f2c7806fb70 msgr2=0x7f2c7806ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:12.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.769+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/3555815083 shutdown_connections 2023-06-22T08:21:12.774 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.770+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/3555815083 wait complete. 2023-06-22T08:21:12.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.773+0000 7f2c7d9ff700 1 Processor -- start 2023-06-22T08:21:12.776 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.774+0000 7f2c7d9ff700 1 -- start start 2023-06-22T08:21:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.775+0000 7f2c7d9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64005200 0x7f2c781545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.775+0000 7f2c7d9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c7814ca30 0x7f2c78155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.775+0000 7f2c7d9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c7814ce80 0x7f2c781f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.775+0000 7f2c7d9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78154f20 con 0x7f2c7814ca30 2023-06-22T08:21:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.775+0000 7f2c7d9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c780cb650 con 0x7f2c7814ce80 2023-06-22T08:21:12.778 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.775+0000 7f2c7d9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c780cc190 con 0x7f2c64005200 2023-06-22T08:21:12.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.776+0000 7f2c767fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c7814ca30 0x7f2c78155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:12.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.776+0000 7f2c767fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c7814ca30 0x7f2c78155810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39518/0 (socket says 172.21.15.23:39518) 2023-06-22T08:21:12.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.776+0000 7f2c767fc700 1 -- 172.21.15.23:0/262194848 learned_addr learned my addr 172.21.15.23:0/262194848 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:12.779 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.776+0000 7f2c76ffd700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64005200 0x7f2c781545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:12.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.776+0000 7f2c76ffd700 1 -- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c7814ce80 msgr2=0x7f2c781f2630 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T08:21:12.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.776+0000 7f2c76ffd700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c7814ce80 0x7f2c781f2630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:12.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.776+0000 7f2c76ffd700 1 -- 172.21.15.23:0/262194848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c7814ca30 msgr2=0x7f2c78155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:12.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.776+0000 7f2c76ffd700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c7814ca30 0x7f2c78155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:12.780 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.776+0000 7f2c76ffd700 1 -- 172.21.15.23:0/262194848 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c78154c60 con 0x7f2c64005200 2023-06-22T08:21:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.777+0000 7f2c76ffd700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64005200 0x7f2c781545d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2c60011bd0 tx=0x7f2c60011ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.777+0000 7f2c6ffff700 1 -- 172.21.15.23:0/262194848 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c60005c30 con 0x7f2c64005200 2023-06-22T08:21:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.777+0000 7f2c6ffff700 1 -- 172.21.15.23:0/262194848 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c60005d90 con 0x7f2c64005200 2023-06-22T08:21:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.777+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/262194848 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c780cb860 con 0x7f2c64005200 2023-06-22T08:21:12.781 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.777+0000 7f2c6ffff700 1 -- 172.21.15.23:0/262194848 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c60004240 con 0x7f2c64005200 2023-06-22T08:21:12.782 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.777+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/262194848 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c780cbe20 con 0x7f2c64005200 2023-06-22T08:21:12.783 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.781+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/262194848 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7804ea20 con 0x7f2c64005200 2023-06-22T08:21:12.784 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.782+0000 7f2c6ffff700 1 -- 172.21.15.23:0/262194848 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2c600043a0 con 0x7f2c64005200 2023-06-22T08:21:12.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.783+0000 7f2c6ffff700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2c5806ad10 0x7f2c5806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:12.786 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.784+0000 7f2c6ffff700 1 -- 172.21.15.23:0/262194848 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f2c60089b70 con 0x7f2c64005200 2023-06-22T08:21:12.787 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.785+0000 7f2c767fc700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2c5806ad10 0x7f2c5806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:12.790 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.788+0000 7f2c6ffff700 1 -- 172.21.15.23:0/262194848 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2c60054500 con 0x7f2c64005200 2023-06-22T08:21:12.796 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:12.794+0000 7f2c767fc700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2c5806ad10 0x7f2c5806d1c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2c68000f60 tx=0x7f2c68005880 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:13.138 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.136+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/262194848 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c781541a0 con 0x7f2c64005200 2023-06-22T08:21:13.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.137+0000 7f2c6ffff700 1 -- 172.21.15.23:0/262194848 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5520 (secure 0 0 0) 0x7f2c60019070 con 0x7f2c64005200 2023-06-22T08:21:13.140 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:13.140 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:08.110246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:active","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:13.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2c5806ad10 msgr2=0x7f2c5806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:13.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2c5806ad10 0x7f2c5806d1c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2c68000f60 tx=0x7f2c68005880 comp rx=0 tx=0).stop 2023-06-22T08:21:13.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64005200 msgr2=0x7f2c781545d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:13.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64005200 0x7f2c781545d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2c60011bd0 tx=0x7f2c60011ee0 comp rx=0 tx=0).stop 2023-06-22T08:21:13.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/262194848 shutdown_connections 2023-06-22T08:21:13.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2c5806ad10 0x7f2c5806d1c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:13.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c7814ce80 0x7f2c781f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:13.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2c64005200 0x7f2c781545d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:13.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 --2- 172.21.15.23:0/262194848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c7814ca30 0x7f2c78155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:13.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/262194848 >> 172.21.15.23:0/262194848 conn(0x7f2c7806fb70 msgr2=0x7f2c7814d750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:13.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/262194848 shutdown_connections 2023-06-22T08:21:13.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.140+0000 7f2c7d9ff700 1 -- 172.21.15.23:0/262194848 wait complete. 2023-06-22T08:21:13.146 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 167 2023-06-22T08:21:13.165 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 5 -> 4 2023-06-22T08:21:13.166 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 4 2023-06-22T08:21:13.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.443+0000 7f363b7ff700 1 Processor -- start 2023-06-22T08:21:13.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.443+0000 7f363b7ff700 1 -- start start 2023-06-22T08:21:13.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.445+0000 7f363b7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3634152ba0 0x7f36341593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:13.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.445+0000 7f363b7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f36340c9260 0x7f36340c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:13.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.445+0000 7f363b7ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3634159930 con 0x7f36340c4650 2023-06-22T08:21:13.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.445+0000 7f363b7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36340c9aa0 con 0x7f36340c9260 2023-06-22T08:21:13.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.445+0000 7f363b7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363406aed0 con 0x7f3634152ba0 2023-06-22T08:21:13.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.445+0000 7f363959b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3634152ba0 0x7f36341593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:13.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.446+0000 7f363959b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3634152ba0 0x7f36341593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:37638/0 (socket says 172.21.15.23:37638) 2023-06-22T08:21:13.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.446+0000 7f363959b700 1 -- 172.21.15.23:0/3405248371 learned_addr learned my addr 172.21.15.23:0/3405248371 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:13.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.446+0000 7f3638d9a700 1 --2- 172.21.15.23:0/3405248371 >> v2:172.21.15.67:3300/0 conn(0x7f36340c9260 0x7f36340c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:13.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.446+0000 7f363959b700 1 -- 172.21.15.23:0/3405248371 >> v2:172.21.15.67:3300/0 conn(0x7f36340c9260 msgr2=0x7f36340c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:13.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.446+0000 7f363959b700 1 --2- 172.21.15.23:0/3405248371 >> v2:172.21.15.67:3300/0 conn(0x7f36340c9260 0x7f36340c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:13.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.446+0000 7f363959b700 1 -- 172.21.15.23:0/3405248371 >> v1:172.21.15.23:6789/0 conn(0x7f36340c4650 legacy=0x7f36340c9ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:21:13.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.446+0000 7f363959b700 1 -- 172.21.15.23:0/3405248371 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363406b1b0 con 0x7f3634152ba0 2023-06-22T08:21:13.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.446+0000 7f3638d9a700 1 --2- 172.21.15.23:0/3405248371 >> v2:172.21.15.67:3300/0 conn(0x7f36340c9260 0x7f36340c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:13.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.447+0000 7f363959b700 1 --2- 172.21.15.23:0/3405248371 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3634152ba0 0x7f36341593f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3624014500 tx=0x7f3624014810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca5b7facd0cb8d6b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.447+0000 7f362bfff700 1 -- 172.21.15.23:0/3405248371 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624004230 con 0x7f3634152ba0 2023-06-22T08:21:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.447+0000 7f362bfff700 1 -- 172.21.15.23:0/3405248371 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3624004390 con 0x7f3634152ba0 2023-06-22T08:21:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.447+0000 7f362bfff700 1 -- 172.21.15.23:0/3405248371 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36240036a0 con 0x7f3634152ba0 2023-06-22T08:21:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.448+0000 7f363b7ff700 1 -- 172.21.15.23:0/3405248371 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3634152ba0 msgr2=0x7f36341593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:13.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.448+0000 7f363b7ff700 1 --2- 172.21.15.23:0/3405248371 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3634152ba0 0x7f36341593f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3624014500 tx=0x7f3624014810 comp rx=0 tx=0).stop 2023-06-22T08:21:13.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.449+0000 7f363b7ff700 1 -- 172.21.15.23:0/3405248371 shutdown_connections 2023-06-22T08:21:13.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.449+0000 7f363b7ff700 1 --2- 172.21.15.23:0/3405248371 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3634152ba0 0x7f36341593f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:13.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.449+0000 7f363b7ff700 1 --2- 172.21.15.23:0/3405248371 >> v2:172.21.15.67:3300/0 conn(0x7f36340c9260 0x7f36340c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:13.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.449+0000 7f363b7ff700 1 -- 172.21.15.23:0/3405248371 >> 172.21.15.23:0/3405248371 conn(0x7f3634065b90 msgr2=0x7f3634065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:13.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.449+0000 7f363b7ff700 1 -- 172.21.15.23:0/3405248371 shutdown_connections 2023-06-22T08:21:13.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.449+0000 7f363b7ff700 1 -- 172.21.15.23:0/3405248371 wait complete. 2023-06-22T08:21:13.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.452+0000 7f363b7ff700 1 Processor -- start 2023-06-22T08:21:13.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.452+0000 7f363b7ff700 1 -- start start 2023-06-22T08:21:13.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.454+0000 7f363b7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36340c4650 0x7f36340b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.454+0000 7f363b7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f36340c9260 0x7f36341ed140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.454+0000 7f363b7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3634152ba0 0x7f36341ed680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.454+0000 7f363b7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36340c8590 con 0x7f36340c4650 2023-06-22T08:21:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.454+0000 7f363b7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36340cd910 con 0x7f3634152ba0 2023-06-22T08:21:13.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.454+0000 7f363b7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363414af20 con 0x7f36340c9260 2023-06-22T08:21:13.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.455+0000 7f363959b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36340c4650 0x7f36340b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:13.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.455+0000 7f363959b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36340c4650 0x7f36340b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39534/0 (socket says 172.21.15.23:39534) 2023-06-22T08:21:13.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.455+0000 7f363959b700 1 -- 172.21.15.23:0/3510297897 learned_addr learned my addr 172.21.15.23:0/3510297897 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:13.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.455+0000 7f3638d9a700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f36340c9260 0x7f36341ed140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:13.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.455+0000 7f3638d9a700 1 -- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3634152ba0 msgr2=0x7f36341ed680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T08:21:13.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.455+0000 7f3638d9a700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3634152ba0 0x7f36341ed680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:13.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.456+0000 7f3638d9a700 1 -- 172.21.15.23:0/3510297897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36340c4650 msgr2=0x7f36340b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:13.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.456+0000 7f3638d9a700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36340c4650 0x7f36340b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:13.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.456+0000 7f3638d9a700 1 -- 172.21.15.23:0/3510297897 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362c009a70 con 0x7f36340c9260 2023-06-22T08:21:13.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.456+0000 7f3638d9a700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f36340c9260 0x7f36341ed140 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f362c00efc0 tx=0x7f362c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:13.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.457+0000 7f3629ffb700 1 -- 172.21.15.23:0/3510297897 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c00cdc0 con 0x7f36340c9260 2023-06-22T08:21:13.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.457+0000 7f3629ffb700 1 -- 172.21.15.23:0/3510297897 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362c00f040 con 0x7f36340c9260 2023-06-22T08:21:13.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.457+0000 7f363b7ff700 1 -- 172.21.15.23:0/3510297897 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362400fd90 con 0x7f36340c9260 2023-06-22T08:21:13.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.457+0000 7f3629ffb700 1 -- 172.21.15.23:0/3510297897 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c0146c0 con 0x7f36340c9260 2023-06-22T08:21:13.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.457+0000 7f363b7ff700 1 -- 172.21.15.23:0/3510297897 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36340ccb70 con 0x7f36340c9260 2023-06-22T08:21:13.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.458+0000 7f363b7ff700 1 -- 172.21.15.23:0/3510297897 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36340cbed0 con 0x7f36340c9260 2023-06-22T08:21:13.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.460+0000 7f3629ffb700 1 -- 172.21.15.23:0/3510297897 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f362c014820 con 0x7f36340c9260 2023-06-22T08:21:13.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.461+0000 7f3629ffb700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f361806ad30 0x7f361806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:13.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.461+0000 7f3629ffb700 1 -- 172.21.15.23:0/3510297897 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f362c08a820 con 0x7f36340c9260 2023-06-22T08:21:13.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.462+0000 7f363959b700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f361806ad30 0x7f361806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:13.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.464+0000 7f363959b700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f361806ad30 0x7f361806d1e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3624014850 tx=0x7f3624019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:13.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.467+0000 7f3629ffb700 1 -- 172.21.15.23:0/3510297897 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f362c055280 con 0x7f36340c9260 2023-06-22T08:21:13.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:13.852+0000 7f363b7ff700 1 -- 172.21.15.23:0/3510297897 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "4"} v 0) v1 -- 0x7f363404ea20 con 0x7f36340c9260 2023-06-22T08:21:14.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.035+0000 7f3629ffb700 1 -- 172.21.15.23:0/3510297897 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "4"}]=0 v168) v1 ==== 105+0+0 (secure 0 0 0) 0x7f362c0588a0 con 0x7f36340c9260 2023-06-22T08:21:14.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.045+0000 7f363b7ff700 1 -- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f361806ad30 msgr2=0x7f361806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:14.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.045+0000 7f363b7ff700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f361806ad30 0x7f361806d1e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3624014850 tx=0x7f3624019000 comp rx=0 tx=0).stop 2023-06-22T08:21:14.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.045+0000 7f363b7ff700 1 -- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f36340c9260 msgr2=0x7f36341ed140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:14.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.045+0000 7f363b7ff700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f36340c9260 0x7f36341ed140 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f362c00efc0 tx=0x7f362c00c3a0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.047+0000 7f363b7ff700 1 -- 172.21.15.23:0/3510297897 shutdown_connections 2023-06-22T08:21:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.047+0000 7f363b7ff700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f361806ad30 0x7f361806d1e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.047+0000 7f363b7ff700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3634152ba0 0x7f36341ed680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.047+0000 7f363b7ff700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36340c4650 0x7f36340b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.047+0000 7f363b7ff700 1 --2- 172.21.15.23:0/3510297897 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f36340c9260 0x7f36341ed140 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.047+0000 7f363b7ff700 1 -- 172.21.15.23:0/3510297897 >> 172.21.15.23:0/3510297897 conn(0x7f3634065b90 msgr2=0x7f36340c9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.047+0000 7f363b7ff700 1 -- 172.21.15.23:0/3510297897 shutdown_connections 2023-06-22T08:21:14.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.047+0000 7f363b7ff700 1 -- 172.21.15.23:0/3510297897 wait complete. 2023-06-22T08:21:14.069 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:21:14.070 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.385+0000 7f8c2d9ff700 1 Processor -- start 2023-06-22T08:21:14.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.386+0000 7f8c2d9ff700 1 -- start start 2023-06-22T08:21:14.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.388+0000 7f8c2d9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8c28152ce0 0x7f8c280cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:14.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.388+0000 7f8c2d9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c280c9fd0 0x7f8c2815c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:14.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.388+0000 7f8c2d9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c28158880 0x7f8c28158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:14.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.388+0000 7f8c2d9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c281530b0 con 0x7f8c28152ce0 2023-06-22T08:21:14.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.388+0000 7f8c2d9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c280cf6f0 con 0x7f8c28158880 2023-06-22T08:21:14.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.388+0000 7f8c2d9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2815c370 con 0x7f8c280c9fd0 2023-06-22T08:21:14.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.388+0000 7f8c277fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c28158880 0x7f8c28158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:14.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.388+0000 7f8c26ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8c28152ce0 0x7f8c280cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:14.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.388+0000 7f8c267fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c280c9fd0 0x7f8c2815c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:14.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.389+0000 7f8c267fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c280c9fd0 0x7f8c2815c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:37656/0 (socket says 172.21.15.23:37656) 2023-06-22T08:21:14.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.389+0000 7f8c277fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c28158880 0x7f8c28158d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:47516/0 (socket says 172.21.15.23:47516) 2023-06-22T08:21:14.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.389+0000 7f8c267fc700 1 -- 172.21.15.23:0/1399973898 learned_addr learned my addr 172.21.15.23:0/1399973898 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:14.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.389+0000 7f8c267fc700 1 -- 172.21.15.23:0/1399973898 >> v2:172.21.15.67:3300/0 conn(0x7f8c28158880 msgr2=0x7f8c28158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:14.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.389+0000 7f8c267fc700 1 --2- 172.21.15.23:0/1399973898 >> v2:172.21.15.67:3300/0 conn(0x7f8c28158880 0x7f8c28158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.389+0000 7f8c267fc700 1 -- 172.21.15.23:0/1399973898 >> v2:172.21.15.23:3300/0 conn(0x7f8c28152ce0 msgr2=0x7f8c280cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:14.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.389+0000 7f8c267fc700 1 --2- 172.21.15.23:0/1399973898 >> v2:172.21.15.23:3300/0 conn(0x7f8c28152ce0 0x7f8c280cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.389+0000 7f8c267fc700 1 -- 172.21.15.23:0/1399973898 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2815bda0 con 0x7f8c280c9fd0 2023-06-22T08:21:14.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.389+0000 7f8c267fc700 1 --2- 172.21.15.23:0/1399973898 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c280c9fd0 0x7f8c2815c800 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8c2808b9e0 tx=0x7f8c1c009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=550f58aaf0d08047 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:14.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.390+0000 7f8c25ffb700 1 -- 172.21.15.23:0/1399973898 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c011830 con 0x7f8c280c9fd0 2023-06-22T08:21:14.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.390+0000 7f8c25ffb700 1 -- 172.21.15.23:0/1399973898 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1c011990 con 0x7f8c280c9fd0 2023-06-22T08:21:14.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.390+0000 7f8c25ffb700 1 -- 172.21.15.23:0/1399973898 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c011d20 con 0x7f8c280c9fd0 2023-06-22T08:21:14.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.390+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/1399973898 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c280c9fd0 msgr2=0x7f8c2815c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:14.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.390+0000 7f8c2d9ff700 1 --2- 172.21.15.23:0/1399973898 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c280c9fd0 0x7f8c2815c800 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8c2808b9e0 tx=0x7f8c1c009bd0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.391+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/1399973898 shutdown_connections 2023-06-22T08:21:14.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.391+0000 7f8c2d9ff700 1 --2- 172.21.15.23:0/1399973898 >> v2:172.21.15.67:3300/0 conn(0x7f8c28158880 0x7f8c28158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.391+0000 7f8c2d9ff700 1 --2- 172.21.15.23:0/1399973898 >> v2:172.21.15.23:3300/0 conn(0x7f8c28152ce0 0x7f8c280cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.391+0000 7f8c2d9ff700 1 --2- 172.21.15.23:0/1399973898 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c280c9fd0 0x7f8c2815c800 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.391+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/1399973898 >> 172.21.15.23:0/1399973898 conn(0x7f8c2806ae30 msgr2=0x7f8c2806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:14.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.391+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/1399973898 shutdown_connections 2023-06-22T08:21:14.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.391+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/1399973898 wait complete. 2023-06-22T08:21:14.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.393+0000 7f8c2d9ff700 1 Processor -- start 2023-06-22T08:21:14.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.393+0000 7f8c2d9ff700 1 -- start start 2023-06-22T08:21:14.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.394+0000 7f8c2d9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280c9fd0 0x7f8c280b2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:14.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.394+0000 7f8c2d9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c28152ce0 0x7f8c280c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:14.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.394+0000 7f8c2d9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c28158880 0x7f8c281f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:14.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.394+0000 7f8c2d9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280cdc40 con 0x7f8c28158880 2023-06-22T08:21:14.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.394+0000 7f8c2d9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c281482c0 con 0x7f8c280c9fd0 2023-06-22T08:21:14.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.394+0000 7f8c2d9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c280cd580 con 0x7f8c28152ce0 2023-06-22T08:21:14.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.394+0000 7f8c26ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280c9fd0 0x7f8c280b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:14.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.394+0000 7f8c26ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280c9fd0 0x7f8c280b2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:47524/0 (socket says 172.21.15.23:47524) 2023-06-22T08:21:14.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.394+0000 7f8c26ffd700 1 -- 172.21.15.23:0/993165413 learned_addr learned my addr 172.21.15.23:0/993165413 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:14.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.394+0000 7f8c277fe700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c28158880 0x7f8c281f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:14.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.395+0000 7f8c267fc700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c28152ce0 0x7f8c280c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:14.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.395+0000 7f8c26ffd700 1 -- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c28152ce0 msgr2=0x7f8c280c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:14.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.395+0000 7f8c26ffd700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c28152ce0 0x7f8c280c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.395+0000 7f8c26ffd700 1 -- 172.21.15.23:0/993165413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c28158880 msgr2=0x7f8c281f1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:14.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.395+0000 7f8c26ffd700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c28158880 0x7f8c281f1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.395+0000 7f8c26ffd700 1 -- 172.21.15.23:0/993165413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c009550 con 0x7f8c280c9fd0 2023-06-22T08:21:14.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.396+0000 7f8c26ffd700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280c9fd0 0x7f8c280b2360 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800e9b0 tx=0x7f8c1800ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:14.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.402+0000 7f8c0ffff700 1 -- 172.21.15.23:0/993165413 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18009710 con 0x7f8c280c9fd0 2023-06-22T08:21:14.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.402+0000 7f8c0ffff700 1 -- 172.21.15.23:0/993165413 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c18017c60 con 0x7f8c280c9fd0 2023-06-22T08:21:14.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.402+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/993165413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c28147fe0 con 0x7f8c280c9fd0 2023-06-22T08:21:14.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.402+0000 7f8c0ffff700 1 -- 172.21.15.23:0/993165413 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800c640 con 0x7f8c280c9fd0 2023-06-22T08:21:14.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.402+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/993165413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c280ccaa0 con 0x7f8c280c9fd0 2023-06-22T08:21:14.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.404+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/993165413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2804ea20 con 0x7f8c280c9fd0 2023-06-22T08:21:14.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.406+0000 7f8c0ffff700 1 -- 172.21.15.23:0/993165413 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8c1800c7a0 con 0x7f8c280c9fd0 2023-06-22T08:21:14.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.407+0000 7f8c0ffff700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c1406adb0 0x7f8c1406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:14.411 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.408+0000 7f8c0ffff700 1 -- 172.21.15.23:0/993165413 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f8c18092300 con 0x7f8c280c9fd0 2023-06-22T08:21:14.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.409+0000 7f8c267fc700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c1406adb0 0x7f8c1406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:14.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.410+0000 7f8c267fc700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c1406adb0 0x7f8c1406d260 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8c28157090 tx=0x7f8c1c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:14.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.411+0000 7f8c0ffff700 1 -- 172.21.15.23:0/993165413 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8c1805cd10 con 0x7f8c280c9fd0 2023-06-22T08:21:14.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.821+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/993165413 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c28158620 con 0x7f8c280c9fd0 2023-06-22T08:21:14.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.822+0000 7f8c0ffff700 1 -- 172.21.15.23:0/993165413 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7f8c18024020 con 0x7f8c280c9fd0 2023-06-22T08:21:14.825 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:14.825 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.825+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c1406adb0 msgr2=0x7f8c1406d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.825+0000 7f8c2d9ff700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c1406adb0 0x7f8c1406d260 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8c28157090 tx=0x7f8c1c008030 comp rx=0 tx=0).stop 2023-06-22T08:21:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.825+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280c9fd0 msgr2=0x7f8c280b2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.825+0000 7f8c2d9ff700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280c9fd0 0x7f8c280b2360 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800e9b0 tx=0x7f8c1800ecc0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.826+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/993165413 shutdown_connections 2023-06-22T08:21:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.826+0000 7f8c2d9ff700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c28158880 0x7f8c281f1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.826+0000 7f8c2d9ff700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8c1406adb0 0x7f8c1406d260 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.826+0000 7f8c2d9ff700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c280c9fd0 0x7f8c280b2360 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.826+0000 7f8c2d9ff700 1 --2- 172.21.15.23:0/993165413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8c28152ce0 0x7f8c280c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.826+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/993165413 >> 172.21.15.23:0/993165413 conn(0x7f8c2806ae30 msgr2=0x7f8c280cb550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.826+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/993165413 shutdown_connections 2023-06-22T08:21:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:14.826+0000 7f8c2d9ff700 1 -- 172.21.15.23:0/993165413 wait complete. 2023-06-22T08:21:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:14.865 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-06-22T08:21:14.866 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 169, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 169, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 }, "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 36 }, "gid_6318": { "addr": "172.21.15.23:6834/904774922", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 904774922, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 904774922, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 151, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:stopping", "state_seq": 39 }, "gid_6616": { "addr": "172.21.15.67:6837/1855579717", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1855579717, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1855579717, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6616, "incarnation": 162, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 4, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:21:14.036557+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6616, "mds_1": 5707, "mds_2": 6313, "mds_3": 5995, "mds_4": 6318 } } } ], "standbys": [] } 2023-06-22T08:21:15.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:21:15.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:21:16.867 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:17.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.093+0000 7f49155ff700 1 Processor -- start 2023-06-22T08:21:17.097 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.093+0000 7f49155ff700 1 -- start start 2023-06-22T08:21:17.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.096+0000 7f49155ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4910071c20 0x7f49100c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:17.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.096+0000 7f49155ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491015bf50 con 0x7f49100c4a30 2023-06-22T08:21:17.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.096+0000 7f49155ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49100c9550 con 0x7f4910071850 2023-06-22T08:21:17.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.096+0000 7f49155ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49100c4eb0 con 0x7f4910071c20 2023-06-22T08:21:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.096+0000 7f490effd700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4910071850 0x7f4910152750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:45594/0 (socket says 172.21.15.23:45594) 2023-06-22T08:21:17.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.096+0000 7f490effd700 1 -- 172.21.15.23:0/3775896696 learned_addr learned my addr 172.21.15.23:0/3775896696 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:17.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.096+0000 7f490e7fc700 1 --2- 172.21.15.23:0/3775896696 >> v2:172.21.15.67:3300/0 conn(0x7f4910071c20 0x7f49100c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:17.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.097+0000 7f490e7fc700 1 -- 172.21.15.23:0/3775896696 >> v1:172.21.15.67:6789/0 conn(0x7f4910071850 legacy=0x7f4910152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:17.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.097+0000 7f490e7fc700 1 -- 172.21.15.23:0/3775896696 >> v1:172.21.15.23:6789/0 conn(0x7f49100c4a30 legacy=0x7f491015c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:17.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.097+0000 7f490e7fc700 1 -- 172.21.15.23:0/3775896696 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4910158c00 con 0x7f4910071c20 2023-06-22T08:21:17.102 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.100+0000 7f490e7fc700 1 --2- 172.21.15.23:0/3775896696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4910071c20 0x7f49100c9880 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4904009840 tx=0x7f4904009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7c8fc657943af8c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:17.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.100+0000 7f490dffb700 1 -- 172.21.15.23:0/3775896696 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904011830 con 0x7f4910071c20 2023-06-22T08:21:17.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.100+0000 7f490dffb700 1 -- 172.21.15.23:0/3775896696 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4904011990 con 0x7f4910071c20 2023-06-22T08:21:17.103 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.100+0000 7f490dffb700 1 -- 172.21.15.23:0/3775896696 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904011d20 con 0x7f4910071c20 2023-06-22T08:21:17.105 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.103+0000 7f49155ff700 1 -- 172.21.15.23:0/3775896696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4910071c20 msgr2=0x7f49100c9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:17.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.103+0000 7f49155ff700 1 --2- 172.21.15.23:0/3775896696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4910071c20 0x7f49100c9880 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4904009840 tx=0x7f4904009b50 comp rx=0 tx=0).stop 2023-06-22T08:21:17.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.103+0000 7f49155ff700 1 -- 172.21.15.23:0/3775896696 shutdown_connections 2023-06-22T08:21:17.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.103+0000 7f49155ff700 1 --2- 172.21.15.23:0/3775896696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4910071c20 0x7f49100c9880 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:17.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.103+0000 7f49155ff700 1 -- 172.21.15.23:0/3775896696 >> 172.21.15.23:0/3775896696 conn(0x7f491006b8b0 msgr2=0x7f491006bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:17.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.106+0000 7f49155ff700 1 -- 172.21.15.23:0/3775896696 shutdown_connections 2023-06-22T08:21:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.106+0000 7f49155ff700 1 -- 172.21.15.23:0/3775896696 wait complete. 2023-06-22T08:21:17.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.109+0000 7f49155ff700 1 Processor -- start 2023-06-22T08:21:17.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.109+0000 7f49155ff700 1 -- start start 2023-06-22T08:21:17.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f49155ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4910071850 0x7f49101ef220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:17.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f49155ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4910071c20 0x7f491015b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:17.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f49155ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49100c4a30 0x7f4910152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:17.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f49155ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4910156740 con 0x7f4910071c20 2023-06-22T08:21:17.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f49155ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4910156c70 con 0x7f49100c4a30 2023-06-22T08:21:17.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f49155ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4910156990 con 0x7f4910071850 2023-06-22T08:21:17.114 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f490f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49100c4a30 0x7f4910152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:17.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f490e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4910071c20 0x7f491015b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:17.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f490effd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4910071850 0x7f49101ef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:17.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f490f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49100c4a30 0x7f4910152520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:58528/0 (socket says 172.21.15.23:58528) 2023-06-22T08:21:17.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.111+0000 7f490f7fe700 1 -- 172.21.15.23:0/189946007 learned_addr learned my addr 172.21.15.23:0/189946007 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:17.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.112+0000 7f490e7fc700 1 -- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4910071850 msgr2=0x7f49101ef220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:17.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.112+0000 7f490e7fc700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4910071850 0x7f49101ef220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:17.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.112+0000 7f490e7fc700 1 -- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49100c4a30 msgr2=0x7f4910152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:17.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.112+0000 7f490e7fc700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49100c4a30 0x7f4910152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:17.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.112+0000 7f490e7fc700 1 -- 172.21.15.23:0/189946007 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49040094a0 con 0x7f4910071c20 2023-06-22T08:21:17.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.112+0000 7f490f7fe700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49100c4a30 0x7f4910152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:17.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.112+0000 7f490e7fc700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4910071c20 0x7f491015b790 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4904016000 tx=0x7f4904004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:17.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.112+0000 7f48f77fe700 1 -- 172.21.15.23:0/189946007 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904004510 con 0x7f4910071c20 2023-06-22T08:21:17.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.113+0000 7f48f77fe700 1 -- 172.21.15.23:0/189946007 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490401f070 con 0x7f4910071c20 2023-06-22T08:21:17.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.113+0000 7f48f77fe700 1 -- 172.21.15.23:0/189946007 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49040038a0 con 0x7f4910071c20 2023-06-22T08:21:17.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.113+0000 7f49155ff700 1 -- 172.21.15.23:0/189946007 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49101ee740 con 0x7f4910071c20 2023-06-22T08:21:17.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.113+0000 7f49155ff700 1 -- 172.21.15.23:0/189946007 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49101eed00 con 0x7f4910071c20 2023-06-22T08:21:17.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.114+0000 7f49155ff700 1 -- 172.21.15.23:0/189946007 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491004ea20 con 0x7f4910071c20 2023-06-22T08:21:17.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.116+0000 7f48f77fe700 1 -- 172.21.15.23:0/189946007 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4904003a00 con 0x7f4910071c20 2023-06-22T08:21:17.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.118+0000 7f48f77fe700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48fc06adb0 0x7f48fc06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:17.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.118+0000 7f48f77fe700 1 -- 172.21.15.23:0/189946007 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f4904021070 con 0x7f4910071c20 2023-06-22T08:21:17.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.119+0000 7f490effd700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48fc06adb0 0x7f48fc06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:17.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.119+0000 7f490effd700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48fc06adb0 0x7f48fc06d260 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4900005910 tx=0x7f4900009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:17.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.123+0000 7f48f77fe700 1 -- 172.21.15.23:0/189946007 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4904055ca0 con 0x7f4910071c20 2023-06-22T08:21:17.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~55s 2023-06-22T08:21:17.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.454+0000 7f49155ff700 1 -- 172.21.15.23:0/189946007 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49100c3bf0 con 0x7f4910071c20 2023-06-22T08:21:17.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.455+0000 7f48f77fe700 1 -- 172.21.15.23:0/189946007 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7f4904015030 con 0x7f4910071c20 2023-06-22T08:21:17.458 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:17.458 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:17.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.458+0000 7f49155ff700 1 -- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48fc06adb0 msgr2=0x7f48fc06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:17.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.458+0000 7f49155ff700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48fc06adb0 0x7f48fc06d260 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4900005910 tx=0x7f4900009040 comp rx=0 tx=0).stop 2023-06-22T08:21:17.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.458+0000 7f49155ff700 1 -- 172.21.15.23:0/189946007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4910071c20 msgr2=0x7f491015b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:17.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.458+0000 7f49155ff700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4910071c20 0x7f491015b790 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4904016000 tx=0x7f4904004750 comp rx=0 tx=0).stop 2023-06-22T08:21:17.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.459+0000 7f49155ff700 1 -- 172.21.15.23:0/189946007 shutdown_connections 2023-06-22T08:21:17.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.459+0000 7f49155ff700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f48fc06adb0 0x7f48fc06d260 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:17.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.459+0000 7f49155ff700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49100c4a30 0x7f4910152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:17.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.459+0000 7f49155ff700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4910071850 0x7f49101ef220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:17.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.459+0000 7f49155ff700 1 --2- 172.21.15.23:0/189946007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4910071c20 0x7f491015b790 secure :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4904016000 tx=0x7f4904004750 comp rx=0 tx=0).stop 2023-06-22T08:21:17.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.459+0000 7f49155ff700 1 -- 172.21.15.23:0/189946007 >> 172.21.15.23:0/189946007 conn(0x7f491006b8b0 msgr2=0x7f491014ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:17.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.459+0000 7f49155ff700 1 -- 172.21.15.23:0/189946007 shutdown_connections 2023-06-22T08:21:17.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:17.459+0000 7f49155ff700 1 -- 172.21.15.23:0/189946007 wait complete. 2023-06-22T08:21:17.464 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:17.490 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:19.491 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:19.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.852+0000 7f81a71ff700 1 Processor -- start 2023-06-22T08:21:19.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.852+0000 7f81a71ff700 1 -- start start 2023-06-22T08:21:19.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.854+0000 7f81a71ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81a0152d70 0x7f81a00c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:19.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.854+0000 7f81a71ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f81a00c45f0 0x7f81a00c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:19.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.854+0000 7f81a71ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a0159790 con 0x7f81a0159330 2023-06-22T08:21:19.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.854+0000 7f81a71ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a00c95e0 con 0x7f81a00c45f0 2023-06-22T08:21:19.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.854+0000 7f81a71ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a015bea0 con 0x7f81a0152d70 2023-06-22T08:21:19.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.854+0000 7f81a4f9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81a0152d70 0x7f81a00c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:19.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.854+0000 7f819ffff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f81a0159330 0x7f81a00cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:45610/0 (socket says 172.21.15.23:45610) 2023-06-22T08:21:19.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.854+0000 7f819ffff700 1 -- 172.21.15.23:0/3514143599 learned_addr learned my addr 172.21.15.23:0/3514143599 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:19.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.854+0000 7f81a579c700 1 --2- 172.21.15.23:0/3514143599 >> v2:172.21.15.23:3300/0 conn(0x7f81a00c45f0 0x7f81a00c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:19.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.854+0000 7f81a4f9b700 1 -- 172.21.15.23:0/3514143599 >> v2:172.21.15.23:3300/0 conn(0x7f81a00c45f0 msgr2=0x7f81a00c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:19.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.855+0000 7f81a4f9b700 1 --2- 172.21.15.23:0/3514143599 >> v2:172.21.15.23:3300/0 conn(0x7f81a00c45f0 0x7f81a00c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:19.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.855+0000 7f81a4f9b700 1 -- 172.21.15.23:0/3514143599 >> v1:172.21.15.67:6789/0 conn(0x7f81a0159330 legacy=0x7f81a00cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:19.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.855+0000 7f81a4f9b700 1 -- 172.21.15.23:0/3514143599 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a015c160 con 0x7f81a0152d70 2023-06-22T08:21:19.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.855+0000 7f819f7fe700 1 -- 172.21.15.23:0/3514143599 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142898040 0 0) 0x7f81a0159790 con 0x7f81a0159330 2023-06-22T08:21:19.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.855+0000 7f81a4f9b700 1 --2- 172.21.15.23:0/3514143599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a0152d70 0x7f81a00c97d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f81a00c8ef0 tx=0x7f8194009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f47545bcac996ae0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:19.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.855+0000 7f819f7fe700 1 -- 172.21.15.23:0/3514143599 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194011830 con 0x7f81a0152d70 2023-06-22T08:21:19.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.856+0000 7f819f7fe700 1 -- 172.21.15.23:0/3514143599 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8194011990 con 0x7f81a0152d70 2023-06-22T08:21:19.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.856+0000 7f819f7fe700 1 -- 172.21.15.23:0/3514143599 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194011d20 con 0x7f81a0152d70 2023-06-22T08:21:19.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.856+0000 7f81a71ff700 1 -- 172.21.15.23:0/3514143599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a0152d70 msgr2=0x7f81a00c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:19.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.856+0000 7f81a71ff700 1 --2- 172.21.15.23:0/3514143599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a0152d70 0x7f81a00c97d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f81a00c8ef0 tx=0x7f8194009b20 comp rx=0 tx=0).stop 2023-06-22T08:21:19.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.857+0000 7f81a71ff700 1 -- 172.21.15.23:0/3514143599 shutdown_connections 2023-06-22T08:21:19.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.857+0000 7f81a71ff700 1 --2- 172.21.15.23:0/3514143599 >> v2:172.21.15.23:3300/0 conn(0x7f81a00c45f0 0x7f81a00c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:19.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.857+0000 7f81a71ff700 1 --2- 172.21.15.23:0/3514143599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a0152d70 0x7f81a00c97d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:19.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.857+0000 7f81a71ff700 1 -- 172.21.15.23:0/3514143599 >> 172.21.15.23:0/3514143599 conn(0x7f81a006ae30 msgr2=0x7f81a006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:19.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.857+0000 7f81a71ff700 1 -- 172.21.15.23:0/3514143599 shutdown_connections 2023-06-22T08:21:19.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.857+0000 7f81a71ff700 1 -- 172.21.15.23:0/3514143599 wait complete. 2023-06-22T08:21:19.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.859+0000 7f81a71ff700 1 Processor -- start 2023-06-22T08:21:19.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.860+0000 7f81a71ff700 1 -- start start 2023-06-22T08:21:19.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.861+0000 7f81a71ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00c45f0 0x7f81a01e51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:19.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.861+0000 7f81a71ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f81a0152d70 0x7f81a014bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:19.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.861+0000 7f81a71ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81a0159330 0x7f81a014c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:19.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.861+0000 7f81a71ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a01ef770 con 0x7f81a0159330 2023-06-22T08:21:19.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.861+0000 7f81a71ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a00b2a50 con 0x7f81a00c45f0 2023-06-22T08:21:19.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.861+0000 7f81a71ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a0157d20 con 0x7f81a0152d70 2023-06-22T08:21:19.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.861+0000 7f819ffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f81a0152d70 0x7f81a014bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:19.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.861+0000 7f81a4f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00c45f0 0x7f81a01e51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:19.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.861+0000 7f81a4f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00c45f0 0x7f81a01e51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:58548/0 (socket says 172.21.15.23:58548) 2023-06-22T08:21:19.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.861+0000 7f81a4f9b700 1 -- 172.21.15.23:0/1730776100 learned_addr learned my addr 172.21.15.23:0/1730776100 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:19.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.862+0000 7f81a579c700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81a0159330 0x7f81a014c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:19.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.862+0000 7f819ffff700 1 -- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00c45f0 msgr2=0x7f81a01e51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:19.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.862+0000 7f819ffff700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00c45f0 0x7f81a01e51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:19.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.862+0000 7f819ffff700 1 -- 172.21.15.23:0/1730776100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81a0159330 msgr2=0x7f81a014c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:19.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.862+0000 7f819ffff700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81a0159330 0x7f81a014c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:19.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.862+0000 7f819ffff700 1 -- 172.21.15.23:0/1730776100 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81940094a0 con 0x7f81a0152d70 2023-06-22T08:21:19.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.862+0000 7f81a4f9b700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00c45f0 0x7f81a01e51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:19.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.862+0000 7f819ffff700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f81a0152d70 0x7f81a014bcb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f818c00c410 tx=0x7f818c00c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:19.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.863+0000 7f819d7fa700 1 -- 172.21.15.23:0/1730776100 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c004c80 con 0x7f81a0152d70 2023-06-22T08:21:19.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.863+0000 7f819d7fa700 1 -- 172.21.15.23:0/1730776100 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c004de0 con 0x7f81a0152d70 2023-06-22T08:21:19.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.863+0000 7f81a71ff700 1 -- 172.21.15.23:0/1730776100 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a01eebf0 con 0x7f81a0152d70 2023-06-22T08:21:19.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.863+0000 7f819d7fa700 1 -- 172.21.15.23:0/1730776100 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c007310 con 0x7f81a0152d70 2023-06-22T08:21:19.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.863+0000 7f81a71ff700 1 -- 172.21.15.23:0/1730776100 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a01ef1b0 con 0x7f81a0152d70 2023-06-22T08:21:19.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.864+0000 7f81a71ff700 1 -- 172.21.15.23:0/1730776100 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a01528d0 con 0x7f81a0152d70 2023-06-22T08:21:19.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.866+0000 7f819d7fa700 1 -- 172.21.15.23:0/1730776100 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f818c007470 con 0x7f81a0152d70 2023-06-22T08:21:19.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.867+0000 7f819d7fa700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f818406ace0 0x7f818406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:19.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.868+0000 7f819d7fa700 1 -- 172.21.15.23:0/1730776100 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f818c089780 con 0x7f81a0152d70 2023-06-22T08:21:19.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.868+0000 7f81a4f9b700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f818406ace0 0x7f818406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:19.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.869+0000 7f81a4f9b700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f818406ace0 0x7f818406d190 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8194000c00 tx=0x7f8194008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:19.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:19.874+0000 7f819d7fa700 1 -- 172.21.15.23:0/1730776100 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f818c054080 con 0x7f81a0152d70 2023-06-22T08:21:20.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.272+0000 7f81a71ff700 1 -- 172.21.15.23:0/1730776100 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81a008bf10 con 0x7f81a0152d70 2023-06-22T08:21:20.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.276+0000 7f819d7fa700 1 -- 172.21.15.23:0/1730776100 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7f818c0576a0 con 0x7f81a0152d70 2023-06-22T08:21:20.279 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:20.280 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:20.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.282+0000 7f81a71ff700 1 -- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f818406ace0 msgr2=0x7f818406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:20.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.282+0000 7f81a71ff700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f818406ace0 0x7f818406d190 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8194000c00 tx=0x7f8194008030 comp rx=0 tx=0).stop 2023-06-22T08:21:20.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.282+0000 7f81a71ff700 1 -- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f81a0152d70 msgr2=0x7f81a014bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:20.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.282+0000 7f81a71ff700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f81a0152d70 0x7f81a014bcb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f818c00c410 tx=0x7f818c00c720 comp rx=0 tx=0).stop 2023-06-22T08:21:20.285 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.283+0000 7f81a71ff700 1 -- 172.21.15.23:0/1730776100 shutdown_connections 2023-06-22T08:21:20.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.283+0000 7f81a71ff700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81a0159330 0x7f81a014c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:20.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.283+0000 7f81a71ff700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f818406ace0 0x7f818406d190 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:20.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.283+0000 7f81a71ff700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81a00c45f0 0x7f81a01e51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:20.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.283+0000 7f81a71ff700 1 --2- 172.21.15.23:0/1730776100 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f81a0152d70 0x7f81a014bcb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:20.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.283+0000 7f81a71ff700 1 -- 172.21.15.23:0/1730776100 >> 172.21.15.23:0/1730776100 conn(0x7f81a006ae30 msgr2=0x7f81a00cdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:20.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.283+0000 7f81a71ff700 1 -- 172.21.15.23:0/1730776100 shutdown_connections 2023-06-22T08:21:20.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:20.283+0000 7f81a71ff700 1 -- 172.21.15.23:0/1730776100 wait complete. 2023-06-22T08:21:20.287 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:20.330 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:22.331 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:22.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.614+0000 7fac8abff700 1 Processor -- start 2023-06-22T08:21:22.617 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.614+0000 7fac8abff700 1 -- start start 2023-06-22T08:21:22.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.615+0000 7fac8abff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac84152dd0 0x7fac840aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:22.618 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.615+0000 7fac8abff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fac840c8bd0 0x7fac8406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:22.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.615+0000 7fac8abff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac840c7400 con 0x7fac8406b500 2023-06-22T08:21:22.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.615+0000 7fac8abff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac840c81c0 con 0x7fac840c8bd0 2023-06-22T08:21:22.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.616+0000 7fac8abff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac840c84a0 con 0x7fac84152dd0 2023-06-22T08:21:22.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.616+0000 7fac8899b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac84152dd0 0x7fac840aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:22.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.616+0000 7fac8899b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac84152dd0 0x7fac840aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:47236/0 (socket says 172.21.15.23:47236) 2023-06-22T08:21:22.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.616+0000 7fac8899b700 1 -- 172.21.15.23:0/760801583 learned_addr learned my addr 172.21.15.23:0/760801583 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.616+0000 7fac83fff700 1 --2- 172.21.15.23:0/760801583 >> v2:172.21.15.23:3300/0 conn(0x7fac840c8bd0 0x7fac8406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.616+0000 7fac8899b700 1 -- 172.21.15.23:0/760801583 >> v2:172.21.15.23:3300/0 conn(0x7fac840c8bd0 msgr2=0x7fac8406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.616+0000 7fac8899b700 1 --2- 172.21.15.23:0/760801583 >> v2:172.21.15.23:3300/0 conn(0x7fac840c8bd0 0x7fac8406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.616+0000 7fac8899b700 1 -- 172.21.15.23:0/760801583 >> v1:172.21.15.23:6789/0 conn(0x7fac8406b500 legacy=0x7fac840cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.616+0000 7fac8899b700 1 -- 172.21.15.23:0/760801583 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac840cd370 con 0x7fac84152dd0 2023-06-22T08:21:22.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.617+0000 7fac8899b700 1 --2- 172.21.15.23:0/760801583 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac84152dd0 0x7fac840aedc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fac78006fd0 tx=0x7fac78009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2972fb6dd21e8d0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:22.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.617+0000 7fac837fe700 1 -- 172.21.15.23:0/760801583 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78005e30 con 0x7fac84152dd0 2023-06-22T08:21:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.617+0000 7fac837fe700 1 -- 172.21.15.23:0/760801583 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac78004030 con 0x7fac84152dd0 2023-06-22T08:21:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.617+0000 7fac837fe700 1 -- 172.21.15.23:0/760801583 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac780043e0 con 0x7fac84152dd0 2023-06-22T08:21:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.618+0000 7fac8abff700 1 -- 172.21.15.23:0/760801583 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac84152dd0 msgr2=0x7fac840aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.618+0000 7fac8abff700 1 --2- 172.21.15.23:0/760801583 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac84152dd0 0x7fac840aedc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fac78006fd0 tx=0x7fac78009bd0 comp rx=0 tx=0).stop 2023-06-22T08:21:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.618+0000 7fac8abff700 1 -- 172.21.15.23:0/760801583 shutdown_connections 2023-06-22T08:21:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.618+0000 7fac8abff700 1 --2- 172.21.15.23:0/760801583 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac84152dd0 0x7fac840aedc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.618+0000 7fac8abff700 1 --2- 172.21.15.23:0/760801583 >> v2:172.21.15.23:3300/0 conn(0x7fac840c8bd0 0x7fac8406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.618+0000 7fac8abff700 1 -- 172.21.15.23:0/760801583 >> 172.21.15.23:0/760801583 conn(0x7fac8406fb70 msgr2=0x7fac8406ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.618+0000 7fac8abff700 1 -- 172.21.15.23:0/760801583 shutdown_connections 2023-06-22T08:21:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.618+0000 7fac8abff700 1 -- 172.21.15.23:0/760801583 wait complete. 2023-06-22T08:21:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.621+0000 7fac8abff700 1 Processor -- start 2023-06-22T08:21:22.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.621+0000 7fac8abff700 1 -- start start 2023-06-22T08:21:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8abff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac8406b500 0x7fac840cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8abff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac840c8bd0 0x7fac840af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8abff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac84152dd0 0x7fac8414fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8abff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac84155ae0 con 0x7fac840c8bd0 2023-06-22T08:21:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8abff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac840b0da0 con 0x7fac84152dd0 2023-06-22T08:21:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8abff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac840b1f20 con 0x7fac8406b500 2023-06-22T08:21:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8899b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac8406b500 0x7fac840cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8899b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac8406b500 0x7fac840cbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:47244/0 (socket says 172.21.15.23:47244) 2023-06-22T08:21:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8899b700 1 -- 172.21.15.23:0/4112702812 learned_addr learned my addr 172.21.15.23:0/4112702812 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8919c700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac84152dd0 0x7fac8414fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac83fff700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac840c8bd0 0x7fac840af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8899b700 1 -- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac84152dd0 msgr2=0x7fac8414fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8899b700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac84152dd0 0x7fac8414fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8899b700 1 -- 172.21.15.23:0/4112702812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac840c8bd0 msgr2=0x7fac840af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8899b700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac840c8bd0 0x7fac840af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.623+0000 7fac8899b700 1 -- 172.21.15.23:0/4112702812 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac78009500 con 0x7fac8406b500 2023-06-22T08:21:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.624+0000 7fac8899b700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac8406b500 0x7fac840cbd10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac78009c10 tx=0x7fac78004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.624+0000 7fac817fa700 1 -- 172.21.15.23:0/4112702812 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78003d20 con 0x7fac8406b500 2023-06-22T08:21:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.624+0000 7fac817fa700 1 -- 172.21.15.23:0/4112702812 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac78003e80 con 0x7fac8406b500 2023-06-22T08:21:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.624+0000 7fac817fa700 1 -- 172.21.15.23:0/4112702812 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7801d630 con 0x7fac8406b500 2023-06-22T08:21:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.624+0000 7fac8abff700 1 -- 172.21.15.23:0/4112702812 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac840b2200 con 0x7fac8406b500 2023-06-22T08:21:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.624+0000 7fac8abff700 1 -- 172.21.15.23:0/4112702812 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac841ee790 con 0x7fac8406b500 2023-06-22T08:21:22.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.626+0000 7fac8abff700 1 -- 172.21.15.23:0/4112702812 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8415c690 con 0x7fac8406b500 2023-06-22T08:21:22.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.627+0000 7fac817fa700 1 -- 172.21.15.23:0/4112702812 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fac7801d790 con 0x7fac8406b500 2023-06-22T08:21:22.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.628+0000 7fac817fa700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac6806ad10 0x7fac6806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.628+0000 7fac817fa700 1 -- 172.21.15.23:0/4112702812 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7871+0+0 (secure 0 0 0) 0x7fac7808aeb0 con 0x7fac8406b500 2023-06-22T08:21:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.628+0000 7fac83fff700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac6806ad10 0x7fac6806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.629+0000 7fac83fff700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac6806ad10 0x7fac6806d1c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fac840cc800 tx=0x7fac70008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:22.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.632+0000 7fac817fa700 1 -- 172.21.15.23:0/4112702812 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fac78055840 con 0x7fac8406b500 2023-06-22T08:21:22.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.988+0000 7fac8abff700 1 -- 172.21.15.23:0/4112702812 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac8404ea20 con 0x7fac8406b500 2023-06-22T08:21:22.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.989+0000 7fac817fa700 1 -- 172.21.15.23:0/4112702812 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7fac78058e60 con 0x7fac8406b500 2023-06-22T08:21:22.992 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:22.992 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 -- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac6806ad10 msgr2=0x7fac6806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac6806ad10 0x7fac6806d1c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fac840cc800 tx=0x7fac70008040 comp rx=0 tx=0).stop 2023-06-22T08:21:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 -- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac8406b500 msgr2=0x7fac840cbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac8406b500 0x7fac840cbd10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fac78009c10 tx=0x7fac78004750 comp rx=0 tx=0).stop 2023-06-22T08:21:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 -- 172.21.15.23:0/4112702812 shutdown_connections 2023-06-22T08:21:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fac6806ad10 0x7fac6806d1c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac84152dd0 0x7fac8414fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:22.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fac8406b500 0x7fac840cbd10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:22.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 --2- 172.21.15.23:0/4112702812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac840c8bd0 0x7fac840af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:22.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 -- 172.21.15.23:0/4112702812 >> 172.21.15.23:0/4112702812 conn(0x7fac8406fb70 msgr2=0x7fac84159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:22.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 -- 172.21.15.23:0/4112702812 shutdown_connections 2023-06-22T08:21:22.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:22.992+0000 7fac8abff700 1 -- 172.21.15.23:0/4112702812 wait complete. 2023-06-22T08:21:22.997 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:23.022 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:23.952 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-hardlink.sh... 2023-06-22T08:21:23.952 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-hardlink.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-hardlink.sh 2023-06-22T08:21:24.013 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:21:24.013 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir 1 2 2023-06-22T08:21:24.018 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo asdf 2023-06-22T08:21:24.020 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo asdf 2023-06-22T08:21:24.020 INFO:tasks.workunit.client.0.smithi023.stderr:+ ln 1/file1 2/file1 2023-06-22T08:21:24.024 INFO:tasks.workunit.client.0.smithi023.stderr:+ ln 1/file2 2/file2 2023-06-22T08:21:24.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir 2/.snap/s1 2023-06-22T08:21:24.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~61s 2023-06-22T08:21:24.061 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo qwer 2023-06-22T08:21:24.075 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep asdf 2/.snap/s1/file1 2023-06-22T08:21:24.165 INFO:tasks.workunit.client.0.smithi023.stdout:asdf 2023-06-22T08:21:24.166 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -f 1/file2 2023-06-22T08:21:24.225 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep asdf 2/.snap/s1/file2 2023-06-22T08:21:24.281 INFO:tasks.workunit.client.0.smithi023.stdout:asdf 2023-06-22T08:21:24.281 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -f 2/file2 2023-06-22T08:21:24.296 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep asdf 2/.snap/s1/file2 2023-06-22T08:21:24.298 INFO:tasks.workunit.client.0.smithi023.stdout:asdf 2023-06-22T08:21:24.299 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir 2/.snap/s1 2023-06-22T08:21:24.313 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -rf 1 2 2023-06-22T08:21:24.368 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:21:24.370 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:21:24.370 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:21:24.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:21:24.456 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-intodir.sh... 2023-06-22T08:21:24.457 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-intodir.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-intodir.sh 2023-06-22T08:21:24.554 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir foo 2023-06-22T08:21:24.559 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir foo/.snap/one 2023-06-22T08:21:24.618 INFO:tasks.workunit.client.0.smithi023.stderr:+ touch bar 2023-06-22T08:21:24.629 INFO:tasks.workunit.client.0.smithi023.stderr:+ mv bar foo 2023-06-22T08:21:24.640 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:21:25.024 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:25.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.445+0000 7f7a40bff700 1 Processor -- start 2023-06-22T08:21:25.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.445+0000 7f7a40bff700 1 -- start start 2023-06-22T08:21:25.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.447+0000 7f7a40bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a3c15d9b0 0x7f7a3c06d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:25.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.447+0000 7f7a40bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c06a280 0x7f7a3c0c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:25.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.447+0000 7f7a40bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c0c4d70 con 0x7f7a3c069eb0 2023-06-22T08:21:25.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.447+0000 7f7a40bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3c0703c0 con 0x7f7a3c15d9b0 2023-06-22T08:21:25.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.447+0000 7f7a40bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c0c5610 con 0x7f7a3c06a280 2023-06-22T08:21:25.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.450+0000 7f7a3ad9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c06a280 0x7f7a3c0c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:25.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.450+0000 7f7a3ad9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c06a280 0x7f7a3c0c5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:47254/0 (socket says 172.21.15.23:47254) 2023-06-22T08:21:25.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.450+0000 7f7a3ad9d700 1 -- 172.21.15.23:0/4252981393 learned_addr learned my addr 172.21.15.23:0/4252981393 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:25.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.450+0000 7f7a3ad9d700 1 -- 172.21.15.23:0/4252981393 >> v2:172.21.15.67:3300/0 conn(0x7f7a3c15d9b0 msgr2=0x7f7a3c06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:25.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.450+0000 7f7a3ad9d700 1 --2- 172.21.15.23:0/4252981393 >> v2:172.21.15.67:3300/0 conn(0x7f7a3c15d9b0 0x7f7a3c06d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:25.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.450+0000 7f7a3ad9d700 1 -- 172.21.15.23:0/4252981393 >> v1:172.21.15.67:6789/0 conn(0x7f7a3c069eb0 legacy=0x7f7a3c06fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:25.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.450+0000 7f7a3ad9d700 1 -- 172.21.15.23:0/4252981393 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c0c5fe0 con 0x7f7a3c06a280 2023-06-22T08:21:25.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.451+0000 7f7a3ad9d700 1 --2- 172.21.15.23:0/4252981393 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c06a280 0x7f7a3c0c5aa0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c0098f0 tx=0x7f7a2c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ace4995d7491ec10 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:25.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.451+0000 7f7a3959a700 1 -- 172.21.15.23:0/4252981393 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c004230 con 0x7f7a3c06a280 2023-06-22T08:21:25.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.452+0000 7f7a3959a700 1 -- 172.21.15.23:0/4252981393 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c004390 con 0x7f7a3c06a280 2023-06-22T08:21:25.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.452+0000 7f7a3959a700 1 -- 172.21.15.23:0/4252981393 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0036a0 con 0x7f7a3c06a280 2023-06-22T08:21:25.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.453+0000 7f7a40bff700 1 -- 172.21.15.23:0/4252981393 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c06a280 msgr2=0x7f7a3c0c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:25.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.453+0000 7f7a40bff700 1 --2- 172.21.15.23:0/4252981393 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c06a280 0x7f7a3c0c5aa0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c0098f0 tx=0x7f7a2c009c00 comp rx=0 tx=0).stop 2023-06-22T08:21:25.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.453+0000 7f7a40bff700 1 -- 172.21.15.23:0/4252981393 shutdown_connections 2023-06-22T08:21:25.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.453+0000 7f7a40bff700 1 --2- 172.21.15.23:0/4252981393 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c06a280 0x7f7a3c0c5aa0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:25.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.453+0000 7f7a40bff700 1 --2- 172.21.15.23:0/4252981393 >> v2:172.21.15.67:3300/0 conn(0x7f7a3c15d9b0 0x7f7a3c06d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:25.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.453+0000 7f7a40bff700 1 -- 172.21.15.23:0/4252981393 >> 172.21.15.23:0/4252981393 conn(0x7f7a3c0ca3f0 msgr2=0x7f7a3c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.453+0000 7f7a40bff700 1 -- 172.21.15.23:0/4252981393 shutdown_connections 2023-06-22T08:21:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.453+0000 7f7a40bff700 1 -- 172.21.15.23:0/4252981393 wait complete. 2023-06-22T08:21:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.456+0000 7f7a40bff700 1 Processor -- start 2023-06-22T08:21:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.456+0000 7f7a40bff700 1 -- start start 2023-06-22T08:21:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.458+0000 7f7a40bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c069eb0 0x7f7a3c1609c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.458+0000 7f7a40bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a3c06a280 0x7f7a3c159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.458+0000 7f7a40bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3c15d9b0 0x7f7a3c0c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:25.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.458+0000 7f7a40bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c15d310 con 0x7f7a3c06a280 2023-06-22T08:21:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.458+0000 7f7a40bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c13f9a0 con 0x7f7a3c15d9b0 2023-06-22T08:21:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.458+0000 7f7a40bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c15ae30 con 0x7f7a3c069eb0 2023-06-22T08:21:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.458+0000 7f7a3a59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c069eb0 0x7f7a3c1609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:25.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.458+0000 7f7a3a59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c069eb0 0x7f7a3c1609c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:47260/0 (socket says 172.21.15.23:47260) 2023-06-22T08:21:25.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.458+0000 7f7a3a59c700 1 -- 172.21.15.23:0/3704033375 learned_addr learned my addr 172.21.15.23:0/3704033375 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:25.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.459+0000 7f7a3a59c700 1 -- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3c15d9b0 msgr2=0x7f7a3c0c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:25.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.459+0000 7f7a3a59c700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3c15d9b0 0x7f7a3c0c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:25.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.459+0000 7f7a3a59c700 1 -- 172.21.15.23:0/3704033375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a3c06a280 msgr2=0x7f7a3c159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:25.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.459+0000 7f7a3a59c700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a3c06a280 0x7f7a3c159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:25.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.459+0000 7f7a3a59c700 1 -- 172.21.15.23:0/3704033375 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c009550 con 0x7f7a3c069eb0 2023-06-22T08:21:25.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.463+0000 7f7a3a59c700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c069eb0 0x7f7a3c1609c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400ba80 tx=0x7f7a3400be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:25.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.463+0000 7f7a2affd700 1 -- 172.21.15.23:0/3704033375 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3400ca80 con 0x7f7a3c069eb0 2023-06-22T08:21:25.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.463+0000 7f7a2affd700 1 -- 172.21.15.23:0/3704033375 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3400cbe0 con 0x7f7a3c069eb0 2023-06-22T08:21:25.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.463+0000 7f7a2affd700 1 -- 172.21.15.23:0/3704033375 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a34014440 con 0x7f7a3c069eb0 2023-06-22T08:21:25.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.465+0000 7f7a40bff700 1 -- 172.21.15.23:0/3704033375 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3c15b110 con 0x7f7a3c069eb0 2023-06-22T08:21:25.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.465+0000 7f7a40bff700 1 -- 172.21.15.23:0/3704033375 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c15b6d0 con 0x7f7a3c069eb0 2023-06-22T08:21:25.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.466+0000 7f7a40bff700 1 -- 172.21.15.23:0/3704033375 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3c04ea20 con 0x7f7a3c069eb0 2023-06-22T08:21:25.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.468+0000 7f7a2affd700 1 -- 172.21.15.23:0/3704033375 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f7a34006d50 con 0x7f7a3c069eb0 2023-06-22T08:21:25.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.470+0000 7f7a2affd700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1c06adb0 0x7f7a1c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:25.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.470+0000 7f7a2affd700 1 -- 172.21.15.23:0/3704033375 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f7a340897f0 con 0x7f7a3c069eb0 2023-06-22T08:21:25.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.470+0000 7f7a39d9b700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1c06adb0 0x7f7a1c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:25.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.471+0000 7f7a39d9b700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1c06adb0 0x7f7a1c06d260 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0c5f70 tx=0x7f7a30006d20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:25.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.475+0000 7f7a2affd700 1 -- 172.21.15.23:0/3704033375 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7a34057b70 con 0x7f7a3c069eb0 2023-06-22T08:21:25.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.832+0000 7f7a40bff700 1 -- 172.21.15.23:0/3704033375 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a3c0cb300 con 0x7f7a3c069eb0 2023-06-22T08:21:25.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.833+0000 7f7a2affd700 1 -- 172.21.15.23:0/3704033375 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7f7a34019020 con 0x7f7a3c069eb0 2023-06-22T08:21:25.836 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:25.836 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:25.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 -- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1c06adb0 msgr2=0x7f7a1c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:25.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1c06adb0 0x7f7a1c06d260 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c0c5f70 tx=0x7f7a30006d20 comp rx=0 tx=0).stop 2023-06-22T08:21:25.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 -- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c069eb0 msgr2=0x7f7a3c1609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:25.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c069eb0 0x7f7a3c1609c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400ba80 tx=0x7f7a3400be40 comp rx=0 tx=0).stop 2023-06-22T08:21:25.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 -- 172.21.15.23:0/3704033375 shutdown_connections 2023-06-22T08:21:25.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f7a1c06adb0 0x7f7a1c06d260 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:25.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3c15d9b0 0x7f7a3c0c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:25.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7a3c069eb0 0x7f7a3c1609c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:25.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 --2- 172.21.15.23:0/3704033375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a3c06a280 0x7f7a3c159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:25.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 -- 172.21.15.23:0/3704033375 >> 172.21.15.23:0/3704033375 conn(0x7f7a3c0ca3f0 msgr2=0x7f7a3c0c3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:25.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 -- 172.21.15.23:0/3704033375 shutdown_connections 2023-06-22T08:21:25.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:25.837+0000 7f7a40bff700 1 -- 172.21.15.23:0/3704033375 wait complete. 2023-06-22T08:21:25.841 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:25.867 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:27.868 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:28.116 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir baz 2023-06-22T08:21:28.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.114+0000 7f682fdff700 1 Processor -- start 2023-06-22T08:21:28.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.114+0000 7f682fdff700 1 -- start start 2023-06-22T08:21:28.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.115+0000 7f682fdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 0x7f68280c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:28.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.115+0000 7f682fdff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f682806d050 0x7f6828160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:28.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.115+0000 7f682fdff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68280c6a60 con 0x7f68280c3a40 2023-06-22T08:21:28.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.115+0000 7f682fdff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6828157570 con 0x7f682806d050 2023-06-22T08:21:28.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.115+0000 7f682fdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68280c4350 con 0x7f682815a320 2023-06-22T08:21:28.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682db9b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f68280c3a40 0x7f6828159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:39986/0 (socket says 172.21.15.23:39986) 2023-06-22T08:21:28.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682db9b700 1 -- 172.21.15.23:0/1270662615 learned_addr learned my addr 172.21.15.23:0/1270662615 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:28.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682e39c700 1 --2- 172.21.15.23:0/1270662615 >> v2:172.21.15.23:3300/0 conn(0x7f682806d050 0x7f6828160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:28.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682d39a700 1 --2- 172.21.15.23:0/1270662615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 0x7f68280c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:28.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682cb99700 1 -- 172.21.15.23:0/1270662615 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251523932 0 0) 0x7f68280c6a60 con 0x7f68280c3a40 2023-06-22T08:21:28.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682cb99700 1 -- 172.21.15.23:0/1270662615 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68140035b0 con 0x7f68280c3a40 2023-06-22T08:21:28.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682d39a700 1 -- 172.21.15.23:0/1270662615 >> v2:172.21.15.23:3300/0 conn(0x7f682806d050 msgr2=0x7f6828160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:28.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682d39a700 1 --2- 172.21.15.23:0/1270662615 >> v2:172.21.15.23:3300/0 conn(0x7f682806d050 0x7f6828160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682d39a700 1 -- 172.21.15.23:0/1270662615 >> v1:172.21.15.67:6789/0 conn(0x7f68280c3a40 legacy=0x7f6828159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:28.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682d39a700 1 -- 172.21.15.23:0/1270662615 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682814d690 con 0x7f682815a320 2023-06-22T08:21:28.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.116+0000 7f682e39c700 1 --2- 172.21.15.23:0/1270662615 >> v2:172.21.15.23:3300/0 conn(0x7f682806d050 0x7f6828160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:28.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.117+0000 7f682d39a700 1 --2- 172.21.15.23:0/1270662615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 0x7f68280c3e10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6818009840 tx=0x7f6818009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db4265705b1d556a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:28.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.117+0000 7f682cb99700 1 -- 172.21.15.23:0/1270662615 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6818011830 con 0x7f682815a320 2023-06-22T08:21:28.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.117+0000 7f682cb99700 1 -- 172.21.15.23:0/1270662615 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6818011990 con 0x7f682815a320 2023-06-22T08:21:28.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.117+0000 7f682cb99700 1 -- 172.21.15.23:0/1270662615 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6818011d20 con 0x7f682815a320 2023-06-22T08:21:28.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.118+0000 7f682fdff700 1 -- 172.21.15.23:0/1270662615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 msgr2=0x7f68280c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:28.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.118+0000 7f682fdff700 1 --2- 172.21.15.23:0/1270662615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 0x7f68280c3e10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6818009840 tx=0x7f6818009b50 comp rx=0 tx=0).stop 2023-06-22T08:21:28.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.118+0000 7f682fdff700 1 -- 172.21.15.23:0/1270662615 shutdown_connections 2023-06-22T08:21:28.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.118+0000 7f682fdff700 1 --2- 172.21.15.23:0/1270662615 >> v2:172.21.15.23:3300/0 conn(0x7f682806d050 0x7f6828160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.118+0000 7f682fdff700 1 --2- 172.21.15.23:0/1270662615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 0x7f68280c3e10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.118+0000 7f682fdff700 1 -- 172.21.15.23:0/1270662615 >> 172.21.15.23:0/1270662615 conn(0x7f68280ca3f0 msgr2=0x7f68280ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:28.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.118+0000 7f682fdff700 1 -- 172.21.15.23:0/1270662615 shutdown_connections 2023-06-22T08:21:28.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.118+0000 7f682fdff700 1 -- 172.21.15.23:0/1270662615 wait complete. 2023-06-22T08:21:28.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.120+0000 7f682fdff700 1 Processor -- start 2023-06-22T08:21:28.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.121+0000 7f682fdff700 1 -- start start 2023-06-22T08:21:28.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682fdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f682806d050 0x7f68281e3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:28.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682fdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68280c3a40 0x7f68280c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:28.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682fdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 0x7f68281ee620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:28.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682fdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68281f1b90 con 0x7f682806d050 2023-06-22T08:21:28.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682fdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68281f2710 con 0x7f68280c3a40 2023-06-22T08:21:28.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682fdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68281dfa00 con 0x7f682815a320 2023-06-22T08:21:28.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682d39a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68280c3a40 0x7f68280c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:28.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682db9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f682806d050 0x7f68281e3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:28.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682d39a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68280c3a40 0x7f68280c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:53234/0 (socket says 172.21.15.23:53234) 2023-06-22T08:21:28.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682d39a700 1 -- 172.21.15.23:0/23489821 learned_addr learned my addr 172.21.15.23:0/23489821 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:28.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.122+0000 7f682e39c700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 0x7f68281ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:28.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f682e39c700 1 -- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68280c3a40 msgr2=0x7f68280c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:28.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f682e39c700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68280c3a40 0x7f68280c6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f682e39c700 1 -- 172.21.15.23:0/23489821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f682806d050 msgr2=0x7f68281e3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:28.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f682e39c700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f682806d050 0x7f68281e3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f682e39c700 1 -- 172.21.15.23:0/23489821 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6820009a70 con 0x7f682815a320 2023-06-22T08:21:28.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f682d39a700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68280c3a40 0x7f68280c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:28.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f682db9b700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f682806d050 0x7f68281e3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:28.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f682e39c700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 0x7f68281ee620 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6820009db0 tx=0x7f682000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:28.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f681e7fc700 1 -- 172.21.15.23:0/23489821 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820009c20 con 0x7f682815a320 2023-06-22T08:21:28.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f681e7fc700 1 -- 172.21.15.23:0/23489821 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682000f040 con 0x7f682815a320 2023-06-22T08:21:28.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f681e7fc700 1 -- 172.21.15.23:0/23489821 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682000ccb0 con 0x7f682815a320 2023-06-22T08:21:28.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f682fdff700 1 -- 172.21.15.23:0/23489821 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68180094a0 con 0x7f682815a320 2023-06-22T08:21:28.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.123+0000 7f682fdff700 1 -- 172.21.15.23:0/23489821 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68281f1de0 con 0x7f682815a320 2023-06-22T08:21:28.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.124+0000 7f682fdff700 1 -- 172.21.15.23:0/23489821 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68281f2aa0 con 0x7f682815a320 2023-06-22T08:21:28.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.125+0000 7f681e7fc700 1 -- 172.21.15.23:0/23489821 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6820014480 con 0x7f682815a320 2023-06-22T08:21:28.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.126+0000 7f681e7fc700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f680c06ace0 0x7f680c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:28.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.126+0000 7f681e7fc700 1 -- 172.21.15.23:0/23489821 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f682008a230 con 0x7f682815a320 2023-06-22T08:21:28.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.126+0000 7f682db9b700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f680c06ace0 0x7f680c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:28.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.127+0000 7f682db9b700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f680c06ace0 0x7f680c06d190 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6824007fd0 tx=0x7f6824007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:28.130 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir baz/.snap/two 2023-06-22T08:21:28.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.131+0000 7f681e7fc700 1 -- 172.21.15.23:0/23489821 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6820054aa0 con 0x7f682815a320 2023-06-22T08:21:28.150 INFO:tasks.workunit.client.0.smithi023.stderr:+ mv baz foo 2023-06-22T08:21:28.156 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:21:28.332 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir foo/baz/.snap/two 2023-06-22T08:21:28.341 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir foo/.snap/one 2023-06-22T08:21:28.351 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -r foo 2023-06-22T08:21:28.373 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:21:28.374 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:21:28.374 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:21:28.374 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:21:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.453+0000 7f682fdff700 1 -- 172.21.15.23:0/23489821 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f682804ea20 con 0x7f682815a320 2023-06-22T08:21:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.453+0000 7f681e7fc700 1 -- 172.21.15.23:0/23489821 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7f68200580c0 con 0x7f682815a320 2023-06-22T08:21:28.456 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:28.457 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 -- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f680c06ace0 msgr2=0x7f680c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:28.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f680c06ace0 0x7f680c06d190 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6824007fd0 tx=0x7f6824007dc0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 -- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 msgr2=0x7f68281ee620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 0x7f68281ee620 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6820009db0 tx=0x7f682000c3a0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 -- 172.21.15.23:0/23489821 shutdown_connections 2023-06-22T08:21:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f682815a320 0x7f68281ee620 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f682806d050 0x7f68281e3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f680c06ace0 0x7f680c06d190 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 --2- 172.21.15.23:0/23489821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68280c3a40 0x7f68280c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 -- 172.21.15.23:0/23489821 >> 172.21.15.23:0/23489821 conn(0x7f68280ca3f0 msgr2=0x7f6828148f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 -- 172.21.15.23:0/23489821 shutdown_connections 2023-06-22T08:21:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:28.457+0000 7f682fdff700 1 -- 172.21.15.23:0/23489821 wait complete. 2023-06-22T08:21:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:28.483 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-06-22T08:21:28.484 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 169, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 169, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 }, "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 36 }, "gid_6318": { "addr": "172.21.15.23:6834/904774922", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 904774922, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 904774922, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 151, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:stopping", "state_seq": 39 }, "gid_6616": { "addr": "172.21.15.67:6837/1855579717", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1855579717, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1855579717, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6616, "incarnation": 162, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 4, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:21:14.036557+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6616, "mds_1": 5707, "mds_2": 6313, "mds_3": 5995, "mds_4": 6318 } } } ], "standbys": [] } 2023-06-22T08:21:28.485 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-multiple-capsnaps.sh... 2023-06-22T08:21:28.485 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-multiple-capsnaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh 2023-06-22T08:21:30.485 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:30.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.725+0000 7fc83bdff700 1 Processor -- start 2023-06-22T08:21:30.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.725+0000 7fc83bdff700 1 -- start start 2023-06-22T08:21:30.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.727+0000 7fc83bdff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc834071070 0x7fc834160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:30.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.727+0000 7fc83bdff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8340c51a0 con 0x7fc83414f850 2023-06-22T08:21:30.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.727+0000 7fc83bdff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8340c44e0 con 0x7fc8340c69e0 2023-06-22T08:21:30.730 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.727+0000 7fc83bdff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8340c5440 con 0x7fc834071070 2023-06-22T08:21:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.728+0000 7fc83a39c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc834071070 0x7fc834160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.728+0000 7fc839b9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc83414f850 0x7fc8340c3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:42968/0 (socket says 172.21.15.23:42968) 2023-06-22T08:21:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.728+0000 7fc839b9b700 1 -- 172.21.15.23:0/1426226605 learned_addr learned my addr 172.21.15.23:0/1426226605 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:30.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.728+0000 7fc83939a700 1 --1- 172.21.15.23:0/1426226605 >> v1:172.21.15.67:6789/0 conn(0x7fc8340c69e0 0x7fc834155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:40000/0 (socket says 172.21.15.23:40000) 2023-06-22T08:21:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.728+0000 7fc83a39c700 1 -- 172.21.15.23:0/1426226605 >> v1:172.21.15.67:6789/0 conn(0x7fc8340c69e0 legacy=0x7fc834155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.728+0000 7fc83a39c700 1 -- 172.21.15.23:0/1426226605 >> v1:172.21.15.23:6789/0 conn(0x7fc83414f850 legacy=0x7fc8340c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:30.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.728+0000 7fc838b99700 1 -- 172.21.15.23:0/1426226605 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696932269 0 0) 0x7fc8340c44e0 con 0x7fc8340c69e0 2023-06-22T08:21:30.734 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.728+0000 7fc83a39c700 1 -- 172.21.15.23:0/1426226605 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8340c5720 con 0x7fc834071070 2023-06-22T08:21:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.729+0000 7fc83a39c700 1 --2- 172.21.15.23:0/1426226605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc834071070 0x7fc834160eb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009d90 tx=0x7fc82c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb14fab7b061acc server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.729+0000 7fc838b99700 1 -- 172.21.15.23:0/1426226605 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c00e070 con 0x7fc834071070 2023-06-22T08:21:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.729+0000 7fc838b99700 1 -- 172.21.15.23:0/1426226605 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c00bad0 con 0x7fc834071070 2023-06-22T08:21:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.729+0000 7fc838b99700 1 -- 172.21.15.23:0/1426226605 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c00be80 con 0x7fc834071070 2023-06-22T08:21:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.730+0000 7fc83bdff700 1 -- 172.21.15.23:0/1426226605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc834071070 msgr2=0x7fc834160eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.730+0000 7fc83bdff700 1 --2- 172.21.15.23:0/1426226605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc834071070 0x7fc834160eb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009d90 tx=0x7fc82c00b330 comp rx=0 tx=0).stop 2023-06-22T08:21:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.730+0000 7fc83bdff700 1 -- 172.21.15.23:0/1426226605 shutdown_connections 2023-06-22T08:21:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.730+0000 7fc83bdff700 1 --2- 172.21.15.23:0/1426226605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc834071070 0x7fc834160eb0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.730+0000 7fc83bdff700 1 -- 172.21.15.23:0/1426226605 >> 172.21.15.23:0/1426226605 conn(0x7fc8340ca3f0 msgr2=0x7fc8340ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:30.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.730+0000 7fc83bdff700 1 -- 172.21.15.23:0/1426226605 shutdown_connections 2023-06-22T08:21:30.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.730+0000 7fc83bdff700 1 -- 172.21.15.23:0/1426226605 wait complete. 2023-06-22T08:21:30.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.733+0000 7fc83bdff700 1 Processor -- start 2023-06-22T08:21:30.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.733+0000 7fc83bdff700 1 -- start start 2023-06-22T08:21:30.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.734+0000 7fc83bdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc834071070 0x7fc83414da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:30.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.735+0000 7fc83bdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8340c69e0 0x7fc83414dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:30.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.735+0000 7fc83bdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc83414f850 0x7fc8340c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:30.738 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.735+0000 7fc83bdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83414eb50 con 0x7fc8340c69e0 2023-06-22T08:21:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.735+0000 7fc83bdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340c9b60 con 0x7fc834071070 2023-06-22T08:21:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.735+0000 7fc83bdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8341492c0 con 0x7fc83414f850 2023-06-22T08:21:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.735+0000 7fc83a39c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc83414f850 0x7fc8340c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.735+0000 7fc839b9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc834071070 0x7fc83414da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.735+0000 7fc83a39c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc83414f850 0x7fc8340c8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:41034/0 (socket says 172.21.15.23:41034) 2023-06-22T08:21:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.735+0000 7fc83a39c700 1 -- 172.21.15.23:0/1441574385 learned_addr learned my addr 172.21.15.23:0/1441574385 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.735+0000 7fc83939a700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8340c69e0 0x7fc83414dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.736+0000 7fc83939a700 1 -- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc83414f850 msgr2=0x7fc8340c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.736+0000 7fc83939a700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc83414f850 0x7fc8340c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.736+0000 7fc83939a700 1 -- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc834071070 msgr2=0x7fc83414da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.736+0000 7fc83939a700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc834071070 0x7fc83414da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.736+0000 7fc83939a700 1 -- 172.21.15.23:0/1441574385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82c009a70 con 0x7fc8340c69e0 2023-06-22T08:21:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.736+0000 7fc83939a700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8340c69e0 0x7fc83414dfc0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc82400c400 tx=0x7fc82400c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.736+0000 7fc82a7fc700 1 -- 172.21.15.23:0/1441574385 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824004c80 con 0x7fc8340c69e0 2023-06-22T08:21:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.736+0000 7fc82a7fc700 1 -- 172.21.15.23:0/1441574385 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc824004de0 con 0x7fc8340c69e0 2023-06-22T08:21:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.736+0000 7fc82a7fc700 1 -- 172.21.15.23:0/1441574385 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824007310 con 0x7fc8340c69e0 2023-06-22T08:21:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.736+0000 7fc83bdff700 1 -- 172.21.15.23:0/1441574385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8340c81d0 con 0x7fc8340c69e0 2023-06-22T08:21:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.737+0000 7fc83bdff700 1 -- 172.21.15.23:0/1441574385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8340c9d70 con 0x7fc8340c69e0 2023-06-22T08:21:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.738+0000 7fc83bdff700 1 -- 172.21.15.23:0/1441574385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8340c8360 con 0x7fc8340c69e0 2023-06-22T08:21:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.741+0000 7fc82a7fc700 1 -- 172.21.15.23:0/1441574385 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc824007470 con 0x7fc8340c69e0 2023-06-22T08:21:30.745 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.743+0000 7fc82a7fc700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc81806ad10 0x7fc81806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.743+0000 7fc82a7fc700 1 -- 172.21.15.23:0/1441574385 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7871+0+0 (secure 0 0 0) 0x7fc824089ce0 con 0x7fc8340c69e0 2023-06-22T08:21:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.744+0000 7fc839b9b700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc81806ad10 0x7fc81806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:30.747 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.744+0000 7fc839b9b700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc81806ad10 0x7fc81806d1c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc830000f60 tx=0x7fc830004a50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:30.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:30.746+0000 7fc82a7fc700 1 -- 172.21.15.23:0/1441574385 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc8240546e0 con 0x7fc8340c69e0 2023-06-22T08:21:30.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~68s 2023-06-22T08:21:31.139 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:31.140 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:31.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.136+0000 7fc83bdff700 1 -- 172.21.15.23:0/1441574385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc83404ea20 con 0x7fc8340c69e0 2023-06-22T08:21:31.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.137+0000 7fc82a7fc700 1 -- 172.21.15.23:0/1441574385 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7fc824057d00 con 0x7fc8340c69e0 2023-06-22T08:21:31.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.140+0000 7fc83bdff700 1 -- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc81806ad10 msgr2=0x7fc81806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:31.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.140+0000 7fc83bdff700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc81806ad10 0x7fc81806d1c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc830000f60 tx=0x7fc830004a50 comp rx=0 tx=0).stop 2023-06-22T08:21:31.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.140+0000 7fc83bdff700 1 -- 172.21.15.23:0/1441574385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8340c69e0 msgr2=0x7fc83414dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:31.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.140+0000 7fc83bdff700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8340c69e0 0x7fc83414dfc0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc82400c400 tx=0x7fc82400c710 comp rx=0 tx=0).stop 2023-06-22T08:21:31.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.140+0000 7fc83bdff700 1 -- 172.21.15.23:0/1441574385 shutdown_connections 2023-06-22T08:21:31.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.140+0000 7fc83bdff700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc83414f850 0x7fc8340c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:31.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.140+0000 7fc83bdff700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc81806ad10 0x7fc81806d1c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:31.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.140+0000 7fc83bdff700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc834071070 0x7fc83414da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:31.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.140+0000 7fc83bdff700 1 --2- 172.21.15.23:0/1441574385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8340c69e0 0x7fc83414dfc0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:31.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.140+0000 7fc83bdff700 1 -- 172.21.15.23:0/1441574385 >> 172.21.15.23:0/1441574385 conn(0x7fc8340ca3f0 msgr2=0x7fc83413b050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:31.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.141+0000 7fc83bdff700 1 -- 172.21.15.23:0/1441574385 shutdown_connections 2023-06-22T08:21:31.146 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:31.141+0000 7fc83bdff700 1 -- 172.21.15.23:0/1441574385 wait complete. 2023-06-22T08:21:31.147 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:31.174 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:33.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:21:33.092 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo asdf 2023-06-22T08:21:33.093 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/1 2023-06-22T08:21:33.110 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 777 a 2023-06-22T08:21:33.113 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/2 2023-06-22T08:21:33.125 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo qwer 2023-06-22T08:21:33.139 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/3 2023-06-22T08:21:33.150 INFO:tasks.workunit.client.0.smithi023.stderr:+ chmod 666 a 2023-06-22T08:21:33.152 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/4 2023-06-22T08:21:33.162 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo zxcv 2023-06-22T08:21:33.175 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:33.209 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/5 2023-06-22T08:21:33.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.391+0000 7fcfcc5ff700 1 Processor -- start 2023-06-22T08:21:33.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.391+0000 7fcfcc5ff700 1 -- start start 2023-06-22T08:21:33.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.393+0000 7fcfcc5ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcfc4152d70 0x7fcfc40c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:33.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.393+0000 7fcfcc5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfc40c45f0 0x7fcfc40c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:33.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.393+0000 7fcfcc5ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc4159790 con 0x7fcfc4159330 2023-06-22T08:21:33.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.393+0000 7fcfcc5ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc40c95e0 con 0x7fcfc4152d70 2023-06-22T08:21:33.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.393+0000 7fcfcc5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc415bea0 con 0x7fcfc40c45f0 2023-06-22T08:21:33.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.393+0000 7fcfcab9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfc40c45f0 0x7fcfc40c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:33.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.393+0000 7fcfca39b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcfc4152d70 0x7fcfc40c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:33.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.393+0000 7fcfcab9c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfc40c45f0 0x7fcfc40c4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:53254/0 (socket says 172.21.15.23:53254) 2023-06-22T08:21:33.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.393+0000 7fcfc9b9a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcfc4159330 0x7fcfc40cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:40002/0 (socket says 172.21.15.23:40002) 2023-06-22T08:21:33.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.393+0000 7fcfcab9c700 1 -- 172.21.15.23:0/3171715642 learned_addr learned my addr 172.21.15.23:0/3171715642 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:33.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.394+0000 7fcfcab9c700 1 -- 172.21.15.23:0/3171715642 >> v2:172.21.15.23:3300/0 conn(0x7fcfc4152d70 msgr2=0x7fcfc40c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:33.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.394+0000 7fcfcab9c700 1 --2- 172.21.15.23:0/3171715642 >> v2:172.21.15.23:3300/0 conn(0x7fcfc4152d70 0x7fcfc40c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:33.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.394+0000 7fcfcab9c700 1 -- 172.21.15.23:0/3171715642 >> v1:172.21.15.67:6789/0 conn(0x7fcfc4159330 legacy=0x7fcfc40cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:33.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.394+0000 7fcfcab9c700 1 -- 172.21.15.23:0/3171715642 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc415c160 con 0x7fcfc40c45f0 2023-06-22T08:21:33.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.394+0000 7fcfcab9c700 1 --2- 172.21.15.23:0/3171715642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc40c45f0 0x7fcfc40c4a10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcfc408c840 tx=0x7fcfbc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70695e5669bd440e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:33.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.394+0000 7fcfc9399700 1 -- 172.21.15.23:0/3171715642 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc00e070 con 0x7fcfc40c45f0 2023-06-22T08:21:33.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.394+0000 7fcfc9399700 1 -- 172.21.15.23:0/3171715642 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfbc003a20 con 0x7fcfc40c45f0 2023-06-22T08:21:33.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.394+0000 7fcfc9399700 1 -- 172.21.15.23:0/3171715642 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc003dd0 con 0x7fcfc40c45f0 2023-06-22T08:21:33.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.395+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/3171715642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc40c45f0 msgr2=0x7fcfc40c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:33.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.395+0000 7fcfcc5ff700 1 --2- 172.21.15.23:0/3171715642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc40c45f0 0x7fcfc40c4a10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcfc408c840 tx=0x7fcfbc00b330 comp rx=0 tx=0).stop 2023-06-22T08:21:33.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.395+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/3171715642 shutdown_connections 2023-06-22T08:21:33.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.395+0000 7fcfcc5ff700 1 --2- 172.21.15.23:0/3171715642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc40c45f0 0x7fcfc40c4a10 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:33.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.395+0000 7fcfcc5ff700 1 --2- 172.21.15.23:0/3171715642 >> v2:172.21.15.23:3300/0 conn(0x7fcfc4152d70 0x7fcfc40c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:33.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.395+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/3171715642 >> 172.21.15.23:0/3171715642 conn(0x7fcfc406ae30 msgr2=0x7fcfc406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:33.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.395+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/3171715642 shutdown_connections 2023-06-22T08:21:33.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.395+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/3171715642 wait complete. 2023-06-22T08:21:33.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.397+0000 7fcfcc5ff700 1 Processor -- start 2023-06-22T08:21:33.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.397+0000 7fcfcc5ff700 1 -- start start 2023-06-22T08:21:33.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.398+0000 7fcfcc5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc40c45f0 0x7fcfc414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:33.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.398+0000 7fcfcc5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc4152d70 0x7fcfc41ee670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:33.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.398+0000 7fcfcc5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcfc4159330 0x7fcfc41eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:33.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.398+0000 7fcfcc5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41f1970 con 0x7fcfc40c45f0 2023-06-22T08:21:33.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.398+0000 7fcfcc5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc414ef80 con 0x7fcfc4152d70 2023-06-22T08:21:33.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.398+0000 7fcfcc5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41f1bc0 con 0x7fcfc4159330 2023-06-22T08:21:33.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfc9b9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc4152d70 0x7fcfc41ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:33.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfca39b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc40c45f0 0x7fcfc414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:33.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfc9b9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc4152d70 0x7fcfc41ee670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:53262/0 (socket says 172.21.15.23:53262) 2023-06-22T08:21:33.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfc9b9a700 1 -- 172.21.15.23:0/2781419559 learned_addr learned my addr 172.21.15.23:0/2781419559 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:33.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfcab9c700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcfc4159330 0x7fcfc41eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:33.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfc9b9a700 1 -- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcfc4159330 msgr2=0x7fcfc41eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:33.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfc9b9a700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcfc4159330 0x7fcfc41eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:33.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfc9b9a700 1 -- 172.21.15.23:0/2781419559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc40c45f0 msgr2=0x7fcfc414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:33.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfc9b9a700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc40c45f0 0x7fcfc414e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:33.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfc9b9a700 1 -- 172.21.15.23:0/2781419559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfbc009b30 con 0x7fcfc4152d70 2023-06-22T08:21:33.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfca39b700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc40c45f0 0x7fcfc414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:33.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.399+0000 7fcfc9b9a700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc4152d70 0x7fcfc41ee670 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400c780 tx=0x7fcfb400ca90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:33.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.400+0000 7fcfbaffd700 1 -- 172.21.15.23:0/2781419559 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4004c80 con 0x7fcfc4152d70 2023-06-22T08:21:33.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.400+0000 7fcfbaffd700 1 -- 172.21.15.23:0/2781419559 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb4004de0 con 0x7fcfc4152d70 2023-06-22T08:21:33.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.400+0000 7fcfbaffd700 1 -- 172.21.15.23:0/2781419559 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40072f0 con 0x7fcfc4152d70 2023-06-22T08:21:33.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.400+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/2781419559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc41f1ea0 con 0x7fcfc4152d70 2023-06-22T08:21:33.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.400+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/2781419559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc41f2460 con 0x7fcfc4152d70 2023-06-22T08:21:33.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.401+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/2781419559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc404ea20 con 0x7fcfc4152d70 2023-06-22T08:21:33.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.402+0000 7fcfbaffd700 1 -- 172.21.15.23:0/2781419559 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcfb4007450 con 0x7fcfc4152d70 2023-06-22T08:21:33.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.403+0000 7fcfbaffd700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcfa806ace0 0x7fcfa806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:33.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.403+0000 7fcfbaffd700 1 -- 172.21.15.23:0/2781419559 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7871+0+0 (secure 0 0 0) 0x7fcfb40899f0 con 0x7fcfc4152d70 2023-06-22T08:21:33.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.403+0000 7fcfca39b700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcfa806ace0 0x7fcfa806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:33.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.404+0000 7fcfca39b700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcfa806ace0 0x7fcfa806d190 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009fd0 tx=0x7fcfc0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:33.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.407+0000 7fcfbaffd700 1 -- 172.21.15.23:0/2781419559 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcfb4054260 con 0x7fcfc4152d70 2023-06-22T08:21:33.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.694+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/2781419559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfc40c4200 con 0x7fcfc4152d70 2023-06-22T08:21:33.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.695+0000 7fcfbaffd700 1 -- 172.21.15.23:0/2781419559 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7fcfb4057880 con 0x7fcfc4152d70 2023-06-22T08:21:33.698 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:33.698 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:33.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcfa806ace0 msgr2=0x7fcfa806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:33.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcfa806ace0 0x7fcfa806d190 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009fd0 tx=0x7fcfc0009210 comp rx=0 tx=0).stop 2023-06-22T08:21:33.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc4152d70 msgr2=0x7fcfc41ee670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:33.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc4152d70 0x7fcfc41ee670 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400c780 tx=0x7fcfb400ca90 comp rx=0 tx=0).stop 2023-06-22T08:21:33.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/2781419559 shutdown_connections 2023-06-22T08:21:33.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcfc4159330 0x7fcfc41eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:33.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc40c45f0 0x7fcfc414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:33.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcfa806ace0 0x7fcfa806d190 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:33.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 --2- 172.21.15.23:0/2781419559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfc4152d70 0x7fcfc41ee670 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:33.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/2781419559 >> 172.21.15.23:0/2781419559 conn(0x7fcfc406ae30 msgr2=0x7fcfc414f790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:33.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.697+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/2781419559 shutdown_connections 2023-06-22T08:21:33.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:33.698+0000 7fcfcc5ff700 1 -- 172.21.15.23:0/2781419559 wait complete. 2023-06-22T08:21:33.702 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:33.720 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:35.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:35.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.923+0000 7f424cbff700 1 Processor -- start 2023-06-22T08:21:35.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.923+0000 7f424cbff700 1 -- start start 2023-06-22T08:21:35.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.924+0000 7f424cbff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4248152a10 0x7f42480c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:35.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.924+0000 7f424cbff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42480c9200 0x7f42480cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:35.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.924+0000 7f424cbff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4248159730 con 0x7f42480cb7a0 2023-06-22T08:21:35.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.924+0000 7f424cbff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42480c5470 con 0x7f4248152a10 2023-06-22T08:21:35.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.924+0000 7f424cbff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4248158910 con 0x7f42480c9200 2023-06-22T08:21:35.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f4245d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42480c9200 0x7f42480cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:35.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f424659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4248152a10 0x7f42480c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:35.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f4246d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f42480cb7a0 0x7f424815c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:39490/0 (socket says 172.21.15.23:39490) 2023-06-22T08:21:35.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f4246d9d700 1 -- 172.21.15.23:0/2515802887 learned_addr learned my addr 172.21.15.23:0/2515802887 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:35.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f4245d9b700 1 -- 172.21.15.23:0/2515802887 >> v2:172.21.15.23:3300/0 conn(0x7f4248152a10 msgr2=0x7f42480c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:35.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f4245d9b700 1 --2- 172.21.15.23:0/2515802887 >> v2:172.21.15.23:3300/0 conn(0x7f4248152a10 0x7f42480c4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:35.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f4245d9b700 1 -- 172.21.15.23:0/2515802887 >> v1:172.21.15.23:6789/0 conn(0x7f42480cb7a0 legacy=0x7f424815c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:35.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f4245d9b700 1 -- 172.21.15.23:0/2515802887 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4248158bb0 con 0x7f42480c9200 2023-06-22T08:21:35.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f424659c700 1 --2- 172.21.15.23:0/2515802887 >> v2:172.21.15.23:3300/0 conn(0x7f4248152a10 0x7f42480c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:35.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f4245d9b700 1 --2- 172.21.15.23:0/2515802887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480c9200 0x7f42480cb260 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f424808c4c0 tx=0x7f423c009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caa8a540e1536828 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:35.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.925+0000 7f424559a700 1 -- 172.21.15.23:0/2515802887 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c011830 con 0x7f42480c9200 2023-06-22T08:21:35.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.926+0000 7f424559a700 1 -- 172.21.15.23:0/2515802887 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c011990 con 0x7f42480c9200 2023-06-22T08:21:35.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.926+0000 7f424559a700 1 -- 172.21.15.23:0/2515802887 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c011d20 con 0x7f42480c9200 2023-06-22T08:21:35.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.926+0000 7f424cbff700 1 -- 172.21.15.23:0/2515802887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480c9200 msgr2=0x7f42480cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:35.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.926+0000 7f424cbff700 1 --2- 172.21.15.23:0/2515802887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480c9200 0x7f42480cb260 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f424808c4c0 tx=0x7f423c009b20 comp rx=0 tx=0).stop 2023-06-22T08:21:35.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.926+0000 7f424cbff700 1 -- 172.21.15.23:0/2515802887 shutdown_connections 2023-06-22T08:21:35.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.926+0000 7f424cbff700 1 --2- 172.21.15.23:0/2515802887 >> v2:172.21.15.23:3300/0 conn(0x7f4248152a10 0x7f42480c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:35.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.926+0000 7f424cbff700 1 --2- 172.21.15.23:0/2515802887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480c9200 0x7f42480cb260 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:35.932 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.926+0000 7f424cbff700 1 -- 172.21.15.23:0/2515802887 >> 172.21.15.23:0/2515802887 conn(0x7f4248065b90 msgr2=0x7f4248065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:35.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.926+0000 7f424cbff700 1 -- 172.21.15.23:0/2515802887 shutdown_connections 2023-06-22T08:21:35.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.926+0000 7f424cbff700 1 -- 172.21.15.23:0/2515802887 wait complete. 2023-06-22T08:21:35.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.928+0000 7f424cbff700 1 Processor -- start 2023-06-22T08:21:35.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.928+0000 7f424cbff700 1 -- start start 2023-06-22T08:21:35.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f424cbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f42480c9200 0x7f424814e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:35.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f424cbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480cb7a0 0x7f424814cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:35.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f424cbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4248152a10 0x7f424814d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:35.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f424cbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42480cb080 con 0x7f4248152a10 2023-06-22T08:21:35.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f424cbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424815a500 con 0x7f42480cb7a0 2023-06-22T08:21:35.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f424cbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f424814d9a0 con 0x7f42480c9200 2023-06-22T08:21:35.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f4245d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480cb7a0 0x7f424814cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:35.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f424659c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f42480c9200 0x7f424814e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:35.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f4246d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4248152a10 0x7f424814d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:35.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f4245d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480cb7a0 0x7f424814cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:43268/0 (socket says 172.21.15.23:43268) 2023-06-22T08:21:35.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.929+0000 7f4245d9b700 1 -- 172.21.15.23:0/1549608127 learned_addr learned my addr 172.21.15.23:0/1549608127 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:35.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f4245d9b700 1 -- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f42480c9200 msgr2=0x7f424814e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:35.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f4245d9b700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f42480c9200 0x7f424814e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:35.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f4245d9b700 1 -- 172.21.15.23:0/1549608127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4248152a10 msgr2=0x7f424814d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:35.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f4245d9b700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4248152a10 0x7f424814d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:35.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f4245d9b700 1 -- 172.21.15.23:0/1549608127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42380094a0 con 0x7f42480cb7a0 2023-06-22T08:21:35.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f4246d9d700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4248152a10 0x7f424814d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:35.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f424659c700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f42480c9200 0x7f424814e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:35.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f4245d9b700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480cb7a0 0x7f424814cf20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f42481598c0 tx=0x7f423c004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:35.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f4236ffd700 1 -- 172.21.15.23:0/1549608127 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c0036a0 con 0x7f42480cb7a0 2023-06-22T08:21:35.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f4236ffd700 1 -- 172.21.15.23:0/1549608127 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c01f070 con 0x7f42480cb7a0 2023-06-22T08:21:35.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f424cbff700 1 -- 172.21.15.23:0/1549608127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423c0094a0 con 0x7f42480cb7a0 2023-06-22T08:21:35.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f4236ffd700 1 -- 172.21.15.23:0/1549608127 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c003a60 con 0x7f42480cb7a0 2023-06-22T08:21:35.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.930+0000 7f424cbff700 1 -- 172.21.15.23:0/1549608127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4248159910 con 0x7f42480cb7a0 2023-06-22T08:21:35.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.931+0000 7f424cbff700 1 -- 172.21.15.23:0/1549608127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4248159150 con 0x7f42480cb7a0 2023-06-22T08:21:35.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.932+0000 7f4236ffd700 1 -- 172.21.15.23:0/1549608127 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f423c003bc0 con 0x7f42480cb7a0 2023-06-22T08:21:35.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.933+0000 7f4236ffd700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f422806ace0 0x7f422806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:35.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.933+0000 7f4236ffd700 1 -- 172.21.15.23:0/1549608127 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f423c021070 con 0x7f42480cb7a0 2023-06-22T08:21:35.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.934+0000 7f424659c700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f422806ace0 0x7f422806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:35.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.934+0000 7f424659c700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f422806ace0 0x7f422806d190 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4238011380 tx=0x7f4238009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:35.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:35.940+0000 7f4236ffd700 1 -- 172.21.15.23:0/1549608127 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f423c055e50 con 0x7f42480cb7a0 2023-06-22T08:21:36.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.195+0000 7f424cbff700 1 -- 172.21.15.23:0/1549608127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4248159150 con 0x7f42480cb7a0 2023-06-22T08:21:36.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.196+0000 7f4236ffd700 1 -- 172.21.15.23:0/1549608127 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7f4248159150 con 0x7f42480cb7a0 2023-06-22T08:21:36.198 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:36.199 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:36.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 -- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f422806ace0 msgr2=0x7f422806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:36.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f422806ace0 0x7f422806d190 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4238011380 tx=0x7f4238009210 comp rx=0 tx=0).stop 2023-06-22T08:21:36.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 -- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480cb7a0 msgr2=0x7f424814cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:36.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480cb7a0 0x7f424814cf20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f42481598c0 tx=0x7f423c004750 comp rx=0 tx=0).stop 2023-06-22T08:21:36.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 -- 172.21.15.23:0/1549608127 shutdown_connections 2023-06-22T08:21:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4248152a10 0x7f424814d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f42480c9200 0x7f424814e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f422806ace0 0x7f422806d190 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 --2- 172.21.15.23:0/1549608127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42480cb7a0 0x7f424814cf20 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 -- 172.21.15.23:0/1549608127 >> 172.21.15.23:0/1549608127 conn(0x7f4248065b90 msgr2=0x7f42480ca440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 -- 172.21.15.23:0/1549608127 shutdown_connections 2023-06-22T08:21:36.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:36.198+0000 7f424cbff700 1 -- 172.21.15.23:0/1549608127 wait complete. 2023-06-22T08:21:36.203 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:36.219 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:37.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~75s 2023-06-22T08:21:38.220 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:38.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.423+0000 7f84e03ff700 1 Processor -- start 2023-06-22T08:21:38.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.423+0000 7f84e03ff700 1 -- start start 2023-06-22T08:21:38.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.424+0000 7f84e03ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f84d815cb00 0x7f84d80c9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:38.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.424+0000 7f84e03ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d80c9540 con 0x7f84d8071c20 2023-06-22T08:21:38.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.424+0000 7f84e03ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d8158c10 con 0x7f84d8071850 2023-06-22T08:21:38.427 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.425+0000 7f84e03ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d8158ef0 con 0x7f84d815cb00 2023-06-22T08:21:38.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.425+0000 7f84de19b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f84d8071850 0x7f84d815c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:59542/0 (socket says 172.21.15.23:59542) 2023-06-22T08:21:38.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.425+0000 7f84de19b700 1 -- 172.21.15.23:0/1971322298 learned_addr learned my addr 172.21.15.23:0/1971322298 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:38.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.425+0000 7f84de99c700 1 --2- 172.21.15.23:0/1971322298 >> v2:172.21.15.23:3300/0 conn(0x7f84d815cb00 0x7f84d80c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:38.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.425+0000 7f84de99c700 1 -- 172.21.15.23:0/1971322298 >> v1:172.21.15.67:6789/0 conn(0x7f84d8071850 legacy=0x7f84d815c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:38.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.425+0000 7f84dd199700 1 -- 172.21.15.23:0/1971322298 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401081769 0 0) 0x7f84d8158c10 con 0x7f84d8071850 2023-06-22T08:21:38.429 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.425+0000 7f84de99c700 1 -- 172.21.15.23:0/1971322298 >> v1:172.21.15.23:6789/0 conn(0x7f84d8071c20 legacy=0x7f84d8152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:38.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.425+0000 7f84de99c700 1 -- 172.21.15.23:0/1971322298 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d81591d0 con 0x7f84d815cb00 2023-06-22T08:21:38.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.425+0000 7f84de99c700 1 --2- 172.21.15.23:0/1971322298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d815cb00 0x7f84d80c9830 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009d90 tx=0x7f84d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfce89ce623bf740 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:38.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.426+0000 7f84dd199700 1 -- 172.21.15.23:0/1971322298 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d000e070 con 0x7f84d815cb00 2023-06-22T08:21:38.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.426+0000 7f84dd199700 1 -- 172.21.15.23:0/1971322298 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d000bad0 con 0x7f84d815cb00 2023-06-22T08:21:38.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.426+0000 7f84dd199700 1 -- 172.21.15.23:0/1971322298 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d000be80 con 0x7f84d815cb00 2023-06-22T08:21:38.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.426+0000 7f84e03ff700 1 -- 172.21.15.23:0/1971322298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d815cb00 msgr2=0x7f84d80c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:38.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.426+0000 7f84e03ff700 1 --2- 172.21.15.23:0/1971322298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d815cb00 0x7f84d80c9830 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009d90 tx=0x7f84d000b330 comp rx=0 tx=0).stop 2023-06-22T08:21:38.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.426+0000 7f84e03ff700 1 -- 172.21.15.23:0/1971322298 shutdown_connections 2023-06-22T08:21:38.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.426+0000 7f84e03ff700 1 --2- 172.21.15.23:0/1971322298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d815cb00 0x7f84d80c9830 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:38.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.426+0000 7f84e03ff700 1 -- 172.21.15.23:0/1971322298 >> 172.21.15.23:0/1971322298 conn(0x7f84d806b890 msgr2=0x7f84d806bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:38.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.426+0000 7f84e03ff700 1 -- 172.21.15.23:0/1971322298 shutdown_connections 2023-06-22T08:21:38.431 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.427+0000 7f84e03ff700 1 -- 172.21.15.23:0/1971322298 wait complete. 2023-06-22T08:21:38.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.428+0000 7f84e03ff700 1 Processor -- start 2023-06-22T08:21:38.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.428+0000 7f84e03ff700 1 -- start start 2023-06-22T08:21:38.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.429+0000 7f84e03ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d8071850 0x7f84d80b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.429+0000 7f84e03ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d8071c20 0x7f84d81f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.429+0000 7f84e03ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d815cb00 0x7f84d814d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.429+0000 7f84e03ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d81515f0 con 0x7f84d8071c20 2023-06-22T08:21:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.429+0000 7f84e03ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d8151030 con 0x7f84d8071850 2023-06-22T08:21:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.429+0000 7f84e03ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d8151310 con 0x7f84d815cb00 2023-06-22T08:21:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84de19b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d8071850 0x7f84d80b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84de19b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d8071850 0x7f84d80b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:43284/0 (socket says 172.21.15.23:43284) 2023-06-22T08:21:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84de19b700 1 -- 172.21.15.23:0/3477490420 learned_addr learned my addr 172.21.15.23:0/3477490420 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:38.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84de99c700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d815cb00 0x7f84d814d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:38.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84dd99a700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d8071c20 0x7f84d81f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:38.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84de19b700 1 -- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d815cb00 msgr2=0x7f84d814d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:38.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84de19b700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d815cb00 0x7f84d814d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:38.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84de19b700 1 -- 172.21.15.23:0/3477490420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d8071c20 msgr2=0x7f84d81f1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:38.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84de19b700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d8071c20 0x7f84d81f1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:38.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84de19b700 1 -- 172.21.15.23:0/3477490420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d0009a70 con 0x7f84d8071850 2023-06-22T08:21:38.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84de19b700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d8071850 0x7f84d80b36d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f84d40011e0 tx=0x7f84d400ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:38.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.430+0000 7f84ceffd700 1 -- 172.21.15.23:0/3477490420 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4004c80 con 0x7f84d8071850 2023-06-22T08:21:38.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.431+0000 7f84ceffd700 1 -- 172.21.15.23:0/3477490420 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d4004e00 con 0x7f84d8071850 2023-06-22T08:21:38.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.431+0000 7f84ceffd700 1 -- 172.21.15.23:0/3477490420 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4007310 con 0x7f84d8071850 2023-06-22T08:21:38.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.431+0000 7f84e03ff700 1 -- 172.21.15.23:0/3477490420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d80c5d50 con 0x7f84d8071850 2023-06-22T08:21:38.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.431+0000 7f84e03ff700 1 -- 172.21.15.23:0/3477490420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d80c6310 con 0x7f84d8071850 2023-06-22T08:21:38.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.432+0000 7f84e03ff700 1 -- 172.21.15.23:0/3477490420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d80c5bb0 con 0x7f84d8071850 2023-06-22T08:21:38.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.433+0000 7f84ceffd700 1 -- 172.21.15.23:0/3477490420 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f84d4007470 con 0x7f84d8071850 2023-06-22T08:21:38.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.434+0000 7f84ceffd700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f84bc06ad10 0x7f84bc06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:38.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.434+0000 7f84ceffd700 1 -- 172.21.15.23:0/3477490420 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f84d40897f0 con 0x7f84d8071850 2023-06-22T08:21:38.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.434+0000 7f84dd99a700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f84bc06ad10 0x7f84bc06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:38.438 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.435+0000 7f84dd99a700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f84bc06ad10 0x7f84bc06d1c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f84d80c9300 tx=0x7f84c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.442+0000 7f84ceffd700 1 -- 172.21.15.23:0/3477490420 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f84d80c5bb0 con 0x7f84d8071850 2023-06-22T08:21:38.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.696+0000 7f84e03ff700 1 -- 172.21.15.23:0/3477490420 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84d80c5bb0 con 0x7f84d8071850 2023-06-22T08:21:38.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.697+0000 7f84ceffd700 1 -- 172.21.15.23:0/3477490420 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7f84d80c5bb0 con 0x7f84d8071850 2023-06-22T08:21:38.700 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:38.700 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:38.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 -- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f84bc06ad10 msgr2=0x7f84bc06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:38.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f84bc06ad10 0x7f84bc06d1c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f84d80c9300 tx=0x7f84c8009040 comp rx=0 tx=0).stop 2023-06-22T08:21:38.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 -- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d8071850 msgr2=0x7f84d80b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:38.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d8071850 0x7f84d80b36d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f84d40011e0 tx=0x7f84d400ca30 comp rx=0 tx=0).stop 2023-06-22T08:21:38.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 -- 172.21.15.23:0/3477490420 shutdown_connections 2023-06-22T08:21:38.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d815cb00 0x7f84d814d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:38.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f84bc06ad10 0x7f84bc06d1c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:38.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d8071850 0x7f84d80b36d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:38.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 --2- 172.21.15.23:0/3477490420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84d8071c20 0x7f84d81f1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:38.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 -- 172.21.15.23:0/3477490420 >> 172.21.15.23:0/3477490420 conn(0x7f84d806b890 msgr2=0x7f84d8152430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:38.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 -- 172.21.15.23:0/3477490420 shutdown_connections 2023-06-22T08:21:38.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:38.700+0000 7f84e03ff700 1 -- 172.21.15.23:0/3477490420 wait complete. 2023-06-22T08:21:38.705 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:38.721 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:40.722 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:40.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.930+0000 7f4c989ff700 1 Processor -- start 2023-06-22T08:21:40.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.930+0000 7f4c989ff700 1 -- start start 2023-06-22T08:21:40.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.931+0000 7f4c989ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c94152d70 0x7f4c940c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:40.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c989ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4c940c45f0 0x7f4c940c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:40.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c989ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c940c4fb0 con 0x7f4c94159330 2023-06-22T08:21:40.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c989ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c940c95e0 con 0x7f4c940c45f0 2023-06-22T08:21:40.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c989ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9415bea0 con 0x7f4c94152d70 2023-06-22T08:21:40.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c9259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c94152d70 0x7f4c940c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:40.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c8bfff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c94159330 0x7f4c940cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:59556/0 (socket says 172.21.15.23:59556) 2023-06-22T08:21:40.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c8bfff700 1 -- 172.21.15.23:0/3898268217 learned_addr learned my addr 172.21.15.23:0/3898268217 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:40.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c92d9d700 1 --2- 172.21.15.23:0/3898268217 >> v2:172.21.15.23:3300/0 conn(0x7f4c940c45f0 0x7f4c940c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:40.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c91d9b700 1 -- 172.21.15.23:0/3898268217 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062864282 0 0) 0x7f4c940c4fb0 con 0x7f4c94159330 2023-06-22T08:21:40.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c91d9b700 1 -- 172.21.15.23:0/3898268217 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c800035b0 con 0x7f4c94159330 2023-06-22T08:21:40.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c9259c700 1 -- 172.21.15.23:0/3898268217 >> v2:172.21.15.23:3300/0 conn(0x7f4c940c45f0 msgr2=0x7f4c940c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:40.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c9259c700 1 --2- 172.21.15.23:0/3898268217 >> v2:172.21.15.23:3300/0 conn(0x7f4c940c45f0 0x7f4c940c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:40.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c9259c700 1 -- 172.21.15.23:0/3898268217 >> v1:172.21.15.67:6789/0 conn(0x7f4c94159330 legacy=0x7f4c940cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:40.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c9259c700 1 -- 172.21.15.23:0/3898268217 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9415c160 con 0x7f4c94152d70 2023-06-22T08:21:40.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.932+0000 7f4c92d9d700 1 --2- 172.21.15.23:0/3898268217 >> v2:172.21.15.23:3300/0 conn(0x7f4c940c45f0 0x7f4c940c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:40.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.933+0000 7f4c9259c700 1 --2- 172.21.15.23:0/3898268217 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c94152d70 0x7f4c940c97d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009840 tx=0x7f4c84009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=564bd61f407f62b9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:40.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.933+0000 7f4c91d9b700 1 -- 172.21.15.23:0/3898268217 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c84011830 con 0x7f4c94152d70 2023-06-22T08:21:40.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.933+0000 7f4c91d9b700 1 -- 172.21.15.23:0/3898268217 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c84011990 con 0x7f4c94152d70 2023-06-22T08:21:40.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.933+0000 7f4c91d9b700 1 -- 172.21.15.23:0/3898268217 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c84011d20 con 0x7f4c94152d70 2023-06-22T08:21:40.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.933+0000 7f4c989ff700 1 -- 172.21.15.23:0/3898268217 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c94152d70 msgr2=0x7f4c940c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:40.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.933+0000 7f4c989ff700 1 --2- 172.21.15.23:0/3898268217 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c94152d70 0x7f4c940c97d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009840 tx=0x7f4c84009b50 comp rx=0 tx=0).stop 2023-06-22T08:21:40.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.933+0000 7f4c989ff700 1 -- 172.21.15.23:0/3898268217 shutdown_connections 2023-06-22T08:21:40.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.933+0000 7f4c989ff700 1 --2- 172.21.15.23:0/3898268217 >> v2:172.21.15.23:3300/0 conn(0x7f4c940c45f0 0x7f4c940c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:40.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.933+0000 7f4c989ff700 1 --2- 172.21.15.23:0/3898268217 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c94152d70 0x7f4c940c97d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:40.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.933+0000 7f4c989ff700 1 -- 172.21.15.23:0/3898268217 >> 172.21.15.23:0/3898268217 conn(0x7f4c9406ae30 msgr2=0x7f4c9406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:40.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.934+0000 7f4c989ff700 1 -- 172.21.15.23:0/3898268217 shutdown_connections 2023-06-22T08:21:40.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.934+0000 7f4c989ff700 1 -- 172.21.15.23:0/3898268217 wait complete. 2023-06-22T08:21:40.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.935+0000 7f4c989ff700 1 Processor -- start 2023-06-22T08:21:40.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.935+0000 7f4c989ff700 1 -- start start 2023-06-22T08:21:40.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.936+0000 7f4c989ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c940c45f0 0x7f4c941f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:40.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.936+0000 7f4c989ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c94152d70 0x7f4c9414eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:40.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.936+0000 7f4c989ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c94159330 0x7f4c940c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:40.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.936+0000 7f4c989ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9414b0b0 con 0x7f4c94159330 2023-06-22T08:21:40.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.936+0000 7f4c989ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c94158460 con 0x7f4c94152d70 2023-06-22T08:21:40.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.936+0000 7f4c989ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941f1c50 con 0x7f4c940c45f0 2023-06-22T08:21:40.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.936+0000 7f4c9259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c940c45f0 0x7f4c941f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:40.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.936+0000 7f4c9259c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c940c45f0 0x7f4c941f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:60790/0 (socket says 172.21.15.23:60790) 2023-06-22T08:21:40.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.936+0000 7f4c9259c700 1 -- 172.21.15.23:0/2236988286 learned_addr learned my addr 172.21.15.23:0/2236988286 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:40.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.936+0000 7f4c8bfff700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c94152d70 0x7f4c9414eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:40.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c92d9d700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c94159330 0x7f4c940c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:40.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c9259c700 1 -- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c94152d70 msgr2=0x7f4c9414eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:40.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c9259c700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c94152d70 0x7f4c9414eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:40.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c9259c700 1 -- 172.21.15.23:0/2236988286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c94159330 msgr2=0x7f4c940c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:40.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c9259c700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c94159330 0x7f4c940c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:40.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c9259c700 1 -- 172.21.15.23:0/2236988286 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c009a70 con 0x7f4c940c45f0 2023-06-22T08:21:40.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c9259c700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c940c45f0 0x7f4c941f20e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4c84000c00 tx=0x7f4c84004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:40.945 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c8affd700 1 -- 172.21.15.23:0/2236988286 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c84004550 con 0x7f4c940c45f0 2023-06-22T08:21:40.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c8affd700 1 -- 172.21.15.23:0/2236988286 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8401f070 con 0x7f4c940c45f0 2023-06-22T08:21:40.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c8affd700 1 -- 172.21.15.23:0/2236988286 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c840038c0 con 0x7f4c940c45f0 2023-06-22T08:21:40.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.937+0000 7f4c989ff700 1 -- 172.21.15.23:0/2236988286 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c840094a0 con 0x7f4c940c45f0 2023-06-22T08:21:40.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.938+0000 7f4c989ff700 1 -- 172.21.15.23:0/2236988286 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c94158990 con 0x7f4c940c45f0 2023-06-22T08:21:40.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.938+0000 7f4c989ff700 1 -- 172.21.15.23:0/2236988286 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9404ea20 con 0x7f4c940c45f0 2023-06-22T08:21:40.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.939+0000 7f4c8affd700 1 -- 172.21.15.23:0/2236988286 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4c84003a20 con 0x7f4c940c45f0 2023-06-22T08:21:40.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.940+0000 7f4c8affd700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c7406ace0 0x7f4c7406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:40.946 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.940+0000 7f4c8affd700 1 -- 172.21.15.23:0/2236988286 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f4c84021070 con 0x7f4c940c45f0 2023-06-22T08:21:40.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.941+0000 7f4c8bfff700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c7406ace0 0x7f4c7406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:40.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.941+0000 7f4c8bfff700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c7406ace0 0x7f4c7406d190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4c940ce080 tx=0x7f4c7c007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:40.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:40.945+0000 7f4c8affd700 1 -- 172.21.15.23:0/2236988286 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4c84055d10 con 0x7f4c940c45f0 2023-06-22T08:21:41.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.200+0000 7f4c989ff700 1 -- 172.21.15.23:0/2236988286 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c940c9e00 con 0x7f4c940c45f0 2023-06-22T08:21:41.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.201+0000 7f4c8affd700 1 -- 172.21.15.23:0/2236988286 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7f4c84059330 con 0x7f4c940c45f0 2023-06-22T08:21:41.204 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:41.204 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:41.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.203+0000 7f4c989ff700 1 -- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c7406ace0 msgr2=0x7f4c7406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:41.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.203+0000 7f4c989ff700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c7406ace0 0x7f4c7406d190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4c940ce080 tx=0x7f4c7c007dc0 comp rx=0 tx=0).stop 2023-06-22T08:21:41.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.203+0000 7f4c989ff700 1 -- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c940c45f0 msgr2=0x7f4c941f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:41.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.203+0000 7f4c989ff700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c940c45f0 0x7f4c941f20e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4c84000c00 tx=0x7f4c84004750 comp rx=0 tx=0).stop 2023-06-22T08:21:41.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.204+0000 7f4c989ff700 1 -- 172.21.15.23:0/2236988286 shutdown_connections 2023-06-22T08:21:41.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.204+0000 7f4c989ff700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c94159330 0x7f4c940c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:41.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.204+0000 7f4c989ff700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4c940c45f0 0x7f4c941f20e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:41.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.204+0000 7f4c989ff700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4c7406ace0 0x7f4c7406d190 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:41.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.204+0000 7f4c989ff700 1 --2- 172.21.15.23:0/2236988286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c94152d70 0x7f4c9414eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:41.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.204+0000 7f4c989ff700 1 -- 172.21.15.23:0/2236988286 >> 172.21.15.23:0/2236988286 conn(0x7f4c9406ae30 msgr2=0x7f4c9406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:41.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.204+0000 7f4c989ff700 1 -- 172.21.15.23:0/2236988286 shutdown_connections 2023-06-22T08:21:41.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:41.204+0000 7f4c989ff700 1 -- 172.21.15.23:0/2236988286 wait complete. 2023-06-22T08:21:41.209 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:41.224 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-06-22T08:21:41.226 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 169, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 169, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5707": { "addr": "172.21.15.23:6835/934828076", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 934828076, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 934828076, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5707, "incarnation": 72, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 19 }, "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 }, "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 36 }, "gid_6318": { "addr": "172.21.15.23:6834/904774922", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 904774922, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 904774922, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 151, "join_fscid": -1, "name": "c", "rank": 4, "state": "up:stopping", "state_seq": 39 }, "gid_6616": { "addr": "172.21.15.67:6837/1855579717", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1855579717, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1855579717, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6616, "incarnation": 162, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 4, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:21:14.036557+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6616, "mds_1": 5707, "mds_2": 6313, "mds_3": 5995, "mds_4": 6318 } } } ], "standbys": [] } 2023-06-22T08:21:43.228 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.441+0000 7f2ff05ff700 1 Processor -- start 2023-06-22T08:21:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.442+0000 7f2ff05ff700 1 -- start start 2023-06-22T08:21:43.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.443+0000 7f2ff05ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fe815a320 0x7f2fe80c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:43.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.443+0000 7f2ff05ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2fe806d050 0x7f2fe8160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:43.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.443+0000 7f2ff05ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe80c6a60 con 0x7f2fe80c3a40 2023-06-22T08:21:43.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.443+0000 7f2ff05ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe8157570 con 0x7f2fe806d050 2023-06-22T08:21:43.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.443+0000 7f2ff05ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe80c4350 con 0x7f2fe815a320 2023-06-22T08:21:43.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.443+0000 7f2fee39b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2fe80c3a40 0x7f2fe8159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:59572/0 (socket says 172.21.15.23:59572) 2023-06-22T08:21:43.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.443+0000 7f2fee39b700 1 -- 172.21.15.23:0/803518905 learned_addr learned my addr 172.21.15.23:0/803518905 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:43.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.443+0000 7f2feeb9c700 1 --2- 172.21.15.23:0/803518905 >> v2:172.21.15.23:3300/0 conn(0x7f2fe806d050 0x7f2fe8160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:43.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.443+0000 7f2fedb9a700 1 --2- 172.21.15.23:0/803518905 >> v2:172.21.15.67:3300/0 conn(0x7f2fe815a320 0x7f2fe80c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:43.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.444+0000 7f2fed399700 1 -- 172.21.15.23:0/803518905 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094260007 0 0) 0x7f2fe80c6a60 con 0x7f2fe80c3a40 2023-06-22T08:21:43.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.444+0000 7f2fed399700 1 -- 172.21.15.23:0/803518905 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd40035b0 con 0x7f2fe80c3a40 2023-06-22T08:21:43.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.444+0000 7f2feeb9c700 1 -- 172.21.15.23:0/803518905 >> v2:172.21.15.67:3300/0 conn(0x7f2fe815a320 msgr2=0x7f2fe80c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:43.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.444+0000 7f2feeb9c700 1 --2- 172.21.15.23:0/803518905 >> v2:172.21.15.67:3300/0 conn(0x7f2fe815a320 0x7f2fe80c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.444+0000 7f2feeb9c700 1 -- 172.21.15.23:0/803518905 >> v1:172.21.15.67:6789/0 conn(0x7f2fe80c3a40 legacy=0x7f2fe8159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.444+0000 7f2feeb9c700 1 -- 172.21.15.23:0/803518905 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe814d690 con 0x7f2fe806d050 2023-06-22T08:21:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.444+0000 7f2feeb9c700 1 --2- 172.21.15.23:0/803518905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fe806d050 0x7f2fe8160ec0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8066820 tx=0x7f2fe000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7108de0e365ef02 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.444+0000 7f2fed399700 1 -- 172.21.15.23:0/803518905 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000e070 con 0x7f2fe806d050 2023-06-22T08:21:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.444+0000 7f2fed399700 1 -- 172.21.15.23:0/803518905 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe000bb20 con 0x7f2fe806d050 2023-06-22T08:21:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.444+0000 7f2fed399700 1 -- 172.21.15.23:0/803518905 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0012430 con 0x7f2fe806d050 2023-06-22T08:21:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.445+0000 7f2ff05ff700 1 -- 172.21.15.23:0/803518905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fe806d050 msgr2=0x7f2fe8160ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.445+0000 7f2ff05ff700 1 --2- 172.21.15.23:0/803518905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fe806d050 0x7f2fe8160ec0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8066820 tx=0x7f2fe000b330 comp rx=0 tx=0).stop 2023-06-22T08:21:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.445+0000 7f2ff05ff700 1 -- 172.21.15.23:0/803518905 shutdown_connections 2023-06-22T08:21:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.445+0000 7f2ff05ff700 1 --2- 172.21.15.23:0/803518905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fe806d050 0x7f2fe8160ec0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.445+0000 7f2ff05ff700 1 --2- 172.21.15.23:0/803518905 >> v2:172.21.15.67:3300/0 conn(0x7f2fe815a320 0x7f2fe80c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.445+0000 7f2ff05ff700 1 -- 172.21.15.23:0/803518905 >> 172.21.15.23:0/803518905 conn(0x7f2fe80ca3f0 msgr2=0x7f2fe80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.445+0000 7f2ff05ff700 1 -- 172.21.15.23:0/803518905 shutdown_connections 2023-06-22T08:21:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.445+0000 7f2ff05ff700 1 -- 172.21.15.23:0/803518905 wait complete. 2023-06-22T08:21:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.447+0000 7f2ff05ff700 1 Processor -- start 2023-06-22T08:21:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.447+0000 7f2ff05ff700 1 -- start start 2023-06-22T08:21:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2ff05ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2fe806d050 0x7f2fe81ee720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2ff05ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fe80c3a40 0x7f2fe81eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2ff05ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe815a320 0x7f2fe81e5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2ff05ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81e2c70 con 0x7f2fe80c3a40 2023-06-22T08:21:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2ff05ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81e3af0 con 0x7f2fe815a320 2023-06-22T08:21:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2ff05ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81e2ee0 con 0x7f2fe806d050 2023-06-22T08:21:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2fee39b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2fe806d050 0x7f2fe81ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2feeb9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe815a320 0x7f2fe81e5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2fedb9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fe80c3a40 0x7f2fe81eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2fee39b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2fe806d050 0x7f2fe81ee720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:60798/0 (socket says 172.21.15.23:60798) 2023-06-22T08:21:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2fee39b700 1 -- 172.21.15.23:0/648276779 learned_addr learned my addr 172.21.15.23:0/648276779 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2fee39b700 1 -- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe815a320 msgr2=0x7f2fe81e5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:43.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2fee39b700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe815a320 0x7f2fe81e5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:43.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2fee39b700 1 -- 172.21.15.23:0/648276779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fe80c3a40 msgr2=0x7f2fe81eefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:43.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2fee39b700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fe80c3a40 0x7f2fe81eefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:43.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.448+0000 7f2fee39b700 1 -- 172.21.15.23:0/648276779 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0009a70 con 0x7f2fe806d050 2023-06-22T08:21:43.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.449+0000 7f2fee39b700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2fe806d050 0x7f2fe81ee720 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400c700 tx=0x7f2fe400ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:43.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.449+0000 7f2fdeffd700 1 -- 172.21.15.23:0/648276779 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4005d70 con 0x7f2fe806d050 2023-06-22T08:21:43.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.449+0000 7f2fdeffd700 1 -- 172.21.15.23:0/648276779 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe40070d0 con 0x7f2fe806d050 2023-06-22T08:21:43.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.449+0000 7f2ff05ff700 1 -- 172.21.15.23:0/648276779 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe81e31c0 con 0x7f2fe806d050 2023-06-22T08:21:43.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.449+0000 7f2fdeffd700 1 -- 172.21.15.23:0/648276779 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4007460 con 0x7f2fe806d050 2023-06-22T08:21:43.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.449+0000 7f2ff05ff700 1 -- 172.21.15.23:0/648276779 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe81e3780 con 0x7f2fe806d050 2023-06-22T08:21:43.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.450+0000 7f2ff05ff700 1 -- 172.21.15.23:0/648276779 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe806cc00 con 0x7f2fe806d050 2023-06-22T08:21:43.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.451+0000 7f2fdeffd700 1 -- 172.21.15.23:0/648276779 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2fe40075c0 con 0x7f2fe806d050 2023-06-22T08:21:43.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.452+0000 7f2fdeffd700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2fcc06ace0 0x7f2fcc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:43.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.452+0000 7f2fdeffd700 1 -- 172.21.15.23:0/648276779 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f2fe4088ff0 con 0x7f2fe806d050 2023-06-22T08:21:43.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.452+0000 7f2fedb9a700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2fcc06ace0 0x7f2fcc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:43.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.453+0000 7f2fedb9a700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2fcc06ace0 0x7f2fcc06d190 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8145180 tx=0x7f2fd8009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:43.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.456+0000 7f2fdeffd700 1 -- 172.21.15.23:0/648276779 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2fe4053980 con 0x7f2fe806d050 2023-06-22T08:21:43.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.714+0000 7f2ff05ff700 1 -- 172.21.15.23:0/648276779 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fe804ea20 con 0x7f2fe806d050 2023-06-22T08:21:43.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.715+0000 7f2fdeffd700 1 -- 172.21.15.23:0/648276779 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7f2fe4017020 con 0x7f2fe806d050 2023-06-22T08:21:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:43.718 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:43.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.717+0000 7f2ff05ff700 1 -- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2fcc06ace0 msgr2=0x7f2fcc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:43.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.717+0000 7f2ff05ff700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2fcc06ace0 0x7f2fcc06d190 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8145180 tx=0x7f2fd8009210 comp rx=0 tx=0).stop 2023-06-22T08:21:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.717+0000 7f2ff05ff700 1 -- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2fe806d050 msgr2=0x7f2fe81ee720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.717+0000 7f2ff05ff700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2fe806d050 0x7f2fe81ee720 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400c700 tx=0x7f2fe400ca10 comp rx=0 tx=0).stop 2023-06-22T08:21:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.718+0000 7f2ff05ff700 1 -- 172.21.15.23:0/648276779 shutdown_connections 2023-06-22T08:21:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.718+0000 7f2ff05ff700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f2fcc06ace0 0x7f2fcc06d190 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.718+0000 7f2ff05ff700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fe815a320 0x7f2fe81e5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.718+0000 7f2ff05ff700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2fe806d050 0x7f2fe81ee720 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.718+0000 7f2ff05ff700 1 --2- 172.21.15.23:0/648276779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fe80c3a40 0x7f2fe81eefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.718+0000 7f2ff05ff700 1 -- 172.21.15.23:0/648276779 >> 172.21.15.23:0/648276779 conn(0x7f2fe80ca3f0 msgr2=0x7f2fe8155970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.718+0000 7f2ff05ff700 1 -- 172.21.15.23:0/648276779 shutdown_connections 2023-06-22T08:21:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:43.718+0000 7f2ff05ff700 1 -- 172.21.15.23:0/648276779 wait complete. 2023-06-22T08:21:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:43.738 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:44.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~82s 2023-06-22T08:21:45.739 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:45.948 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:21:45.954 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:21:45.955 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.949+0000 7f89697ff700 1 Processor -- start 2023-06-22T08:21:45.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.949+0000 7f89697ff700 1 -- start start 2023-06-22T08:21:45.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.950+0000 7f89697ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f89640c3a40 0x7f89640c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:45.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.950+0000 7f89697ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 0x7f8964160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:45.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.950+0000 7f89697ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89640c6a60 con 0x7f896406d050 2023-06-22T08:21:45.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.950+0000 7f89697ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8964157570 con 0x7f89640c3a40 2023-06-22T08:21:45.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.950+0000 7f89697ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89640c4350 con 0x7f896415a320 2023-06-22T08:21:45.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.951+0000 7f8962ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f89640c3a40 0x7f89640c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:45.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.951+0000 7f89637fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 0x7f8964160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:45.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.951+0000 7f89627fc700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f896406d050 0x7f8964159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:44962/0 (socket says 172.21.15.23:44962) 2023-06-22T08:21:45.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.951+0000 7f89627fc700 1 -- 172.21.15.23:0/3774928034 learned_addr learned my addr 172.21.15.23:0/3774928034 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:45.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.951+0000 7f89637fe700 1 -- 172.21.15.23:0/3774928034 >> v2:172.21.15.23:3300/0 conn(0x7f89640c3a40 msgr2=0x7f89640c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:45.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.951+0000 7f89637fe700 1 --2- 172.21.15.23:0/3774928034 >> v2:172.21.15.23:3300/0 conn(0x7f89640c3a40 0x7f89640c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:45.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.951+0000 7f8961ffb700 1 -- 172.21.15.23:0/3774928034 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446496319 0 0) 0x7f89640c6a60 con 0x7f896406d050 2023-06-22T08:21:45.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.951+0000 7f89637fe700 1 -- 172.21.15.23:0/3774928034 >> v1:172.21.15.67:6789/0 conn(0x7f896406d050 legacy=0x7f8964159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:45.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.951+0000 7f89637fe700 1 -- 172.21.15.23:0/3774928034 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896414d690 con 0x7f896415a320 2023-06-22T08:21:45.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.952+0000 7f89637fe700 1 --2- 172.21.15.23:0/3774928034 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 0x7f8964160ec0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f895c009d90 tx=0x7f895c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbb7c14dcb0b984a server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:45.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.952+0000 7f8961ffb700 1 -- 172.21.15.23:0/3774928034 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c00e070 con 0x7f896415a320 2023-06-22T08:21:45.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.952+0000 7f8961ffb700 1 -- 172.21.15.23:0/3774928034 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895c00bb20 con 0x7f896415a320 2023-06-22T08:21:45.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.952+0000 7f8961ffb700 1 -- 172.21.15.23:0/3774928034 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c012430 con 0x7f896415a320 2023-06-22T08:21:45.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.952+0000 7f89697ff700 1 -- 172.21.15.23:0/3774928034 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 msgr2=0x7f8964160ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.952+0000 7f89697ff700 1 --2- 172.21.15.23:0/3774928034 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 0x7f8964160ec0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f895c009d90 tx=0x7f895c00b330 comp rx=0 tx=0).stop 2023-06-22T08:21:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.953+0000 7f89697ff700 1 -- 172.21.15.23:0/3774928034 shutdown_connections 2023-06-22T08:21:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.953+0000 7f89697ff700 1 --2- 172.21.15.23:0/3774928034 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 0x7f8964160ec0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.953+0000 7f89697ff700 1 --2- 172.21.15.23:0/3774928034 >> v2:172.21.15.23:3300/0 conn(0x7f89640c3a40 0x7f89640c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.953+0000 7f89697ff700 1 -- 172.21.15.23:0/3774928034 >> 172.21.15.23:0/3774928034 conn(0x7f89640ca3f0 msgr2=0x7f89640ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.953+0000 7f89697ff700 1 -- 172.21.15.23:0/3774928034 shutdown_connections 2023-06-22T08:21:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.953+0000 7f89697ff700 1 -- 172.21.15.23:0/3774928034 wait complete. 2023-06-22T08:21:45.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.954+0000 7f89697ff700 1 Processor -- start 2023-06-22T08:21:45.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.955+0000 7f89697ff700 1 -- start start 2023-06-22T08:21:45.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.955+0000 7f89697ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f896406d050 0x7f89641595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:45.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.955+0000 7f89697ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89640c3a40 0x7f8964140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:45.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.955+0000 7f89697ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 0x7f8964157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:45.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.955+0000 7f89697ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8964140a80 con 0x7f896406d050 2023-06-22T08:21:45.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.955+0000 7f89697ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89641f1f30 con 0x7f89640c3a40 2023-06-22T08:21:45.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.955+0000 7f89697ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89640c9dd0 con 0x7f896415a320 2023-06-22T08:21:45.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f89627fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89640c3a40 0x7f8964140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:45.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f8962ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f896406d050 0x7f89641595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:45.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f89627fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89640c3a40 0x7f8964140540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50644/0 (socket says 172.21.15.23:50644) 2023-06-22T08:21:45.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f89627fc700 1 -- 172.21.15.23:0/4042471981 learned_addr learned my addr 172.21.15.23:0/4042471981 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:45.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f89637fe700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 0x7f8964157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:45.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f89627fc700 1 -- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 msgr2=0x7f8964157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:45.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f89627fc700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 0x7f8964157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:45.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f89627fc700 1 -- 172.21.15.23:0/4042471981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f896406d050 msgr2=0x7f89641595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:45.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f89627fc700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f896406d050 0x7f89641595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:45.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f89627fc700 1 -- 172.21.15.23:0/4042471981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895c009a70 con 0x7f89640c3a40 2023-06-22T08:21:45.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.956+0000 7f89637fe700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 0x7f8964157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:45.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.957+0000 7f89627fc700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89640c3a40 0x7f8964140540 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f895800c550 tx=0x7f895800c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:45.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.957+0000 7f894b7fe700 1 -- 172.21.15.23:0/4042471981 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958004c80 con 0x7f89640c3a40 2023-06-22T08:21:45.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.957+0000 7f894b7fe700 1 -- 172.21.15.23:0/4042471981 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8958004e00 con 0x7f89640c3a40 2023-06-22T08:21:45.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.957+0000 7f89697ff700 1 -- 172.21.15.23:0/4042471981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89640ca090 con 0x7f89640c3a40 2023-06-22T08:21:45.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.957+0000 7f894b7fe700 1 -- 172.21.15.23:0/4042471981 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958007310 con 0x7f89640c3a40 2023-06-22T08:21:45.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.957+0000 7f89697ff700 1 -- 172.21.15.23:0/4042471981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89641f1600 con 0x7f89640c3a40 2023-06-22T08:21:45.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.958+0000 7f89697ff700 1 -- 172.21.15.23:0/4042471981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896404ea20 con 0x7f89640c3a40 2023-06-22T08:21:45.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.959+0000 7f894b7fe700 1 -- 172.21.15.23:0/4042471981 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8958007470 con 0x7f89640c3a40 2023-06-22T08:21:45.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.960+0000 7f894b7fe700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f895006ace0 0x7f895006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:45.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.960+0000 7f894b7fe700 1 -- 172.21.15.23:0/4042471981 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f8958089880 con 0x7f89640c3a40 2023-06-22T08:21:45.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.960+0000 7f8962ffd700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f895006ace0 0x7f895006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:45.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.961+0000 7f8962ffd700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f895006ace0 0x7f895006d190 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8954009be0 tx=0x7f8954009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:45.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:45.965+0000 7f894b7fe700 1 -- 172.21.15.23:0/4042471981 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f89580540f0 con 0x7f89640c3a40 2023-06-22T08:21:46.227 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.224+0000 7f89697ff700 1 -- 172.21.15.23:0/4042471981 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89640c5f30 con 0x7f89640c3a40 2023-06-22T08:21:46.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.224+0000 7f894b7fe700 1 -- 172.21.15.23:0/4042471981 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7f8958057710 con 0x7f89640c3a40 2023-06-22T08:21:46.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:46.228 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:46.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 -- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f895006ace0 msgr2=0x7f895006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:46.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f895006ace0 0x7f895006d190 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8954009be0 tx=0x7f8954009210 comp rx=0 tx=0).stop 2023-06-22T08:21:46.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 -- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89640c3a40 msgr2=0x7f8964140540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:46.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89640c3a40 0x7f8964140540 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f895800c550 tx=0x7f895800c860 comp rx=0 tx=0).stop 2023-06-22T08:21:46.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 -- 172.21.15.23:0/4042471981 shutdown_connections 2023-06-22T08:21:46.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f896415a320 0x7f8964157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:46.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f896406d050 0x7f89641595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:46.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f895006ace0 0x7f895006d190 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:46.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 --2- 172.21.15.23:0/4042471981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89640c3a40 0x7f8964140540 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:46.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 -- 172.21.15.23:0/4042471981 >> 172.21.15.23:0/4042471981 conn(0x7f89640ca3f0 msgr2=0x7f8964156100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:46.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 -- 172.21.15.23:0/4042471981 shutdown_connections 2023-06-22T08:21:46.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:46.227+0000 7f89697ff700 1 -- 172.21.15.23:0/4042471981 wait complete. 2023-06-22T08:21:46.234 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:46.248 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:48.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:48.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.457+0000 7fa403bff700 1 Processor -- start 2023-06-22T08:21:48.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.457+0000 7fa403bff700 1 -- start start 2023-06-22T08:21:48.463 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.458+0000 7fa403bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3fc0cb6b0 0x7fa3fc069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:48.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa403bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3fc152e10 0x7fa3fc1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:48.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa403bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3fc158f60 con 0x7fa3fc0c5740 2023-06-22T08:21:48.464 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa403bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc06bd20 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa403bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3fc0cc450 con 0x7fa3fc152e10 2023-06-22T08:21:48.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa40199b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3fc0cb6b0 0x7fa3fc069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:48.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa40119a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa3fc0c5740 0x7fa3fc152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:44974/0 (socket says 172.21.15.23:44974) 2023-06-22T08:21:48.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa40119a700 1 -- 172.21.15.23:0/1788383179 learned_addr learned my addr 172.21.15.23:0/1788383179 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:48.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa40219c700 1 --2- 172.21.15.23:0/1788383179 >> v2:172.21.15.67:3300/0 conn(0x7fa3fc152e10 0x7fa3fc1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:48.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa40199b700 1 -- 172.21.15.23:0/1788383179 >> v2:172.21.15.67:3300/0 conn(0x7fa3fc152e10 msgr2=0x7fa3fc1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:48.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa40199b700 1 --2- 172.21.15.23:0/1788383179 >> v2:172.21.15.67:3300/0 conn(0x7fa3fc152e10 0x7fa3fc1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:48.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa40199b700 1 -- 172.21.15.23:0/1788383179 >> v1:172.21.15.67:6789/0 conn(0x7fa3fc0c5740 legacy=0x7fa3fc152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:48.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa40199b700 1 -- 172.21.15.23:0/1788383179 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc158980 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.459+0000 7fa400999700 1 -- 172.21.15.23:0/1788383179 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042334681 0 0) 0x7fa3fc158f60 con 0x7fa3fc0c5740 2023-06-22T08:21:48.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.460+0000 7fa40199b700 1 --2- 172.21.15.23:0/1788383179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 0x7fa3fc069eb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0c2ed0 tx=0x7fa3f8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cdd31720852842 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:48.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.460+0000 7fa400999700 1 -- 172.21.15.23:0/1788383179 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8011830 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.460+0000 7fa400999700 1 -- 172.21.15.23:0/1788383179 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f8011990 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.460+0000 7fa400999700 1 -- 172.21.15.23:0/1788383179 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8011d20 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.460+0000 7fa403bff700 1 -- 172.21.15.23:0/1788383179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 msgr2=0x7fa3fc069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:48.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.460+0000 7fa403bff700 1 --2- 172.21.15.23:0/1788383179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 0x7fa3fc069eb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0c2ed0 tx=0x7fa3f8009b20 comp rx=0 tx=0).stop 2023-06-22T08:21:48.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.460+0000 7fa403bff700 1 -- 172.21.15.23:0/1788383179 shutdown_connections 2023-06-22T08:21:48.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.460+0000 7fa403bff700 1 --2- 172.21.15.23:0/1788383179 >> v2:172.21.15.67:3300/0 conn(0x7fa3fc152e10 0x7fa3fc1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:48.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.460+0000 7fa403bff700 1 --2- 172.21.15.23:0/1788383179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 0x7fa3fc069eb0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:48.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.460+0000 7fa403bff700 1 -- 172.21.15.23:0/1788383179 >> 172.21.15.23:0/1788383179 conn(0x7fa3fc070d80 msgr2=0x7fa3fc071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:48.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.461+0000 7fa403bff700 1 -- 172.21.15.23:0/1788383179 shutdown_connections 2023-06-22T08:21:48.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.461+0000 7fa403bff700 1 -- 172.21.15.23:0/1788383179 wait complete. 2023-06-22T08:21:48.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.462+0000 7fa403bff700 1 Processor -- start 2023-06-22T08:21:48.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.463+0000 7fa403bff700 1 -- start start 2023-06-22T08:21:48.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.463+0000 7fa403bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 0x7fa3fc151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:48.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.463+0000 7fa403bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc152e10 0x7fa3fc0c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:48.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.463+0000 7fa403bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3fc0c70c0 0x7fa3fc150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:48.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.463+0000 7fa403bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c4110 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.463+0000 7fa403bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc1eec20 con 0x7fa3fc152e10 2023-06-22T08:21:48.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.463+0000 7fa403bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc0c7830 con 0x7fa3fc0c70c0 2023-06-22T08:21:48.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40199b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 0x7fa3fc151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:48.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40199b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 0x7fa3fc151a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41246/0 (socket says 172.21.15.23:41246) 2023-06-22T08:21:48.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40199b700 1 -- 172.21.15.23:0/1428639574 learned_addr learned my addr 172.21.15.23:0/1428639574 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:48.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40219c700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3fc0c70c0 0x7fa3fc150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:48.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40119a700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc152e10 0x7fa3fc0c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:48.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40199b700 1 -- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3fc0c70c0 msgr2=0x7fa3fc150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:48.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40199b700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3fc0c70c0 0x7fa3fc150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:48.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40199b700 1 -- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc152e10 msgr2=0x7fa3fc0c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:48.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40199b700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc152e10 0x7fa3fc0c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:48.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40199b700 1 -- 172.21.15.23:0/1428639574 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f80094a0 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.464+0000 7fa40199b700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 0x7fa3fc151a80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0708f0 tx=0x7fa3f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:48.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.465+0000 7fa3f27fc700 1 -- 172.21.15.23:0/1428639574 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80036a0 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.465+0000 7fa3f27fc700 1 -- 172.21.15.23:0/1428639574 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f801f070 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.465+0000 7fa3f27fc700 1 -- 172.21.15.23:0/1428639574 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8003a60 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.465+0000 7fa403bff700 1 -- 172.21.15.23:0/1428639574 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc1ef0b0 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.465+0000 7fa403bff700 1 -- 172.21.15.23:0/1428639574 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc1ee2f0 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.466+0000 7fa403bff700 1 -- 172.21.15.23:0/1428639574 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc04ea20 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.467+0000 7fa3f27fc700 1 -- 172.21.15.23:0/1428639574 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa3f8003bc0 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.468+0000 7fa3f27fc700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3e006ad10 0x7fa3e006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:48.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.469+0000 7fa3f27fc700 1 -- 172.21.15.23:0/1428639574 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7871+0+0 (secure 0 0 0) 0x7fa3f8021070 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.469+0000 7fa40119a700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3e006ad10 0x7fa3e006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:48.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.469+0000 7fa40119a700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3e006ad10 0x7fa3e006d1c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0c5180 tx=0x7fa3ec007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:48.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.473+0000 7fa3f27fc700 1 -- 172.21.15.23:0/1428639574 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa3f8055c90 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.732 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.730+0000 7fa403bff700 1 -- 172.21.15.23:0/1428639574 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3fc0c6960 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.733 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.730+0000 7fa3f27fc700 1 -- 172.21.15.23:0/1428639574 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5522 (secure 0 0 0) 0x7fa3f80592b0 con 0x7fa3fc0cb6b0 2023-06-22T08:21:48.733 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:48.733 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:14.036557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3,4],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995,"mds_4":6318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6318":{"gid":6318,"name":"c","rank":4,"incarnation":151,"state":"up:stopping","state_seq":39,"addr":"172.21.15.23:6834/904774922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":904774922},{"type":"v1","addr":"172.21.15.23:6834","nonce":904774922}]},"join_fscid":-1,"export_targets":[0,1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:48.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.732+0000 7fa403bff700 1 -- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3e006ad10 msgr2=0x7fa3e006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:48.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.732+0000 7fa403bff700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3e006ad10 0x7fa3e006d1c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0c5180 tx=0x7fa3ec007de0 comp rx=0 tx=0).stop 2023-06-22T08:21:48.735 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.732+0000 7fa403bff700 1 -- 172.21.15.23:0/1428639574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 msgr2=0x7fa3fc151a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:48.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.732+0000 7fa403bff700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 0x7fa3fc151a80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0708f0 tx=0x7fa3f8004750 comp rx=0 tx=0).stop 2023-06-22T08:21:48.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.732+0000 7fa403bff700 1 -- 172.21.15.23:0/1428639574 shutdown_connections 2023-06-22T08:21:48.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.732+0000 7fa403bff700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3fc0c70c0 0x7fa3fc150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:48.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.732+0000 7fa403bff700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3fc0cb6b0 0x7fa3fc151a80 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:48.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.732+0000 7fa403bff700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3e006ad10 0x7fa3e006d1c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:48.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.732+0000 7fa403bff700 1 --2- 172.21.15.23:0/1428639574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc152e10 0x7fa3fc0c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:48.736 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.732+0000 7fa403bff700 1 -- 172.21.15.23:0/1428639574 >> 172.21.15.23:0/1428639574 conn(0x7fa3fc070d80 msgr2=0x7fa3fc156530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:48.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.733+0000 7fa403bff700 1 -- 172.21.15.23:0/1428639574 shutdown_connections 2023-06-22T08:21:48.737 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:48.733+0000 7fa403bff700 1 -- 172.21.15.23:0/1428639574 wait complete. 2023-06-22T08:21:48.737 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 169 2023-06-22T08:21:48.753 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:21:49.805 INFO:tasks.workunit.client.0.smithi023.stderr:+ ls -al .snap/1/a .snap/2/a .snap/3/a .snap/4/a .snap/5/a 2023-06-22T08:21:49.809 INFO:tasks.workunit.client.0.smithi023.stdout:-rw-rw-r--. 1 ubuntu ubuntu 5 Jun 22 08:21 .snap/1/a 2023-06-22T08:21:49.809 INFO:tasks.workunit.client.0.smithi023.stdout:-rwxrwxrwx. 1 ubuntu ubuntu 5 Jun 22 08:21 .snap/2/a 2023-06-22T08:21:49.809 INFO:tasks.workunit.client.0.smithi023.stdout:-rwxrwxrwx. 1 ubuntu ubuntu 5 Jun 22 08:21 .snap/3/a 2023-06-22T08:21:49.809 INFO:tasks.workunit.client.0.smithi023.stdout:-rw-rw-rw-. 1 ubuntu ubuntu 5 Jun 22 08:21 .snap/4/a 2023-06-22T08:21:49.810 INFO:tasks.workunit.client.0.smithi023.stdout:-rw-rw-rw-. 1 ubuntu ubuntu 5 Jun 22 08:21 .snap/5/a 2023-06-22T08:21:49.810 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep asdf .snap/1/a 2023-06-22T08:21:49.813 INFO:tasks.workunit.client.0.smithi023.stdout:asdf 2023-06-22T08:21:49.813 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/1/a 2023-06-22T08:21:49.814 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'Size: 5' 2023-06-22T08:21:49.816 INFO:tasks.workunit.client.0.smithi023.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-06-22T08:21:49.817 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep asdf .snap/2/a 2023-06-22T08:21:49.820 INFO:tasks.workunit.client.0.smithi023.stdout:asdf 2023-06-22T08:21:49.821 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/2/a 2023-06-22T08:21:49.821 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'Size: 5' 2023-06-22T08:21:49.823 INFO:tasks.workunit.client.0.smithi023.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-06-22T08:21:49.824 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/2/a 2023-06-22T08:21:49.825 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep -- -rwxrwxrwx 2023-06-22T08:21:49.828 INFO:tasks.workunit.client.0.smithi023.stdout:Access: (0777/-rwxrwxrwx) Uid: ( 1000/ ubuntu) Gid: ( 1265/ ubuntu) 2023-06-22T08:21:49.828 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep qwer .snap/3/a 2023-06-22T08:21:49.831 INFO:tasks.workunit.client.0.smithi023.stdout:qwer 2023-06-22T08:21:49.833 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/3/a 2023-06-22T08:21:49.833 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'Size: 5' 2023-06-22T08:21:49.834 INFO:tasks.workunit.client.0.smithi023.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-06-22T08:21:49.835 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/3/a 2023-06-22T08:21:49.837 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep -- -rwxrwxrwx 2023-06-22T08:21:49.838 INFO:tasks.workunit.client.0.smithi023.stdout:Access: (0777/-rwxrwxrwx) Uid: ( 1000/ ubuntu) Gid: ( 1265/ ubuntu) 2023-06-22T08:21:49.838 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep qwer .snap/4/a 2023-06-22T08:21:49.842 INFO:tasks.workunit.client.0.smithi023.stdout:qwer 2023-06-22T08:21:49.843 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/4/a 2023-06-22T08:21:49.843 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'Size: 5' 2023-06-22T08:21:49.845 INFO:tasks.workunit.client.0.smithi023.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-06-22T08:21:49.846 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/4/a 2023-06-22T08:21:49.847 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep -- -rw-rw-rw- 2023-06-22T08:21:49.849 INFO:tasks.workunit.client.0.smithi023.stdout:Access: (0666/-rw-rw-rw-) Uid: ( 1000/ ubuntu) Gid: ( 1265/ ubuntu) 2023-06-22T08:21:49.850 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep zxcv .snap/5/a 2023-06-22T08:21:49.853 INFO:tasks.workunit.client.0.smithi023.stdout:zxcv 2023-06-22T08:21:49.854 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/5/a 2023-06-22T08:21:49.854 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep 'Size: 5' 2023-06-22T08:21:49.856 INFO:tasks.workunit.client.0.smithi023.stdout: Size: 5 Blocks: 1 IO Block: 4194304 regular file 2023-06-22T08:21:49.857 INFO:tasks.workunit.client.0.smithi023.stderr:+ stat .snap/5/a 2023-06-22T08:21:49.858 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep -- -rw-rw-rw- 2023-06-22T08:21:49.860 INFO:tasks.workunit.client.0.smithi023.stdout:Access: (0666/-rw-rw-rw-) Uid: ( 1000/ ubuntu) Gid: ( 1265/ ubuntu) 2023-06-22T08:21:49.862 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/1 .snap/2 .snap/3 .snap/4 .snap/5 2023-06-22T08:21:49.877 INFO:tasks.ceph.mds.c.smithi023.stdout:starting mds.c at 2023-06-22T08:21:49.903 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:21:49.904 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:21:49.905 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:21:49.905 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:21:49.986 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-name-limits.sh... 2023-06-22T08:21:49.986 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-name-limits.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-name-limits.sh 2023-06-22T08:21:50.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:50.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.958+0000 7fb5e71ff700 1 Processor -- start 2023-06-22T08:21:50.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.958+0000 7fb5e71ff700 1 -- start start 2023-06-22T08:21:50.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.959+0000 7fb5e71ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5e00c97d0 0x7fb5e0159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:50.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.959+0000 7fb5e71ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00c9c30 con 0x7fb5e00c45f0 2023-06-22T08:21:50.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.959+0000 7fb5e71ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e015bea0 con 0x7fb5e0152d70 2023-06-22T08:21:50.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.959+0000 7fb5e71ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e00c95e0 con 0x7fb5e00c97d0 2023-06-22T08:21:50.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.959+0000 7fb5dffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5e00c97d0 0x7fb5e0159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:50.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.959+0000 7fb5e4f9b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb5e0152d70 0x7fb5e00cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:44984/0 (socket says 172.21.15.23:44984) 2023-06-22T08:21:50.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.960+0000 7fb5dffff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5e00c97d0 0x7fb5e0159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50684/0 (socket says 172.21.15.23:50684) 2023-06-22T08:21:50.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.960+0000 7fb5e4f9b700 1 -- 172.21.15.23:0/1142574688 learned_addr learned my addr 172.21.15.23:0/1142574688 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:50.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.960+0000 7fb5df7fe700 1 -- 172.21.15.23:0/1142574688 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089820752 0 0) 0x7fb5e015bea0 con 0x7fb5e0152d70 2023-06-22T08:21:50.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.960+0000 7fb5df7fe700 1 -- 172.21.15.23:0/1142574688 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d00035b0 con 0x7fb5e0152d70 2023-06-22T08:21:50.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.960+0000 7fb5df7fe700 1 -- 172.21.15.23:0/1142574688 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445746186 0 0) 0x7fb5e00c9c30 con 0x7fb5e00c45f0 2023-06-22T08:21:50.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.960+0000 7fb5dffff700 1 -- 172.21.15.23:0/1142574688 >> v1:172.21.15.67:6789/0 conn(0x7fb5e0152d70 legacy=0x7fb5e00cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:50.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.960+0000 7fb5dffff700 1 -- 172.21.15.23:0/1142574688 >> v1:172.21.15.23:6789/0 conn(0x7fb5e00c45f0 legacy=0x7fb5e00c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:50.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.960+0000 7fb5dffff700 1 -- 172.21.15.23:0/1142574688 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e015c160 con 0x7fb5e00c97d0 2023-06-22T08:21:50.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.960+0000 7fb5dffff700 1 --2- 172.21.15.23:0/1142574688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c97d0 0x7fb5e0159330 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009840 tx=0x7fb5cc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9e2cc2daa0e0fb2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:50.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.960+0000 7fb5df7fe700 1 -- 172.21.15.23:0/1142574688 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc011830 con 0x7fb5e00c97d0 2023-06-22T08:21:50.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.961+0000 7fb5df7fe700 1 -- 172.21.15.23:0/1142574688 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5cc011990 con 0x7fb5e00c97d0 2023-06-22T08:21:50.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.961+0000 7fb5df7fe700 1 -- 172.21.15.23:0/1142574688 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc011d20 con 0x7fb5e00c97d0 2023-06-22T08:21:50.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.961+0000 7fb5e71ff700 1 -- 172.21.15.23:0/1142574688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c97d0 msgr2=0x7fb5e0159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:50.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.961+0000 7fb5e71ff700 1 --2- 172.21.15.23:0/1142574688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c97d0 0x7fb5e0159330 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009840 tx=0x7fb5cc009b50 comp rx=0 tx=0).stop 2023-06-22T08:21:50.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.961+0000 7fb5e71ff700 1 -- 172.21.15.23:0/1142574688 shutdown_connections 2023-06-22T08:21:50.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.961+0000 7fb5e71ff700 1 --2- 172.21.15.23:0/1142574688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c97d0 0x7fb5e0159330 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:50.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.961+0000 7fb5e71ff700 1 -- 172.21.15.23:0/1142574688 >> 172.21.15.23:0/1142574688 conn(0x7fb5e006ae30 msgr2=0x7fb5e006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:50.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.961+0000 7fb5e71ff700 1 -- 172.21.15.23:0/1142574688 shutdown_connections 2023-06-22T08:21:50.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.961+0000 7fb5e71ff700 1 -- 172.21.15.23:0/1142574688 wait complete. 2023-06-22T08:21:50.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.963+0000 7fb5e71ff700 1 Processor -- start 2023-06-22T08:21:50.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.963+0000 7fb5e71ff700 1 -- start start 2023-06-22T08:21:50.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5e71ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c45f0 0x7fb5e01583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:50.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5e71ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e00c97d0 0x7fb5e00b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5e71ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5e0152d70 0x7fb5e00c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5e71ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0157d20 con 0x7fb5e00c97d0 2023-06-22T08:21:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5e71ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01f2230 con 0x7fb5e00c45f0 2023-06-22T08:21:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5e71ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00c8a30 con 0x7fb5e0152d70 2023-06-22T08:21:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5dffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e00c97d0 0x7fb5e00b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5e4f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c45f0 0x7fb5e01583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5e4f9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c45f0 0x7fb5e01583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50694/0 (socket says 172.21.15.23:50694) 2023-06-22T08:21:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5dffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e00c97d0 0x7fb5e00b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41254/0 (socket says 172.21.15.23:41254) 2023-06-22T08:21:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5e4f9b700 1 -- 172.21.15.23:0/3655181124 learned_addr learned my addr 172.21.15.23:0/3655181124 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.964+0000 7fb5e579c700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5e0152d70 0x7fb5e00c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5dffff700 1 -- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5e0152d70 msgr2=0x7fb5e00c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5dffff700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5e0152d70 0x7fb5e00c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5dffff700 1 -- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c45f0 msgr2=0x7fb5e01583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5dffff700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c45f0 0x7fb5e01583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5dffff700 1 -- 172.21.15.23:0/3655181124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc0094a0 con 0x7fb5e00c97d0 2023-06-22T08:21:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5e4f9b700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c45f0 0x7fb5e01583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5dffff700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e00c97d0 0x7fb5e00b36b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc016000 tx=0x7fb5cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5dd7fa700 1 -- 172.21.15.23:0/3655181124 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc01a7e0 con 0x7fb5e00c97d0 2023-06-22T08:21:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5dd7fa700 1 -- 172.21.15.23:0/3655181124 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5cc01a960 con 0x7fb5e00c97d0 2023-06-22T08:21:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5e71ff700 1 -- 172.21.15.23:0/3655181124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e01f1340 con 0x7fb5e00c97d0 2023-06-22T08:21:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5dd7fa700 1 -- 172.21.15.23:0/3655181124 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc01ad10 con 0x7fb5e00c97d0 2023-06-22T08:21:50.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.965+0000 7fb5e71ff700 1 -- 172.21.15.23:0/3655181124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e01f1900 con 0x7fb5e00c97d0 2023-06-22T08:21:50.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.966+0000 7fb5e71ff700 1 -- 172.21.15.23:0/3655181124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e00c68a0 con 0x7fb5e00c97d0 2023-06-22T08:21:50.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.967+0000 7fb5dd7fa700 1 -- 172.21.15.23:0/3655181124 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb5cc022070 con 0x7fb5e00c97d0 2023-06-22T08:21:50.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.968+0000 7fb5dd7fa700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb5c406ace0 0x7fb5c406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:50.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.968+0000 7fb5dd7fa700 1 -- 172.21.15.23:0/3655181124 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7871+0+0 (secure 0 0 0) 0x7fb5cc08aaf0 con 0x7fb5e00c97d0 2023-06-22T08:21:50.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.969+0000 7fb5e4f9b700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb5c406ace0 0x7fb5c406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:50.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.969+0000 7fb5e4f9b700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb5c406ace0 0x7fb5c406d190 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4005910 tx=0x7fb5d4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:50.975 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:50.973+0000 7fb5dd7fa700 1 -- 172.21.15.23:0/3655181124 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb5e00c68a0 con 0x7fb5e00c97d0 2023-06-22T08:21:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.230+0000 7fb5e71ff700 1 -- 172.21.15.23:0/3655181124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5e00c68a0 con 0x7fb5e00c97d0 2023-06-22T08:21:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.231+0000 7fb5dd7fa700 1 -- 172.21.15.23:0/3655181124 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+5498 (secure 0 0 0) 0x7fb5e00c68a0 con 0x7fb5e00c97d0 2023-06-22T08:21:51.234 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:51.234 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6730,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:49.783257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:51.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 -- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb5c406ace0 msgr2=0x7fb5c406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:51.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb5c406ace0 0x7fb5c406d190 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4005910 tx=0x7fb5d4009040 comp rx=0 tx=0).stop 2023-06-22T08:21:51.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 -- 172.21.15.23:0/3655181124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e00c97d0 msgr2=0x7fb5e00b36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:51.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e00c97d0 0x7fb5e00b36b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc016000 tx=0x7fb5cc004750 comp rx=0 tx=0).stop 2023-06-22T08:21:51.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 -- 172.21.15.23:0/3655181124 shutdown_connections 2023-06-22T08:21:51.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb5e0152d70 0x7fb5e00c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb5c406ace0 0x7fb5c406d190 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e00c45f0 0x7fb5e01583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 --2- 172.21.15.23:0/3655181124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e00c97d0 0x7fb5e00b36b0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 -- 172.21.15.23:0/3655181124 >> 172.21.15.23:0/3655181124 conn(0x7fb5e006ae30 msgr2=0x7fb5e014fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:51.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 -- 172.21.15.23:0/3655181124 shutdown_connections 2023-06-22T08:21:51.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.233+0000 7fb5e71ff700 1 -- 172.21.15.23:0/3655181124 wait complete. 2023-06-22T08:21:51.238 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 171 2023-06-22T08:21:51.255 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 4 alive and active, now stable! 2023-06-22T08:21:51.255 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2023-06-22T08:21:51.256 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-22T08:21:51.256 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:21:51.289 INFO:tasks.ceph.mds.a:Stopped 2023-06-22T08:21:51.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:51.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~89s 2023-06-22T08:21:51.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.527+0000 7f43339ff700 1 Processor -- start 2023-06-22T08:21:51.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.527+0000 7f43339ff700 1 -- start start 2023-06-22T08:21:51.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.528+0000 7f43339ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f432c0c4d00 0x7f432c159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:51.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.528+0000 7f43339ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432c0c9c30 con 0x7f432c152d70 2023-06-22T08:21:51.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.528+0000 7f43339ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432c15bea0 con 0x7f432c0c97d0 2023-06-22T08:21:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.528+0000 7f43339ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432c0c95e0 con 0x7f432c0c4d00 2023-06-22T08:21:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.529+0000 7f4331f9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f432c0c4d00 0x7f432c159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.529+0000 7f433179b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f432c152d70 0x7f432c0cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41398/0 (socket says 172.21.15.23:41398) 2023-06-22T08:21:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.529+0000 7f433179b700 1 -- 172.21.15.23:0/1355628489 learned_addr learned my addr 172.21.15.23:0/1355628489 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.529+0000 7f4331f9c700 1 -- 172.21.15.23:0/1355628489 >> v1:172.21.15.67:6789/0 conn(0x7f432c0c97d0 legacy=0x7f432c0c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.529+0000 7f4323fff700 1 -- 172.21.15.23:0/1355628489 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806823045 0 0) 0x7f432c15bea0 con 0x7f432c0c97d0 2023-06-22T08:21:51.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.529+0000 7f4331f9c700 1 -- 172.21.15.23:0/1355628489 >> v1:172.21.15.23:6789/0 conn(0x7f432c152d70 legacy=0x7f432c0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:51.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.529+0000 7f4331f9c700 1 -- 172.21.15.23:0/1355628489 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c15c160 con 0x7f432c0c4d00 2023-06-22T08:21:51.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.529+0000 7f4331f9c700 1 --2- 172.21.15.23:0/1355628489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f432c0c4d00 0x7f432c159330 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4324009d90 tx=0x7f432400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b5147f1bd7af3c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:51.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.530+0000 7f4323fff700 1 -- 172.21.15.23:0/1355628489 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432400e070 con 0x7f432c0c4d00 2023-06-22T08:21:51.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.530+0000 7f4323fff700 1 -- 172.21.15.23:0/1355628489 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432400bad0 con 0x7f432c0c4d00 2023-06-22T08:21:51.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.530+0000 7f4323fff700 1 -- 172.21.15.23:0/1355628489 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432400be80 con 0x7f432c0c4d00 2023-06-22T08:21:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.530+0000 7f43339ff700 1 -- 172.21.15.23:0/1355628489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f432c0c4d00 msgr2=0x7f432c159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.530+0000 7f43339ff700 1 --2- 172.21.15.23:0/1355628489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f432c0c4d00 0x7f432c159330 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4324009d90 tx=0x7f432400b330 comp rx=0 tx=0).stop 2023-06-22T08:21:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.530+0000 7f43339ff700 1 -- 172.21.15.23:0/1355628489 shutdown_connections 2023-06-22T08:21:51.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.530+0000 7f43339ff700 1 --2- 172.21.15.23:0/1355628489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f432c0c4d00 0x7f432c159330 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.530+0000 7f43339ff700 1 -- 172.21.15.23:0/1355628489 >> 172.21.15.23:0/1355628489 conn(0x7f432c06ae30 msgr2=0x7f432c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.530+0000 7f43339ff700 1 -- 172.21.15.23:0/1355628489 shutdown_connections 2023-06-22T08:21:51.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.531+0000 7f43339ff700 1 -- 172.21.15.23:0/1355628489 wait complete. 2023-06-22T08:21:51.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.533+0000 7f43339ff700 1 Processor -- start 2023-06-22T08:21:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.533+0000 7f43339ff700 1 -- start start 2023-06-22T08:21:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.534+0000 7f43339ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f432c0c4d00 0x7f432c154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.534+0000 7f43339ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f432c0c97d0 0x7f432c14e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.534+0000 7f43339ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f432c152d70 0x7f432c0b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.534+0000 7f43339ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c1f2830 con 0x7f432c152d70 2023-06-22T08:21:51.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.534+0000 7f43339ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c1ee580 con 0x7f432c0c4d00 2023-06-22T08:21:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.534+0000 7f43339ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c1ee860 con 0x7f432c0c97d0 2023-06-22T08:21:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f433179b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f432c0c4d00 0x7f432c154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f4331f9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f432c152d70 0x7f432c0b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:51.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f4330f9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f432c0c97d0 0x7f432c14e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:51.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f433179b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f432c0c4d00 0x7f432c154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50710/0 (socket says 172.21.15.23:50710) 2023-06-22T08:21:51.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f433179b700 1 -- 172.21.15.23:0/4071527029 learned_addr learned my addr 172.21.15.23:0/4071527029 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:51.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f433179b700 1 -- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f432c0c97d0 msgr2=0x7f432c14e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:51.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f433179b700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f432c0c97d0 0x7f432c14e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f433179b700 1 -- 172.21.15.23:0/4071527029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f432c152d70 msgr2=0x7f432c0b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:51.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f433179b700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f432c152d70 0x7f432c0b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f433179b700 1 -- 172.21.15.23:0/4071527029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4324009a70 con 0x7f432c0c4d00 2023-06-22T08:21:51.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f4331f9c700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f432c152d70 0x7f432c0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:51.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.535+0000 7f433179b700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f432c0c4d00 0x7f432c154230 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f432800c780 tx=0x7f432800ca90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:51.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.536+0000 7f4321ffb700 1 -- 172.21.15.23:0/4071527029 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4328004c80 con 0x7f432c0c4d00 2023-06-22T08:21:51.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.536+0000 7f4321ffb700 1 -- 172.21.15.23:0/4071527029 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4328004de0 con 0x7f432c0c4d00 2023-06-22T08:21:51.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.536+0000 7f43339ff700 1 -- 172.21.15.23:0/4071527029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c1eeb40 con 0x7f432c0c4d00 2023-06-22T08:21:51.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.536+0000 7f4321ffb700 1 -- 172.21.15.23:0/4071527029 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43280072f0 con 0x7f432c0c4d00 2023-06-22T08:21:51.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.536+0000 7f43339ff700 1 -- 172.21.15.23:0/4071527029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c1ef100 con 0x7f432c0c4d00 2023-06-22T08:21:51.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.537+0000 7f43339ff700 1 -- 172.21.15.23:0/4071527029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c04ea20 con 0x7f432c0c4d00 2023-06-22T08:21:51.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.538+0000 7f4321ffb700 1 -- 172.21.15.23:0/4071527029 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4328007450 con 0x7f432c0c4d00 2023-06-22T08:21:51.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.539+0000 7f4321ffb700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431006ad60 0x7f431006d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:51.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.539+0000 7f4321ffb700 1 -- 172.21.15.23:0/4071527029 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f4328089790 con 0x7f432c0c4d00 2023-06-22T08:21:51.547 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.539+0000 7f4330f9a700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431006ad60 0x7f431006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:51.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.539+0000 7f4330f9a700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431006ad60 0x7f431006d210 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f432c0cde10 tx=0x7f431c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:51.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.543+0000 7f4321ffb700 1 -- 172.21.15.23:0/4071527029 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f43280541a0 con 0x7f432c0c4d00 2023-06-22T08:21:51.797 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.794+0000 7f43339ff700 1 -- 172.21.15.23:0/4071527029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f432c0c9e00 con 0x7f432c0c4d00 2023-06-22T08:21:51.798 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.795+0000 7f4321ffb700 1 -- 172.21.15.23:0/4071527029 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+5498 (secure 0 0 0) 0x7f43280577c0 con 0x7f432c0c4d00 2023-06-22T08:21:51.798 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:51.798 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6730,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:49.783257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:51.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 -- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431006ad60 msgr2=0x7f431006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:51.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431006ad60 0x7f431006d210 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f432c0cde10 tx=0x7f431c009040 comp rx=0 tx=0).stop 2023-06-22T08:21:51.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 -- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f432c0c4d00 msgr2=0x7f432c154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:51.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f432c0c4d00 0x7f432c154230 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f432800c780 tx=0x7f432800ca90 comp rx=0 tx=0).stop 2023-06-22T08:21:51.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 -- 172.21.15.23:0/4071527029 shutdown_connections 2023-06-22T08:21:51.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f432c152d70 0x7f432c0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f431006ad60 0x7f431006d210 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f432c0c4d00 0x7f432c154230 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 --2- 172.21.15.23:0/4071527029 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f432c0c97d0 0x7f432c14e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:51.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 -- 172.21.15.23:0/4071527029 >> 172.21.15.23:0/4071527029 conn(0x7f432c06ae30 msgr2=0x7f432c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:51.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 -- 172.21.15.23:0/4071527029 shutdown_connections 2023-06-22T08:21:51.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:51.798+0000 7f43339ff700 1 -- 172.21.15.23:0/4071527029 wait complete. 2023-06-22T08:21:51.803 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 171 2023-06-22T08:21:51.820 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:21:53.820 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:54.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.031+0000 7f07af1ff700 1 Processor -- start 2023-06-22T08:21:54.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.031+0000 7f07af1ff700 1 -- start start 2023-06-22T08:21:54.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.032+0000 7f07af1ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f07a80c9fd0 0x7f07a80cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:54.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.032+0000 7f07af1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07a80c40b0 0x7f07a80c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:54.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.032+0000 7f07af1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a815c370 con 0x7f07a8152ce0 2023-06-22T08:21:54.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.032+0000 7f07af1ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a81530b0 con 0x7f07a80c9fd0 2023-06-22T08:21:54.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.032+0000 7f07af1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80cf6f0 con 0x7f07a80c40b0 2023-06-22T08:21:54.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.032+0000 7f07a7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f07a80c9fd0 0x7f07a80cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:54.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.032+0000 7f07a7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f07a80c9fd0 0x7f07a80cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41282/0 (socket says 172.21.15.23:41282) 2023-06-22T08:21:54.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.033+0000 7f07a7fff700 1 -- 172.21.15.23:0/2828864019 learned_addr learned my addr 172.21.15.23:0/2828864019 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:54.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.033+0000 7f07ad79c700 1 --2- 172.21.15.23:0/2828864019 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07a80c40b0 0x7f07a80c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:54.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.033+0000 7f07a7fff700 1 -- 172.21.15.23:0/2828864019 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07a80c40b0 msgr2=0x7f07a80c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:54.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.033+0000 7f07a7fff700 1 --2- 172.21.15.23:0/2828864019 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07a80c40b0 0x7f07a80c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:54.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.033+0000 7f07a7fff700 1 -- 172.21.15.23:0/2828864019 >> v1:172.21.15.67:6789/0 conn(0x7f07a8152ce0 legacy=0x7f07a815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:54.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.033+0000 7f07a7fff700 1 -- 172.21.15.23:0/2828864019 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a815bda0 con 0x7f07a80c9fd0 2023-06-22T08:21:54.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.033+0000 7f07a7fff700 1 --2- 172.21.15.23:0/2828864019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a80c9fd0 0x7f07a80cf1b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f07940098f0 tx=0x7f0794009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8746a1c0bf329972 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:54.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.033+0000 7f07a77fe700 1 -- 172.21.15.23:0/2828864019 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794011830 con 0x7f07a80c9fd0 2023-06-22T08:21:54.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.033+0000 7f07a77fe700 1 -- 172.21.15.23:0/2828864019 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794011990 con 0x7f07a80c9fd0 2023-06-22T08:21:54.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.033+0000 7f07a77fe700 1 -- 172.21.15.23:0/2828864019 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794011d20 con 0x7f07a80c9fd0 2023-06-22T08:21:54.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.034+0000 7f07af1ff700 1 -- 172.21.15.23:0/2828864019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a80c9fd0 msgr2=0x7f07a80cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:54.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.034+0000 7f07af1ff700 1 --2- 172.21.15.23:0/2828864019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a80c9fd0 0x7f07a80cf1b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f07940098f0 tx=0x7f0794009c00 comp rx=0 tx=0).stop 2023-06-22T08:21:54.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.034+0000 7f07af1ff700 1 -- 172.21.15.23:0/2828864019 shutdown_connections 2023-06-22T08:21:54.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.034+0000 7f07af1ff700 1 --2- 172.21.15.23:0/2828864019 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07a80c40b0 0x7f07a80c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:54.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.034+0000 7f07af1ff700 1 --2- 172.21.15.23:0/2828864019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a80c9fd0 0x7f07a80cf1b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:54.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.034+0000 7f07af1ff700 1 -- 172.21.15.23:0/2828864019 >> 172.21.15.23:0/2828864019 conn(0x7f07a806ae30 msgr2=0x7f07a806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:54.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.034+0000 7f07af1ff700 1 -- 172.21.15.23:0/2828864019 shutdown_connections 2023-06-22T08:21:54.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.034+0000 7f07af1ff700 1 -- 172.21.15.23:0/2828864019 wait complete. 2023-06-22T08:21:54.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.036+0000 7f07af1ff700 1 Processor -- start 2023-06-22T08:21:54.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.036+0000 7f07af1ff700 1 -- start start 2023-06-22T08:21:54.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07af1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a80c40b0 0x7f07a814a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:54.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07af1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07a80c9fd0 0x7f07a80cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:54.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07af1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a8152ce0 0x7f07a80b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:54.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07af1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a80cee60 con 0x7f07a8152ce0 2023-06-22T08:21:54.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07af1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a814ab10 con 0x7f07a80c40b0 2023-06-22T08:21:54.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07af1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a81ef290 con 0x7f07a80c9fd0 2023-06-22T08:21:54.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07ad79c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a8152ce0 0x7f07a80b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:54.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07ad79c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a8152ce0 0x7f07a80b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41294/0 (socket says 172.21.15.23:41294) 2023-06-22T08:21:54.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07ad79c700 1 -- 172.21.15.23:0/2551965992 learned_addr learned my addr 172.21.15.23:0/2551965992 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:54.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07acf9b700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a80c40b0 0x7f07a814a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:54.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.037+0000 7f07a7fff700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07a80c9fd0 0x7f07a80cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:54.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.038+0000 7f07ad79c700 1 -- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07a80c9fd0 msgr2=0x7f07a80cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:54.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.038+0000 7f07ad79c700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07a80c9fd0 0x7f07a80cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:54.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.038+0000 7f07ad79c700 1 -- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a80c40b0 msgr2=0x7f07a814a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:54.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.038+0000 7f07ad79c700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a80c40b0 0x7f07a814a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:54.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.038+0000 7f07ad79c700 1 -- 172.21.15.23:0/2551965992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0794009550 con 0x7f07a8152ce0 2023-06-22T08:21:54.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.038+0000 7f07ad79c700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a8152ce0 0x7f07a80b33b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009d30 tx=0x7f07a000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:54.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.038+0000 7f07a57fa700 1 -- 172.21.15.23:0/2551965992 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a000cc90 con 0x7f07a8152ce0 2023-06-22T08:21:54.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.038+0000 7f07a57fa700 1 -- 172.21.15.23:0/2551965992 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a000f040 con 0x7f07a8152ce0 2023-06-22T08:21:54.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.038+0000 7f07a57fa700 1 -- 172.21.15.23:0/2551965992 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0009e80 con 0x7f07a8152ce0 2023-06-22T08:21:54.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.038+0000 7f07af1ff700 1 -- 172.21.15.23:0/2551965992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a80b13f0 con 0x7f07a8152ce0 2023-06-22T08:21:54.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.039+0000 7f07af1ff700 1 -- 172.21.15.23:0/2551965992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a81ee680 con 0x7f07a8152ce0 2023-06-22T08:21:54.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.040+0000 7f07af1ff700 1 -- 172.21.15.23:0/2551965992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a80742e0 con 0x7f07a8152ce0 2023-06-22T08:21:54.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.041+0000 7f07a57fa700 1 -- 172.21.15.23:0/2551965992 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f07a0014480 con 0x7f07a8152ce0 2023-06-22T08:21:54.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.042+0000 7f07a57fa700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f078c06ace0 0x7f078c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:54.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.042+0000 7f07a57fa700 1 -- 172.21.15.23:0/2551965992 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f07a008a320 con 0x7f07a8152ce0 2023-06-22T08:21:54.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.042+0000 7f07acf9b700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f078c06ace0 0x7f078c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:54.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.043+0000 7f07acf9b700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f078c06ace0 0x7f078c06d190 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f079c007fd0 tx=0x7f079c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:54.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.046+0000 7f07a57fa700 1 -- 172.21.15.23:0/2551965992 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f07a0054cc0 con 0x7f07a8152ce0 2023-06-22T08:21:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.448+0000 7f07af1ff700 1 -- 172.21.15.23:0/2551965992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07a806b200 con 0x7f07a8152ce0 2023-06-22T08:21:54.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.449+0000 7f07a57fa700 1 -- 172.21.15.23:0/2551965992 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+5498 (secure 0 0 0) 0x7f07a0018090 con 0x7f07a8152ce0 2023-06-22T08:21:54.455 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:54.456 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6730,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:49.783257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:54.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.454+0000 7f07af1ff700 1 -- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f078c06ace0 msgr2=0x7f078c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:54.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.454+0000 7f07af1ff700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f078c06ace0 0x7f078c06d190 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f079c007fd0 tx=0x7f079c009040 comp rx=0 tx=0).stop 2023-06-22T08:21:54.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.454+0000 7f07af1ff700 1 -- 172.21.15.23:0/2551965992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a8152ce0 msgr2=0x7f07a80b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:54.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.454+0000 7f07af1ff700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a8152ce0 0x7f07a80b33b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009d30 tx=0x7f07a000c3a0 comp rx=0 tx=0).stop 2023-06-22T08:21:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.459+0000 7f07af1ff700 1 -- 172.21.15.23:0/2551965992 shutdown_connections 2023-06-22T08:21:54.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.459+0000 7f07af1ff700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f07a8152ce0 0x7f07a80b33b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:54.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.459+0000 7f07af1ff700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f078c06ace0 0x7f078c06d190 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:54.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.459+0000 7f07af1ff700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07a80c40b0 0x7f07a814a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:54.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.459+0000 7f07af1ff700 1 --2- 172.21.15.23:0/2551965992 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f07a80c9fd0 0x7f07a80cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:54.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.459+0000 7f07af1ff700 1 -- 172.21.15.23:0/2551965992 >> 172.21.15.23:0/2551965992 conn(0x7f07a806ae30 msgr2=0x7f07a80ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:54.462 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.459+0000 7f07af1ff700 1 -- 172.21.15.23:0/2551965992 shutdown_connections 2023-06-22T08:21:54.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:54.470+0000 7f07af1ff700 1 -- 172.21.15.23:0/2551965992 wait complete. 2023-06-22T08:21:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 171 2023-06-22T08:21:54.507 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:21:56.509 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:56.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.872+0000 7fad54bff700 1 Processor -- start 2023-06-22T08:21:56.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.873+0000 7fad54bff700 1 -- start start 2023-06-22T08:21:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.875+0000 7fad54bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad5006e310 0x7fad500c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.875+0000 7fad54bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad501548b0 0x7fad50154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.875+0000 7fad54bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad500c3dd0 con 0x7fad500caf00 2023-06-22T08:21:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.875+0000 7fad54bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad50158d10 con 0x7fad5006e310 2023-06-22T08:21:56.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.875+0000 7fad54bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad500c4800 con 0x7fad501548b0 2023-06-22T08:21:56.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.875+0000 7fad4dd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad5006e310 0x7fad500c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:56.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.876+0000 7fad4ed9d700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad501548b0 0x7fad50154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:56.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.876+0000 7fad4dd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad5006e310 0x7fad500c5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:37140/0 (socket says 172.21.15.23:37140) 2023-06-22T08:21:56.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.876+0000 7fad4dd9b700 1 -- 172.21.15.23:0/2275610606 learned_addr learned my addr 172.21.15.23:0/2275610606 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:56.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.876+0000 7fad4dd9b700 1 -- 172.21.15.23:0/2275610606 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad501548b0 msgr2=0x7fad50154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:56.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.876+0000 7fad4dd9b700 1 --2- 172.21.15.23:0/2275610606 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad501548b0 0x7fad50154cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:56.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.876+0000 7fad4dd9b700 1 -- 172.21.15.23:0/2275610606 >> v1:172.21.15.23:6789/0 conn(0x7fad500caf00 legacy=0x7fad5014eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:21:56.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.876+0000 7fad4dd9b700 1 -- 172.21.15.23:0/2275610606 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad500c4520 con 0x7fad5006e310 2023-06-22T08:21:56.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.876+0000 7fad4dd9b700 1 --2- 172.21.15.23:0/2275610606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5006e310 0x7fad500c5f20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fad440098f0 tx=0x7fad44009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bfa3227bdfd6653 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:56.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.881+0000 7fad4d59a700 1 -- 172.21.15.23:0/2275610606 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44004230 con 0x7fad5006e310 2023-06-22T08:21:56.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.881+0000 7fad4d59a700 1 -- 172.21.15.23:0/2275610606 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad44004390 con 0x7fad5006e310 2023-06-22T08:21:56.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.881+0000 7fad4d59a700 1 -- 172.21.15.23:0/2275610606 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad440036a0 con 0x7fad5006e310 2023-06-22T08:21:56.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.882+0000 7fad54bff700 1 -- 172.21.15.23:0/2275610606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5006e310 msgr2=0x7fad500c5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:56.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.882+0000 7fad54bff700 1 --2- 172.21.15.23:0/2275610606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5006e310 0x7fad500c5f20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fad440098f0 tx=0x7fad44009c00 comp rx=0 tx=0).stop 2023-06-22T08:21:56.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.882+0000 7fad54bff700 1 -- 172.21.15.23:0/2275610606 shutdown_connections 2023-06-22T08:21:56.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.882+0000 7fad54bff700 1 --2- 172.21.15.23:0/2275610606 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad501548b0 0x7fad50154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:56.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.882+0000 7fad54bff700 1 --2- 172.21.15.23:0/2275610606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad5006e310 0x7fad500c5f20 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:56.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.882+0000 7fad54bff700 1 -- 172.21.15.23:0/2275610606 >> 172.21.15.23:0/2275610606 conn(0x7fad500ca3f0 msgr2=0x7fad500ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:56.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.886+0000 7fad54bff700 1 -- 172.21.15.23:0/2275610606 shutdown_connections 2023-06-22T08:21:56.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.886+0000 7fad54bff700 1 -- 172.21.15.23:0/2275610606 wait complete. 2023-06-22T08:21:56.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.889+0000 7fad54bff700 1 Processor -- start 2023-06-22T08:21:56.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.889+0000 7fad54bff700 1 -- start start 2023-06-22T08:21:56.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.890+0000 7fad54bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad5006e310 0x7fad50151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:56.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.890+0000 7fad54bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad500caf00 0x7fad50156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:56.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.890+0000 7fad54bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad501548b0 0x7fad500c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:56.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.890+0000 7fad54bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad50157b50 con 0x7fad501548b0 2023-06-22T08:21:56.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.890+0000 7fad54bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad501413b0 con 0x7fad500caf00 2023-06-22T08:21:56.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.890+0000 7fad54bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad50150d20 con 0x7fad5006e310 2023-06-22T08:21:56.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.891+0000 7fad4ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad501548b0 0x7fad500c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:56.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.891+0000 7fad4ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad501548b0 0x7fad500c9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35126/0 (socket says 172.21.15.23:35126) 2023-06-22T08:21:56.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.891+0000 7fad4ed9d700 1 -- 172.21.15.23:0/4016328820 learned_addr learned my addr 172.21.15.23:0/4016328820 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:56.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.891+0000 7fad4dd9b700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad500caf00 0x7fad50156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:56.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.891+0000 7fad4e59c700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad5006e310 0x7fad50151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:56.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.892+0000 7fad4e59c700 1 -- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad500caf00 msgr2=0x7fad50156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:56.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.892+0000 7fad4e59c700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad500caf00 0x7fad50156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:56.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.892+0000 7fad4e59c700 1 -- 172.21.15.23:0/4016328820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad501548b0 msgr2=0x7fad500c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:56.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.892+0000 7fad4e59c700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad501548b0 0x7fad500c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:56.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.892+0000 7fad4e59c700 1 -- 172.21.15.23:0/4016328820 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44009550 con 0x7fad5006e310 2023-06-22T08:21:56.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.892+0000 7fad4e59c700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad5006e310 0x7fad50151db0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fad4000c3e0 tx=0x7fad4000c6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:56.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.893+0000 7fad3effd700 1 -- 172.21.15.23:0/4016328820 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad400052d0 con 0x7fad5006e310 2023-06-22T08:21:56.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.893+0000 7fad3effd700 1 -- 172.21.15.23:0/4016328820 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40005430 con 0x7fad5006e310 2023-06-22T08:21:56.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.893+0000 7fad3effd700 1 -- 172.21.15.23:0/4016328820 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad400038c0 con 0x7fad5006e310 2023-06-22T08:21:56.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.894+0000 7fad54bff700 1 -- 172.21.15.23:0/4016328820 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad50157220 con 0x7fad5006e310 2023-06-22T08:21:56.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.894+0000 7fad54bff700 1 -- 172.21.15.23:0/4016328820 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad50157870 con 0x7fad5006e310 2023-06-22T08:21:56.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.896+0000 7fad54bff700 1 -- 172.21.15.23:0/4016328820 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad501538a0 con 0x7fad5006e310 2023-06-22T08:21:56.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.897+0000 7fad3effd700 1 -- 172.21.15.23:0/4016328820 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fad40003a20 con 0x7fad5006e310 2023-06-22T08:21:56.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.899+0000 7fad3effd700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fad3006ad30 0x7fad3006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:56.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.899+0000 7fad3effd700 1 -- 172.21.15.23:0/4016328820 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7871+0+0 (secure 0 0 0) 0x7fad40088e80 con 0x7fad5006e310 2023-06-22T08:21:56.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.899+0000 7fad4dd9b700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fad3006ad30 0x7fad3006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:56.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.900+0000 7fad4dd9b700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fad3006ad30 0x7fad3006d1e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fad44000c00 tx=0x7fad44008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:56.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:56.904+0000 7fad3effd700 1 -- 172.21.15.23:0/4016328820 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fad40053890 con 0x7fad5006e310 2023-06-22T08:21:57.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.259+0000 7fad54bff700 1 -- 172.21.15.23:0/4016328820 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad5004ea20 con 0x7fad5006e310 2023-06-22T08:21:57.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.260+0000 7fad3effd700 1 -- 172.21.15.23:0/4016328820 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+5498 (secure 0 0 0) 0x7fad40056eb0 con 0x7fad5006e310 2023-06-22T08:21:57.263 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:57.264 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6730,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:49.783257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:57.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 -- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fad3006ad30 msgr2=0x7fad3006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fad3006ad30 0x7fad3006d1e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fad44000c00 tx=0x7fad44008030 comp rx=0 tx=0).stop 2023-06-22T08:21:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 -- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad5006e310 msgr2=0x7fad50151db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad5006e310 0x7fad50151db0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fad4000c3e0 tx=0x7fad4000c6f0 comp rx=0 tx=0).stop 2023-06-22T08:21:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 -- 172.21.15.23:0/4016328820 shutdown_connections 2023-06-22T08:21:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad501548b0 0x7fad500c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fad5006e310 0x7fad50151db0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:57.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fad3006ad30 0x7fad3006d1e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:57.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 --2- 172.21.15.23:0/4016328820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad500caf00 0x7fad50156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:57.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 -- 172.21.15.23:0/4016328820 >> 172.21.15.23:0/4016328820 conn(0x7fad500ca3f0 msgr2=0x7fad5014eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:57.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 -- 172.21.15.23:0/4016328820 shutdown_connections 2023-06-22T08:21:57.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:57.263+0000 7fad54bff700 1 -- 172.21.15.23:0/4016328820 wait complete. 2023-06-22T08:21:57.267 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 171 2023-06-22T08:21:57.293 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:21:58.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~95s 2023-06-22T08:21:59.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:21:59.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.511+0000 7fbac07ff700 1 Processor -- start 2023-06-22T08:21:59.514 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.511+0000 7fbac07ff700 1 -- start start 2023-06-22T08:21:59.515 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.513+0000 7fbac07ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbab80cf1b0 0x7fbab80c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:59.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.513+0000 7fbac07ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab815c370 con 0x7fbab80c9fd0 2023-06-22T08:21:59.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.513+0000 7fbac07ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab815bda0 con 0x7fbab8152ce0 2023-06-22T08:21:59.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.513+0000 7fbac07ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab81530b0 con 0x7fbab80cf1b0 2023-06-22T08:21:59.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.513+0000 7fbabdd9a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbab80c9fd0 0x7fbab80c40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52080/0 (socket says 172.21.15.23:52080) 2023-06-22T08:21:59.516 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.513+0000 7fbabdd9a700 1 -- 172.21.15.23:0/3651456242 learned_addr learned my addr 172.21.15.23:0/3651456242 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:21:59.517 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.513+0000 7fbabed9c700 1 --2- 172.21.15.23:0/3651456242 >> v2:172.21.15.23:3300/0 conn(0x7fbab80cf1b0 0x7fbab80c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:59.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.513+0000 7fbabd599700 1 -- 172.21.15.23:0/3651456242 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298081653 0 0) 0x7fbab815c370 con 0x7fbab80c9fd0 2023-06-22T08:21:59.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.514+0000 7fbabed9c700 1 -- 172.21.15.23:0/3651456242 >> v1:172.21.15.67:6789/0 conn(0x7fbab8152ce0 legacy=0x7fbab815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:59.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.514+0000 7fbabed9c700 1 -- 172.21.15.23:0/3651456242 >> v1:172.21.15.23:6789/0 conn(0x7fbab80c9fd0 legacy=0x7fbab80c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:59.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.514+0000 7fbabed9c700 1 -- 172.21.15.23:0/3651456242 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab80cf630 con 0x7fbab80cf1b0 2023-06-22T08:21:59.518 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.514+0000 7fbabed9c700 1 --2- 172.21.15.23:0/3651456242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 0x7fbab80c47c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009d90 tx=0x7fbab000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85394320d023d4f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:59.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.514+0000 7fbabd599700 1 -- 172.21.15.23:0/3651456242 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000e070 con 0x7fbab80cf1b0 2023-06-22T08:21:59.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.514+0000 7fbabd599700 1 -- 172.21.15.23:0/3651456242 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab000bad0 con 0x7fbab80cf1b0 2023-06-22T08:21:59.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.514+0000 7fbabd599700 1 -- 172.21.15.23:0/3651456242 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000be80 con 0x7fbab80cf1b0 2023-06-22T08:21:59.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.515+0000 7fbac07ff700 1 -- 172.21.15.23:0/3651456242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 msgr2=0x7fbab80c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:59.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.515+0000 7fbac07ff700 1 --2- 172.21.15.23:0/3651456242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 0x7fbab80c47c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009d90 tx=0x7fbab000b330 comp rx=0 tx=0).stop 2023-06-22T08:21:59.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.515+0000 7fbac07ff700 1 -- 172.21.15.23:0/3651456242 shutdown_connections 2023-06-22T08:21:59.519 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.515+0000 7fbac07ff700 1 --2- 172.21.15.23:0/3651456242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 0x7fbab80c47c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:59.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.515+0000 7fbac07ff700 1 -- 172.21.15.23:0/3651456242 >> 172.21.15.23:0/3651456242 conn(0x7fbab806ae30 msgr2=0x7fbab806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:59.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.515+0000 7fbac07ff700 1 -- 172.21.15.23:0/3651456242 shutdown_connections 2023-06-22T08:21:59.520 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.515+0000 7fbac07ff700 1 -- 172.21.15.23:0/3651456242 wait complete. 2023-06-22T08:21:59.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.516+0000 7fbac07ff700 1 Processor -- start 2023-06-22T08:21:59.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.517+0000 7fbac07ff700 1 -- start start 2023-06-22T08:21:59.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.517+0000 7fbac07ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab80c9fd0 0x7fbab80ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:59.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbac07ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 0x7fbab80ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:59.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbac07ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbab8152ce0 0x7fbab814c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:59.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbac07ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab80cdd10 con 0x7fbab80cf1b0 2023-06-22T08:21:59.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbac07ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab81ee7c0 con 0x7fbab80c9fd0 2023-06-22T08:21:59.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbac07ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab81eea80 con 0x7fbab8152ce0 2023-06-22T08:21:59.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabdd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 0x7fbab80ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:59.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabed9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbab8152ce0 0x7fbab814c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:59.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabe59b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab80c9fd0 0x7fbab80ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:59.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabe59b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab80c9fd0 0x7fbab80ce1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:37158/0 (socket says 172.21.15.23:37158) 2023-06-22T08:21:59.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabe59b700 1 -- 172.21.15.23:0/418875126 learned_addr learned my addr 172.21.15.23:0/418875126 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:21:59.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabed9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbab8152ce0 0x7fbab814c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:46590/0 (socket says 172.21.15.23:46590) 2023-06-22T08:21:59.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabdd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 0x7fbab80ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35148/0 (socket says 172.21.15.23:35148) 2023-06-22T08:21:59.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabe59b700 1 -- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbab8152ce0 msgr2=0x7fbab814c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:59.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabe59b700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbab8152ce0 0x7fbab814c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:59.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabe59b700 1 -- 172.21.15.23:0/418875126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 msgr2=0x7fbab80ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:59.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabe59b700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 0x7fbab80ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:59.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabe59b700 1 -- 172.21.15.23:0/418875126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab0009a70 con 0x7fbab80c9fd0 2023-06-22T08:21:59.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabed9c700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbab8152ce0 0x7fbab814c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:59.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.518+0000 7fbabdd9a700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 0x7fbab80ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:21:59.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.519+0000 7fbabe59b700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab80c9fd0 0x7fbab80ce1a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbab400c700 tx=0x7fbab400ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:59.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.519+0000 7fbaaeffd700 1 -- 172.21.15.23:0/418875126 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab40054e0 con 0x7fbab80c9fd0 2023-06-22T08:21:59.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.519+0000 7fbaaeffd700 1 -- 172.21.15.23:0/418875126 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab4003680 con 0x7fbab80c9fd0 2023-06-22T08:21:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.519+0000 7fbaaeffd700 1 -- 172.21.15.23:0/418875126 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4003a30 con 0x7fbab80c9fd0 2023-06-22T08:21:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.519+0000 7fbac07ff700 1 -- 172.21.15.23:0/418875126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab81eed60 con 0x7fbab80c9fd0 2023-06-22T08:21:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.519+0000 7fbac07ff700 1 -- 172.21.15.23:0/418875126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab81ef320 con 0x7fbab80c9fd0 2023-06-22T08:21:59.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.520+0000 7fbac07ff700 1 -- 172.21.15.23:0/418875126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab804ea20 con 0x7fbab80c9fd0 2023-06-22T08:21:59.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.521+0000 7fbaaeffd700 1 -- 172.21.15.23:0/418875126 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbab4003b90 con 0x7fbab80c9fd0 2023-06-22T08:21:59.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.522+0000 7fbaaeffd700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba9c06ace0 0x7fba9c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:21:59.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.522+0000 7fbaaeffd700 1 -- 172.21.15.23:0/418875126 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7871+0+0 (secure 0 0 0) 0x7fbab4088b30 con 0x7fbab80c9fd0 2023-06-22T08:21:59.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.522+0000 7fbabdd9a700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba9c06ace0 0x7fba9c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:21:59.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.523+0000 7fbabdd9a700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba9c06ace0 0x7fba9c06d190 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbab8152560 tx=0x7fbaa8007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:21:59.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.527+0000 7fbaaeffd700 1 -- 172.21.15.23:0/418875126 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbab40534c0 con 0x7fbab80c9fd0 2023-06-22T08:21:59.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.798+0000 7fbac07ff700 1 -- 172.21.15.23:0/418875126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbab80b1e60 con 0x7fbab80c9fd0 2023-06-22T08:21:59.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.800+0000 7fbaaeffd700 1 -- 172.21.15.23:0/418875126 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+5498 (secure 0 0 0) 0x7fbab4017020 con 0x7fbab80c9fd0 2023-06-22T08:21:59.803 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:21:59.803 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6730,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:49.783257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:21:59.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 -- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba9c06ace0 msgr2=0x7fba9c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:59.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba9c06ace0 0x7fba9c06d190 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbab8152560 tx=0x7fbaa8007de0 comp rx=0 tx=0).stop 2023-06-22T08:21:59.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 -- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab80c9fd0 msgr2=0x7fbab80ce1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:21:59.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab80c9fd0 0x7fbab80ce1a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbab400c700 tx=0x7fbab400ca10 comp rx=0 tx=0).stop 2023-06-22T08:21:59.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 -- 172.21.15.23:0/418875126 shutdown_connections 2023-06-22T08:21:59.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbab8152ce0 0x7fbab814c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:59.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fba9c06ace0 0x7fba9c06d190 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:59.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbab80c9fd0 0x7fbab80ce1a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:59.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 --2- 172.21.15.23:0/418875126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbab80cf1b0 0x7fbab80ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:21:59.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 -- 172.21.15.23:0/418875126 >> 172.21.15.23:0/418875126 conn(0x7fbab806ae30 msgr2=0x7fbab814e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:21:59.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 -- 172.21.15.23:0/418875126 shutdown_connections 2023-06-22T08:21:59.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:21:59.801+0000 7fbac07ff700 1 -- 172.21.15.23:0/418875126 wait complete. 2023-06-22T08:21:59.807 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 171 2023-06-22T08:21:59.823 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:22:01.824 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.032+0000 7f5a85dff700 1 Processor -- start 2023-06-22T08:22:02.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.032+0000 7f5a85dff700 1 -- start start 2023-06-22T08:22:02.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.033+0000 7f5a85dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a80152d70 0x7f5a800c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:02.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.033+0000 7f5a85dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5a800c45f0 0x7f5a800c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:02.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.033+0000 7f5a85dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a800c4fb0 con 0x7f5a80159330 2023-06-22T08:22:02.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.033+0000 7f5a85dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a800c95e0 con 0x7f5a800c45f0 2023-06-22T08:22:02.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.033+0000 7f5a85dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8015bea0 con 0x7f5a80152d70 2023-06-22T08:22:02.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.034+0000 7f5a7ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5a800c45f0 0x7f5a800c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:02.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.034+0000 7f5a7effd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5a80159330 0x7f5a800cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52090/0 (socket says 172.21.15.23:52090) 2023-06-22T08:22:02.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.034+0000 7f5a7effd700 1 -- 172.21.15.23:0/3093815494 learned_addr learned my addr 172.21.15.23:0/3093815494 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:22:02.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.034+0000 7f5a7f7fe700 1 --2- 172.21.15.23:0/3093815494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a80152d70 0x7f5a800c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:02.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.034+0000 7f5a7ffff700 1 -- 172.21.15.23:0/3093815494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a80152d70 msgr2=0x7f5a800c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:02.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.034+0000 7f5a7ffff700 1 --2- 172.21.15.23:0/3093815494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a80152d70 0x7f5a800c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:02.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.034+0000 7f5a7ffff700 1 -- 172.21.15.23:0/3093815494 >> v1:172.21.15.23:6789/0 conn(0x7f5a80159330 legacy=0x7f5a800cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:02.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.034+0000 7f5a7ffff700 1 -- 172.21.15.23:0/3093815494 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8015c160 con 0x7f5a800c45f0 2023-06-22T08:22:02.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.034+0000 7f5a7ffff700 1 --2- 172.21.15.23:0/3093815494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a800c45f0 0x7f5a800c4a70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5a78009d90 tx=0x7f5a7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbfa2a7dda4c49f server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:02.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.035+0000 7f5a7e7fc700 1 -- 172.21.15.23:0/3093815494 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7800e070 con 0x7f5a800c45f0 2023-06-22T08:22:02.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.035+0000 7f5a7e7fc700 1 -- 172.21.15.23:0/3093815494 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7800bb20 con 0x7f5a800c45f0 2023-06-22T08:22:02.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.035+0000 7f5a7e7fc700 1 -- 172.21.15.23:0/3093815494 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78012430 con 0x7f5a800c45f0 2023-06-22T08:22:02.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.035+0000 7f5a85dff700 1 -- 172.21.15.23:0/3093815494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a800c45f0 msgr2=0x7f5a800c4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:02.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.035+0000 7f5a85dff700 1 --2- 172.21.15.23:0/3093815494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a800c45f0 0x7f5a800c4a70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5a78009d90 tx=0x7f5a7800b330 comp rx=0 tx=0).stop 2023-06-22T08:22:02.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.035+0000 7f5a85dff700 1 -- 172.21.15.23:0/3093815494 shutdown_connections 2023-06-22T08:22:02.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.035+0000 7f5a85dff700 1 --2- 172.21.15.23:0/3093815494 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a80152d70 0x7f5a800c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:02.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.035+0000 7f5a85dff700 1 --2- 172.21.15.23:0/3093815494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a800c45f0 0x7f5a800c4a70 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:02.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.035+0000 7f5a85dff700 1 -- 172.21.15.23:0/3093815494 >> 172.21.15.23:0/3093815494 conn(0x7f5a8006ae30 msgr2=0x7f5a8006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:02.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.036+0000 7f5a85dff700 1 -- 172.21.15.23:0/3093815494 shutdown_connections 2023-06-22T08:22:02.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.036+0000 7f5a85dff700 1 -- 172.21.15.23:0/3093815494 wait complete. 2023-06-22T08:22:02.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.037+0000 7f5a85dff700 1 Processor -- start 2023-06-22T08:22:02.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.037+0000 7f5a85dff700 1 -- start start 2023-06-22T08:22:02.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.038+0000 7f5a85dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a800c45f0 0x7f5a8014dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:02.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.038+0000 7f5a85dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a80152d70 0x7f5a80157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:02.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.038+0000 7f5a85dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a80159330 0x7f5a80157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:02.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.038+0000 7f5a85dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a801ef0c0 con 0x7f5a80152d70 2023-06-22T08:22:02.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.038+0000 7f5a85dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a801eeb20 con 0x7f5a80159330 2023-06-22T08:22:02.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.038+0000 7f5a85dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800c8f10 con 0x7f5a800c45f0 2023-06-22T08:22:02.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.038+0000 7f5a7effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a80152d70 0x7f5a80157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:02.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a7effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a80152d70 0x7f5a80157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35156/0 (socket says 172.21.15.23:35156) 2023-06-22T08:22:02.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a7effd700 1 -- 172.21.15.23:0/2434032282 learned_addr learned my addr 172.21.15.23:0/2434032282 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:02.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a7f7fe700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a800c45f0 0x7f5a8014dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:02.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a7ffff700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a80159330 0x7f5a80157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:02.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a7effd700 1 -- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a800c45f0 msgr2=0x7f5a8014dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:02.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a7effd700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a800c45f0 0x7f5a8014dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:02.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a7effd700 1 -- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a80159330 msgr2=0x7f5a80157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:02.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a7effd700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a80159330 0x7f5a80157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:02.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a7effd700 1 -- 172.21.15.23:0/2434032282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a78009a70 con 0x7f5a80152d70 2023-06-22T08:22:02.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a7effd700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a80152d70 0x7f5a80157240 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5a7400c480 tx=0x7f5a7400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:02.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a67fff700 1 -- 172.21.15.23:0/2434032282 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a74005d70 con 0x7f5a80152d70 2023-06-22T08:22:02.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a67fff700 1 -- 172.21.15.23:0/2434032282 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7400ce90 con 0x7f5a80152d70 2023-06-22T08:22:02.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.039+0000 7f5a67fff700 1 -- 172.21.15.23:0/2434032282 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a740072d0 con 0x7f5a80152d70 2023-06-22T08:22:02.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.040+0000 7f5a85dff700 1 -- 172.21.15.23:0/2434032282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8014ea00 con 0x7f5a80152d70 2023-06-22T08:22:02.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.040+0000 7f5a85dff700 1 -- 172.21.15.23:0/2434032282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8014d3f0 con 0x7f5a80152d70 2023-06-22T08:22:02.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.041+0000 7f5a85dff700 1 -- 172.21.15.23:0/2434032282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8014d250 con 0x7f5a80152d70 2023-06-22T08:22:02.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.042+0000 7f5a67fff700 1 -- 172.21.15.23:0/2434032282 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5a74007430 con 0x7f5a80152d70 2023-06-22T08:22:02.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.043+0000 7f5a67fff700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5a6c06ace0 0x7f5a6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:02.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.043+0000 7f5a67fff700 1 -- 172.21.15.23:0/2434032282 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7871+0+0 (secure 0 0 0) 0x7f5a74088ed0 con 0x7f5a80152d70 2023-06-22T08:22:02.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.043+0000 7f5a7f7fe700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5a6c06ace0 0x7f5a6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:02.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.043+0000 7f5a7f7fe700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5a6c06ace0 0x7f5a6c06d190 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009be0 tx=0x7f5a70009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:02.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.047+0000 7f5a67fff700 1 -- 172.21.15.23:0/2434032282 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5a8014d250 con 0x7f5a80152d70 2023-06-22T08:22:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.297+0000 7f5a85dff700 1 -- 172.21.15.23:0/2434032282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a8014d250 con 0x7f5a80152d70 2023-06-22T08:22:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.298+0000 7f5a67fff700 1 -- 172.21.15.23:0/2434032282 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+5498 (secure 0 0 0) 0x7f5a8014d250 con 0x7f5a80152d70 2023-06-22T08:22:02.301 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:02.301 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6730,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:21:49.783257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":5707,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5707":{"gid":5707,"name":"a","rank":1,"incarnation":72,"state":"up:active","state_seq":19,"addr":"172.21.15.23:6835/934828076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":934828076},{"type":"v1","addr":"172.21.15.23:6835","nonce":934828076}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:02.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.300+0000 7f5a85dff700 1 -- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5a6c06ace0 msgr2=0x7f5a6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:02.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.300+0000 7f5a85dff700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5a6c06ace0 0x7f5a6c06d190 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009be0 tx=0x7f5a70009210 comp rx=0 tx=0).stop 2023-06-22T08:22:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.300+0000 7f5a85dff700 1 -- 172.21.15.23:0/2434032282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a80152d70 msgr2=0x7f5a80157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.300+0000 7f5a85dff700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a80152d70 0x7f5a80157240 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5a7400c480 tx=0x7f5a7400c790 comp rx=0 tx=0).stop 2023-06-22T08:22:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.301+0000 7f5a85dff700 1 -- 172.21.15.23:0/2434032282 shutdown_connections 2023-06-22T08:22:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.301+0000 7f5a85dff700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5a6c06ace0 0x7f5a6c06d190 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.301+0000 7f5a85dff700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a80159330 0x7f5a80157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.301+0000 7f5a85dff700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a800c45f0 0x7f5a8014dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.301+0000 7f5a85dff700 1 --2- 172.21.15.23:0/2434032282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a80152d70 0x7f5a80157240 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:02.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.301+0000 7f5a85dff700 1 -- 172.21.15.23:0/2434032282 >> 172.21.15.23:0/2434032282 conn(0x7f5a8006ae30 msgr2=0x7f5a80150420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:02.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.301+0000 7f5a85dff700 1 -- 172.21.15.23:0/2434032282 shutdown_connections 2023-06-22T08:22:02.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:02.301+0000 7f5a85dff700 1 -- 172.21.15.23:0/2434032282 wait complete. 2023-06-22T08:22:02.305 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 171 2023-06-22T08:22:02.322 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-06-22T08:22:04.323 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:04.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.531+0000 7fa8f13ff700 1 Processor -- start 2023-06-22T08:22:04.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.531+0000 7fa8f13ff700 1 -- start start 2023-06-22T08:22:04.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.532+0000 7fa8f13ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa8ec152ce0 0x7fa8ec0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:04.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.532+0000 7fa8f13ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8ec0c9fd0 0x7fa8ec15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:04.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8f13ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ec15c370 con 0x7fa8ec158880 2023-06-22T08:22:04.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8f13ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ec1530b0 con 0x7fa8ec152ce0 2023-06-22T08:22:04.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8f13ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ec0cf6f0 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8eaffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa8ec152ce0 0x7fa8ec0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:04.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8eaffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa8ec152ce0 0x7fa8ec0cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35170/0 (socket says 172.21.15.23:35170) 2023-06-22T08:22:04.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8eaffd700 1 -- 172.21.15.23:0/3469264345 learned_addr learned my addr 172.21.15.23:0/3469264345 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:04.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8ea7fc700 1 --2- 172.21.15.23:0/3469264345 >> v2:172.21.15.67:3300/0 conn(0x7fa8ec0c9fd0 0x7fa8ec15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:04.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8eaffd700 1 -- 172.21.15.23:0/3469264345 >> v2:172.21.15.67:3300/0 conn(0x7fa8ec0c9fd0 msgr2=0x7fa8ec15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:04.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8eaffd700 1 --2- 172.21.15.23:0/3469264345 >> v2:172.21.15.67:3300/0 conn(0x7fa8ec0c9fd0 0x7fa8ec15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:04.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8eaffd700 1 -- 172.21.15.23:0/3469264345 >> v1:172.21.15.67:6789/0 conn(0x7fa8ec158880 legacy=0x7fa8ec158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:04.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.533+0000 7fa8eaffd700 1 -- 172.21.15.23:0/3469264345 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec15bda0 con 0x7fa8ec152ce0 2023-06-22T08:22:04.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8eaffd700 1 --2- 172.21.15.23:0/3469264345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec152ce0 0x7fa8ec0cf1b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0cc380 tx=0x7fa8dc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2168dae85054a4a1 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:04.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8e9ffb700 1 -- 172.21.15.23:0/3469264345 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc011830 con 0x7fa8ec152ce0 2023-06-22T08:22:04.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8e9ffb700 1 -- 172.21.15.23:0/3469264345 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc011990 con 0x7fa8ec152ce0 2023-06-22T08:22:04.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8e9ffb700 1 -- 172.21.15.23:0/3469264345 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc011d20 con 0x7fa8ec152ce0 2023-06-22T08:22:04.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8f13ff700 1 -- 172.21.15.23:0/3469264345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec152ce0 msgr2=0x7fa8ec0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:04.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8f13ff700 1 --2- 172.21.15.23:0/3469264345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec152ce0 0x7fa8ec0cf1b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0cc380 tx=0x7fa8dc009bd0 comp rx=0 tx=0).stop 2023-06-22T08:22:04.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8f13ff700 1 -- 172.21.15.23:0/3469264345 shutdown_connections 2023-06-22T08:22:04.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8f13ff700 1 --2- 172.21.15.23:0/3469264345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec152ce0 0x7fa8ec0cf1b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:04.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8f13ff700 1 --2- 172.21.15.23:0/3469264345 >> v2:172.21.15.67:3300/0 conn(0x7fa8ec0c9fd0 0x7fa8ec15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:04.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8f13ff700 1 -- 172.21.15.23:0/3469264345 >> 172.21.15.23:0/3469264345 conn(0x7fa8ec06ae30 msgr2=0x7fa8ec06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:04.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8f13ff700 1 -- 172.21.15.23:0/3469264345 shutdown_connections 2023-06-22T08:22:04.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.534+0000 7fa8f13ff700 1 -- 172.21.15.23:0/3469264345 wait complete. 2023-06-22T08:22:04.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.536+0000 7fa8f13ff700 1 Processor -- start 2023-06-22T08:22:04.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.536+0000 7fa8f13ff700 1 -- start start 2023-06-22T08:22:04.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.537+0000 7fa8f13ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec0c9fd0 0x7fa8ec1eead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:04.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.537+0000 7fa8f13ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8ec152ce0 0x7fa8ec0ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:04.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.537+0000 7fa8f13ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa8ec158880 0x7fa8ec15aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:04.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.537+0000 7fa8f13ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0ce1c0 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.537+0000 7fa8f13ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec1ef010 con 0x7fa8ec152ce0 2023-06-22T08:22:04.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.537+0000 7fa8f13ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ec0cc9c0 con 0x7fa8ec158880 2023-06-22T08:22:04.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.537+0000 7fa8eaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec0c9fd0 0x7fa8ec1eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:04.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8eaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec0c9fd0 0x7fa8ec1eead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35178/0 (socket says 172.21.15.23:35178) 2023-06-22T08:22:04.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8eaffd700 1 -- 172.21.15.23:0/201392654 learned_addr learned my addr 172.21.15.23:0/201392654 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:04.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8ea7fc700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8ec152ce0 0x7fa8ec0ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:04.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8eb7fe700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa8ec158880 0x7fa8ec15aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:04.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8eaffd700 1 -- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa8ec158880 msgr2=0x7fa8ec15aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:04.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8eaffd700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa8ec158880 0x7fa8ec15aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:04.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8eaffd700 1 -- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8ec152ce0 msgr2=0x7fa8ec0ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:04.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8eaffd700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8ec152ce0 0x7fa8ec0ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:04.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8eaffd700 1 -- 172.21.15.23:0/201392654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8dc009550 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8eb7fe700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa8ec158880 0x7fa8ec15aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:22:04.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8eaffd700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec0c9fd0 0x7fa8ec1eead0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec08bc30 tx=0x7fa8dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:04.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8d37fe700 1 -- 172.21.15.23:0/201392654 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc004380 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8d37fe700 1 -- 172.21.15.23:0/201392654 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc01f070 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8d37fe700 1 -- 172.21.15.23:0/201392654 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc0036a0 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.538+0000 7fa8f13ff700 1 -- 172.21.15.23:0/201392654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ec0ccc10 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.539+0000 7fa8f13ff700 1 -- 172.21.15.23:0/201392654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ec0cd1d0 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.540+0000 7fa8f13ff700 1 -- 172.21.15.23:0/201392654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ec15c100 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.541+0000 7fa8d37fe700 1 -- 172.21.15.23:0/201392654 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa8dc021030 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.541+0000 7fa8d37fe700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa8d806ad10 0x7fa8d806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:04.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.542+0000 7fa8d37fe700 1 -- 172.21.15.23:0/201392654 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7871+0+0 (secure 0 0 0) 0x7fa8dc08b0e0 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.542+0000 7fa8ea7fc700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa8d806ad10 0x7fa8d806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:04.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.542+0000 7fa8ea7fc700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa8d806ad10 0x7fa8d806d1c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0ca900 tx=0x7fa8e0009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:04.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.546+0000 7fa8d37fe700 1 -- 172.21.15.23:0/201392654 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa8dc055b90 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~102s 2023-06-22T08:22:04.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.804+0000 7fa8f13ff700 1 -- 172.21.15.23:0/201392654 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8ec04ea20 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.805+0000 7fa8d37fe700 1 -- 172.21.15.23:0/201392654 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4737 (secure 0 0 0) 0x7fa8dc0591b0 con 0x7fa8ec0c9fd0 2023-06-22T08:22:04.808 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:04.808 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:04.632858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:04.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.807+0000 7fa8f13ff700 1 -- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa8d806ad10 msgr2=0x7fa8d806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:04.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.807+0000 7fa8f13ff700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa8d806ad10 0x7fa8d806d1c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0ca900 tx=0x7fa8e0009210 comp rx=0 tx=0).stop 2023-06-22T08:22:04.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.807+0000 7fa8f13ff700 1 -- 172.21.15.23:0/201392654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec0c9fd0 msgr2=0x7fa8ec1eead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:04.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.807+0000 7fa8f13ff700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec0c9fd0 0x7fa8ec1eead0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec08bc30 tx=0x7fa8dc004750 comp rx=0 tx=0).stop 2023-06-22T08:22:04.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.808+0000 7fa8f13ff700 1 -- 172.21.15.23:0/201392654 shutdown_connections 2023-06-22T08:22:04.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.808+0000 7fa8f13ff700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa8ec158880 0x7fa8ec15aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:04.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.808+0000 7fa8f13ff700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8ec0c9fd0 0x7fa8ec1eead0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:04.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.808+0000 7fa8f13ff700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa8d806ad10 0x7fa8d806d1c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:04.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.808+0000 7fa8f13ff700 1 --2- 172.21.15.23:0/201392654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8ec152ce0 0x7fa8ec0ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:04.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.808+0000 7fa8f13ff700 1 -- 172.21.15.23:0/201392654 >> 172.21.15.23:0/201392654 conn(0x7fa8ec06ae30 msgr2=0x7fa8ec0cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:04.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.808+0000 7fa8f13ff700 1 -- 172.21.15.23:0/201392654 shutdown_connections 2023-06-22T08:22:04.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:04.808+0000 7fa8f13ff700 1 -- 172.21.15.23:0/201392654 wait complete. 2023-06-22T08:22:04.812 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 172 2023-06-22T08:22:04.829 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-06-22T08:22:04.829 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:22:04.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:05.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.026+0000 7f669bdff700 1 Processor -- start 2023-06-22T08:22:05.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.026+0000 7f669bdff700 1 -- start start 2023-06-22T08:22:05.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.027+0000 7f669bdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66940c3a40 0x7f66940c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:05.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.027+0000 7f669bdff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f669415a320 0x7f6694160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:05.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.027+0000 7f669bdff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66940c6a60 con 0x7f669406d050 2023-06-22T08:22:05.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.027+0000 7f669bdff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6694157570 con 0x7f669415a320 2023-06-22T08:22:05.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.027+0000 7f669bdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66940c4350 con 0x7f66940c3a40 2023-06-22T08:22:05.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.027+0000 7f669939a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f669406d050 0x7f6694159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:48210/0 (socket says 172.21.15.23:48210) 2023-06-22T08:22:05.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.027+0000 7f669939a700 1 -- 172.21.15.23:0/4007519560 learned_addr learned my addr 172.21.15.23:0/4007519560 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:22:05.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.027+0000 7f6699b9b700 1 --2- 172.21.15.23:0/4007519560 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66940c3a40 0x7f66940c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:05.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.027+0000 7f669a39c700 1 --2- 172.21.15.23:0/4007519560 >> v2:172.21.15.67:3300/0 conn(0x7f669415a320 0x7f6694160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:05.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f6698b99700 1 -- 172.21.15.23:0/4007519560 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114253583 0 0) 0x7f66940c6a60 con 0x7f669406d050 2023-06-22T08:22:05.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f6698b99700 1 -- 172.21.15.23:0/4007519560 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66800035b0 con 0x7f669406d050 2023-06-22T08:22:05.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f669a39c700 1 -- 172.21.15.23:0/4007519560 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66940c3a40 msgr2=0x7f66940c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:05.032 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f669a39c700 1 --2- 172.21.15.23:0/4007519560 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66940c3a40 0x7f66940c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:05.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f669a39c700 1 -- 172.21.15.23:0/4007519560 >> v1:172.21.15.67:6789/0 conn(0x7f669406d050 legacy=0x7f6694159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:05.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f669a39c700 1 -- 172.21.15.23:0/4007519560 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669414d690 con 0x7f669415a320 2023-06-22T08:22:05.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f6699b9b700 1 --2- 172.21.15.23:0/4007519560 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66940c3a40 0x7f66940c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:22:05.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f669a39c700 1 --2- 172.21.15.23:0/4007519560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669415a320 0x7f6694160ec0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f668c009d90 tx=0x7f668c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c0ddc5633c35987 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:05.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f6698b99700 1 -- 172.21.15.23:0/4007519560 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c00e070 con 0x7f669415a320 2023-06-22T08:22:05.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f6698b99700 1 -- 172.21.15.23:0/4007519560 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668c00bb20 con 0x7f669415a320 2023-06-22T08:22:05.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.028+0000 7f6698b99700 1 -- 172.21.15.23:0/4007519560 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668c012430 con 0x7f669415a320 2023-06-22T08:22:05.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.029+0000 7f669bdff700 1 -- 172.21.15.23:0/4007519560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669415a320 msgr2=0x7f6694160ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:05.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.029+0000 7f669bdff700 1 --2- 172.21.15.23:0/4007519560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669415a320 0x7f6694160ec0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f668c009d90 tx=0x7f668c00b330 comp rx=0 tx=0).stop 2023-06-22T08:22:05.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.029+0000 7f669bdff700 1 -- 172.21.15.23:0/4007519560 shutdown_connections 2023-06-22T08:22:05.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.029+0000 7f669bdff700 1 --2- 172.21.15.23:0/4007519560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669415a320 0x7f6694160ec0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:05.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.029+0000 7f669bdff700 1 --2- 172.21.15.23:0/4007519560 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f66940c3a40 0x7f66940c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:05.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.029+0000 7f669bdff700 1 -- 172.21.15.23:0/4007519560 >> 172.21.15.23:0/4007519560 conn(0x7f66940ca3f0 msgr2=0x7f66940ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:05.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.029+0000 7f669bdff700 1 -- 172.21.15.23:0/4007519560 shutdown_connections 2023-06-22T08:22:05.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.029+0000 7f669bdff700 1 -- 172.21.15.23:0/4007519560 wait complete. 2023-06-22T08:22:05.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.031+0000 7f669bdff700 1 Processor -- start 2023-06-22T08:22:05.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.031+0000 7f669bdff700 1 -- start start 2023-06-22T08:22:05.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f669bdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f669406d050 0x7f66941e1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:05.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f669bdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66940c3a40 0x7f66941ef730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:05.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f669bdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669415a320 0x7f66941dff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:05.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f669bdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66941f1b90 con 0x7f66940c3a40 2023-06-22T08:22:05.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f669bdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66941e07c0 con 0x7f669415a320 2023-06-22T08:22:05.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f669bdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66941e12e0 con 0x7f669406d050 2023-06-22T08:22:05.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f669939a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66940c3a40 0x7f66941ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:05.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f6699b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f669406d050 0x7f66941e1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:05.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f6699b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f669406d050 0x7f66941e1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:46632/0 (socket says 172.21.15.23:46632) 2023-06-22T08:22:05.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f669a39c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669415a320 0x7f66941dff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:05.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f6699b9b700 1 -- 172.21.15.23:0/2860783521 learned_addr learned my addr 172.21.15.23:0/2860783521 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:05.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f6699b9b700 1 -- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669415a320 msgr2=0x7f66941dff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:05.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f6699b9b700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669415a320 0x7f66941dff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:05.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f6699b9b700 1 -- 172.21.15.23:0/2860783521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66940c3a40 msgr2=0x7f66941ef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:05.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f6699b9b700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66940c3a40 0x7f66941ef730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:05.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f6699b9b700 1 -- 172.21.15.23:0/2860783521 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66900094a0 con 0x7f669406d050 2023-06-22T08:22:05.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f669a39c700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669415a320 0x7f66941dff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:22:05.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.032+0000 7f669939a700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66940c3a40 0x7f66941ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:22:05.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.033+0000 7f6699b9b700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f669406d050 0x7f66941e1770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f669000ec10 tx=0x7f669000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:05.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.033+0000 7f668a7fc700 1 -- 172.21.15.23:0/2860783521 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669000c850 con 0x7f669406d050 2023-06-22T08:22:05.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.033+0000 7f668a7fc700 1 -- 172.21.15.23:0/2860783521 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669000c9b0 con 0x7f669406d050 2023-06-22T08:22:05.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.033+0000 7f668a7fc700 1 -- 172.21.15.23:0/2860783521 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669000cd20 con 0x7f669406d050 2023-06-22T08:22:05.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.033+0000 7f669bdff700 1 -- 172.21.15.23:0/2860783521 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668c009a70 con 0x7f669406d050 2023-06-22T08:22:05.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.033+0000 7f669bdff700 1 -- 172.21.15.23:0/2860783521 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66941e09b0 con 0x7f669406d050 2023-06-22T08:22:05.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.034+0000 7f669bdff700 1 -- 172.21.15.23:0/2860783521 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669404ea20 con 0x7f669406d050 2023-06-22T08:22:05.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.035+0000 7f668a7fc700 1 -- 172.21.15.23:0/2860783521 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f6690013470 con 0x7f669406d050 2023-06-22T08:22:05.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.036+0000 7f668a7fc700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667806ad10 0x7f667806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:05.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.036+0000 7f668a7fc700 1 -- 172.21.15.23:0/2860783521 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f6690089fc0 con 0x7f669406d050 2023-06-22T08:22:05.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.036+0000 7f669939a700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667806ad10 0x7f667806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:05.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.037+0000 7f669939a700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667806ad10 0x7f667806d1c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f66940c68a0 tx=0x7f6684007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:05.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.041+0000 7f668a7fc700 1 -- 172.21.15.23:0/2860783521 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f6690054930 con 0x7f669406d050 2023-06-22T08:22:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.294+0000 7f669bdff700 1 -- 172.21.15.23:0/2860783521 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66940c5f30 con 0x7f669406d050 2023-06-22T08:22:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.296+0000 7f668a7fc700 1 -- 172.21.15.23:0/2860783521 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4737 (secure 0 0 0) 0x7f6690021090 con 0x7f669406d050 2023-06-22T08:22:05.298 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:05.298 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:04.632858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 -- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667806ad10 msgr2=0x7f667806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667806ad10 0x7f667806d1c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f66940c68a0 tx=0x7f6684007dc0 comp rx=0 tx=0).stop 2023-06-22T08:22:05.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 -- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f669406d050 msgr2=0x7f66941e1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:05.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f669406d050 0x7f66941e1770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f669000ec10 tx=0x7f669000ef20 comp rx=0 tx=0).stop 2023-06-22T08:22:05.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 -- 172.21.15.23:0/2860783521 shutdown_connections 2023-06-22T08:22:05.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f667806ad10 0x7f667806d1c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:05.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f669415a320 0x7f66941dff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:05.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f669406d050 0x7f66941e1770 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:05.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 --2- 172.21.15.23:0/2860783521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66940c3a40 0x7f66941ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:05.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 -- 172.21.15.23:0/2860783521 >> 172.21.15.23:0/2860783521 conn(0x7f66940ca3f0 msgr2=0x7f66940c6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:05.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 -- 172.21.15.23:0/2860783521 shutdown_connections 2023-06-22T08:22:05.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:05.298+0000 7f669bdff700 1 -- 172.21.15.23:0/2860783521 wait complete. 2023-06-22T08:22:05.304 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 172 2023-06-22T08:22:05.319 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 172, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 172, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 }, "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 36 }, "gid_6616": { "addr": "172.21.15.67:6837/1855579717", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1855579717, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1855579717, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6616, "incarnation": 162, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_6730": { "addr": "172.21.15.23:6834/456927633", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 456927633, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 456927633, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6730, "incarnation": 172, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 4, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:22:04.632858+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 6616, "mds_1": 6730, "mds_2": 6313, "mds_3": 5995 } } } ], "standbys": [] } 2023-06-22T08:22:07.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:07.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.530+0000 7fc4b3dff700 1 Processor -- start 2023-06-22T08:22:07.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.531+0000 7fc4b3dff700 1 -- start start 2023-06-22T08:22:07.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.532+0000 7fc4b3dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc4ac152d70 0x7fc4ac0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:07.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.532+0000 7fc4b3dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac159330 0x7fc4ac0cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:07.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.532+0000 7fc4b3dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ac15bea0 con 0x7fc4ac0cb800 2023-06-22T08:22:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.532+0000 7fc4b3dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ac0c95e0 con 0x7fc4ac152d70 2023-06-22T08:22:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.532+0000 7fc4b3dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac15c160 con 0x7fc4ac159330 2023-06-22T08:22:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.532+0000 7fc4b139a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac159330 0x7fc4ac0cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.532+0000 7fc4b1b9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc4ac152d70 0x7fc4ac0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:07.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.532+0000 7fc4b239c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc4ac0cb800 0x7fc4ac0c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:53216/0 (socket says 172.21.15.23:53216) 2023-06-22T08:22:07.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.532+0000 7fc4b239c700 1 -- 172.21.15.23:0/1363698601 learned_addr learned my addr 172.21.15.23:0/1363698601 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:22:07.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.533+0000 7fc4b139a700 1 -- 172.21.15.23:0/1363698601 >> v2:172.21.15.67:3300/0 conn(0x7fc4ac152d70 msgr2=0x7fc4ac0c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:07.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.533+0000 7fc4b139a700 1 --2- 172.21.15.23:0/1363698601 >> v2:172.21.15.67:3300/0 conn(0x7fc4ac152d70 0x7fc4ac0c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:07.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.533+0000 7fc4b139a700 1 -- 172.21.15.23:0/1363698601 >> v1:172.21.15.67:6789/0 conn(0x7fc4ac0cb800 legacy=0x7fc4ac0c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:07.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.533+0000 7fc4b139a700 1 -- 172.21.15.23:0/1363698601 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac15c440 con 0x7fc4ac159330 2023-06-22T08:22:07.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.533+0000 7fc4b139a700 1 --2- 172.21.15.23:0/1363698601 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac159330 0x7fc4ac0cb2c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009840 tx=0x7fc49c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aef7d6a1667b6fd9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:07.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.533+0000 7fc4b0b99700 1 -- 172.21.15.23:0/1363698601 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c011830 con 0x7fc4ac159330 2023-06-22T08:22:07.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.533+0000 7fc4b0b99700 1 -- 172.21.15.23:0/1363698601 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c011990 con 0x7fc4ac159330 2023-06-22T08:22:07.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.533+0000 7fc4b0b99700 1 -- 172.21.15.23:0/1363698601 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c011d20 con 0x7fc4ac159330 2023-06-22T08:22:07.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.534+0000 7fc4b3dff700 1 -- 172.21.15.23:0/1363698601 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac159330 msgr2=0x7fc4ac0cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:07.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.534+0000 7fc4b3dff700 1 --2- 172.21.15.23:0/1363698601 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac159330 0x7fc4ac0cb2c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009840 tx=0x7fc49c009b50 comp rx=0 tx=0).stop 2023-06-22T08:22:07.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.534+0000 7fc4b3dff700 1 -- 172.21.15.23:0/1363698601 shutdown_connections 2023-06-22T08:22:07.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.534+0000 7fc4b3dff700 1 --2- 172.21.15.23:0/1363698601 >> v2:172.21.15.67:3300/0 conn(0x7fc4ac152d70 0x7fc4ac0c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:07.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.534+0000 7fc4b3dff700 1 --2- 172.21.15.23:0/1363698601 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac159330 0x7fc4ac0cb2c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:07.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.534+0000 7fc4b3dff700 1 -- 172.21.15.23:0/1363698601 >> 172.21.15.23:0/1363698601 conn(0x7fc4ac06ae30 msgr2=0x7fc4ac06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:07.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.534+0000 7fc4b3dff700 1 -- 172.21.15.23:0/1363698601 shutdown_connections 2023-06-22T08:22:07.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.534+0000 7fc4b3dff700 1 -- 172.21.15.23:0/1363698601 wait complete. 2023-06-22T08:22:07.546 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.536+0000 7fc4b3dff700 1 Processor -- start 2023-06-22T08:22:07.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.536+0000 7fc4b3dff700 1 -- start start 2023-06-22T08:22:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b3dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ac0cb800 0x7fc4ac1f1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b3dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac152d70 0x7fc4ac149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b3dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4ac159330 0x7fc4ac148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b3dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac0cf730 con 0x7fc4ac159330 2023-06-22T08:22:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b3dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac1588a0 con 0x7fc4ac0cb800 2023-06-22T08:22:07.549 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b3dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac0cf200 con 0x7fc4ac152d70 2023-06-22T08:22:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b239c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4ac159330 0x7fc4ac148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b139a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac152d70 0x7fc4ac149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b1b9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ac0cb800 0x7fc4ac1f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b1b9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ac0cb800 0x7fc4ac1f1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:42842/0 (socket says 172.21.15.23:42842) 2023-06-22T08:22:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b139a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac152d70 0x7fc4ac149ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:56472/0 (socket says 172.21.15.23:56472) 2023-06-22T08:22:07.550 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.537+0000 7fc4b1b9b700 1 -- 172.21.15.23:0/4214762813 learned_addr learned my addr 172.21.15.23:0/4214762813 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4b139a700 1 -- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ac0cb800 msgr2=0x7fc4ac1f1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4b139a700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ac0cb800 0x7fc4ac1f1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4b139a700 1 -- 172.21.15.23:0/4214762813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4ac159330 msgr2=0x7fc4ac148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4b139a700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4ac159330 0x7fc4ac148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:07.551 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4b139a700 1 -- 172.21.15.23:0/4214762813 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c0094a0 con 0x7fc4ac152d70 2023-06-22T08:22:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4b1b9b700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ac0cb800 0x7fc4ac1f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:22:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4b139a700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac152d70 0x7fc4ac149ab0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc49c016000 tx=0x7fc49c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4a27fc700 1 -- 172.21.15.23:0/4214762813 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c0036a0 con 0x7fc4ac152d70 2023-06-22T08:22:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4a27fc700 1 -- 172.21.15.23:0/4214762813 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49c01f070 con 0x7fc4ac152d70 2023-06-22T08:22:07.552 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4a27fc700 1 -- 172.21.15.23:0/4214762813 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49c003a20 con 0x7fc4ac152d70 2023-06-22T08:22:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.538+0000 7fc4b3dff700 1 -- 172.21.15.23:0/4214762813 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac1f2250 con 0x7fc4ac152d70 2023-06-22T08:22:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.539+0000 7fc4b3dff700 1 -- 172.21.15.23:0/4214762813 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac1f24c0 con 0x7fc4ac152d70 2023-06-22T08:22:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.539+0000 7fc4b3dff700 1 -- 172.21.15.23:0/4214762813 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac0679d0 con 0x7fc4ac152d70 2023-06-22T08:22:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.541+0000 7fc4a27fc700 1 -- 172.21.15.23:0/4214762813 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc49c003b80 con 0x7fc4ac152d70 2023-06-22T08:22:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.542+0000 7fc4a27fc700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc49006ad10 0x7fc49006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.542+0000 7fc4a27fc700 1 -- 172.21.15.23:0/4214762813 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7901+0+0 (secure 0 0 0) 0x7fc49c021070 con 0x7fc4ac152d70 2023-06-22T08:22:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.542+0000 7fc4b1b9b700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc49006ad10 0x7fc49006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:07.554 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.542+0000 7fc4b1b9b700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc49006ad10 0x7fc49006d1c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80113d0 tx=0x7fc4a8009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:07.555 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.546+0000 7fc4a27fc700 1 -- 172.21.15.23:0/4214762813 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc49c0559d0 con 0x7fc4ac152d70 2023-06-22T08:22:07.801 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.798+0000 7fc4b3dff700 1 -- 172.21.15.23:0/4214762813 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4ac04ea20 con 0x7fc4ac152d70 2023-06-22T08:22:07.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.799+0000 7fc4a27fc700 1 -- 172.21.15.23:0/4214762813 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4737 (secure 0 0 0) 0x7fc49c058ff0 con 0x7fc4ac152d70 2023-06-22T08:22:07.802 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:07.802 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:04.632858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 -- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc49006ad10 msgr2=0x7fc49006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc49006ad10 0x7fc49006d1c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80113d0 tx=0x7fc4a8009210 comp rx=0 tx=0).stop 2023-06-22T08:22:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 -- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac152d70 msgr2=0x7fc4ac149ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac152d70 0x7fc4ac149ab0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc49c016000 tx=0x7fc49c004750 comp rx=0 tx=0).stop 2023-06-22T08:22:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 -- 172.21.15.23:0/4214762813 shutdown_connections 2023-06-22T08:22:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4ac159330 0x7fc4ac148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc49006ad10 0x7fc49006d1c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc4ac0cb800 0x7fc4ac1f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 --2- 172.21.15.23:0/4214762813 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc4ac152d70 0x7fc4ac149ab0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 -- 172.21.15.23:0/4214762813 >> 172.21.15.23:0/4214762813 conn(0x7fc4ac06ae30 msgr2=0x7fc4ac14fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 -- 172.21.15.23:0/4214762813 shutdown_connections 2023-06-22T08:22:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:07.802+0000 7fc4b3dff700 1 -- 172.21.15.23:0/4214762813 wait complete. 2023-06-22T08:22:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 172 2023-06-22T08:22:07.822 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:22:09.823 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:10.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.037+0000 7f1dbb9ff700 1 Processor -- start 2023-06-22T08:22:10.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.037+0000 7f1dbb9ff700 1 -- start start 2023-06-22T08:22:10.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1dbb9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db4152dd0 0x7f1db40aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:10.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1dbb9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1db40c8bd0 0x7f1db406afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:10.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1dbb9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1db406b500 0x7f1db40cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:10.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1dbb9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db40c7400 con 0x7f1db40c8bd0 2023-06-22T08:22:10.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1dbb9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db40c81c0 con 0x7f1db406b500 2023-06-22T08:22:10.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1dbb9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db40c84a0 con 0x7f1db4152dd0 2023-06-22T08:22:10.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1db979b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db4152dd0 0x7f1db40aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:10.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1db8f9a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1db40c8bd0 0x7f1db406afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:10.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1db979b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db4152dd0 0x7f1db40aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:56482/0 (socket says 172.21.15.23:56482) 2023-06-22T08:22:10.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1db979b700 1 -- 172.21.15.23:0/3660474961 learned_addr learned my addr 172.21.15.23:0/3660474961 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:10.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1db9f9c700 1 --2- 172.21.15.23:0/3660474961 >> v2:172.21.15.67:3300/0 conn(0x7f1db406b500 0x7f1db40cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:10.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.040+0000 7f1db9f9c700 1 -- 172.21.15.23:0/3660474961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db4152dd0 msgr2=0x7f1db40aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:10.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.041+0000 7f1db9f9c700 1 --2- 172.21.15.23:0/3660474961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db4152dd0 0x7f1db40aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.041+0000 7f1db9f9c700 1 -- 172.21.15.23:0/3660474961 >> v2:172.21.15.23:3300/0 conn(0x7f1db40c8bd0 msgr2=0x7f1db406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:10.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.041+0000 7f1db9f9c700 1 --2- 172.21.15.23:0/3660474961 >> v2:172.21.15.23:3300/0 conn(0x7f1db40c8bd0 0x7f1db406afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.041+0000 7f1db9f9c700 1 -- 172.21.15.23:0/3660474961 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db40cd370 con 0x7f1db406b500 2023-06-22T08:22:10.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.041+0000 7f1db979b700 1 --2- 172.21.15.23:0/3660474961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db4152dd0 0x7f1db40aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:22:10.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.041+0000 7f1db9f9c700 1 --2- 172.21.15.23:0/3660474961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db406b500 0x7f1db40cc640 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1db40c1700 tx=0x7f1dac00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7908c8b67b7c27d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:10.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.041+0000 7f1dabfff700 1 -- 172.21.15.23:0/3660474961 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dac00e070 con 0x7f1db406b500 2023-06-22T08:22:10.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.041+0000 7f1dabfff700 1 -- 172.21.15.23:0/3660474961 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dac00bb20 con 0x7f1db406b500 2023-06-22T08:22:10.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.041+0000 7f1dabfff700 1 -- 172.21.15.23:0/3660474961 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dac012430 con 0x7f1db406b500 2023-06-22T08:22:10.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.042+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/3660474961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db406b500 msgr2=0x7f1db40cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:10.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.042+0000 7f1dbb9ff700 1 --2- 172.21.15.23:0/3660474961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db406b500 0x7f1db40cc640 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1db40c1700 tx=0x7f1dac00b330 comp rx=0 tx=0).stop 2023-06-22T08:22:10.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.042+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/3660474961 shutdown_connections 2023-06-22T08:22:10.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.042+0000 7f1dbb9ff700 1 --2- 172.21.15.23:0/3660474961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db406b500 0x7f1db40cc640 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.042+0000 7f1dbb9ff700 1 --2- 172.21.15.23:0/3660474961 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db4152dd0 0x7f1db40aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.042+0000 7f1dbb9ff700 1 --2- 172.21.15.23:0/3660474961 >> v2:172.21.15.23:3300/0 conn(0x7f1db40c8bd0 0x7f1db406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.042+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/3660474961 >> 172.21.15.23:0/3660474961 conn(0x7f1db406fb70 msgr2=0x7f1db406ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:10.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.042+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/3660474961 shutdown_connections 2023-06-22T08:22:10.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.042+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/3660474961 wait complete. 2023-06-22T08:22:10.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.044+0000 7f1dbb9ff700 1 Processor -- start 2023-06-22T08:22:10.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.044+0000 7f1dbb9ff700 1 -- start start 2023-06-22T08:22:10.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1dbb9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1db406b500 0x7f1db40ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:10.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1dbb9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db40c8bd0 0x7f1db40cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:10.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1dbb9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db4152dd0 0x7f1db40cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:10.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1dbb9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db40afef0 con 0x7f1db406b500 2023-06-22T08:22:10.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1dbb9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db41eeb10 con 0x7f1db4152dd0 2023-06-22T08:22:10.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1dbb9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db41ef040 con 0x7f1db40c8bd0 2023-06-22T08:22:10.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1db979b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1db406b500 0x7f1db40ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:10.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1db979b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1db406b500 0x7f1db40ae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32776/0 (socket says 172.21.15.23:32776) 2023-06-22T08:22:10.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1db979b700 1 -- 172.21.15.23:0/4207300270 learned_addr learned my addr 172.21.15.23:0/4207300270 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:10.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1db8f9a700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db40c8bd0 0x7f1db40cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:10.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.045+0000 7f1db9f9c700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db4152dd0 0x7f1db40cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:10.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.046+0000 7f1db979b700 1 -- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db40c8bd0 msgr2=0x7f1db40cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:10.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.046+0000 7f1db979b700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db40c8bd0 0x7f1db40cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.046+0000 7f1db979b700 1 -- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db4152dd0 msgr2=0x7f1db40cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:10.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.046+0000 7f1db979b700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db4152dd0 0x7f1db40cbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.046+0000 7f1db979b700 1 -- 172.21.15.23:0/4207300270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db0009500 con 0x7f1db406b500 2023-06-22T08:22:10.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.046+0000 7f1db979b700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1db406b500 0x7f1db40ae270 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1db000e9d0 tx=0x7f1db000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:10.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.046+0000 7f1da9ffb700 1 -- 172.21.15.23:0/4207300270 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db000ee70 con 0x7f1db406b500 2023-06-22T08:22:10.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.046+0000 7f1da9ffb700 1 -- 172.21.15.23:0/4207300270 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db0005a20 con 0x7f1db406b500 2023-06-22T08:22:10.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.046+0000 7f1da9ffb700 1 -- 172.21.15.23:0/4207300270 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db0005db0 con 0x7f1db406b500 2023-06-22T08:22:10.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.046+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/4207300270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac009a70 con 0x7f1db406b500 2023-06-22T08:22:10.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.047+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/4207300270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db40af930 con 0x7f1db406b500 2023-06-22T08:22:10.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.048+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/4207300270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db415c080 con 0x7f1db406b500 2023-06-22T08:22:10.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.049+0000 7f1da9ffb700 1 -- 172.21.15.23:0/4207300270 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1db000c850 con 0x7f1db406b500 2023-06-22T08:22:10.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.050+0000 7f1da9ffb700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1d9806ace0 0x7f1d9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:10.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.050+0000 7f1da9ffb700 1 -- 172.21.15.23:0/4207300270 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f1db0089d00 con 0x7f1db406b500 2023-06-22T08:22:10.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.050+0000 7f1db8f9a700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1d9806ace0 0x7f1d9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:10.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.051+0000 7f1db8f9a700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1d9806ace0 0x7f1d9806d190 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1db40cc630 tx=0x7f1da4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:10.061 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.059+0000 7f1da9ffb700 1 -- 172.21.15.23:0/4207300270 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1db0054670 con 0x7f1db406b500 2023-06-22T08:22:10.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.316+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/4207300270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1db404ea20 con 0x7f1db406b500 2023-06-22T08:22:10.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.317+0000 7f1da9ffb700 1 -- 172.21.15.23:0/4207300270 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4737 (secure 0 0 0) 0x7f1db0020020 con 0x7f1db406b500 2023-06-22T08:22:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:10.320 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:04.632858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:10.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1d9806ace0 msgr2=0x7f1d9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:10.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1d9806ace0 0x7f1d9806d190 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1db40cc630 tx=0x7f1da4009210 comp rx=0 tx=0).stop 2023-06-22T08:22:10.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/4207300270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1db406b500 msgr2=0x7f1db40ae270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:10.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1db406b500 0x7f1db40ae270 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1db000e9d0 tx=0x7f1db000ece0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/4207300270 shutdown_connections 2023-06-22T08:22:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1d9806ace0 0x7f1d9806d190 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1db4152dd0 0x7f1db40cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1db406b500 0x7f1db40ae270 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 --2- 172.21.15.23:0/4207300270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1db40c8bd0 0x7f1db40cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/4207300270 >> 172.21.15.23:0/4207300270 conn(0x7f1db406fb70 msgr2=0x7f1db406e150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/4207300270 shutdown_connections 2023-06-22T08:22:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:10.319+0000 7f1dbb9ff700 1 -- 172.21.15.23:0/4207300270 wait complete. 2023-06-22T08:22:10.324 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 172 2023-06-22T08:22:10.341 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:22:11.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~109s 2023-06-22T08:22:12.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:12.560 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.557+0000 7f3ea85ff700 1 Processor -- start 2023-06-22T08:22:12.561 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.557+0000 7f3ea85ff700 1 -- start start 2023-06-22T08:22:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.558+0000 7f3ea85ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3ea00cf1b0 0x7f3ea00c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.558+0000 7f3ea85ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea015c370 con 0x7f3ea00c9fd0 2023-06-22T08:22:12.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.558+0000 7f3ea85ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea015bda0 con 0x7f3ea0152ce0 2023-06-22T08:22:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.558+0000 7f3ea85ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea01530b0 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.559+0000 7f3ea639b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3ea0152ce0 0x7f3ea015c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:53222/0 (socket says 172.21.15.23:53222) 2023-06-22T08:22:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.559+0000 7f3ea639b700 1 -- 172.21.15.23:0/1483583259 learned_addr learned my addr 172.21.15.23:0/1483583259 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:22:12.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.559+0000 7f3ea6b9c700 1 --2- 172.21.15.23:0/1483583259 >> v2:172.21.15.23:3300/0 conn(0x7f3ea00cf1b0 0x7f3ea00c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:12.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.559+0000 7f3ea6b9c700 1 -- 172.21.15.23:0/1483583259 >> v1:172.21.15.67:6789/0 conn(0x7f3ea0152ce0 legacy=0x7f3ea015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:12.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.559+0000 7f3ea6b9c700 1 -- 172.21.15.23:0/1483583259 >> v1:172.21.15.23:6789/0 conn(0x7f3ea00c9fd0 legacy=0x7f3ea00c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:12.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.559+0000 7f3ea5399700 1 -- 172.21.15.23:0/1483583259 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255221185 0 0) 0x7f3ea015bda0 con 0x7f3ea0152ce0 2023-06-22T08:22:12.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.559+0000 7f3ea6b9c700 1 -- 172.21.15.23:0/1483583259 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea00cf630 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.560+0000 7f3ea6b9c700 1 --2- 172.21.15.23:0/1483583259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 0x7f3ea00c47c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009d90 tx=0x7f3e9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf05c3a7caf56fe server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.560+0000 7f3ea5399700 1 -- 172.21.15.23:0/1483583259 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9800e070 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.560+0000 7f3ea5399700 1 -- 172.21.15.23:0/1483583259 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e9800bad0 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.565 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.560+0000 7f3ea5399700 1 -- 172.21.15.23:0/1483583259 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9800be80 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.561+0000 7f3ea85ff700 1 -- 172.21.15.23:0/1483583259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 msgr2=0x7f3ea00c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.561+0000 7f3ea85ff700 1 --2- 172.21.15.23:0/1483583259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 0x7f3ea00c47c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009d90 tx=0x7f3e9800b330 comp rx=0 tx=0).stop 2023-06-22T08:22:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.561+0000 7f3ea85ff700 1 -- 172.21.15.23:0/1483583259 shutdown_connections 2023-06-22T08:22:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.561+0000 7f3ea85ff700 1 --2- 172.21.15.23:0/1483583259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 0x7f3ea00c47c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.561+0000 7f3ea85ff700 1 -- 172.21.15.23:0/1483583259 >> 172.21.15.23:0/1483583259 conn(0x7f3ea006ae30 msgr2=0x7f3ea006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:12.566 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.561+0000 7f3ea85ff700 1 -- 172.21.15.23:0/1483583259 shutdown_connections 2023-06-22T08:22:12.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.561+0000 7f3ea85ff700 1 -- 172.21.15.23:0/1483583259 wait complete. 2023-06-22T08:22:12.567 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.564+0000 7f3ea85ff700 1 Processor -- start 2023-06-22T08:22:12.568 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.564+0000 7f3ea85ff700 1 -- start start 2023-06-22T08:22:12.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea85ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea00c9fd0 0x7f3ea0153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:12.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea85ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 0x7f3ea01ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:12.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea85ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ea0152ce0 0x7f3ea00cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:12.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea85ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea0154070 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.570 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea85ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea01560b0 con 0x7f3ea0152ce0 2023-06-22T08:22:12.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea85ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea01f2550 con 0x7f3ea00c9fd0 2023-06-22T08:22:12.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea5b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 0x7f3ea01ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:12.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea639b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea00c9fd0 0x7f3ea0153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:12.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea5b9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 0x7f3ea01ee7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32796/0 (socket says 172.21.15.23:32796) 2023-06-22T08:22:12.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea5b9a700 1 -- 172.21.15.23:0/4098777023 learned_addr learned my addr 172.21.15.23:0/4098777023 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:12.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.566+0000 7f3ea6b9c700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ea0152ce0 0x7f3ea00cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:12.571 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3ea5b9a700 1 -- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea00c9fd0 msgr2=0x7f3ea0153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3ea5b9a700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea00c9fd0 0x7f3ea0153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3ea5b9a700 1 -- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ea0152ce0 msgr2=0x7f3ea00cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3ea5b9a700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ea0152ce0 0x7f3ea00cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:12.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3ea5b9a700 1 -- 172.21.15.23:0/4098777023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e98009a70 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3ea5b9a700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 0x7f3ea01ee7f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000c780 tx=0x7f3e9000ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:12.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3e96ffd700 1 -- 172.21.15.23:0/4098777023 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e900054e0 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3e96ffd700 1 -- 172.21.15.23:0/4098777023 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e90003680 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3ea85ff700 1 -- 172.21.15.23:0/4098777023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea0153ab0 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.573 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3e96ffd700 1 -- 172.21.15.23:0/4098777023 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e90003a30 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.567+0000 7f3ea85ff700 1 -- 172.21.15.23:0/4098777023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea01f1f00 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.569+0000 7f3ea85ff700 1 -- 172.21.15.23:0/4098777023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea01f1d60 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.574 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.570+0000 7f3e96ffd700 1 -- 172.21.15.23:0/4098777023 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3e90003b90 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.571+0000 7f3e96ffd700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3e8406ace0 0x7f3e8406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:12.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.571+0000 7f3e96ffd700 1 -- 172.21.15.23:0/4098777023 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f3e900890d0 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.572+0000 7f3ea639b700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3e8406ace0 0x7f3e8406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:12.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.572+0000 7f3ea639b700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3e8406ace0 0x7f3e8406d190 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c000f60 tx=0x7f3e9c007e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:12.581 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.578+0000 7f3e96ffd700 1 -- 172.21.15.23:0/4098777023 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3ea01f1d60 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.868+0000 7f3ea85ff700 1 -- 172.21.15.23:0/4098777023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ea01f1d60 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.868+0000 7f3e96ffd700 1 -- 172.21.15.23:0/4098777023 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4737 (secure 0 0 0) 0x7f3ea01f1d60 con 0x7f3ea00cf1b0 2023-06-22T08:22:12.872 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:12.873 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:04.632858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:12.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 -- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3e8406ace0 msgr2=0x7f3e8406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:12.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3e8406ace0 0x7f3e8406d190 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c000f60 tx=0x7f3e9c007e70 comp rx=0 tx=0).stop 2023-06-22T08:22:12.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 -- 172.21.15.23:0/4098777023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 msgr2=0x7f3ea01ee7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:12.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 0x7f3ea01ee7f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000c780 tx=0x7f3e9000ca90 comp rx=0 tx=0).stop 2023-06-22T08:22:12.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 -- 172.21.15.23:0/4098777023 shutdown_connections 2023-06-22T08:22:12.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3e8406ace0 0x7f3e8406d190 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:12.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ea0152ce0 0x7f3ea00cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:12.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea00c9fd0 0x7f3ea0153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:12.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 --2- 172.21.15.23:0/4098777023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ea00cf1b0 0x7f3ea01ee7f0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:12.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 -- 172.21.15.23:0/4098777023 >> 172.21.15.23:0/4098777023 conn(0x7f3ea006ae30 msgr2=0x7f3ea0151fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:12.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 -- 172.21.15.23:0/4098777023 shutdown_connections 2023-06-22T08:22:12.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:12.871+0000 7f3ea85ff700 1 -- 172.21.15.23:0/4098777023 wait complete. 2023-06-22T08:22:12.877 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 172 2023-06-22T08:22:12.899 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:22:14.900 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:15.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.107+0000 7fb4e13ff700 1 Processor -- start 2023-06-22T08:22:15.111 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.107+0000 7fb4e13ff700 1 -- start start 2023-06-22T08:22:15.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.109+0000 7fb4e13ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 0x7fb4dc0c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:15.112 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.109+0000 7fb4e13ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc0c51a0 con 0x7fb4dc071070 2023-06-22T08:22:15.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.109+0000 7fb4e13ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc0c44e0 con 0x7fb4dc0c69e0 2023-06-22T08:22:15.113 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.109+0000 7fb4e13ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc0c5440 con 0x7fb4dc14f850 2023-06-22T08:22:15.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.109+0000 7fb4d3fff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb4dc0c69e0 0x7fb4dc155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:53238/0 (socket says 172.21.15.23:53238) 2023-06-22T08:22:15.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.109+0000 7fb4db7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb4dc071070 0x7fb4dc15fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:44284/0 (socket says 172.21.15.23:44284) 2023-06-22T08:22:15.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.109+0000 7fb4d3fff700 1 -- 172.21.15.23:0/4099435562 learned_addr learned my addr 172.21.15.23:0/4099435562 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:22:15.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.109+0000 7fb4daffd700 1 --2- 172.21.15.23:0/4099435562 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 0x7fb4dc0c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:15.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.110+0000 7fb4da7fc700 1 -- 172.21.15.23:0/4099435562 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214779948 0 0) 0x7fb4dc0c44e0 con 0x7fb4dc0c69e0 2023-06-22T08:22:15.115 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.110+0000 7fb4da7fc700 1 -- 172.21.15.23:0/4099435562 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c80035b0 con 0x7fb4dc0c69e0 2023-06-22T08:22:15.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.110+0000 7fb4da7fc700 1 -- 172.21.15.23:0/4099435562 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266010918 0 0) 0x7fb4dc0c51a0 con 0x7fb4dc071070 2023-06-22T08:22:15.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.110+0000 7fb4daffd700 1 -- 172.21.15.23:0/4099435562 >> v1:172.21.15.67:6789/0 conn(0x7fb4dc0c69e0 legacy=0x7fb4dc155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:15.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.110+0000 7fb4daffd700 1 -- 172.21.15.23:0/4099435562 >> v1:172.21.15.23:6789/0 conn(0x7fb4dc071070 legacy=0x7fb4dc15fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:15.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.110+0000 7fb4daffd700 1 -- 172.21.15.23:0/4099435562 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc0c5720 con 0x7fb4dc14f850 2023-06-22T08:22:15.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.110+0000 7fb4daffd700 1 --2- 172.21.15.23:0/4099435562 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 0x7fb4dc0c3080 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc009870 tx=0x7fb4cc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d12fc9e7ec257c20 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:15.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.110+0000 7fb4da7fc700 1 -- 172.21.15.23:0/4099435562 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc00f8a0 con 0x7fb4dc14f850 2023-06-22T08:22:15.116 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.110+0000 7fb4da7fc700 1 -- 172.21.15.23:0/4099435562 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4cc00fa00 con 0x7fb4dc14f850 2023-06-22T08:22:15.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.110+0000 7fb4da7fc700 1 -- 172.21.15.23:0/4099435562 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc00fd90 con 0x7fb4dc14f850 2023-06-22T08:22:15.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.111+0000 7fb4e13ff700 1 -- 172.21.15.23:0/4099435562 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 msgr2=0x7fb4dc0c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:15.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.111+0000 7fb4e13ff700 1 --2- 172.21.15.23:0/4099435562 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 0x7fb4dc0c3080 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc009870 tx=0x7fb4cc009b80 comp rx=0 tx=0).stop 2023-06-22T08:22:15.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.111+0000 7fb4e13ff700 1 -- 172.21.15.23:0/4099435562 shutdown_connections 2023-06-22T08:22:15.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.111+0000 7fb4e13ff700 1 --2- 172.21.15.23:0/4099435562 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 0x7fb4dc0c3080 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:15.117 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.111+0000 7fb4e13ff700 1 -- 172.21.15.23:0/4099435562 >> 172.21.15.23:0/4099435562 conn(0x7fb4dc0ca3f0 msgr2=0x7fb4dc0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:15.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.111+0000 7fb4e13ff700 1 -- 172.21.15.23:0/4099435562 shutdown_connections 2023-06-22T08:22:15.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.111+0000 7fb4e13ff700 1 -- 172.21.15.23:0/4099435562 wait complete. 2023-06-22T08:22:15.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.113+0000 7fb4e13ff700 1 Processor -- start 2023-06-22T08:22:15.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.113+0000 7fb4e13ff700 1 -- start start 2023-06-22T08:22:15.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4e13ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4dc071070 0x7fb4dc1f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:15.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4e13ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4dc0c69e0 0x7fb4dc1dca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:15.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4e13ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 0x7fb4dc0c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:15.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4e13ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc0c8300 con 0x7fb4dc0c69e0 2023-06-22T08:22:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4e13ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc1f2590 con 0x7fb4dc071070 2023-06-22T08:22:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4e13ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc1e3380 con 0x7fb4dc14f850 2023-06-22T08:22:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4d3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4dc0c69e0 0x7fb4dc1dca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4daffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4dc071070 0x7fb4dc1f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4db7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 0x7fb4dc0c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4daffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4dc071070 0x7fb4dc1f2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:42886/0 (socket says 172.21.15.23:42886) 2023-06-22T08:22:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4daffd700 1 -- 172.21.15.23:0/3335261064 learned_addr learned my addr 172.21.15.23:0/3335261064 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:15.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4daffd700 1 -- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 msgr2=0x7fb4dc0c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4daffd700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 0x7fb4dc0c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4daffd700 1 -- 172.21.15.23:0/3335261064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4dc0c69e0 msgr2=0x7fb4dc1dca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4daffd700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4dc0c69e0 0x7fb4dc1dca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.114+0000 7fb4daffd700 1 -- 172.21.15.23:0/3335261064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0094a0 con 0x7fb4dc071070 2023-06-22T08:22:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.115+0000 7fb4d3fff700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4dc0c69e0 0x7fb4dc1dca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:22:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.115+0000 7fb4daffd700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4dc071070 0x7fb4dc1f2050 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc0090b0 tx=0x7fb4cc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:15.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.115+0000 7fb4d37fe700 1 -- 172.21.15.23:0/3335261064 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc010cb0 con 0x7fb4dc071070 2023-06-22T08:22:15.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.115+0000 7fb4d37fe700 1 -- 172.21.15.23:0/3335261064 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4cc010e10 con 0x7fb4dc071070 2023-06-22T08:22:15.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.115+0000 7fb4d37fe700 1 -- 172.21.15.23:0/3335261064 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc019620 con 0x7fb4dc071070 2023-06-22T08:22:15.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.115+0000 7fb4e13ff700 1 -- 172.21.15.23:0/3335261064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4dc1ee430 con 0x7fb4dc071070 2023-06-22T08:22:15.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.115+0000 7fb4e13ff700 1 -- 172.21.15.23:0/3335261064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4dc1ee9f0 con 0x7fb4dc071070 2023-06-22T08:22:15.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.116+0000 7fb4e13ff700 1 -- 172.21.15.23:0/3335261064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4dc0c85a0 con 0x7fb4dc071070 2023-06-22T08:22:15.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.117+0000 7fb4d37fe700 1 -- 172.21.15.23:0/3335261064 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb4cc019780 con 0x7fb4dc071070 2023-06-22T08:22:15.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.118+0000 7fb4d37fe700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb4bc06ad10 0x7fb4bc06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:15.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.118+0000 7fb4d37fe700 1 -- 172.21.15.23:0/3335261064 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7901+0+0 (secure 0 0 0) 0x7fb4cc08afe0 con 0x7fb4dc071070 2023-06-22T08:22:15.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.119+0000 7fb4d3fff700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb4bc06ad10 0x7fb4bc06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:15.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.119+0000 7fb4d3fff700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb4bc06ad10 0x7fb4bc06d1c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc1dd390 tx=0x7fb4c4007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:15.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.123+0000 7fb4d37fe700 1 -- 172.21.15.23:0/3335261064 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb4cc055940 con 0x7fb4dc071070 2023-06-22T08:22:15.386 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.383+0000 7fb4e13ff700 1 -- 172.21.15.23:0/3335261064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4dc04ea20 con 0x7fb4dc071070 2023-06-22T08:22:15.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.385+0000 7fb4d37fe700 1 -- 172.21.15.23:0/3335261064 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4738 (secure 0 0 0) 0x7fb4cc058f60 con 0x7fb4dc071070 2023-06-22T08:22:15.389 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:15.389 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:13.923632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:15.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.386+0000 7fb4e13ff700 1 -- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb4bc06ad10 msgr2=0x7fb4bc06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:15.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.386+0000 7fb4e13ff700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb4bc06ad10 0x7fb4bc06d1c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc1dd390 tx=0x7fb4c4007dc0 comp rx=0 tx=0).stop 2023-06-22T08:22:15.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.386+0000 7fb4e13ff700 1 -- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4dc071070 msgr2=0x7fb4dc1f2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:15.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.386+0000 7fb4e13ff700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4dc071070 0x7fb4dc1f2050 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc0090b0 tx=0x7fb4cc004750 comp rx=0 tx=0).stop 2023-06-22T08:22:15.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.387+0000 7fb4e13ff700 1 -- 172.21.15.23:0/3335261064 shutdown_connections 2023-06-22T08:22:15.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.387+0000 7fb4e13ff700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb4dc14f850 0x7fb4dc0c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:15.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.387+0000 7fb4e13ff700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb4bc06ad10 0x7fb4bc06d1c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:15.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.387+0000 7fb4e13ff700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4dc071070 0x7fb4dc1f2050 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:15.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.387+0000 7fb4e13ff700 1 --2- 172.21.15.23:0/3335261064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4dc0c69e0 0x7fb4dc1dca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:15.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.387+0000 7fb4e13ff700 1 -- 172.21.15.23:0/3335261064 >> 172.21.15.23:0/3335261064 conn(0x7fb4dc0ca3f0 msgr2=0x7fb4dc06a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:15.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.387+0000 7fb4e13ff700 1 -- 172.21.15.23:0/3335261064 shutdown_connections 2023-06-22T08:22:15.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:15.387+0000 7fb4e13ff700 1 -- 172.21.15.23:0/3335261064 wait complete. 2023-06-22T08:22:15.392 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 173 2023-06-22T08:22:15.408 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:22:16.049 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:22:16.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:22:17.409 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:17.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.616+0000 7f3838bff700 1 Processor -- start 2023-06-22T08:22:17.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.616+0000 7f3838bff700 1 -- start start 2023-06-22T08:22:17.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.617+0000 7f3838bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38340c5740 0x7f3834069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:17.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.617+0000 7f3838bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3834158f60 con 0x7f38340cb6b0 2023-06-22T08:22:17.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.617+0000 7f3838bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38340cc450 con 0x7f3834152e10 2023-06-22T08:22:17.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.617+0000 7f3838bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383406bd20 con 0x7f38340c5740 2023-06-22T08:22:17.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.618+0000 7f383259c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f38340cb6b0 0x7f3834152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:38148/0 (socket says 172.21.15.23:38148) 2023-06-22T08:22:17.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.618+0000 7f383259c700 1 -- 172.21.15.23:0/1916270824 learned_addr learned my addr 172.21.15.23:0/1916270824 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:22:17.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.618+0000 7f3832d9d700 1 --2- 172.21.15.23:0/1916270824 >> v2:172.21.15.23:3300/0 conn(0x7f38340c5740 0x7f3834069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:17.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.618+0000 7f383159a700 1 -- 172.21.15.23:0/1916270824 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199516319 0 0) 0x7f3834158f60 con 0x7f38340cb6b0 2023-06-22T08:22:17.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.618+0000 7f383159a700 1 -- 172.21.15.23:0/1916270824 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f381c0035b0 con 0x7f38340cb6b0 2023-06-22T08:22:17.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.618+0000 7f3832d9d700 1 -- 172.21.15.23:0/1916270824 >> v1:172.21.15.67:6789/0 conn(0x7f3834152e10 legacy=0x7f3834161100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:17.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.618+0000 7f3832d9d700 1 -- 172.21.15.23:0/1916270824 >> v1:172.21.15.23:6789/0 conn(0x7f38340cb6b0 legacy=0x7f3834152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:17.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.618+0000 7f3832d9d700 1 -- 172.21.15.23:0/1916270824 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3834158980 con 0x7f38340c5740 2023-06-22T08:22:17.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.618+0000 7f3832d9d700 1 --2- 172.21.15.23:0/1916270824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38340c5740 0x7f3834069eb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f382c009d90 tx=0x7f382c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4191b19001fb4d7b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:17.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.619+0000 7f383159a700 1 -- 172.21.15.23:0/1916270824 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c00e070 con 0x7f38340c5740 2023-06-22T08:22:17.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.619+0000 7f383159a700 1 -- 172.21.15.23:0/1916270824 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c00bb20 con 0x7f38340c5740 2023-06-22T08:22:17.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.619+0000 7f383159a700 1 -- 172.21.15.23:0/1916270824 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c012430 con 0x7f38340c5740 2023-06-22T08:22:17.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.619+0000 7f3838bff700 1 -- 172.21.15.23:0/1916270824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38340c5740 msgr2=0x7f3834069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:17.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.619+0000 7f3838bff700 1 --2- 172.21.15.23:0/1916270824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38340c5740 0x7f3834069eb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f382c009d90 tx=0x7f382c00b330 comp rx=0 tx=0).stop 2023-06-22T08:22:17.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.619+0000 7f3838bff700 1 -- 172.21.15.23:0/1916270824 shutdown_connections 2023-06-22T08:22:17.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.619+0000 7f3838bff700 1 --2- 172.21.15.23:0/1916270824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38340c5740 0x7f3834069eb0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:17.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.619+0000 7f3838bff700 1 -- 172.21.15.23:0/1916270824 >> 172.21.15.23:0/1916270824 conn(0x7f3834070d80 msgr2=0x7f3834071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:17.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.619+0000 7f3838bff700 1 -- 172.21.15.23:0/1916270824 shutdown_connections 2023-06-22T08:22:17.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.619+0000 7f3838bff700 1 -- 172.21.15.23:0/1916270824 wait complete. 2023-06-22T08:22:17.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.621+0000 7f3838bff700 1 Processor -- start 2023-06-22T08:22:17.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.621+0000 7f3838bff700 1 -- start start 2023-06-22T08:22:17.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.622+0000 7f3838bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38340cb6b0 0x7f3834157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:17.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.622+0000 7f3838bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3834152e10 0x7f38341e3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:17.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.622+0000 7f3838bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3834156b30 0x7f38341e3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:17.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.622+0000 7f3838bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38341e5560 con 0x7f3834152e10 2023-06-22T08:22:17.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.622+0000 7f3838bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38341599d0 con 0x7f38340cb6b0 2023-06-22T08:22:17.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.622+0000 7f3838bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38341e2cb0 con 0x7f3834156b30 2023-06-22T08:22:17.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.622+0000 7f3831d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3834152e10 0x7f38341e3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:17.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3831d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3834152e10 0x7f38341e3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52800/0 (socket says 172.21.15.23:52800) 2023-06-22T08:22:17.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3831d9b700 1 -- 172.21.15.23:0/2249235681 learned_addr learned my addr 172.21.15.23:0/2249235681 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:17.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f383259c700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38340cb6b0 0x7f3834157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:17.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3832d9d700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3834156b30 0x7f38341e3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:17.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3831d9b700 1 -- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3834156b30 msgr2=0x7f38341e3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:17.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3831d9b700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3834156b30 0x7f38341e3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:17.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3831d9b700 1 -- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38340cb6b0 msgr2=0x7f3834157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:17.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3831d9b700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38340cb6b0 0x7f3834157d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:17.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3831d9b700 1 -- 172.21.15.23:0/2249235681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c009a70 con 0x7f3834152e10 2023-06-22T08:22:17.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3831d9b700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3834152e10 0x7f38341e3420 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f382800c420 tx=0x7f382800c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:17.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3822ffd700 1 -- 172.21.15.23:0/2249235681 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38280052d0 con 0x7f3834152e10 2023-06-22T08:22:17.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3822ffd700 1 -- 172.21.15.23:0/2249235681 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3828005430 con 0x7f3834152e10 2023-06-22T08:22:17.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.623+0000 7f3838bff700 1 -- 172.21.15.23:0/2249235681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38341e2f90 con 0x7f3834152e10 2023-06-22T08:22:17.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.624+0000 7f3822ffd700 1 -- 172.21.15.23:0/2249235681 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38280038c0 con 0x7f3834152e10 2023-06-22T08:22:17.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.624+0000 7f3838bff700 1 -- 172.21.15.23:0/2249235681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3834159c20 con 0x7f3834152e10 2023-06-22T08:22:17.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.625+0000 7f3838bff700 1 -- 172.21.15.23:0/2249235681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38341e3120 con 0x7f3834152e10 2023-06-22T08:22:17.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.626+0000 7f3822ffd700 1 -- 172.21.15.23:0/2249235681 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3828003a20 con 0x7f3834152e10 2023-06-22T08:22:17.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.626+0000 7f3822ffd700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f381406ace0 0x7f381406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:17.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.627+0000 7f3822ffd700 1 -- 172.21.15.23:0/2249235681 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f3828088ea0 con 0x7f3834152e10 2023-06-22T08:22:17.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.627+0000 7f383259c700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f381406ace0 0x7f381406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:17.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.627+0000 7f383259c700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f381406ace0 0x7f381406d190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3824007fd0 tx=0x7f3824007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:17.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.631+0000 7f3822ffd700 1 -- 172.21.15.23:0/2249235681 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3828053810 con 0x7f3834152e10 2023-06-22T08:22:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.886+0000 7f3838bff700 1 -- 172.21.15.23:0/2249235681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f383404ea20 con 0x7f3834152e10 2023-06-22T08:22:17.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.887+0000 7f3822ffd700 1 -- 172.21.15.23:0/2249235681 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+4738 (secure 0 0 0) 0x7f3828017090 con 0x7f3834152e10 2023-06-22T08:22:17.891 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:17.891 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:13.923632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:resolve","state_seq":7,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:17.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 -- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f381406ace0 msgr2=0x7f381406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:17.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f381406ace0 0x7f381406d190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3824007fd0 tx=0x7f3824007dc0 comp rx=0 tx=0).stop 2023-06-22T08:22:17.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 -- 172.21.15.23:0/2249235681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3834152e10 msgr2=0x7f38341e3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3834152e10 0x7f38341e3420 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f382800c420 tx=0x7f382800c730 comp rx=0 tx=0).stop 2023-06-22T08:22:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 -- 172.21.15.23:0/2249235681 shutdown_connections 2023-06-22T08:22:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3834156b30 0x7f38341e3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f381406ace0 0x7f381406d190 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38340cb6b0 0x7f3834157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 --2- 172.21.15.23:0/2249235681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3834152e10 0x7f38341e3420 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:17.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 -- 172.21.15.23:0/2249235681 >> 172.21.15.23:0/2249235681 conn(0x7f3834070d80 msgr2=0x7f383406a900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 -- 172.21.15.23:0/2249235681 shutdown_connections 2023-06-22T08:22:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:17.889+0000 7f3838bff700 1 -- 172.21.15.23:0/2249235681 wait complete. 2023-06-22T08:22:17.895 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 173 2023-06-22T08:22:17.911 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 173, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 173, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 }, "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 36 }, "gid_6616": { "addr": "172.21.15.67:6837/1855579717", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1855579717, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1855579717, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6616, "incarnation": 162, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_6730": { "addr": "172.21.15.23:6834/456927633", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 456927633, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 456927633, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6730, "incarnation": 172, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:resolve", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 4, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:22:13.923632+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 6616, "mds_1": 6730, "mds_2": 6313, "mds_3": 5995 } } } ], "standbys": [] } 2023-06-22T08:22:18.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~116s 2023-06-22T08:22:19.913 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:20.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.117+0000 7f5ec6bff700 1 Processor -- start 2023-06-22T08:22:20.121 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.118+0000 7f5ec6bff700 1 -- start start 2023-06-22T08:22:20.122 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.119+0000 7f5ec6bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ec00cf1b0 0x7f5ec00c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:20.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.119+0000 7f5ec6bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec015c370 con 0x7f5ec0152ce0 2023-06-22T08:22:20.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.119+0000 7f5ec6bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ec015bda0 con 0x7f5ec00c9fd0 2023-06-22T08:22:20.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.119+0000 7f5ec6bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec01530b0 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.119+0000 7f5ec499b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5ec0152ce0 0x7f5ec015c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:38162/0 (socket says 172.21.15.23:38162) 2023-06-22T08:22:20.123 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.119+0000 7f5ec499b700 1 -- 172.21.15.23:0/1857482153 learned_addr learned my addr 172.21.15.23:0/1857482153 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:22:20.124 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.119+0000 7f5ec519c700 1 --2- 172.21.15.23:0/1857482153 >> v2:172.21.15.67:3300/0 conn(0x7f5ec00cf1b0 0x7f5ec00c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.119+0000 7f5ebf7fe700 1 -- 172.21.15.23:0/1857482153 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926770444 0 0) 0x7f5ec015c370 con 0x7f5ec0152ce0 2023-06-22T08:22:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.120+0000 7f5ebf7fe700 1 -- 172.21.15.23:0/1857482153 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb00035b0 con 0x7f5ec0152ce0 2023-06-22T08:22:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.120+0000 7f5ec519c700 1 -- 172.21.15.23:0/1857482153 >> v1:172.21.15.67:6789/0 conn(0x7f5ec00c9fd0 legacy=0x7f5ec00c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.120+0000 7f5ec519c700 1 -- 172.21.15.23:0/1857482153 >> v1:172.21.15.23:6789/0 conn(0x7f5ec0152ce0 legacy=0x7f5ec015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.120+0000 7f5ec519c700 1 -- 172.21.15.23:0/1857482153 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec00cf630 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.120+0000 7f5ec519c700 1 --2- 172.21.15.23:0/1857482153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec00cf1b0 0x7f5ec00c47c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009d90 tx=0x7f5eb800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e3e6a904e937201 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.120+0000 7f5ebf7fe700 1 -- 172.21.15.23:0/1857482153 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb800e070 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.120+0000 7f5ebf7fe700 1 -- 172.21.15.23:0/1857482153 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb800bb20 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.120+0000 7f5ebf7fe700 1 -- 172.21.15.23:0/1857482153 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8012430 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.121+0000 7f5ec6bff700 1 -- 172.21.15.23:0/1857482153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec00cf1b0 msgr2=0x7f5ec00c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.121+0000 7f5ec6bff700 1 --2- 172.21.15.23:0/1857482153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec00cf1b0 0x7f5ec00c47c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009d90 tx=0x7f5eb800b330 comp rx=0 tx=0).stop 2023-06-22T08:22:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.121+0000 7f5ec6bff700 1 -- 172.21.15.23:0/1857482153 shutdown_connections 2023-06-22T08:22:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.121+0000 7f5ec6bff700 1 --2- 172.21.15.23:0/1857482153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec00cf1b0 0x7f5ec00c47c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.121+0000 7f5ec6bff700 1 -- 172.21.15.23:0/1857482153 >> 172.21.15.23:0/1857482153 conn(0x7f5ec006ae30 msgr2=0x7f5ec006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.121+0000 7f5ec6bff700 1 -- 172.21.15.23:0/1857482153 shutdown_connections 2023-06-22T08:22:20.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.121+0000 7f5ec6bff700 1 -- 172.21.15.23:0/1857482153 wait complete. 2023-06-22T08:22:20.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.123+0000 7f5ec6bff700 1 Processor -- start 2023-06-22T08:22:20.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.123+0000 7f5ec6bff700 1 -- start start 2023-06-22T08:22:20.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ec6bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec00c9fd0 0x7f5ec00cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:20.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ec6bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec00cf1b0 0x7f5ec014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:20.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ec6bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5ec0152ce0 0x7f5ec014c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:20.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ec6bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0157740 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ec6bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec01ee4b0 con 0x7f5ec00c9fd0 2023-06-22T08:22:20.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ec6bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0156680 con 0x7f5ec0152ce0 2023-06-22T08:22:20.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ebffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec00cf1b0 0x7f5ec014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:20.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ebffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec00cf1b0 0x7f5ec014c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52806/0 (socket says 172.21.15.23:52806) 2023-06-22T08:22:20.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ebffff700 1 -- 172.21.15.23:0/2909860050 learned_addr learned my addr 172.21.15.23:0/2909860050 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:20.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ec499b700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec00c9fd0 0x7f5ec00cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:20.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ec519c700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5ec0152ce0 0x7f5ec014c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:20.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ebffff700 1 -- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5ec0152ce0 msgr2=0x7f5ec014c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:20.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ebffff700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5ec0152ce0 0x7f5ec014c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:20.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ebffff700 1 -- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec00c9fd0 msgr2=0x7f5ec00cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:20.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ebffff700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec00c9fd0 0x7f5ec00cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:20.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.124+0000 7f5ebffff700 1 -- 172.21.15.23:0/2909860050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8009a70 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.125+0000 7f5ebffff700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec00cf1b0 0x7f5ec014c2c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f5eac00c4e0 tx=0x7f5eac00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:20.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.125+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/2909860050 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac0052d0 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.125+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/2909860050 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eac005430 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.125+0000 7f5ec6bff700 1 -- 172.21.15.23:0/2909860050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec01ee6c0 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.125+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/2909860050 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac0038c0 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.125+0000 7f5ec6bff700 1 -- 172.21.15.23:0/2909860050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec01eec80 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.126+0000 7f5ec6bff700 1 -- 172.21.15.23:0/2909860050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec004ea20 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.127+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/2909860050 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5eac003a20 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.128+0000 7f5ebd7fa700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ea406ace0 0x7f5ea406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:20.136 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.128+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/2909860050 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f5eac088f60 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.129+0000 7f5ec499b700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ea406ace0 0x7f5ea406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:20.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.129+0000 7f5ec499b700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ea406ace0 0x7f5ea406d190 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4007fd0 tx=0x7f5eb4007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:20.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.133+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/2909860050 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5eac0538d0 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.386+0000 7f5ec6bff700 1 -- 172.21.15.23:0/2909860050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ec00b13d0 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.386+0000 7f5ebd7fa700 1 -- 172.21.15.23:0/2909860050 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+4740 (secure 0 0 0) 0x7f5eac017090 con 0x7f5ec00cf1b0 2023-06-22T08:22:20.390 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:20.391 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:19.063816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:20.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 -- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ea406ace0 msgr2=0x7f5ea406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:20.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ea406ace0 0x7f5ea406d190 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4007fd0 tx=0x7f5eb4007dc0 comp rx=0 tx=0).stop 2023-06-22T08:22:20.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 -- 172.21.15.23:0/2909860050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec00cf1b0 msgr2=0x7f5ec014c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:20.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec00cf1b0 0x7f5ec014c2c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f5eac00c4e0 tx=0x7f5eac00c7f0 comp rx=0 tx=0).stop 2023-06-22T08:22:20.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 -- 172.21.15.23:0/2909860050 shutdown_connections 2023-06-22T08:22:20.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5ec0152ce0 0x7f5ec014c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:20.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5ea406ace0 0x7f5ea406d190 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ec00c9fd0 0x7f5ec00cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 --2- 172.21.15.23:0/2909860050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ec00cf1b0 0x7f5ec014c2c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 -- 172.21.15.23:0/2909860050 >> 172.21.15.23:0/2909860050 conn(0x7f5ec006ae30 msgr2=0x7f5ec014e840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 -- 172.21.15.23:0/2909860050 shutdown_connections 2023-06-22T08:22:20.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:20.389+0000 7f5ec6bff700 1 -- 172.21.15.23:0/2909860050 wait complete. 2023-06-22T08:22:20.395 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 174 2023-06-22T08:22:20.411 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:22:22.412 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.619+0000 7f3fb3dff700 1 Processor -- start 2023-06-22T08:22:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.619+0000 7f3fb3dff700 1 -- start start 2023-06-22T08:22:22.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb3dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3fac152ce0 0x7f3fac0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb3dff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3fac0c9fd0 0x7f3fac15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb3dff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fac15c370 con 0x7f3fac158880 2023-06-22T08:22:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb3dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fac1530b0 con 0x7f3fac152ce0 2023-06-22T08:22:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb3dff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fac0cf6f0 con 0x7f3fac0c9fd0 2023-06-22T08:22:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb1b9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3fac152ce0 0x7f3fac0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb1b9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3fac152ce0 0x7f3fac0cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52814/0 (socket says 172.21.15.23:52814) 2023-06-22T08:22:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb239c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f3fac158880 0x7f3fac158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:38174/0 (socket says 172.21.15.23:38174) 2023-06-22T08:22:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb1b9b700 1 -- 172.21.15.23:0/2609539299 learned_addr learned my addr 172.21.15.23:0/2609539299 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb139a700 1 --2- 172.21.15.23:0/2609539299 >> v2:172.21.15.67:3300/0 conn(0x7f3fac0c9fd0 0x7f3fac15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.622+0000 7f3fb1b9b700 1 -- 172.21.15.23:0/2609539299 >> v2:172.21.15.67:3300/0 conn(0x7f3fac0c9fd0 msgr2=0x7f3fac15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.622+0000 7f3fb1b9b700 1 --2- 172.21.15.23:0/2609539299 >> v2:172.21.15.67:3300/0 conn(0x7f3fac0c9fd0 0x7f3fac15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.621+0000 7f3fb0b99700 1 -- 172.21.15.23:0/2609539299 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907249338 0 0) 0x7f3fac15c370 con 0x7f3fac158880 2023-06-22T08:22:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.622+0000 7f3fb1b9b700 1 -- 172.21.15.23:0/2609539299 >> v1:172.21.15.23:6789/0 conn(0x7f3fac158880 legacy=0x7f3fac158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.622+0000 7f3fb1b9b700 1 -- 172.21.15.23:0/2609539299 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac15bda0 con 0x7f3fac152ce0 2023-06-22T08:22:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.622+0000 7f3fb1b9b700 1 --2- 172.21.15.23:0/2609539299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 0x7f3fac0cf1b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0cc380 tx=0x7f3fa8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b1dde63ffae543 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.622+0000 7f3fb0b99700 1 -- 172.21.15.23:0/2609539299 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8011830 con 0x7f3fac152ce0 2023-06-22T08:22:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.622+0000 7f3fb0b99700 1 -- 172.21.15.23:0/2609539299 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa8011990 con 0x7f3fac152ce0 2023-06-22T08:22:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.622+0000 7f3fb0b99700 1 -- 172.21.15.23:0/2609539299 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8011d20 con 0x7f3fac152ce0 2023-06-22T08:22:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.623+0000 7f3fb3dff700 1 -- 172.21.15.23:0/2609539299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 msgr2=0x7f3fac0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.623+0000 7f3fb3dff700 1 --2- 172.21.15.23:0/2609539299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 0x7f3fac0cf1b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0cc380 tx=0x7f3fa8009bd0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.623+0000 7f3fb3dff700 1 -- 172.21.15.23:0/2609539299 shutdown_connections 2023-06-22T08:22:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.623+0000 7f3fb3dff700 1 --2- 172.21.15.23:0/2609539299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 0x7f3fac0cf1b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.623+0000 7f3fb3dff700 1 --2- 172.21.15.23:0/2609539299 >> v2:172.21.15.67:3300/0 conn(0x7f3fac0c9fd0 0x7f3fac15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.623+0000 7f3fb3dff700 1 -- 172.21.15.23:0/2609539299 >> 172.21.15.23:0/2609539299 conn(0x7f3fac06ae30 msgr2=0x7f3fac06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.623+0000 7f3fb3dff700 1 -- 172.21.15.23:0/2609539299 shutdown_connections 2023-06-22T08:22:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.623+0000 7f3fb3dff700 1 -- 172.21.15.23:0/2609539299 wait complete. 2023-06-22T08:22:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.624+0000 7f3fb3dff700 1 Processor -- start 2023-06-22T08:22:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.625+0000 7f3fb3dff700 1 -- start start 2023-06-22T08:22:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.625+0000 7f3fb3dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fac0c9fd0 0x7f3fac1510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.625+0000 7f3fb3dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 0x7f3fac151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.625+0000 7f3fb3dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3fac158880 0x7f3fac147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb3dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac1506b0 con 0x7f3fac152ce0 2023-06-22T08:22:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb3dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac0b2290 con 0x7f3fac0c9fd0 2023-06-22T08:22:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb3dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac1f1400 con 0x7f3fac158880 2023-06-22T08:22:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb139a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 0x7f3fac151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb139a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 0x7f3fac151cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52826/0 (socket says 172.21.15.23:52826) 2023-06-22T08:22:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb139a700 1 -- 172.21.15.23:0/478935323 learned_addr learned my addr 172.21.15.23:0/478935323 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb1b9b700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fac0c9fd0 0x7f3fac1510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb239c700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3fac158880 0x7f3fac147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb139a700 1 -- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3fac158880 msgr2=0x7f3fac147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb139a700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3fac158880 0x7f3fac147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb139a700 1 -- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fac0c9fd0 msgr2=0x7f3fac1510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb139a700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fac0c9fd0 0x7f3fac1510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb139a700 1 -- 172.21.15.23:0/478935323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa8009550 con 0x7f3fac152ce0 2023-06-22T08:22:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb239c700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3fac158880 0x7f3fac147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:22:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb1b9b700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fac0c9fd0 0x7f3fac1510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:22:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.626+0000 7f3fb139a700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 0x7f3fac151cb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c00e9b0 tx=0x7f3f9c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.627+0000 7f3fa27fc700 1 -- 172.21.15.23:0/478935323 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c009710 con 0x7f3fac152ce0 2023-06-22T08:22:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.627+0000 7f3fa27fc700 1 -- 172.21.15.23:0/478935323 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9c00c850 con 0x7f3fac152ce0 2023-06-22T08:22:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.627+0000 7f3fb3dff700 1 -- 172.21.15.23:0/478935323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac147a30 con 0x7f3fac152ce0 2023-06-22T08:22:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.627+0000 7f3fa27fc700 1 -- 172.21.15.23:0/478935323 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c00cbc0 con 0x7f3fac152ce0 2023-06-22T08:22:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.627+0000 7f3fb3dff700 1 -- 172.21.15.23:0/478935323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fac1476c0 con 0x7f3fac152ce0 2023-06-22T08:22:22.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.628+0000 7f3fb3dff700 1 -- 172.21.15.23:0/478935323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fac147bc0 con 0x7f3fac152ce0 2023-06-22T08:22:22.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.629+0000 7f3fa27fc700 1 -- 172.21.15.23:0/478935323 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3f9c00cd20 con 0x7f3fac152ce0 2023-06-22T08:22:22.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.630+0000 7f3fa27fc700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f9006ace0 0x7f3f9006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:22.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.630+0000 7f3fa27fc700 1 -- 172.21.15.23:0/478935323 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f3f9c089cb0 con 0x7f3fac152ce0 2023-06-22T08:22:22.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.630+0000 7f3fb1b9b700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f9006ace0 0x7f3f9006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:22.636 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.631+0000 7f3fb1b9b700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f9006ace0 0x7f3f9006d190 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8000c00 tx=0x7f3fa8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:22.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.634+0000 7f3fa27fc700 1 -- 172.21.15.23:0/478935323 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3f9c054620 con 0x7f3fac152ce0 2023-06-22T08:22:22.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.888+0000 7f3fb3dff700 1 -- 172.21.15.23:0/478935323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fac04ea20 con 0x7f3fac152ce0 2023-06-22T08:22:22.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.889+0000 7f3fa27fc700 1 -- 172.21.15.23:0/478935323 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+4740 (secure 0 0 0) 0x7f3f9c021090 con 0x7f3fac152ce0 2023-06-22T08:22:22.893 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:22.893 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:19.063816+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 -- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f9006ace0 msgr2=0x7f3f9006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f9006ace0 0x7f3f9006d190 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8000c00 tx=0x7f3fa8008030 comp rx=0 tx=0).stop 2023-06-22T08:22:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 -- 172.21.15.23:0/478935323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 msgr2=0x7f3fac151cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 0x7f3fac151cb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c00e9b0 tx=0x7f3f9c00ecc0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 -- 172.21.15.23:0/478935323 shutdown_connections 2023-06-22T08:22:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3fac158880 0x7f3fac147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f3f9006ace0 0x7f3f9006d190 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3fac0c9fd0 0x7f3fac1510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 --2- 172.21.15.23:0/478935323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fac152ce0 0x7f3fac151cb0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 -- 172.21.15.23:0/478935323 >> 172.21.15.23:0/478935323 conn(0x7f3fac06ae30 msgr2=0x7f3fac06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 -- 172.21.15.23:0/478935323 shutdown_connections 2023-06-22T08:22:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:22.891+0000 7f3fb3dff700 1 -- 172.21.15.23:0/478935323 wait complete. 2023-06-22T08:22:22.897 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 174 2023-06-22T08:22:22.913 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:22:24.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~122s 2023-06-22T08:22:24.914 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.123+0000 7f5e343ff700 1 Processor -- start 2023-06-22T08:22:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.124+0000 7f5e343ff700 1 -- start start 2023-06-22T08:22:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.125+0000 7f5e343ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e2c152ce0 0x7f5e2c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.125+0000 7f5e343ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5e2c0c9fd0 0x7f5e2c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.125+0000 7f5e343ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c15c370 con 0x7f5e2c158880 2023-06-22T08:22:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.125+0000 7f5e343ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c1530b0 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.125+0000 7f5e343ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c0cf6f0 con 0x7f5e2c152ce0 2023-06-22T08:22:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.125+0000 7f5e3199a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5e2c0c9fd0 0x7f5e2c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:25.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.125+0000 7f5e3199a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5e2c0c9fd0 0x7f5e2c15c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52840/0 (socket says 172.21.15.23:52840) 2023-06-22T08:22:25.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.125+0000 7f5e3199a700 1 -- 172.21.15.23:0/2594647494 learned_addr learned my addr 172.21.15.23:0/2594647494 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:25.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.125+0000 7f5e3219b700 1 --2- 172.21.15.23:0/2594647494 >> v2:172.21.15.67:3300/0 conn(0x7f5e2c152ce0 0x7f5e2c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:25.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.126+0000 7f5e3199a700 1 -- 172.21.15.23:0/2594647494 >> v2:172.21.15.67:3300/0 conn(0x7f5e2c152ce0 msgr2=0x7f5e2c0cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:25.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.126+0000 7f5e3199a700 1 --2- 172.21.15.23:0/2594647494 >> v2:172.21.15.67:3300/0 conn(0x7f5e2c152ce0 0x7f5e2c0cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:25.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.126+0000 7f5e3199a700 1 -- 172.21.15.23:0/2594647494 >> v1:172.21.15.67:6789/0 conn(0x7f5e2c158880 legacy=0x7f5e2c158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:25.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.126+0000 7f5e3199a700 1 -- 172.21.15.23:0/2594647494 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c15bda0 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.126+0000 7f5e3199a700 1 --2- 172.21.15.23:0/2594647494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 0x7f5e2c15c800 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c08ba10 tx=0x7f5e1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db689ed8f58a6746 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:25.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.126+0000 7f5e31199700 1 -- 172.21.15.23:0/2594647494 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c011830 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.126+0000 7f5e31199700 1 -- 172.21.15.23:0/2594647494 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c011990 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.126+0000 7f5e31199700 1 -- 172.21.15.23:0/2594647494 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c011d20 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.127+0000 7f5e343ff700 1 -- 172.21.15.23:0/2594647494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 msgr2=0x7f5e2c15c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:25.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.127+0000 7f5e343ff700 1 --2- 172.21.15.23:0/2594647494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 0x7f5e2c15c800 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c08ba10 tx=0x7f5e1c009bd0 comp rx=0 tx=0).stop 2023-06-22T08:22:25.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.127+0000 7f5e343ff700 1 -- 172.21.15.23:0/2594647494 shutdown_connections 2023-06-22T08:22:25.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.127+0000 7f5e343ff700 1 --2- 172.21.15.23:0/2594647494 >> v2:172.21.15.67:3300/0 conn(0x7f5e2c152ce0 0x7f5e2c0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:25.134 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.127+0000 7f5e343ff700 1 --2- 172.21.15.23:0/2594647494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 0x7f5e2c15c800 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:25.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.127+0000 7f5e343ff700 1 -- 172.21.15.23:0/2594647494 >> 172.21.15.23:0/2594647494 conn(0x7f5e2c06ae30 msgr2=0x7f5e2c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:25.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.127+0000 7f5e343ff700 1 -- 172.21.15.23:0/2594647494 shutdown_connections 2023-06-22T08:22:25.135 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.127+0000 7f5e343ff700 1 -- 172.21.15.23:0/2594647494 wait complete. 2023-06-22T08:22:25.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.129+0000 7f5e343ff700 1 Processor -- start 2023-06-22T08:22:25.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.129+0000 7f5e343ff700 1 -- start start 2023-06-22T08:22:25.139 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e343ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 0x7f5e2c1577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:25.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e343ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c152ce0 0x7f5e2c0cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:25.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e343ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e2c158880 0x7f5e2c0ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:25.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e343ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c156070 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.140 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e343ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c1554a0 con 0x7f5e2c152ce0 2023-06-22T08:22:25.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e343ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c153d80 con 0x7f5e2c158880 2023-06-22T08:22:25.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e3219b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 0x7f5e2c1577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:25.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e3219b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 0x7f5e2c1577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52842/0 (socket says 172.21.15.23:52842) 2023-06-22T08:22:25.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e3219b700 1 -- 172.21.15.23:0/852912406 learned_addr learned my addr 172.21.15.23:0/852912406 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:25.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e3299c700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e2c158880 0x7f5e2c0ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:25.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e3199a700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c152ce0 0x7f5e2c0cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e3219b700 1 -- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e2c158880 msgr2=0x7f5e2c0ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e3219b700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e2c158880 0x7f5e2c0ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e3219b700 1 -- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c152ce0 msgr2=0x7f5e2c0cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e3219b700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c152ce0 0x7f5e2c0cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.130+0000 7f5e3219b700 1 -- 172.21.15.23:0/852912406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c009550 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.131+0000 7f5e3219b700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 0x7f5e2c1577a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800e9b0 tx=0x7f5e2800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.131+0000 7f5e22ffd700 1 -- 172.21.15.23:0/852912406 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28009710 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.131+0000 7f5e22ffd700 1 -- 172.21.15.23:0/852912406 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2800c850 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.131+0000 7f5e343ff700 1 -- 172.21.15.23:0/852912406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2c155760 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.131+0000 7f5e22ffd700 1 -- 172.21.15.23:0/852912406 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2800cbc0 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.131+0000 7f5e343ff700 1 -- 172.21.15.23:0/852912406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2c155a20 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.132+0000 7f5e343ff700 1 -- 172.21.15.23:0/852912406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e2c0cc690 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.133+0000 7f5e22ffd700 1 -- 172.21.15.23:0/852912406 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5e2800cd20 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.134+0000 7f5e22ffd700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5e1006ace0 0x7f5e1006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:25.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.134+0000 7f5e22ffd700 1 -- 172.21.15.23:0/852912406 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f5e28089cd0 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.134+0000 7f5e3199a700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5e1006ace0 0x7f5e1006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:25.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.135+0000 7f5e3199a700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5e1006ace0 0x7f5e1006d190 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c000c00 tx=0x7f5e1c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:25.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.139+0000 7f5e22ffd700 1 -- 172.21.15.23:0/852912406 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5e280580d0 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.394+0000 7f5e343ff700 1 -- 172.21.15.23:0/852912406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e2c04ea20 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.394+0000 7f5e22ffd700 1 -- 172.21.15.23:0/852912406 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+4738 (secure 0 0 0) 0x7f5e28017090 con 0x7f5e2c0c9fd0 2023-06-22T08:22:25.397 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:25.397 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:24.955757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:25.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 -- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5e1006ace0 msgr2=0x7f5e1006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:25.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5e1006ace0 0x7f5e1006d190 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c000c00 tx=0x7f5e1c008030 comp rx=0 tx=0).stop 2023-06-22T08:22:25.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 -- 172.21.15.23:0/852912406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 msgr2=0x7f5e2c1577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:25.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 0x7f5e2c1577a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800e9b0 tx=0x7f5e2800ed70 comp rx=0 tx=0).stop 2023-06-22T08:22:25.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 -- 172.21.15.23:0/852912406 shutdown_connections 2023-06-22T08:22:25.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e2c158880 0x7f5e2c0ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e2c0c9fd0 0x7f5e2c1577a0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5e1006ace0 0x7f5e1006d190 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 --2- 172.21.15.23:0/852912406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e2c152ce0 0x7f5e2c0cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 -- 172.21.15.23:0/852912406 >> 172.21.15.23:0/852912406 conn(0x7f5e2c06ae30 msgr2=0x7f5e2c14b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 -- 172.21.15.23:0/852912406 shutdown_connections 2023-06-22T08:22:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:25.397+0000 7f5e343ff700 1 -- 172.21.15.23:0/852912406 wait complete. 2023-06-22T08:22:25.402 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 175 2023-06-22T08:22:25.418 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:22:27.419 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.626+0000 7fa3df7ff700 1 Processor -- start 2023-06-22T08:22:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.626+0000 7fa3df7ff700 1 -- start start 2023-06-22T08:22:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.627+0000 7fa3df7ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3d814f850 0x7fa3d80c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.627+0000 7fa3df7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 0x7fa3d80c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.627+0000 7fa3df7ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d80c51a0 con 0x7fa3d80c69e0 2023-06-22T08:22:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.628+0000 7fa3df7ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d80c44e0 con 0x7fa3d814f850 2023-06-22T08:22:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.628+0000 7fa3df7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d80c5440 con 0x7fa3d8071070 2023-06-22T08:22:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.628+0000 7fa3ddd9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 0x7fa3d80c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.628+0000 7fa3ddd9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 0x7fa3d80c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:33668/0 (socket says 172.21.15.23:33668) 2023-06-22T08:22:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.628+0000 7fa3ddd9c700 1 -- 172.21.15.23:0/3222322475 learned_addr learned my addr 172.21.15.23:0/3222322475 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.628+0000 7fa3dd59b700 1 --2- 172.21.15.23:0/3222322475 >> v2:172.21.15.67:3300/0 conn(0x7fa3d814f850 0x7fa3d80c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.628+0000 7fa3ddd9c700 1 -- 172.21.15.23:0/3222322475 >> v2:172.21.15.67:3300/0 conn(0x7fa3d814f850 msgr2=0x7fa3d80c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.628+0000 7fa3ddd9c700 1 --2- 172.21.15.23:0/3222322475 >> v2:172.21.15.67:3300/0 conn(0x7fa3d814f850 0x7fa3d80c3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:27.633 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.628+0000 7fa3ddd9c700 1 -- 172.21.15.23:0/3222322475 >> v1:172.21.15.67:6789/0 conn(0x7fa3d80c69e0 legacy=0x7fa3d8155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:27.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.628+0000 7fa3ddd9c700 1 -- 172.21.15.23:0/3222322475 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d80c5720 con 0x7fa3d8071070 2023-06-22T08:22:27.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.629+0000 7fa3ddd9c700 1 --2- 172.21.15.23:0/3222322475 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 0x7fa3d80c35c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000d150 tx=0x7fa3d000d460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c997f9518a862555 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:27.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.629+0000 7fa3cffff700 1 -- 172.21.15.23:0/3222322475 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0010070 con 0x7fa3d8071070 2023-06-22T08:22:27.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.629+0000 7fa3cffff700 1 -- 172.21.15.23:0/3222322475 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d000dd10 con 0x7fa3d8071070 2023-06-22T08:22:27.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.629+0000 7fa3cffff700 1 -- 172.21.15.23:0/3222322475 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d000b310 con 0x7fa3d8071070 2023-06-22T08:22:27.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.629+0000 7fa3df7ff700 1 -- 172.21.15.23:0/3222322475 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 msgr2=0x7fa3d80c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:27.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.629+0000 7fa3df7ff700 1 --2- 172.21.15.23:0/3222322475 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 0x7fa3d80c35c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000d150 tx=0x7fa3d000d460 comp rx=0 tx=0).stop 2023-06-22T08:22:27.634 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.630+0000 7fa3df7ff700 1 -- 172.21.15.23:0/3222322475 shutdown_connections 2023-06-22T08:22:27.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.630+0000 7fa3df7ff700 1 --2- 172.21.15.23:0/3222322475 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 0x7fa3d80c35c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:27.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.630+0000 7fa3df7ff700 1 --2- 172.21.15.23:0/3222322475 >> v2:172.21.15.67:3300/0 conn(0x7fa3d814f850 0x7fa3d80c3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:27.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.630+0000 7fa3df7ff700 1 -- 172.21.15.23:0/3222322475 >> 172.21.15.23:0/3222322475 conn(0x7fa3d80ca3f0 msgr2=0x7fa3d80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:27.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.630+0000 7fa3df7ff700 1 -- 172.21.15.23:0/3222322475 shutdown_connections 2023-06-22T08:22:27.635 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.630+0000 7fa3df7ff700 1 -- 172.21.15.23:0/3222322475 wait complete. 2023-06-22T08:22:27.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.631+0000 7fa3df7ff700 1 Processor -- start 2023-06-22T08:22:27.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.632+0000 7fa3df7ff700 1 -- start start 2023-06-22T08:22:27.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.632+0000 7fa3df7ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 0x7fa3d80c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:27.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.632+0000 7fa3df7ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3d80c69e0 0x7fa3d81f14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:27.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.632+0000 7fa3df7ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d814f850 0x7fa3d8146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:27.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.632+0000 7fa3df7ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d8150750 con 0x7fa3d814f850 2023-06-22T08:22:27.637 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.632+0000 7fa3df7ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d8154b80 con 0x7fa3d80c69e0 2023-06-22T08:22:27.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3df7ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d8151e50 con 0x7fa3d8071070 2023-06-22T08:22:27.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3ddd9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d814f850 0x7fa3d8146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:27.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3ddd9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d814f850 0x7fa3d8146cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41260/0 (socket says 172.21.15.23:41260) 2023-06-22T08:22:27.638 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3ddd9c700 1 -- 172.21.15.23:0/1967633214 learned_addr learned my addr 172.21.15.23:0/1967633214 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:27.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3dd59b700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 0x7fa3d80c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:27.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3dcd9a700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3d80c69e0 0x7fa3d81f14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:27.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3ddd9c700 1 -- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 msgr2=0x7fa3d80c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:27.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3ddd9c700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 0x7fa3d80c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:27.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3ddd9c700 1 -- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3d80c69e0 msgr2=0x7fa3d81f14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:27.639 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3ddd9c700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3d80c69e0 0x7fa3d81f14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:27.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3ddd9c700 1 -- 172.21.15.23:0/1967633214 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d000bd40 con 0x7fa3d814f850 2023-06-22T08:22:27.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.633+0000 7fa3ddd9c700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d814f850 0x7fa3d8146cc0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000d4a0 tx=0x7fa3d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:27.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.634+0000 7fa3cdffb700 1 -- 172.21.15.23:0/1967633214 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0009a10 con 0x7fa3d814f850 2023-06-22T08:22:27.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.634+0000 7fa3cdffb700 1 -- 172.21.15.23:0/1967633214 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d0009b70 con 0x7fa3d814f850 2023-06-22T08:22:27.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.634+0000 7fa3df7ff700 1 -- 172.21.15.23:0/1967633214 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d81522e0 con 0x7fa3d814f850 2023-06-22T08:22:27.640 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.634+0000 7fa3cdffb700 1 -- 172.21.15.23:0/1967633214 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d000b4c0 con 0x7fa3d814f850 2023-06-22T08:22:27.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.634+0000 7fa3df7ff700 1 -- 172.21.15.23:0/1967633214 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d81501f0 con 0x7fa3d814f850 2023-06-22T08:22:27.641 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.635+0000 7fa3df7ff700 1 -- 172.21.15.23:0/1967633214 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d80c2950 con 0x7fa3d814f850 2023-06-22T08:22:27.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.636+0000 7fa3cdffb700 1 -- 172.21.15.23:0/1967633214 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa3d0004040 con 0x7fa3d814f850 2023-06-22T08:22:27.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.637+0000 7fa3cdffb700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3bc06ace0 0x7fa3bc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:27.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.637+0000 7fa3cdffb700 1 -- 172.21.15.23:0/1967633214 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7901+0+0 (secure 0 0 0) 0x7fa3d008aed0 con 0x7fa3d814f850 2023-06-22T08:22:27.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.637+0000 7fa3dd59b700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3bc06ace0 0x7fa3bc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:27.642 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.638+0000 7fa3dd59b700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3bc06ace0 0x7fa3bc06d190 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009b40 tx=0x7fa3d4009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:27.644 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.641+0000 7fa3cdffb700 1 -- 172.21.15.23:0/1967633214 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa3d00557b0 con 0x7fa3d814f850 2023-06-22T08:22:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.893+0000 7fa3df7ff700 1 -- 172.21.15.23:0/1967633214 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3d804ea20 con 0x7fa3d814f850 2023-06-22T08:22:27.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.893+0000 7fa3cdffb700 1 -- 172.21.15.23:0/1967633214 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+4738 (secure 0 0 0) 0x7fa3d0058dd0 con 0x7fa3d814f850 2023-06-22T08:22:27.896 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:27.896 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:24.955757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:27.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 -- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3bc06ace0 msgr2=0x7fa3bc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:27.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3bc06ace0 0x7fa3bc06d190 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009b40 tx=0x7fa3d4009210 comp rx=0 tx=0).stop 2023-06-22T08:22:27.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 -- 172.21.15.23:0/1967633214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d814f850 msgr2=0x7fa3d8146cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:27.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d814f850 0x7fa3d8146cc0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000d4a0 tx=0x7fa3d0004ad0 comp rx=0 tx=0).stop 2023-06-22T08:22:27.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 -- 172.21.15.23:0/1967633214 shutdown_connections 2023-06-22T08:22:27.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3d814f850 0x7fa3d8146cc0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:27.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa3d8071070 0x7fa3d80c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:27.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa3bc06ace0 0x7fa3bc06d190 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 --2- 172.21.15.23:0/1967633214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3d80c69e0 0x7fa3d81f14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 -- 172.21.15.23:0/1967633214 >> 172.21.15.23:0/1967633214 conn(0x7fa3d80ca3f0 msgr2=0x7fa3d80c2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 -- 172.21.15.23:0/1967633214 shutdown_connections 2023-06-22T08:22:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:27.896+0000 7fa3df7ff700 1 -- 172.21.15.23:0/1967633214 wait complete. 2023-06-22T08:22:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 175 2023-06-22T08:22:27.917 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:22:29.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:30.005 INFO:tasks.workunit.client.0.smithi023.stderr:rmdir: failed to remove 'd1/.snap/*': No such file or directory 2023-06-22T08:22:30.009 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:22:30.010 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:22:30.010 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:22:30.110 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-parents.sh... 2023-06-22T08:22:30.111 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-parents.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-parents.sh 2023-06-22T08:22:30.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.157+0000 7f05f79ff700 1 Processor -- start 2023-06-22T08:22:30.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.157+0000 7f05f79ff700 1 -- start start 2023-06-22T08:22:30.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.158+0000 7f05f79ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05f0152d70 0x7f05f00c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:30.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f79ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f00c45f0 0x7f05f00c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:30.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f79ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f00c4f90 con 0x7f05f0159330 2023-06-22T08:22:30.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f79ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f00c95e0 con 0x7f05f0152d70 2023-06-22T08:22:30.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f79ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f0159790 con 0x7f05f00c45f0 2023-06-22T08:22:30.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f4f9a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f05f0159330 0x7f05f00cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:57010/0 (socket says 172.21.15.23:57010) 2023-06-22T08:22:30.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f4f9a700 1 -- 172.21.15.23:0/1099018773 learned_addr learned my addr 172.21.15.23:0/1099018773 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:22:30.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f579b700 1 --2- 172.21.15.23:0/1099018773 >> v2:172.21.15.23:3300/0 conn(0x7f05f0152d70 0x7f05f00c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:30.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f5f9c700 1 --2- 172.21.15.23:0/1099018773 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f00c45f0 0x7f05f00c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:30.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05e7fff700 1 -- 172.21.15.23:0/1099018773 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367307863 0 0) 0x7f05f00c4f90 con 0x7f05f0159330 2023-06-22T08:22:30.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05e7fff700 1 -- 172.21.15.23:0/1099018773 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05dc0035b0 con 0x7f05f0159330 2023-06-22T08:22:30.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f579b700 1 -- 172.21.15.23:0/1099018773 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f00c45f0 msgr2=0x7f05f00c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:30.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f579b700 1 --2- 172.21.15.23:0/1099018773 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f00c45f0 0x7f05f00c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:30.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f579b700 1 -- 172.21.15.23:0/1099018773 >> v1:172.21.15.67:6789/0 conn(0x7f05f0159330 legacy=0x7f05f00cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:30.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f579b700 1 -- 172.21.15.23:0/1099018773 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f015bea0 con 0x7f05f0152d70 2023-06-22T08:22:30.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.159+0000 7f05f5f9c700 1 --2- 172.21.15.23:0/1099018773 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f00c45f0 0x7f05f00c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:22:30.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.160+0000 7f05f579b700 1 --2- 172.21.15.23:0/1099018773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0152d70 0x7f05f00c97d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009840 tx=0x7f05ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d82b01d0823fd0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:30.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.160+0000 7f05e7fff700 1 -- 172.21.15.23:0/1099018773 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec011830 con 0x7f05f0152d70 2023-06-22T08:22:30.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.160+0000 7f05e7fff700 1 -- 172.21.15.23:0/1099018773 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec011990 con 0x7f05f0152d70 2023-06-22T08:22:30.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.160+0000 7f05e7fff700 1 -- 172.21.15.23:0/1099018773 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec011d20 con 0x7f05f0152d70 2023-06-22T08:22:30.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.160+0000 7f05f79ff700 1 -- 172.21.15.23:0/1099018773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0152d70 msgr2=0x7f05f00c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:30.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.160+0000 7f05f79ff700 1 --2- 172.21.15.23:0/1099018773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0152d70 0x7f05f00c97d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009840 tx=0x7f05ec009b50 comp rx=0 tx=0).stop 2023-06-22T08:22:30.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.161+0000 7f05f79ff700 1 -- 172.21.15.23:0/1099018773 shutdown_connections 2023-06-22T08:22:30.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.161+0000 7f05f79ff700 1 --2- 172.21.15.23:0/1099018773 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f00c45f0 0x7f05f00c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:30.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.161+0000 7f05f79ff700 1 --2- 172.21.15.23:0/1099018773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f0152d70 0x7f05f00c97d0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:30.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.161+0000 7f05f79ff700 1 -- 172.21.15.23:0/1099018773 >> 172.21.15.23:0/1099018773 conn(0x7f05f006ae30 msgr2=0x7f05f006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:30.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.161+0000 7f05f79ff700 1 -- 172.21.15.23:0/1099018773 shutdown_connections 2023-06-22T08:22:30.171 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.161+0000 7f05f79ff700 1 -- 172.21.15.23:0/1099018773 wait complete. 2023-06-22T08:22:30.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.162+0000 7f05f79ff700 1 Processor -- start 2023-06-22T08:22:30.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.163+0000 7f05f79ff700 1 -- start start 2023-06-22T08:22:30.174 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.163+0000 7f05f79ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f00c45f0 0x7f05f014e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:30.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.163+0000 7f05f79ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f0152d70 0x7f05f015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:30.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.163+0000 7f05f79ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f0159330 0x7f05f0157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:30.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.163+0000 7f05f79ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f014cb10 con 0x7f05f00c45f0 2023-06-22T08:22:30.175 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.163+0000 7f05f79ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f014f880 con 0x7f05f0159330 2023-06-22T08:22:30.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.163+0000 7f05f79ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00c8cc0 con 0x7f05f0152d70 2023-06-22T08:22:30.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f579b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f00c45f0 0x7f05f014e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:30.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f579b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f00c45f0 0x7f05f014e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41276/0 (socket says 172.21.15.23:41276) 2023-06-22T08:22:30.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f579b700 1 -- 172.21.15.23:0/1202256789 learned_addr learned my addr 172.21.15.23:0/1202256789 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:30.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f4f9a700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f0152d70 0x7f05f015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:30.176 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f5f9c700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f0159330 0x7f05f0157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f579b700 1 -- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f0152d70 msgr2=0x7f05f015b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f579b700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f0152d70 0x7f05f015b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f579b700 1 -- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f0159330 msgr2=0x7f05f0157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f579b700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f0159330 0x7f05f0157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f579b700 1 -- 172.21.15.23:0/1202256789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e8009a70 con 0x7f05f00c45f0 2023-06-22T08:22:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f4f9a700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f0152d70 0x7f05f015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:22:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f5f9c700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f0159330 0x7f05f0157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:22:30.177 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.164+0000 7f05f579b700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f00c45f0 0x7f05f014e060 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0090b0 tx=0x7f05ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.165+0000 7f05e5ffb700 1 -- 172.21.15.23:0/1202256789 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec0036a0 con 0x7f05f00c45f0 2023-06-22T08:22:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.165+0000 7f05e5ffb700 1 -- 172.21.15.23:0/1202256789 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec01f070 con 0x7f05f00c45f0 2023-06-22T08:22:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.165+0000 7f05e5ffb700 1 -- 172.21.15.23:0/1202256789 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec003a40 con 0x7f05f00c45f0 2023-06-22T08:22:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.165+0000 7f05f79ff700 1 -- 172.21.15.23:0/1202256789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ec0094a0 con 0x7f05f00c45f0 2023-06-22T08:22:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.165+0000 7f05f79ff700 1 -- 172.21.15.23:0/1202256789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f014cfe0 con 0x7f05f00c45f0 2023-06-22T08:22:30.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.166+0000 7f05f79ff700 1 -- 172.21.15.23:0/1202256789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f004ea20 con 0x7f05f00c45f0 2023-06-22T08:22:30.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.167+0000 7f05e5ffb700 1 -- 172.21.15.23:0/1202256789 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f05ec003ba0 con 0x7f05f00c45f0 2023-06-22T08:22:30.180 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'making directory tree and files' 2023-06-22T08:22:30.180 INFO:tasks.workunit.client.0.smithi023.stdout:making directory tree and files 2023-06-22T08:22:30.181 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir -p 1/a/b/c/ 2023-06-22T08:22:30.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.168+0000 7f05e5ffb700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05d406ad10 0x7f05d406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:30.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.168+0000 7f05e5ffb700 1 -- 172.21.15.23:0/1202256789 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f05ec021070 con 0x7f05f00c45f0 2023-06-22T08:22:30.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.168+0000 7f05f4f9a700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05d406ad10 0x7f05d406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:30.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.169+0000 7f05f4f9a700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05d406ad10 0x7f05d406d1c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f05f015cbe0 tx=0x7f05e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:30.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.172+0000 7f05e5ffb700 1 -- 172.21.15.23:0/1202256789 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f05ec055a00 con 0x7f05f00c45f0 2023-06-22T08:22:30.183 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'i'\''m file1' 2023-06-22T08:22:30.184 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'i'\''m file2' 2023-06-22T08:22:30.184 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'i'\''m file3' 2023-06-22T08:22:30.184 INFO:tasks.workunit.client.0.smithi023.stdout:snapshotting 2023-06-22T08:22:30.185 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo snapshotting 2023-06-22T08:22:30.185 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir 1/.snap/foosnap1 2023-06-22T08:22:30.199 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir 2 2023-06-22T08:22:30.202 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'moving tree' 2023-06-22T08:22:30.203 INFO:tasks.workunit.client.0.smithi023.stdout:moving tree 2023-06-22T08:22:30.203 INFO:tasks.workunit.client.0.smithi023.stderr:+ mv 1/a 2 2023-06-22T08:22:30.207 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'checking snapshot contains tree...' 2023-06-22T08:22:30.210 INFO:tasks.workunit.client.0.smithi023.stdout:checking snapshot contains tree... 2023-06-22T08:22:30.210 INFO:tasks.workunit.client.0.smithi023.stderr:++ find 1/.snap/foosnap1 2023-06-22T08:22:30.210 INFO:tasks.workunit.client.0.smithi023.stderr:++ wc -w 2023-06-22T08:22:30.432 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.429+0000 7f05f79ff700 1 -- 172.21.15.23:0/1202256789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05f00c9e00 con 0x7f05f00c45f0 2023-06-22T08:22:30.433 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.430+0000 7f05e5ffb700 1 -- 172.21.15.23:0/1202256789 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+4738 (secure 0 0 0) 0x7f05ec015030 con 0x7f05f00c45f0 2023-06-22T08:22:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:30.433 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:24.955757+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:30.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 -- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05d406ad10 msgr2=0x7f05d406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:30.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05d406ad10 0x7f05d406d1c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f05f015cbe0 tx=0x7f05e0009040 comp rx=0 tx=0).stop 2023-06-22T08:22:30.435 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 -- 172.21.15.23:0/1202256789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f00c45f0 msgr2=0x7f05f014e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:30.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f00c45f0 0x7f05f014e060 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0090b0 tx=0x7f05ec004750 comp rx=0 tx=0).stop 2023-06-22T08:22:30.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 -- 172.21.15.23:0/1202256789 shutdown_connections 2023-06-22T08:22:30.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f05d406ad10 0x7f05d406d1c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:30.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05f0159330 0x7f05f0157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:30.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05f00c45f0 0x7f05f014e060 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:30.436 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 --2- 172.21.15.23:0/1202256789 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f05f0152d70 0x7f05f015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:30.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 -- 172.21.15.23:0/1202256789 >> 172.21.15.23:0/1202256789 conn(0x7f05f006ae30 msgr2=0x7f05f0151500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:30.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 -- 172.21.15.23:0/1202256789 shutdown_connections 2023-06-22T08:22:30.437 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:30.433+0000 7f05f79ff700 1 -- 172.21.15.23:0/1202256789 wait complete. 2023-06-22T08:22:30.437 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 175 2023-06-22T08:22:30.454 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 175, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 175, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_5995": { "addr": "172.21.15.67:6839/298860192", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6838", "nonce": 298860192, "type": "v2" }, { "addr": "172.21.15.67:6839", "nonce": 298860192, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5995, "incarnation": 101, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 34 }, "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 36 }, "gid_6616": { "addr": "172.21.15.67:6837/1855579717", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1855579717, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1855579717, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6616, "incarnation": 162, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_6730": { "addr": "172.21.15.23:6834/456927633", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 456927633, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 456927633, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6730, "incarnation": 172, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 4, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:22:24.955757+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 6616, "mds_1": 6730, "mds_2": 6313, "mds_3": 5995 } } } ], "standbys": [] } 2023-06-22T08:22:31.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~129s 2023-06-22T08:22:32.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:32.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.663+0000 7f91db3ff700 1 Processor -- start 2023-06-22T08:22:32.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.663+0000 7f91db3ff700 1 -- start start 2023-06-22T08:22:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.664+0000 7f91db3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f91d40c3a40 0x7f91d40c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.664+0000 7f91db3ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d40c6a60 con 0x7f91d415a320 2023-06-22T08:22:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.664+0000 7f91db3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d414d690 con 0x7f91d406d050 2023-06-22T08:22:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.664+0000 7f91db3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d4157570 con 0x7f91d40c3a40 2023-06-22T08:22:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.664+0000 7f91d919b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f91d40c3a40 0x7f91d40c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.664+0000 7f91d919b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f91d40c3a40 0x7f91d40c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41284/0 (socket says 172.21.15.23:41284) 2023-06-22T08:22:32.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.664+0000 7f91d919b700 1 -- 172.21.15.23:0/381761509 learned_addr learned my addr 172.21.15.23:0/381761509 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:32.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.664+0000 7f91d999c700 1 --1- 172.21.15.23:0/381761509 >> v1:172.21.15.23:6789/0 conn(0x7f91d415a320 0x7f91d4160ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:44784/0 (socket says 172.21.15.23:44784) 2023-06-22T08:22:32.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.665+0000 7f91d919b700 1 -- 172.21.15.23:0/381761509 >> v1:172.21.15.67:6789/0 conn(0x7f91d406d050 legacy=0x7f91d4159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:32.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.665+0000 7f91d919b700 1 -- 172.21.15.23:0/381761509 >> v1:172.21.15.23:6789/0 conn(0x7f91d415a320 legacy=0x7f91d4160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:32.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.665+0000 7f91d919b700 1 -- 172.21.15.23:0/381761509 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d40c6430 con 0x7f91d40c3a40 2023-06-22T08:22:32.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.665+0000 7f91d919b700 1 --2- 172.21.15.23:0/381761509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d40c3a40 0x7f91d40c3e10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f91d00098f0 tx=0x7f91d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236df9b58492b69b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:32.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.665+0000 7f91cbfff700 1 -- 172.21.15.23:0/381761509 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d000b8a0 con 0x7f91d40c3a40 2023-06-22T08:22:32.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.665+0000 7f91cbfff700 1 -- 172.21.15.23:0/381761509 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d000ba00 con 0x7f91d40c3a40 2023-06-22T08:22:32.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.665+0000 7f91cbfff700 1 -- 172.21.15.23:0/381761509 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d000bd90 con 0x7f91d40c3a40 2023-06-22T08:22:32.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.666+0000 7f91db3ff700 1 -- 172.21.15.23:0/381761509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d40c3a40 msgr2=0x7f91d40c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:32.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.666+0000 7f91db3ff700 1 --2- 172.21.15.23:0/381761509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d40c3a40 0x7f91d40c3e10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f91d00098f0 tx=0x7f91d0009c00 comp rx=0 tx=0).stop 2023-06-22T08:22:32.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.666+0000 7f91db3ff700 1 -- 172.21.15.23:0/381761509 shutdown_connections 2023-06-22T08:22:32.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.666+0000 7f91db3ff700 1 --2- 172.21.15.23:0/381761509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d40c3a40 0x7f91d40c3e10 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:32.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.666+0000 7f91db3ff700 1 -- 172.21.15.23:0/381761509 >> 172.21.15.23:0/381761509 conn(0x7f91d40ca3f0 msgr2=0x7f91d40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:32.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.666+0000 7f91db3ff700 1 -- 172.21.15.23:0/381761509 shutdown_connections 2023-06-22T08:22:32.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.666+0000 7f91db3ff700 1 -- 172.21.15.23:0/381761509 wait complete. 2023-06-22T08:22:32.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.668+0000 7f91db3ff700 1 Processor -- start 2023-06-22T08:22:32.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.668+0000 7f91db3ff700 1 -- start start 2023-06-22T08:22:32.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91db3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d406d050 0x7f91d4144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:32.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91db3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f91d40c3a40 0x7f91d4158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:32.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91db3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d415a320 0x7f91d4159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:32.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91db3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d4149750 con 0x7f91d406d050 2023-06-22T08:22:32.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91db3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d414dc40 con 0x7f91d415a320 2023-06-22T08:22:32.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91db3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d414ee40 con 0x7f91d40c3a40 2023-06-22T08:22:32.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91d919b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d406d050 0x7f91d4144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:32.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91d919b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d406d050 0x7f91d4144030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41288/0 (socket says 172.21.15.23:41288) 2023-06-22T08:22:32.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91d919b700 1 -- 172.21.15.23:0/2548500223 learned_addr learned my addr 172.21.15.23:0/2548500223 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:32.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91d999c700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d415a320 0x7f91d4159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:32.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91d899a700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f91d40c3a40 0x7f91d4158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:32.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91d919b700 1 -- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f91d40c3a40 msgr2=0x7f91d4158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:32.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91d919b700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f91d40c3a40 0x7f91d4158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:32.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91d919b700 1 -- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d415a320 msgr2=0x7f91d4159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:32.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91d919b700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d415a320 0x7f91d4159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:32.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.669+0000 7f91d919b700 1 -- 172.21.15.23:0/2548500223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d0009550 con 0x7f91d406d050 2023-06-22T08:22:32.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.670+0000 7f91d919b700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d406d050 0x7f91d4144030 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f91d0000c00 tx=0x7f91d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:32.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.670+0000 7f91c9ffb700 1 -- 172.21.15.23:0/2548500223 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0004570 con 0x7f91d406d050 2023-06-22T08:22:32.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.670+0000 7f91c9ffb700 1 -- 172.21.15.23:0/2548500223 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d001f070 con 0x7f91d406d050 2023-06-22T08:22:32.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.670+0000 7f91c9ffb700 1 -- 172.21.15.23:0/2548500223 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d00038e0 con 0x7f91d406d050 2023-06-22T08:22:32.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.670+0000 7f91db3ff700 1 -- 172.21.15.23:0/2548500223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d4156fe0 con 0x7f91d406d050 2023-06-22T08:22:32.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.670+0000 7f91db3ff700 1 -- 172.21.15.23:0/2548500223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d41582d0 con 0x7f91d406d050 2023-06-22T08:22:32.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.671+0000 7f91db3ff700 1 -- 172.21.15.23:0/2548500223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d4158130 con 0x7f91d406d050 2023-06-22T08:22:32.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.672+0000 7f91c9ffb700 1 -- 172.21.15.23:0/2548500223 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f91d0021030 con 0x7f91d406d050 2023-06-22T08:22:32.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.673+0000 7f91c9ffb700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f91b806ace0 0x7f91b806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:32.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.673+0000 7f91c9ffb700 1 -- 172.21.15.23:0/2548500223 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f91d008b130 con 0x7f91d406d050 2023-06-22T08:22:32.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.673+0000 7f91d899a700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f91b806ace0 0x7f91b806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:32.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.674+0000 7f91d899a700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f91b806ace0 0x7f91b806d190 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f91d41563c0 tx=0x7f91c4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:32.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.678+0000 7f91c9ffb700 1 -- 172.21.15.23:0/2548500223 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f91d4158130 con 0x7f91d406d050 2023-06-22T08:22:32.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.933+0000 7f91db3ff700 1 -- 172.21.15.23:0/2548500223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91d4158130 con 0x7f91d406d050 2023-06-22T08:22:32.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.933+0000 7f91c9ffb700 1 -- 172.21.15.23:0/2548500223 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+4738 (secure 0 0 0) 0x7f91d4158130 con 0x7f91d406d050 2023-06-22T08:22:32.937 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:32.937 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:30.642319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:32.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 -- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f91b806ace0 msgr2=0x7f91b806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:32.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f91b806ace0 0x7f91b806d190 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f91d41563c0 tx=0x7f91c4009040 comp rx=0 tx=0).stop 2023-06-22T08:22:32.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 -- 172.21.15.23:0/2548500223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d406d050 msgr2=0x7f91d4144030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:32.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d406d050 0x7f91d4144030 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f91d0000c00 tx=0x7f91d0004750 comp rx=0 tx=0).stop 2023-06-22T08:22:32.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 -- 172.21.15.23:0/2548500223 shutdown_connections 2023-06-22T08:22:32.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f91b806ace0 0x7f91b806d190 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:32.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91d415a320 0x7f91d4159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:32.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d406d050 0x7f91d4144030 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:32.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 --2- 172.21.15.23:0/2548500223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f91d40c3a40 0x7f91d4158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:32.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 -- 172.21.15.23:0/2548500223 >> 172.21.15.23:0/2548500223 conn(0x7f91d40ca3f0 msgr2=0x7f91d4147f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:32.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 -- 172.21.15.23:0/2548500223 shutdown_connections 2023-06-22T08:22:32.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:32.936+0000 7f91db3ff700 1 -- 172.21.15.23:0/2548500223 wait complete. 2023-06-22T08:22:32.942 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 176 2023-06-22T08:22:32.957 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=5707 2023-06-22T08:22:32.957 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.a 2023-06-22T08:22:32.976 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T08:22:32.974+0000 7f6838c81700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-06-22T08:22:35.724 INFO:tasks.workunit.client.0.smithi023.stderr:+ dir1=7 2023-06-22T08:22:35.725 INFO:tasks.workunit.client.0.smithi023.stderr:++ find 2/ 2023-06-22T08:22:35.726 INFO:tasks.workunit.client.0.smithi023.stderr:++ wc -w 2023-06-22T08:22:35.733 INFO:tasks.workunit.client.0.smithi023.stderr:+ dir2=7 2023-06-22T08:22:35.733 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 7==7 2023-06-22T08:22:35.734 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'Success!' 2023-06-22T08:22:35.734 INFO:tasks.workunit.client.0.smithi023.stdout:Success! 2023-06-22T08:22:35.734 INFO:tasks.workunit.client.0.smithi023.stdout:adding folder and file to tree... 2023-06-22T08:22:35.735 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'adding folder and file to tree...' 2023-06-22T08:22:35.735 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir 2/a/b/c/d 2023-06-22T08:22:35.737 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'i'\''m file 4!' 2023-06-22T08:22:35.738 INFO:tasks.workunit.client.0.smithi023.stdout:snapshotting tree 2 2023-06-22T08:22:35.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'snapshotting tree 2' 2023-06-22T08:22:35.739 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir 2/.snap/barsnap2 2023-06-22T08:22:35.752 INFO:tasks.workunit.client.0.smithi023.stdout:comparing snapshots 2023-06-22T08:22:35.752 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'comparing snapshots' 2023-06-22T08:22:35.753 INFO:tasks.workunit.client.0.smithi023.stderr:++ find 1/.snap/foosnap1/ -maxdepth 2 2023-06-22T08:22:35.753 INFO:tasks.workunit.client.0.smithi023.stderr:++ wc -w 2023-06-22T08:22:35.760 INFO:tasks.workunit.client.0.smithi023.stderr:+ dir1=4 2023-06-22T08:22:35.761 INFO:tasks.workunit.client.0.smithi023.stderr:++ find 2/.snap/barsnap2/ -maxdepth 2 2023-06-22T08:22:35.762 INFO:tasks.workunit.client.0.smithi023.stderr:++ wc -w 2023-06-22T08:22:35.778 INFO:tasks.workunit.client.0.smithi023.stderr:+ dir2=4 2023-06-22T08:22:35.778 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 4==4 2023-06-22T08:22:35.779 INFO:tasks.workunit.client.0.smithi023.stdout:Success! 2023-06-22T08:22:35.779 INFO:tasks.workunit.client.0.smithi023.stdout:moving subtree to first folder 2023-06-22T08:22:35.779 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'Success!' 2023-06-22T08:22:35.779 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'moving subtree to first folder' 2023-06-22T08:22:35.779 INFO:tasks.workunit.client.0.smithi023.stderr:+ mv 2/a/b/c 1 2023-06-22T08:22:35.784 INFO:tasks.workunit.client.0.smithi023.stdout:comparing snapshots and new tree 2023-06-22T08:22:35.784 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'comparing snapshots and new tree' 2023-06-22T08:22:35.785 INFO:tasks.workunit.client.0.smithi023.stderr:++ find 1/ 2023-06-22T08:22:35.786 INFO:tasks.workunit.client.0.smithi023.stderr:++ wc -w 2023-06-22T08:22:35.794 INFO:tasks.workunit.client.0.smithi023.stderr:+ dir1=5 2023-06-22T08:22:35.795 INFO:tasks.workunit.client.0.smithi023.stderr:++ find 2/.snap/barsnap2/a/b/c 2023-06-22T08:22:35.796 INFO:tasks.workunit.client.0.smithi023.stderr:++ wc -w 2023-06-22T08:22:35.811 INFO:tasks.workunit.client.0.smithi023.stderr:+ dir2=4 2023-06-22T08:22:35.812 INFO:tasks.workunit.client.0.smithi023.stdout:Success! 2023-06-22T08:22:35.812 INFO:tasks.workunit.client.0.smithi023.stderr:+ test 5==4 2023-06-22T08:22:35.813 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'Success!' 2023-06-22T08:22:35.813 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir 1/.snap/foosnap1 2023-06-22T08:22:35.823 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir 2/.snap/barsnap2 2023-06-22T08:22:35.833 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:22:35.834 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:22:35.834 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:22:35.834 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:22:35.970 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-realm-split.sh... 2023-06-22T08:22:35.970 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-realm-split.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-realm-split.sh 2023-06-22T08:22:37.958 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-06-22T08:22:37.958 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-22T08:22:37.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-22T08:22:37.963 INFO:tasks.ceph.mds.a:Started 2023-06-22T08:22:37.963 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:38.096 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:22:38.096 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir -p 1/a 2023-06-22T08:22:38.122 INFO:tasks.workunit.client.0.smithi023.stderr:+ exec 2023-06-22T08:22:38.134 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo -n a 2023-06-22T08:22:38.134 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir 1/.snap/s1 2023-06-22T08:22:38.174 INFO:tasks.ceph.mds.a.smithi023.stdout:starting mds.a at 2023-06-22T08:22:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.290+0000 7f4ccc1ff700 1 Processor -- start 2023-06-22T08:22:38.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.291+0000 7f4ccc1ff700 1 -- start start 2023-06-22T08:22:38.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.293+0000 7f4ccc1ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4cc414eaf0 0x7f4cc40c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:38.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.293+0000 7f4ccc1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cc4069eb0 0x7f4cc406a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:38.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.293+0000 7f4ccc1ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc40c3dd0 con 0x7f4cc406e310 2023-06-22T08:22:38.296 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.293+0000 7f4ccc1ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc4158d10 con 0x7f4cc414eaf0 2023-06-22T08:22:38.297 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.293+0000 7f4ccc1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40c4800 con 0x7f4cc4069eb0 2023-06-22T08:22:38.297 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.294+0000 7f4cca79c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cc4069eb0 0x7f4cc406a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:38.297 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.295+0000 7f4cc9f9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4cc414eaf0 0x7f4cc40c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:38.298 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.296+0000 7f4cc979a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4cc406e310 0x7f4cc41548b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55246/0 (socket says 172.21.15.23:55246) 2023-06-22T08:22:38.299 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.296+0000 7f4cc979a700 1 -- 172.21.15.23:0/1079123044 learned_addr learned my addr 172.21.15.23:0/1079123044 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:22:38.299 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.297+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462112819 0 0) 0x7f4cc40c3dd0 con 0x7f4cc406e310 2023-06-22T08:22:38.299 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.297+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb00035b0 con 0x7f4cc406e310 2023-06-22T08:22:38.300 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.297+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 760288166 0 0) 0x7f4cb00035b0 con 0x7f4cc406e310 2023-06-22T08:22:38.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.297+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cc4069eb0 msgr2=0x7f4cc406a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:38.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.297+0000 7f4cc8f99700 1 --2- 172.21.15.23:0/1079123044 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cc4069eb0 0x7f4cc406a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.297+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 >> v2:172.21.15.67:3300/0 conn(0x7f4cc414eaf0 msgr2=0x7f4cc40c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:22:38.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.297+0000 7f4cc8f99700 1 --2- 172.21.15.23:0/1079123044 >> v2:172.21.15.67:3300/0 conn(0x7f4cc414eaf0 0x7f4cc40c5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.298+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc40c4520 con 0x7f4cc406e310 2023-06-22T08:22:38.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.298+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7f4cbc0019f0 con 0x7f4cc406e310 2023-06-22T08:22:38.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.298+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 >> v1:172.21.15.23:6789/0 conn(0x7f4cc406e310 legacy=0x7f4cc41548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:38.301 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.298+0000 7f4cc8f99700 1 --2- 172.21.15.23:0/1079123044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cb0003a00 0x7f4cb0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:38.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.298+0000 7f4cc8f99700 1 --2- 172.21.15.23:0/1079123044 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cb0004850 0x7f4cb0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:38.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.298+0000 7f4cc8f99700 1 --2- 172.21.15.23:0/1079123044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb0005250 0x7f4cb000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:38.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.298+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc4158d10 con 0x7f4cb0003a00 2023-06-22T08:22:38.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.298+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40c4800 con 0x7f4cb0005250 2023-06-22T08:22:38.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.298+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb000c1b0 con 0x7f4cb0004850 2023-06-22T08:22:38.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~136s 2023-06-22T08:22:38.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.300+0000 7f4cc9f9b700 1 --2- 172.21.15.23:0/1079123044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cb0003a00 0x7f4cb0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:38.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.300+0000 7f4cc9f9b700 1 -- 172.21.15.23:0/1079123044 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cb0004850 msgr2=0x7f4cb0004cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-22T08:22:38.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.300+0000 7f4cc9f9b700 1 --2- 172.21.15.23:0/1079123044 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cb0004850 0x7f4cb0004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.300+0000 7f4cc9f9b700 1 -- 172.21.15.23:0/1079123044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb0005250 msgr2=0x7f4cb000ba10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-22T08:22:38.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.300+0000 7f4cc9f9b700 1 --2- 172.21.15.23:0/1079123044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb0005250 0x7f4cb000ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.300+0000 7f4cc9f9b700 1 -- 172.21.15.23:0/1079123044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc40c4520 con 0x7f4cb0003a00 2023-06-22T08:22:38.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.301+0000 7f4cc9f9b700 1 --2- 172.21.15.23:0/1079123044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cb0003a00 0x7f4cb0003e10 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400aa30 tx=0x7f4cb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6325cde5070aa server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:38.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.302+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4004230 con 0x7f4cb0003a00 2023-06-22T08:22:38.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.302+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4004390 con 0x7f4cb0003a00 2023-06-22T08:22:38.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.302+0000 7f4cc8f99700 1 -- 172.21.15.23:0/1079123044 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400ae60 con 0x7f4cb0003a00 2023-06-22T08:22:38.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.303+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/1079123044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cb0003a00 msgr2=0x7f4cb0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:38.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.303+0000 7f4ccc1ff700 1 --2- 172.21.15.23:0/1079123044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cb0003a00 0x7f4cb0003e10 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400aa30 tx=0x7f4cb0005880 comp rx=0 tx=0).stop 2023-06-22T08:22:38.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.303+0000 7f4cca79c700 1 -- 172.21.15.23:0/1079123044 reap_dead start 2023-06-22T08:22:38.312 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo -n b 2023-06-22T08:22:38.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.307+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/1079123044 shutdown_connections 2023-06-22T08:22:38.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.307+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/1079123044 >> 172.21.15.23:0/1079123044 conn(0x7f4cc40ca3f0 msgr2=0x7f4cc40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:38.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.324+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/1079123044 shutdown_connections 2023-06-22T08:22:38.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.325+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/1079123044 wait complete. 2023-06-22T08:22:38.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.327+0000 7f4ccc1ff700 1 Processor -- start 2023-06-22T08:22:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.329+0000 7f4ccc1ff700 1 -- start start 2023-06-22T08:22:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.330+0000 7f4ccc1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cb0003a00 0x7f4cc41538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.330+0000 7f4ccc1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cc4155a10 0x7f4cc4157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.330+0000 7f4ccc1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4151a10 0x7f4cc414baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.330+0000 7f4ccc1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc4152d50 con 0x7f4cc4151a10 2023-06-22T08:22:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.330+0000 7f4ccc1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc4153e10 con 0x7f4cc4155a10 2023-06-22T08:22:38.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.330+0000 7f4ccc1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc41ed1c0 con 0x7f4cb0003a00 2023-06-22T08:22:38.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.331+0000 7f4cc979a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cc4155a10 0x7f4cc4157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:38.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.331+0000 7f4cc9f9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cb0003a00 0x7f4cc41538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:38.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.331+0000 7f4cc9f9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cb0003a00 0x7f4cc41538d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:56126/0 (socket says 172.21.15.23:56126) 2023-06-22T08:22:38.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.331+0000 7f4cc9f9b700 1 -- 172.21.15.23:0/3375153684 learned_addr learned my addr 172.21.15.23:0/3375153684 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:38.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.332+0000 7f4cca79c700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4151a10 0x7f4cc414baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:38.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.333+0000 7f4cc9f9b700 1 -- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cc4155a10 msgr2=0x7f4cc4157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:38.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.333+0000 7f4cc9f9b700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cc4155a10 0x7f4cc4157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.333+0000 7f4cc9f9b700 1 -- 172.21.15.23:0/3375153684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4151a10 msgr2=0x7f4cc414baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:38.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.333+0000 7f4cc9f9b700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4151a10 0x7f4cc414baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.333+0000 7f4cc9f9b700 1 -- 172.21.15.23:0/3375153684 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc4154670 con 0x7f4cb0003a00 2023-06-22T08:22:38.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.354+0000 7f4cc9f9b700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cb0003a00 0x7f4cc41538d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4008010 tx=0x7f4cb40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.370+0000 7f4cba7fc700 1 -- 172.21.15.23:0/3375153684 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb401f070 con 0x7f4cb0003a00 2023-06-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.370+0000 7f4cba7fc700 1 -- 172.21.15.23:0/3375153684 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4004580 con 0x7f4cb0003a00 2023-06-22T08:22:38.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.370+0000 7f4cba7fc700 1 -- 172.21.15.23:0/3375153684 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400c8e0 con 0x7f4cb0003a00 2023-06-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.370+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/3375153684 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc41dea90 con 0x7f4cb0003a00 2023-06-22T08:22:38.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.370+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/3375153684 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc41de7b0 con 0x7f4cb0003a00 2023-06-22T08:22:38.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.374+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/3375153684 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cac0052f0 con 0x7f4cb0003a00 2023-06-22T08:22:38.387 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.383+0000 7f4cba7fc700 1 -- 172.21.15.23:0/3375153684 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4cb400ca40 con 0x7f4cb0003a00 2023-06-22T08:22:38.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.384+0000 7f4cba7fc700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ca806adb0 0x7f4ca806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:38.388 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.384+0000 7f4cba7fc700 1 -- 172.21.15.23:0/3375153684 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7957+0+0 (secure 0 0 0) 0x7f4cb408ab00 con 0x7f4cb0003a00 2023-06-22T08:22:38.400 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir 2 2023-06-22T08:22:38.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.398+0000 7f4cc979a700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ca806adb0 0x7f4ca806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:38.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.402+0000 7f4cba7fc700 1 -- 172.21.15.23:0/3375153684 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4cb40555d0 con 0x7f4cb0003a00 2023-06-22T08:22:38.426 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.416+0000 7f4cc979a700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ca806adb0 0x7f4ca806d260 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc001b20 tx=0x7f4cbc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:38.429 INFO:tasks.workunit.client.0.smithi023.stderr:+ mv 1/a 2 2023-06-22T08:22:38.496 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir 2/.snap/s2 2023-06-22T08:22:38.867 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo -n c 2023-06-22T08:22:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.866+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/3375153684 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cac005160 con 0x7f4cb0003a00 2023-06-22T08:22:38.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.876+0000 7f4cba7fc700 1 -- 172.21.15.23:0/3375153684 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+4738 (secure 0 0 0) 0x7f4cb4028020 con 0x7f4cb0003a00 2023-06-22T08:22:38.881 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:38.881 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:30.642319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.908+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ca806adb0 msgr2=0x7f4ca806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.908+0000 7f4ccc1ff700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ca806adb0 0x7f4ca806d260 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc001b20 tx=0x7f4cbc009040 comp rx=0 tx=0).stop 2023-06-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.908+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cb0003a00 msgr2=0x7f4cc41538d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.908+0000 7f4ccc1ff700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cb0003a00 0x7f4cc41538d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4008010 tx=0x7f4cb40047c0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.926+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/3375153684 shutdown_connections 2023-06-22T08:22:38.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.926+0000 7f4ccc1ff700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cc4151a10 0x7f4cc414baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.926+0000 7f4ccc1ff700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4cb0003a00 0x7f4cc41538d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.926+0000 7f4ccc1ff700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4ca806adb0 0x7f4ca806d260 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.926+0000 7f4ccc1ff700 1 --2- 172.21.15.23:0/3375153684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cc4155a10 0x7f4cc4157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:38.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.926+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/3375153684 >> 172.21.15.23:0/3375153684 conn(0x7f4cc40ca3f0 msgr2=0x7f4cc40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:38.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.926+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/3375153684 shutdown_connections 2023-06-22T08:22:38.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:38.926+0000 7f4ccc1ff700 1 -- 172.21.15.23:0/3375153684 wait complete. 2023-06-22T08:22:38.942 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 176 2023-06-22T08:22:38.968 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-06-22T08:22:39.063 INFO:tasks.workunit.client.0.smithi023.stderr:+ exec 2023-06-22T08:22:39.063 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep '^a$' 1/.snap/s1/a/file1 2023-06-22T08:22:39.075 INFO:tasks.workunit.client.0.smithi023.stdout:a 2023-06-22T08:22:39.077 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep '^ab$' 2/.snap/s2/a/file1 2023-06-22T08:22:39.556 INFO:tasks.workunit.client.0.smithi023.stdout:ab 2023-06-22T08:22:39.557 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep '^abc$' 2/a/file1 2023-06-22T08:22:39.559 INFO:tasks.workunit.client.0.smithi023.stdout:abc 2023-06-22T08:22:39.559 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir 1/.snap/s1 2023-06-22T08:22:39.572 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir 2/.snap/s2 2023-06-22T08:22:39.581 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -rf 1 2 2023-06-22T08:22:39.604 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:22:39.605 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:22:39.605 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:22:39.605 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:22:39.702 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-snap-rename.sh... 2023-06-22T08:22:39.704 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-snap-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap-rename.sh 2023-06-22T08:22:39.762 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:22:39.762 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir -p d1/d2 2023-06-22T08:22:39.766 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir -p d1/d3 2023-06-22T08:22:39.769 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1/.snap/foo 2023-06-22T08:22:39.785 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1/d2/.snap/foo 2023-06-22T08:22:39.794 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1/d3/.snap/foo 2023-06-22T08:22:39.803 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir d1/d3/.snap/bar 2023-06-22T08:22:39.813 INFO:tasks.workunit.client.0.smithi023.stderr:+ mv d1/d2/.snap/foo d1/d2/.snap/bar 2023-06-22T08:22:39.822 INFO:tasks.workunit.client.0.smithi023.stderr:+ expect_failure mv d1/d2/.snap/bar d1/d2/.snap/_bar 2023-06-22T08:22:39.822 INFO:tasks.workunit.client.0.smithi023.stderr:+ mv d1/d2/.snap/bar d1/d2/.snap/_bar 2023-06-22T08:22:39.827 INFO:tasks.workunit.client.0.smithi023.stderr:mv: cannot move 'd1/d2/.snap/bar' to a subdirectory of itself, 'd1/d2/.snap/_bar' 2023-06-22T08:22:39.827 INFO:tasks.workunit.client.0.smithi023.stderr:+ return 0 2023-06-22T08:22:39.828 INFO:tasks.workunit.client.0.smithi023.stderr:+ expect_failure mv d1/d2/.snap/_foo_1099511651583 d1/d2/.snap/foo 2023-06-22T08:22:39.828 INFO:tasks.workunit.client.0.smithi023.stderr:+ mv d1/d2/.snap/_foo_1099511651583 d1/d2/.snap/foo 2023-06-22T08:22:39.833 INFO:tasks.workunit.client.0.smithi023.stderr:mv: cannot move 'd1/d2/.snap/_foo_1099511651583' to a subdirectory of itself, 'd1/d2/.snap/foo' 2023-06-22T08:22:39.834 INFO:tasks.workunit.client.0.smithi023.stderr:+ return 0 2023-06-22T08:22:39.834 INFO:tasks.workunit.client.0.smithi023.stderr:+ expect_failure mv d1/d2/.snap/_foo_1099511651583 d1/d2/.snap/_foo_1 2023-06-22T08:22:39.834 INFO:tasks.workunit.client.0.smithi023.stderr:+ mv d1/d2/.snap/_foo_1099511651583 d1/d2/.snap/_foo_1 2023-06-22T08:22:39.838 INFO:tasks.workunit.client.0.smithi023.stderr:mv: cannot move 'd1/d2/.snap/_foo_1099511651583' to a subdirectory of itself, 'd1/d2/.snap/_foo_1' 2023-06-22T08:22:39.838 INFO:tasks.workunit.client.0.smithi023.stderr:+ return 0 2023-06-22T08:22:39.838 INFO:tasks.workunit.client.0.smithi023.stderr:+ expect_failure mv d1/d2/.snap/bar d1/.snap/ 2023-06-22T08:22:39.839 INFO:tasks.workunit.client.0.smithi023.stderr:+ mv d1/d2/.snap/bar d1/.snap/ 2023-06-22T08:22:39.842 INFO:tasks.workunit.client.0.smithi023.stderr:mv: cannot move 'd1/d2/.snap/bar' to 'd1/.snap/bar': Read-only file system 2023-06-22T08:22:39.842 INFO:tasks.workunit.client.0.smithi023.stderr:+ return 0 2023-06-22T08:22:39.843 INFO:tasks.workunit.client.0.smithi023.stderr:+ expect_failure python3 -c 'import os; os.rename('\''d1/d3/.snap/foo'\'', '\''d1/d3/.snap/bar'\'')' 2023-06-22T08:22:39.843 INFO:tasks.workunit.client.0.smithi023.stderr:+ python3 -c 'import os; os.rename('\''d1/d3/.snap/foo'\'', '\''d1/d3/.snap/bar'\'')' 2023-06-22T08:22:39.868 INFO:tasks.workunit.client.0.smithi023.stderr:Traceback (most recent call last): 2023-06-22T08:22:39.868 INFO:tasks.workunit.client.0.smithi023.stderr: File "", line 1, in 2023-06-22T08:22:39.868 INFO:tasks.workunit.client.0.smithi023.stderr:FileExistsError: [Errno 17] File exists: 'd1/d3/.snap/foo' -> 'd1/d3/.snap/bar' 2023-06-22T08:22:39.870 INFO:tasks.workunit.client.0.smithi023.stderr:+ return 0 2023-06-22T08:22:39.871 INFO:tasks.workunit.client.0.smithi023.stderr:+ expect_failure python3 -c 'import os; os.rename('\''d1/.snap/foo'\'', '\''d1/foo'\'')' 2023-06-22T08:22:39.871 INFO:tasks.workunit.client.0.smithi023.stderr:+ python3 -c 'import os; os.rename('\''d1/.snap/foo'\'', '\''d1/foo'\'')' 2023-06-22T08:22:39.894 INFO:tasks.workunit.client.0.smithi023.stderr:Traceback (most recent call last): 2023-06-22T08:22:39.895 INFO:tasks.workunit.client.0.smithi023.stderr: File "", line 1, in 2023-06-22T08:22:39.895 INFO:tasks.workunit.client.0.smithi023.stderr:OSError: [Errno 18] Invalid cross-device link: 'd1/.snap/foo' -> 'd1/foo' 2023-06-22T08:22:39.897 INFO:tasks.workunit.client.0.smithi023.stderr:+ return 0 2023-06-22T08:22:39.897 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1/.snap/foo 2023-06-22T08:22:39.906 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1/d2/.snap/bar 2023-06-22T08:22:39.917 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1/d3/.snap/foo 2023-06-22T08:22:39.924 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir d1/d3/.snap/bar 2023-06-22T08:22:39.936 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -rf d1 2023-06-22T08:22:39.961 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:22:39.962 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:22:39.963 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:22:39.963 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:22:40.047 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-snap-rm-cmp.sh... 2023-06-22T08:22:40.048 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-snap-rm-cmp.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-snap-rm-cmp.sh 2023-06-22T08:22:40.969 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:22:41.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.181+0000 7f9e49fff700 1 Processor -- start 2023-06-22T08:22:41.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.182+0000 7f9e49fff700 1 -- start start 2023-06-22T08:22:41.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.183+0000 7f9e49fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9e440c9fd0 0x7f9e440cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:41.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.184+0000 7f9e49fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e4415c370 con 0x7f9e440c40b0 2023-06-22T08:22:41.186 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.184+0000 7f9e49fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e4415bda0 con 0x7f9e44152ce0 2023-06-22T08:22:41.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.184+0000 7f9e49fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e441530b0 con 0x7f9e440c9fd0 2023-06-22T08:22:41.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.184+0000 7f9e43fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9e440c40b0 0x7f9e440c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55248/0 (socket says 172.21.15.23:55248) 2023-06-22T08:22:41.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.184+0000 7f9e43fff700 1 -- 172.21.15.23:0/2117988453 learned_addr learned my addr 172.21.15.23:0/2117988453 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:22:41.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.184+0000 7f9e42ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9e440c9fd0 0x7f9e440cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:41.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.184+0000 7f9e42ffd700 1 -- 172.21.15.23:0/2117988453 >> v1:172.21.15.67:6789/0 conn(0x7f9e44152ce0 legacy=0x7f9e4415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:41.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.184+0000 7f9e42ffd700 1 -- 172.21.15.23:0/2117988453 >> v1:172.21.15.23:6789/0 conn(0x7f9e440c40b0 legacy=0x7f9e440c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:41.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.184+0000 7f9e42ffd700 1 -- 172.21.15.23:0/2117988453 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e440c9460 con 0x7f9e440c9fd0 2023-06-22T08:22:41.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.184+0000 7f9e427fc700 1 -- 172.21.15.23:0/2117988453 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983534617 0 0) 0x7f9e4415bda0 con 0x7f9e44152ce0 2023-06-22T08:22:41.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.185+0000 7f9e42ffd700 1 --2- 172.21.15.23:0/2117988453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c9fd0 0x7f9e440cf1b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009840 tx=0x7f9e38009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52940f8122bcd997 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:41.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.185+0000 7f9e427fc700 1 -- 172.21.15.23:0/2117988453 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e38004230 con 0x7f9e440c9fd0 2023-06-22T08:22:41.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.185+0000 7f9e427fc700 1 -- 172.21.15.23:0/2117988453 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e38004390 con 0x7f9e440c9fd0 2023-06-22T08:22:41.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.185+0000 7f9e427fc700 1 -- 172.21.15.23:0/2117988453 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e380036a0 con 0x7f9e440c9fd0 2023-06-22T08:22:41.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.185+0000 7f9e49fff700 1 -- 172.21.15.23:0/2117988453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c9fd0 msgr2=0x7f9e440cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:41.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.185+0000 7f9e49fff700 1 --2- 172.21.15.23:0/2117988453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c9fd0 0x7f9e440cf1b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009840 tx=0x7f9e38009b50 comp rx=0 tx=0).stop 2023-06-22T08:22:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.185+0000 7f9e49fff700 1 -- 172.21.15.23:0/2117988453 shutdown_connections 2023-06-22T08:22:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.185+0000 7f9e49fff700 1 --2- 172.21.15.23:0/2117988453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c9fd0 0x7f9e440cf1b0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.185+0000 7f9e49fff700 1 -- 172.21.15.23:0/2117988453 >> 172.21.15.23:0/2117988453 conn(0x7f9e4406ae30 msgr2=0x7f9e4406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.186+0000 7f9e49fff700 1 -- 172.21.15.23:0/2117988453 shutdown_connections 2023-06-22T08:22:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.186+0000 7f9e49fff700 1 -- 172.21.15.23:0/2117988453 wait complete. 2023-06-22T08:22:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.187+0000 7f9e49fff700 1 Processor -- start 2023-06-22T08:22:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.187+0000 7f9e49fff700 1 -- start start 2023-06-22T08:22:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.188+0000 7f9e49fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c40b0 0x7f9e441f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.188+0000 7f9e49fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e440c9fd0 0x7f9e4414a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.188+0000 7f9e49fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e44152ce0 0x7f9e441ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.188+0000 7f9e49fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e440cd180 con 0x7f9e440c40b0 2023-06-22T08:22:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.188+0000 7f9e49fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e44154e60 con 0x7f9e440c9fd0 2023-06-22T08:22:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.188+0000 7f9e49fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e441edec0 con 0x7f9e44152ce0 2023-06-22T08:22:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e437fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c40b0 0x7f9e441f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e42ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e440c9fd0 0x7f9e4414a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e437fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c40b0 0x7f9e441f1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52670/0 (socket says 172.21.15.23:52670) 2023-06-22T08:22:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e437fe700 1 -- 172.21.15.23:0/2411122590 learned_addr learned my addr 172.21.15.23:0/2411122590 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:22:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e43fff700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e44152ce0 0x7f9e441ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e437fe700 1 -- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e44152ce0 msgr2=0x7f9e441ee5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e437fe700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e44152ce0 0x7f9e441ee5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e437fe700 1 -- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e440c9fd0 msgr2=0x7f9e4414a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e437fe700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e440c9fd0 0x7f9e4414a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e437fe700 1 -- 172.21.15.23:0/2411122590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e380094a0 con 0x7f9e440c40b0 2023-06-22T08:22:41.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.189+0000 7f9e437fe700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c40b0 0x7f9e441f1e40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c480 tx=0x7f9e3400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:41.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.190+0000 7f9e2bfff700 1 -- 172.21.15.23:0/2411122590 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e340054e0 con 0x7f9e440c40b0 2023-06-22T08:22:41.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.190+0000 7f9e2bfff700 1 -- 172.21.15.23:0/2411122590 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e3400ce50 con 0x7f9e440c40b0 2023-06-22T08:22:41.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.190+0000 7f9e2bfff700 1 -- 172.21.15.23:0/2411122590 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e340038c0 con 0x7f9e440c40b0 2023-06-22T08:22:41.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.190+0000 7f9e49fff700 1 -- 172.21.15.23:0/2411122590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e440cc690 con 0x7f9e440c40b0 2023-06-22T08:22:41.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.190+0000 7f9e49fff700 1 -- 172.21.15.23:0/2411122590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e440ccbc0 con 0x7f9e440c40b0 2023-06-22T08:22:41.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.191+0000 7f9e49fff700 1 -- 172.21.15.23:0/2411122590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4404ea20 con 0x7f9e440c40b0 2023-06-22T08:22:41.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.192+0000 7f9e2bfff700 1 -- 172.21.15.23:0/2411122590 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f9e34003a20 con 0x7f9e440c40b0 2023-06-22T08:22:41.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.192+0000 7f9e2bfff700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e3006ad10 0x7f9e3006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:22:41.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.193+0000 7f9e2bfff700 1 -- 172.21.15.23:0/2411122590 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7957+0+0 (secure 0 0 0) 0x7f9e34088ff0 con 0x7f9e440c40b0 2023-06-22T08:22:41.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.193+0000 7f9e42ffd700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e3006ad10 0x7f9e3006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:22:41.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.194+0000 7f9e42ffd700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e3006ad10 0x7f9e3006d1c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9e380090b0 tx=0x7f9e38008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:22:41.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.197+0000 7f9e2bfff700 1 -- 172.21.15.23:0/2411122590 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f9e340538a0 con 0x7f9e440c40b0 2023-06-22T08:22:41.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.453+0000 7f9e49fff700 1 -- 172.21.15.23:0/2411122590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e441560d0 con 0x7f9e440c40b0 2023-06-22T08:22:41.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.453+0000 7f9e2bfff700 1 -- 172.21.15.23:0/2411122590 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5495 (secure 0 0 0) 0x7f9e34056ec0 con 0x7f9e440c40b0 2023-06-22T08:22:41.457 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:22:41.457 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:22:30.642319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:22:41.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 -- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e3006ad10 msgr2=0x7f9e3006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:41.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e3006ad10 0x7f9e3006d1c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9e380090b0 tx=0x7f9e38008030 comp rx=0 tx=0).stop 2023-06-22T08:22:41.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 -- 172.21.15.23:0/2411122590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c40b0 msgr2=0x7f9e441f1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:22:41.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c40b0 0x7f9e441f1e40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c480 tx=0x7f9e3400c790 comp rx=0 tx=0).stop 2023-06-22T08:22:41.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 -- 172.21.15.23:0/2411122590 shutdown_connections 2023-06-22T08:22:41.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9e44152ce0 0x7f9e441ee5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:41.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e440c40b0 0x7f9e441f1e40 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:41.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f9e3006ad10 0x7f9e3006d1c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:41.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 --2- 172.21.15.23:0/2411122590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e440c9fd0 0x7f9e4414a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:22:41.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 -- 172.21.15.23:0/2411122590 >> 172.21.15.23:0/2411122590 conn(0x7f9e4406ae30 msgr2=0x7f9e4415b730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:22:41.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 -- 172.21.15.23:0/2411122590 shutdown_connections 2023-06-22T08:22:41.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:22:41.456+0000 7f9e49fff700 1 -- 172.21.15.23:0/2411122590 wait complete. 2023-06-22T08:22:41.461 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 177 2023-06-22T08:22:41.476 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-06-22T08:22:41.477 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 37 secs before thrashing 2023-06-22T08:22:43.087 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:22:43.087 INFO:tasks.workunit.client.0.smithi023.stderr:+ file=linux-2.6.33.tar.bz2 2023-06-22T08:22:43.087 INFO:tasks.workunit.client.0.smithi023.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-06-22T08:22:45.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~142s 2023-06-22T08:22:46.158 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:22:46.164 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:22:49.724 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum linux-2.6.33.tar.bz2 2023-06-22T08:22:49.724 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:22:49.851 INFO:tasks.workunit.client.0.smithi023.stderr:+ real=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:22:49.851 INFO:tasks.workunit.client.0.smithi023.stderr:++ seq 1 20 2023-06-22T08:22:49.853 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:22:49.853 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 1 2023-06-22T08:22:49.853 INFO:tasks.workunit.client.0.smithi023.stdout:1 2023-06-22T08:22:49.854 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:22:49.959 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:22:49.975 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:22:49.980 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:22:50.394 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:22:50.395 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:22:50.501 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:22:50.501 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:22:50.501 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:22:50.530 INFO:tasks.workunit.client.0.smithi023.stdout:2 2023-06-22T08:22:50.531 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:22:50.532 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 2 2023-06-22T08:22:50.535 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:22:50.627 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:22:50.637 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:22:50.640 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:22:51.073 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:22:51.073 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:22:51.184 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:22:51.185 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:22:51.185 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:22:51.241 INFO:tasks.workunit.client.0.smithi023.stdout:3 2023-06-22T08:22:51.242 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:22:51.242 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 3 2023-06-22T08:22:51.242 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:22:51.338 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:22:51.348 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:22:51.352 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:22:51.766 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:22:51.766 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:22:51.873 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:22:51.874 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:22:51.874 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:22:51.900 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:22:51.900 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 4 2023-06-22T08:22:51.901 INFO:tasks.workunit.client.0.smithi023.stdout:4 2023-06-22T08:22:51.901 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:22:51.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~149s 2023-06-22T08:22:51.997 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:22:52.008 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:22:52.011 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:22:52.487 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:22:52.488 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:22:52.594 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:22:52.594 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:22:52.594 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:22:52.618 INFO:tasks.workunit.client.0.smithi023.stdout:5 2023-06-22T08:22:52.619 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:22:52.619 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 5 2023-06-22T08:22:52.619 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:22:52.713 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:22:52.723 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:22:52.727 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:22:53.149 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:22:53.149 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:22:53.268 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:22:53.268 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:22:53.268 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:22:53.291 INFO:tasks.workunit.client.0.smithi023.stdout:6 2023-06-22T08:22:53.292 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:22:53.292 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 6 2023-06-22T08:22:53.292 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:22:53.386 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:22:53.396 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:22:53.400 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:22:53.807 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:22:53.808 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:22:53.914 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:22:53.914 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:22:53.914 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:22:53.942 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:22:53.942 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 7 2023-06-22T08:22:53.942 INFO:tasks.workunit.client.0.smithi023.stdout:7 2023-06-22T08:22:53.943 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:22:54.052 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:22:54.119 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:22:54.133 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:22:55.352 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:22:55.353 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:22:55.472 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:22:55.472 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:22:55.472 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:22:55.574 INFO:tasks.workunit.client.0.smithi023.stdout:8 2023-06-22T08:22:55.575 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:22:55.575 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 8 2023-06-22T08:22:55.575 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:22:55.827 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:22:55.915 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:22:55.926 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:22:57.046 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:22:57.046 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:22:57.165 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:22:57.165 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:22:57.166 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:22:57.255 INFO:tasks.workunit.client.0.smithi023.stdout:9 2023-06-22T08:22:57.255 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:22:57.256 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 9 2023-06-22T08:22:57.256 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:22:57.530 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:22:57.694 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:22:57.703 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:22:58.741 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~156s 2023-06-22T08:22:58.900 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:22:58.908 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:22:59.054 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:22:59.055 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:22:59.055 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:22:59.112 INFO:tasks.workunit.client.0.smithi023.stdout:10 2023-06-22T08:22:59.112 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:22:59.113 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 10 2023-06-22T08:22:59.113 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:22:59.331 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:22:59.423 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:22:59.438 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:00.809 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:00.811 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:00.934 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:00.934 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:00.934 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:00.999 INFO:tasks.workunit.client.0.smithi023.stdout:11 2023-06-22T08:23:01.000 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:23:01.000 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 11 2023-06-22T08:23:01.000 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:23:01.337 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:23:01.480 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:23:01.493 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:03.002 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:03.003 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:03.119 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:03.120 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:03.120 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:03.158 INFO:tasks.workunit.client.0.smithi023.stdout:12 2023-06-22T08:23:03.159 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:23:03.159 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 12 2023-06-22T08:23:03.159 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:23:03.344 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:23:03.356 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:23:03.361 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:03.901 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:03.902 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:04.008 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:04.008 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:04.009 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:04.036 INFO:tasks.workunit.client.0.smithi023.stdout:13 2023-06-22T08:23:04.036 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:23:04.037 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 13 2023-06-22T08:23:04.037 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:23:04.153 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:23:04.168 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:23:04.173 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:05.160 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:05.161 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:05.287 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:05.287 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:05.287 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:05.409 INFO:tasks.workunit.client.0.smithi023.stdout:14 2023-06-22T08:23:05.409 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:23:05.409 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 14 2023-06-22T08:23:05.410 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:23:05.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~163s 2023-06-22T08:23:05.581 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:23:05.606 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:23:05.611 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:06.468 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:06.469 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:06.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:06.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:06.595 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:06.661 INFO:tasks.workunit.client.0.smithi023.stdout:15 2023-06-22T08:23:06.661 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:23:06.661 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 15 2023-06-22T08:23:06.662 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:23:06.859 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:23:07.057 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:23:07.064 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:07.967 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:07.967 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:08.098 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:08.098 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:08.099 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:08.235 INFO:tasks.workunit.client.0.smithi023.stdout:16 2023-06-22T08:23:08.236 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:23:08.236 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 16 2023-06-22T08:23:08.236 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:23:08.511 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:23:08.593 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:23:08.605 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:09.363 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:09.363 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:09.483 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:09.483 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:09.483 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:09.519 INFO:tasks.workunit.client.0.smithi023.stdout:17 2023-06-22T08:23:09.520 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:23:09.520 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 17 2023-06-22T08:23:09.520 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:23:09.667 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:23:09.680 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:23:09.687 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:10.220 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:10.221 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:10.329 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:10.329 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:10.329 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:10.359 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:23:10.359 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 18 2023-06-22T08:23:10.360 INFO:tasks.workunit.client.0.smithi023.stdout:18 2023-06-22T08:23:10.360 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:23:10.455 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:23:10.466 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:23:10.469 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:11.018 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:11.018 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:11.138 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:11.138 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:11.139 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:11.258 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:23:11.259 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 19 2023-06-22T08:23:11.259 INFO:tasks.workunit.client.0.smithi023.stdout:19 2023-06-22T08:23:11.259 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:23:11.434 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:23:11.587 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:23:11.604 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:12.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~170s 2023-06-22T08:23:12.747 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:12.752 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:12.861 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:12.861 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:12.862 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:13.026 INFO:tasks.workunit.client.0.smithi023.stdout:20 2023-06-22T08:23:13.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ for f in `seq 1 20` 2023-06-22T08:23:13.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 20 2023-06-22T08:23:13.027 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp linux-2.6.33.tar.bz2 a 2023-06-22T08:23:13.235 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s 2023-06-22T08:23:13.405 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm a 2023-06-22T08:23:13.424 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp .snap/s/a /tmp/a 2023-06-22T08:23:14.268 INFO:tasks.workunit.client.0.smithi023.stderr:++ md5sum /tmp/a 2023-06-22T08:23:14.270 INFO:tasks.workunit.client.0.smithi023.stderr:++ awk '{print $1}' 2023-06-22T08:23:14.391 INFO:tasks.workunit.client.0.smithi023.stderr:+ cur=c3883760b18d50e8d78819c54d579b00 2023-06-22T08:23:14.392 INFO:tasks.workunit.client.0.smithi023.stderr:+ '[' c3883760b18d50e8d78819c54d579b00 '!=' c3883760b18d50e8d78819c54d579b00 ']' 2023-06-22T08:23:14.392 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s 2023-06-22T08:23:14.448 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm linux-2.6.33.tar.bz2 2023-06-22T08:23:14.456 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:23:14.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:23:14.648 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-upchildrealms.sh... 2023-06-22T08:23:14.648 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-upchildrealms.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-upchildrealms.sh 2023-06-22T08:23:14.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:23:14.781 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:23:14.798 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/b 2023-06-22T08:23:14.809 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/a1 2023-06-22T08:23:14.845 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/b/.snap/b1 2023-06-22T08:23:14.856 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo asdf 2023-06-22T08:23:14.857 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/a2 2023-06-22T08:23:14.876 INFO:tasks.workunit.client.0.smithi023.stderr:+ ln a/b/foo a/b/bar 2023-06-22T08:23:14.881 INFO:tasks.workunit.client.0.smithi023.stdout:this should not hang... 2023-06-22T08:23:14.882 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'this should not hang...' 2023-06-22T08:23:14.946 INFO:tasks.workunit.client.0.smithi023.stderr:+ cat a/b/.snap/_a2_1099511651608/foo 2023-06-22T08:23:14.952 INFO:tasks.workunit.client.0.smithi023.stdout:asdf 2023-06-22T08:23:14.953 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'good, it did not hang.' 2023-06-22T08:23:14.953 INFO:tasks.workunit.client.0.smithi023.stdout:good, it did not hang. 2023-06-22T08:23:14.954 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/b/.snap/b1 2023-06-22T08:23:14.962 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/a1 2023-06-22T08:23:14.971 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/a2 2023-06-22T08:23:14.985 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -r a 2023-06-22T08:23:15.010 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:23:15.011 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:23:15.011 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:23:15.011 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:23:15.103 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-xattrwb.sh... 2023-06-22T08:23:15.103 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/snaptest-xattrwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-xattrwb.sh 2023-06-22T08:23:16.317 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:23:16.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:23:17.163 INFO:tasks.workunit.client.0.smithi023.stderr:+ set -e 2023-06-22T08:23:17.164 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'testing simple xattr wb' 2023-06-22T08:23:17.164 INFO:tasks.workunit.client.0.smithi023.stdout:testing simple xattr wb 2023-06-22T08:23:17.165 INFO:tasks.workunit.client.0.smithi023.stderr:+ touch x 2023-06-22T08:23:17.172 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n user.foo x 2023-06-22T08:23:17.175 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/s1 2023-06-22T08:23:17.372 INFO:tasks.workunit.client.0.smithi023.stderr:+ getfattr -n user.foo .snap/s1/x 2023-06-22T08:23:17.380 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep user.foo 2023-06-22T08:23:17.425 INFO:tasks.workunit.client.0.smithi023.stdout:user.foo="" 2023-06-22T08:23:17.425 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm x 2023-06-22T08:23:17.464 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/s1 2023-06-22T08:23:17.658 INFO:tasks.workunit.client.0.smithi023.stdout:testing wb with pre-wb server cow 2023-06-22T08:23:17.659 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo 'testing wb with pre-wb server cow' 2023-06-22T08:23:17.659 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a 2023-06-22T08:23:17.667 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/b 2023-06-22T08:23:17.713 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/b/c 2023-06-22T08:23:17.735 INFO:tasks.workunit.client.0.smithi023.stderr:+ setfattr -n user.foo a/b 2023-06-22T08:23:17.753 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/.snap/s 2023-06-22T08:23:18.062 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir a/b/cc 2023-06-22T08:23:18.071 INFO:tasks.workunit.client.0.smithi023.stderr:+ getfattr -n user.foo a/b 2023-06-22T08:23:18.073 INFO:tasks.workunit.client.0.smithi023.stdout:# file: a/b 2023-06-22T08:23:18.073 INFO:tasks.workunit.client.0.smithi023.stdout:user.foo="" 2023-06-22T08:23:18.073 INFO:tasks.workunit.client.0.smithi023.stdout: 2023-06-22T08:23:18.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ getfattr -n user.foo a/.snap/s/b 2023-06-22T08:23:18.079 INFO:tasks.workunit.client.0.smithi023.stderr:+ grep user.foo 2023-06-22T08:23:18.197 INFO:tasks.workunit.client.0.smithi023.stdout:user.foo="" 2023-06-22T08:23:18.198 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir a/.snap/s 2023-06-22T08:23:18.292 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -r a 2023-06-22T08:23:18.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:18.622 INFO:tasks.workunit.client.0.smithi023.stdout:OK 2023-06-22T08:23:18.622 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo OK 2023-06-22T08:23:18.623 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-06-22T08:23:18.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-06-22T08:23:18.893 INFO:tasks.workunit:Running workunit fs/snaps/untar_snap_rm.sh... 2023-06-22T08:23:18.894 DEBUG:teuthology.orchestra.run.smithi023:workunit test fs/snaps/untar_snap_rm.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh 2023-06-22T08:23:18.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.907+0000 7f8e13fff700 1 Processor -- start 2023-06-22T08:23:18.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.907+0000 7f8e13fff700 1 -- start start 2023-06-22T08:23:18.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.909+0000 7f8e13fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c06e310 0x7f8e0c0c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:18.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.909+0000 7f8e13fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c0c3dd0 con 0x7f8e0c069eb0 2023-06-22T08:23:18.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.909+0000 7f8e13fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c0c4800 con 0x7f8e0c14eaf0 2023-06-22T08:23:18.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.909+0000 7f8e13fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c158d10 con 0x7f8e0c06e310 2023-06-22T08:23:18.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.910+0000 7f8e1159a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c06e310 0x7f8e0c0c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:18.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.910+0000 7f8e1159a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c06e310 0x7f8e0c0c5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:56480/0 (socket says 172.21.15.23:56480) 2023-06-22T08:23:18.914 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.910+0000 7f8e1159a700 1 -- 172.21.15.23:0/3139664530 learned_addr learned my addr 172.21.15.23:0/3139664530 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:18.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.912+0000 7f8e10d99700 1 -- 172.21.15.23:0/3139664530 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981826236 0 0) 0x7f8e0c0c4800 con 0x7f8e0c14eaf0 2023-06-22T08:23:18.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.912+0000 7f8e10d99700 1 -- 172.21.15.23:0/3139664530 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df80035b0 con 0x7f8e0c14eaf0 2023-06-22T08:23:18.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.913+0000 7f8e1159a700 1 -- 172.21.15.23:0/3139664530 >> v1:172.21.15.67:6789/0 conn(0x7f8e0c14eaf0 legacy=0x7f8e0c1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:18.915 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.913+0000 7f8e1159a700 1 -- 172.21.15.23:0/3139664530 >> v1:172.21.15.23:6789/0 conn(0x7f8e0c069eb0 legacy=0x7f8e0c15f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:18.916 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.913+0000 7f8e1159a700 1 -- 172.21.15.23:0/3139664530 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c0c4520 con 0x7f8e0c06e310 2023-06-22T08:23:18.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.921+0000 7f8e1159a700 1 --2- 172.21.15.23:0/3139664530 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c06e310 0x7f8e0c0c5f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc008920 tx=0x7f8dfc00d330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d021638dbf9bd707 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:18.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.924+0000 7f8e10d99700 1 -- 172.21.15.23:0/3139664530 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc010070 con 0x7f8e0c06e310 2023-06-22T08:23:18.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.924+0000 7f8e10d99700 1 -- 172.21.15.23:0/3139664530 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dfc00d990 con 0x7f8e0c06e310 2023-06-22T08:23:18.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.924+0000 7f8e10d99700 1 -- 172.21.15.23:0/3139664530 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc00dd20 con 0x7f8e0c06e310 2023-06-22T08:23:18.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.925+0000 7f8e13fff700 1 -- 172.21.15.23:0/3139664530 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c06e310 msgr2=0x7f8e0c0c5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:18.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.925+0000 7f8e13fff700 1 --2- 172.21.15.23:0/3139664530 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c06e310 0x7f8e0c0c5f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc008920 tx=0x7f8dfc00d330 comp rx=0 tx=0).stop 2023-06-22T08:23:18.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.925+0000 7f8e13fff700 1 -- 172.21.15.23:0/3139664530 shutdown_connections 2023-06-22T08:23:18.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.925+0000 7f8e13fff700 1 --2- 172.21.15.23:0/3139664530 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c06e310 0x7f8e0c0c5f20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:18.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.925+0000 7f8e13fff700 1 -- 172.21.15.23:0/3139664530 >> 172.21.15.23:0/3139664530 conn(0x7f8e0c0ca3f0 msgr2=0x7f8e0c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:18.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.926+0000 7f8e13fff700 1 -- 172.21.15.23:0/3139664530 shutdown_connections 2023-06-22T08:23:18.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.926+0000 7f8e13fff700 1 -- 172.21.15.23:0/3139664530 wait complete. 2023-06-22T08:23:18.931 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.928+0000 7f8e13fff700 1 Processor -- start 2023-06-22T08:23:18.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.929+0000 7f8e13fff700 1 -- start start 2023-06-22T08:23:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.930+0000 7f8e13fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c069eb0 0x7f8e0c150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.930+0000 7f8e13fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c06e310 0x7f8e0c1f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.930+0000 7f8e13fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e0c14eaf0 0x7f8e0c0c0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.930+0000 7f8e13fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c0b75d0 con 0x7f8e0c14eaf0 2023-06-22T08:23:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.930+0000 7f8e13fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c0c15c0 con 0x7f8e0c06e310 2023-06-22T08:23:18.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.930+0000 7f8e13fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c1f62e0 con 0x7f8e0c069eb0 2023-06-22T08:23:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.931+0000 7f8e1159a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c06e310 0x7f8e0c1f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.931+0000 7f8e11d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c069eb0 0x7f8e0c150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.931+0000 7f8e11d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c069eb0 0x7f8e0c150a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:56482/0 (socket says 172.21.15.23:56482) 2023-06-22T08:23:18.935 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.931+0000 7f8e11d9b700 1 -- 172.21.15.23:0/1408469773 learned_addr learned my addr 172.21.15.23:0/1408469773 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:18.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.932+0000 7f8e11d9b700 1 -- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c06e310 msgr2=0x7f8e0c1f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:18.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.932+0000 7f8e11d9b700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c06e310 0x7f8e0c1f5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:18.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.932+0000 7f8e11d9b700 1 -- 172.21.15.23:0/1408469773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e0c14eaf0 msgr2=0x7f8e0c0c0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:18.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.932+0000 7f8e11d9b700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e0c14eaf0 0x7f8e0c0c0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:18.936 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.932+0000 7f8e11d9b700 1 -- 172.21.15.23:0/1408469773 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc0085d0 con 0x7f8e0c069eb0 2023-06-22T08:23:18.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.933+0000 7f8e11d9b700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c069eb0 0x7f8e0c150a20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8e0800c500 tx=0x7f8e0800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:18.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.933+0000 7f8e027fc700 1 -- 172.21.15.23:0/1408469773 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e080052d0 con 0x7f8e0c069eb0 2023-06-22T08:23:18.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.933+0000 7f8e027fc700 1 -- 172.21.15.23:0/1408469773 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e08005450 con 0x7f8e0c069eb0 2023-06-22T08:23:18.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.933+0000 7f8e027fc700 1 -- 172.21.15.23:0/1408469773 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e080038c0 con 0x7f8e0c069eb0 2023-06-22T08:23:18.937 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.933+0000 7f8e13fff700 1 -- 172.21.15.23:0/1408469773 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0c0c12e0 con 0x7f8e0c069eb0 2023-06-22T08:23:18.938 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.934+0000 7f8e13fff700 1 -- 172.21.15.23:0/1408469773 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0c1ded10 con 0x7f8e0c069eb0 2023-06-22T08:23:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.936+0000 7f8e13fff700 1 -- 172.21.15.23:0/1408469773 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e0c0c1140 con 0x7f8e0c069eb0 2023-06-22T08:23:18.940 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.938+0000 7f8e027fc700 1 -- 172.21.15.23:0/1408469773 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8e08003a20 con 0x7f8e0c069eb0 2023-06-22T08:23:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.939+0000 7f8e027fc700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8df006adb0 0x7f8df006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.939+0000 7f8e027fc700 1 -- 172.21.15.23:0/1408469773 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 7957+0+0 (secure 0 0 0) 0x7f8e080890e0 con 0x7f8e0c069eb0 2023-06-22T08:23:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.940+0000 7f8e1159a700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8df006adb0 0x7f8df006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:18.943 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.941+0000 7f8e1159a700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8df006adb0 0x7f8df006d260 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc0088f0 tx=0x7f8dfc012000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:18.944+0000 7f8e027fc700 1 -- 172.21.15.23:0/1408469773 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f8e0c0c1140 con 0x7f8e0c069eb0 2023-06-22T08:23:19.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~176s 2023-06-22T08:23:19.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.373+0000 7f8e13fff700 1 -- 172.21.15.23:0/1408469773 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e0c14fe60 con 0x7f8e0c069eb0 2023-06-22T08:23:19.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.375+0000 7f8e027fc700 1 -- 172.21.15.23:0/1408469773 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+5483 (secure 0 0 0) 0x7f8e0c14fe60 con 0x7f8e0c069eb0 2023-06-22T08:23:19.378 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:19.378 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:15.702250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:19.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.378+0000 7f8e13fff700 1 -- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8df006adb0 msgr2=0x7f8df006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:19.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.378+0000 7f8e13fff700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8df006adb0 0x7f8df006d260 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc0088f0 tx=0x7f8dfc012000 comp rx=0 tx=0).stop 2023-06-22T08:23:19.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.378+0000 7f8e13fff700 1 -- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c069eb0 msgr2=0x7f8e0c150a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:19.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.378+0000 7f8e13fff700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c069eb0 0x7f8e0c150a20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8e0800c500 tx=0x7f8e0800c8c0 comp rx=0 tx=0).stop 2023-06-22T08:23:19.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.379+0000 7f8e13fff700 1 -- 172.21.15.23:0/1408469773 shutdown_connections 2023-06-22T08:23:19.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.379+0000 7f8e13fff700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e0c14eaf0 0x7f8e0c0c0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:19.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.379+0000 7f8e13fff700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8e0c069eb0 0x7f8e0c150a20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:19.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.379+0000 7f8e13fff700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f8df006adb0 0x7f8df006d260 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:19.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.379+0000 7f8e13fff700 1 --2- 172.21.15.23:0/1408469773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c06e310 0x7f8e0c1f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:19.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.379+0000 7f8e13fff700 1 -- 172.21.15.23:0/1408469773 >> 172.21.15.23:0/1408469773 conn(0x7f8e0c0ca3f0 msgr2=0x7f8e0c1de400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:19.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.379+0000 7f8e13fff700 1 -- 172.21.15.23:0/1408469773 shutdown_connections 2023-06-22T08:23:19.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.379+0000 7f8e13fff700 1 -- 172.21.15.23:0/1408469773 wait complete. 2023-06-22T08:23:19.390 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 181 2023-06-22T08:23:19.414 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=3) 2023-06-22T08:23:19.415 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-06-22T08:23:19.415 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:23:19.435 INFO:tasks.ceph.mds.f:Stopped 2023-06-22T08:23:19.435 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:19.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.844+0000 7effc1bff700 1 Processor -- start 2023-06-22T08:23:19.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.844+0000 7effc1bff700 1 -- start start 2023-06-22T08:23:19.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.846+0000 7effc1bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7effbc15d9b0 0x7effbc06d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:19.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.846+0000 7effc1bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7effbc06a280 0x7effbc0c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:19.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.847+0000 7effc1bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effbc0c4d70 con 0x7effbc069eb0 2023-06-22T08:23:19.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.847+0000 7effc1bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effbc0703c0 con 0x7effbc06a280 2023-06-22T08:23:19.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.847+0000 7effc1bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0c5610 con 0x7effbc15d9b0 2023-06-22T08:23:19.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.848+0000 7effbb7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7effbc069eb0 0x7effbc06fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:36666/0 (socket says 172.21.15.23:36666) 2023-06-22T08:23:19.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.848+0000 7effbb7fe700 1 -- 172.21.15.23:0/2555794473 learned_addr learned my addr 172.21.15.23:0/2555794473 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:19.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.850+0000 7effbaffd700 1 --2- 172.21.15.23:0/2555794473 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7effbc15d9b0 0x7effbc06d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:19.853 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.850+0000 7effbbfff700 1 --2- 172.21.15.23:0/2555794473 >> v2:172.21.15.67:3300/0 conn(0x7effbc06a280 0x7effbc0c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:19.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.852+0000 7effbbfff700 1 -- 172.21.15.23:0/2555794473 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7effbc15d9b0 msgr2=0x7effbc06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:19.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.852+0000 7effbbfff700 1 --2- 172.21.15.23:0/2555794473 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7effbc15d9b0 0x7effbc06d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:19.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.852+0000 7effbbfff700 1 -- 172.21.15.23:0/2555794473 >> v1:172.21.15.23:6789/0 conn(0x7effbc069eb0 legacy=0x7effbc06fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:19.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.852+0000 7effbbfff700 1 -- 172.21.15.23:0/2555794473 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc0c5fe0 con 0x7effbc06a280 2023-06-22T08:23:19.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.863+0000 7effbbfff700 1 --2- 172.21.15.23:0/2555794473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effbc06a280 0x7effbc0c5aa0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7effac009840 tx=0x7effac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=444e5b35d7540aa0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:19.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.864+0000 7effba7fc700 1 -- 172.21.15.23:0/2555794473 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac011830 con 0x7effbc06a280 2023-06-22T08:23:19.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.864+0000 7effba7fc700 1 -- 172.21.15.23:0/2555794473 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac011990 con 0x7effbc06a280 2023-06-22T08:23:19.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.864+0000 7effba7fc700 1 -- 172.21.15.23:0/2555794473 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac011d20 con 0x7effbc06a280 2023-06-22T08:23:19.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.865+0000 7effc1bff700 1 -- 172.21.15.23:0/2555794473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effbc06a280 msgr2=0x7effbc0c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:19.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.865+0000 7effc1bff700 1 --2- 172.21.15.23:0/2555794473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effbc06a280 0x7effbc0c5aa0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7effac009840 tx=0x7effac009b50 comp rx=0 tx=0).stop 2023-06-22T08:23:19.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.865+0000 7effc1bff700 1 -- 172.21.15.23:0/2555794473 shutdown_connections 2023-06-22T08:23:19.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.865+0000 7effc1bff700 1 --2- 172.21.15.23:0/2555794473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effbc06a280 0x7effbc0c5aa0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:19.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.865+0000 7effc1bff700 1 --2- 172.21.15.23:0/2555794473 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7effbc15d9b0 0x7effbc06d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:19.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.865+0000 7effc1bff700 1 -- 172.21.15.23:0/2555794473 >> 172.21.15.23:0/2555794473 conn(0x7effbc0ca3f0 msgr2=0x7effbc0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:19.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.874+0000 7effc1bff700 1 -- 172.21.15.23:0/2555794473 shutdown_connections 2023-06-22T08:23:19.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.874+0000 7effc1bff700 1 -- 172.21.15.23:0/2555794473 wait complete. 2023-06-22T08:23:19.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.877+0000 7effc1bff700 1 Processor -- start 2023-06-22T08:23:19.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.878+0000 7effc1bff700 1 -- start start 2023-06-22T08:23:19.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.879+0000 7effc1bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7effbc069eb0 0x7effbc13d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:19.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.880+0000 7effc1bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effbc06a280 0x7effbc152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:19.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.880+0000 7effc1bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effbc15d9b0 0x7effbc153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:19.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.880+0000 7effc1bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc153c00 con 0x7effbc15d9b0 2023-06-22T08:23:19.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.880+0000 7effc1bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0c9af0 con 0x7effbc06a280 2023-06-22T08:23:19.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.880+0000 7effc1bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0c9dd0 con 0x7effbc069eb0 2023-06-22T08:23:19.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.880+0000 7effbaffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effbc06a280 0x7effbc152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:19.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.882+0000 7effbbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effbc15d9b0 0x7effbc153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:19.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.882+0000 7effbbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effbc15d9b0 0x7effbc153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36196/0 (socket says 172.21.15.23:36196) 2023-06-22T08:23:19.885 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.882+0000 7effbbfff700 1 -- 172.21.15.23:0/4131311018 learned_addr learned my addr 172.21.15.23:0/4131311018 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:19.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.883+0000 7effbbfff700 1 -- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7effbc069eb0 msgr2=0x7effbc13d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:19.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.883+0000 7effbbfff700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7effbc069eb0 0x7effbc13d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:19.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.883+0000 7effbbfff700 1 -- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effbc06a280 msgr2=0x7effbc152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:19.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.883+0000 7effbbfff700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effbc06a280 0x7effbc152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:19.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.883+0000 7effbbfff700 1 -- 172.21.15.23:0/4131311018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac0094a0 con 0x7effbc15d9b0 2023-06-22T08:23:19.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.884+0000 7effbbfff700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effbc15d9b0 0x7effbc153400 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7effac009b90 tx=0x7effac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.897+0000 7effa3fff700 1 -- 172.21.15.23:0/4131311018 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac0044d0 con 0x7effbc15d9b0 2023-06-22T08:23:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.897+0000 7effa3fff700 1 -- 172.21.15.23:0/4131311018 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac01f070 con 0x7effbc15d9b0 2023-06-22T08:23:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.897+0000 7effa3fff700 1 -- 172.21.15.23:0/4131311018 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac0038e0 con 0x7effbc15d9b0 2023-06-22T08:23:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.897+0000 7effc1bff700 1 -- 172.21.15.23:0/4131311018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effbc0ca0b0 con 0x7effbc15d9b0 2023-06-22T08:23:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.897+0000 7effc1bff700 1 -- 172.21.15.23:0/4131311018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effbc0c84b0 con 0x7effbc15d9b0 2023-06-22T08:23:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.899+0000 7effa1ffb700 1 -- 172.21.15.23:0/4131311018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff9c0052f0 con 0x7effbc15d9b0 2023-06-22T08:23:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.911+0000 7effa3fff700 1 -- 172.21.15.23:0/4131311018 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7effac003a40 con 0x7effbc15d9b0 2023-06-22T08:23:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.912+0000 7effa3fff700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effa806ad60 0x7effa806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:19.933 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.920+0000 7effa3fff700 1 -- 172.21.15.23:0/4131311018 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 8013+0+0 (secure 0 0 0) 0x7effac021070 con 0x7effbc15d9b0 2023-06-22T08:23:19.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.920+0000 7effa3fff700 1 -- 172.21.15.23:0/4131311018 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7effac0561a0 con 0x7effbc15d9b0 2023-06-22T08:23:19.934 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.922+0000 7effbb7fe700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effa806ad60 0x7effa806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:19.944 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:19.942+0000 7effbb7fe700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effa806ad60 0x7effa806d210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7effb0001670 tx=0x7effb0007ad0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:20.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.392+0000 7effa1ffb700 1 -- 172.21.15.23:0/4131311018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff9c005870 con 0x7effbc15d9b0 2023-06-22T08:23:20.406 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:20.406 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:15.702250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:20.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.399+0000 7effa3fff700 1 -- 172.21.15.23:0/4131311018 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+5483 (secure 0 0 0) 0x7effac015090 con 0x7effbc15d9b0 2023-06-22T08:23:20.417 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.415+0000 7effa1ffb700 1 -- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effa806ad60 msgr2=0x7effa806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:20.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.415+0000 7effa1ffb700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effa806ad60 0x7effa806d210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7effb0001670 tx=0x7effb0007ad0 comp rx=0 tx=0).stop 2023-06-22T08:23:20.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.415+0000 7effa1ffb700 1 -- 172.21.15.23:0/4131311018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effbc15d9b0 msgr2=0x7effbc153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:20.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.415+0000 7effa1ffb700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effbc15d9b0 0x7effbc153400 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7effac009b90 tx=0x7effac004750 comp rx=0 tx=0).stop 2023-06-22T08:23:20.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.415+0000 7effa1ffb700 1 -- 172.21.15.23:0/4131311018 shutdown_connections 2023-06-22T08:23:20.418 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.415+0000 7effa1ffb700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effbc15d9b0 0x7effbc153400 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:20.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.415+0000 7effa1ffb700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7effbc069eb0 0x7effbc13d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:20.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.415+0000 7effa1ffb700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7effa806ad60 0x7effa806d210 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:20.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.415+0000 7effa1ffb700 1 --2- 172.21.15.23:0/4131311018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effbc06a280 0x7effbc152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:20.419 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.415+0000 7effa1ffb700 1 -- 172.21.15.23:0/4131311018 >> 172.21.15.23:0/4131311018 conn(0x7effbc0ca3f0 msgr2=0x7effbc15c140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:20.421 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.419+0000 7effa1ffb700 1 -- 172.21.15.23:0/4131311018 shutdown_connections 2023-06-22T08:23:20.424 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:20.422+0000 7effa1ffb700 1 -- 172.21.15.23:0/4131311018 wait complete. 2023-06-22T08:23:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 181 2023-06-22T08:23:20.457 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:23:22.459 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:22.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.673+0000 7fc379bff700 1 Processor -- start 2023-06-22T08:23:22.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.673+0000 7fc379bff700 1 -- start start 2023-06-22T08:23:22.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.674+0000 7fc379bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc3740c3a40 0x7fc3740c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:22.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.674+0000 7fc379bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 0x7fc374159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:22.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.674+0000 7fc379bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc374156fc0 0x7fc37415a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:22.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.674+0000 7fc379bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc374157570 con 0x7fc3740c3a40 2023-06-22T08:23:22.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.674+0000 7fc379bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3740c4350 con 0x7fc374156fc0 2023-06-22T08:23:22.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.674+0000 7fc379bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740c6a60 con 0x7fc37406d050 2023-06-22T08:23:22.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc373fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc374156fc0 0x7fc37415a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:22.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc3737fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc3740c3a40 0x7fc3740c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:22.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc373fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc374156fc0 0x7fc37415a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:43812/0 (socket says 172.21.15.23:43812) 2023-06-22T08:23:22.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc372ffd700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 0x7fc374159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:22.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc373fff700 1 -- 172.21.15.23:0/3194241925 learned_addr learned my addr 172.21.15.23:0/3194241925 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:22.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc373fff700 1 -- 172.21.15.23:0/3194241925 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 msgr2=0x7fc374159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:22.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc373fff700 1 --2- 172.21.15.23:0/3194241925 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 0x7fc374159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc373fff700 1 -- 172.21.15.23:0/3194241925 >> v2:172.21.15.23:3300/0 conn(0x7fc3740c3a40 msgr2=0x7fc3740c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:22.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc373fff700 1 --2- 172.21.15.23:0/3194241925 >> v2:172.21.15.23:3300/0 conn(0x7fc3740c3a40 0x7fc3740c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc373fff700 1 -- 172.21.15.23:0/3194241925 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37414d690 con 0x7fc374156fc0 2023-06-22T08:23:22.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc3737fe700 1 --2- 172.21.15.23:0/3194241925 >> v2:172.21.15.23:3300/0 conn(0x7fc3740c3a40 0x7fc3740c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:23:22.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.675+0000 7fc373fff700 1 --2- 172.21.15.23:0/3194241925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc374156fc0 0x7fc37415a150 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc37406e5f0 tx=0x7fc36c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aaf3e8b23ab3cea server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:22.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc3727fc700 1 -- 172.21.15.23:0/3194241925 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c00e070 con 0x7fc374156fc0 2023-06-22T08:23:22.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc3727fc700 1 -- 172.21.15.23:0/3194241925 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c00b910 con 0x7fc374156fc0 2023-06-22T08:23:22.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc3727fc700 1 -- 172.21.15.23:0/3194241925 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c00bcc0 con 0x7fc374156fc0 2023-06-22T08:23:22.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc379bff700 1 -- 172.21.15.23:0/3194241925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc374156fc0 msgr2=0x7fc37415a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:22.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc379bff700 1 --2- 172.21.15.23:0/3194241925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc374156fc0 0x7fc37415a150 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc37406e5f0 tx=0x7fc36c00b330 comp rx=0 tx=0).stop 2023-06-22T08:23:22.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc379bff700 1 -- 172.21.15.23:0/3194241925 shutdown_connections 2023-06-22T08:23:22.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc379bff700 1 --2- 172.21.15.23:0/3194241925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc374156fc0 0x7fc37415a150 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc379bff700 1 --2- 172.21.15.23:0/3194241925 >> v2:172.21.15.23:3300/0 conn(0x7fc3740c3a40 0x7fc3740c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc379bff700 1 --2- 172.21.15.23:0/3194241925 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 0x7fc374159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc379bff700 1 -- 172.21.15.23:0/3194241925 >> 172.21.15.23:0/3194241925 conn(0x7fc3740ca3f0 msgr2=0x7fc3740ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:22.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc379bff700 1 -- 172.21.15.23:0/3194241925 shutdown_connections 2023-06-22T08:23:22.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.676+0000 7fc379bff700 1 -- 172.21.15.23:0/3194241925 wait complete. 2023-06-22T08:23:22.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.679+0000 7fc379bff700 1 Processor -- start 2023-06-22T08:23:22.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.679+0000 7fc379bff700 1 -- start start 2023-06-22T08:23:22.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.680+0000 7fc379bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 0x7fc3741502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:22.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.680+0000 7fc379bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3740c3a40 0x7fc374144bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:22.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.680+0000 7fc379bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc374156fc0 0x7fc374145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:22.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.680+0000 7fc379bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc374145640 con 0x7fc3740c3a40 2023-06-22T08:23:22.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.680+0000 7fc379bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc374155d30 con 0x7fc374156fc0 2023-06-22T08:23:22.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.680+0000 7fc379bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc374156260 con 0x7fc37406d050 2023-06-22T08:23:22.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.680+0000 7fc3737fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 0x7fc3741502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:22.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.680+0000 7fc372ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3740c3a40 0x7fc374144bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:22.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc3737fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 0x7fc3741502b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:56504/0 (socket says 172.21.15.23:56504) 2023-06-22T08:23:22.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc3737fe700 1 -- 172.21.15.23:0/1329874148 learned_addr learned my addr 172.21.15.23:0/1329874148 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:22.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc373fff700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc374156fc0 0x7fc374145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:22.686 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc372ffd700 1 -- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 msgr2=0x7fc3741502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:22.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc372ffd700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 0x7fc3741502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc372ffd700 1 -- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc374156fc0 msgr2=0x7fc374145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:22.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc372ffd700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc374156fc0 0x7fc374145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc372ffd700 1 -- 172.21.15.23:0/1329874148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3640094a0 con 0x7fc3740c3a40 2023-06-22T08:23:22.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc372ffd700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3740c3a40 0x7fc374144bc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e870 tx=0x7fc36800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:22.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc35bfff700 1 -- 172.21.15.23:0/1329874148 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc368013680 con 0x7fc3740c3a40 2023-06-22T08:23:22.687 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc35bfff700 1 -- 172.21.15.23:0/1329874148 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3680137e0 con 0x7fc3740c3a40 2023-06-22T08:23:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc35bfff700 1 -- 172.21.15.23:0/1329874148 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc368013b50 con 0x7fc3740c3a40 2023-06-22T08:23:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.681+0000 7fc379bff700 1 -- 172.21.15.23:0/1329874148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c009b30 con 0x7fc3740c3a40 2023-06-22T08:23:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.682+0000 7fc379bff700 1 -- 172.21.15.23:0/1329874148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3741f25c0 con 0x7fc3740c3a40 2023-06-22T08:23:22.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.683+0000 7fc379bff700 1 -- 172.21.15.23:0/1329874148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37404ea20 con 0x7fc3740c3a40 2023-06-22T08:23:22.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.684+0000 7fc35bfff700 1 -- 172.21.15.23:0/1329874148 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc368013cb0 con 0x7fc3740c3a40 2023-06-22T08:23:22.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.685+0000 7fc35bfff700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc36006ad10 0x7fc36006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:22.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.685+0000 7fc35bfff700 1 -- 172.21.15.23:0/1329874148 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 8013+0+0 (secure 0 0 0) 0x7fc368089940 con 0x7fc3740c3a40 2023-06-22T08:23:22.689 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.685+0000 7fc3737fe700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc36006ad10 0x7fc36006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:22.690 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.686+0000 7fc3737fe700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc36006ad10 0x7fc36006d1c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc364009e70 tx=0x7fc364009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:22.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.689+0000 7fc35bfff700 1 -- 172.21.15.23:0/1329874148 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc3680541c0 con 0x7fc3740c3a40 2023-06-22T08:23:22.947 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.944+0000 7fc379bff700 1 -- 172.21.15.23:0/1329874148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3740c5f30 con 0x7fc3740c3a40 2023-06-22T08:23:22.948 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.945+0000 7fc35bfff700 1 -- 172.21.15.23:0/1329874148 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5482 (secure 0 0 0) 0x7fc3680577e0 con 0x7fc3740c3a40 2023-06-22T08:23:22.948 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:22.948 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:20.921145+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 -- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc36006ad10 msgr2=0x7fc36006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc36006ad10 0x7fc36006d1c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc364009e70 tx=0x7fc364009210 comp rx=0 tx=0).stop 2023-06-22T08:23:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 -- 172.21.15.23:0/1329874148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3740c3a40 msgr2=0x7fc374144bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:22.950 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3740c3a40 0x7fc374144bc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e870 tx=0x7fc36800eb80 comp rx=0 tx=0).stop 2023-06-22T08:23:22.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 -- 172.21.15.23:0/1329874148 shutdown_connections 2023-06-22T08:23:22.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc36006ad10 0x7fc36006d1c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc374156fc0 0x7fc374145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc37406d050 0x7fc3741502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 --2- 172.21.15.23:0/1329874148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3740c3a40 0x7fc374144bc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:22.951 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 -- 172.21.15.23:0/1329874148 >> 172.21.15.23:0/1329874148 conn(0x7fc3740ca3f0 msgr2=0x7fc3740c6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:22.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 -- 172.21.15.23:0/1329874148 shutdown_connections 2023-06-22T08:23:22.952 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:22.947+0000 7fc379bff700 1 -- 172.21.15.23:0/1329874148 wait complete. 2023-06-22T08:23:22.952 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 182 2023-06-22T08:23:22.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:23:24.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:25.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.180+0000 7fcac77ff700 1 Processor -- start 2023-06-22T08:23:25.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.181+0000 7fcac77ff700 1 -- start start 2023-06-22T08:23:25.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.182+0000 7fcac77ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcac015a320 0x7fcac00c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:25.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.182+0000 7fcac77ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcac006d050 0x7fcac0160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:25.187 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.182+0000 7fcac77ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac00c6a60 con 0x7fcac00c3a40 2023-06-22T08:23:25.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.182+0000 7fcac77ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac0157570 con 0x7fcac006d050 2023-06-22T08:23:25.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.182+0000 7fcac77ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac00c4350 con 0x7fcac015a320 2023-06-22T08:23:25.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.182+0000 7fcac5d9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcac006d050 0x7fcac0160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:25.188 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.182+0000 7fcac559b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcac00c3a40 0x7fcac0159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:45060/0 (socket says 172.21.15.23:45060) 2023-06-22T08:23:25.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.182+0000 7fcac559b700 1 -- 172.21.15.23:0/2604997530 learned_addr learned my addr 172.21.15.23:0/2604997530 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:25.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.182+0000 7fcac4d9a700 1 --2- 172.21.15.23:0/2604997530 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcac015a320 0x7fcac00c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:25.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.183+0000 7fcab7fff700 1 -- 172.21.15.23:0/2604997530 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868911911 0 0) 0x7fcac00c6a60 con 0x7fcac00c3a40 2023-06-22T08:23:25.189 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.183+0000 7fcac5d9c700 1 -- 172.21.15.23:0/2604997530 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcac015a320 msgr2=0x7fcac00c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:25.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.183+0000 7fcac5d9c700 1 --2- 172.21.15.23:0/2604997530 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcac015a320 0x7fcac00c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:25.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.183+0000 7fcac5d9c700 1 -- 172.21.15.23:0/2604997530 >> v1:172.21.15.67:6789/0 conn(0x7fcac00c3a40 legacy=0x7fcac0159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:25.190 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.183+0000 7fcac5d9c700 1 -- 172.21.15.23:0/2604997530 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac014d690 con 0x7fcac006d050 2023-06-22T08:23:25.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.183+0000 7fcac5d9c700 1 --2- 172.21.15.23:0/2604997530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac006d050 0x7fcac0160ec0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009d90 tx=0x7fcab800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624c29e76376fed8 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:25.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.183+0000 7fcab7fff700 1 -- 172.21.15.23:0/2604997530 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab800e070 con 0x7fcac006d050 2023-06-22T08:23:25.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.183+0000 7fcab7fff700 1 -- 172.21.15.23:0/2604997530 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab800bb20 con 0x7fcac006d050 2023-06-22T08:23:25.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.183+0000 7fcab7fff700 1 -- 172.21.15.23:0/2604997530 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab8012430 con 0x7fcac006d050 2023-06-22T08:23:25.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.184+0000 7fcac77ff700 1 -- 172.21.15.23:0/2604997530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac006d050 msgr2=0x7fcac0160ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:25.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.184+0000 7fcac77ff700 1 --2- 172.21.15.23:0/2604997530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac006d050 0x7fcac0160ec0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fcab8009d90 tx=0x7fcab800b330 comp rx=0 tx=0).stop 2023-06-22T08:23:25.191 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.184+0000 7fcac77ff700 1 -- 172.21.15.23:0/2604997530 shutdown_connections 2023-06-22T08:23:25.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.184+0000 7fcac77ff700 1 --2- 172.21.15.23:0/2604997530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac006d050 0x7fcac0160ec0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:25.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.184+0000 7fcac77ff700 1 --2- 172.21.15.23:0/2604997530 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcac015a320 0x7fcac00c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:25.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.184+0000 7fcac77ff700 1 -- 172.21.15.23:0/2604997530 >> 172.21.15.23:0/2604997530 conn(0x7fcac00ca3f0 msgr2=0x7fcac00ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:25.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.184+0000 7fcac77ff700 1 -- 172.21.15.23:0/2604997530 shutdown_connections 2023-06-22T08:23:25.192 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.184+0000 7fcac77ff700 1 -- 172.21.15.23:0/2604997530 wait complete. 2023-06-22T08:23:25.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.186+0000 7fcac77ff700 1 Processor -- start 2023-06-22T08:23:25.194 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.186+0000 7fcac77ff700 1 -- start start 2023-06-22T08:23:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac77ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcac006d050 0x7fcac0147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac77ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac00c3a40 0x7fcac0147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac77ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac015a320 0x7fcac0155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac77ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac00c2f90 con 0x7fcac00c3a40 2023-06-22T08:23:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac77ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac01ee930 con 0x7fcac015a320 2023-06-22T08:23:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac77ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac0149110 con 0x7fcac006d050 2023-06-22T08:23:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac4d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac00c3a40 0x7fcac0147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac5d9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac015a320 0x7fcac0155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac559b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcac006d050 0x7fcac0147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac5d9c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac015a320 0x7fcac0155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:43826/0 (socket says 172.21.15.23:43826) 2023-06-22T08:23:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac4d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac00c3a40 0x7fcac0147ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36236/0 (socket says 172.21.15.23:36236) 2023-06-22T08:23:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.187+0000 7fcac5d9c700 1 -- 172.21.15.23:0/1516337719 learned_addr learned my addr 172.21.15.23:0/1516337719 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcac4d9a700 1 -- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcac006d050 msgr2=0x7fcac0147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcac4d9a700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcac006d050 0x7fcac0147660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcac4d9a700 1 -- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac015a320 msgr2=0x7fcac0155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcac4d9a700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac015a320 0x7fcac0155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcac4d9a700 1 -- 172.21.15.23:0/1516337719 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab8009a70 con 0x7fcac00c3a40 2023-06-22T08:23:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcac4d9a700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac00c3a40 0x7fcac0147ba0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fcab000e900 tx=0x7fcab000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcab5ffb700 1 -- 172.21.15.23:0/1516337719 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0013680 con 0x7fcac00c3a40 2023-06-22T08:23:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcab5ffb700 1 -- 172.21.15.23:0/1516337719 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab00137e0 con 0x7fcac00c3a40 2023-06-22T08:23:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcab5ffb700 1 -- 172.21.15.23:0/1516337719 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0013b50 con 0x7fcac00c3a40 2023-06-22T08:23:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcac77ff700 1 -- 172.21.15.23:0/1516337719 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac01493d0 con 0x7fcac00c3a40 2023-06-22T08:23:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.188+0000 7fcac77ff700 1 -- 172.21.15.23:0/1516337719 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac01eee90 con 0x7fcac00c3a40 2023-06-22T08:23:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.189+0000 7fcac77ff700 1 -- 172.21.15.23:0/1516337719 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac0157210 con 0x7fcac00c3a40 2023-06-22T08:23:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.190+0000 7fcab5ffb700 1 -- 172.21.15.23:0/1516337719 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fcab0013cb0 con 0x7fcac00c3a40 2023-06-22T08:23:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.191+0000 7fcab5ffb700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcaa406ace0 0x7fcaa406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.191+0000 7fcab5ffb700 1 -- 172.21.15.23:0/1516337719 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 7957+0+0 (secure 0 0 0) 0x7fcab0089ef0 con 0x7fcac00c3a40 2023-06-22T08:23:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.192+0000 7fcac559b700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcaa406ace0 0x7fcaa406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:25.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.192+0000 7fcac559b700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcaa406ace0 0x7fcaa406d190 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fcabc007fd0 tx=0x7fcabc007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:25.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.196+0000 7fcab5ffb700 1 -- 172.21.15.23:0/1516337719 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fcab0054790 con 0x7fcac00c3a40 2023-06-22T08:23:25.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.452+0000 7fcac77ff700 1 -- 172.21.15.23:0/1516337719 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcac00c6930 con 0x7fcac00c3a40 2023-06-22T08:23:25.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.452+0000 7fcab5ffb700 1 -- 172.21.15.23:0/1516337719 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5482 (secure 0 0 0) 0x7fcab0017020 con 0x7fcac00c3a40 2023-06-22T08:23:25.455 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:25.455 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:20.921145+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 -- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcaa406ace0 msgr2=0x7fcaa406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:25.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcaa406ace0 0x7fcaa406d190 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fcabc007fd0 tx=0x7fcabc007de0 comp rx=0 tx=0).stop 2023-06-22T08:23:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 -- 172.21.15.23:0/1516337719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac00c3a40 msgr2=0x7fcac0147ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac00c3a40 0x7fcac0147ba0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fcab000e900 tx=0x7fcab000ec10 comp rx=0 tx=0).stop 2023-06-22T08:23:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 -- 172.21.15.23:0/1516337719 shutdown_connections 2023-06-22T08:23:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fcaa406ace0 0x7fcaa406d190 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcac015a320 0x7fcac0155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcac006d050 0x7fcac0147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:25.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 --2- 172.21.15.23:0/1516337719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcac00c3a40 0x7fcac0147ba0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 -- 172.21.15.23:0/1516337719 >> 172.21.15.23:0/1516337719 conn(0x7fcac00ca3f0 msgr2=0x7fcac00708a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 -- 172.21.15.23:0/1516337719 shutdown_connections 2023-06-22T08:23:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:25.455+0000 7fcac77ff700 1 -- 172.21.15.23:0/1516337719 wait complete. 2023-06-22T08:23:25.459 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 182 2023-06-22T08:23:25.476 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:23:25.897 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~183s 2023-06-22T08:23:27.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:27.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.688+0000 7fbba0bff700 1 Processor -- start 2023-06-22T08:23:27.691 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.688+0000 7fbba0bff700 1 -- start start 2023-06-22T08:23:27.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.689+0000 7fbba0bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0cb260 0x7fbb9c0c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:27.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.689+0000 7fbba0bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb9c0c9200 0x7fbb9c0cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:27.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.689+0000 7fbba0bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c0cbd60 con 0x7fbb9c0b0fc0 2023-06-22T08:23:27.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.689+0000 7fbba0bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9c0cbb70 con 0x7fbb9c0c9200 2023-06-22T08:23:27.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.689+0000 7fbba0bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c158980 con 0x7fbb9c0cb260 2023-06-22T08:23:27.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb9ad9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbb9c0b0fc0 0x7fbb9c15be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:50112/0 (socket says 172.21.15.23:50112) 2023-06-22T08:23:27.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb9ad9d700 1 -- 172.21.15.23:0/2633554084 learned_addr learned my addr 172.21.15.23:0/2633554084 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:27.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb99d9b700 1 --2- 172.21.15.23:0/2633554084 >> v2:172.21.15.67:3300/0 conn(0x7fbb9c0c9200 0x7fbb9c0cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:27.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb9a59c700 1 --2- 172.21.15.23:0/2633554084 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0cb260 0x7fbb9c0c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:27.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb9959a700 1 -- 172.21.15.23:0/2633554084 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930104766 0 0) 0x7fbb9c0cbd60 con 0x7fbb9c0b0fc0 2023-06-22T08:23:27.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb9959a700 1 -- 172.21.15.23:0/2633554084 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb840035b0 con 0x7fbb9c0b0fc0 2023-06-22T08:23:27.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb9a59c700 1 -- 172.21.15.23:0/2633554084 >> v2:172.21.15.67:3300/0 conn(0x7fbb9c0c9200 msgr2=0x7fbb9c0cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:27.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb9a59c700 1 --2- 172.21.15.23:0/2633554084 >> v2:172.21.15.67:3300/0 conn(0x7fbb9c0c9200 0x7fbb9c0cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:27.698 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb9a59c700 1 -- 172.21.15.23:0/2633554084 >> v1:172.21.15.23:6789/0 conn(0x7fbb9c0b0fc0 legacy=0x7fbb9c15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:27.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb9a59c700 1 -- 172.21.15.23:0/2633554084 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c159550 con 0x7fbb9c0cb260 2023-06-22T08:23:27.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.690+0000 7fbb99d9b700 1 --2- 172.21.15.23:0/2633554084 >> v2:172.21.15.67:3300/0 conn(0x7fbb9c0c9200 0x7fbb9c0cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:23:27.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbb9a59c700 1 --2- 172.21.15.23:0/2633554084 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0cb260 0x7fbb9c0c4a50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009840 tx=0x7fbb8c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b12c6076f0182 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:27.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbb9959a700 1 -- 172.21.15.23:0/2633554084 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c011830 con 0x7fbb9c0cb260 2023-06-22T08:23:27.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbb9959a700 1 -- 172.21.15.23:0/2633554084 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c011990 con 0x7fbb9c0cb260 2023-06-22T08:23:27.699 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbb9959a700 1 -- 172.21.15.23:0/2633554084 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c011d20 con 0x7fbb9c0cb260 2023-06-22T08:23:27.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbba0bff700 1 -- 172.21.15.23:0/2633554084 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0cb260 msgr2=0x7fbb9c0c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:27.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbba0bff700 1 --2- 172.21.15.23:0/2633554084 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0cb260 0x7fbb9c0c4a50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009840 tx=0x7fbb8c009b50 comp rx=0 tx=0).stop 2023-06-22T08:23:27.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbba0bff700 1 -- 172.21.15.23:0/2633554084 shutdown_connections 2023-06-22T08:23:27.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbba0bff700 1 --2- 172.21.15.23:0/2633554084 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0cb260 0x7fbb9c0c4a50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:27.700 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbba0bff700 1 --2- 172.21.15.23:0/2633554084 >> v2:172.21.15.67:3300/0 conn(0x7fbb9c0c9200 0x7fbb9c0cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:27.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbba0bff700 1 -- 172.21.15.23:0/2633554084 >> 172.21.15.23:0/2633554084 conn(0x7fbb9c065b90 msgr2=0x7fbb9c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:27.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.691+0000 7fbba0bff700 1 -- 172.21.15.23:0/2633554084 shutdown_connections 2023-06-22T08:23:27.701 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.692+0000 7fbba0bff700 1 -- 172.21.15.23:0/2633554084 wait complete. 2023-06-22T08:23:27.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.693+0000 7fbba0bff700 1 Processor -- start 2023-06-22T08:23:27.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.693+0000 7fbba0bff700 1 -- start start 2023-06-22T08:23:27.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.694+0000 7fbba0bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0b0fc0 0x7fbb9c1f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:27.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.694+0000 7fbba0bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c0c9200 0x7fbb9c157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:27.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.694+0000 7fbba0bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb9c0cb260 0x7fbb9c1481b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:27.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.694+0000 7fbba0bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c1f1d40 con 0x7fbb9c0cb260 2023-06-22T08:23:27.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.694+0000 7fbba0bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c0b45f0 con 0x7fbb9c0c9200 2023-06-22T08:23:27.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.694+0000 7fbba0bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c1f26a0 con 0x7fbb9c0b0fc0 2023-06-22T08:23:27.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb9c0cb260 0x7fbb9c1481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:27.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9a59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0b0fc0 0x7fbb9c1f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:27.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb99d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c0c9200 0x7fbb9c157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:27.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb9c0cb260 0x7fbb9c1481b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48298/0 (socket says 172.21.15.23:48298) 2023-06-22T08:23:27.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9ad9d700 1 -- 172.21.15.23:0/3393864026 learned_addr learned my addr 172.21.15.23:0/3393864026 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:27.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9a59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0b0fc0 0x7fbb9c1f2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:45162/0 (socket says 172.21.15.23:45162) 2023-06-22T08:23:27.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9ad9d700 1 -- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0b0fc0 msgr2=0x7fbb9c1f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:27.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9ad9d700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0b0fc0 0x7fbb9c1f2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:27.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9ad9d700 1 -- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c0c9200 msgr2=0x7fbb9c157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:27.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9ad9d700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c0c9200 0x7fbb9c157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:27.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9ad9d700 1 -- 172.21.15.23:0/3393864026 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb900094a0 con 0x7fbb9c0cb260 2023-06-22T08:23:27.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.695+0000 7fbb9ad9d700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb9c0cb260 0x7fbb9c1481b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400cb40 tx=0x7fbb9400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:27.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.696+0000 7fbb8affd700 1 -- 172.21.15.23:0/3393864026 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9400a9e0 con 0x7fbb9c0cb260 2023-06-22T08:23:27.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.696+0000 7fbb8affd700 1 -- 172.21.15.23:0/3393864026 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9400ab40 con 0x7fbb9c0cb260 2023-06-22T08:23:27.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.696+0000 7fbba0bff700 1 -- 172.21.15.23:0/3393864026 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8c0094a0 con 0x7fbb9c0cb260 2023-06-22T08:23:27.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.696+0000 7fbb8affd700 1 -- 172.21.15.23:0/3393864026 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94012400 con 0x7fbb9c0cb260 2023-06-22T08:23:27.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.696+0000 7fbba0bff700 1 -- 172.21.15.23:0/3393864026 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9c0ccf10 con 0x7fbb9c0cb260 2023-06-22T08:23:27.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.697+0000 7fbba0bff700 1 -- 172.21.15.23:0/3393864026 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9c1f1e80 con 0x7fbb9c0cb260 2023-06-22T08:23:27.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.698+0000 7fbb8affd700 1 -- 172.21.15.23:0/3393864026 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbb9401b3b0 con 0x7fbb9c0cb260 2023-06-22T08:23:27.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.699+0000 7fbb8affd700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbb7c06ace0 0x7fbb7c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:27.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.699+0000 7fbb8affd700 1 -- 172.21.15.23:0/3393864026 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 7957+0+0 (secure 0 0 0) 0x7fbb94089880 con 0x7fbb9c0cb260 2023-06-22T08:23:27.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.699+0000 7fbb9a59c700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbb7c06ace0 0x7fbb7c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:27.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.700+0000 7fbb9a59c700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbb7c06ace0 0x7fbb7c06d190 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c000c00 tx=0x7fbb8c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:27.708 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.703+0000 7fbb8affd700 1 -- 172.21.15.23:0/3393864026 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbb9c1f1e80 con 0x7fbb9c0cb260 2023-06-22T08:23:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.959+0000 7fbba0bff700 1 -- 172.21.15.23:0/3393864026 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb9c1f1e80 con 0x7fbb9c0cb260 2023-06-22T08:23:27.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.959+0000 7fbb8affd700 1 -- 172.21.15.23:0/3393864026 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5482 (secure 0 0 0) 0x7fbb9c1f1e80 con 0x7fbb9c0cb260 2023-06-22T08:23:27.962 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:27.962 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:20.921145+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 -- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbb7c06ace0 msgr2=0x7fbb7c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbb7c06ace0 0x7fbb7c06d190 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c000c00 tx=0x7fbb8c008030 comp rx=0 tx=0).stop 2023-06-22T08:23:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 -- 172.21.15.23:0/3393864026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb9c0cb260 msgr2=0x7fbb9c1481b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb9c0cb260 0x7fbb9c1481b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400cb40 tx=0x7fbb9400ce50 comp rx=0 tx=0).stop 2023-06-22T08:23:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 -- 172.21.15.23:0/3393864026 shutdown_connections 2023-06-22T08:23:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb9c0cb260 0x7fbb9c1481b0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbb9c0b0fc0 0x7fbb9c1f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbb7c06ace0 0x7fbb7c06d190 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 --2- 172.21.15.23:0/3393864026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb9c0c9200 0x7fbb9c157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 -- 172.21.15.23:0/3393864026 >> 172.21.15.23:0/3393864026 conn(0x7fbb9c065b90 msgr2=0x7fbb9c06adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 -- 172.21.15.23:0/3393864026 shutdown_connections 2023-06-22T08:23:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:27.962+0000 7fbba0bff700 1 -- 172.21.15.23:0/3393864026 wait complete. 2023-06-22T08:23:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 182 2023-06-22T08:23:27.983 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:23:29.984 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:30.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.194+0000 7f1e985ff700 1 Processor -- start 2023-06-22T08:23:30.197 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.194+0000 7f1e985ff700 1 -- start start 2023-06-22T08:23:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.196+0000 7f1e985ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1e90152ce0 0x7f1e900cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.196+0000 7f1e985ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e9015c370 con 0x7f1e900c9fd0 2023-06-22T08:23:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.196+0000 7f1e985ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e9015bda0 con 0x7f1e900c40b0 2023-06-22T08:23:30.198 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.196+0000 7f1e985ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e901530b0 con 0x7f1e90152ce0 2023-06-22T08:23:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.196+0000 7f1e9639b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1e90152ce0 0x7f1e900cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:30.199 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.196+0000 7f1e95b9a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1e900c9fd0 0x7f1e9015c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:50114/0 (socket says 172.21.15.23:50114) 2023-06-22T08:23:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.196+0000 7f1e95b9a700 1 -- 172.21.15.23:0/1684415325 learned_addr learned my addr 172.21.15.23:0/1684415325 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.196+0000 7f1e9639b700 1 -- 172.21.15.23:0/1684415325 >> v1:172.21.15.67:6789/0 conn(0x7f1e900c40b0 legacy=0x7f1e900c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.196+0000 7f1e9639b700 1 -- 172.21.15.23:0/1684415325 >> v1:172.21.15.23:6789/0 conn(0x7f1e900c9fd0 legacy=0x7f1e9015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.196+0000 7f1e9639b700 1 -- 172.21.15.23:0/1684415325 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e900c9460 con 0x7f1e90152ce0 2023-06-22T08:23:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.197+0000 7f1e9639b700 1 --2- 172.21.15.23:0/1684415325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e90152ce0 0x7f1e900cf1b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009840 tx=0x7f1e8c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be21145fe3090f5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:30.200 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.197+0000 7f1e95399700 1 -- 172.21.15.23:0/1684415325 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c011830 con 0x7f1e90152ce0 2023-06-22T08:23:30.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.197+0000 7f1e95399700 1 -- 172.21.15.23:0/1684415325 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8c011990 con 0x7f1e90152ce0 2023-06-22T08:23:30.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.197+0000 7f1e95399700 1 -- 172.21.15.23:0/1684415325 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c011d20 con 0x7f1e90152ce0 2023-06-22T08:23:30.201 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.197+0000 7f1e985ff700 1 -- 172.21.15.23:0/1684415325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e90152ce0 msgr2=0x7f1e900cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.197+0000 7f1e985ff700 1 --2- 172.21.15.23:0/1684415325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e90152ce0 0x7f1e900cf1b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009840 tx=0x7f1e8c009b50 comp rx=0 tx=0).stop 2023-06-22T08:23:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.198+0000 7f1e985ff700 1 -- 172.21.15.23:0/1684415325 shutdown_connections 2023-06-22T08:23:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.198+0000 7f1e985ff700 1 --2- 172.21.15.23:0/1684415325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e90152ce0 0x7f1e900cf1b0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.198+0000 7f1e985ff700 1 -- 172.21.15.23:0/1684415325 >> 172.21.15.23:0/1684415325 conn(0x7f1e9006ae30 msgr2=0x7f1e9006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.198+0000 7f1e985ff700 1 -- 172.21.15.23:0/1684415325 shutdown_connections 2023-06-22T08:23:30.203 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.198+0000 7f1e985ff700 1 -- 172.21.15.23:0/1684415325 wait complete. 2023-06-22T08:23:30.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.199+0000 7f1e985ff700 1 Processor -- start 2023-06-22T08:23:30.204 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.200+0000 7f1e985ff700 1 -- start start 2023-06-22T08:23:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.200+0000 7f1e985ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e900c40b0 0x7f1e901e1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.200+0000 7f1e985ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e900c9fd0 0x7f1e90154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.200+0000 7f1e985ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e90152ce0 0x7f1e90147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.200+0000 7f1e985ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e90153610 con 0x7f1e900c40b0 2023-06-22T08:23:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.200+0000 7f1e985ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e901599f0 con 0x7f1e900c9fd0 2023-06-22T08:23:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.200+0000 7f1e985ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e90154fc0 con 0x7f1e90152ce0 2023-06-22T08:23:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.201+0000 7f1e9639b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e900c40b0 0x7f1e901e1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.201+0000 7f1e9639b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e900c40b0 0x7f1e901e1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48322/0 (socket says 172.21.15.23:48322) 2023-06-22T08:23:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.201+0000 7f1e9639b700 1 -- 172.21.15.23:0/1146772213 learned_addr learned my addr 172.21.15.23:0/1146772213 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.201+0000 7f1e95b9a700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e900c9fd0 0x7f1e90154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.201+0000 7f1e96b9c700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e90152ce0 0x7f1e90147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.201+0000 7f1e9639b700 1 -- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e90152ce0 msgr2=0x7f1e90147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.201+0000 7f1e9639b700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e90152ce0 0x7f1e90147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.201+0000 7f1e9639b700 1 -- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e900c9fd0 msgr2=0x7f1e90154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.201+0000 7f1e9639b700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e900c9fd0 0x7f1e90154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.201+0000 7f1e9639b700 1 -- 172.21.15.23:0/1146772213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c0094a0 con 0x7f1e900c40b0 2023-06-22T08:23:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.202+0000 7f1e9639b700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e900c40b0 0x7f1e901e1510 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c000c00 tx=0x7f1e8c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.202+0000 7f1e86ffd700 1 -- 172.21.15.23:0/1146772213 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c0036a0 con 0x7f1e900c40b0 2023-06-22T08:23:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.202+0000 7f1e86ffd700 1 -- 172.21.15.23:0/1146772213 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8c01e070 con 0x7f1e900c40b0 2023-06-22T08:23:30.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.202+0000 7f1e985ff700 1 -- 172.21.15.23:0/1146772213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e901552a0 con 0x7f1e900c40b0 2023-06-22T08:23:30.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.202+0000 7f1e86ffd700 1 -- 172.21.15.23:0/1146772213 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c003a00 con 0x7f1e900c40b0 2023-06-22T08:23:30.208 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.202+0000 7f1e985ff700 1 -- 172.21.15.23:0/1146772213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9014a1e0 con 0x7f1e900c40b0 2023-06-22T08:23:30.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.203+0000 7f1e985ff700 1 -- 172.21.15.23:0/1146772213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e90067a00 con 0x7f1e900c40b0 2023-06-22T08:23:30.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.204+0000 7f1e86ffd700 1 -- 172.21.15.23:0/1146772213 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1e8c003b60 con 0x7f1e900c40b0 2023-06-22T08:23:30.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.205+0000 7f1e86ffd700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e7406ad10 0x7f1e7406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:30.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.205+0000 7f1e86ffd700 1 -- 172.21.15.23:0/1146772213 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f1e8c08ad20 con 0x7f1e900c40b0 2023-06-22T08:23:30.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.205+0000 7f1e95b9a700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e7406ad10 0x7f1e7406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:30.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.206+0000 7f1e95b9a700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e7406ad10 0x7f1e7406d1c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1e90152500 tx=0x7f1e80009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:30.212 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.210+0000 7f1e86ffd700 1 -- 172.21.15.23:0/1146772213 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1e8c055600 con 0x7f1e900c40b0 2023-06-22T08:23:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.463+0000 7f1e985ff700 1 -- 172.21.15.23:0/1146772213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e9004ea20 con 0x7f1e900c40b0 2023-06-22T08:23:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.464+0000 7f1e86ffd700 1 -- 172.21.15.23:0/1146772213 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5481 (secure 0 0 0) 0x7f1e8c020020 con 0x7f1e900c40b0 2023-06-22T08:23:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:30.467 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:29.047319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:30.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.466+0000 7f1e985ff700 1 -- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e7406ad10 msgr2=0x7f1e7406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:30.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.466+0000 7f1e985ff700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e7406ad10 0x7f1e7406d1c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1e90152500 tx=0x7f1e80009040 comp rx=0 tx=0).stop 2023-06-22T08:23:30.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.466+0000 7f1e985ff700 1 -- 172.21.15.23:0/1146772213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e900c40b0 msgr2=0x7f1e901e1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:30.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.466+0000 7f1e985ff700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e900c40b0 0x7f1e901e1510 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c000c00 tx=0x7f1e8c004750 comp rx=0 tx=0).stop 2023-06-22T08:23:30.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.466+0000 7f1e985ff700 1 -- 172.21.15.23:0/1146772213 shutdown_connections 2023-06-22T08:23:30.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.466+0000 7f1e985ff700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1e90152ce0 0x7f1e90147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:30.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.466+0000 7f1e985ff700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e900c40b0 0x7f1e901e1510 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:30.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.466+0000 7f1e985ff700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1e7406ad10 0x7f1e7406d1c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:30.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.466+0000 7f1e985ff700 1 --2- 172.21.15.23:0/1146772213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e900c9fd0 0x7f1e90154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:30.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.466+0000 7f1e985ff700 1 -- 172.21.15.23:0/1146772213 >> 172.21.15.23:0/1146772213 conn(0x7f1e9006ae30 msgr2=0x7f1e9014d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:30.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.467+0000 7f1e985ff700 1 -- 172.21.15.23:0/1146772213 shutdown_connections 2023-06-22T08:23:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:30.467+0000 7f1e985ff700 1 -- 172.21.15.23:0/1146772213 wait complete. 2023-06-22T08:23:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 183 2023-06-22T08:23:30.487 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:23:32.488 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:32.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~190s 2023-06-22T08:23:32.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.699+0000 7f1414bff700 1 Processor -- start 2023-06-22T08:23:32.702 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.699+0000 7f1414bff700 1 -- start start 2023-06-22T08:23:32.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.701+0000 7f1414bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1410152ce0 0x7f14100cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:32.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.701+0000 7f1414bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141015c370 con 0x7f14100c40b0 2023-06-22T08:23:32.703 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.701+0000 7f1414bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141015bda0 con 0x7f14100c9fd0 2023-06-22T08:23:32.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.701+0000 7f1414bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14101530b0 con 0x7f1410152ce0 2023-06-22T08:23:32.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.701+0000 7f140ed9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f14100c40b0 0x7f14100c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:50124/0 (socket says 172.21.15.23:50124) 2023-06-22T08:23:32.704 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.701+0000 7f140ed9d700 1 -- 172.21.15.23:0/1662632525 learned_addr learned my addr 172.21.15.23:0/1662632525 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:32.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.701+0000 7f140e59c700 1 --2- 172.21.15.23:0/1662632525 >> v2:172.21.15.67:3300/0 conn(0x7f1410152ce0 0x7f14100cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:32.705 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.701+0000 7f140dd9b700 1 -- 172.21.15.23:0/1662632525 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101517212 0 0) 0x7f141015c370 con 0x7f14100c40b0 2023-06-22T08:23:32.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.702+0000 7f140dd9b700 1 -- 172.21.15.23:0/1662632525 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13fc0035b0 con 0x7f14100c40b0 2023-06-22T08:23:32.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.702+0000 7f140e59c700 1 -- 172.21.15.23:0/1662632525 >> v1:172.21.15.67:6789/0 conn(0x7f14100c9fd0 legacy=0x7f141015c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:32.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.702+0000 7f140e59c700 1 -- 172.21.15.23:0/1662632525 >> v1:172.21.15.23:6789/0 conn(0x7f14100c40b0 legacy=0x7f14100c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:32.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.702+0000 7f140e59c700 1 -- 172.21.15.23:0/1662632525 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14100c9460 con 0x7f1410152ce0 2023-06-22T08:23:32.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.702+0000 7f140e59c700 1 --2- 172.21.15.23:0/1662632525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1410152ce0 0x7f14100cf1b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1400009840 tx=0x7f1400009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3f6abae5f2b03cb server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:32.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.702+0000 7f140dd9b700 1 -- 172.21.15.23:0/1662632525 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1400011830 con 0x7f1410152ce0 2023-06-22T08:23:32.706 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.702+0000 7f140dd9b700 1 -- 172.21.15.23:0/1662632525 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1400011990 con 0x7f1410152ce0 2023-06-22T08:23:32.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.702+0000 7f140dd9b700 1 -- 172.21.15.23:0/1662632525 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1400011d20 con 0x7f1410152ce0 2023-06-22T08:23:32.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.703+0000 7f1414bff700 1 -- 172.21.15.23:0/1662632525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1410152ce0 msgr2=0x7f14100cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:32.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.703+0000 7f1414bff700 1 --2- 172.21.15.23:0/1662632525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1410152ce0 0x7f14100cf1b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1400009840 tx=0x7f1400009b50 comp rx=0 tx=0).stop 2023-06-22T08:23:32.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.703+0000 7f1414bff700 1 -- 172.21.15.23:0/1662632525 shutdown_connections 2023-06-22T08:23:32.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.703+0000 7f1414bff700 1 --2- 172.21.15.23:0/1662632525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1410152ce0 0x7f14100cf1b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:32.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.703+0000 7f1414bff700 1 -- 172.21.15.23:0/1662632525 >> 172.21.15.23:0/1662632525 conn(0x7f141006ae30 msgr2=0x7f141006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:32.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.703+0000 7f1414bff700 1 -- 172.21.15.23:0/1662632525 shutdown_connections 2023-06-22T08:23:32.707 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.703+0000 7f1414bff700 1 -- 172.21.15.23:0/1662632525 wait complete. 2023-06-22T08:23:32.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.705+0000 7f1414bff700 1 Processor -- start 2023-06-22T08:23:32.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.705+0000 7f1414bff700 1 -- start start 2023-06-22T08:23:32.709 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f1414bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14100c40b0 0x7f141014ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:32.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f1414bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14100c9fd0 0x7f14101474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:32.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f1414bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1410152ce0 0x7f14100c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:32.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f1414bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14101eee70 con 0x7f14100c40b0 2023-06-22T08:23:32.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f1414bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141014d440 con 0x7f14100c9fd0 2023-06-22T08:23:32.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f1414bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14101ee910 con 0x7f1410152ce0 2023-06-22T08:23:32.710 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f140e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14100c40b0 0x7f141014ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:32.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f140e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14100c40b0 0x7f141014ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48332/0 (socket says 172.21.15.23:48332) 2023-06-22T08:23:32.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f140e59c700 1 -- 172.21.15.23:0/2552313134 learned_addr learned my addr 172.21.15.23:0/2552313134 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:32.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f1407fff700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14100c9fd0 0x7f14101474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:32.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f140ed9d700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1410152ce0 0x7f14100c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:32.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f140e59c700 1 -- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1410152ce0 msgr2=0x7f14100c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:32.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f140e59c700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1410152ce0 0x7f14100c6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:32.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f140e59c700 1 -- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14100c9fd0 msgr2=0x7f14101474c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:32.711 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f140e59c700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14100c9fd0 0x7f14101474c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:32.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.706+0000 7f140e59c700 1 -- 172.21.15.23:0/2552313134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14000094a0 con 0x7f14100c40b0 2023-06-22T08:23:32.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.707+0000 7f140e59c700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14100c40b0 0x7f141014ef50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1400014040 tx=0x7f1400004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:32.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.707+0000 7f1406ffd700 1 -- 172.21.15.23:0/2552313134 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000038a0 con 0x7f14100c40b0 2023-06-22T08:23:32.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.707+0000 7f1406ffd700 1 -- 172.21.15.23:0/2552313134 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140001e070 con 0x7f14100c40b0 2023-06-22T08:23:32.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.707+0000 7f1406ffd700 1 -- 172.21.15.23:0/2552313134 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1400003c20 con 0x7f14100c40b0 2023-06-22T08:23:32.712 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.707+0000 7f1414bff700 1 -- 172.21.15.23:0/2552313134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141014a760 con 0x7f14100c40b0 2023-06-22T08:23:32.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.707+0000 7f1414bff700 1 -- 172.21.15.23:0/2552313134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14100c87d0 con 0x7f14100c40b0 2023-06-22T08:23:32.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.708+0000 7f1414bff700 1 -- 172.21.15.23:0/2552313134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f141014d5b0 con 0x7f14100c40b0 2023-06-22T08:23:32.713 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.709+0000 7f1406ffd700 1 -- 172.21.15.23:0/2552313134 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f140002d450 con 0x7f14100c40b0 2023-06-22T08:23:32.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.710+0000 7f1406ffd700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f13f006ace0 0x7f13f006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:32.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.710+0000 7f1406ffd700 1 -- 172.21.15.23:0/2552313134 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 7901+0+0 (secure 0 0 0) 0x7f140008ac90 con 0x7f14100c40b0 2023-06-22T08:23:32.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.710+0000 7f1407fff700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f13f006ace0 0x7f13f006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:32.714 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.711+0000 7f1407fff700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f13f006ace0 0x7f13f006d190 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f141015c540 tx=0x7f13f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:32.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.715+0000 7f1406ffd700 1 -- 172.21.15.23:0/2552313134 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1400055600 con 0x7f14100c40b0 2023-06-22T08:23:32.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.966+0000 7f1414bff700 1 -- 172.21.15.23:0/2552313134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f141004ea20 con 0x7f14100c40b0 2023-06-22T08:23:32.969 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.966+0000 7f1406ffd700 1 -- 172.21.15.23:0/2552313134 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5481 (secure 0 0 0) 0x7f1400020020 con 0x7f14100c40b0 2023-06-22T08:23:32.970 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:32.970 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:29.047319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":5995},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_5995":{"gid":5995,"name":"f","rank":3,"incarnation":101,"state":"up:active","state_seq":34,"addr":"172.21.15.67:6839/298860192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":298860192},{"type":"v1","addr":"172.21.15.67:6839","nonce":298860192}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:32.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 -- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f13f006ace0 msgr2=0x7f13f006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:32.971 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f13f006ace0 0x7f13f006d190 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f141015c540 tx=0x7f13f8009040 comp rx=0 tx=0).stop 2023-06-22T08:23:32.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 -- 172.21.15.23:0/2552313134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14100c40b0 msgr2=0x7f141014ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:32.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14100c40b0 0x7f141014ef50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1400014040 tx=0x7f1400004750 comp rx=0 tx=0).stop 2023-06-22T08:23:32.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 -- 172.21.15.23:0/2552313134 shutdown_connections 2023-06-22T08:23:32.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1410152ce0 0x7f14100c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:32.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14100c40b0 0x7f141014ef50 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:32.972 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f13f006ace0 0x7f13f006d190 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:32.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 --2- 172.21.15.23:0/2552313134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14100c9fd0 0x7f14101474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:32.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 -- 172.21.15.23:0/2552313134 >> 172.21.15.23:0/2552313134 conn(0x7f141006ae30 msgr2=0x7f1410150c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:32.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 -- 172.21.15.23:0/2552313134 shutdown_connections 2023-06-22T08:23:32.973 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:32.969+0000 7f1414bff700 1 -- 172.21.15.23:0/2552313134 wait complete. 2023-06-22T08:23:32.973 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 183 2023-06-22T08:23:32.990 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-06-22T08:23:34.991 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:35.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.206+0000 7f350c9ff700 1 Processor -- start 2023-06-22T08:23:35.209 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.206+0000 7f350c9ff700 1 -- start start 2023-06-22T08:23:35.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f350c9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35081558b0 0x7f350806df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.210 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f350c9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f350806e450 0x7f350806d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f350c9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35080c4ea0 con 0x7f350814f0c0 2023-06-22T08:23:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f350c9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35080c1f70 con 0x7f350806e450 2023-06-22T08:23:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f350c9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35080c68f0 con 0x7f35081558b0 2023-06-22T08:23:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f350659c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f350814f0c0 0x7f35080c61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:50138/0 (socket says 172.21.15.23:50138) 2023-06-22T08:23:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f350659c700 1 -- 172.21.15.23:0/2882301668 learned_addr learned my addr 172.21.15.23:0/2882301668 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f34fffff700 1 --2- 172.21.15.23:0/2882301668 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35081558b0 0x7f350806df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f3506d9d700 1 --2- 172.21.15.23:0/2882301668 >> v2:172.21.15.67:3300/0 conn(0x7f350806e450 0x7f350806d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f3505d9b700 1 -- 172.21.15.23:0/2882301668 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653229557 0 0) 0x7f35080c4ea0 con 0x7f350814f0c0 2023-06-22T08:23:35.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f3505d9b700 1 -- 172.21.15.23:0/2882301668 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f40035b0 con 0x7f350814f0c0 2023-06-22T08:23:35.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f34fffff700 1 -- 172.21.15.23:0/2882301668 >> v2:172.21.15.67:3300/0 conn(0x7f350806e450 msgr2=0x7f350806d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:35.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f34fffff700 1 --2- 172.21.15.23:0/2882301668 >> v2:172.21.15.67:3300/0 conn(0x7f350806e450 0x7f350806d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f34fffff700 1 -- 172.21.15.23:0/2882301668 >> v1:172.21.15.23:6789/0 conn(0x7f350814f0c0 legacy=0x7f35080c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.213 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.208+0000 7f34fffff700 1 -- 172.21.15.23:0/2882301668 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3508155ce0 con 0x7f35081558b0 2023-06-22T08:23:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.209+0000 7f3506d9d700 1 --2- 172.21.15.23:0/2882301668 >> v2:172.21.15.67:3300/0 conn(0x7f350806e450 0x7f350806d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:23:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.209+0000 7f34fffff700 1 --2- 172.21.15.23:0/2882301668 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35081558b0 0x7f350806df10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009840 tx=0x7f34f0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25093c35ee7bb2e6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.209+0000 7f3505d9b700 1 -- 172.21.15.23:0/2882301668 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f0011830 con 0x7f35081558b0 2023-06-22T08:23:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.209+0000 7f3505d9b700 1 -- 172.21.15.23:0/2882301668 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f0011990 con 0x7f35081558b0 2023-06-22T08:23:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.209+0000 7f3505d9b700 1 -- 172.21.15.23:0/2882301668 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f0011d20 con 0x7f35081558b0 2023-06-22T08:23:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.210+0000 7f350c9ff700 1 -- 172.21.15.23:0/2882301668 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35081558b0 msgr2=0x7f350806df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.210+0000 7f350c9ff700 1 --2- 172.21.15.23:0/2882301668 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35081558b0 0x7f350806df10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009840 tx=0x7f34f0009b50 comp rx=0 tx=0).stop 2023-06-22T08:23:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.210+0000 7f350c9ff700 1 -- 172.21.15.23:0/2882301668 shutdown_connections 2023-06-22T08:23:35.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.210+0000 7f350c9ff700 1 --2- 172.21.15.23:0/2882301668 >> v2:172.21.15.67:3300/0 conn(0x7f350806e450 0x7f350806d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.210+0000 7f350c9ff700 1 --2- 172.21.15.23:0/2882301668 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f35081558b0 0x7f350806df10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.210+0000 7f350c9ff700 1 -- 172.21.15.23:0/2882301668 >> 172.21.15.23:0/2882301668 conn(0x7f35080ca3f0 msgr2=0x7f35080ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:35.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.210+0000 7f350c9ff700 1 -- 172.21.15.23:0/2882301668 shutdown_connections 2023-06-22T08:23:35.215 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.210+0000 7f350c9ff700 1 -- 172.21.15.23:0/2882301668 wait complete. 2023-06-22T08:23:35.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.212+0000 7f350c9ff700 1 Processor -- start 2023-06-22T08:23:35.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.212+0000 7f350c9ff700 1 -- start start 2023-06-22T08:23:35.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f350c9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f350806e450 0x7f35081e1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f350c9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f350814f0c0 0x7f35080c3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f350c9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35081558b0 0x7f3508144bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f350c9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35081e1800 con 0x7f35081558b0 2023-06-22T08:23:35.217 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f350c9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3508151b40 con 0x7f350814f0c0 2023-06-22T08:23:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f350c9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3508151d90 con 0x7f350806e450 2023-06-22T08:23:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f34fffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f350814f0c0 0x7f35080c3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f3506d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35081558b0 0x7f3508144bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f350659c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f350806e450 0x7f35081e1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f34fffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f350814f0c0 0x7f35080c3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:59764/0 (socket says 172.21.15.23:59764) 2023-06-22T08:23:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f34fffff700 1 -- 172.21.15.23:0/2674917940 learned_addr learned my addr 172.21.15.23:0/2674917940 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f34fffff700 1 -- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f350806e450 msgr2=0x7f35081e1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f34fffff700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f350806e450 0x7f35081e1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f34fffff700 1 -- 172.21.15.23:0/2674917940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35081558b0 msgr2=0x7f3508144bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f34fffff700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35081558b0 0x7f3508144bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f34fffff700 1 -- 172.21.15.23:0/2674917940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350000ab80 con 0x7f350814f0c0 2023-06-22T08:23:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.213+0000 7f3506d9d700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35081558b0 0x7f3508144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:23:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.214+0000 7f34fffff700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f350814f0c0 0x7f35080c3bf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f34f00090b0 tx=0x7f34f0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.214+0000 7f34feffd700 1 -- 172.21.15.23:0/2674917940 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f0004590 con 0x7f350814f0c0 2023-06-22T08:23:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.214+0000 7f34feffd700 1 -- 172.21.15.23:0/2674917940 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f001f070 con 0x7f350814f0c0 2023-06-22T08:23:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.214+0000 7f34feffd700 1 -- 172.21.15.23:0/2674917940 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f00038e0 con 0x7f350814f0c0 2023-06-22T08:23:35.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.214+0000 7f350c9ff700 1 -- 172.21.15.23:0/2674917940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f00094a0 con 0x7f350814f0c0 2023-06-22T08:23:35.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.214+0000 7f350c9ff700 1 -- 172.21.15.23:0/2674917940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35080c33f0 con 0x7f350814f0c0 2023-06-22T08:23:35.220 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.215+0000 7f350c9ff700 1 -- 172.21.15.23:0/2674917940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35081522b0 con 0x7f350814f0c0 2023-06-22T08:23:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.216+0000 7f34feffd700 1 -- 172.21.15.23:0/2674917940 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f34f0003a40 con 0x7f350814f0c0 2023-06-22T08:23:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.217+0000 7f34feffd700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f34e806ace0 0x7f34e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.217+0000 7f34feffd700 1 -- 172.21.15.23:0/2674917940 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f34f0021070 con 0x7f350814f0c0 2023-06-22T08:23:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.217+0000 7f350659c700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f34e806ace0 0x7f34e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.218+0000 7f350659c700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f34e806ace0 0x7f34e806d190 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f34f8007fd0 tx=0x7f34f8007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.222+0000 7f34feffd700 1 -- 172.21.15.23:0/2674917940 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f35081522b0 con 0x7f350814f0c0 2023-06-22T08:23:35.485 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.482+0000 7f350c9ff700 1 -- 172.21.15.23:0/2674917940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35081522b0 con 0x7f350814f0c0 2023-06-22T08:23:35.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.483+0000 7f34feffd700 1 -- 172.21.15.23:0/2674917940 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+4726 (secure 0 0 0) 0x7f35081522b0 con 0x7f350814f0c0 2023-06-22T08:23:35.486 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:35.487 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:34.706065+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:35.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 -- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f34e806ace0 msgr2=0x7f34e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f34e806ace0 0x7f34e806d190 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f34f8007fd0 tx=0x7f34f8007dc0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.489 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 -- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f350814f0c0 msgr2=0x7f35080c3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f350814f0c0 0x7f35080c3bf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f34f00090b0 tx=0x7f34f0004750 comp rx=0 tx=0).stop 2023-06-22T08:23:35.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 -- 172.21.15.23:0/2674917940 shutdown_connections 2023-06-22T08:23:35.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35081558b0 0x7f3508144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f350806e450 0x7f35081e1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f34e806ace0 0x7f34e806d190 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 --2- 172.21.15.23:0/2674917940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f350814f0c0 0x7f35080c3bf0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 -- 172.21.15.23:0/2674917940 >> 172.21.15.23:0/2674917940 conn(0x7f35080ca3f0 msgr2=0x7f3508070420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:35.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 -- 172.21.15.23:0/2674917940 shutdown_connections 2023-06-22T08:23:35.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.486+0000 7f350c9ff700 1 -- 172.21.15.23:0/2674917940 wait complete. 2023-06-22T08:23:35.492 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 184 2023-06-22T08:23:35.507 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-06-22T08:23:35.507 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:23:35.508 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:35.715 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.713+0000 7fde809ff700 1 Processor -- start 2023-06-22T08:23:35.716 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.713+0000 7fde809ff700 1 -- start start 2023-06-22T08:23:35.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.714+0000 7fde809ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fde7c152ce0 0x7fde7c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.714+0000 7fde809ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde7c0c9fd0 0x7fde7c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.714+0000 7fde809ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c15c370 con 0x7fde7c158880 2023-06-22T08:23:35.717 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.714+0000 7fde809ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7c1530b0 con 0x7fde7c152ce0 2023-06-22T08:23:35.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.714+0000 7fde809ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0cf6f0 con 0x7fde7c0c9fd0 2023-06-22T08:23:35.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.715+0000 7fde7a59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fde7c152ce0 0x7fde7c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.715+0000 7fde79d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde7c0c9fd0 0x7fde7c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.715+0000 7fde7ad9d700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fde7c158880 0x7fde7c158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:56090/0 (socket says 172.21.15.23:56090) 2023-06-22T08:23:35.718 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.715+0000 7fde7ad9d700 1 -- 172.21.15.23:0/2386822714 learned_addr learned my addr 172.21.15.23:0/2386822714 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:35.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.715+0000 7fde7a59c700 1 -- 172.21.15.23:0/2386822714 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde7c0c9fd0 msgr2=0x7fde7c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:35.719 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.715+0000 7fde7a59c700 1 --2- 172.21.15.23:0/2386822714 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde7c0c9fd0 0x7fde7c15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.715+0000 7fde7a59c700 1 -- 172.21.15.23:0/2386822714 >> v1:172.21.15.67:6789/0 conn(0x7fde7c158880 legacy=0x7fde7c158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.715+0000 7fde7a59c700 1 -- 172.21.15.23:0/2386822714 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c15bda0 con 0x7fde7c152ce0 2023-06-22T08:23:35.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.715+0000 7fde7959a700 1 -- 172.21.15.23:0/2386822714 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517965608 0 0) 0x7fde7c15c370 con 0x7fde7c158880 2023-06-22T08:23:35.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.715+0000 7fde7a59c700 1 --2- 172.21.15.23:0/2386822714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde7c152ce0 0x7fde7c0cf1b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009840 tx=0x7fde6c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fe126d34e0ca58 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:35.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde7959a700 1 -- 172.21.15.23:0/2386822714 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c011830 con 0x7fde7c152ce0 2023-06-22T08:23:35.720 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde7959a700 1 -- 172.21.15.23:0/2386822714 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6c011990 con 0x7fde7c152ce0 2023-06-22T08:23:35.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde7959a700 1 -- 172.21.15.23:0/2386822714 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c011d20 con 0x7fde7c152ce0 2023-06-22T08:23:35.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde809ff700 1 -- 172.21.15.23:0/2386822714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde7c152ce0 msgr2=0x7fde7c0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde809ff700 1 --2- 172.21.15.23:0/2386822714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde7c152ce0 0x7fde7c0cf1b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009840 tx=0x7fde6c009b50 comp rx=0 tx=0).stop 2023-06-22T08:23:35.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde809ff700 1 -- 172.21.15.23:0/2386822714 shutdown_connections 2023-06-22T08:23:35.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde809ff700 1 --2- 172.21.15.23:0/2386822714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde7c152ce0 0x7fde7c0cf1b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde809ff700 1 --2- 172.21.15.23:0/2386822714 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde7c0c9fd0 0x7fde7c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde809ff700 1 -- 172.21.15.23:0/2386822714 >> 172.21.15.23:0/2386822714 conn(0x7fde7c06ae30 msgr2=0x7fde7c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:35.721 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde809ff700 1 -- 172.21.15.23:0/2386822714 shutdown_connections 2023-06-22T08:23:35.722 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.716+0000 7fde809ff700 1 -- 172.21.15.23:0/2386822714 wait complete. 2023-06-22T08:23:35.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.718+0000 7fde809ff700 1 Processor -- start 2023-06-22T08:23:35.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.719+0000 7fde809ff700 1 -- start start 2023-06-22T08:23:35.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.719+0000 7fde809ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c0c9fd0 0x7fde7c147940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.723 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.719+0000 7fde809ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde7c152ce0 0x7fde7c150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.719+0000 7fde809ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde7c158880 0x7fde7c150940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde809ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c151b20 con 0x7fde7c158880 2023-06-22T08:23:35.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde809ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c151e70 con 0x7fde7c0c9fd0 2023-06-22T08:23:35.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde809ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c15a7b0 con 0x7fde7c152ce0 2023-06-22T08:23:35.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde7a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c0c9fd0 0x7fde7c147940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde7a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c0c9fd0 0x7fde7c147940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:59776/0 (socket says 172.21.15.23:59776) 2023-06-22T08:23:35.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde7a59c700 1 -- 172.21.15.23:0/3270048729 learned_addr learned my addr 172.21.15.23:0/3270048729 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:35.724 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde79d9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde7c152ce0 0x7fde7c150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde7ad9d700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde7c158880 0x7fde7c150940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde7ad9d700 1 -- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde7c152ce0 msgr2=0x7fde7c150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde7ad9d700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde7c152ce0 0x7fde7c150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde7ad9d700 1 -- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c0c9fd0 msgr2=0x7fde7c147940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde7ad9d700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c0c9fd0 0x7fde7c147940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde7ad9d700 1 -- 172.21.15.23:0/3270048729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0094a0 con 0x7fde7c158880 2023-06-22T08:23:35.725 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.720+0000 7fde7ad9d700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde7c158880 0x7fde7c150940 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fde7400caf0 tx=0x7fde7400ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:35.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.721+0000 7fde6affd700 1 -- 172.21.15.23:0/3270048729 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7400a9e0 con 0x7fde7c158880 2023-06-22T08:23:35.726 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.721+0000 7fde6affd700 1 -- 172.21.15.23:0/3270048729 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7400ab60 con 0x7fde7c158880 2023-06-22T08:23:35.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.721+0000 7fde6affd700 1 -- 172.21.15.23:0/3270048729 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde740070d0 con 0x7fde7c158880 2023-06-22T08:23:35.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.721+0000 7fde809ff700 1 -- 172.21.15.23:0/3270048729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c159ba0 con 0x7fde7c158880 2023-06-22T08:23:35.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.721+0000 7fde809ff700 1 -- 172.21.15.23:0/3270048729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c15a160 con 0x7fde7c158880 2023-06-22T08:23:35.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.723+0000 7fde809ff700 1 -- 172.21.15.23:0/3270048729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7c04ea20 con 0x7fde7c158880 2023-06-22T08:23:35.727 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.724+0000 7fde6affd700 1 -- 172.21.15.23:0/3270048729 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fde74007230 con 0x7fde7c158880 2023-06-22T08:23:35.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.724+0000 7fde6affd700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde5c06ad10 0x7fde5c06d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:35.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.725+0000 7fde6affd700 1 -- 172.21.15.23:0/3270048729 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7fde740895a0 con 0x7fde7c158880 2023-06-22T08:23:35.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.725+0000 7fde7a59c700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde5c06ad10 0x7fde5c06d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:35.728 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.726+0000 7fde7a59c700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde5c06ad10 0x7fde5c06d1c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fde6c000c00 tx=0x7fde6c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:35.731 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.729+0000 7fde6affd700 1 -- 172.21.15.23:0/3270048729 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fde74053dd0 con 0x7fde7c158880 2023-06-22T08:23:35.986 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.983+0000 7fde809ff700 1 -- 172.21.15.23:0/3270048729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde7c15a8f0 con 0x7fde7c158880 2023-06-22T08:23:35.987 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.984+0000 7fde6affd700 1 -- 172.21.15.23:0/3270048729 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+4726 (secure 0 0 0) 0x7fde74018070 con 0x7fde7c158880 2023-06-22T08:23:35.987 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:35.987 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:34.706065+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:35.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 -- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde5c06ad10 msgr2=0x7fde5c06d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde5c06ad10 0x7fde5c06d1c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fde6c000c00 tx=0x7fde6c008030 comp rx=0 tx=0).stop 2023-06-22T08:23:35.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 -- 172.21.15.23:0/3270048729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde7c158880 msgr2=0x7fde7c150940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:35.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde7c158880 0x7fde7c150940 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fde7400caf0 tx=0x7fde7400ce00 comp rx=0 tx=0).stop 2023-06-22T08:23:35.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 -- 172.21.15.23:0/3270048729 shutdown_connections 2023-06-22T08:23:35.990 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fde7c158880 0x7fde7c150940 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fde5c06ad10 0x7fde5c06d1c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde7c0c9fd0 0x7fde7c147940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 --2- 172.21.15.23:0/3270048729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fde7c152ce0 0x7fde7c150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:35.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 -- 172.21.15.23:0/3270048729 >> 172.21.15.23:0/3270048729 conn(0x7fde7c06ae30 msgr2=0x7fde7c0cb820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:35.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 -- 172.21.15.23:0/3270048729 shutdown_connections 2023-06-22T08:23:35.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:35.986+0000 7fde809ff700 1 -- 172.21.15.23:0/3270048729 wait complete. 2023-06-22T08:23:35.992 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 184 2023-06-22T08:23:36.008 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 184, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 184, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 36 }, "gid_6616": { "addr": "172.21.15.67:6837/1855579717", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1855579717, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1855579717, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6616, "incarnation": 162, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_6705": { "addr": "172.21.15.23:6835/3148174349", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 3148174349, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 3148174349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6705, "incarnation": 184, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:replay", "state_seq": 1 }, "gid_6730": { "addr": "172.21.15.23:6834/456927633", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 456927633, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 456927633, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6730, "incarnation": 172, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 110, "max_file_size": 1099511627776, "max_mds": 4, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:23:34.706065+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 6616, "mds_1": 6730, "mds_2": 6313, "mds_3": 6705 } } } ], "standbys": [] } 2023-06-22T08:23:38.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:38.222 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.220+0000 7ff7667ff700 1 Processor -- start 2023-06-22T08:23:38.223 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.220+0000 7ff7667ff700 1 -- start start 2023-06-22T08:23:38.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.221+0000 7ff7667ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff7600cb6b0 0x7ff760069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:38.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.221+0000 7ff7667ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff7600c5740 0x7ff7601593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:38.224 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.221+0000 7ff7667ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff760158f60 con 0x7ff7600cba80 2023-06-22T08:23:38.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.221+0000 7ff7667ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76006bd20 con 0x7ff7600c5740 2023-06-22T08:23:38.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.221+0000 7ff7667ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7600cc450 con 0x7ff7600cb6b0 2023-06-22T08:23:38.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.222+0000 7ff75f7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff7600c5740 0x7ff7601593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:38.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.222+0000 7ff764d9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff7600cba80 0x7ff760152700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40834/0 (socket says 172.21.15.23:40834) 2023-06-22T08:23:38.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.222+0000 7ff764d9c700 1 -- 172.21.15.23:0/2500740816 learned_addr learned my addr 172.21.15.23:0/2500740816 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:38.225 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.222+0000 7ff75ffff700 1 --2- 172.21.15.23:0/2500740816 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff7600cb6b0 0x7ff760069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:38.228 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.222+0000 7ff75effd700 1 -- 172.21.15.23:0/2500740816 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707964045 0 0) 0x7ff760158f60 con 0x7ff7600cba80 2023-06-22T08:23:38.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.222+0000 7ff75effd700 1 -- 172.21.15.23:0/2500740816 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7480035b0 con 0x7ff7600cba80 2023-06-22T08:23:38.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.222+0000 7ff75f7fe700 1 -- 172.21.15.23:0/2500740816 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff7600cb6b0 msgr2=0x7ff760069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:38.229 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.222+0000 7ff75f7fe700 1 --2- 172.21.15.23:0/2500740816 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff7600cb6b0 0x7ff760069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:38.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.222+0000 7ff75f7fe700 1 -- 172.21.15.23:0/2500740816 >> v1:172.21.15.23:6789/0 conn(0x7ff7600cba80 legacy=0x7ff760152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:38.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.222+0000 7ff75f7fe700 1 -- 172.21.15.23:0/2500740816 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff760158980 con 0x7ff7600c5740 2023-06-22T08:23:38.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.223+0000 7ff75f7fe700 1 --2- 172.21.15.23:0/2500740816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600c5740 0x7ff7601593f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff754009840 tx=0x7ff754009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70fb40353aa4a6b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:38.230 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.223+0000 7ff75effd700 1 -- 172.21.15.23:0/2500740816 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754011830 con 0x7ff7600c5740 2023-06-22T08:23:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.223+0000 7ff75effd700 1 -- 172.21.15.23:0/2500740816 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff754011990 con 0x7ff7600c5740 2023-06-22T08:23:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.223+0000 7ff75effd700 1 -- 172.21.15.23:0/2500740816 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754011d20 con 0x7ff7600c5740 2023-06-22T08:23:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.223+0000 7ff7667ff700 1 -- 172.21.15.23:0/2500740816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600c5740 msgr2=0x7ff7601593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.223+0000 7ff7667ff700 1 --2- 172.21.15.23:0/2500740816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600c5740 0x7ff7601593f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff754009840 tx=0x7ff754009b50 comp rx=0 tx=0).stop 2023-06-22T08:23:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.224+0000 7ff7667ff700 1 -- 172.21.15.23:0/2500740816 shutdown_connections 2023-06-22T08:23:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.224+0000 7ff7667ff700 1 --2- 172.21.15.23:0/2500740816 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff7600cb6b0 0x7ff760069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.224+0000 7ff7667ff700 1 --2- 172.21.15.23:0/2500740816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600c5740 0x7ff7601593f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:38.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.224+0000 7ff7667ff700 1 -- 172.21.15.23:0/2500740816 >> 172.21.15.23:0/2500740816 conn(0x7ff760070d80 msgr2=0x7ff760071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:38.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.224+0000 7ff7667ff700 1 -- 172.21.15.23:0/2500740816 shutdown_connections 2023-06-22T08:23:38.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.224+0000 7ff7667ff700 1 -- 172.21.15.23:0/2500740816 wait complete. 2023-06-22T08:23:38.233 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.225+0000 7ff7667ff700 1 Processor -- start 2023-06-22T08:23:38.234 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.226+0000 7ff7667ff700 1 -- start start 2023-06-22T08:23:38.236 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.226+0000 7ff7667ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600cb6b0 0x7ff7600c4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:38.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.226+0000 7ff7667ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600cba80 0x7ff76014a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:38.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.226+0000 7ff7667ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff7601583d0 0x7ff760157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:38.237 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.226+0000 7ff7667ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7601ef1c0 con 0x7ff7600cba80 2023-06-22T08:23:38.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.226+0000 7ff7667ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7600c42f0 con 0x7ff7600cb6b0 2023-06-22T08:23:38.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.226+0000 7ff7667ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff760149e70 con 0x7ff7601583d0 2023-06-22T08:23:38.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600cba80 0x7ff76014a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:38.238 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600cb6b0 0x7ff7600c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:38.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff764d9c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff7601583d0 0x7ff760157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:38.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600cb6b0 0x7ff7600c4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:50592/0 (socket says 172.21.15.23:50592) 2023-06-22T08:23:38.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75ffff700 1 -- 172.21.15.23:0/2217631696 learned_addr learned my addr 172.21.15.23:0/2217631696 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:38.239 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600cba80 0x7ff76014a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35456/0 (socket says 172.21.15.23:35456) 2023-06-22T08:23:38.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75f7fe700 1 -- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff7601583d0 msgr2=0x7ff760157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:38.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75f7fe700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff7601583d0 0x7ff760157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:38.240 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75f7fe700 1 -- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600cb6b0 msgr2=0x7ff7600c4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:38.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75f7fe700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600cb6b0 0x7ff7600c4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:38.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75f7fe700 1 -- 172.21.15.23:0/2217631696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7540094a0 con 0x7ff7600cba80 2023-06-22T08:23:38.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.227+0000 7ff75f7fe700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600cba80 0x7ff76014a5a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff754016000 tx=0x7ff754004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:38.241 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.228+0000 7ff75cff9700 1 -- 172.21.15.23:0/2217631696 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754004590 con 0x7ff7600cba80 2023-06-22T08:23:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.228+0000 7ff75cff9700 1 -- 172.21.15.23:0/2217631696 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75401e070 con 0x7ff7600cba80 2023-06-22T08:23:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.228+0000 7ff7667ff700 1 -- 172.21.15.23:0/2217631696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7601eec20 con 0x7ff7600cba80 2023-06-22T08:23:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.228+0000 7ff75cff9700 1 -- 172.21.15.23:0/2217631696 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7540038e0 con 0x7ff7600cba80 2023-06-22T08:23:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.228+0000 7ff7667ff700 1 -- 172.21.15.23:0/2217631696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7600c3ca0 con 0x7ff7600cba80 2023-06-22T08:23:38.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.229+0000 7ff7667ff700 1 -- 172.21.15.23:0/2217631696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7600c3b00 con 0x7ff7600cba80 2023-06-22T08:23:38.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.230+0000 7ff75cff9700 1 -- 172.21.15.23:0/2217631696 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff754003a40 con 0x7ff7600cba80 2023-06-22T08:23:38.243 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.231+0000 7ff75cff9700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff74c06ace0 0x7ff74c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:38.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.231+0000 7ff75cff9700 1 -- 172.21.15.23:0/2217631696 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7ff75408ab60 con 0x7ff7600cba80 2023-06-22T08:23:38.244 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.231+0000 7ff75ffff700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff74c06ace0 0x7ff74c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:38.245 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.232+0000 7ff75ffff700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff74c06ace0 0x7ff74c06d190 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff750009ca0 tx=0x7ff750009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.235+0000 7ff75cff9700 1 -- 172.21.15.23:0/2217631696 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff7600c3b00 con 0x7ff7600cba80 2023-06-22T08:23:38.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.490+0000 7ff7667ff700 1 -- 172.21.15.23:0/2217631696 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7600c3b00 con 0x7ff7600cba80 2023-06-22T08:23:38.494 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.490+0000 7ff75cff9700 1 -- 172.21.15.23:0/2217631696 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+4726 (secure 0 0 0) 0x7ff7600c3b00 con 0x7ff7600cba80 2023-06-22T08:23:38.494 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:38.495 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:34.706065+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:38.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 -- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff74c06ace0 msgr2=0x7ff74c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:38.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff74c06ace0 0x7ff74c06d190 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff750009ca0 tx=0x7ff750009210 comp rx=0 tx=0).stop 2023-06-22T08:23:38.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 -- 172.21.15.23:0/2217631696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600cba80 msgr2=0x7ff76014a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:38.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600cba80 0x7ff76014a5a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff754016000 tx=0x7ff754004750 comp rx=0 tx=0).stop 2023-06-22T08:23:38.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 -- 172.21.15.23:0/2217631696 shutdown_connections 2023-06-22T08:23:38.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff7601583d0 0x7ff760157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:38.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff74c06ace0 0x7ff74c06d190 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:38.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7600cb6b0 0x7ff7600c4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:38.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 --2- 172.21.15.23:0/2217631696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600cba80 0x7ff76014a5a0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:38.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 -- 172.21.15.23:0/2217631696 >> 172.21.15.23:0/2217631696 conn(0x7ff760070d80 msgr2=0x7ff760151390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:38.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 -- 172.21.15.23:0/2217631696 shutdown_connections 2023-06-22T08:23:38.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:38.493+0000 7ff7667ff700 1 -- 172.21.15.23:0/2217631696 wait complete. 2023-06-22T08:23:38.500 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 184 2023-06-22T08:23:38.515 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:23:39.324 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~197s 2023-06-22T08:23:40.517 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:40.748 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.746+0000 7fa9abfff700 1 Processor -- start 2023-06-22T08:23:40.749 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.746+0000 7fa9abfff700 1 -- start start 2023-06-22T08:23:40.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.748+0000 7fa9abfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa9ac06a120 0x7fa9ac15c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:40.751 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.748+0000 7fa9abfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9ac06f390 0x7fa9ac152550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:40.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9abfff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9ac072010 con 0x7fa9ac06a4f0 2023-06-22T08:23:40.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9abfff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9ac06e860 con 0x7fa9ac06a120 2023-06-22T08:23:40.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9abfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0c7280 con 0x7fa9ac06f390 2023-06-22T08:23:40.752 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9aaffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa9ac06a120 0x7fa9ac15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9aaffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa9ac06a120 0x7fa9ac15c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35462/0 (socket says 172.21.15.23:35462) 2023-06-22T08:23:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9aaffd700 1 -- 172.21.15.23:0/3588261239 learned_addr learned my addr 172.21.15.23:0/3588261239 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9ab7fe700 1 --2- 172.21.15.23:0/3588261239 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9ac06f390 0x7fa9ac152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9aaffd700 1 -- 172.21.15.23:0/3588261239 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9ac06f390 msgr2=0x7fa9ac152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9aaffd700 1 --2- 172.21.15.23:0/3588261239 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9ac06f390 0x7fa9ac152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9aaffd700 1 -- 172.21.15.23:0/3588261239 >> v1:172.21.15.67:6789/0 conn(0x7fa9ac06a4f0 legacy=0x7fa9ac06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.749+0000 7fa9aaffd700 1 -- 172.21.15.23:0/3588261239 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac15cd70 con 0x7fa9ac06a120 2023-06-22T08:23:40.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.750+0000 7fa9aaffd700 1 --2- 172.21.15.23:0/3588261239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 0x7fa9ac15c830 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0098f0 tx=0x7fa99c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0ce1641f0ff0ed server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:40.754 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.750+0000 7fa9a9ffb700 1 -- 172.21.15.23:0/3588261239 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c00b8a0 con 0x7fa9ac06a120 2023-06-22T08:23:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.750+0000 7fa9a9ffb700 1 -- 172.21.15.23:0/3588261239 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99c00ba00 con 0x7fa9ac06a120 2023-06-22T08:23:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.750+0000 7fa9a9ffb700 1 -- 172.21.15.23:0/3588261239 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c00bd90 con 0x7fa9ac06a120 2023-06-22T08:23:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.751+0000 7fa9abfff700 1 -- 172.21.15.23:0/3588261239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 msgr2=0x7fa9ac15c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.751+0000 7fa9abfff700 1 --2- 172.21.15.23:0/3588261239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 0x7fa9ac15c830 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0098f0 tx=0x7fa99c009c00 comp rx=0 tx=0).stop 2023-06-22T08:23:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.751+0000 7fa9abfff700 1 -- 172.21.15.23:0/3588261239 shutdown_connections 2023-06-22T08:23:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.751+0000 7fa9abfff700 1 --2- 172.21.15.23:0/3588261239 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9ac06f390 0x7fa9ac152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.751+0000 7fa9abfff700 1 --2- 172.21.15.23:0/3588261239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 0x7fa9ac15c830 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.751+0000 7fa9abfff700 1 -- 172.21.15.23:0/3588261239 >> 172.21.15.23:0/3588261239 conn(0x7fa9ac06afc0 msgr2=0x7fa9ac06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.751+0000 7fa9abfff700 1 -- 172.21.15.23:0/3588261239 shutdown_connections 2023-06-22T08:23:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.751+0000 7fa9abfff700 1 -- 172.21.15.23:0/3588261239 wait complete. 2023-06-22T08:23:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.754+0000 7fa9abfff700 1 Processor -- start 2023-06-22T08:23:40.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.754+0000 7fa9abfff700 1 -- start start 2023-06-22T08:23:40.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.755+0000 7fa9abfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 0x7fa9ac157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:40.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.755+0000 7fa9abfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9ac06a4f0 0x7fa9ac0c2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:40.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.755+0000 7fa9abfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9ac06f390 0x7fa9ac0c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:40.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.755+0000 7fa9abfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac0c4570 con 0x7fa9ac06a120 2023-06-22T08:23:40.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.755+0000 7fa9abfff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac141810 con 0x7fa9ac06f390 2023-06-22T08:23:40.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.755+0000 7fa9abfff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ac141af0 con 0x7fa9ac06a4f0 2023-06-22T08:23:40.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.755+0000 7fa9aaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 0x7fa9ac157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:40.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.756+0000 7fa9aaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 0x7fa9ac157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35464/0 (socket says 172.21.15.23:35464) 2023-06-22T08:23:40.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.756+0000 7fa9ab7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9ac06f390 0x7fa9ac0c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:40.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.756+0000 7fa9aaffd700 1 -- 172.21.15.23:0/3370908413 learned_addr learned my addr 172.21.15.23:0/3370908413 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:40.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.756+0000 7fa9aa7fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9ac06a4f0 0x7fa9ac0c2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:40.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.756+0000 7fa9aaffd700 1 -- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9ac06a4f0 msgr2=0x7fa9ac0c2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:40.760 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.756+0000 7fa9aaffd700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9ac06a4f0 0x7fa9ac0c2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:40.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.756+0000 7fa9aaffd700 1 -- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9ac06f390 msgr2=0x7fa9ac0c3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:40.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.756+0000 7fa9aaffd700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9ac06f390 0x7fa9ac0c3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:40.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.756+0000 7fa9aaffd700 1 -- 172.21.15.23:0/3370908413 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c009550 con 0x7fa9ac06a120 2023-06-22T08:23:40.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.756+0000 7fa9aaffd700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 0x7fa9ac157920 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa99c000c00 tx=0x7fa99c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:40.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.757+0000 7fa9937fe700 1 -- 172.21.15.23:0/3370908413 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c004300 con 0x7fa9ac06a120 2023-06-22T08:23:40.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.757+0000 7fa9937fe700 1 -- 172.21.15.23:0/3370908413 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99c01f070 con 0x7fa9ac06a120 2023-06-22T08:23:40.761 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.757+0000 7fa9937fe700 1 -- 172.21.15.23:0/3370908413 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99c0036a0 con 0x7fa9ac06a120 2023-06-22T08:23:40.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.757+0000 7fa9abfff700 1 -- 172.21.15.23:0/3370908413 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ac141dd0 con 0x7fa9ac06a120 2023-06-22T08:23:40.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.757+0000 7fa9abfff700 1 -- 172.21.15.23:0/3370908413 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ac142390 con 0x7fa9ac06a120 2023-06-22T08:23:40.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.758+0000 7fa9abfff700 1 -- 172.21.15.23:0/3370908413 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ac061b00 con 0x7fa9ac06a120 2023-06-22T08:23:40.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.759+0000 7fa9937fe700 1 -- 172.21.15.23:0/3370908413 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fa99c021030 con 0x7fa9ac06a120 2023-06-22T08:23:40.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.760+0000 7fa9937fe700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa99806ace0 0x7fa99806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:40.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.760+0000 7fa9937fe700 1 -- 172.21.15.23:0/3370908413 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7fa99c08a9f0 con 0x7fa9ac06a120 2023-06-22T08:23:40.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.760+0000 7fa9aa7fc700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa99806ace0 0x7fa99806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:40.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.761+0000 7fa9aa7fc700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa99806ace0 0x7fa99806d190 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac140be0 tx=0x7fa9a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:40.767 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:40.765+0000 7fa9937fe700 1 -- 172.21.15.23:0/3370908413 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fa99c0552d0 con 0x7fa9ac06a120 2023-06-22T08:23:41.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.050+0000 7fa9abfff700 1 -- 172.21.15.23:0/3370908413 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9ac156250 con 0x7fa9ac06a120 2023-06-22T08:23:41.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.051+0000 7fa9937fe700 1 -- 172.21.15.23:0/3370908413 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+4726 (secure 0 0 0) 0x7fa99c02a090 con 0x7fa9ac06a120 2023-06-22T08:23:41.055 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:41.055 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:34.706065+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:replay","state_seq":1,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:41.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.054+0000 7fa9abfff700 1 -- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa99806ace0 msgr2=0x7fa99806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:41.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.054+0000 7fa9abfff700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa99806ace0 0x7fa99806d190 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac140be0 tx=0x7fa9a0009040 comp rx=0 tx=0).stop 2023-06-22T08:23:41.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.054+0000 7fa9abfff700 1 -- 172.21.15.23:0/3370908413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 msgr2=0x7fa9ac157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:41.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.054+0000 7fa9abfff700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 0x7fa9ac157920 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa99c000c00 tx=0x7fa99c004750 comp rx=0 tx=0).stop 2023-06-22T08:23:41.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.055+0000 7fa9abfff700 1 -- 172.21.15.23:0/3370908413 shutdown_connections 2023-06-22T08:23:41.058 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.055+0000 7fa9abfff700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fa99806ace0 0x7fa99806d190 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:41.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.055+0000 7fa9abfff700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9ac06f390 0x7fa9ac0c3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:41.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.055+0000 7fa9abfff700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ac06a120 0x7fa9ac157920 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:41.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.055+0000 7fa9abfff700 1 --2- 172.21.15.23:0/3370908413 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa9ac06a4f0 0x7fa9ac0c2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:41.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.055+0000 7fa9abfff700 1 -- 172.21.15.23:0/3370908413 >> 172.21.15.23:0/3370908413 conn(0x7fa9ac06afc0 msgr2=0x7fa9ac0c9620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:41.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.055+0000 7fa9abfff700 1 -- 172.21.15.23:0/3370908413 shutdown_connections 2023-06-22T08:23:41.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:41.055+0000 7fa9abfff700 1 -- 172.21.15.23:0/3370908413 wait complete. 2023-06-22T08:23:41.060 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 184 2023-06-22T08:23:41.080 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:23:43.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:43.302 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.300+0000 7ff35bbff700 1 Processor -- start 2023-06-22T08:23:43.303 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.300+0000 7ff35bbff700 1 -- start start 2023-06-22T08:23:43.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.301+0000 7ff35bbff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff3540c9fd0 0x7ff3540cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:43.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.301+0000 7ff35bbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c40b0 0x7ff3540c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:43.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.301+0000 7ff35bbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35415c370 con 0x7ff354152ce0 2023-06-22T08:23:43.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.301+0000 7ff35bbff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3541530b0 con 0x7ff3540c9fd0 2023-06-22T08:23:43.304 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.301+0000 7ff35bbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540cf6f0 con 0x7ff3540c40b0 2023-06-22T08:23:43.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff35919a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff3540c9fd0 0x7ff3540cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:43.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff35999b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff354152ce0 0x7ff35415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:40200/0 (socket says 172.21.15.23:40200) 2023-06-22T08:23:43.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff35999b700 1 -- 172.21.15.23:0/1585337729 learned_addr learned my addr 172.21.15.23:0/1585337729 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:43.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff35a19c700 1 --2- 172.21.15.23:0/1585337729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c40b0 0x7ff3540c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:43.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff358999700 1 -- 172.21.15.23:0/1585337729 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527732350 0 0) 0x7ff35415c370 con 0x7ff354152ce0 2023-06-22T08:23:43.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff358999700 1 -- 172.21.15.23:0/1585337729 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3400035b0 con 0x7ff354152ce0 2023-06-22T08:23:43.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff35919a700 1 -- 172.21.15.23:0/1585337729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c40b0 msgr2=0x7ff3540c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:43.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff35919a700 1 --2- 172.21.15.23:0/1585337729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c40b0 0x7ff3540c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:43.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff35919a700 1 -- 172.21.15.23:0/1585337729 >> v1:172.21.15.67:6789/0 conn(0x7ff354152ce0 legacy=0x7ff35415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:43.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff35919a700 1 -- 172.21.15.23:0/1585337729 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35415bda0 con 0x7ff3540c9fd0 2023-06-22T08:23:43.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff35a19c700 1 --2- 172.21.15.23:0/1585337729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c40b0 0x7ff3540c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:23:43.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.302+0000 7ff35919a700 1 --2- 172.21.15.23:0/1585337729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540c9fd0 0x7ff3540cf1b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff344009840 tx=0x7ff344009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aeb17f49537a916 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:43.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.303+0000 7ff358999700 1 -- 172.21.15.23:0/1585337729 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344011830 con 0x7ff3540c9fd0 2023-06-22T08:23:43.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.303+0000 7ff358999700 1 -- 172.21.15.23:0/1585337729 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff344011990 con 0x7ff3540c9fd0 2023-06-22T08:23:43.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.303+0000 7ff358999700 1 -- 172.21.15.23:0/1585337729 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344011d00 con 0x7ff3540c9fd0 2023-06-22T08:23:43.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.303+0000 7ff35bbff700 1 -- 172.21.15.23:0/1585337729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540c9fd0 msgr2=0x7ff3540cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:43.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.303+0000 7ff35bbff700 1 --2- 172.21.15.23:0/1585337729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540c9fd0 0x7ff3540cf1b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff344009840 tx=0x7ff344009b50 comp rx=0 tx=0).stop 2023-06-22T08:23:43.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.303+0000 7ff35bbff700 1 -- 172.21.15.23:0/1585337729 shutdown_connections 2023-06-22T08:23:43.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.303+0000 7ff35bbff700 1 --2- 172.21.15.23:0/1585337729 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c40b0 0x7ff3540c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:43.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.303+0000 7ff35bbff700 1 --2- 172.21.15.23:0/1585337729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540c9fd0 0x7ff3540cf1b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:43.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.303+0000 7ff35bbff700 1 -- 172.21.15.23:0/1585337729 >> 172.21.15.23:0/1585337729 conn(0x7ff35406ae30 msgr2=0x7ff35406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:43.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.303+0000 7ff35bbff700 1 -- 172.21.15.23:0/1585337729 shutdown_connections 2023-06-22T08:23:43.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.304+0000 7ff35bbff700 1 -- 172.21.15.23:0/1585337729 wait complete. 2023-06-22T08:23:43.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.305+0000 7ff35bbff700 1 Processor -- start 2023-06-22T08:23:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.305+0000 7ff35bbff700 1 -- start start 2023-06-22T08:23:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.306+0000 7ff35bbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540c40b0 0x7ff35415aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.306+0000 7ff35bbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c9fd0 0x7ff3541f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.306+0000 7ff35bbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff354152ce0 0x7ff3541471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.306+0000 7ff35bbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3541f1cf0 con 0x7ff3540c40b0 2023-06-22T08:23:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.306+0000 7ff35bbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3541f19e0 con 0x7ff354152ce0 2023-06-22T08:23:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.306+0000 7ff35bbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540b2360 con 0x7ff3540c9fd0 2023-06-22T08:23:43.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35919a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c9fd0 0x7ff3541f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:43.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35999b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540c40b0 0x7ff35415aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:43.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35919a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c9fd0 0x7ff3541f2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49446/0 (socket says 172.21.15.23:49446) 2023-06-22T08:23:43.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35919a700 1 -- 172.21.15.23:0/1106472768 learned_addr learned my addr 172.21.15.23:0/1106472768 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:43.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35a19c700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff354152ce0 0x7ff3541471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:43.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35919a700 1 -- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff354152ce0 msgr2=0x7ff3541471b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:43.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35919a700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff354152ce0 0x7ff3541471b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:43.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35919a700 1 -- 172.21.15.23:0/1106472768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540c40b0 msgr2=0x7ff35415aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:43.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35919a700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540c40b0 0x7ff35415aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:43.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35919a700 1 -- 172.21.15.23:0/1106472768 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c009a70 con 0x7ff3540c9fd0 2023-06-22T08:23:43.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35999b700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540c40b0 0x7ff35415aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:23:43.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.307+0000 7ff35919a700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c9fd0 0x7ff3541f2110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff3440090b0 tx=0x7ff344004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:43.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.308+0000 7ff34a7fc700 1 -- 172.21.15.23:0/1106472768 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344004550 con 0x7ff3540c9fd0 2023-06-22T08:23:43.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.308+0000 7ff34a7fc700 1 -- 172.21.15.23:0/1106472768 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34401f070 con 0x7ff3540c9fd0 2023-06-22T08:23:43.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.308+0000 7ff35bbff700 1 -- 172.21.15.23:0/1106472768 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3440094a0 con 0x7ff3540c9fd0 2023-06-22T08:23:43.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.308+0000 7ff34a7fc700 1 -- 172.21.15.23:0/1106472768 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3440038a0 con 0x7ff3540c9fd0 2023-06-22T08:23:43.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.308+0000 7ff35bbff700 1 -- 172.21.15.23:0/1106472768 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3541e1520 con 0x7ff3540c9fd0 2023-06-22T08:23:43.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.309+0000 7ff35bbff700 1 -- 172.21.15.23:0/1106472768 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35404ea20 con 0x7ff3540c9fd0 2023-06-22T08:23:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.310+0000 7ff34a7fc700 1 -- 172.21.15.23:0/1106472768 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff344003a00 con 0x7ff3540c9fd0 2023-06-22T08:23:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.311+0000 7ff34a7fc700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff33806ad10 0x7ff33806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.311+0000 7ff34a7fc700 1 -- 172.21.15.23:0/1106472768 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7ff344021070 con 0x7ff3540c9fd0 2023-06-22T08:23:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.311+0000 7ff35999b700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff33806ad10 0x7ff33806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:43.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.312+0000 7ff35999b700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff33806ad10 0x7ff33806d1c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff350007fd0 tx=0x7ff350009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:43.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.316+0000 7ff34a7fc700 1 -- 172.21.15.23:0/1106472768 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff344055960 con 0x7ff3540c9fd0 2023-06-22T08:23:43.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.569+0000 7ff35bbff700 1 -- 172.21.15.23:0/1106472768 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3540b45e0 con 0x7ff3540c9fd0 2023-06-22T08:23:43.572 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.571+0000 7ff34a7fc700 1 -- 172.21.15.23:0/1106472768 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+4728 (secure 0 0 0) 0x7ff344015090 con 0x7ff3540c9fd0 2023-06-22T08:23:43.573 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:43.573 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:42.750608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:resolve","state_seq":18,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.572+0000 7ff35bbff700 1 -- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff33806ad10 msgr2=0x7ff33806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.572+0000 7ff35bbff700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff33806ad10 0x7ff33806d1c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff350007fd0 tx=0x7ff350009040 comp rx=0 tx=0).stop 2023-06-22T08:23:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.572+0000 7ff35bbff700 1 -- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c9fd0 msgr2=0x7ff3541f2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.572+0000 7ff35bbff700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c9fd0 0x7ff3541f2110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff3440090b0 tx=0x7ff344004750 comp rx=0 tx=0).stop 2023-06-22T08:23:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.572+0000 7ff35bbff700 1 -- 172.21.15.23:0/1106472768 shutdown_connections 2023-06-22T08:23:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.572+0000 7ff35bbff700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff33806ad10 0x7ff33806d1c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.572+0000 7ff35bbff700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff354152ce0 0x7ff3541471b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.572+0000 7ff35bbff700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540c40b0 0x7ff35415aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.572+0000 7ff35bbff700 1 --2- 172.21.15.23:0/1106472768 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff3540c9fd0 0x7ff3541f2110 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:43.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.572+0000 7ff35bbff700 1 -- 172.21.15.23:0/1106472768 >> 172.21.15.23:0/1106472768 conn(0x7ff35406ae30 msgr2=0x7ff3541511b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:43.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.573+0000 7ff35bbff700 1 -- 172.21.15.23:0/1106472768 shutdown_connections 2023-06-22T08:23:43.577 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:43.573+0000 7ff35bbff700 1 -- 172.21.15.23:0/1106472768 wait complete. 2023-06-22T08:23:43.577 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 185 2023-06-22T08:23:43.593 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:23:45.594 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.804+0000 7fb1411ff700 1 Processor -- start 2023-06-22T08:23:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.804+0000 7fb1411ff700 1 -- start start 2023-06-22T08:23:45.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.805+0000 7fb1411ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb13c0c9fd0 0x7fb13c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:45.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.805+0000 7fb1411ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c15c370 con 0x7fb13c152ce0 2023-06-22T08:23:45.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.805+0000 7fb1411ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c15bda0 con 0x7fb13c0c40b0 2023-06-22T08:23:45.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.805+0000 7fb1411ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13c1530b0 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.805+0000 7fb13a59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb13c0c9fd0 0x7fb13c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:45.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.805+0000 7fb13b59e700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb13c0c40b0 0x7fb13c0c4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:40216/0 (socket says 172.21.15.23:40216) 2023-06-22T08:23:45.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.805+0000 7fb13ad9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb13c152ce0 0x7fb13c15c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:40836/0 (socket says 172.21.15.23:40836) 2023-06-22T08:23:45.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.805+0000 7fb13b59e700 1 -- 172.21.15.23:0/3566622403 learned_addr learned my addr 172.21.15.23:0/3566622403 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:45.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.806+0000 7fb139d9b700 1 -- 172.21.15.23:0/3566622403 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246385414 0 0) 0x7fb13c15bda0 con 0x7fb13c0c40b0 2023-06-22T08:23:45.809 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.806+0000 7fb13a59c700 1 -- 172.21.15.23:0/3566622403 >> v1:172.21.15.67:6789/0 conn(0x7fb13c0c40b0 legacy=0x7fb13c0c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:45.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.806+0000 7fb13a59c700 1 -- 172.21.15.23:0/3566622403 >> v1:172.21.15.23:6789/0 conn(0x7fb13c152ce0 legacy=0x7fb13c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:45.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.806+0000 7fb13a59c700 1 -- 172.21.15.23:0/3566622403 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c0c9460 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.806+0000 7fb13a59c700 1 --2- 172.21.15.23:0/3566622403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 0x7fb13c0cf1b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb130009840 tx=0x7fb130009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e47e5be0b6031d server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:45.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.806+0000 7fb139d9b700 1 -- 172.21.15.23:0/3566622403 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130011830 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.806+0000 7fb139d9b700 1 -- 172.21.15.23:0/3566622403 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb130011990 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.810 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.806+0000 7fb139d9b700 1 -- 172.21.15.23:0/3566622403 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb130011d20 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.807+0000 7fb1411ff700 1 -- 172.21.15.23:0/3566622403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 msgr2=0x7fb13c0cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:45.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.807+0000 7fb1411ff700 1 --2- 172.21.15.23:0/3566622403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 0x7fb13c0cf1b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb130009840 tx=0x7fb130009b50 comp rx=0 tx=0).stop 2023-06-22T08:23:45.811 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.807+0000 7fb1411ff700 1 -- 172.21.15.23:0/3566622403 shutdown_connections 2023-06-22T08:23:45.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.807+0000 7fb1411ff700 1 --2- 172.21.15.23:0/3566622403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 0x7fb13c0cf1b0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:45.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.807+0000 7fb1411ff700 1 -- 172.21.15.23:0/3566622403 >> 172.21.15.23:0/3566622403 conn(0x7fb13c06ae30 msgr2=0x7fb13c06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:45.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.807+0000 7fb1411ff700 1 -- 172.21.15.23:0/3566622403 shutdown_connections 2023-06-22T08:23:45.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.807+0000 7fb1411ff700 1 -- 172.21.15.23:0/3566622403 wait complete. 2023-06-22T08:23:45.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.809+0000 7fb1411ff700 1 Processor -- start 2023-06-22T08:23:45.812 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.809+0000 7fb1411ff700 1 -- start start 2023-06-22T08:23:45.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb1411ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb13c0c40b0 0x7fb13c0cac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:45.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb1411ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 0x7fb13c156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:45.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb1411ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c152ce0 0x7fb13c157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:45.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb1411ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c1e3b90 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.814 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb1411ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c156b00 con 0x7fb13c152ce0 2023-06-22T08:23:45.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb1411ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c1e24b0 con 0x7fb13c0c40b0 2023-06-22T08:23:45.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb13a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 0x7fb13c156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:45.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb13a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 0x7fb13c156ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35504/0 (socket says 172.21.15.23:35504) 2023-06-22T08:23:45.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb13a59c700 1 -- 172.21.15.23:0/3554472200 learned_addr learned my addr 172.21.15.23:0/3554472200 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:45.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb13b59e700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c152ce0 0x7fb13c157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:45.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.810+0000 7fb13ad9d700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb13c0c40b0 0x7fb13c0cac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:45.815 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb13a59c700 1 -- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb13c0c40b0 msgr2=0x7fb13c0cac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:45.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb13a59c700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb13c0c40b0 0x7fb13c0cac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:45.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb13a59c700 1 -- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c152ce0 msgr2=0x7fb13c157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:45.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb13a59c700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c152ce0 0x7fb13c157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:45.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb13a59c700 1 -- 172.21.15.23:0/3554472200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1300094a0 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb13a59c700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 0x7fb13c156ec0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb1300090b0 tx=0x7fb130004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:45.816 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb1237fe700 1 -- 172.21.15.23:0/3554472200 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13001a860 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.817 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb1237fe700 1 -- 172.21.15.23:0/3554472200 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13001a9c0 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.817 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb1411ff700 1 -- 172.21.15.23:0/3554472200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c1e21d0 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.817 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb1237fe700 1 -- 172.21.15.23:0/3554472200 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13001ad50 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.817 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.811+0000 7fb1411ff700 1 -- 172.21.15.23:0/3554472200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c1e3540 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.812+0000 7fb1411ff700 1 -- 172.21.15.23:0/3554472200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c1e3d40 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.813+0000 7fb1237fe700 1 -- 172.21.15.23:0/3554472200 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb130022070 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.815+0000 7fb1237fe700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb12806ad10 0x7fb12806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:45.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.815+0000 7fb1237fe700 1 -- 172.21.15.23:0/3554472200 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7fb13008add0 con 0x7fb13c0c9fd0 2023-06-22T08:23:45.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.815+0000 7fb13ad9d700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb12806ad10 0x7fb12806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:45.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.816+0000 7fb13ad9d700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb12806ad10 0x7fb12806d1c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb12c000f60 tx=0x7fb12c007ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:45.820+0000 7fb1237fe700 1 -- 172.21.15.23:0/3554472200 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb13c1e3d40 con 0x7fb13c0c9fd0 2023-06-22T08:23:46.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~203s 2023-06-22T08:23:46.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.078+0000 7fb1411ff700 1 -- 172.21.15.23:0/3554472200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb13c1e3d40 con 0x7fb13c0c9fd0 2023-06-22T08:23:46.081 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.078+0000 7fb1237fe700 1 -- 172.21.15.23:0/3554472200 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+4728 (secure 0 0 0) 0x7fb13c1e3d40 con 0x7fb13c0c9fd0 2023-06-22T08:23:46.081 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:46.081 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:42.750608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:resolve","state_seq":18,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:46.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 -- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb12806ad10 msgr2=0x7fb12806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:46.083 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb12806ad10 0x7fb12806d1c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb12c000f60 tx=0x7fb12c007ed0 comp rx=0 tx=0).stop 2023-06-22T08:23:46.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 -- 172.21.15.23:0/3554472200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 msgr2=0x7fb13c156ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:46.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 0x7fb13c156ec0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb1300090b0 tx=0x7fb130004750 comp rx=0 tx=0).stop 2023-06-22T08:23:46.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 -- 172.21.15.23:0/3554472200 shutdown_connections 2023-06-22T08:23:46.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb12806ad10 0x7fb12806d1c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:46.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb13c152ce0 0x7fb13c157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:46.084 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb13c0c40b0 0x7fb13c0cac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:46.085 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 --2- 172.21.15.23:0/3554472200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c0c9fd0 0x7fb13c156ec0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:46.085 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 -- 172.21.15.23:0/3554472200 >> 172.21.15.23:0/3554472200 conn(0x7fb13c06ae30 msgr2=0x7fb13c14c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:46.085 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 -- 172.21.15.23:0/3554472200 shutdown_connections 2023-06-22T08:23:46.085 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:46.081+0000 7fb1411ff700 1 -- 172.21.15.23:0/3554472200 wait complete. 2023-06-22T08:23:46.085 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 185 2023-06-22T08:23:46.102 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:23:46.498 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:23:46.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:23:48.102 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:48.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.311+0000 7f1c7a5ff700 1 Processor -- start 2023-06-22T08:23:48.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.311+0000 7f1c7a5ff700 1 -- start start 2023-06-22T08:23:48.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.312+0000 7f1c7a5ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1c7415a320 0x7f1c740c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:48.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.312+0000 7f1c7a5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c7406d050 0x7f1c74160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:48.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.312+0000 7f1c7a5ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c740c6a60 con 0x7f1c740c3a40 2023-06-22T08:23:48.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.312+0000 7f1c7a5ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c74157570 con 0x7f1c7415a320 2023-06-22T08:23:48.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.312+0000 7f1c7a5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740c4350 con 0x7f1c7406d050 2023-06-22T08:23:48.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c73fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1c740c3a40 0x7f1c74159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55252/0 (socket says 172.21.15.23:55252) 2023-06-22T08:23:48.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c73fff700 1 -- 172.21.15.23:0/2080049826 learned_addr learned my addr 172.21.15.23:0/2080049826 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:48.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c737fe700 1 --2- 172.21.15.23:0/2080049826 >> v2:172.21.15.67:3300/0 conn(0x7f1c7415a320 0x7f1c740c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:48.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c78b9c700 1 --2- 172.21.15.23:0/2080049826 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c7406d050 0x7f1c74160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:48.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c72ffd700 1 -- 172.21.15.23:0/2080049826 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236430247 0 0) 0x7f1c740c6a60 con 0x7f1c740c3a40 2023-06-22T08:23:48.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c72ffd700 1 -- 172.21.15.23:0/2080049826 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c5c0035b0 con 0x7f1c740c3a40 2023-06-22T08:23:48.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c78b9c700 1 -- 172.21.15.23:0/2080049826 >> v2:172.21.15.67:3300/0 conn(0x7f1c7415a320 msgr2=0x7f1c740c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:48.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c78b9c700 1 --2- 172.21.15.23:0/2080049826 >> v2:172.21.15.67:3300/0 conn(0x7f1c7415a320 0x7f1c740c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:48.319 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c78b9c700 1 -- 172.21.15.23:0/2080049826 >> v1:172.21.15.23:6789/0 conn(0x7f1c740c3a40 legacy=0x7f1c74159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:48.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c72ffd700 1 -- 172.21.15.23:0/2080049826 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3553809676 0 0) 0x7f1c5c0035b0 con 0x7f1c740c3a40 2023-06-22T08:23:48.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c78b9c700 1 -- 172.21.15.23:0/2080049826 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7414d690 con 0x7f1c7406d050 2023-06-22T08:23:48.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.313+0000 7f1c737fe700 1 --2- 172.21.15.23:0/2080049826 >> v2:172.21.15.67:3300/0 conn(0x7f1c7415a320 0x7f1c740c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:23:48.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.314+0000 7f1c78b9c700 1 --2- 172.21.15.23:0/2080049826 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c7406d050 0x7f1c74160ec0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009d90 tx=0x7f1c6c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4500326f0bdeae76 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:48.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.314+0000 7f1c72ffd700 1 -- 172.21.15.23:0/2080049826 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c00e070 con 0x7f1c7406d050 2023-06-22T08:23:48.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.314+0000 7f1c72ffd700 1 -- 172.21.15.23:0/2080049826 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6c003a20 con 0x7f1c7406d050 2023-06-22T08:23:48.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.314+0000 7f1c72ffd700 1 -- 172.21.15.23:0/2080049826 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64001670 con 0x7f1c7406d050 2023-06-22T08:23:48.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.314+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/2080049826 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c7406d050 msgr2=0x7f1c74160ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:48.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.314+0000 7f1c7a5ff700 1 --2- 172.21.15.23:0/2080049826 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c7406d050 0x7f1c74160ec0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c009d90 tx=0x7f1c6c00b330 comp rx=0 tx=0).stop 2023-06-22T08:23:48.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.315+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/2080049826 shutdown_connections 2023-06-22T08:23:48.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.315+0000 7f1c7a5ff700 1 --2- 172.21.15.23:0/2080049826 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c7406d050 0x7f1c74160ec0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:48.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.315+0000 7f1c7a5ff700 1 --2- 172.21.15.23:0/2080049826 >> v2:172.21.15.67:3300/0 conn(0x7f1c7415a320 0x7f1c740c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:48.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.315+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/2080049826 >> 172.21.15.23:0/2080049826 conn(0x7f1c740ca3f0 msgr2=0x7f1c740ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:48.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.315+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/2080049826 shutdown_connections 2023-06-22T08:23:48.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.315+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/2080049826 wait complete. 2023-06-22T08:23:48.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.317+0000 7f1c7a5ff700 1 Processor -- start 2023-06-22T08:23:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.317+0000 7f1c7a5ff700 1 -- start start 2023-06-22T08:23:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.318+0000 7f1c7a5ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c7406d050 0x7f1c740c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.318+0000 7f1c7a5ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c740c3a40 0x7f1c7414d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.318+0000 7f1c7a5ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c7415a320 0x7f1c741483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.318+0000 7f1c7a5ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c741591f0 con 0x7f1c7415a320 2023-06-22T08:23:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.318+0000 7f1c7a5ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c740c9510 con 0x7f1c7406d050 2023-06-22T08:23:48.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c7a5ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c74158e40 con 0x7f1c740c3a40 2023-06-22T08:23:48.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c78b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c7415a320 0x7f1c741483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:48.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c73fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c7406d050 0x7f1c740c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:48.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c737fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c740c3a40 0x7f1c7414d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:48.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c737fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c740c3a40 0x7f1c7414d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:38920/0 (socket says 172.21.15.23:38920) 2023-06-22T08:23:48.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c737fe700 1 -- 172.21.15.23:0/3191535908 learned_addr learned my addr 172.21.15.23:0/3191535908 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:48.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c78b9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c7415a320 0x7f1c741483e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59016/0 (socket says 172.21.15.23:59016) 2023-06-22T08:23:48.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c73fff700 1 -- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c740c3a40 msgr2=0x7f1c7414d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:48.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c73fff700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c740c3a40 0x7f1c7414d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:48.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c73fff700 1 -- 172.21.15.23:0/3191535908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c7415a320 msgr2=0x7f1c741483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:48.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c73fff700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c7415a320 0x7f1c741483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:48.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c73fff700 1 -- 172.21.15.23:0/3191535908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c680094a0 con 0x7f1c7406d050 2023-06-22T08:23:48.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.319+0000 7f1c737fe700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c740c3a40 0x7f1c7414d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:23:48.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.320+0000 7f1c73fff700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c7406d050 0x7f1c740c9b70 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400c730 tx=0x7f1c6400ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:48.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.320+0000 7f1c70ff9700 1 -- 172.21.15.23:0/3191535908 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64004c80 con 0x7f1c7406d050 2023-06-22T08:23:48.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.320+0000 7f1c70ff9700 1 -- 172.21.15.23:0/3191535908 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c64004de0 con 0x7f1c7406d050 2023-06-22T08:23:48.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.320+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/3191535908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c009a70 con 0x7f1c7406d050 2023-06-22T08:23:48.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.320+0000 7f1c70ff9700 1 -- 172.21.15.23:0/3191535908 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64007310 con 0x7f1c7406d050 2023-06-22T08:23:48.327 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.320+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/3191535908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c740c8470 con 0x7f1c7406d050 2023-06-22T08:23:48.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.321+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/3191535908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c740c50d0 con 0x7f1c7406d050 2023-06-22T08:23:48.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.322+0000 7f1c70ff9700 1 -- 172.21.15.23:0/3191535908 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1c64007470 con 0x7f1c7406d050 2023-06-22T08:23:48.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.323+0000 7f1c70ff9700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6006ace0 0x7f1c6006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:48.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.323+0000 7f1c70ff9700 1 -- 172.21.15.23:0/3191535908 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f1c640899b0 con 0x7f1c7406d050 2023-06-22T08:23:48.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.323+0000 7f1c737fe700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6006ace0 0x7f1c6006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:48.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.324+0000 7f1c737fe700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6006ace0 0x7f1c6006d190 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1c740c13c0 tx=0x7f1c68009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:48.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.328+0000 7f1c70ff9700 1 -- 172.21.15.23:0/3191535908 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1c640541e0 con 0x7f1c7406d050 2023-06-22T08:23:48.586 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.584+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/3191535908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c74158fb0 con 0x7f1c7406d050 2023-06-22T08:23:48.587 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.585+0000 7f1c70ff9700 1 -- 172.21.15.23:0/3191535908 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+4728 (secure 0 0 0) 0x7f1c64057800 con 0x7f1c7406d050 2023-06-22T08:23:48.588 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:48.588 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:42.750608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:resolve","state_seq":18,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:48.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.587+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6006ace0 msgr2=0x7f1c6006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:48.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.587+0000 7f1c7a5ff700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6006ace0 0x7f1c6006d190 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1c740c13c0 tx=0x7f1c68009210 comp rx=0 tx=0).stop 2023-06-22T08:23:48.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.588+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c7406d050 msgr2=0x7f1c740c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.588+0000 7f1c7a5ff700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c7406d050 0x7f1c740c9b70 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400c730 tx=0x7f1c6400ca40 comp rx=0 tx=0).stop 2023-06-22T08:23:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.588+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/3191535908 shutdown_connections 2023-06-22T08:23:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.588+0000 7f1c7a5ff700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c7415a320 0x7f1c741483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.588+0000 7f1c7a5ff700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f1c6006ace0 0x7f1c6006d190 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.588+0000 7f1c7a5ff700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c7406d050 0x7f1c740c9b70 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.588+0000 7f1c7a5ff700 1 --2- 172.21.15.23:0/3191535908 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1c740c3a40 0x7f1c7414d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.588+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/3191535908 >> 172.21.15.23:0/3191535908 conn(0x7f1c740ca3f0 msgr2=0x7f1c7414a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:48.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.588+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/3191535908 shutdown_connections 2023-06-22T08:23:48.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:48.588+0000 7f1c7a5ff700 1 -- 172.21.15.23:0/3191535908 wait complete. 2023-06-22T08:23:48.592 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 185 2023-06-22T08:23:48.609 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 185, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 185, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_6313": { "addr": "172.21.15.23:6837/1502668188", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6836", "nonce": 1502668188, "type": "v2" }, { "addr": "172.21.15.23:6837", "nonce": 1502668188, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6313, "incarnation": 132, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 36 }, "gid_6616": { "addr": "172.21.15.67:6837/1855579717", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1855579717, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1855579717, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6616, "incarnation": 162, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_6705": { "addr": "172.21.15.23:6835/3148174349", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 3148174349, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 3148174349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6705, "incarnation": 184, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:resolve", "state_seq": 18 }, "gid_6730": { "addr": "172.21.15.23:6834/456927633", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 456927633, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 456927633, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6730, "incarnation": 172, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 110, "max_file_size": 1099511627776, "max_mds": 4, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:23:42.750608+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 6616, "mds_1": 6730, "mds_2": 6313, "mds_3": 6705 } } } ], "standbys": [] } 2023-06-22T08:23:50.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:50.822 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.820+0000 7f21939ff700 1 Processor -- start 2023-06-22T08:23:50.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.820+0000 7f21939ff700 1 -- start start 2023-06-22T08:23:50.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.821+0000 7f21939ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 0x7f218c0c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:50.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.821+0000 7f21939ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f218c15c0b0 0x7f218c15c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:50.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.821+0000 7f21939ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218c159730 con 0x7f218c0c9200 2023-06-22T08:23:50.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.821+0000 7f21939ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218c0c5470 con 0x7f218c15c0b0 2023-06-22T08:23:50.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.821+0000 7f21939ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218c158910 con 0x7f218c152a10 2023-06-22T08:23:50.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.821+0000 7f2190f9a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f218c0c9200 0x7f218c0cb260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55258/0 (socket says 172.21.15.23:55258) 2023-06-22T08:23:50.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.821+0000 7f219179b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 0x7f218c0c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:50.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.821+0000 7f2190f9a700 1 -- 172.21.15.23:0/1899125374 learned_addr learned my addr 172.21.15.23:0/1899125374 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:50.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.821+0000 7f2191f9c700 1 --2- 172.21.15.23:0/1899125374 >> v2:172.21.15.23:3300/0 conn(0x7f218c15c0b0 0x7f218c15c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:50.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.822+0000 7f2183fff700 1 -- 172.21.15.23:0/1899125374 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277015110 0 0) 0x7f218c159730 con 0x7f218c0c9200 2023-06-22T08:23:50.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.822+0000 7f2183fff700 1 -- 172.21.15.23:0/1899125374 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21780035b0 con 0x7f218c0c9200 2023-06-22T08:23:50.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.822+0000 7f2191f9c700 1 -- 172.21.15.23:0/1899125374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 msgr2=0x7f218c0c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:50.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.822+0000 7f2191f9c700 1 --2- 172.21.15.23:0/1899125374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 0x7f218c0c4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:50.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.822+0000 7f2191f9c700 1 -- 172.21.15.23:0/1899125374 >> v1:172.21.15.23:6789/0 conn(0x7f218c0c9200 legacy=0x7f218c0cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:50.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.822+0000 7f2191f9c700 1 -- 172.21.15.23:0/1899125374 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c15ca20 con 0x7f218c15c0b0 2023-06-22T08:23:50.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.822+0000 7f2191f9c700 1 --2- 172.21.15.23:0/1899125374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f218c15c0b0 0x7f218c15c4e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2184009d90 tx=0x7f218400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3447452394fbdfaf server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:50.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.822+0000 7f2183fff700 1 -- 172.21.15.23:0/1899125374 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218400e070 con 0x7f218c15c0b0 2023-06-22T08:23:50.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.823+0000 7f2183fff700 1 -- 172.21.15.23:0/1899125374 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218400bb20 con 0x7f218c15c0b0 2023-06-22T08:23:50.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.823+0000 7f2183fff700 1 -- 172.21.15.23:0/1899125374 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184012430 con 0x7f218c15c0b0 2023-06-22T08:23:50.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.823+0000 7f21939ff700 1 -- 172.21.15.23:0/1899125374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f218c15c0b0 msgr2=0x7f218c15c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:50.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.823+0000 7f21939ff700 1 --2- 172.21.15.23:0/1899125374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f218c15c0b0 0x7f218c15c4e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2184009d90 tx=0x7f218400b330 comp rx=0 tx=0).stop 2023-06-22T08:23:50.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.823+0000 7f21939ff700 1 -- 172.21.15.23:0/1899125374 shutdown_connections 2023-06-22T08:23:50.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.823+0000 7f21939ff700 1 --2- 172.21.15.23:0/1899125374 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 0x7f218c0c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:50.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.823+0000 7f21939ff700 1 --2- 172.21.15.23:0/1899125374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f218c15c0b0 0x7f218c15c4e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:50.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.823+0000 7f21939ff700 1 -- 172.21.15.23:0/1899125374 >> 172.21.15.23:0/1899125374 conn(0x7f218c065b90 msgr2=0x7f218c065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:50.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.823+0000 7f21939ff700 1 -- 172.21.15.23:0/1899125374 shutdown_connections 2023-06-22T08:23:50.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.823+0000 7f21939ff700 1 -- 172.21.15.23:0/1899125374 wait complete. 2023-06-22T08:23:50.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.826+0000 7f21939ff700 1 Processor -- start 2023-06-22T08:23:50.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.826+0000 7f21939ff700 1 -- start start 2023-06-22T08:23:50.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f21939ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f218c0c9200 0x7f218c1e2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:50.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f21939ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 0x7f218c0b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:50.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f21939ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f218c15c0b0 0x7f218c159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:50.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f21939ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c157920 con 0x7f218c15c0b0 2023-06-22T08:23:50.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f21939ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c159910 con 0x7f218c0c9200 2023-06-22T08:23:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f21939ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0b2ee0 con 0x7f218c152a10 2023-06-22T08:23:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f2190f9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 0x7f218c0b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f219179b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f218c0c9200 0x7f218c1e2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f2190f9a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 0x7f218c0b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:38938/0 (socket says 172.21.15.23:38938) 2023-06-22T08:23:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f2190f9a700 1 -- 172.21.15.23:0/3288996780 learned_addr learned my addr 172.21.15.23:0/3288996780 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:50.831 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.827+0000 7f2191f9c700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f218c15c0b0 0x7f218c159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:50.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f2191f9c700 1 -- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 msgr2=0x7f218c0b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:50.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f2191f9c700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 0x7f218c0b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:50.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f2191f9c700 1 -- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f218c0c9200 msgr2=0x7f218c1e2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:50.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f2191f9c700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f218c0c9200 0x7f218c1e2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:50.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f2191f9c700 1 -- 172.21.15.23:0/3288996780 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2184009a70 con 0x7f218c15c0b0 2023-06-22T08:23:50.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f219179b700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f218c0c9200 0x7f218c1e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:23:50.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f2191f9c700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f218c15c0b0 0x7f218c159da0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2184016000 tx=0x7f2184004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:50.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f2181ffb700 1 -- 172.21.15.23:0/3288996780 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184012430 con 0x7f218c15c0b0 2023-06-22T08:23:50.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f2181ffb700 1 -- 172.21.15.23:0/3288996780 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2184020070 con 0x7f218c15c0b0 2023-06-22T08:23:50.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f2181ffb700 1 -- 172.21.15.23:0/3288996780 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184003e30 con 0x7f218c15c0b0 2023-06-22T08:23:50.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.828+0000 7f21939ff700 1 -- 172.21.15.23:0/3288996780 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218c157b70 con 0x7f218c15c0b0 2023-06-22T08:23:50.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.829+0000 7f21939ff700 1 -- 172.21.15.23:0/3288996780 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f218c158130 con 0x7f218c15c0b0 2023-06-22T08:23:50.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.830+0000 7f21939ff700 1 -- 172.21.15.23:0/3288996780 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218c159a80 con 0x7f218c15c0b0 2023-06-22T08:23:50.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.830+0000 7f2181ffb700 1 -- 172.21.15.23:0/3288996780 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2184004040 con 0x7f218c15c0b0 2023-06-22T08:23:50.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.831+0000 7f2181ffb700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f217006ace0 0x7f217006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:50.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.831+0000 7f2181ffb700 1 -- 172.21.15.23:0/3288996780 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f2184022070 con 0x7f218c15c0b0 2023-06-22T08:23:50.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.832+0000 7f219179b700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f217006ace0 0x7f217006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:50.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.832+0000 7f219179b700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f217006ace0 0x7f217006d190 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2188009fd0 tx=0x7f2188009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:50.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:50.836+0000 7f2181ffb700 1 -- 172.21.15.23:0/3288996780 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f2184056090 con 0x7f218c15c0b0 2023-06-22T08:23:51.087 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.085+0000 7f21939ff700 1 -- 172.21.15.23:0/3288996780 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f218c04ea20 con 0x7f218c15c0b0 2023-06-22T08:23:51.088 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.085+0000 7f2181ffb700 1 -- 172.21.15.23:0/3288996780 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+4730 (secure 0 0 0) 0x7f218402c090 con 0x7f218c15c0b0 2023-06-22T08:23:51.088 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:51.088 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:48.740000+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:51.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 -- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f217006ace0 msgr2=0x7f217006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:51.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f217006ace0 0x7f217006d190 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2188009fd0 tx=0x7f2188009210 comp rx=0 tx=0).stop 2023-06-22T08:23:51.091 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 -- 172.21.15.23:0/3288996780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f218c15c0b0 msgr2=0x7f218c159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:51.092 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f218c15c0b0 0x7f218c159da0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2184016000 tx=0x7f2184004ad0 comp rx=0 tx=0).stop 2023-06-22T08:23:51.092 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 -- 172.21.15.23:0/3288996780 shutdown_connections 2023-06-22T08:23:51.092 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f218c15c0b0 0x7f218c159da0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:51.092 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f217006ace0 0x7f217006d190 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:51.092 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f218c0c9200 0x7f218c1e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:51.092 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 --2- 172.21.15.23:0/3288996780 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f218c152a10 0x7f218c0b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:51.092 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 -- 172.21.15.23:0/3288996780 >> 172.21.15.23:0/3288996780 conn(0x7f218c065b90 msgr2=0x7f218c0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:51.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 -- 172.21.15.23:0/3288996780 shutdown_connections 2023-06-22T08:23:51.093 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:51.088+0000 7f21939ff700 1 -- 172.21.15.23:0/3288996780 wait complete. 2023-06-22T08:23:51.093 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 186 2023-06-22T08:23:51.108 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:23:52.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~210s 2023-06-22T08:23:53.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:53.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.317+0000 7f59ff9ff700 1 Processor -- start 2023-06-22T08:23:53.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.318+0000 7f59ff9ff700 1 -- start start 2023-06-22T08:23:53.321 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.319+0000 7f59ff9ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f59f80c4d00 0x7f59f8159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.319+0000 7f59ff9ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f80c9c30 con 0x7f59f8152d70 2023-06-22T08:23:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.319+0000 7f59ff9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f815bea0 con 0x7f59f80c97d0 2023-06-22T08:23:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.319+0000 7f59ff9ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f80c95e0 con 0x7f59f80c4d00 2023-06-22T08:23:53.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.319+0000 7f59fdf9c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f59f80c4d00 0x7f59f8159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.319+0000 7f59fd79b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f59f8152d70 0x7f59f80cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55262/0 (socket says 172.21.15.23:55262) 2023-06-22T08:23:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.319+0000 7f59fd79b700 1 -- 172.21.15.23:0/2439857746 learned_addr learned my addr 172.21.15.23:0/2439857746 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:53.323 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.319+0000 7f59fcf9a700 1 --1- 172.21.15.23:0/2439857746 >> v1:172.21.15.67:6789/0 conn(0x7f59f80c97d0 0x7f59f80c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:58418/0 (socket says 172.21.15.23:58418) 2023-06-22T08:23:53.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.320+0000 7f59effff700 1 -- 172.21.15.23:0/2439857746 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074795530 0 0) 0x7f59f815bea0 con 0x7f59f80c97d0 2023-06-22T08:23:53.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.320+0000 7f59fdf9c700 1 -- 172.21.15.23:0/2439857746 >> v1:172.21.15.67:6789/0 conn(0x7f59f80c97d0 legacy=0x7f59f80c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:53.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.320+0000 7f59fdf9c700 1 -- 172.21.15.23:0/2439857746 >> v1:172.21.15.23:6789/0 conn(0x7f59f8152d70 legacy=0x7f59f80cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:53.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.320+0000 7f59fdf9c700 1 -- 172.21.15.23:0/2439857746 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f815c160 con 0x7f59f80c4d00 2023-06-22T08:23:53.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.320+0000 7f59fdf9c700 1 --2- 172.21.15.23:0/2439857746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 0x7f59f8159330 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009d90 tx=0x7f59f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1953893b115f37af server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:53.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.320+0000 7f59effff700 1 -- 172.21.15.23:0/2439857746 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f000e070 con 0x7f59f80c4d00 2023-06-22T08:23:53.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.320+0000 7f59effff700 1 -- 172.21.15.23:0/2439857746 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f000bbd0 con 0x7f59f80c4d00 2023-06-22T08:23:53.325 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.320+0000 7f59effff700 1 -- 172.21.15.23:0/2439857746 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f0012430 con 0x7f59f80c4d00 2023-06-22T08:23:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.321+0000 7f59ff9ff700 1 -- 172.21.15.23:0/2439857746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 msgr2=0x7f59f8159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.321+0000 7f59ff9ff700 1 --2- 172.21.15.23:0/2439857746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 0x7f59f8159330 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009d90 tx=0x7f59f000b330 comp rx=0 tx=0).stop 2023-06-22T08:23:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.321+0000 7f59ff9ff700 1 -- 172.21.15.23:0/2439857746 shutdown_connections 2023-06-22T08:23:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.321+0000 7f59ff9ff700 1 --2- 172.21.15.23:0/2439857746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 0x7f59f8159330 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.321+0000 7f59ff9ff700 1 -- 172.21.15.23:0/2439857746 >> 172.21.15.23:0/2439857746 conn(0x7f59f806ae30 msgr2=0x7f59f806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.321+0000 7f59ff9ff700 1 -- 172.21.15.23:0/2439857746 shutdown_connections 2023-06-22T08:23:53.326 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.321+0000 7f59ff9ff700 1 -- 172.21.15.23:0/2439857746 wait complete. 2023-06-22T08:23:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.323+0000 7f59ff9ff700 1 Processor -- start 2023-06-22T08:23:53.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.324+0000 7f59ff9ff700 1 -- start start 2023-06-22T08:23:53.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59ff9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 0x7f59f814e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:53.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59ff9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80c97d0 0x7f59f81ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:53.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59ff9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59f8152d70 0x7f59f81eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59ff9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f80c6a30 con 0x7f59f80c4d00 2023-06-22T08:23:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59ff9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f814c200 con 0x7f59f80c97d0 2023-06-22T08:23:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59ff9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f814c4c0 con 0x7f59f8152d70 2023-06-22T08:23:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fd79b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 0x7f59f814e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fd79b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 0x7f59f814e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59054/0 (socket says 172.21.15.23:59054) 2023-06-22T08:23:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fd79b700 1 -- 172.21.15.23:0/449068209 learned_addr learned my addr 172.21.15.23:0/449068209 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fcf9a700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80c97d0 0x7f59f81ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fdf9c700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59f8152d70 0x7f59f81eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fd79b700 1 -- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59f8152d70 msgr2=0x7f59f81eec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fd79b700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59f8152d70 0x7f59f81eec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fd79b700 1 -- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80c97d0 msgr2=0x7f59f81ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fd79b700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80c97d0 0x7f59f81ee750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fd79b700 1 -- 172.21.15.23:0/449068209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f0009a70 con 0x7f59f80c4d00 2023-06-22T08:23:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fdf9c700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59f8152d70 0x7f59f81eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:23:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.325+0000 7f59fcf9a700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80c97d0 0x7f59f81ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:23:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.326+0000 7f59fd79b700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 0x7f59f814e100 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f59f400c480 tx=0x7f59f400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.326+0000 7f59edffb700 1 -- 172.21.15.23:0/449068209 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4005d70 con 0x7f59f80c4d00 2023-06-22T08:23:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.326+0000 7f59edffb700 1 -- 172.21.15.23:0/449068209 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f40070d0 con 0x7f59f80c4d00 2023-06-22T08:23:53.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.326+0000 7f59ff9ff700 1 -- 172.21.15.23:0/449068209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f80c5e20 con 0x7f59f80c4d00 2023-06-22T08:23:53.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.326+0000 7f59edffb700 1 -- 172.21.15.23:0/449068209 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4007440 con 0x7f59f80c4d00 2023-06-22T08:23:53.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.326+0000 7f59ff9ff700 1 -- 172.21.15.23:0/449068209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f80c63e0 con 0x7f59f80c4d00 2023-06-22T08:23:53.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.327+0000 7f59ff9ff700 1 -- 172.21.15.23:0/449068209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f804ea20 con 0x7f59f80c4d00 2023-06-22T08:23:53.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.328+0000 7f59edffb700 1 -- 172.21.15.23:0/449068209 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f59f40075a0 con 0x7f59f80c4d00 2023-06-22T08:23:53.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.329+0000 7f59edffb700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59dc06ace0 0x7f59dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:53.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.329+0000 7f59edffb700 1 -- 172.21.15.23:0/449068209 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f59f4088fe0 con 0x7f59f80c4d00 2023-06-22T08:23:53.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.329+0000 7f59fcf9a700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59dc06ace0 0x7f59dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:53.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.330+0000 7f59fcf9a700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59dc06ace0 0x7f59dc06d190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f59f80c8ec0 tx=0x7f59e8007ce0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:53.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.333+0000 7f59edffb700 1 -- 172.21.15.23:0/449068209 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f59f4053a00 con 0x7f59f80c4d00 2023-06-22T08:23:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.585+0000 7f59ff9ff700 1 -- 172.21.15.23:0/449068209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59f80c9e00 con 0x7f59f80c4d00 2023-06-22T08:23:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.586+0000 7f59edffb700 1 -- 172.21.15.23:0/449068209 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+4730 (secure 0 0 0) 0x7f59f4017090 con 0x7f59f80c4d00 2023-06-22T08:23:53.588 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:53.589 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:48.740000+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:53.590 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 -- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59dc06ace0 msgr2=0x7f59dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:53.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59dc06ace0 0x7f59dc06d190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f59f80c8ec0 tx=0x7f59e8007ce0 comp rx=0 tx=0).stop 2023-06-22T08:23:53.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 -- 172.21.15.23:0/449068209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 msgr2=0x7f59f814e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:53.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 0x7f59f814e100 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f59f400c480 tx=0x7f59f400c840 comp rx=0 tx=0).stop 2023-06-22T08:23:53.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 -- 172.21.15.23:0/449068209 shutdown_connections 2023-06-22T08:23:53.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f59f8152d70 0x7f59f81eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:53.591 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59f80c4d00 0x7f59f814e100 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:53.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f59dc06ace0 0x7f59dc06d190 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:53.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 --2- 172.21.15.23:0/449068209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59f80c97d0 0x7f59f81ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:53.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 -- 172.21.15.23:0/449068209 >> 172.21.15.23:0/449068209 conn(0x7f59f806ae30 msgr2=0x7f59f806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:53.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 -- 172.21.15.23:0/449068209 shutdown_connections 2023-06-22T08:23:53.592 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:53.588+0000 7f59ff9ff700 1 -- 172.21.15.23:0/449068209 wait complete. 2023-06-22T08:23:53.593 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 186 2023-06-22T08:23:53.609 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:23:55.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:55.823 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.821+0000 7f5b155ff700 1 Processor -- start 2023-06-22T08:23:55.824 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.821+0000 7f5b155ff700 1 -- start start 2023-06-22T08:23:55.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.822+0000 7f5b155ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5b10071850 0x7f5b100c9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:55.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.822+0000 7f5b155ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b10071c20 0x7f5b1015c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:55.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.822+0000 7f5b155ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5b100c4160 0x7f5b10158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:55.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.822+0000 7f5b155ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1015c340 con 0x7f5b10071850 2023-06-22T08:23:55.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.822+0000 7f5b155ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b100cb840 con 0x7f5b10071c20 2023-06-22T08:23:55.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.822+0000 7f5b155ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b100cba90 con 0x7f5b100c4160 2023-06-22T08:23:55.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0effd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5b10071850 0x7f5b100c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:55.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0effd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5b10071850 0x7f5b100c9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59060/0 (socket says 172.21.15.23:59060) 2023-06-22T08:23:55.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0effd700 1 -- 172.21.15.23:0/1661267657 learned_addr learned my addr 172.21.15.23:0/1661267657 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:55.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0f7fe700 1 --2- 172.21.15.23:0/1661267657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5b100c4160 0x7f5b10158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:55.826 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0e7fc700 1 --2- 172.21.15.23:0/1661267657 >> v2:172.21.15.67:3300/0 conn(0x7f5b10071c20 0x7f5b1015c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:55.827 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0effd700 1 -- 172.21.15.23:0/1661267657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5b100c4160 msgr2=0x7f5b10158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:55.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0effd700 1 --2- 172.21.15.23:0/1661267657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5b100c4160 0x7f5b10158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:55.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0effd700 1 -- 172.21.15.23:0/1661267657 >> v2:172.21.15.67:3300/0 conn(0x7f5b10071c20 msgr2=0x7f5b1015c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:23:55.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0effd700 1 --2- 172.21.15.23:0/1661267657 >> v2:172.21.15.67:3300/0 conn(0x7f5b10071c20 0x7f5b1015c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:55.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0effd700 1 -- 172.21.15.23:0/1661267657 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100cbd70 con 0x7f5b10071850 2023-06-22T08:23:55.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.823+0000 7f5b0effd700 1 --2- 172.21.15.23:0/1661267657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b10071850 0x7f5b100c9190 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5b1008c3a0 tx=0x7f5b00009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336c1f6eb157597c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:55.828 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b0dffb700 1 -- 172.21.15.23:0/1661267657 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00011830 con 0x7f5b10071850 2023-06-22T08:23:55.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b0dffb700 1 -- 172.21.15.23:0/1661267657 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b00011990 con 0x7f5b10071850 2023-06-22T08:23:55.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b0dffb700 1 -- 172.21.15.23:0/1661267657 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00011d20 con 0x7f5b10071850 2023-06-22T08:23:55.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b155ff700 1 -- 172.21.15.23:0/1661267657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b10071850 msgr2=0x7f5b100c9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:55.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b155ff700 1 --2- 172.21.15.23:0/1661267657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b10071850 0x7f5b100c9190 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5b1008c3a0 tx=0x7f5b00009bd0 comp rx=0 tx=0).stop 2023-06-22T08:23:55.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b155ff700 1 -- 172.21.15.23:0/1661267657 shutdown_connections 2023-06-22T08:23:55.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b155ff700 1 --2- 172.21.15.23:0/1661267657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5b100c4160 0x7f5b10158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:55.829 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b155ff700 1 --2- 172.21.15.23:0/1661267657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b10071850 0x7f5b100c9190 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:55.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b155ff700 1 --2- 172.21.15.23:0/1661267657 >> v2:172.21.15.67:3300/0 conn(0x7f5b10071c20 0x7f5b1015c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:55.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b155ff700 1 -- 172.21.15.23:0/1661267657 >> 172.21.15.23:0/1661267657 conn(0x7f5b1006ad30 msgr2=0x7f5b1006b870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:55.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b155ff700 1 -- 172.21.15.23:0/1661267657 shutdown_connections 2023-06-22T08:23:55.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.824+0000 7f5b155ff700 1 -- 172.21.15.23:0/1661267657 wait complete. 2023-06-22T08:23:55.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.826+0000 7f5b155ff700 1 Processor -- start 2023-06-22T08:23:55.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.826+0000 7f5b155ff700 1 -- start start 2023-06-22T08:23:55.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.827+0000 7f5b155ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b10071850 0x7f5b1015ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:55.832 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.827+0000 7f5b155ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5b10071c20 0x7f5b1014e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:55.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.827+0000 7f5b155ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b100c4160 0x7f5b1014e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:55.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.827+0000 7f5b155ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b101ef080 con 0x7f5b100c4160 2023-06-22T08:23:55.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.827+0000 7f5b155ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b101559f0 con 0x7f5b10071850 2023-06-22T08:23:55.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.827+0000 7f5b155ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b10155cd0 con 0x7f5b10071c20 2023-06-22T08:23:55.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.827+0000 7f5b0f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b100c4160 0x7f5b1014e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:55.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.827+0000 7f5b0f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b100c4160 0x7f5b1014e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59062/0 (socket says 172.21.15.23:59062) 2023-06-22T08:23:55.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.827+0000 7f5b0f7fe700 1 -- 172.21.15.23:0/3572596167 learned_addr learned my addr 172.21.15.23:0/3572596167 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:55.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.827+0000 7f5b0e7fc700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5b10071c20 0x7f5b1014e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:55.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5b0effd700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b10071850 0x7f5b1015ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:55.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5b0f7fe700 1 -- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5b10071c20 msgr2=0x7f5b1014e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:55.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5b0f7fe700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5b10071c20 0x7f5b1014e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:55.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5b0f7fe700 1 -- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b10071850 msgr2=0x7f5b1015ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:55.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5b0f7fe700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b10071850 0x7f5b1015ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:55.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5b0f7fe700 1 -- 172.21.15.23:0/3572596167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b00009550 con 0x7f5b100c4160 2023-06-22T08:23:55.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5b0f7fe700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b100c4160 0x7f5b1014e810 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800efc0 tx=0x7f5b0800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:55.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5af77fe700 1 -- 172.21.15.23:0/3572596167 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0800cad0 con 0x7f5b100c4160 2023-06-22T08:23:55.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5af77fe700 1 -- 172.21.15.23:0/3572596167 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0800f040 con 0x7f5b100c4160 2023-06-22T08:23:55.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5af77fe700 1 -- 172.21.15.23:0/3572596167 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0800ce70 con 0x7f5b100c4160 2023-06-22T08:23:55.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.828+0000 7f5b155ff700 1 -- 172.21.15.23:0/3572596167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b10155fb0 con 0x7f5b100c4160 2023-06-22T08:23:55.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.829+0000 7f5b155ff700 1 -- 172.21.15.23:0/3572596167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b10153420 con 0x7f5b100c4160 2023-06-22T08:23:55.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.830+0000 7f5b155ff700 1 -- 172.21.15.23:0/3572596167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1004ea20 con 0x7f5b100c4160 2023-06-22T08:23:55.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.831+0000 7f5af77fe700 1 -- 172.21.15.23:0/3572596167 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5b08014480 con 0x7f5b100c4160 2023-06-22T08:23:55.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.832+0000 7f5af77fe700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5afc06ace0 0x7f5afc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:55.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.832+0000 7f5af77fe700 1 -- 172.21.15.23:0/3572596167 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f5b0808a620 con 0x7f5b100c4160 2023-06-22T08:23:55.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.832+0000 7f5b0effd700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5afc06ace0 0x7f5afc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:55.837 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.833+0000 7f5b0effd700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5afc06ace0 0x7f5afc06d190 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009c10 tx=0x7f5b00008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:55.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:55.837+0000 7f5af77fe700 1 -- 172.21.15.23:0/3572596167 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5b08058aa0 con 0x7f5b100c4160 2023-06-22T08:23:56.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.092+0000 7f5b155ff700 1 -- 172.21.15.23:0/3572596167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b1006ab10 con 0x7f5b100c4160 2023-06-22T08:23:56.095 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.093+0000 7f5af77fe700 1 -- 172.21.15.23:0/3572596167 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+4727 (secure 0 0 0) 0x7f5b08058630 con 0x7f5b100c4160 2023-06-22T08:23:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:56.096 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:54.053005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:56.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 -- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5afc06ace0 msgr2=0x7f5afc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:56.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5afc06ace0 0x7f5afc06d190 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009c10 tx=0x7f5b00008030 comp rx=0 tx=0).stop 2023-06-22T08:23:56.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 -- 172.21.15.23:0/3572596167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b100c4160 msgr2=0x7f5b1014e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:56.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b100c4160 0x7f5b1014e810 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800efc0 tx=0x7f5b0800c0c0 comp rx=0 tx=0).stop 2023-06-22T08:23:56.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 -- 172.21.15.23:0/3572596167 shutdown_connections 2023-06-22T08:23:56.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b100c4160 0x7f5b1014e810 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:56.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f5afc06ace0 0x7f5afc06d190 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:56.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b10071850 0x7f5b1015ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:56.099 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 --2- 172.21.15.23:0/3572596167 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5b10071c20 0x7f5b1014e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:56.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 -- 172.21.15.23:0/3572596167 >> 172.21.15.23:0/3572596167 conn(0x7f5b1006ad30 msgr2=0x7f5b10151460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:56.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 -- 172.21.15.23:0/3572596167 shutdown_connections 2023-06-22T08:23:56.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:56.096+0000 7f5b155ff700 1 -- 172.21.15.23:0/3572596167 wait complete. 2023-06-22T08:23:56.100 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 187 2023-06-22T08:23:56.117 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:23:58.118 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:23:58.328 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.326+0000 7fbcbbdff700 1 Processor -- start 2023-06-22T08:23:58.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.326+0000 7fbcbbdff700 1 -- start start 2023-06-22T08:23:58.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.327+0000 7fbcbbdff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbcb40cf1b0 0x7fbcb40c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:58.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.327+0000 7fbcbbdff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb415c370 con 0x7fbcb40c9fd0 2023-06-22T08:23:58.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.327+0000 7fbcbbdff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb415bda0 con 0x7fbcb4152ce0 2023-06-22T08:23:58.330 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.327+0000 7fbcbbdff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb41530b0 con 0x7fbcb40cf1b0 2023-06-22T08:23:58.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.328+0000 7fbcba39c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbcb40cf1b0 0x7fbcb40c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:58.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.328+0000 7fbcb939a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbcb40c9fd0 0x7fbcb40c40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:47248/0 (socket says 172.21.15.23:47248) 2023-06-22T08:23:58.331 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.328+0000 7fbcb939a700 1 -- 172.21.15.23:0/3021300257 learned_addr learned my addr 172.21.15.23:0/3021300257 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:23:58.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.328+0000 7fbcb8b99700 1 -- 172.21.15.23:0/3021300257 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462384214 0 0) 0x7fbcb415c370 con 0x7fbcb40c9fd0 2023-06-22T08:23:58.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.328+0000 7fbcb8b99700 1 -- 172.21.15.23:0/3021300257 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca00035b0 con 0x7fbcb40c9fd0 2023-06-22T08:23:58.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.328+0000 7fbcb8b99700 1 -- 172.21.15.23:0/3021300257 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776171504 0 0) 0x7fbcb415bda0 con 0x7fbcb4152ce0 2023-06-22T08:23:58.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.328+0000 7fbcba39c700 1 -- 172.21.15.23:0/3021300257 >> v1:172.21.15.67:6789/0 conn(0x7fbcb4152ce0 legacy=0x7fbcb415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:58.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.328+0000 7fbcba39c700 1 -- 172.21.15.23:0/3021300257 >> v1:172.21.15.23:6789/0 conn(0x7fbcb40c9fd0 legacy=0x7fbcb40c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:58.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.328+0000 7fbcba39c700 1 -- 172.21.15.23:0/3021300257 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb40cf630 con 0x7fbcb40cf1b0 2023-06-22T08:23:58.332 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.328+0000 7fbcba39c700 1 --2- 172.21.15.23:0/3021300257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb40cf1b0 0x7fbcb40c47c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009d90 tx=0x7fbcac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8770c8e5dd9a929 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:58.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.329+0000 7fbcb8b99700 1 -- 172.21.15.23:0/3021300257 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac00e070 con 0x7fbcb40cf1b0 2023-06-22T08:23:58.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.329+0000 7fbcb8b99700 1 -- 172.21.15.23:0/3021300257 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcac00bb20 con 0x7fbcb40cf1b0 2023-06-22T08:23:58.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.329+0000 7fbcb8b99700 1 -- 172.21.15.23:0/3021300257 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac00be90 con 0x7fbcb40cf1b0 2023-06-22T08:23:58.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.329+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3021300257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb40cf1b0 msgr2=0x7fbcb40c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:58.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.329+0000 7fbcbbdff700 1 --2- 172.21.15.23:0/3021300257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb40cf1b0 0x7fbcb40c47c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009d90 tx=0x7fbcac00b330 comp rx=0 tx=0).stop 2023-06-22T08:23:58.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.329+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3021300257 shutdown_connections 2023-06-22T08:23:58.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.329+0000 7fbcbbdff700 1 --2- 172.21.15.23:0/3021300257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb40cf1b0 0x7fbcb40c47c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:58.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.329+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3021300257 >> 172.21.15.23:0/3021300257 conn(0x7fbcb406ae30 msgr2=0x7fbcb406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:58.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.329+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3021300257 shutdown_connections 2023-06-22T08:23:58.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.329+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3021300257 wait complete. 2023-06-22T08:23:58.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.331+0000 7fbcbbdff700 1 Processor -- start 2023-06-22T08:23:58.335 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.331+0000 7fbcbbdff700 1 -- start start 2023-06-22T08:23:58.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.332+0000 7fbcbbdff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcb40c9fd0 0x7fbcb41e2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:58.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.332+0000 7fbcbbdff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcb40cf1b0 0x7fbcb41f11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:58.336 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.332+0000 7fbcbbdff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb4152ce0 0x7fbcb41f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.332+0000 7fbcbbdff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb40cd7a0 con 0x7fbcb4152ce0 2023-06-22T08:23:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.332+0000 7fbcbbdff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb4151ca0 con 0x7fbcb40cf1b0 2023-06-22T08:23:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.332+0000 7fbcbbdff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb41e2c70 con 0x7fbcb40c9fd0 2023-06-22T08:23:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.332+0000 7fbcb939a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcb40cf1b0 0x7fbcb41f11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.332+0000 7fbcba39c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb4152ce0 0x7fbcb41f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.332+0000 7fbcb9b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcb40c9fd0 0x7fbcb41e2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.332+0000 7fbcb939a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcb40cf1b0 0x7fbcb41f11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:52296/0 (socket says 172.21.15.23:52296) 2023-06-22T08:23:58.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcb939a700 1 -- 172.21.15.23:0/3844620831 learned_addr learned my addr 172.21.15.23:0/3844620831 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:23:58.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcba39c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb4152ce0 0x7fbcb41f16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59596/0 (socket says 172.21.15.23:59596) 2023-06-22T08:23:58.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcba39c700 1 -- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcb40c9fd0 msgr2=0x7fbcb41e2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:58.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcba39c700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcb40c9fd0 0x7fbcb41e2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:58.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcba39c700 1 -- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcb40cf1b0 msgr2=0x7fbcb41f11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:58.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcba39c700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcb40cf1b0 0x7fbcb41f11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:58.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcba39c700 1 -- 172.21.15.23:0/3844620831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcac009a70 con 0x7fbcb4152ce0 2023-06-22T08:23:58.338 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcba39c700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb4152ce0 0x7fbcb41f16f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fbcac016000 tx=0x7fbcac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:58.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcaa7fc700 1 -- 172.21.15.23:0/3844620831 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac004670 con 0x7fbcb4152ce0 2023-06-22T08:23:58.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcaa7fc700 1 -- 172.21.15.23:0/3844620831 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcac021070 con 0x7fbcb4152ce0 2023-06-22T08:23:58.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3844620831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb41f1c30 con 0x7fbcb4152ce0 2023-06-22T08:23:58.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.333+0000 7fbcaa7fc700 1 -- 172.21.15.23:0/3844620831 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac0098f0 con 0x7fbcb4152ce0 2023-06-22T08:23:58.339 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.334+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3844620831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb4150460 con 0x7fbcb4152ce0 2023-06-22T08:23:58.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.335+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3844620831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb404ea20 con 0x7fbcb4152ce0 2023-06-22T08:23:58.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.335+0000 7fbcaa7fc700 1 -- 172.21.15.23:0/3844620831 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbcac023030 con 0x7fbcb4152ce0 2023-06-22T08:23:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.336+0000 7fbcaa7fc700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbc9806ace0 0x7fbc9806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:23:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.337+0000 7fbcaa7fc700 1 -- 172.21.15.23:0/3844620831 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7fbcac08bae0 con 0x7fbcb4152ce0 2023-06-22T08:23:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.337+0000 7fbcb9b9b700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbc9806ace0 0x7fbc9806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:23:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.337+0000 7fbcb9b9b700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbc9806ace0 0x7fbc9806d190 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0007fd0 tx=0x7fbcb0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:23:58.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.341+0000 7fbcaa7fc700 1 -- 172.21.15.23:0/3844620831 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbcac056550 con 0x7fbcb4152ce0 2023-06-22T08:23:58.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.594+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3844620831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcb40c9db0 con 0x7fbcb4152ce0 2023-06-22T08:23:58.597 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.594+0000 7fbcaa7fc700 1 -- 172.21.15.23:0/3844620831 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+4727 (secure 0 0 0) 0x7fbcac059b70 con 0x7fbcb4152ce0 2023-06-22T08:23:58.597 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:23:58.598 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:23:54.053005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:23:58.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbc9806ace0 msgr2=0x7fbc9806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:58.600 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbc9806ace0 0x7fbc9806d190 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0007fd0 tx=0x7fbcb0009040 comp rx=0 tx=0).stop 2023-06-22T08:23:58.601 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3844620831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb4152ce0 msgr2=0x7fbcb41f16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:23:58.601 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb4152ce0 0x7fbcb41f16f0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fbcac016000 tx=0x7fbcac004ad0 comp rx=0 tx=0).stop 2023-06-22T08:23:58.601 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3844620831 shutdown_connections 2023-06-22T08:23:58.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcb4152ce0 0x7fbcb41f16f0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:58.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcb40c9fd0 0x7fbcb41e2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:58.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbc9806ace0 0x7fbc9806d190 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:58.602 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 --2- 172.21.15.23:0/3844620831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcb40cf1b0 0x7fbcb41f11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:23:58.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3844620831 >> 172.21.15.23:0/3844620831 conn(0x7fbcb406ae30 msgr2=0x7fbcb40c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:23:58.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3844620831 shutdown_connections 2023-06-22T08:23:58.603 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:23:58.597+0000 7fbcbbdff700 1 -- 172.21.15.23:0/3844620831 wait complete. 2023-06-22T08:23:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 187 2023-06-22T08:23:58.618 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:23:59.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~217s 2023-06-22T08:24:00.620 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:00.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.827+0000 7f0a777ff700 1 Processor -- start 2023-06-22T08:24:00.830 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.827+0000 7f0a777ff700 1 -- start start 2023-06-22T08:24:00.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.829+0000 7f0a777ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700aedc0 0x7f0a70164290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:00.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.829+0000 7f0a777ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a700c7400 con 0x7f0a700c8bd0 2023-06-22T08:24:00.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.829+0000 7f0a777ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a700c81c0 con 0x7f0a70152dd0 2023-06-22T08:24:00.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.829+0000 7f0a777ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a700c84a0 con 0x7f0a700aedc0 2023-06-22T08:24:00.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.829+0000 7f0a74d9a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f0a700c8bd0 0x7f0a700cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:47258/0 (socket says 172.21.15.23:47258) 2023-06-22T08:24:00.833 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.829+0000 7f0a74d9a700 1 -- 172.21.15.23:0/3183763538 learned_addr learned my addr 172.21.15.23:0/3183763538 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:00.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.829+0000 7f0a7559b700 1 --1- 172.21.15.23:0/3183763538 >> v1:172.21.15.67:6789/0 conn(0x7f0a70152dd0 0x7f0a7006afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:47732/0 (socket says 172.21.15.23:47732) 2023-06-22T08:24:00.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.829+0000 7f0a75d9c700 1 --2- 172.21.15.23:0/3183763538 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700aedc0 0x7f0a70164290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:00.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.829+0000 7f0a67fff700 1 -- 172.21.15.23:0/3183763538 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035369893 0 0) 0x7f0a700c81c0 con 0x7f0a70152dd0 2023-06-22T08:24:00.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.829+0000 7f0a67fff700 1 -- 172.21.15.23:0/3183763538 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a5c0035b0 con 0x7f0a70152dd0 2023-06-22T08:24:00.834 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.830+0000 7f0a67fff700 1 -- 172.21.15.23:0/3183763538 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059539009 0 0) 0x7f0a700c7400 con 0x7f0a700c8bd0 2023-06-22T08:24:00.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.830+0000 7f0a67fff700 1 -- 172.21.15.23:0/3183763538 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a700c81c0 con 0x7f0a700c8bd0 2023-06-22T08:24:00.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.830+0000 7f0a75d9c700 1 -- 172.21.15.23:0/3183763538 >> v1:172.21.15.67:6789/0 conn(0x7f0a70152dd0 legacy=0x7f0a7006afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:00.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.830+0000 7f0a75d9c700 1 -- 172.21.15.23:0/3183763538 >> v1:172.21.15.23:6789/0 conn(0x7f0a700c8bd0 legacy=0x7f0a700cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:00.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.830+0000 7f0a75d9c700 1 -- 172.21.15.23:0/3183763538 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a700af240 con 0x7f0a700aedc0 2023-06-22T08:24:00.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.830+0000 7f0a75d9c700 1 --2- 172.21.15.23:0/3183763538 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700aedc0 0x7f0a70164290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009dc0 tx=0x7f0a6800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c9622c897344eac server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:00.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.830+0000 7f0a67fff700 1 -- 172.21.15.23:0/3183763538 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6800e070 con 0x7f0a700aedc0 2023-06-22T08:24:00.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.830+0000 7f0a67fff700 1 -- 172.21.15.23:0/3183763538 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6800bb20 con 0x7f0a700aedc0 2023-06-22T08:24:00.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.830+0000 7f0a67fff700 1 -- 172.21.15.23:0/3183763538 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68012430 con 0x7f0a700aedc0 2023-06-22T08:24:00.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.831+0000 7f0a777ff700 1 -- 172.21.15.23:0/3183763538 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700aedc0 msgr2=0x7f0a70164290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:00.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.831+0000 7f0a777ff700 1 --2- 172.21.15.23:0/3183763538 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700aedc0 0x7f0a70164290 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009dc0 tx=0x7f0a6800b330 comp rx=0 tx=0).stop 2023-06-22T08:24:00.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.831+0000 7f0a777ff700 1 -- 172.21.15.23:0/3183763538 shutdown_connections 2023-06-22T08:24:00.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.831+0000 7f0a777ff700 1 --2- 172.21.15.23:0/3183763538 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700aedc0 0x7f0a70164290 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:00.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.831+0000 7f0a777ff700 1 -- 172.21.15.23:0/3183763538 >> 172.21.15.23:0/3183763538 conn(0x7f0a7006fb70 msgr2=0x7f0a7006ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:00.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.831+0000 7f0a777ff700 1 -- 172.21.15.23:0/3183763538 shutdown_connections 2023-06-22T08:24:00.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.831+0000 7f0a777ff700 1 -- 172.21.15.23:0/3183763538 wait complete. 2023-06-22T08:24:00.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.833+0000 7f0a777ff700 1 Processor -- start 2023-06-22T08:24:00.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.833+0000 7f0a777ff700 1 -- start start 2023-06-22T08:24:00.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a777ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700aedc0 0x7f0a700c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:00.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a777ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700c8bd0 0x7f0a700c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:00.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a777ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a70152dd0 0x7f0a701f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:00.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a777ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7015c560 con 0x7f0a70152dd0 2023-06-22T08:24:00.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a777ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a700c5ed0 con 0x7f0a700aedc0 2023-06-22T08:24:00.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a777ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7015bfe0 con 0x7f0a700c8bd0 2023-06-22T08:24:00.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a75d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a70152dd0 0x7f0a701f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:00.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a75d9c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a70152dd0 0x7f0a701f1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59604/0 (socket says 172.21.15.23:59604) 2023-06-22T08:24:00.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a75d9c700 1 -- 172.21.15.23:0/1110786819 learned_addr learned my addr 172.21.15.23:0/1110786819 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:00.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a7559b700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700aedc0 0x7f0a700c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:00.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a74d9a700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700c8bd0 0x7f0a700c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:00.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a75d9c700 1 -- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700c8bd0 msgr2=0x7f0a700c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:00.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a75d9c700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700c8bd0 0x7f0a700c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:00.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a75d9c700 1 -- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700aedc0 msgr2=0x7f0a700c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:00.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a75d9c700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700aedc0 0x7f0a700c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:00.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.834+0000 7f0a75d9c700 1 -- 172.21.15.23:0/1110786819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a68009a70 con 0x7f0a70152dd0 2023-06-22T08:24:00.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.835+0000 7f0a75d9c700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a70152dd0 0x7f0a701f1900 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0a68014040 tx=0x7f0a68004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:00.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.835+0000 7f0a65ffb700 1 -- 172.21.15.23:0/1110786819 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68012430 con 0x7f0a70152dd0 2023-06-22T08:24:00.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.835+0000 7f0a65ffb700 1 -- 172.21.15.23:0/1110786819 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a68020070 con 0x7f0a70152dd0 2023-06-22T08:24:00.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.835+0000 7f0a65ffb700 1 -- 172.21.15.23:0/1110786819 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6801ca50 con 0x7f0a70152dd0 2023-06-22T08:24:00.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.835+0000 7f0a777ff700 1 -- 172.21.15.23:0/1110786819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a700c4bd0 con 0x7f0a70152dd0 2023-06-22T08:24:00.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.835+0000 7f0a777ff700 1 -- 172.21.15.23:0/1110786819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7015c280 con 0x7f0a70152dd0 2023-06-22T08:24:00.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.836+0000 7f0a777ff700 1 -- 172.21.15.23:0/1110786819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a700cd580 con 0x7f0a70152dd0 2023-06-22T08:24:00.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.837+0000 7f0a65ffb700 1 -- 172.21.15.23:0/1110786819 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0a6801cbb0 con 0x7f0a70152dd0 2023-06-22T08:24:00.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.838+0000 7f0a65ffb700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a5406acb0 0x7f0a5406d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:00.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.838+0000 7f0a65ffb700 1 -- 172.21.15.23:0/1110786819 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f0a68022070 con 0x7f0a70152dd0 2023-06-22T08:24:00.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.838+0000 7f0a7559b700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a5406acb0 0x7f0a5406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:00.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.839+0000 7f0a7559b700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a5406acb0 0x7f0a5406d160 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c000fb0 tx=0x7f0a6c007d90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:00.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:00.843+0000 7f0a65ffb700 1 -- 172.21.15.23:0/1110786819 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0a680564b0 con 0x7f0a70152dd0 2023-06-22T08:24:01.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.098+0000 7f0a777ff700 1 -- 172.21.15.23:0/1110786819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a70070470 con 0x7f0a70152dd0 2023-06-22T08:24:01.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.098+0000 7f0a65ffb700 1 -- 172.21.15.23:0/1110786819 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+4727 (secure 0 0 0) 0x7f0a6802c070 con 0x7f0a70152dd0 2023-06-22T08:24:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:01.101 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:00.074498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:01.106 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 -- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a5406acb0 msgr2=0x7f0a5406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a5406acb0 0x7f0a5406d160 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c000fb0 tx=0x7f0a6c007d90 comp rx=0 tx=0).stop 2023-06-22T08:24:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 -- 172.21.15.23:0/1110786819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a70152dd0 msgr2=0x7f0a701f1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a70152dd0 0x7f0a701f1900 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0a68014040 tx=0x7f0a68004ad0 comp rx=0 tx=0).stop 2023-06-22T08:24:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 -- 172.21.15.23:0/1110786819 shutdown_connections 2023-06-22T08:24:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a70152dd0 0x7f0a701f1900 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:01.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0a5406acb0 0x7f0a5406d160 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a700aedc0 0x7f0a700c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 --2- 172.21.15.23:0/1110786819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0a700c8bd0 0x7f0a700c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 -- 172.21.15.23:0/1110786819 >> 172.21.15.23:0/1110786819 conn(0x7f0a7006fb70 msgr2=0x7f0a701523d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 -- 172.21.15.23:0/1110786819 shutdown_connections 2023-06-22T08:24:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:01.101+0000 7f0a777ff700 1 -- 172.21.15.23:0/1110786819 wait complete. 2023-06-22T08:24:01.108 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 188 2023-06-22T08:24:01.122 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=3, replacing gid=5995 2023-06-22T08:24:01.122 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.f 2023-06-22T08:24:02.991 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T08:24:02.989+0000 7f6838c81700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-06-22T08:24:05.265 INFO:tasks.workunit.client.0.smithi023.stderr:+ do_tarball coreutils_8.5.orig.tar.gz z coreutils-8.5 2023-06-22T08:24:05.265 INFO:tasks.workunit.client.0.smithi023.stderr:+ wget http://download.ceph.com/qa/coreutils_8.5.orig.tar.gz 2023-06-22T08:24:05.271 INFO:tasks.workunit.client.0.smithi023.stderr:--2023-06-22 08:24:05-- http://download.ceph.com/qa/coreutils_8.5.orig.tar.gz 2023-06-22T08:24:05.272 INFO:tasks.workunit.client.0.smithi023.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2023-06-22T08:24:05.293 INFO:tasks.workunit.client.0.smithi023.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2023-06-22T08:24:05.314 INFO:tasks.workunit.client.0.smithi023.stderr:HTTP request sent, awaiting response... 200 OK 2023-06-22T08:24:05.314 INFO:tasks.workunit.client.0.smithi023.stderr:Length: 10740866 (10M) [application/octet-stream] 2023-06-22T08:24:05.316 INFO:tasks.workunit.client.0.smithi023.stderr:Saving to: ‘coreutils_8.5.orig.tar.gz’ 2023-06-22T08:24:05.316 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-06-22T08:24:05.355 INFO:tasks.workunit.client.0.smithi023.stderr: 0K .......... .......... .......... .......... .......... 0% 1.24M 8s 2023-06-22T08:24:05.376 INFO:tasks.workunit.client.0.smithi023.stderr: 50K .......... .......... .......... .......... .......... 0% 2.41M 6s 2023-06-22T08:24:05.376 INFO:tasks.workunit.client.0.smithi023.stderr: 100K .......... .......... .......... .......... .......... 1% 169M 4s 2023-06-22T08:24:05.398 INFO:tasks.workunit.client.0.smithi023.stderr: 150K .......... .......... .......... .......... .......... 1% 2.30M 4s 2023-06-22T08:24:05.399 INFO:tasks.workunit.client.0.smithi023.stderr: 200K .......... .......... .......... .......... .......... 2% 181M 3s 2023-06-22T08:24:05.399 INFO:tasks.workunit.client.0.smithi023.stderr: 250K .......... .......... .......... .......... .......... 2% 202M 3s 2023-06-22T08:24:05.416 INFO:tasks.workunit.client.0.smithi023.stderr: 300K .......... .......... .......... .......... .......... 3% 2.59M 3s 2023-06-22T08:24:05.419 INFO:tasks.workunit.client.0.smithi023.stderr: 350K .......... .......... .......... .......... .......... 3% 52.4M 3s 2023-06-22T08:24:05.419 INFO:tasks.workunit.client.0.smithi023.stderr: 400K .......... .......... .......... .......... .......... 4% 152M 2s 2023-06-22T08:24:05.420 INFO:tasks.workunit.client.0.smithi023.stderr: 450K .......... .......... .......... .......... .......... 4% 247M 2s 2023-06-22T08:24:05.420 INFO:tasks.workunit.client.0.smithi023.stderr: 500K .......... .......... .......... .......... .......... 5% 192M 2s 2023-06-22T08:24:05.439 INFO:tasks.workunit.client.0.smithi023.stderr: 550K .......... .......... .......... .......... .......... 5% 2.51M 2s 2023-06-22T08:24:05.439 INFO:tasks.workunit.client.0.smithi023.stderr: 600K .......... .......... .......... .......... .......... 6% 140M 2s 2023-06-22T08:24:05.439 INFO:tasks.workunit.client.0.smithi023.stderr: 650K .......... .......... .......... .......... .......... 6% 210M 2s 2023-06-22T08:24:05.440 INFO:tasks.workunit.client.0.smithi023.stderr: 700K .......... .......... .......... .......... .......... 7% 256M 2s 2023-06-22T08:24:05.460 INFO:tasks.workunit.client.0.smithi023.stderr: 750K .......... .......... .......... .......... .......... 7% 2.49M 2s 2023-06-22T08:24:05.460 INFO:tasks.workunit.client.0.smithi023.stderr: 800K .......... .......... .......... .......... .......... 8% 209M 2s 2023-06-22T08:24:05.460 INFO:tasks.workunit.client.0.smithi023.stderr: 850K .......... .......... .......... .......... .......... 8% 105M 2s 2023-06-22T08:24:05.461 INFO:tasks.workunit.client.0.smithi023.stderr: 900K .......... .......... .......... .......... .......... 9% 209M 1s 2023-06-22T08:24:05.461 INFO:tasks.workunit.client.0.smithi023.stderr: 950K .......... .......... .......... .......... .......... 9% 250M 1s 2023-06-22T08:24:05.480 INFO:tasks.workunit.client.0.smithi023.stderr: 1000K .......... .......... .......... .......... .......... 10% 2.53M 1s 2023-06-22T08:24:05.480 INFO:tasks.workunit.client.0.smithi023.stderr: 1050K .......... .......... .......... .......... .......... 10% 180M 1s 2023-06-22T08:24:05.481 INFO:tasks.workunit.client.0.smithi023.stderr: 1100K .......... .......... .......... .......... .......... 10% 187M 1s 2023-06-22T08:24:05.481 INFO:tasks.workunit.client.0.smithi023.stderr: 1150K .......... .......... .......... .......... .......... 11% 209M 1s 2023-06-22T08:24:05.481 INFO:tasks.workunit.client.0.smithi023.stderr: 1200K .......... .......... .......... .......... .......... 11% 228M 1s 2023-06-22T08:24:05.501 INFO:tasks.workunit.client.0.smithi023.stderr: 1250K .......... .......... .......... .......... .......... 12% 2.52M 1s 2023-06-22T08:24:05.501 INFO:tasks.workunit.client.0.smithi023.stderr: 1300K .......... .......... .......... .......... .......... 12% 220M 1s 2023-06-22T08:24:05.501 INFO:tasks.workunit.client.0.smithi023.stderr: 1350K .......... .......... .......... .......... .......... 13% 109M 1s 2023-06-22T08:24:05.502 INFO:tasks.workunit.client.0.smithi023.stderr: 1400K .......... .......... .......... .......... .......... 13% 200M 1s 2023-06-22T08:24:05.502 INFO:tasks.workunit.client.0.smithi023.stderr: 1450K .......... .......... .......... .......... .......... 14% 216M 1s 2023-06-22T08:24:05.521 INFO:tasks.workunit.client.0.smithi023.stderr: 1500K .......... .......... .......... .......... .......... 14% 2.55M 1s 2023-06-22T08:24:05.521 INFO:tasks.workunit.client.0.smithi023.stderr: 1550K .......... .......... .......... .......... .......... 15% 184M 1s 2023-06-22T08:24:05.522 INFO:tasks.workunit.client.0.smithi023.stderr: 1600K .......... .......... .......... .......... .......... 15% 134M 1s 2023-06-22T08:24:05.522 INFO:tasks.workunit.client.0.smithi023.stderr: 1650K .......... .......... .......... .......... .......... 16% 177M 1s 2023-06-22T08:24:05.522 INFO:tasks.workunit.client.0.smithi023.stderr: 1700K .......... .......... .......... .......... .......... 16% 232M 1s 2023-06-22T08:24:05.541 INFO:tasks.workunit.client.0.smithi023.stderr: 1750K .......... .......... .......... .......... .......... 17% 2.53M 1s 2023-06-22T08:24:05.542 INFO:tasks.workunit.client.0.smithi023.stderr: 1800K .......... .......... .......... .......... .......... 17% 222M 1s 2023-06-22T08:24:05.542 INFO:tasks.workunit.client.0.smithi023.stderr: 1850K .......... .......... .......... .......... .......... 18% 209M 1s 2023-06-22T08:24:05.542 INFO:tasks.workunit.client.0.smithi023.stderr: 1900K .......... .......... .......... .......... .......... 18% 211M 1s 2023-06-22T08:24:05.542 INFO:tasks.workunit.client.0.smithi023.stderr: 1950K .......... .......... .......... .......... .......... 19% 187M 1s 2023-06-22T08:24:05.542 INFO:tasks.workunit.client.0.smithi023.stderr: 2000K .......... .......... .......... .......... .......... 19% 240M 1s 2023-06-22T08:24:05.563 INFO:tasks.workunit.client.0.smithi023.stderr: 2050K .......... .......... .......... .......... .......... 20% 2.54M 1s 2023-06-22T08:24:05.563 INFO:tasks.workunit.client.0.smithi023.stderr: 2100K .......... .......... .......... .......... .......... 20% 222M 1s 2023-06-22T08:24:05.564 INFO:tasks.workunit.client.0.smithi023.stderr: 2150K .......... .......... .......... .......... .......... 20% 208M 1s 2023-06-22T08:24:05.564 INFO:tasks.workunit.client.0.smithi023.stderr: 2200K .......... .......... .......... .......... .......... 21% 185M 1s 2023-06-22T08:24:05.564 INFO:tasks.workunit.client.0.smithi023.stderr: 2250K .......... .......... .......... .......... .......... 21% 228M 1s 2023-06-22T08:24:05.564 INFO:tasks.workunit.client.0.smithi023.stderr: 2300K .......... .......... .......... .......... .......... 22% 228M 1s 2023-06-22T08:24:05.587 INFO:tasks.workunit.client.0.smithi023.stderr: 2350K .......... .......... .......... .......... .......... 22% 2.52M 1s 2023-06-22T08:24:05.587 INFO:tasks.workunit.client.0.smithi023.stderr: 2400K .......... .......... .......... .......... .......... 23% 163M 1s 2023-06-22T08:24:05.588 INFO:tasks.workunit.client.0.smithi023.stderr: 2450K .......... .......... .......... .......... .......... 23% 130M 1s 2023-06-22T08:24:05.588 INFO:tasks.workunit.client.0.smithi023.stderr: 2500K .......... .......... .......... .......... .......... 24% 237M 1s 2023-06-22T08:24:05.588 INFO:tasks.workunit.client.0.smithi023.stderr: 2550K .......... .......... .......... .......... .......... 24% 206M 1s 2023-06-22T08:24:05.588 INFO:tasks.workunit.client.0.smithi023.stderr: 2600K .......... .......... .......... .......... .......... 25% 255M 1s 2023-06-22T08:24:05.604 INFO:tasks.workunit.client.0.smithi023.stderr: 2650K .......... .......... .......... .......... .......... 25% 2.58M 1s 2023-06-22T08:24:05.604 INFO:tasks.workunit.client.0.smithi023.stderr: 2700K .......... .......... .......... .......... .......... 26% 213M 1s 2023-06-22T08:24:05.604 INFO:tasks.workunit.client.0.smithi023.stderr: 2750K .......... .......... .......... .......... .......... 26% 194M 1s 2023-06-22T08:24:05.605 INFO:tasks.workunit.client.0.smithi023.stderr: 2800K .......... .......... .......... .......... .......... 27% 214M 1s 2023-06-22T08:24:05.605 INFO:tasks.workunit.client.0.smithi023.stderr: 2850K .......... .......... .......... .......... .......... 27% 211M 1s 2023-06-22T08:24:05.605 INFO:tasks.workunit.client.0.smithi023.stderr: 2900K .......... .......... .......... .......... .......... 28% 237M 1s 2023-06-22T08:24:05.605 INFO:tasks.workunit.client.0.smithi023.stderr: 2950K .......... .......... .......... .......... .......... 28% 217M 1s 2023-06-22T08:24:05.628 INFO:tasks.workunit.client.0.smithi023.stderr: 3000K .......... .......... .......... .......... .......... 29% 2.57M 1s 2023-06-22T08:24:05.628 INFO:tasks.workunit.client.0.smithi023.stderr: 3050K .......... .......... .......... .......... .......... 29% 203M 1s 2023-06-22T08:24:05.629 INFO:tasks.workunit.client.0.smithi023.stderr: 3100K .......... .......... .......... .......... .......... 30% 201M 1s 2023-06-22T08:24:05.629 INFO:tasks.workunit.client.0.smithi023.stderr: 3150K .......... .......... .......... .......... .......... 30% 189M 1s 2023-06-22T08:24:05.629 INFO:tasks.workunit.client.0.smithi023.stderr: 3200K .......... .......... .......... .......... .......... 30% 248M 1s 2023-06-22T08:24:05.630 INFO:tasks.workunit.client.0.smithi023.stderr: 3250K .......... .......... .......... .......... .......... 31% 250M 1s 2023-06-22T08:24:05.644 INFO:tasks.workunit.client.0.smithi023.stderr: 3300K .......... .......... .......... .......... .......... 31% 2.80M 1s 2023-06-22T08:24:05.644 INFO:tasks.workunit.client.0.smithi023.stderr: 3350K .......... .......... .......... .......... .......... 32% 27.8M 1s 2023-06-22T08:24:05.645 INFO:tasks.workunit.client.0.smithi023.stderr: 3400K .......... .......... .......... .......... .......... 32% 184M 1s 2023-06-22T08:24:05.645 INFO:tasks.workunit.client.0.smithi023.stderr: 3450K .......... .......... .......... .......... .......... 33% 228M 1s 2023-06-22T08:24:05.645 INFO:tasks.workunit.client.0.smithi023.stderr: 3500K .......... .......... .......... .......... .......... 33% 217M 1s 2023-06-22T08:24:05.645 INFO:tasks.workunit.client.0.smithi023.stderr: 3550K .......... .......... .......... .......... .......... 34% 239M 1s 2023-06-22T08:24:05.645 INFO:tasks.workunit.client.0.smithi023.stderr: 3600K .......... .......... .......... .......... .......... 34% 31.3M 1s 2023-06-22T08:24:05.646 INFO:tasks.workunit.client.0.smithi023.stderr: 3650K .......... .......... .......... .......... .......... 35% 28.3M 1s 2023-06-22T08:24:05.664 INFO:tasks.workunit.client.0.smithi023.stderr: 3700K .......... .......... .......... .......... .......... 35% 3.04M 1s 2023-06-22T08:24:05.664 INFO:tasks.workunit.client.0.smithi023.stderr: 3750K .......... .......... .......... .......... .......... 36% 212M 1s 2023-06-22T08:24:05.665 INFO:tasks.workunit.client.0.smithi023.stderr: 3800K .......... .......... .......... .......... .......... 36% 211M 1s 2023-06-22T08:24:05.665 INFO:tasks.workunit.client.0.smithi023.stderr: 3850K .......... .......... .......... .......... .......... 37% 194M 1s 2023-06-22T08:24:05.665 INFO:tasks.workunit.client.0.smithi023.stderr: 3900K .......... .......... .......... .......... .......... 37% 218M 1s 2023-06-22T08:24:05.665 INFO:tasks.workunit.client.0.smithi023.stderr: 3950K .......... .......... .......... .......... .......... 38% 59.4M 1s 2023-06-22T08:24:05.665 INFO:tasks.workunit.client.0.smithi023.stderr: 4000K .......... .......... .......... .......... .......... 38% 35.8M 1s 2023-06-22T08:24:05.686 INFO:tasks.workunit.client.0.smithi023.stderr: 4050K .......... .......... .......... .......... .......... 39% 2.87M 1s 2023-06-22T08:24:05.686 INFO:tasks.workunit.client.0.smithi023.stderr: 4100K .......... .......... .......... .......... .......... 39% 213M 1s 2023-06-22T08:24:05.686 INFO:tasks.workunit.client.0.smithi023.stderr: 4150K .......... .......... .......... .......... .......... 40% 182M 1s 2023-06-22T08:24:05.686 INFO:tasks.workunit.client.0.smithi023.stderr: 4200K .......... .......... .......... .......... .......... 40% 211M 1s 2023-06-22T08:24:05.687 INFO:tasks.workunit.client.0.smithi023.stderr: 4250K .......... .......... .......... .......... .......... 40% 239M 1s 2023-06-22T08:24:05.687 INFO:tasks.workunit.client.0.smithi023.stderr: 4300K .......... .......... .......... .......... .......... 41% 237M 1s 2023-06-22T08:24:05.687 INFO:tasks.workunit.client.0.smithi023.stderr: 4350K .......... .......... .......... .......... .......... 41% 32.1M 1s 2023-06-22T08:24:05.687 INFO:tasks.workunit.client.0.smithi023.stderr: 4400K .......... .......... .......... .......... .......... 42% 37.3M 1s 2023-06-22T08:24:05.707 INFO:tasks.workunit.client.0.smithi023.stderr: 4450K .......... .......... .......... .......... .......... 42% 2.99M 1s 2023-06-22T08:24:05.707 INFO:tasks.workunit.client.0.smithi023.stderr: 4500K .......... .......... .......... .......... .......... 43% 220M 1s 2023-06-22T08:24:05.707 INFO:tasks.workunit.client.0.smithi023.stderr: 4550K .......... .......... .......... .......... .......... 43% 142M 0s 2023-06-22T08:24:05.708 INFO:tasks.workunit.client.0.smithi023.stderr: 4600K .......... .......... .......... .......... .......... 44% 175M 0s 2023-06-22T08:24:05.708 INFO:tasks.workunit.client.0.smithi023.stderr: 4650K .......... .......... .......... .......... .......... 44% 218M 0s 2023-06-22T08:24:05.708 INFO:tasks.workunit.client.0.smithi023.stderr: 4700K .......... .......... .......... .......... .......... 45% 248M 0s 2023-06-22T08:24:05.708 INFO:tasks.workunit.client.0.smithi023.stderr: 4750K .......... .......... .......... .......... .......... 45% 34.9M 0s 2023-06-22T08:24:05.708 INFO:tasks.workunit.client.0.smithi023.stderr: 4800K .......... .......... .......... .......... .......... 46% 34.0M 0s 2023-06-22T08:24:05.709 INFO:tasks.workunit.client.0.smithi023.stderr: 4850K .......... .......... .......... .......... .......... 46% 17.4M 0s 2023-06-22T08:24:05.725 INFO:tasks.workunit.client.0.smithi023.stderr: 4900K .......... .......... .......... .......... .......... 47% 3.58M 0s 2023-06-22T08:24:05.726 INFO:tasks.workunit.client.0.smithi023.stderr: 4950K .......... .......... .......... .......... .......... 47% 217M 0s 2023-06-22T08:24:05.726 INFO:tasks.workunit.client.0.smithi023.stderr: 5000K .......... .......... .......... .......... .......... 48% 206M 0s 2023-06-22T08:24:05.726 INFO:tasks.workunit.client.0.smithi023.stderr: 5050K .......... .......... .......... .......... .......... 48% 223M 0s 2023-06-22T08:24:05.726 INFO:tasks.workunit.client.0.smithi023.stderr: 5100K .......... .......... .......... .......... .......... 49% 189M 0s 2023-06-22T08:24:05.726 INFO:tasks.workunit.client.0.smithi023.stderr: 5150K .......... .......... .......... .......... .......... 49% 37.7M 0s 2023-06-22T08:24:05.726 INFO:tasks.workunit.client.0.smithi023.stderr: 5200K .......... .......... .......... .......... .......... 50% 46.2M 0s 2023-06-22T08:24:05.727 INFO:tasks.workunit.client.0.smithi023.stderr: 5250K .......... .......... .......... .......... .......... 50% 21.0M 0s 2023-06-22T08:24:05.746 INFO:tasks.workunit.client.0.smithi023.stderr: 5300K .......... .......... .......... .......... .......... 51% 3.32M 0s 2023-06-22T08:24:05.746 INFO:tasks.workunit.client.0.smithi023.stderr: 5350K .......... .......... .......... .......... .......... 51% 194M 0s 2023-06-22T08:24:05.746 INFO:tasks.workunit.client.0.smithi023.stderr: 5400K .......... .......... .......... .......... .......... 51% 207M 0s 2023-06-22T08:24:05.746 INFO:tasks.workunit.client.0.smithi023.stderr: 5450K .......... .......... .......... .......... .......... 52% 202M 0s 2023-06-22T08:24:05.746 INFO:tasks.workunit.client.0.smithi023.stderr: 5500K .......... .......... .......... .......... .......... 52% 217M 0s 2023-06-22T08:24:05.747 INFO:tasks.workunit.client.0.smithi023.stderr: 5550K .......... .......... .......... .......... .......... 53% 257M 0s 2023-06-22T08:24:05.747 INFO:tasks.workunit.client.0.smithi023.stderr: 5600K .......... .......... .......... .......... .......... 53% 37.0M 0s 2023-06-22T08:24:05.748 INFO:tasks.workunit.client.0.smithi023.stderr: 5650K .......... .......... .......... .......... .......... 54% 15.0M 0s 2023-06-22T08:24:05.751 INFO:tasks.workunit.client.0.smithi023.stderr: 5700K .......... .......... .......... .......... .......... 54% 21.0M 0s 2023-06-22T08:24:05.753 INFO:tasks.workunit.client.0.smithi023.stderr: 5750K .......... .......... .......... .......... .......... 55% 19.9M 0s 2023-06-22T08:24:05.770 INFO:tasks.workunit.client.0.smithi023.stderr: 5800K .......... .......... .......... .......... .......... 55% 4.32M 0s 2023-06-22T08:24:05.770 INFO:tasks.workunit.client.0.smithi023.stderr: 5850K .......... .......... .......... .......... .......... 56% 26.9M 0s 2023-06-22T08:24:05.771 INFO:tasks.workunit.client.0.smithi023.stderr: 5900K .......... .......... .......... .......... .......... 56% 23.7M 0s 2023-06-22T08:24:05.771 INFO:tasks.workunit.client.0.smithi023.stderr: 5950K .......... .......... .......... .......... .......... 57% 36.6M 0s 2023-06-22T08:24:05.773 INFO:tasks.workunit.client.0.smithi023.stderr: 6000K .......... .......... .......... .......... .......... 57% 23.9M 0s 2023-06-22T08:24:05.777 INFO:tasks.workunit.client.0.smithi023.stderr: 6050K .......... .......... .......... .......... .......... 58% 49.3M 0s 2023-06-22T08:24:05.777 INFO:tasks.workunit.client.0.smithi023.stderr: 6100K .......... .......... .......... .......... .......... 58% 16.0M 0s 2023-06-22T08:24:05.778 INFO:tasks.workunit.client.0.smithi023.stderr: 6150K .......... .......... .......... .......... .......... 59% 23.5M 0s 2023-06-22T08:24:05.779 INFO:tasks.workunit.client.0.smithi023.stderr: 6200K .......... .......... .......... .......... .......... 59% 28.6M 0s 2023-06-22T08:24:05.784 INFO:tasks.workunit.client.0.smithi023.stderr: 6250K .......... .......... .......... .......... .......... 60% 9.78M 0s 2023-06-22T08:24:05.785 INFO:tasks.workunit.client.0.smithi023.stderr: 6300K .......... .......... .......... .......... .......... 60% 41.9M 0s 2023-06-22T08:24:05.787 INFO:tasks.workunit.client.0.smithi023.stderr: 6350K .......... .......... .......... .......... .......... 61% 39.6M 0s 2023-06-22T08:24:05.789 INFO:tasks.workunit.client.0.smithi023.stderr: 6400K .......... .......... .......... .......... .......... 61% 25.0M 0s 2023-06-22T08:24:05.790 INFO:tasks.workunit.client.0.smithi023.stderr: 6450K .......... .......... .......... .......... .......... 61% 36.7M 0s 2023-06-22T08:24:05.792 INFO:tasks.workunit.client.0.smithi023.stderr: 6500K .......... .......... .......... .......... .......... 62% 24.8M 0s 2023-06-22T08:24:05.793 INFO:tasks.workunit.client.0.smithi023.stderr: 6550K .......... .......... .......... .......... .......... 62% 50.0M 0s 2023-06-22T08:24:05.796 INFO:tasks.workunit.client.0.smithi023.stderr: 6600K .......... .......... .......... .......... .......... 63% 16.6M 0s 2023-06-22T08:24:05.799 INFO:tasks.workunit.client.0.smithi023.stderr: 6650K .......... .......... .......... .......... .......... 63% 17.1M 0s 2023-06-22T08:24:05.801 INFO:tasks.workunit.client.0.smithi023.stderr: 6700K .......... .......... .......... .......... .......... 64% 36.7M 0s 2023-06-22T08:24:05.805 INFO:tasks.workunit.client.0.smithi023.stderr: 6750K .......... .......... .......... .......... .......... 64% 10.8M 0s 2023-06-22T08:24:05.806 INFO:tasks.workunit.client.0.smithi023.stderr: 6800K .......... .......... .......... .......... .......... 65% 44.9M 0s 2023-06-22T08:24:05.808 INFO:tasks.workunit.client.0.smithi023.stderr: 6850K .......... .......... .......... .......... .......... 65% 37.2M 0s 2023-06-22T08:24:05.809 INFO:tasks.workunit.client.0.smithi023.stderr: 6900K .......... .......... .......... .......... .......... 66% 24.5M 0s 2023-06-22T08:24:05.810 INFO:tasks.workunit.client.0.smithi023.stderr: 6950K .......... .......... .......... .......... .......... 66% 38.6M 0s 2023-06-22T08:24:05.812 INFO:tasks.workunit.client.0.smithi023.stderr: 7000K .......... .......... .......... .......... .......... 67% 25.5M 0s 2023-06-22T08:24:05.813 INFO:tasks.workunit.client.0.smithi023.stderr: 7050K .......... .......... .......... .......... .......... 67% 56.3M 0s 2023-06-22T08:24:05.817 INFO:tasks.workunit.client.0.smithi023.stderr: 7100K .......... .......... .......... .......... .......... 68% 15.9M 0s 2023-06-22T08:24:05.817 INFO:tasks.workunit.client.0.smithi023.stderr: 7150K .......... .......... .......... .......... .......... 68% 35.7M 0s 2023-06-22T08:24:05.819 INFO:tasks.workunit.client.0.smithi023.stderr: 7200K .......... .......... .......... .......... .......... 69% 27.6M 0s 2023-06-22T08:24:05.823 INFO:tasks.workunit.client.0.smithi023.stderr: 7250K .......... .......... .......... .......... .......... 69% 11.4M 0s 2023-06-22T08:24:05.825 INFO:tasks.workunit.client.0.smithi023.stderr: 7300K .......... .......... .......... .......... .......... 70% 31.3M 0s 2023-06-22T08:24:05.828 INFO:tasks.workunit.client.0.smithi023.stderr: 7350K .......... .......... .......... .......... .......... 70% 25.2M 0s 2023-06-22T08:24:05.829 INFO:tasks.workunit.client.0.smithi023.stderr: 7400K .......... .......... .......... .......... .......... 71% 24.2M 0s 2023-06-22T08:24:05.830 INFO:tasks.workunit.client.0.smithi023.stderr: 7450K .......... .......... .......... .......... .......... 71% 168M 0s 2023-06-22T08:24:05.832 INFO:tasks.workunit.client.0.smithi023.stderr: 7500K .......... .......... .......... .......... .......... 71% 16.6M 0s 2023-06-22T08:24:05.833 INFO:tasks.workunit.client.0.smithi023.stderr: 7550K .......... .......... .......... .......... .......... 72% 42.9M 0s 2023-06-22T08:24:05.837 INFO:tasks.workunit.client.0.smithi023.stderr: 7600K .......... .......... .......... .......... .......... 72% 14.7M 0s 2023-06-22T08:24:05.840 INFO:tasks.workunit.client.0.smithi023.stderr: 7650K .......... .......... .......... .......... .......... 73% 17.0M 0s 2023-06-22T08:24:05.841 INFO:tasks.workunit.client.0.smithi023.stderr: 7700K .......... .......... .......... .......... .......... 73% 31.9M 0s 2023-06-22T08:24:05.844 INFO:tasks.workunit.client.0.smithi023.stderr: 7750K .......... .......... .......... .......... .......... 74% 15.4M 0s 2023-06-22T08:24:05.848 INFO:tasks.workunit.client.0.smithi023.stderr: 7800K .......... .......... .......... .......... .......... 74% 14.2M 0s 2023-06-22T08:24:05.850 INFO:tasks.workunit.client.0.smithi023.stderr: 7850K .......... .......... .......... .......... .......... 75% 26.4M 0s 2023-06-22T08:24:05.851 INFO:tasks.workunit.client.0.smithi023.stderr: 7900K .......... .......... .......... .......... .......... 75% 33.2M 0s 2023-06-22T08:24:05.853 INFO:tasks.workunit.client.0.smithi023.stderr: 7950K .......... .......... .......... .......... .......... 76% 25.0M 0s 2023-06-22T08:24:05.855 INFO:tasks.workunit.client.0.smithi023.stderr: 8000K .......... .......... .......... .......... .......... 76% 23.9M 0s 2023-06-22T08:24:05.856 INFO:tasks.workunit.client.0.smithi023.stderr: 8050K .......... .......... .......... .......... .......... 77% 48.6M 0s 2023-06-22T08:24:05.857 INFO:tasks.workunit.client.0.smithi023.stderr: 8100K .......... .......... .......... .......... .......... 77% 29.3M 0s 2023-06-22T08:24:05.859 INFO:tasks.workunit.client.0.smithi023.stderr: 8150K .......... .......... .......... .......... .......... 78% 24.6M 0s 2023-06-22T08:24:05.862 INFO:tasks.workunit.client.0.smithi023.stderr: 8200K .......... .......... .......... .......... .......... 78% 22.9M 0s 2023-06-22T08:24:05.865 INFO:tasks.workunit.client.0.smithi023.stderr: 8250K .......... .......... .......... .......... .......... 79% 30.8M 0s 2023-06-22T08:24:05.865 INFO:tasks.workunit.client.0.smithi023.stderr: 8300K .......... .......... .......... .......... .......... 79% 49.9M 0s 2023-06-22T08:24:05.868 INFO:tasks.workunit.client.0.smithi023.stderr: 8350K .......... .......... .......... .......... .......... 80% 13.4M 0s 2023-06-22T08:24:05.870 INFO:tasks.workunit.client.0.smithi023.stderr: 8400K .......... .......... .......... .......... .......... 80% 28.1M 0s 2023-06-22T08:24:05.870 INFO:tasks.workunit.client.0.smithi023.stderr: 8450K .......... .......... .......... .......... .......... 81% 171M 0s 2023-06-22T08:24:05.873 INFO:tasks.workunit.client.0.smithi023.stderr: 8500K .......... .......... .......... .......... .......... 81% 14.1M 0s 2023-06-22T08:24:05.875 INFO:tasks.workunit.client.0.smithi023.stderr: 8550K .......... .......... .......... .......... .......... 81% 24.2M 0s 2023-06-22T08:24:05.878 INFO:tasks.workunit.client.0.smithi023.stderr: 8600K .......... .......... .......... .......... .......... 82% 17.4M 0s 2023-06-22T08:24:05.881 INFO:tasks.workunit.client.0.smithi023.stderr: 8650K .......... .......... .......... .......... .......... 82% 18.0M 0s 2023-06-22T08:24:05.883 INFO:tasks.workunit.client.0.smithi023.stderr: 8700K .......... .......... .......... .......... .......... 83% 20.6M 0s 2023-06-22T08:24:05.886 INFO:tasks.workunit.client.0.smithi023.stderr: 8750K .......... .......... .......... .......... .......... 83% 17.6M 0s 2023-06-22T08:24:05.888 INFO:tasks.workunit.client.0.smithi023.stderr: 8800K .......... .......... .......... .......... .......... 84% 26.9M 0s 2023-06-22T08:24:05.892 INFO:tasks.workunit.client.0.smithi023.stderr: 8850K .......... .......... .......... .......... .......... 84% 11.1M 0s 2023-06-22T08:24:05.897 INFO:tasks.workunit.client.0.smithi023.stderr: 8900K .......... .......... .......... .......... .......... 85% 11.8M 0s 2023-06-22T08:24:05.899 INFO:tasks.workunit.client.0.smithi023.stderr: 8950K .......... .......... .......... .......... .......... 85% 16.7M 0s 2023-06-22T08:24:05.904 INFO:tasks.workunit.client.0.smithi023.stderr: 9000K .......... .......... .......... .......... .......... 86% 11.4M 0s 2023-06-22T08:24:05.905 INFO:tasks.workunit.client.0.smithi023.stderr: 9050K .......... .......... .......... .......... .......... 86% 46.5M 0s 2023-06-22T08:24:05.907 INFO:tasks.workunit.client.0.smithi023.stderr: 9100K .......... .......... .......... .......... .......... 87% 24.5M 0s 2023-06-22T08:24:05.909 INFO:tasks.workunit.client.0.smithi023.stderr: 9150K .......... .......... .......... .......... .......... 87% 24.5M 0s 2023-06-22T08:24:05.910 INFO:tasks.workunit.client.0.smithi023.stderr: 9200K .......... .......... .......... .......... .......... 88% 42.9M 0s 2023-06-22T08:24:05.912 INFO:tasks.workunit.client.0.smithi023.stderr: 9250K .......... .......... .......... .......... .......... 88% 21.5M 0s 2023-06-22T08:24:05.914 INFO:tasks.workunit.client.0.smithi023.stderr: 9300K .......... .......... .......... .......... .......... 89% 24.3M 0s 2023-06-22T08:24:05.917 INFO:tasks.workunit.client.0.smithi023.stderr: 9350K .......... .......... .......... .......... .......... 89% 18.0M 0s 2023-06-22T08:24:05.920 INFO:tasks.workunit.client.0.smithi023.stderr: 9400K .......... .......... .......... .......... .......... 90% 16.8M 0s 2023-06-22T08:24:05.921 INFO:tasks.workunit.client.0.smithi023.stderr: 9450K .......... .......... .......... .......... .......... 90% 30.5M 0s 2023-06-22T08:24:05.923 INFO:tasks.workunit.client.0.smithi023.stderr: 9500K .......... .......... .......... .......... .......... 91% 25.2M 0s 2023-06-22T08:24:05.926 INFO:tasks.workunit.client.0.smithi023.stderr: 9550K .......... .......... .......... .......... .......... 91% 42.5M 0s 2023-06-22T08:24:05.927 INFO:tasks.workunit.client.0.smithi023.stderr: 9600K .......... .......... .......... .......... .......... 92% 25.8M 0s 2023-06-22T08:24:05.928 INFO:tasks.workunit.client.0.smithi023.stderr: 9650K .......... .......... .......... .......... .......... 92% 24.5M 0s 2023-06-22T08:24:05.930 INFO:tasks.workunit.client.0.smithi023.stderr: 9700K .......... .......... .......... .......... .......... 92% 33.0M 0s 2023-06-22T08:24:05.932 INFO:tasks.workunit.client.0.smithi023.stderr: 9750K .......... .......... .......... .......... .......... 93% 26.7M 0s 2023-06-22T08:24:05.934 INFO:tasks.workunit.client.0.smithi023.stderr: 9800K .......... .......... .......... .......... .......... 93% 21.7M 0s 2023-06-22T08:24:05.935 INFO:tasks.workunit.client.0.smithi023.stderr: 9850K .......... .......... .......... .......... .......... 94% 39.1M 0s 2023-06-22T08:24:05.937 INFO:tasks.workunit.client.0.smithi023.stderr: 9900K .......... .......... .......... .......... .......... 94% 29.1M 0s 2023-06-22T08:24:05.939 INFO:tasks.workunit.client.0.smithi023.stderr: 9950K .......... .......... .......... .......... .......... 95% 26.7M 0s 2023-06-22T08:24:05.940 INFO:tasks.workunit.client.0.smithi023.stderr: 10000K .......... .......... .......... .......... .......... 95% 30.5M 0s 2023-06-22T08:24:05.942 INFO:tasks.workunit.client.0.smithi023.stderr: 10050K .......... .......... .......... .......... .......... 96% 24.6M 0s 2023-06-22T08:24:05.943 INFO:tasks.workunit.client.0.smithi023.stderr: 10100K .......... .......... .......... .......... .......... 96% 35.5M 0s 2023-06-22T08:24:05.945 INFO:tasks.workunit.client.0.smithi023.stderr: 10150K .......... .......... .......... .......... .......... 97% 29.0M 0s 2023-06-22T08:24:05.948 INFO:tasks.workunit.client.0.smithi023.stderr: 10200K .......... .......... .......... .......... .......... 97% 23.3M 0s 2023-06-22T08:24:05.949 INFO:tasks.workunit.client.0.smithi023.stderr: 10250K .......... .......... .......... .......... .......... 98% 26.1M 0s 2023-06-22T08:24:05.952 INFO:tasks.workunit.client.0.smithi023.stderr: 10300K .......... .......... .......... .......... .......... 98% 16.2M 0s 2023-06-22T08:24:05.954 INFO:tasks.workunit.client.0.smithi023.stderr: 10350K .......... .......... .......... .......... .......... 99% 35.8M 0s 2023-06-22T08:24:05.958 INFO:tasks.workunit.client.0.smithi023.stderr: 10400K .......... .......... .......... .......... .......... 99% 12.4M 0s 2023-06-22T08:24:05.958 INFO:tasks.workunit.client.0.smithi023.stderr: 10450K .......... .......... .......... ......... 100% 41.5M=0.6s 2023-06-22T08:24:05.959 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-06-22T08:24:05.959 INFO:tasks.workunit.client.0.smithi023.stderr:2023-06-22 08:24:05 (15.9 MB/s) - ‘coreutils_8.5.orig.tar.gz’ saved [10740866/10740866] 2023-06-22T08:24:05.959 INFO:tasks.workunit.client.0.smithi023.stderr: 2023-06-22T08:24:05.960 INFO:tasks.workunit.client.0.smithi023.stderr:+ tar xvfz coreutils_8.5.orig.tar.gz 2023-06-22T08:24:05.981 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/ 2023-06-22T08:24:05.982 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/maint.mk 2023-06-22T08:24:05.983 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/THANKS 2023-06-22T08:24:05.983 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/ABOUT-NLS 2023-06-22T08:24:05.984 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/ 2023-06-22T08:24:05.986 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/textutils/ 2023-06-22T08:24:05.987 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/textutils/ChangeLog 2023-06-22T08:24:05.990 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/textutils/NEWS 2023-06-22T08:24:05.990 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/fileutils/ 2023-06-22T08:24:05.991 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/fileutils/ChangeLog 2023-06-22T08:24:05.994 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/fileutils/ChangeLog-1997 2023-06-22T08:24:05.995 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/fileutils/NEWS 2023-06-22T08:24:05.996 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/sh-utils/ 2023-06-22T08:24:05.997 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/sh-utils/ChangeLog 2023-06-22T08:24:05.999 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/sh-utils/NEWS 2023-06-22T08:24:06.000 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/old/sh-utils/ChangeLog.0 2023-06-22T08:24:06.000 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_prohibit_strcmp 2023-06-22T08:24:06.000 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/ 2023-06-22T08:24:06.002 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/config.rpath 2023-06-22T08:24:06.004 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/cvsu 2023-06-22T08:24:06.005 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/mdate-sh 2023-06-22T08:24:06.005 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/gitlog-to-changelog 2023-06-22T08:24:06.006 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/gnu-web-doc-update 2023-06-22T08:24:06.006 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/useless-if-before-free 2023-06-22T08:24:06.006 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/config.sub 2023-06-22T08:24:06.006 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/missing 2023-06-22T08:24:06.006 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/compile 2023-06-22T08:24:06.006 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/install-sh 2023-06-22T08:24:06.007 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/gendocs.sh 2023-06-22T08:24:06.007 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/update-copyright 2023-06-22T08:24:06.007 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/texinfo.tex 2023-06-22T08:24:06.008 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/unused-parameter.h 2023-06-22T08:24:06.008 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/c++defs.h 2023-06-22T08:24:06.008 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/config.guess 2023-06-22T08:24:06.009 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/do-release-commit-and-tag 2023-06-22T08:24:06.009 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/git-version-gen 2023-06-22T08:24:06.009 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/arg-nonnull.h 2023-06-22T08:24:06.010 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/gnupload 2023-06-22T08:24:06.010 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/warn-on-use.h 2023-06-22T08:24:06.010 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/vc-list-files 2023-06-22T08:24:06.010 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/ylwrap 2023-06-22T08:24:06.010 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/ChangeLog-2007 2023-06-22T08:24:06.010 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/announce-gen 2023-06-22T08:24:06.011 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/build-aux/depcomp 2023-06-22T08:24:06.011 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_error_message_uppercase 2023-06-22T08:24:06.011 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_obsolete_symbols 2023-06-22T08:24:06.011 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/INSTALL 2023-06-22T08:24:06.012 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gl/ 2023-06-22T08:24:06.023 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gl/modules/ 2023-06-22T08:24:06.033 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gl/modules/getloadavg.diff 2023-06-22T08:24:06.034 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/configure.ac 2023-06-22T08:24:06.035 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_prohibit_tab_based_indentation 2023-06-22T08:24:06.035 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/GNUmakefile 2023-06-22T08:24:06.035 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/ChangeLog 2023-06-22T08:24:06.037 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/COPYING 2023-06-22T08:24:06.038 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/ 2023-06-22T08:24:06.040 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/closeout.m4 2023-06-22T08:24:06.043 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/dirent-safer.m4 2023-06-22T08:24:06.043 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/settime.m4 2023-06-22T08:24:06.043 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/readtokens.m4 2023-06-22T08:24:06.043 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/clock_time.m4 2023-06-22T08:24:06.044 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/config-h.m4 2023-06-22T08:24:06.044 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/euidaccess.m4 2023-06-22T08:24:06.044 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/printf-frexpl.m4 2023-06-22T08:24:06.044 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/posix-shell.m4 2023-06-22T08:24:06.044 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/servent.m4 2023-06-22T08:24:06.044 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/xstrtol.m4 2023-06-22T08:24:06.044 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mkdir-p.m4 2023-06-22T08:24:06.045 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/00gnulib.m4 2023-06-22T08:24:06.045 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mbsinit.m4 2023-06-22T08:24:06.045 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/dup2.m4 2023-06-22T08:24:06.045 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/eealloc.m4 2023-06-22T08:24:06.045 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/savewd.m4 2023-06-22T08:24:06.045 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/memrchr.m4 2023-06-22T08:24:06.045 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/d-ino.m4 2023-06-22T08:24:06.045 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/bison.m4 2023-06-22T08:24:06.046 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/isnanl.m4 2023-06-22T08:24:06.053 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/lib-ld.m4 2023-06-22T08:24:06.053 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stpcpy.m4 2023-06-22T08:24:06.054 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/progtest.m4 2023-06-22T08:24:06.054 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/unistd_h.m4 2023-06-22T08:24:06.054 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/time_h.m4 2023-06-22T08:24:06.054 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mkancesdirs.m4 2023-06-22T08:24:06.054 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/signalblocking.m4 2023-06-22T08:24:06.054 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/unlink.m4 2023-06-22T08:24:06.054 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/backupfile.m4 2023-06-22T08:24:06.055 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/remove.m4 2023-06-22T08:24:06.055 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fts.m4 2023-06-22T08:24:06.055 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/xalloc.m4 2023-06-22T08:24:06.055 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/inet_ntop.m4 2023-06-22T08:24:06.055 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/iconv_open.m4 2023-06-22T08:24:06.055 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/extensions.m4 2023-06-22T08:24:06.055 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mbrtowc.m4 2023-06-22T08:24:06.055 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/check-decl.m4 2023-06-22T08:24:06.056 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/cycle-check.m4 2023-06-22T08:24:06.056 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/signal_h.m4 2023-06-22T08:24:06.056 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sys_socket_h.m4 2023-06-22T08:24:06.056 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fprintftime.m4 2023-06-22T08:24:06.056 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/lib-check.m4 2023-06-22T08:24:06.056 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/isblank.m4 2023-06-22T08:24:06.056 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/xstrtod.m4 2023-06-22T08:24:06.056 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stdarg.m4 2023-06-22T08:24:06.057 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/selinux-selinux-h.m4 2023-06-22T08:24:06.057 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fstypename.m4 2023-06-22T08:24:06.057 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/link.m4 2023-06-22T08:24:06.057 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/md5.m4 2023-06-22T08:24:06.057 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/multiarch.m4 2023-06-22T08:24:06.057 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/ioctl.m4 2023-06-22T08:24:06.057 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/flexmember.m4 2023-06-22T08:24:06.057 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/locale-fr.m4 2023-06-22T08:24:06.058 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fdopendir.m4 2023-06-22T08:24:06.058 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/exponentf.m4 2023-06-22T08:24:06.058 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/utimecmp.m4 2023-06-22T08:24:06.058 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/human.m4 2023-06-22T08:24:06.058 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strerror.m4 2023-06-22T08:24:06.058 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/socklen.m4 2023-06-22T08:24:06.058 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/dirent_h.m4 2023-06-22T08:24:06.058 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/locale-zh.m4 2023-06-22T08:24:06.059 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getdate.m4 2023-06-22T08:24:06.059 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/cloexec.m4 2023-06-22T08:24:06.059 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/i-ring.m4 2023-06-22T08:24:06.059 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/group-member.m4 2023-06-22T08:24:06.059 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strtod.m4 2023-06-22T08:24:06.059 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/chdir-long.m4 2023-06-22T08:24:06.059 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strnlen.m4 2023-06-22T08:24:06.059 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strtoimax.m4 2023-06-22T08:24:06.060 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getgroups.m4 2023-06-22T08:24:06.060 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/yield.m4 2023-06-22T08:24:06.060 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/write.m4 2023-06-22T08:24:06.060 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/localcharset.m4 2023-06-22T08:24:06.060 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/usleep.m4 2023-06-22T08:24:06.060 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/memmove.m4 2023-06-22T08:24:06.060 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/close.m4 2023-06-22T08:24:06.060 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/memcmp.m4 2023-06-22T08:24:06.061 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/autobuild.m4 2023-06-22T08:24:06.061 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fchdir.m4 2023-06-22T08:24:06.061 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fd-reopen.m4 2023-06-22T08:24:06.061 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/jm-winsz2.m4 2023-06-22T08:24:06.061 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/posixver.m4 2023-06-22T08:24:06.061 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/locale-ja.m4 2023-06-22T08:24:06.061 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stat-prog.m4 2023-06-22T08:24:06.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/wint_t.m4 2023-06-22T08:24:06.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/st_dm_mode.m4 2023-06-22T08:24:06.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/tempname.m4 2023-06-22T08:24:06.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/frexp.m4 2023-06-22T08:24:06.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/dirfd.m4 2023-06-22T08:24:06.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sys_select_h.m4 2023-06-22T08:24:06.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/btowc.m4 2023-06-22T08:24:06.063 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/filenamecat.m4 2023-06-22T08:24:06.063 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/gmp.m4 2023-06-22T08:24:06.063 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/isapipe.m4 2023-06-22T08:24:06.063 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/wctype_h.m4 2023-06-22T08:24:06.063 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/ldexpl.m4 2023-06-22T08:24:06.063 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getloadavg.m4 2023-06-22T08:24:06.063 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mbchar.m4 2023-06-22T08:24:06.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/version-etc.m4 2023-06-22T08:24:06.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fcntl_h.m4 2023-06-22T08:24:06.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/gettime.m4 2023-06-22T08:24:06.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/wchar_h.m4 2023-06-22T08:24:06.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/vasprintf-posix.m4 2023-06-22T08:24:06.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/selinux-context-h.m4 2023-06-22T08:24:06.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getpass.m4 2023-06-22T08:24:06.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mempcpy.m4 2023-06-22T08:24:06.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/save-cwd.m4 2023-06-22T08:24:06.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fclose.m4 2023-06-22T08:24:06.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/modechange.m4 2023-06-22T08:24:06.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strstr.m4 2023-06-22T08:24:06.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getdtablesize.m4 2023-06-22T08:24:06.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/wchar_t.m4 2023-06-22T08:24:06.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mkdir.m4 2023-06-22T08:24:06.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/idcache.m4 2023-06-22T08:24:06.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/tm_gmtoff.m4 2023-06-22T08:24:06.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/dirname.m4 2023-06-22T08:24:06.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mbiter.m4 2023-06-22T08:24:06.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fcntl-safer.m4 2023-06-22T08:24:06.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/manywarnings.m4 2023-06-22T08:24:06.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/same.m4 2023-06-22T08:24:06.067 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getdelim.m4 2023-06-22T08:24:06.067 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mountlist.m4 2023-06-22T08:24:06.067 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/lchmod.m4 2023-06-22T08:24:06.067 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/readlink.m4 2023-06-22T08:24:06.067 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/gnu-make.m4 2023-06-22T08:24:06.067 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/thread.m4 2023-06-22T08:24:06.067 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/hostent.m4 2023-06-22T08:24:06.067 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/math_h.m4 2023-06-22T08:24:06.068 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fpending.m4 2023-06-22T08:24:06.068 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/xgetcwd.m4 2023-06-22T08:24:06.068 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/utimens.m4 2023-06-22T08:24:06.068 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/exponentd.m4 2023-06-22T08:24:06.068 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getaddrinfo.m4 2023-06-22T08:24:06.068 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sys_ioctl_h.m4 2023-06-22T08:24:06.068 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stdlib_h.m4 2023-06-22T08:24:06.069 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mgetgroups.m4 2023-06-22T08:24:06.069 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/open.m4 2023-06-22T08:24:06.069 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strpbrk.m4 2023-06-22T08:24:06.069 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/ssize_t.m4 2023-06-22T08:24:06.069 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/priv-set.m4 2023-06-22T08:24:06.069 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fcntl-o.m4 2023-06-22T08:24:06.069 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/inttostr.m4 2023-06-22T08:24:06.069 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/malloca.m4 2023-06-22T08:24:06.081 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/unlocked-io.m4 2023-06-22T08:24:06.081 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getugroups.m4 2023-06-22T08:24:06.082 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sha1.m4 2023-06-22T08:24:06.082 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fpurge.m4 2023-06-22T08:24:06.082 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/gnulib-comp.m4 2023-06-22T08:24:06.082 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/wcrtomb.m4 2023-06-22T08:24:06.082 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/putenv.m4 2023-06-22T08:24:06.082 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/lstat.m4 2023-06-22T08:24:06.082 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strsignal.m4 2023-06-22T08:24:06.082 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mbslen.m4 2023-06-22T08:24:06.083 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/inttypes.m4 2023-06-22T08:24:06.083 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stdlib-safer.m4 2023-06-22T08:24:06.083 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strtol.m4 2023-06-22T08:24:06.083 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/lib-ignore.m4 2023-06-22T08:24:06.083 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/readutmp.m4 2023-06-22T08:24:06.083 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strtoll.m4 2023-06-22T08:24:06.083 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stdbool.m4 2023-06-22T08:24:06.083 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/setenv.m4 2023-06-22T08:24:06.084 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/alloca.m4 2023-06-22T08:24:06.084 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/size_max.m4 2023-06-22T08:24:06.084 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/include_next.m4 2023-06-22T08:24:06.084 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sys_stat_h.m4 2023-06-22T08:24:06.084 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strndup.m4 2023-06-22T08:24:06.084 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/link-follow.m4 2023-06-22T08:24:06.084 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/ctype.m4 2023-06-22T08:24:06.085 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/timespec.m4 2023-06-22T08:24:06.085 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getline.m4 2023-06-22T08:24:06.085 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/calloc.m4 2023-06-22T08:24:06.085 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strftime.m4 2023-06-22T08:24:06.085 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/arpa_inet_h.m4 2023-06-22T08:24:06.085 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sig2str.m4 2023-06-22T08:24:06.085 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/printf.m4 2023-06-22T08:24:06.085 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/environ.m4 2023-06-22T08:24:06.086 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/vprintf-posix.m4 2023-06-22T08:24:06.086 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/intlmacosx.m4 2023-06-22T08:24:06.086 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/locale-tr.m4 2023-06-22T08:24:06.086 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mktime.m4 2023-06-22T08:24:06.086 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/iconv.m4 2023-06-22T08:24:06.086 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/pathmax.m4 2023-06-22T08:24:06.086 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/printf-frexp.m4 2023-06-22T08:24:06.086 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/userspec.m4 2023-06-22T08:24:06.087 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/host-os.m4 2023-06-22T08:24:06.087 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/file-type.m4 2023-06-22T08:24:06.087 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/ls-mntd-fs.m4 2023-06-22T08:24:06.087 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sha512.m4 2023-06-22T08:24:06.087 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stdio-safer.m4 2023-06-22T08:24:06.087 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/lock.m4 2023-06-22T08:24:06.087 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/c-strtod.m4 2023-06-22T08:24:06.087 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/closein.m4 2023-06-22T08:24:06.088 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/frexpl.m4 2023-06-22T08:24:06.088 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/posixtm.m4 2023-06-22T08:24:06.088 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/rmdir.m4 2023-06-22T08:24:06.088 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/chown.m4 2023-06-22T08:24:06.088 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stdint.m4 2023-06-22T08:24:06.088 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stddef_h.m4 2023-06-22T08:24:06.088 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sigaction.m4 2023-06-22T08:24:06.088 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/error.m4 2023-06-22T08:24:06.089 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/vasnprintf.m4 2023-06-22T08:24:06.089 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/utimbuf.m4 2023-06-22T08:24:06.089 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sockets.m4 2023-06-22T08:24:06.089 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/canonicalize.m4 2023-06-22T08:24:06.089 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sys_time_h.m4 2023-06-22T08:24:06.089 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/unlink-busy.m4 2023-06-22T08:24:06.089 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/gettimeofday.m4 2023-06-22T08:24:06.089 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/write-any-file.m4 2023-06-22T08:24:06.090 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strdup.m4 2023-06-22T08:24:06.090 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/regex.m4 2023-06-22T08:24:06.090 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sleep.m4 2023-06-22T08:24:06.090 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/float_h.m4 2023-06-22T08:24:06.090 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/warn-on-use.m4 2023-06-22T08:24:06.090 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/atexit.m4 2023-06-22T08:24:06.090 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getcwd-path-max.m4 2023-06-22T08:24:06.091 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fnmatch.m4 2023-06-22T08:24:06.091 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strnumcmp.m4 2023-06-22T08:24:06.091 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/gnulib-common.m4 2023-06-22T08:24:06.091 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/codeset.m4 2023-06-22T08:24:06.091 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/time_r.m4 2023-06-22T08:24:06.091 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/memcoll.m4 2023-06-22T08:24:06.091 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/string_h.m4 2023-06-22T08:24:06.091 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sys_utsname_h.m4 2023-06-22T08:24:06.092 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stdint_h.m4 2023-06-22T08:24:06.092 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/nproc.m4 2023-06-22T08:24:06.092 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/symlink.m4 2023-06-22T08:24:06.092 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/rename.m4 2023-06-22T08:24:06.092 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/read-file.m4 2023-06-22T08:24:06.092 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strtoumax.m4 2023-06-22T08:24:06.092 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getopt.m4 2023-06-22T08:24:06.092 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fopen.m4 2023-06-22T08:24:06.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/ungetc.m4 2023-06-22T08:24:06.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/xsize.m4 2023-06-22T08:24:06.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fflush.m4 2023-06-22T08:24:06.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/acl.m4 2023-06-22T08:24:06.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fsusage.m4 2023-06-22T08:24:06.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/gethrxtime.m4 2023-06-22T08:24:06.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/nls.m4 2023-06-22T08:24:06.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/lib-link.m4 2023-06-22T08:24:06.094 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mpsort.m4 2023-06-22T08:24:06.094 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/netinet_in_h.m4 2023-06-22T08:24:06.094 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fseeko.m4 2023-06-22T08:24:06.094 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/inet_pton.m4 2023-06-22T08:24:06.094 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/yesno.m4 2023-06-22T08:24:06.094 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/physmem.m4 2023-06-22T08:24:06.094 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/intmax_t.m4 2023-06-22T08:24:06.094 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/inttypes-pri.m4 2023-06-22T08:24:06.095 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/malloc.m4 2023-06-22T08:24:06.095 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/jm-winsz1.m4 2023-06-22T08:24:06.095 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/close-stream.m4 2023-06-22T08:24:06.095 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/nocrash.m4 2023-06-22T08:24:06.095 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/realloc.m4 2023-06-22T08:24:06.095 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/canon-host.m4 2023-06-22T08:24:06.095 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/memcpy.m4 2023-06-22T08:24:06.095 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/tls.m4 2023-06-22T08:24:06.096 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/perl.m4 2023-06-22T08:24:06.096 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/signbit.m4 2023-06-22T08:24:06.096 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/memchr.m4 2023-06-22T08:24:06.096 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/d-type.m4 2023-06-22T08:24:06.096 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/lchown.m4 2023-06-22T08:24:06.096 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/lib-prefix.m4 2023-06-22T08:24:06.096 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getcwd.m4 2023-06-22T08:24:06.096 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/locale_h.m4 2023-06-22T08:24:06.097 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/memxfrm.m4 2023-06-22T08:24:06.097 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mbstate_t.m4 2023-06-22T08:24:06.097 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/filemode.m4 2023-06-22T08:24:06.097 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/memset.m4 2023-06-22T08:24:06.097 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/gethostname.m4 2023-06-22T08:24:06.097 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/vasprintf.m4 2023-06-22T08:24:06.097 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/unistd-safer.m4 2023-06-22T08:24:06.097 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/xfts.m4 2023-06-22T08:24:06.098 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/uptime.m4 2023-06-22T08:24:06.098 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/prereq.m4 2023-06-22T08:24:06.098 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/symlinkat.m4 2023-06-22T08:24:06.098 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getusershell.m4 2023-06-22T08:24:06.098 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/isnand.m4 2023-06-22T08:24:06.098 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/select.m4 2023-06-22T08:24:06.098 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/wctob.m4 2023-06-22T08:24:06.098 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/double-slash-root.m4 2023-06-22T08:24:06.099 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/long-options.m4 2023-06-22T08:24:06.099 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/glibc21.m4 2023-06-22T08:24:06.099 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stat-time.m4 2023-06-22T08:24:06.099 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/rpmatch.m4 2023-06-22T08:24:06.099 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/xattr.m4 2023-06-22T08:24:06.099 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/gettext.m4 2023-06-22T08:24:06.099 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fsync.m4 2023-06-22T08:24:06.099 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/exclude.m4 2023-06-22T08:24:06.100 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/freopen.m4 2023-06-22T08:24:06.100 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strtoul.m4 2023-06-22T08:24:06.100 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/netdb_h.m4 2023-06-22T08:24:06.100 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/snprintf.m4 2023-06-22T08:24:06.100 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stpncpy.m4 2023-06-22T08:24:06.100 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/xnanosleep.m4 2023-06-22T08:24:06.100 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/warnings.m4 2023-06-22T08:24:06.100 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/lseek.m4 2023-06-22T08:24:06.101 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/hard-locale.m4 2023-06-22T08:24:06.101 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/include-exclude-prog.m4 2023-06-22T08:24:06.101 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mmap-anon.m4 2023-06-22T08:24:06.101 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mkstemp.m4 2023-06-22T08:24:06.101 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/openat.m4 2023-06-22T08:24:06.101 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mbrlen.m4 2023-06-22T08:24:06.101 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/jm-macros.m4 2023-06-22T08:24:06.101 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/ftello.m4 2023-06-22T08:24:06.102 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stdio_h.m4 2023-06-22T08:24:06.102 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/exponentl.m4 2023-06-22T08:24:06.102 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getpagesize.m4 2023-06-22T08:24:06.102 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/argmatch.m4 2023-06-22T08:24:06.102 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/perror.m4 2023-06-22T08:24:06.102 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/inline.m4 2023-06-22T08:24:06.102 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/inttypes_h.m4 2023-06-22T08:24:06.102 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/iconv_h.m4 2023-06-22T08:24:06.103 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/base64.m4 2023-06-22T08:24:06.103 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/stat.m4 2023-06-22T08:24:06.103 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/tzset.m4 2023-06-22T08:24:06.103 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/unicodeio.m4 2023-06-22T08:24:06.103 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sha256.m4 2023-06-22T08:24:06.103 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/errno_h.m4 2023-06-22T08:24:06.103 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/unlinkdir.m4 2023-06-22T08:24:06.103 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/boottime.m4 2023-06-22T08:24:06.104 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/freading.m4 2023-06-22T08:24:06.104 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mode_t.m4 2023-06-22T08:24:06.104 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getndelim2.m4 2023-06-22T08:24:06.104 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/xvasprintf.m4 2023-06-22T08:24:06.104 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mbsrtowcs.m4 2023-06-22T08:24:06.104 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/hash.m4 2023-06-22T08:24:06.104 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/safe-write.m4 2023-06-22T08:24:06.104 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/po.m4 2023-06-22T08:24:06.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/isnanf.m4 2023-06-22T08:24:06.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/longlong.m4 2023-06-22T08:24:06.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/xstrndup.m4 2023-06-22T08:24:06.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/euidaccess-stat.m4 2023-06-22T08:24:06.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/getcwd-abort-bug.m4 2023-06-22T08:24:06.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/faccessat.m4 2023-06-22T08:24:06.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/nanosleep.m4 2023-06-22T08:24:06.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/ChangeLog-2007 2023-06-22T08:24:06.109 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/utimes.m4 2023-06-22T08:24:06.109 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fileblocks.m4 2023-06-22T08:24:06.109 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/sockpfaf.m4 2023-06-22T08:24:06.109 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strtoull.m4 2023-06-22T08:24:06.109 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/assert.m4 2023-06-22T08:24:06.109 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/quotearg.m4 2023-06-22T08:24:06.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/uname.m4 2023-06-22T08:24:06.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/savedir.m4 2023-06-22T08:24:06.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/threadlib.m4 2023-06-22T08:24:06.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/langinfo_h.m4 2023-06-22T08:24:06.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/linkat.m4 2023-06-22T08:24:06.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fpieee.m4 2023-06-22T08:24:06.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/memcasecmp.m4 2023-06-22T08:24:06.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/vfprintf-posix.m4 2023-06-22T08:24:06.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/dos.m4 2023-06-22T08:24:06.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/root-dev-ino.m4 2023-06-22T08:24:06.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/quote.m4 2023-06-22T08:24:06.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/mbswidth.m4 2023-06-22T08:24:06.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/nl_langinfo.m4 2023-06-22T08:24:06.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/fcntl.m4 2023-06-22T08:24:06.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/strcspn.m4 2023-06-22T08:24:06.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/safe-read.m4 2023-06-22T08:24:06.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/m4/wcwidth.m4 2023-06-22T08:24:06.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/Makefile.am 2023-06-22T08:24:06.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_po_check 2023-06-22T08:24:06.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/dist-check.mk 2023-06-22T08:24:06.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_GPL_version 2023-06-22T08:24:06.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_unmarked_diagnostics 2023-06-22T08:24:06.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.prev-version 2023-06-22T08:24:06.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/bootstrap.conf 2023-06-22T08:24:06.113 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/ChangeLog-2005 2023-06-22T08:24:06.113 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/aclocal.m4 2023-06-22T08:24:06.113 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/AUTHORS 2023-06-22T08:24:06.113 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/cfg.mk 2023-06-22T08:24:06.113 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_file_system 2023-06-22T08:24:06.113 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.version 2023-06-22T08:24:06.113 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/TODO 2023-06-22T08:24:06.113 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_space_tab 2023-06-22T08:24:06.114 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_prohibit_empty_lines_at_EOF 2023-06-22T08:24:06.114 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/Makefile.in 2023-06-22T08:24:06.114 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/ 2023-06-22T08:24:06.123 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-06-22T08:24:06.124 INFO:tasks.ceph.mds.f:Restarting daemon 2023-06-22T08:24:06.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-06-22T08:24:06.126 INFO:tasks.ceph.mds.f:Started 2023-06-22T08:24:06.126 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:06.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~224s 2023-06-22T08:24:06.260 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/cat.x 2023-06-22T08:24:06.314 INFO:tasks.ceph.mds.f.smithi067.stdout:starting mds.f at 2023-06-22T08:24:06.340 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.338+0000 7f5073fff700 1 Processor -- start 2023-06-22T08:24:06.341 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.338+0000 7f5073fff700 1 -- start start 2023-06-22T08:24:06.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.339+0000 7f5073fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f506c0c3a40 0x7f506c0c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:06.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.339+0000 7f5073fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f506c15a320 0x7f506c160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:06.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.340+0000 7f5073fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506c0c6a60 con 0x7f506c06d050 2023-06-22T08:24:06.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.340+0000 7f5073fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506c157570 con 0x7f506c15a320 2023-06-22T08:24:06.342 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.340+0000 7f5073fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506c0c4350 con 0x7f506c0c3a40 2023-06-22T08:24:06.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.340+0000 7f507259c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f506c15a320 0x7f506c160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:06.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.340+0000 7f507159a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f506c06d050 0x7f506c159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:56586/0 (socket says 172.21.15.23:56586) 2023-06-22T08:24:06.343 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.340+0000 7f5071d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f506c0c3a40 0x7f506c0c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.340+0000 7f507159a700 1 -- 172.21.15.23:0/2708000465 learned_addr learned my addr 172.21.15.23:0/2708000465 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.341+0000 7f5070d99700 1 -- 172.21.15.23:0/2708000465 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527391894 0 0) 0x7f506c0c6a60 con 0x7f506c06d050 2023-06-22T08:24:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.341+0000 7f5070d99700 1 -- 172.21.15.23:0/2708000465 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50580035b0 con 0x7f506c06d050 2023-06-22T08:24:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.341+0000 7f5071d9b700 1 -- 172.21.15.23:0/2708000465 >> v2:172.21.15.23:3300/0 conn(0x7f506c15a320 msgr2=0x7f506c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.341+0000 7f5071d9b700 1 --2- 172.21.15.23:0/2708000465 >> v2:172.21.15.23:3300/0 conn(0x7f506c15a320 0x7f506c160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:06.344 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.341+0000 7f5071d9b700 1 -- 172.21.15.23:0/2708000465 >> v1:172.21.15.67:6789/0 conn(0x7f506c06d050 legacy=0x7f506c159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.341+0000 7f5071d9b700 1 -- 172.21.15.23:0/2708000465 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c14d690 con 0x7f506c0c3a40 2023-06-22T08:24:06.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.341+0000 7f5071d9b700 1 --2- 172.21.15.23:0/2708000465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0c3a40 0x7f506c0c3e10 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f506c06af80 tx=0x7f5068009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48fd11c1993e77e6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.341+0000 7f5070d99700 1 -- 172.21.15.23:0/2708000465 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068011830 con 0x7f506c0c3a40 2023-06-22T08:24:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.341+0000 7f5070d99700 1 -- 172.21.15.23:0/2708000465 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5068011990 con 0x7f506c0c3a40 2023-06-22T08:24:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.341+0000 7f5070d99700 1 -- 172.21.15.23:0/2708000465 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068011d20 con 0x7f506c0c3a40 2023-06-22T08:24:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.342+0000 7f5073fff700 1 -- 172.21.15.23:0/2708000465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0c3a40 msgr2=0x7f506c0c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.342+0000 7f5073fff700 1 --2- 172.21.15.23:0/2708000465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0c3a40 0x7f506c0c3e10 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f506c06af80 tx=0x7f5068009b20 comp rx=0 tx=0).stop 2023-06-22T08:24:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.342+0000 7f5073fff700 1 -- 172.21.15.23:0/2708000465 shutdown_connections 2023-06-22T08:24:06.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.342+0000 7f5073fff700 1 --2- 172.21.15.23:0/2708000465 >> v2:172.21.15.23:3300/0 conn(0x7f506c15a320 0x7f506c160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:06.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.342+0000 7f5073fff700 1 --2- 172.21.15.23:0/2708000465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c0c3a40 0x7f506c0c3e10 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:06.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.342+0000 7f5073fff700 1 -- 172.21.15.23:0/2708000465 >> 172.21.15.23:0/2708000465 conn(0x7f506c0ca3f0 msgr2=0x7f506c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:06.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.342+0000 7f5073fff700 1 -- 172.21.15.23:0/2708000465 shutdown_connections 2023-06-22T08:24:06.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.342+0000 7f5073fff700 1 -- 172.21.15.23:0/2708000465 wait complete. 2023-06-22T08:24:06.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.344+0000 7f5073fff700 1 Processor -- start 2023-06-22T08:24:06.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.344+0000 7f5073fff700 1 -- start start 2023-06-22T08:24:06.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f5073fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c06d050 0x7f506c0c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:06.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f5073fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f506c0c3a40 0x7f506c157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:06.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f5073fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c15a320 0x7f506c158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:06.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f5073fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c0ca1c0 con 0x7f506c15a320 2023-06-22T08:24:06.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f5073fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c157790 con 0x7f506c06d050 2023-06-22T08:24:06.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f5073fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506c0c9640 con 0x7f506c0c3a40 2023-06-22T08:24:06.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f5071d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c06d050 0x7f506c0c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:06.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f5071d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c06d050 0x7f506c0c8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:46018/0 (socket says 172.21.15.23:46018) 2023-06-22T08:24:06.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f507159a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f506c0c3a40 0x7f506c157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:06.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f5071d9b700 1 -- 172.21.15.23:0/3374529059 learned_addr learned my addr 172.21.15.23:0/3374529059 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:06.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.345+0000 7f507259c700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c15a320 0x7f506c158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:06.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.346+0000 7f5071d9b700 1 -- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f506c0c3a40 msgr2=0x7f506c157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:06.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.346+0000 7f5071d9b700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f506c0c3a40 0x7f506c157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:06.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.346+0000 7f5071d9b700 1 -- 172.21.15.23:0/3374529059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c15a320 msgr2=0x7f506c158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:06.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.346+0000 7f5071d9b700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c15a320 0x7f506c158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:06.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.346+0000 7f5071d9b700 1 -- 172.21.15.23:0/3374529059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50680094a0 con 0x7f506c06d050 2023-06-22T08:24:06.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.346+0000 7f5071d9b700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c06d050 0x7f506c0c8440 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5068009b60 tx=0x7f5068004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:06.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.346+0000 7f50627fc700 1 -- 172.21.15.23:0/3374529059 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50680036a0 con 0x7f506c06d050 2023-06-22T08:24:06.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.346+0000 7f50627fc700 1 -- 172.21.15.23:0/3374529059 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506801f070 con 0x7f506c06d050 2023-06-22T08:24:06.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.346+0000 7f50627fc700 1 -- 172.21.15.23:0/3374529059 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068003a00 con 0x7f506c06d050 2023-06-22T08:24:06.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.346+0000 7f5073fff700 1 -- 172.21.15.23:0/3374529059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506c0c9920 con 0x7f506c06d050 2023-06-22T08:24:06.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.347+0000 7f5073fff700 1 -- 172.21.15.23:0/3374529059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506c0c9ee0 con 0x7f506c06d050 2023-06-22T08:24:06.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.348+0000 7f5073fff700 1 -- 172.21.15.23:0/3374529059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506c04ea20 con 0x7f506c06d050 2023-06-22T08:24:06.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.349+0000 7f50627fc700 1 -- 172.21.15.23:0/3374529059 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f5068003b60 con 0x7f506c06d050 2023-06-22T08:24:06.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.350+0000 7f50627fc700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f505006ad10 0x7f505006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:06.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.350+0000 7f50627fc700 1 -- 172.21.15.23:0/3374529059 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f5068021070 con 0x7f506c06d050 2023-06-22T08:24:06.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.350+0000 7f507159a700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f505006ad10 0x7f505006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:06.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.351+0000 7f507159a700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f505006ad10 0x7f505006d1c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f506c071620 tx=0x7f505c007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:06.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.354+0000 7f50627fc700 1 -- 172.21.15.23:0/3374529059 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f5068055d00 con 0x7f506c06d050 2023-06-22T08:24:06.364 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/date.x 2023-06-22T08:24:06.364 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/basename.1 2023-06-22T08:24:06.364 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/pathchk.x 2023-06-22T08:24:06.365 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tail.x 2023-06-22T08:24:06.366 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/yes.x 2023-06-22T08:24:06.366 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/pinky.1 2023-06-22T08:24:06.366 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/kill.x 2023-06-22T08:24:06.366 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/pathchk.1 2023-06-22T08:24:06.366 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sync.1 2023-06-22T08:24:06.366 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/dir.1 2023-06-22T08:24:06.366 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/mv.1 2023-06-22T08:24:06.366 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/rm.1 2023-06-22T08:24:06.367 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/dircolors.x 2023-06-22T08:24:06.367 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/rmdir.1 2023-06-22T08:24:06.367 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tty.1 2023-06-22T08:24:06.367 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/fold.x 2023-06-22T08:24:06.367 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/fold.1 2023-06-22T08:24:06.369 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/dir.x 2023-06-22T08:24:06.370 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/fmt.x 2023-06-22T08:24:06.370 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/paste.1 2023-06-22T08:24:06.370 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/chmod.x 2023-06-22T08:24:06.370 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/od.1 2023-06-22T08:24:06.370 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/ptx.x 2023-06-22T08:24:06.370 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/nice.1 2023-06-22T08:24:06.371 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/install.1 2023-06-22T08:24:06.371 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/dd.x 2023-06-22T08:24:06.371 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/cksum.1 2023-06-22T08:24:06.371 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/test.1 2023-06-22T08:24:06.371 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/chgrp.1 2023-06-22T08:24:06.371 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/unlink.1 2023-06-22T08:24:06.371 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/chmod.1 2023-06-22T08:24:06.371 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/dd.1 2023-06-22T08:24:06.372 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/du.1 2023-06-22T08:24:06.372 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tsort.1 2023-06-22T08:24:06.372 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/ls.1 2023-06-22T08:24:06.372 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/split.x 2023-06-22T08:24:06.372 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/nohup.x 2023-06-22T08:24:06.372 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sha1sum.1 2023-06-22T08:24:06.372 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tr.1 2023-06-22T08:24:06.373 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/head.x 2023-06-22T08:24:06.373 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/false.1 2023-06-22T08:24:06.373 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/pwd.x 2023-06-22T08:24:06.373 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/logname.x 2023-06-22T08:24:06.373 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/paste.x 2023-06-22T08:24:06.373 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/nl.x 2023-06-22T08:24:06.373 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sha224sum.1 2023-06-22T08:24:06.374 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/mkdir.1 2023-06-22T08:24:06.374 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/nl.1 2023-06-22T08:24:06.374 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sha256sum.1 2023-06-22T08:24:06.374 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/printf.x 2023-06-22T08:24:06.374 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/md5sum.1 2023-06-22T08:24:06.377 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/Makefile.am 2023-06-22T08:24:06.377 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/dirname.1 2023-06-22T08:24:06.377 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/shuf.x 2023-06-22T08:24:06.378 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/echo.1 2023-06-22T08:24:06.378 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/chcon.x 2023-06-22T08:24:06.378 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/chroot.1 2023-06-22T08:24:06.378 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sha384sum.x 2023-06-22T08:24:06.378 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/stdbuf.1 2023-06-22T08:24:06.378 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/users.1 2023-06-22T08:24:06.378 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/readlink.x 2023-06-22T08:24:06.378 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/cut.x 2023-06-22T08:24:06.379 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/groups.1 2023-06-22T08:24:06.379 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/seq.1 2023-06-22T08:24:06.379 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/help2man 2023-06-22T08:24:06.379 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/dirname.x 2023-06-22T08:24:06.379 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/uname.x 2023-06-22T08:24:06.379 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/uptime.x 2023-06-22T08:24:06.379 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/cut.1 2023-06-22T08:24:06.379 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/cksum.x 2023-06-22T08:24:06.380 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/nohup.1 2023-06-22T08:24:06.380 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/mv.x 2023-06-22T08:24:06.380 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sort.1 2023-06-22T08:24:06.380 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/chown.1 2023-06-22T08:24:06.380 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/pr.x 2023-06-22T08:24:06.380 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/true.x 2023-06-22T08:24:06.380 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/base64.1 2023-06-22T08:24:06.381 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/factor.1 2023-06-22T08:24:06.381 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/ln.x 2023-06-22T08:24:06.381 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/head.1 2023-06-22T08:24:06.381 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/unexpand.1 2023-06-22T08:24:06.381 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/df.x 2023-06-22T08:24:06.381 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/join.x 2023-06-22T08:24:06.381 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/du.x 2023-06-22T08:24:06.381 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/dircolors.1 2023-06-22T08:24:06.382 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/uniq.x 2023-06-22T08:24:06.382 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/split.1 2023-06-22T08:24:06.382 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tty.x 2023-06-22T08:24:06.382 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/uname.1 2023-06-22T08:24:06.382 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/uptime.1 2023-06-22T08:24:06.382 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sort.x 2023-06-22T08:24:06.382 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sha256sum.x 2023-06-22T08:24:06.382 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sha512sum.x 2023-06-22T08:24:06.393 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/nice.x 2023-06-22T08:24:06.393 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/pinky.x 2023-06-22T08:24:06.393 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/fmt.1 2023-06-22T08:24:06.393 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/expr.1 2023-06-22T08:24:06.394 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/wc.1 2023-06-22T08:24:06.394 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/true.1 2023-06-22T08:24:06.394 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/comm.x 2023-06-22T08:24:06.394 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/truncate.x 2023-06-22T08:24:06.394 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/install.x 2023-06-22T08:24:06.394 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/wc.x 2023-06-22T08:24:06.394 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/uniq.1 2023-06-22T08:24:06.394 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/arch.x 2023-06-22T08:24:06.395 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sum.x 2023-06-22T08:24:06.395 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/timeout.x 2023-06-22T08:24:06.395 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/link.x 2023-06-22T08:24:06.395 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tac.1 2023-06-22T08:24:06.395 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sync.x 2023-06-22T08:24:06.395 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/Makefile.in 2023-06-22T08:24:06.395 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/mkfifo.1 2023-06-22T08:24:06.395 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/od.x 2023-06-22T08:24:06.396 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/vdir.1 2023-06-22T08:24:06.396 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/timeout.1 2023-06-22T08:24:06.396 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/mkfifo.x 2023-06-22T08:24:06.396 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/basename.x 2023-06-22T08:24:06.396 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/expr.x 2023-06-22T08:24:06.396 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/mktemp.x 2023-06-22T08:24:06.396 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/vdir.x 2023-06-22T08:24:06.397 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/su.x 2023-06-22T08:24:06.397 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/env.1 2023-06-22T08:24:06.397 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/who.x 2023-06-22T08:24:06.397 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/hostid.1 2023-06-22T08:24:06.397 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tee.x 2023-06-22T08:24:06.397 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/rmdir.x 2023-06-22T08:24:06.397 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/hostid.x 2023-06-22T08:24:06.397 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/id.x 2023-06-22T08:24:06.398 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/unexpand.x 2023-06-22T08:24:06.398 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/pr.1 2023-06-22T08:24:06.398 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/csplit.1 2023-06-22T08:24:06.398 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/truncate.1 2023-06-22T08:24:06.398 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tee.1 2023-06-22T08:24:06.398 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/env.x 2023-06-22T08:24:06.398 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/chroot.x 2023-06-22T08:24:06.398 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/cp.x 2023-06-22T08:24:06.399 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/whoami.x 2023-06-22T08:24:06.399 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/logname.1 2023-06-22T08:24:06.399 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/mknod.1 2023-06-22T08:24:06.399 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/groups.x 2023-06-22T08:24:06.399 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/cp.1 2023-06-22T08:24:06.399 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/runcon.x 2023-06-22T08:24:06.399 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/yes.1 2023-06-22T08:24:06.400 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/stat.1 2023-06-22T08:24:06.400 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/nproc.x 2023-06-22T08:24:06.400 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/unlink.x 2023-06-22T08:24:06.400 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sha384sum.1 2023-06-22T08:24:06.400 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/nproc.1 2023-06-22T08:24:06.400 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/seq.x 2023-06-22T08:24:06.400 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sha512sum.1 2023-06-22T08:24:06.400 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/stat.x 2023-06-22T08:24:06.401 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sleep.x 2023-06-22T08:24:06.401 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/printenv.1 2023-06-22T08:24:06.401 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tr.x 2023-06-22T08:24:06.401 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tail.1 2023-06-22T08:24:06.401 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/md5sum.x 2023-06-22T08:24:06.401 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/link.1 2023-06-22T08:24:06.401 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/echo.x 2023-06-22T08:24:06.401 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/ptx.1 2023-06-22T08:24:06.402 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/factor.x 2023-06-22T08:24:06.402 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/who.1 2023-06-22T08:24:06.402 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/mktemp.1 2023-06-22T08:24:06.402 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/shred.1 2023-06-22T08:24:06.402 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/runcon.1 2023-06-22T08:24:06.402 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/stty.x 2023-06-22T08:24:06.402 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/comm.1 2023-06-22T08:24:06.402 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/shuf.1 2023-06-22T08:24:06.403 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/stty.1 2023-06-22T08:24:06.403 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/printenv.x 2023-06-22T08:24:06.403 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/join.1 2023-06-22T08:24:06.403 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/hostname.x 2023-06-22T08:24:06.403 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/ln.1 2023-06-22T08:24:06.403 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/false.x 2023-06-22T08:24:06.403 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/cat.1 2023-06-22T08:24:06.403 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/users.x 2023-06-22T08:24:06.404 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/printf.1 2023-06-22T08:24:06.404 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/ls.x 2023-06-22T08:24:06.404 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sha224sum.x 2023-06-22T08:24:06.404 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/whoami.1 2023-06-22T08:24:06.404 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/expand.1 2023-06-22T08:24:06.404 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/chgrp.x 2023-06-22T08:24:06.404 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sleep.1 2023-06-22T08:24:06.405 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/csplit.x 2023-06-22T08:24:06.405 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/pwd.1 2023-06-22T08:24:06.405 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/kill.1 2023-06-22T08:24:06.405 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/touch.x 2023-06-22T08:24:06.405 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/df.1 2023-06-22T08:24:06.405 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/stdbuf.x 2023-06-22T08:24:06.405 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/mkdir.x 2023-06-22T08:24:06.405 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/id.1 2023-06-22T08:24:06.406 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/mknod.x 2023-06-22T08:24:06.406 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sum.1 2023-06-22T08:24:06.406 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/rm.x 2023-06-22T08:24:06.406 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/date.1 2023-06-22T08:24:06.406 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/chown.x 2023-06-22T08:24:06.406 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/touch.1 2023-06-22T08:24:06.406 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tac.x 2023-06-22T08:24:06.406 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/base64.x 2023-06-22T08:24:06.407 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/tsort.x 2023-06-22T08:24:06.407 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/shred.x 2023-06-22T08:24:06.407 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/readlink.1 2023-06-22T08:24:06.407 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/sha1sum.x 2023-06-22T08:24:06.407 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/expand.x 2023-06-22T08:24:06.407 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/test.x 2023-06-22T08:24:06.407 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/man/chcon.1 2023-06-22T08:24:06.407 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.tarball-version 2023-06-22T08:24:06.408 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/NEWS 2023-06-22T08:24:06.408 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_sun_os_names 2023-06-22T08:24:06.408 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/bootstrap 2023-06-22T08:24:06.408 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/ 2023-06-22T08:24:06.480 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbsalign.h 2023-06-22T08:24:06.536 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/streq.h 2023-06-22T08:24:06.536 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/asprintf.c 2023-06-22T08:24:06.537 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strnlen1.c 2023-06-22T08:24:06.537 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dirfd.c 2023-06-22T08:24:06.537 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/safe-write.h 2023-06-22T08:24:06.537 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/snprintf.c 2023-06-22T08:24:06.537 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbiter.h 2023-06-22T08:24:06.537 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mountlist.c 2023-06-22T08:24:06.537 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/propername.h 2023-06-22T08:24:06.538 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memset.c 2023-06-22T08:24:06.538 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/nproc.c 2023-06-22T08:24:06.538 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getdelim.c 2023-06-22T08:24:06.538 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/signbitl.c 2023-06-22T08:24:06.538 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/group-member.h 2023-06-22T08:24:06.538 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/long-options.c 2023-06-22T08:24:06.538 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fts_.h 2023-06-22T08:24:06.538 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fts.c 2023-06-22T08:24:06.542 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/verify.h 2023-06-22T08:24:06.542 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/str-two-way.h 2023-06-22T08:24:06.542 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/areadlink.c 2023-06-22T08:24:06.542 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/c-strtod.h 2023-06-22T08:24:06.542 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sha256.h 2023-06-22T08:24:06.542 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stddef.in.h 2023-06-22T08:24:06.543 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/randint.c 2023-06-22T08:24:06.543 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dup-safer.c 2023-06-22T08:24:06.543 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getpass.h 2023-06-22T08:24:06.543 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/gettext.h 2023-06-22T08:24:06.543 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/randint.h 2023-06-22T08:24:06.543 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdio-write.c 2023-06-22T08:24:06.543 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xgethostname.h 2023-06-22T08:24:06.543 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/asnprintf.c 2023-06-22T08:24:06.544 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open-solaris.gperf 2023-06-22T08:24:06.544 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdio-impl.h 2023-06-22T08:24:06.544 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mpsort.h 2023-06-22T08:24:06.544 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/closein.c 2023-06-22T08:24:06.544 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/hard-locale.c 2023-06-22T08:24:06.544 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/utimecmp.c 2023-06-22T08:24:06.544 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/vasnprintf.c 2023-06-22T08:24:06.544 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/chown.c 2023-06-22T08:24:06.545 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/printf-args.c 2023-06-22T08:24:06.545 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/netdb.in.h 2023-06-22T08:24:06.545 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strnlen.c 2023-06-22T08:24:06.545 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/errno.in.h 2023-06-22T08:24:06.545 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbslen.c 2023-06-22T08:24:06.545 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/obstack.h 2023-06-22T08:24:06.545 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/file-type.h 2023-06-22T08:24:06.545 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbsrtowcs.c 2023-06-22T08:24:06.546 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fnmatch_loop.c 2023-06-22T08:24:06.546 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/safe-write.c 2023-06-22T08:24:06.546 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mpsort.c 2023-06-22T08:24:06.546 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/netinet_in.in.h 2023-06-22T08:24:06.546 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xfreopen.h 2023-06-22T08:24:06.546 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/wctype.in.h 2023-06-22T08:24:06.546 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stripslash.c 2023-06-22T08:24:06.546 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/closein.h 2023-06-22T08:24:06.547 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memchr2.c 2023-06-22T08:24:06.547 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memcoll.c 2023-06-22T08:24:06.547 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/malloca.h 2023-06-22T08:24:06.547 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stpcpy.c 2023-06-22T08:24:06.547 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/striconv.c 2023-06-22T08:24:06.547 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/basename.c 2023-06-22T08:24:06.547 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sha256.c 2023-06-22T08:24:06.548 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdint.in.h 2023-06-22T08:24:06.548 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/acl-internal.h 2023-06-22T08:24:06.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/atexit.c 2023-06-22T08:24:06.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/size_max.h 2023-06-22T08:24:06.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/buffer-lcm.h 2023-06-22T08:24:06.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/idcache.h 2023-06-22T08:24:06.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/root-dev-ino.c 2023-06-22T08:24:06.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/physmem.c 2023-06-22T08:24:06.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/glthread/ 2023-06-22T08:24:06.570 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/glthread/lock.h 2023-06-22T08:24:06.592 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/glthread/threadlib.c 2023-06-22T08:24:06.593 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/glthread/lock.c 2023-06-22T08:24:06.593 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/glthread/tls.h 2023-06-22T08:24:06.593 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/glthread/tls.c 2023-06-22T08:24:06.593 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unistr/ 2023-06-22T08:24:06.595 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unistr/u8-uctomb.c 2023-06-22T08:24:06.597 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unistr/u8-uctomb-aux.c 2023-06-22T08:24:06.597 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mkancesdirs.c 2023-06-22T08:24:06.598 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/readtokens0.c 2023-06-22T08:24:06.598 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fnmatch.in.h 2023-06-22T08:24:06.598 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dirname.h 2023-06-22T08:24:06.598 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/regex_internal.c 2023-06-22T08:24:06.599 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strcspn.c 2023-06-22T08:24:06.599 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getugroups.h 2023-06-22T08:24:06.599 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/file-set.c 2023-06-22T08:24:06.599 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sys_socket.in.h 2023-06-22T08:24:06.599 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/inttostr.h 2023-06-22T08:24:06.601 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mkdirat.c 2023-06-22T08:24:06.601 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/readtokens.h 2023-06-22T08:24:06.601 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fd-reopen.h 2023-06-22T08:24:06.601 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/w32sock.h 2023-06-22T08:24:06.601 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memcasecmp.c 2023-06-22T08:24:06.601 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/at-func.c 2023-06-22T08:24:06.602 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unistd-safer.h 2023-06-22T08:24:06.602 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/filevercmp.h 2023-06-22T08:24:06.602 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/c-strcasecmp.c 2023-06-22T08:24:06.602 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fopen-safer.c 2023-06-22T08:24:06.602 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/modechange.h 2023-06-22T08:24:06.602 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sys_stat.in.h 2023-06-22T08:24:06.602 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/freading.c 2023-06-22T08:24:06.602 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/utimens.h 2023-06-22T08:24:06.603 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/md5.h 2023-06-22T08:24:06.603 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/isnanf.c 2023-06-22T08:24:06.603 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sha1.c 2023-06-22T08:24:06.603 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/filenamecat-lgpl.c 2023-06-22T08:24:06.603 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/config.hin 2023-06-22T08:24:06.603 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/ignore-value.h 2023-06-22T08:24:06.606 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strstr.c 2023-06-22T08:24:06.606 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/verror.c 2023-06-22T08:24:06.606 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/randperm.h 2023-06-22T08:24:06.607 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/vasprintf.c 2023-06-22T08:24:06.607 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/c-strncasecmp.c 2023-06-22T08:24:06.607 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/basename-lgpl.c 2023-06-22T08:24:06.607 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/human.h 2023-06-22T08:24:06.607 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/regex.h 2023-06-22T08:24:06.607 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/regcomp.c 2023-06-22T08:24:06.607 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/freadahead.c 2023-06-22T08:24:06.608 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/trim.h 2023-06-22T08:24:06.608 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getgroups.c 2023-06-22T08:24:06.608 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xprintf.c 2023-06-22T08:24:06.608 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/euidaccess-stat.h 2023-06-22T08:24:06.608 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/rand-isaac.h 2023-06-22T08:24:06.608 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mgetgroups.h 2023-06-22T08:24:06.608 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/selinux-at.h 2023-06-22T08:24:06.609 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open-solaris.h 2023-06-22T08:24:06.609 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/setenv.c 2023-06-22T08:24:06.609 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/locale.in.h 2023-06-22T08:24:06.609 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/base64.h 2023-06-22T08:24:06.609 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/open-safer.c 2023-06-22T08:24:06.609 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/exitfail.c 2023-06-22T08:24:06.609 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/closeout.h 2023-06-22T08:24:06.609 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/idcache.c 2023-06-22T08:24:06.610 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unlink.c 2023-06-22T08:24:06.610 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xfts.c 2023-06-22T08:24:06.610 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memrchr.c 2023-06-22T08:24:06.610 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/Makefile.am 2023-06-22T08:24:06.610 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrtol.h 2023-06-22T08:24:06.610 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/gettimeofday.c 2023-06-22T08:24:06.610 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/isnanf-nolibm.h 2023-06-22T08:24:06.610 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/chdir-long.c 2023-06-22T08:24:06.611 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbchar.c 2023-06-22T08:24:06.611 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/canon-host.h 2023-06-22T08:24:06.611 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unsetenv.c 2023-06-22T08:24:06.611 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/freadseek.c 2023-06-22T08:24:06.611 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xnanosleep.c 2023-06-22T08:24:06.611 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/nanosleep.c 2023-06-22T08:24:06.614 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/ctype.in.h 2023-06-22T08:24:06.615 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xmalloc.c 2023-06-22T08:24:06.615 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strtoul.c 2023-06-22T08:24:06.615 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sha512.h 2023-06-22T08:24:06.615 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dirchownmod.h 2023-06-22T08:24:06.615 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/freadptr.h 2023-06-22T08:24:06.615 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getfilecon.c 2023-06-22T08:24:06.615 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/alignof.h 2023-06-22T08:24:06.616 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mempcpy.c 2023-06-22T08:24:06.616 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/savedir.h 2023-06-22T08:24:06.616 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/areadlink-with-size.c 2023-06-22T08:24:06.616 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/inttypes.in.h 2023-06-22T08:24:06.616 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/config.charset 2023-06-22T08:24:06.616 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrtold.c 2023-06-22T08:24:06.616 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fcntl-safer.h 2023-06-22T08:24:06.617 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrndup.c 2023-06-22T08:24:06.617 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/write.c 2023-06-22T08:24:06.617 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/math.in.h 2023-06-22T08:24:06.617 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/openat-priv.h 2023-06-22T08:24:06.617 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/vasnprintf.h 2023-06-22T08:24:06.617 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getcwd.c 2023-06-22T08:24:06.617 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdbool.in.h 2023-06-22T08:24:06.617 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strtol.c 2023-06-22T08:24:06.618 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/argmatch.c 2023-06-22T08:24:06.618 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/acl_entries.c 2023-06-22T08:24:06.618 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/progname.c 2023-06-22T08:24:06.618 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mktime-internal.h 2023-06-22T08:24:06.618 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/printf-frexpl.c 2023-06-22T08:24:06.618 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/offtostr.c 2023-06-22T08:24:06.618 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memchr2.h 2023-06-22T08:24:06.618 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrtoumax.c 2023-06-22T08:24:06.619 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getdate.y 2023-06-22T08:24:06.619 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/close-hook.h 2023-06-22T08:24:06.619 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/printf-parse.c 2023-06-22T08:24:06.619 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/isnand.c 2023-06-22T08:24:06.619 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/timespec.h 2023-06-22T08:24:06.619 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getusershell.c 2023-06-22T08:24:06.619 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrtod.c 2023-06-22T08:24:06.619 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/verror.h 2023-06-22T08:24:06.620 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/same.c 2023-06-22T08:24:06.620 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/quote.c 2023-06-22T08:24:06.620 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/raise.c 2023-06-22T08:24:06.620 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/link.c 2023-06-22T08:24:06.620 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/cloexec.h 2023-06-22T08:24:06.620 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sys_utsname.in.h 2023-06-22T08:24:06.620 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/cloexec.c 2023-06-22T08:24:06.620 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/vfprintf.c 2023-06-22T08:24:06.621 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/uniwidth/ 2023-06-22T08:24:06.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.622+0000 7f5073fff700 1 -- 172.21.15.23:0/3374529059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f506c156f50 con 0x7f506c06d050 2023-06-22T08:24:06.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.622+0000 7f50627fc700 1 -- 172.21.15.23:0/3374529059 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+4727 (secure 0 0 0) 0x7f5068059320 con 0x7f506c06d050 2023-06-22T08:24:06.625 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:06.625 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:00.074498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:06.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 -- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f505006ad10 msgr2=0x7f505006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:06.627 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f505006ad10 0x7f505006d1c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f506c071620 tx=0x7f505c007dc0 comp rx=0 tx=0).stop 2023-06-22T08:24:06.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 -- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c06d050 msgr2=0x7f506c0c8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:06.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c06d050 0x7f506c0c8440 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5068009b60 tx=0x7f5068004750 comp rx=0 tx=0).stop 2023-06-22T08:24:06.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 -- 172.21.15.23:0/3374529059 shutdown_connections 2023-06-22T08:24:06.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f506c15a320 0x7f506c158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:06.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f505006ad10 0x7f505006d1c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:06.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f506c06d050 0x7f506c0c8440 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:06.628 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 --2- 172.21.15.23:0/3374529059 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f506c0c3a40 0x7f506c157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:06.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 -- 172.21.15.23:0/3374529059 >> 172.21.15.23:0/3374529059 conn(0x7f506c0ca3f0 msgr2=0x7f506c070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:06.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 -- 172.21.15.23:0/3374529059 shutdown_connections 2023-06-22T08:24:06.629 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:06.625+0000 7f5073fff700 1 -- 172.21.15.23:0/3374529059 wait complete. 2023-06-22T08:24:06.629 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 188 2023-06-22T08:24:06.647 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-06-22T08:24:08.624 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/uniwidth/cjk.h 2023-06-22T08:24:08.627 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/uniwidth/width.c 2023-06-22T08:24:08.627 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strtoull.c 2023-06-22T08:24:08.630 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/opendir-safer.c 2023-06-22T08:24:08.630 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/hash-pjw.h 2023-06-22T08:24:08.630 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/uname.c 2023-06-22T08:24:08.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/full-read.c 2023-06-22T08:24:08.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/priv-set.h 2023-06-22T08:24:08.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/close.c 2023-06-22T08:24:08.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fpending.h 2023-06-22T08:24:08.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/i-ring.h 2023-06-22T08:24:08.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/nl_langinfo.c 2023-06-22T08:24:08.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xmemxfrm.h 2023-06-22T08:24:08.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/c-strcase.h 2023-06-22T08:24:08.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fchdir.c 2023-06-22T08:24:08.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/canon-host.c 2023-06-22T08:24:08.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/signbitf.c 2023-06-22T08:24:08.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xnanosleep.h 2023-06-22T08:24:08.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/tempname.c 2023-06-22T08:24:08.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xreadlink.c 2023-06-22T08:24:08.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xgetcwd.h 2023-06-22T08:24:08.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/regex.c 2023-06-22T08:24:08.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/diacrit.h 2023-06-22T08:24:08.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/posixver.c 2023-06-22T08:24:08.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getugroups.c 2023-06-22T08:24:08.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/readutmp.h 2023-06-22T08:24:08.638 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/quote.h 2023-06-22T08:24:08.638 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/quotearg.c 2023-06-22T08:24:08.638 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/filevercmp.c 2023-06-22T08:24:08.638 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/printf-args.h 2023-06-22T08:24:08.638 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mountlist.h 2023-06-22T08:24:08.638 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrtol-error.c 2023-06-22T08:24:08.638 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/rename.c 2023-06-22T08:24:08.639 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sha1.h 2023-06-22T08:24:08.639 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strndup.c 2023-06-22T08:24:08.639 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/priv-set.c 2023-06-22T08:24:08.639 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xfts.h 2023-06-22T08:24:08.639 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/argv-iter.c 2023-06-22T08:24:08.639 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdio--.h 2023-06-22T08:24:08.639 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/yesno.c 2023-06-22T08:24:08.639 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getopt.c 2023-06-22T08:24:08.640 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strpbrk.c 2023-06-22T08:24:08.640 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dirname-lgpl.c 2023-06-22T08:24:08.640 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/filemode.h 2023-06-22T08:24:08.640 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbrtowc.c 2023-06-22T08:24:08.640 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open.c 2023-06-22T08:24:08.640 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/euidaccess-stat.c 2023-06-22T08:24:08.640 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stat-time.h 2023-06-22T08:24:08.640 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/frexpl.c 2023-06-22T08:24:08.641 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sys_select.in.h 2023-06-22T08:24:08.641 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strftime.h 2023-06-22T08:24:08.641 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strftime.c 2023-06-22T08:24:08.641 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open-aix.gperf 2023-06-22T08:24:08.641 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/buffer-lcm.c 2023-06-22T08:24:08.641 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unistr.h 2023-06-22T08:24:08.641 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/modechange.c 2023-06-22T08:24:08.641 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/tempname.h 2023-06-22T08:24:08.642 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/faccessat.c 2023-06-22T08:24:08.642 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unicodeio.h 2023-06-22T08:24:08.642 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/filemode.c 2023-06-22T08:24:08.642 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strdup.c 2023-06-22T08:24:08.642 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strsignal.c 2023-06-22T08:24:08.642 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/euidaccess.c 2023-06-22T08:24:08.642 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/select.c 2023-06-22T08:24:08.642 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memcoll.h 2023-06-22T08:24:08.643 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fseeko.c 2023-06-22T08:24:08.643 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/langinfo.in.h 2023-06-22T08:24:08.643 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getopt.in.h 2023-06-22T08:24:08.643 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/localcharset.c 2023-06-22T08:24:08.643 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/isapipe.c 2023-06-22T08:24:08.643 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/settime.c 2023-06-22T08:24:08.643 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/bitrotate.h 2023-06-22T08:24:08.643 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/isnanl-nolibm.h 2023-06-22T08:24:08.644 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fcntl.c 2023-06-22T08:24:08.644 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/exclude.h 2023-06-22T08:24:08.644 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/symlinkat.c 2023-06-22T08:24:08.644 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fpending.c 2023-06-22T08:24:08.644 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/savewd.c 2023-06-22T08:24:08.650 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fchownat.c 2023-06-22T08:24:08.650 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/root-dev-ino.h 2023-06-22T08:24:08.650 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/randread.h 2023-06-22T08:24:08.650 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/freadseek.h 2023-06-22T08:24:08.650 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strtoll.c 2023-06-22T08:24:08.650 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdio-safer.h 2023-06-22T08:24:08.650 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strtoumax.c 2023-06-22T08:24:08.651 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/argv-iter.h 2023-06-22T08:24:08.651 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dirent-safer.h 2023-06-22T08:24:08.651 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dirent.in.h 2023-06-22T08:24:08.651 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/malloca.valgrind 2023-06-22T08:24:08.651 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/savewd.h 2023-06-22T08:24:08.651 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/uinttostr.c 2023-06-22T08:24:08.651 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getline.c 2023-06-22T08:24:08.651 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/chdir-long.h 2023-06-22T08:24:08.652 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/cycle-check.h 2023-06-22T08:24:08.652 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fts-cycle.c 2023-06-22T08:24:08.652 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/file-has-acl.c 2023-06-22T08:24:08.652 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/areadlink.h 2023-06-22T08:24:08.652 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sig2str.h 2023-06-22T08:24:08.652 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memcpy.c 2023-06-22T08:24:08.652 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/freading.h 2023-06-22T08:24:08.653 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/full-write.h 2023-06-22T08:24:08.653 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unistd.in.h 2023-06-22T08:24:08.653 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/symlink.c 2023-06-22T08:24:08.653 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/se-selinux.in.h 2023-06-22T08:24:08.653 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/write-any-file.h 2023-06-22T08:24:08.653 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/printf-frexpl.h 2023-06-22T08:24:08.653 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/Makefile.in 2023-06-22T08:24:08.653 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/filenamecat.h 2023-06-22T08:24:08.654 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/malloca.c 2023-06-22T08:24:08.654 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/utimecmp.h 2023-06-22T08:24:08.654 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdlib.in.h 2023-06-22T08:24:08.654 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdio.in.h 2023-06-22T08:24:08.654 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbsinit.c 2023-06-22T08:24:08.654 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/diacrit.c 2023-06-22T08:24:08.654 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/se-context.in.h 2023-06-22T08:24:08.654 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xvasprintf.c 2023-06-22T08:24:08.655 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xgetcwd.c 2023-06-22T08:24:08.655 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sockets.h 2023-06-22T08:24:08.655 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/striconv.h 2023-06-22T08:24:08.655 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mkstemp-safer.c 2023-06-22T08:24:08.655 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/u64.h 2023-06-22T08:24:08.655 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrtoimax.c 2023-06-22T08:24:08.655 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/time_r.c 2023-06-22T08:24:08.655 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbchar.h 2023-06-22T08:24:08.656 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xtime.h 2023-06-22T08:24:08.656 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/set-mode-acl.c 2023-06-22T08:24:08.656 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/version-etc.c 2023-06-22T08:24:08.656 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dup2.c 2023-06-22T08:24:08.656 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unlinkat.c 2023-06-22T08:24:08.656 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sigprocmask.c 2023-06-22T08:24:08.656 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/hash-triple.h 2023-06-22T08:24:08.656 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fprintftime.c 2023-06-22T08:24:08.657 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unistd--.h 2023-06-22T08:24:08.657 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/umaxtostr.c 2023-06-22T08:24:08.657 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/i-ring.c 2023-06-22T08:24:08.657 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/signbitd.c 2023-06-22T08:24:08.657 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/ftello.c 2023-06-22T08:24:08.657 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/lchown.c 2023-06-22T08:24:08.657 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/lstat.c 2023-06-22T08:24:08.657 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fpurge.c 2023-06-22T08:24:08.658 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fstatat.c 2023-06-22T08:24:08.658 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/lseek.c 2023-06-22T08:24:08.658 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fchmodat.c 2023-06-22T08:24:08.658 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/alloca.c 2023-06-22T08:24:08.658 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memcmp.c 2023-06-22T08:24:08.658 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrndup.h 2023-06-22T08:24:08.658 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/frexp.c 2023-06-22T08:24:08.658 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/nproc.h 2023-06-22T08:24:08.659 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fsusage.h 2023-06-22T08:24:08.659 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fd-safer-flag.c 2023-06-22T08:24:08.659 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getdtablesize.c 2023-06-22T08:24:08.659 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/savedir.c 2023-06-22T08:24:08.659 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/wcwidth.c 2023-06-22T08:24:08.659 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbsstr.c 2023-06-22T08:24:08.659 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fpucw.h 2023-06-22T08:24:08.659 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open-hpux.h 2023-06-22T08:24:08.660 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/readtokens0.h 2023-06-22T08:24:08.660 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getdate.c 2023-06-22T08:24:08.660 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/utimens.c 2023-06-22T08:24:08.660 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getndelim2.c 2023-06-22T08:24:08.660 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fflush.c 2023-06-22T08:24:08.660 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/version-etc-fsf.c 2023-06-22T08:24:08.660 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xsize.h 2023-06-22T08:24:08.661 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:08.672 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xgethostname.c 2023-06-22T08:24:08.672 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strnlen1.h 2023-06-22T08:24:08.672 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/isnanl.c 2023-06-22T08:24:08.672 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/same-inode.h 2023-06-22T08:24:08.673 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/regex_internal.h 2023-06-22T08:24:08.673 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/full-read.h 2023-06-22T08:24:08.673 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/intprops.h 2023-06-22T08:24:08.673 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fsusage.c 2023-06-22T08:24:08.673 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/save-cwd.c 2023-06-22T08:24:08.673 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memchr.valgrind 2023-06-22T08:24:08.673 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/exitfail.h 2023-06-22T08:24:08.673 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/printf-frexp.h 2023-06-22T08:24:08.674 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/isblank.c 2023-06-22T08:24:08.674 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/arpa_inet.in.h 2023-06-22T08:24:08.674 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strerror.c 2023-06-22T08:24:08.674 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/freadptr.c 2023-06-22T08:24:08.674 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/openat.h 2023-06-22T08:24:08.674 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open-irix.h 2023-06-22T08:24:08.674 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/open.c 2023-06-22T08:24:08.674 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/vprintf.c 2023-06-22T08:24:08.675 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/same.h 2023-06-22T08:24:08.675 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/gai_strerror.c 2023-06-22T08:24:08.675 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/imaxtostr.c 2023-06-22T08:24:08.675 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memcmp2.h 2023-06-22T08:24:08.675 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xfreopen.c 2023-06-22T08:24:08.675 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/rand-isaac.c 2023-06-22T08:24:08.675 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/full-write.c 2023-06-22T08:24:08.675 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/rmdir.c 2023-06-22T08:24:08.676 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/save-cwd.h 2023-06-22T08:24:08.676 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/at-func2.c 2023-06-22T08:24:08.676 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/gethrxtime.c 2023-06-22T08:24:08.676 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strtod.c 2023-06-22T08:24:08.676 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbsalign.c 2023-06-22T08:24:08.676 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open-osf.gperf 2023-06-22T08:24:08.676 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/ref-add.sin 2023-06-22T08:24:08.676 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memcmp2.c 2023-06-22T08:24:08.677 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/gnulib.mk 2023-06-22T08:24:08.677 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/ref-del.sin 2023-06-22T08:24:08.677 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stpncpy.c 2023-06-22T08:24:08.677 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv.in.h 2023-06-22T08:24:08.677 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/posixtm.h 2023-06-22T08:24:08.677 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrtol.c 2023-06-22T08:24:08.677 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getopt1.c 2023-06-22T08:24:08.677 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/quotearg.h 2023-06-22T08:24:08.678 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/cycle-check.c 2023-06-22T08:24:08.678 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/calloc.c 2023-06-22T08:24:08.678 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/areadlinkat.c 2023-06-22T08:24:08.678 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/hash-triple.c 2023-06-22T08:24:08.678 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getpass.c 2023-06-22T08:24:08.678 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstriconv.c 2023-06-22T08:24:08.678 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dirent--.h 2023-06-22T08:24:08.678 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sigaction.c 2023-06-22T08:24:08.679 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/linkat.c 2023-06-22T08:24:08.679 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unitypes.h 2023-06-22T08:24:08.679 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/linebuffer.c 2023-06-22T08:24:08.679 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fnmatch.c 2023-06-22T08:24:08.679 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xasprintf.c 2023-06-22T08:24:08.679 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/printf-parse.h 2023-06-22T08:24:08.679 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/string.in.h 2023-06-22T08:24:08.679 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mkdir.c 2023-06-22T08:24:08.680 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fchown-stub.c 2023-06-22T08:24:08.680 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/human.c 2023-06-22T08:24:08.680 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sockets.c 2023-06-22T08:24:08.680 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/canonicalize.h 2023-06-22T08:24:08.680 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getopt_int.h 2023-06-22T08:24:08.680 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/version-etc.h 2023-06-22T08:24:08.680 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdlib--.h 2023-06-22T08:24:08.681 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/backupfile.h 2023-06-22T08:24:08.681 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/file-type.c 2023-06-22T08:24:08.681 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/file-set.h 2023-06-22T08:24:08.681 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbuiter.h 2023-06-22T08:24:08.681 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fcntl.in.h 2023-06-22T08:24:08.681 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/inet_ntop.c 2023-06-22T08:24:08.681 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fseterr.c 2023-06-22T08:24:08.681 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/close-stream.h 2023-06-22T08:24:08.682 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/freopen-safer.c 2023-06-22T08:24:08.682 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stat-macros.h 2023-06-22T08:24:08.682 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dirname.c 2023-06-22T08:24:08.682 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dev-ino.h 2023-06-22T08:24:08.682 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xalloc.h 2023-06-22T08:24:08.682 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memcasecmp.h 2023-06-22T08:24:08.682 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/hash.c 2023-06-22T08:24:08.682 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/userspec.c 2023-06-22T08:24:08.683 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/hard-locale.h 2023-06-22T08:24:08.683 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xvasprintf.h 2023-06-22T08:24:08.683 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/wcrtomb.c 2023-06-22T08:24:08.683 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/gethrxtime.h 2023-06-22T08:24:08.683 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fprintftime.h 2023-06-22T08:24:08.683 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mgetgroups.c 2023-06-22T08:24:08.683 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/randread.c 2023-06-22T08:24:08.683 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstriconv.h 2023-06-22T08:24:08.684 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dup-safer-flag.c 2023-06-22T08:24:08.684 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/yesno.h 2023-06-22T08:24:08.684 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbscasecmp.c 2023-06-22T08:24:08.684 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strtoimax.c 2023-06-22T08:24:08.684 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbsrtowcs-state.c 2023-06-22T08:24:08.684 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mkstemp.c 2023-06-22T08:24:08.684 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xmemcoll.c 2023-06-22T08:24:08.684 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strintcmp.c 2023-06-22T08:24:08.685 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fclose.c 2023-06-22T08:24:08.685 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fd-reopen.c 2023-06-22T08:24:08.685 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/inttostr.c 2023-06-22T08:24:08.685 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/c-ctype.c 2023-06-22T08:24:08.685 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/openat-die.c 2023-06-22T08:24:08.685 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/error.c 2023-06-22T08:24:08.685 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sig2str.c 2023-06-22T08:24:08.685 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unlocked-io.h 2023-06-22T08:24:08.686 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memxfrm.c 2023-06-22T08:24:08.686 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/pipe-safer.c 2023-06-22T08:24:08.686 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/backupfile.c 2023-06-22T08:24:08.686 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open-osf.h 2023-06-22T08:24:08.686 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/linebuffer.h 2023-06-22T08:24:08.686 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xprintf.h 2023-06-22T08:24:08.686 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/exclude.c 2023-06-22T08:24:08.686 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/userspec.h 2023-06-22T08:24:08.687 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mkancesdirs.h 2023-06-22T08:24:08.687 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/hash-pjw.c 2023-06-22T08:24:08.687 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fd-safer.c 2023-06-22T08:24:08.687 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/long-options.h 2023-06-22T08:24:08.687 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/freadahead.h 2023-06-22T08:24:08.687 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/gethostname.c 2023-06-22T08:24:08.687 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/putenv.c 2023-06-22T08:24:08.687 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbrlen.c 2023-06-22T08:24:08.688 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/group-member.c 2023-06-22T08:24:08.688 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sha512.c 2023-06-22T08:24:08.688 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/signal.in.h 2023-06-22T08:24:08.688 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mktime.c 2023-06-22T08:24:08.688 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/readlink.c 2023-06-22T08:24:08.688 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mkdir-p.c 2023-06-22T08:24:08.688 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getndelim2.h 2023-06-22T08:24:08.688 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memxfrm.h 2023-06-22T08:24:08.689 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/c-strtod.c 2023-06-22T08:24:08.689 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fseterr.h 2023-06-22T08:24:08.689 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open-hpux.gperf 2023-06-22T08:24:08.689 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/filenamecat.c 2023-06-22T08:24:08.689 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memmove.c 2023-06-22T08:24:08.689 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/str-kmp.h 2023-06-22T08:24:08.689 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/base64.c 2023-06-22T08:24:08.689 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/close-hook.c 2023-06-22T08:24:08.690 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/dirchownmod.c 2023-06-22T08:24:08.690 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/openat-safer.c 2023-06-22T08:24:08.690 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/posixtm.c 2023-06-22T08:24:08.690 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/printf-frexp.c 2023-06-22T08:24:08.690 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/safe-read.h 2023-06-22T08:24:08.690 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/regexec.c 2023-06-22T08:24:08.690 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/uniwidth.h 2023-06-22T08:24:08.690 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/write-any-file.c 2023-06-22T08:24:08.691 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open-aix.h 2023-06-22T08:24:08.691 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/unicodeio.c 2023-06-22T08:24:08.691 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/gettime.c 2023-06-22T08:24:08.691 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strnumcmp.h 2023-06-22T08:24:08.691 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/btowc.c 2023-06-22T08:24:08.691 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sys_time.in.h 2023-06-22T08:24:08.691 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/isnan.c 2023-06-22T08:24:08.691 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getaddrinfo.c 2023-06-22T08:24:08.692 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getpagesize.c 2023-06-22T08:24:08.692 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/posixver.h 2023-06-22T08:24:08.692 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/openat.c 2023-06-22T08:24:08.692 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/closeout.c 2023-06-22T08:24:08.692 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fopen.c 2023-06-22T08:24:08.692 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memchr2.valgrind 2023-06-22T08:24:08.692 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/time.in.h 2023-06-22T08:24:08.692 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/physmem.h 2023-06-22T08:24:08.693 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/iconv_open-irix.gperf 2023-06-22T08:24:08.693 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stat.c 2023-06-22T08:24:08.693 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/siglist.h 2023-06-22T08:24:08.693 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbswidth.h 2023-06-22T08:24:08.693 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrtod.h 2023-06-22T08:24:08.693 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/md5.c 2023-06-22T08:24:08.693 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getloadavg.c 2023-06-22T08:24:08.693 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/getdate.h 2023-06-22T08:24:08.694 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdlib-safer.h 2023-06-22T08:24:08.694 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/remove.c 2023-06-22T08:24:08.694 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/malloc.c 2023-06-22T08:24:08.694 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/progname.h 2023-06-22T08:24:08.694 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/argmatch.h 2023-06-22T08:24:08.694 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/ChangeLog-2007 2023-06-22T08:24:08.697 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mbswidth.c 2023-06-22T08:24:08.697 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/error.h 2023-06-22T08:24:08.697 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/obstack.c 2023-06-22T08:24:08.697 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/acl.h 2023-06-22T08:24:08.697 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strnumcmp-in.h 2023-06-22T08:24:08.697 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xstrtoul.c 2023-06-22T08:24:08.697 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/float+.h 2023-06-22T08:24:08.698 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/canonicalize.c 2023-06-22T08:24:08.698 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fcntl--.h 2023-06-22T08:24:08.698 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/wchar.in.h 2023-06-22T08:24:08.698 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/trim.c 2023-06-22T08:24:08.698 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/rpmatch.c 2023-06-22T08:24:08.698 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/isapipe.h 2023-06-22T08:24:08.698 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/memchr.c 2023-06-22T08:24:08.699 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/localcharset.h 2023-06-22T08:24:08.699 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/sig-handler.h 2023-06-22T08:24:08.699 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/openat-proc.c 2023-06-22T08:24:08.699 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/pathmax.h 2023-06-22T08:24:08.699 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/close-stream.c 2023-06-22T08:24:08.699 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/readutmp.c 2023-06-22T08:24:08.699 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/float.in.h 2023-06-22T08:24:08.699 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/creat-safer.c 2023-06-22T08:24:08.700 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/randperm.c 2023-06-22T08:24:08.700 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/readtokens.c 2023-06-22T08:24:08.700 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/safe-read.c 2023-06-22T08:24:08.700 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xreadlink.h 2023-06-22T08:24:08.700 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/stdarg.in.h 2023-06-22T08:24:08.700 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xalloc-die.c 2023-06-22T08:24:08.700 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/copy-acl.c 2023-06-22T08:24:08.700 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fsync.c 2023-06-22T08:24:08.701 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/strnumcmp.c 2023-06-22T08:24:08.701 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/selinux-at.c 2023-06-22T08:24:08.701 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/realloc.c 2023-06-22T08:24:08.701 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fdopendir.c 2023-06-22T08:24:08.701 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/alloca.in.h 2023-06-22T08:24:08.701 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/freopen.c 2023-06-22T08:24:08.701 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/mkdir-p.h 2023-06-22T08:24:08.701 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/isnand-nolibm.h 2023-06-22T08:24:08.702 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/propername.c 2023-06-22T08:24:08.702 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/fileblocks.c 2023-06-22T08:24:08.702 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xmemcoll.h 2023-06-22T08:24:08.702 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/hash.h 2023-06-22T08:24:08.702 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/c-strtold.c 2023-06-22T08:24:08.702 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/c-ctype.h 2023-06-22T08:24:08.702 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/lib/xmemxfrm.c 2023-06-22T08:24:08.702 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_require_config_h_first 2023-06-22T08:24:08.703 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ 2023-06-22T08:24:08.850 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/pt.po 2023-06-22T08:24:08.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.880+0000 7f50a7bff700 1 Processor -- start 2023-06-22T08:24:08.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.881+0000 7f50a7bff700 1 -- start start 2023-06-22T08:24:08.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.882+0000 7f50a7bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f50a0071070 0x7f50a0160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:08.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.882+0000 7f50a7bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a00c51a0 con 0x7f50a00c69e0 2023-06-22T08:24:08.886 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.882+0000 7f50a7bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a00c44e0 con 0x7f50a014f850 2023-06-22T08:24:08.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.882+0000 7f50a7bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a00c5440 con 0x7f50a0071070 2023-06-22T08:24:08.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.882+0000 7f50a599b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f50a014f850 0x7f50a00c3080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:56600/0 (socket says 172.21.15.23:56600) 2023-06-22T08:24:08.887 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.882+0000 7f50a599b700 1 -- 172.21.15.23:0/425720086 learned_addr learned my addr 172.21.15.23:0/425720086 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:08.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.882+0000 7f50a519a700 1 --1- 172.21.15.23:0/425720086 >> v1:172.21.15.23:6789/0 conn(0x7f50a00c69e0 0x7f50a0155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:53142/0 (socket says 172.21.15.23:53142) 2023-06-22T08:24:08.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.882+0000 7f50a619c700 1 --2- 172.21.15.23:0/425720086 >> v2:172.21.15.23:3300/0 conn(0x7f50a0071070 0x7f50a0160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:08.888 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a4999700 1 -- 172.21.15.23:0/425720086 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112800423 0 0) 0x7f50a00c44e0 con 0x7f50a014f850 2023-06-22T08:24:08.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a4999700 1 -- 172.21.15.23:0/425720086 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f508c0035b0 con 0x7f50a014f850 2023-06-22T08:24:08.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a4999700 1 -- 172.21.15.23:0/425720086 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798158403 0 0) 0x7f50a00c51a0 con 0x7f50a00c69e0 2023-06-22T08:24:08.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a4999700 1 -- 172.21.15.23:0/425720086 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50a00c44e0 con 0x7f50a00c69e0 2023-06-22T08:24:08.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a619c700 1 -- 172.21.15.23:0/425720086 >> v1:172.21.15.67:6789/0 conn(0x7f50a014f850 legacy=0x7f50a00c3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:08.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a619c700 1 -- 172.21.15.23:0/425720086 >> v1:172.21.15.23:6789/0 conn(0x7f50a00c69e0 legacy=0x7f50a0155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:08.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a619c700 1 -- 172.21.15.23:0/425720086 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a00c5720 con 0x7f50a0071070 2023-06-22T08:24:08.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a619c700 1 --2- 172.21.15.23:0/425720086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50a0071070 0x7f50a0160eb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5098009d90 tx=0x7f509800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7988adeb12834bd6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:08.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a4999700 1 -- 172.21.15.23:0/425720086 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509800e070 con 0x7f50a0071070 2023-06-22T08:24:08.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a4999700 1 -- 172.21.15.23:0/425720086 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509800bb20 con 0x7f50a0071070 2023-06-22T08:24:08.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.883+0000 7f50a4999700 1 -- 172.21.15.23:0/425720086 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098012430 con 0x7f50a0071070 2023-06-22T08:24:08.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.884+0000 7f50a7bff700 1 -- 172.21.15.23:0/425720086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50a0071070 msgr2=0x7f50a0160eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:08.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.884+0000 7f50a7bff700 1 --2- 172.21.15.23:0/425720086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50a0071070 0x7f50a0160eb0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5098009d90 tx=0x7f509800b330 comp rx=0 tx=0).stop 2023-06-22T08:24:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.884+0000 7f50a7bff700 1 -- 172.21.15.23:0/425720086 shutdown_connections 2023-06-22T08:24:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.884+0000 7f50a7bff700 1 --2- 172.21.15.23:0/425720086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50a0071070 0x7f50a0160eb0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.884+0000 7f50a7bff700 1 -- 172.21.15.23:0/425720086 >> 172.21.15.23:0/425720086 conn(0x7f50a00ca3f0 msgr2=0x7f50a00ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:08.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.884+0000 7f50a7bff700 1 -- 172.21.15.23:0/425720086 shutdown_connections 2023-06-22T08:24:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.884+0000 7f50a7bff700 1 -- 172.21.15.23:0/425720086 wait complete. 2023-06-22T08:24:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.886+0000 7f50a7bff700 1 Processor -- start 2023-06-22T08:24:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.886+0000 7f50a7bff700 1 -- start start 2023-06-22T08:24:08.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a7bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0071070 0x7f50a01f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a7bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50a00c69e0 0x7f50a014b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a7bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50a014f850 0x7f50a013fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a7bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a01f1f80 con 0x7f50a00c69e0 2023-06-22T08:24:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a7bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a0152910 con 0x7f50a0071070 2023-06-22T08:24:08.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a7bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a0145050 con 0x7f50a014f850 2023-06-22T08:24:08.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a599b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0071070 0x7f50a01f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:08.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a599b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0071070 0x7f50a01f1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:46030/0 (socket says 172.21.15.23:46030) 2023-06-22T08:24:08.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a599b700 1 -- 172.21.15.23:0/2605531864 learned_addr learned my addr 172.21.15.23:0/2605531864 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:08.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a519a700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50a00c69e0 0x7f50a014b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:08.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.887+0000 7f50a619c700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50a014f850 0x7f50a013fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:08.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50a599b700 1 -- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50a014f850 msgr2=0x7f50a013fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:08.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50a599b700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50a014f850 0x7f50a013fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:08.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50a599b700 1 -- 172.21.15.23:0/2605531864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50a00c69e0 msgr2=0x7f50a014b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:08.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50a599b700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50a00c69e0 0x7f50a014b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:08.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50a599b700 1 -- 172.21.15.23:0/2605531864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5098009a70 con 0x7f50a0071070 2023-06-22T08:24:08.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50a599b700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0071070 0x7f50a01f1a40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f509c001eb0 tx=0x7f509c00c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:08.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50967fc700 1 -- 172.21.15.23:0/2605531864 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c005d70 con 0x7f50a0071070 2023-06-22T08:24:08.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50967fc700 1 -- 172.21.15.23:0/2605531864 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c0070d0 con 0x7f50a0071070 2023-06-22T08:24:08.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50a7bff700 1 -- 172.21.15.23:0/2605531864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a0145330 con 0x7f50a0071070 2023-06-22T08:24:08.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50967fc700 1 -- 172.21.15.23:0/2605531864 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f509c007440 con 0x7f50a0071070 2023-06-22T08:24:08.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.888+0000 7f50a7bff700 1 -- 172.21.15.23:0/2605531864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a01525a0 con 0x7f50a0071070 2023-06-22T08:24:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.889+0000 7f50a7bff700 1 -- 172.21.15.23:0/2605531864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a004ea20 con 0x7f50a0071070 2023-06-22T08:24:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.890+0000 7f50967fc700 1 -- 172.21.15.23:0/2605531864 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f509c0075a0 con 0x7f50a0071070 2023-06-22T08:24:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.891+0000 7f50967fc700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f508406ace0 0x7f508406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:08.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.891+0000 7f50967fc700 1 -- 172.21.15.23:0/2605531864 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f509c088ed0 con 0x7f50a0071070 2023-06-22T08:24:08.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.892+0000 7f50a519a700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f508406ace0 0x7f508406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:08.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.892+0000 7f50a519a700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f508406ace0 0x7f508406d190 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f50a014bad0 tx=0x7f5090007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:08.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:08.896+0000 7f50967fc700 1 -- 172.21.15.23:0/2605531864 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f509c053840 con 0x7f50a0071070 2023-06-22T08:24:08.966 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/sl.po 2023-06-22T08:24:08.969 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/zh_TW.po 2023-06-22T08:24:08.972 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/sk.po 2023-06-22T08:24:08.975 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/el.gmo 2023-06-22T08:24:08.976 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/en.gmo 2023-06-22T08:24:08.976 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/lg.po 2023-06-22T08:24:08.978 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/sv.gmo 2023-06-22T08:24:08.980 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/pl.gmo 2023-06-22T08:24:08.983 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/insert-header.sin 2023-06-22T08:24:08.983 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/sk.gmo 2023-06-22T08:24:08.983 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ja.gmo 2023-06-22T08:24:08.984 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ko.po 2023-06-22T08:24:08.986 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/da.gmo 2023-06-22T08:24:08.988 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/LINGUAS 2023-06-22T08:24:08.988 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/zh_TW.gmo 2023-06-22T08:24:08.989 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/tr.gmo 2023-06-22T08:24:08.990 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/gl.gmo 2023-06-22T08:24:08.990 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/fi.po 2023-06-22T08:24:08.992 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/da.po 2023-06-22T08:24:08.995 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/fi.gmo 2023-06-22T08:24:08.996 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/quot.sed 2023-06-22T08:24:08.996 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/stamp-po 2023-06-22T08:24:08.997 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ms.gmo 2023-06-22T08:24:08.997 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/pt_BR.po 2023-06-22T08:24:08.999 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/Makevars 2023-06-22T08:24:08.999 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/boldquot.sed 2023-06-22T08:24:09.000 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/coreutils.pot 2023-06-22T08:24:09.000 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/de.po 2023-06-22T08:24:09.003 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/pt.gmo 2023-06-22T08:24:09.005 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/Makefile.in.in 2023-06-22T08:24:09.005 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/nl.gmo 2023-06-22T08:24:09.007 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/fr.gmo 2023-06-22T08:24:09.009 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ga.gmo 2023-06-22T08:24:09.010 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/cs.po 2023-06-22T08:24:09.014 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ru.po 2023-06-22T08:24:09.015 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/hu.po 2023-06-22T08:24:09.018 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/id.po 2023-06-22T08:24:09.020 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ru.gmo 2023-06-22T08:24:09.023 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/sl.gmo 2023-06-22T08:24:09.025 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/remove-potcdate.sin 2023-06-22T08:24:09.025 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/hu.gmo 2023-06-22T08:24:09.026 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/et.po 2023-06-22T08:24:09.029 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/gl.po 2023-06-22T08:24:09.031 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/vi.gmo 2023-06-22T08:24:09.034 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/bg.po 2023-06-22T08:24:09.036 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/pl.po 2023-06-22T08:24:09.039 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/nl.po 2023-06-22T08:24:09.041 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/pt_BR.gmo 2023-06-22T08:24:09.042 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ko.gmo 2023-06-22T08:24:09.043 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/de.gmo 2023-06-22T08:24:09.045 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/POTFILES.in 2023-06-22T08:24:09.045 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/sv.po 2023-06-22T08:24:09.047 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/it.po 2023-06-22T08:24:09.049 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/eu.gmo 2023-06-22T08:24:09.049 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/en.po 2023-06-22T08:24:09.051 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/it.gmo 2023-06-22T08:24:09.053 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/en@boldquot.header 2023-06-22T08:24:09.053 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/et.gmo 2023-06-22T08:24:09.054 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ja.po 2023-06-22T08:24:09.057 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/zh_CN.gmo 2023-06-22T08:24:09.059 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/uk.po 2023-06-22T08:24:09.061 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ca.gmo 2023-06-22T08:24:09.063 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/Rules-quot 2023-06-22T08:24:09.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ro.po 2023-06-22T08:24:09.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/nb.gmo 2023-06-22T08:24:09.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/bg.gmo 2023-06-22T08:24:09.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/be.po 2023-06-22T08:24:09.068 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/vi.po 2023-06-22T08:24:09.071 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/lt.gmo 2023-06-22T08:24:09.071 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/uk.gmo 2023-06-22T08:24:09.073 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/id.gmo 2023-06-22T08:24:09.075 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/af.po 2023-06-22T08:24:09.076 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/eu.po 2023-06-22T08:24:09.078 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ms.po 2023-06-22T08:24:09.080 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/en@quot.header 2023-06-22T08:24:09.080 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/af.gmo 2023-06-22T08:24:09.080 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/es.gmo 2023-06-22T08:24:09.081 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ga.po 2023-06-22T08:24:09.083 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ro.gmo 2023-06-22T08:24:09.083 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ca.po 2023-06-22T08:24:09.086 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/lg.gmo 2023-06-22T08:24:09.088 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/zh_CN.po 2023-06-22T08:24:09.091 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/cs.gmo 2023-06-22T08:24:09.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/nb.po 2023-06-22T08:24:09.094 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/ChangeLog-2007 2023-06-22T08:24:09.095 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/tr.po 2023-06-22T08:24:09.097 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/be.gmo 2023-06-22T08:24:09.097 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/fr.po 2023-06-22T08:24:09.100 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/el.po 2023-06-22T08:24:09.103 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/lt.po 2023-06-22T08:24:09.104 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/po/es.po 2023-06-22T08:24:09.107 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ 2023-06-22T08:24:09.108 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/ 2023-06-22T08:24:09.109 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/skip-seek 2023-06-22T08:24:09.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/unblock-sync 2023-06-22T08:24:09.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/skip-seek2 2023-06-22T08:24:09.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/unblock 2023-06-22T08:24:09.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/stderr 2023-06-22T08:24:09.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/direct 2023-06-22T08:24:09.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/skip-seek-past-file 2023-06-22T08:24:09.111 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/not-rewound 2023-06-22T08:24:09.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/skip-seek-past-dev 2023-06-22T08:24:09.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/misc 2023-06-22T08:24:09.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/dd/reblock 2023-06-22T08:24:09.112 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ 2023-06-22T08:24:09.116 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tt-FF 2023-06-22T08:24:09.120 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/t_tab_ 2023-06-22T08:24:09.120 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/FF 2023-06-22T08:24:09.120 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/FnFn 2023-06-22T08:24:09.120 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/o3ml24f-bl-tn 2023-06-22T08:24:09.121 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+5-8b3l17f-FF 2023-06-22T08:24:09.121 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+3ml24f-bl-tn 2023-06-22T08:24:09.121 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/FFtn 2023-06-22T08:24:09.121 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/a2l15-FF 2023-06-22T08:24:09.121 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tFnFt 2023-06-22T08:24:09.121 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tn2e5o3-t_tab 2023-06-22T08:24:09.128 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3ml24f-t 2023-06-22T08:24:09.128 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nSml20-t-tFFFF 2023-06-22T08:24:09.128 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tn2e8-t_tab 2023-06-22T08:24:09.128 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/b3f-FF 2023-06-22T08:24:09.128 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/2w60f-t_notab 2023-06-22T08:24:09.128 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/FFn 2023-06-22T08:24:09.129 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/t_tab 2023-06-22T08:24:09.129 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/b3-0F 2023-06-22T08:24:09.129 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+4-8a2l17-FF 2023-06-22T08:24:09.129 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+4b2l17f-0FF 2023-06-22T08:24:09.129 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3a3l15-t 2023-06-22T08:24:09.129 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tFFn 2023-06-22T08:24:09.129 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3l24f-t 2023-06-22T08:24:09.129 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/0FnFt 2023-06-22T08:24:09.130 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3-FF 2023-06-22T08:24:09.130 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/t-FF 2023-06-22T08:24:09.130 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nSml24-bl-FF 2023-06-22T08:24:09.130 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+3l24f-bl 2023-06-22T08:24:09.130 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/b3l15-t 2023-06-22T08:24:09.130 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3b3f-FF 2023-06-22T08:24:09.130 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3b3f-0F 2023-06-22T08:24:09.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W35b3l24f-lm 2023-06-22T08:24:09.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/0Ft 2023-06-22T08:24:09.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tt-bl 2023-06-22T08:24:09.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3ml24-t-FF 2023-06-22T08:24:09.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tFFt-lm 2023-06-22T08:24:09.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nSml20-bl-FF 2023-06-22T08:24:09.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tn2e8o3-t_tab 2023-06-22T08:24:09.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W28l24f-ll 2023-06-22T08:24:09.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nSml24-t-tFFFF 2023-06-22T08:24:09.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/t-0FF 2023-06-22T08:24:09.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tb3-FF 2023-06-22T08:24:09.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/o3b3Sl24f-tn 2023-06-22T08:24:09.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/o3mSl24f-bl-tn 2023-06-22T08:24:09.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3b2l17-FF 2023-06-22T08:24:09.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/2f-t_notab 2023-06-22T08:24:09.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/b2l15-FF 2023-06-22T08:24:09.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3b3f-0FF 2023-06-22T08:24:09.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/a3l15-t 2023-06-22T08:24:09.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/o3Jml24f-lm-lo 2023-06-22T08:24:09.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tb3-0FF 2023-06-22T08:24:09.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/l24-t 2023-06-22T08:24:09.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W35ml24f-lm-lo 2023-06-22T08:24:09.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/a3-0F 2023-06-22T08:24:09.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ml24f-0F 2023-06-22T08:24:09.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/loli 2023-06-22T08:24:09.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3ml24-t 2023-06-22T08:24:09.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/a3l24f-lm 2023-06-22T08:24:09.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/Jb3l24f-lm 2023-06-22T08:24:09.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/t-t 2023-06-22T08:24:09.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/0F 2023-06-22T08:24:09.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/w72l24f-ll 2023-06-22T08:24:09.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ml24f-t 2023-06-22T08:24:09.135 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/b3l15f-t 2023-06-22T08:24:09.135 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/0FnFnt 2023-06-22T08:24:09.135 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/b3l24f-lm 2023-06-22T08:24:09.135 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/o3mSnl24fbltn 2023-06-22T08:24:09.135 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nJml24f-lmlolm 2023-06-22T08:24:09.135 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/o3a3Sl24f-tn 2023-06-22T08:24:09.135 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tne8-t_tab 2023-06-22T08:24:09.135 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3ml24-FF 2023-06-22T08:24:09.136 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/a3f-0FF 2023-06-22T08:24:09.136 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ml20-FF-t 2023-06-22T08:24:09.136 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/pr-tests 2023-06-22T08:24:09.136 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W35Jb3l24f-lm 2023-06-22T08:24:09.136 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ttb3-0FF 2023-06-22T08:24:09.136 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nl24f-bl 2023-06-22T08:24:09.136 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nSml24-t-t-FF 2023-06-22T08:24:09.136 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3b3l15f-t 2023-06-22T08:24:09.137 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/0FF 2023-06-22T08:24:09.137 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3f-0F 2023-06-22T08:24:09.137 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/Ja3l24f-lm 2023-06-22T08:24:09.137 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3a2l17-FF 2023-06-22T08:24:09.137 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ml24f-t-0F 2023-06-22T08:24:09.137 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tn_2e8-t_tab 2023-06-22T08:24:09.137 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/o3a3Snl24f-tn 2023-06-22T08:24:09.137 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/2s_w60f-t_nota 2023-06-22T08:24:09.138 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/t-bl 2023-06-22T08:24:09.138 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ml24-t-FF 2023-06-22T08:24:09.138 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+3-7l24-FF 2023-06-22T08:24:09.138 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/t_notab 2023-06-22T08:24:09.138 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W72Jl24f-ll 2023-06-22T08:24:09.138 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/a2l17-FF 2023-06-22T08:24:09.138 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ml24-FF 2023-06-22T08:24:09.138 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/b3f-0F 2023-06-22T08:24:09.139 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+3l24f-0FF 2023-06-22T08:24:09.139 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+3ml20f-bl-FF 2023-06-22T08:24:09.139 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/a3f-0F 2023-06-22T08:24:09.139 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W35Jml24f-lmlo 2023-06-22T08:24:09.139 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W27l24f-ll 2023-06-22T08:24:09.139 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ml24-t 2023-06-22T08:24:09.139 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/a3f-FF 2023-06-22T08:24:09.139 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tFFt-bl 2023-06-22T08:24:09.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tta3-FF 2023-06-22T08:24:09.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3-0F 2023-06-22T08:24:09.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/2sf-t_notab 2023-06-22T08:24:09.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3a3l15f-t 2023-06-22T08:24:09.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W26l24f-ll 2023-06-22T08:24:09.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+7l24-FF 2023-06-22T08:24:09.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ml24f-lm-lo 2023-06-22T08:24:09.144 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/l24-FF 2023-06-22T08:24:09.144 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+2-5l24f-0FF 2023-06-22T08:24:09.144 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3f-FF 2023-06-22T08:24:09.144 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+3ml24f-tn-bl 2023-06-22T08:24:09.144 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ta3-0FF 2023-06-22T08:24:09.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3b3l15-t 2023-06-22T08:24:09.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+2l24f-0FF 2023-06-22T08:24:09.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tn_2e8S-t_tab 2023-06-22T08:24:09.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ta3-FF 2023-06-22T08:24:09.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tFFt-ll 2023-06-22T08:24:09.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+6b3l13f-FF 2023-06-22T08:24:09.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/2-Sf-t_notab 2023-06-22T08:24:09.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/0FFnt 2023-06-22T08:24:09.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W20l24f-ll 2023-06-22T08:24:09.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/Jml24f-lm-lo 2023-06-22T08:24:09.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3l24-t 2023-06-22T08:24:09.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/o3b3l24f-tn 2023-06-22T08:24:09.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W35a3l24f-lm 2023-06-22T08:24:09.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/2-S_f-t_notab 2023-06-22T08:24:09.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3-5l24f-t 2023-06-22T08:24:09.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/o3b3Snl24f-tn 2023-06-22T08:24:09.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+2l24f-bl 2023-06-22T08:24:09.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W35Ja3l24f-lm 2023-06-22T08:24:09.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+8l20-FF 2023-06-22T08:24:09.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/0FFt 2023-06-22T08:24:09.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/o3a3l24f-tn 2023-06-22T08:24:09.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/2s_f-t_notab 2023-06-22T08:24:09.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nN15l24f-bl 2023-06-22T08:24:09.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/0Fnt 2023-06-22T08:24:09.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/4-7l24-FF 2023-06-22T08:24:09.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tn 2023-06-22T08:24:09.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/b2l17-FF 2023-06-22T08:24:09.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nJml24f-lmlmlo 2023-06-22T08:24:09.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tta3-0FF 2023-06-22T08:24:09.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+5a3l13f-0FF 2023-06-22T08:24:09.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nN1+3l24f-bl 2023-06-22T08:24:09.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/2sw60f-t_notab 2023-06-22T08:24:09.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/W-72l24f-ll 2023-06-22T08:24:09.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tt-0FF 2023-06-22T08:24:09.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/4l24-FF 2023-06-22T08:24:09.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/3a3f-0F 2023-06-22T08:24:09.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tFFt 2023-06-22T08:24:09.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/n+6a2l17-FF 2023-06-22T08:24:09.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/a3l15f-t 2023-06-22T08:24:09.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/l24f-t 2023-06-22T08:24:09.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/b3f-0FF 2023-06-22T08:24:09.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tt-t 2023-06-22T08:24:09.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/ttb3-FF 2023-06-22T08:24:09.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/nSml20-t-t-FF 2023-06-22T08:24:09.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/pr/tne8o3-t_tab 2023-06-22T08:24:09.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/Coreutils.pm 2023-06-22T08:24:09.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.163+0000 7f50a7bff700 1 -- 172.21.15.23:0/2605531864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50a00cb300 con 0x7f50a0071070 2023-06-22T08:24:09.166 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.163+0000 7f50967fc700 1 -- 172.21.15.23:0/2605531864 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5484 (secure 0 0 0) 0x7f509c017090 con 0x7f50a0071070 2023-06-22T08:24:09.166 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:09.166 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6185,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:00.074498+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:09.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 -- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f508406ace0 msgr2=0x7f508406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:09.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f508406ace0 0x7f508406d190 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f50a014bad0 tx=0x7f5090007dc0 comp rx=0 tx=0).stop 2023-06-22T08:24:09.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 -- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0071070 msgr2=0x7f50a01f1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:09.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0071070 0x7f50a01f1a40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f509c001eb0 tx=0x7f509c00c7b0 comp rx=0 tx=0).stop 2023-06-22T08:24:09.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 -- 172.21.15.23:0/2605531864 shutdown_connections 2023-06-22T08:24:09.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50a014f850 0x7f50a013fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:09.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f508406ace0 0x7f508406d190 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:09.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0071070 0x7f50a01f1a40 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:09.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 --2- 172.21.15.23:0/2605531864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50a00c69e0 0x7f50a014b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:09.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 -- 172.21.15.23:0/2605531864 >> 172.21.15.23:0/2605531864 conn(0x7f50a00ca3f0 msgr2=0x7f50a014c370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:09.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 -- 172.21.15.23:0/2605531864 shutdown_connections 2023-06-22T08:24:09.170 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:09.166+0000 7f50a7bff700 1 -- 172.21.15.23:0/2605531864 wait complete. 2023-06-22T08:24:09.170 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 189 2023-06-22T08:24:09.186 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-06-22T08:24:09.187 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 18 secs before thrashing 2023-06-22T08:24:09.539 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rmdir/ 2023-06-22T08:24:09.540 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rmdir/t-slash 2023-06-22T08:24:09.541 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rmdir/ignore 2023-06-22T08:24:09.541 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rmdir/fail-perm 2023-06-22T08:24:09.542 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/ 2023-06-22T08:24:09.545 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/ignorable 2023-06-22T08:24:09.546 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/r-2 2023-06-22T08:24:09.546 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/fail-2eperm 2023-06-22T08:24:09.547 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/empty-inacc 2023-06-22T08:24:09.547 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/cycle 2023-06-22T08:24:09.547 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/sunos-1 2023-06-22T08:24:09.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/v-slash 2023-06-22T08:24:09.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/no-give-up 2023-06-22T08:24:09.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/dot-rel 2023-06-22T08:24:09.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/one-file-system 2023-06-22T08:24:09.549 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/i-never 2023-06-22T08:24:09.550 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/rm1 2023-06-22T08:24:09.550 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/unread2 2023-06-22T08:24:09.550 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/read-only 2023-06-22T08:24:09.550 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/deep-2 2023-06-22T08:24:09.550 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/r-4 2023-06-22T08:24:09.550 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/dir-no-w 2023-06-22T08:24:09.550 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/interactive-always 2023-06-22T08:24:09.551 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/one-file-system2 2023-06-22T08:24:09.551 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/unreadable 2023-06-22T08:24:09.551 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/r-1 2023-06-22T08:24:09.551 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/fail-eacces 2023-06-22T08:24:09.551 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/inaccessible 2023-06-22T08:24:09.551 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/unread3 2023-06-22T08:24:09.551 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/ir-1 2023-06-22T08:24:09.551 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/rm3 2023-06-22T08:24:09.552 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/rm5 2023-06-22T08:24:09.552 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/deep-1 2023-06-22T08:24:09.552 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/ext3-perf 2023-06-22T08:24:09.552 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/dangling-symlink 2023-06-22T08:24:09.552 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/isatty 2023-06-22T08:24:09.552 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/i-1 2023-06-22T08:24:09.552 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/dir-nonrecur 2023-06-22T08:24:09.553 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/interactive-once 2023-06-22T08:24:09.554 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/f-1 2023-06-22T08:24:09.554 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/fail-eperm 2023-06-22T08:24:09.554 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/empty-name 2023-06-22T08:24:09.554 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/hash 2023-06-22T08:24:09.554 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/i-no-r 2023-06-22T08:24:09.554 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/r-3 2023-06-22T08:24:09.554 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/readdir-bug 2023-06-22T08:24:09.555 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/rm2 2023-06-22T08:24:09.555 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/rm/rm4 2023-06-22T08:24:09.555 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/test-lib.sh 2023-06-22T08:24:09.555 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/sample-test 2023-06-22T08:24:09.555 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chown/ 2023-06-22T08:24:09.584 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chown/basic 2023-06-22T08:24:09.614 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chown/separator 2023-06-22T08:24:09.615 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chown/deref 2023-06-22T08:24:09.615 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chown/preserve-root 2023-06-22T08:24:09.615 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/id/ 2023-06-22T08:24:09.617 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/id/no-context 2023-06-22T08:24:09.621 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/CuTmpdir.pm 2023-06-22T08:24:09.621 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/Makefile.am 2023-06-22T08:24:09.621 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/ 2023-06-22T08:24:09.624 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/dir-slash 2023-06-22T08:24:09.627 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/preserve-2 2023-06-22T08:24:09.627 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/dir-vs-file 2023-06-22T08:24:09.628 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/preserve-slink-time 2023-06-22T08:24:09.628 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/same-file 2023-06-22T08:24:09.628 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/src-base-dot 2023-06-22T08:24:09.628 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/cp-a-selinux 2023-06-22T08:24:09.628 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/r-vs-symlink 2023-06-22T08:24:09.628 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/capability 2023-06-22T08:24:09.628 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/existing-perm-race 2023-06-22T08:24:09.629 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/parent-perm-race 2023-06-22T08:24:09.629 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/reflink-auto 2023-06-22T08:24:09.631 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/cp-mv-enotsup-xattr 2023-06-22T08:24:09.631 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/no-deref-link2 2023-06-22T08:24:09.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/cp-mv-backup 2023-06-22T08:24:09.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/link-no-deref 2023-06-22T08:24:09.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/file-perm-race 2023-06-22T08:24:09.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/abuse 2023-06-22T08:24:09.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/link-heap 2023-06-22T08:24:09.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/backup-dir 2023-06-22T08:24:09.632 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/preserve-gid 2023-06-22T08:24:09.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/link 2023-06-22T08:24:09.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/cp-parents 2023-06-22T08:24:09.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/no-deref-link1 2023-06-22T08:24:09.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/into-self 2023-06-22T08:24:09.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/parent-perm 2023-06-22T08:24:09.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/cp-deref 2023-06-22T08:24:09.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/proc-short-read 2023-06-22T08:24:09.633 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/backup-is-src 2023-06-22T08:24:09.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/cp-i 2023-06-22T08:24:09.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/special-f 2023-06-22T08:24:09.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/proc-zero-len 2023-06-22T08:24:09.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/slink-2-slink 2023-06-22T08:24:09.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/special-bits 2023-06-22T08:24:09.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/fail-perm 2023-06-22T08:24:09.634 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/link-preserve 2023-06-22T08:24:09.635 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/dir-rm-dest 2023-06-22T08:24:09.635 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/backup-1 2023-06-22T08:24:09.635 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/cp-HL 2023-06-22T08:24:09.635 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/thru-dangling 2023-06-22T08:24:09.635 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/sparse 2023-06-22T08:24:09.635 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/deref-slink 2023-06-22T08:24:09.635 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/symlink-slash 2023-06-22T08:24:09.635 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/acl 2023-06-22T08:24:09.636 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/perm 2023-06-22T08:24:09.636 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/reflink-perm 2023-06-22T08:24:09.636 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/cp/no-deref-link3 2023-06-22T08:24:09.636 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/ 2023-06-22T08:24:09.671 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/t-slash 2023-06-22T08:24:09.703 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/p-v 2023-06-22T08:24:09.703 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/p-3 2023-06-22T08:24:09.703 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/writable-under-readonly 2023-06-22T08:24:09.704 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/p-slashdot 2023-06-22T08:24:09.704 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/selinux 2023-06-22T08:24:09.704 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/p-1 2023-06-22T08:24:09.704 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/p-2 2023-06-22T08:24:09.704 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/parents 2023-06-22T08:24:09.704 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/special-1 2023-06-22T08:24:09.704 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/perm 2023-06-22T08:24:09.705 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mkdir/p-thru-slink 2023-06-22T08:24:09.705 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/require-perl 2023-06-22T08:24:09.705 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ln/ 2023-06-22T08:24:09.714 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ln/target-1 2023-06-22T08:24:09.726 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ln/hard-to-sym 2023-06-22T08:24:09.726 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ln/hard-backup 2023-06-22T08:24:09.726 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ln/sf-1 2023-06-22T08:24:09.726 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ln/backup-1 2023-06-22T08:24:09.726 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ln/slash-decorated-nonexistent-dest 2023-06-22T08:24:09.727 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ln/misc 2023-06-22T08:24:09.727 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/lang-default 2023-06-22T08:24:09.727 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/check.mk 2023-06-22T08:24:09.727 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/Makefile.in 2023-06-22T08:24:09.728 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/install/ 2023-06-22T08:24:09.737 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/install/create-leading 2023-06-22T08:24:09.745 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/install/install-C-selinux 2023-06-22T08:24:09.746 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/install/trap 2023-06-22T08:24:09.746 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/install/basic-1 2023-06-22T08:24:09.746 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/install/strip-program 2023-06-22T08:24:09.746 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/install/d-slashdot 2023-06-22T08:24:09.746 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/install/install-C 2023-06-22T08:24:09.747 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/install/install-C-root 2023-06-22T08:24:09.747 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/ 2023-06-22T08:24:09.755 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/c-option 2023-06-22T08:24:09.763 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/octal 2023-06-22T08:24:09.763 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/umask-x 2023-06-22T08:24:09.763 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/inaccessible 2023-06-22T08:24:09.764 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/equal-x 2023-06-22T08:24:09.764 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/silent 2023-06-22T08:24:09.764 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/equals 2023-06-22T08:24:09.764 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/usage 2023-06-22T08:24:09.764 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/thru-dangling 2023-06-22T08:24:09.764 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/setgid 2023-06-22T08:24:09.764 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chmod/no-x 2023-06-22T08:24:09.764 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/df/ 2023-06-22T08:24:09.769 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/df/unreadable 2023-06-22T08:24:09.772 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/df/total-verify 2023-06-22T08:24:09.773 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/other-fs-tmpdir 2023-06-22T08:24:09.773 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/ 2023-06-22T08:24:09.774 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/basic 2023-06-22T08:24:09.776 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/exclude 2023-06-22T08:24:09.777 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/one-file-system 2023-06-22T08:24:09.777 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/long-sloop 2023-06-22T08:24:09.777 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/slink 2023-06-22T08:24:09.777 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/inacc-dir 2023-06-22T08:24:09.777 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/inaccessible-cwd 2023-06-22T08:24:09.777 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/inacc-dest 2023-06-22T08:24:09.777 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/deref-args 2023-06-22T08:24:09.778 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/deref 2023-06-22T08:24:09.778 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/trailing-slash 2023-06-22T08:24:09.778 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/files0-from 2023-06-22T08:24:09.779 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/two-args 2023-06-22T08:24:09.779 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/hard-link 2023-06-22T08:24:09.779 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/long-from-unreadable 2023-06-22T08:24:09.780 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/8gb 2023-06-22T08:24:09.780 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/2g 2023-06-22T08:24:09.780 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/restore-wd 2023-06-22T08:24:09.780 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/slash 2023-06-22T08:24:09.780 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/no-x 2023-06-22T08:24:09.780 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/fd-leak 2023-06-22T08:24:09.780 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/du/no-deref 2023-06-22T08:24:09.781 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/ 2023-06-22T08:24:09.788 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/recursive 2023-06-22T08:24:09.796 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/readdir-mountpoint-inode 2023-06-22T08:24:09.796 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/multihardlink 2023-06-22T08:24:09.797 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/color-dtype-dir 2023-06-22T08:24:09.797 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/file-type 2023-06-22T08:24:09.797 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/rt-1 2023-06-22T08:24:09.797 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/stat-free-symlinks 2023-06-22T08:24:09.797 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/no-cap 2023-06-22T08:24:09.797 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/capability 2023-06-22T08:24:09.797 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/inode 2023-06-22T08:24:09.798 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/color-norm 2023-06-22T08:24:09.798 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/abmon-align 2023-06-22T08:24:09.798 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/dangle 2023-06-22T08:24:09.799 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/dired 2023-06-22T08:24:09.799 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/m-option 2023-06-22T08:24:09.799 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/nameless-uid 2023-06-22T08:24:09.800 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/proc-selinux-segfault 2023-06-22T08:24:09.800 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/stat-vs-dirent 2023-06-22T08:24:09.800 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/color-clear-to-eol 2023-06-22T08:24:09.800 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/no-arg 2023-06-22T08:24:09.800 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/follow-slink 2023-06-22T08:24:09.800 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/symlink-slash 2023-06-22T08:24:09.800 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/stat-dtype 2023-06-22T08:24:09.801 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/x-option 2023-06-22T08:24:09.801 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/infloop 2023-06-22T08:24:09.801 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/ls/stat-failed 2023-06-22T08:24:09.801 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/ 2023-06-22T08:24:09.817 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/hard-3 2023-06-22T08:24:09.831 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/mv-n 2023-06-22T08:24:09.831 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/to-symlink 2023-06-22T08:24:09.831 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/part-fail 2023-06-22T08:24:09.831 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/hard-4 2023-06-22T08:24:09.831 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/leak-fd 2023-06-22T08:24:09.831 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/no-target-dir 2023-06-22T08:24:09.832 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/part-rename 2023-06-22T08:24:09.832 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/dir-file 2023-06-22T08:24:09.832 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/hard-2 2023-06-22T08:24:09.832 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/part-hardlink 2023-06-22T08:24:09.832 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/sticky-to-xpart 2023-06-22T08:24:09.832 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/mv-special-1 2023-06-22T08:24:09.834 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/trailing-slash 2023-06-22T08:24:09.834 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/dir2dir 2023-06-22T08:24:09.835 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/into-self-2 2023-06-22T08:24:09.835 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/backup-dir 2023-06-22T08:24:09.835 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/diag 2023-06-22T08:24:09.835 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/atomic 2023-06-22T08:24:09.835 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/force 2023-06-22T08:24:09.835 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/into-self-3 2023-06-22T08:24:09.835 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/into-self 2023-06-22T08:24:09.836 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/i-2 2023-06-22T08:24:09.836 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/childproof 2023-06-22T08:24:09.836 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/perm-1 2023-06-22T08:24:09.836 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/i-3 2023-06-22T08:24:09.836 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/partition-perm 2023-06-22T08:24:09.836 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/part-symlink 2023-06-22T08:24:09.836 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/backup-is-src 2023-06-22T08:24:09.837 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/i-1 2023-06-22T08:24:09.837 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/update 2023-06-22T08:24:09.837 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/i-4 2023-06-22T08:24:09.837 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/hard-verbose 2023-06-22T08:24:09.837 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/dup-source 2023-06-22T08:24:09.837 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/i-link-no 2023-06-22T08:24:09.837 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/atomic2 2023-06-22T08:24:09.837 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/into-self-4 2023-06-22T08:24:09.838 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/hard-link-1 2023-06-22T08:24:09.838 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/i-5 2023-06-22T08:24:09.838 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/mv/acl 2023-06-22T08:24:09.838 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/ 2023-06-22T08:24:09.852 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/fifo 2023-06-22T08:24:09.867 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/empty-file 2023-06-22T08:24:09.867 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/read-only 2023-06-22T08:24:09.867 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/fail-diag 2023-06-22T08:24:09.867 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/trailing-slash 2023-06-22T08:24:09.867 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/no-dereference 2023-06-22T08:24:09.868 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/dir-1 2023-06-22T08:24:09.868 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/dangling-symlink 2023-06-22T08:24:09.868 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/now-owned-by-other 2023-06-22T08:24:09.868 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/no-create-missing 2023-06-22T08:24:09.868 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/no-rights 2023-06-22T08:24:09.868 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/obsolescent 2023-06-22T08:24:09.868 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/60-seconds 2023-06-22T08:24:09.869 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/not-owner 2023-06-22T08:24:09.869 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/touch/relative 2023-06-22T08:24:09.869 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/ 2023-06-22T08:24:09.874 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sort-merge-fdlimit 2023-06-22T08:24:09.881 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/basename 2023-06-22T08:24:09.881 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/stat-fmt 2023-06-22T08:24:09.881 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/pr 2023-06-22T08:24:09.882 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/dirname 2023-06-22T08:24:09.882 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/chcon-fail 2023-06-22T08:24:09.882 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/head-elide-tail 2023-06-22T08:24:09.882 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/chroot-credentials 2023-06-22T08:24:09.882 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/nl 2023-06-22T08:24:09.882 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/tee-dash 2023-06-22T08:24:09.882 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/stat-hyphen 2023-06-22T08:24:09.882 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sort-rand 2023-06-22T08:24:09.883 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/expr 2023-06-22T08:24:09.883 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/fmt-long-line 2023-06-22T08:24:09.883 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/split-fail 2023-06-22T08:24:09.883 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/truncate-relative 2023-06-22T08:24:09.883 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/truncate-dir-fail 2023-06-22T08:24:09.883 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/csplit 2023-06-22T08:24:09.883 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/od-N 2023-06-22T08:24:09.883 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/xattr 2023-06-22T08:24:09.884 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/mktemp 2023-06-22T08:24:09.888 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/cut 2023-06-22T08:24:09.888 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/cat-proc 2023-06-22T08:24:09.889 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/comm 2023-06-22T08:24:09.889 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/timeout-parameters 2023-06-22T08:24:09.889 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/tac-continue 2023-06-22T08:24:09.889 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/tty-eof 2023-06-22T08:24:09.889 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/df-P 2023-06-22T08:24:09.889 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/shuf 2023-06-22T08:24:09.889 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/env 2023-06-22T08:24:09.890 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/od 2023-06-22T08:24:09.890 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/split-a 2023-06-22T08:24:09.890 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/tail 2023-06-22T08:24:09.890 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/stat-slash 2023-06-22T08:24:09.890 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/timeout 2023-06-22T08:24:09.890 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/close-stdout 2023-06-22T08:24:09.890 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/wc-files0-from 2023-06-22T08:24:09.890 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sum 2023-06-22T08:24:09.891 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/expand 2023-06-22T08:24:09.891 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sum-sysv 2023-06-22T08:24:09.891 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/wc-parallel 2023-06-22T08:24:09.891 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/md5sum-newline 2023-06-22T08:24:09.891 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sha224sum 2023-06-22T08:24:09.891 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/stty 2023-06-22T08:24:09.891 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/printf-hex 2023-06-22T08:24:09.891 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/id-context 2023-06-22T08:24:09.892 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/stat-printf 2023-06-22T08:24:09.892 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/pwd-option 2023-06-22T08:24:09.892 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sha256sum 2023-06-22T08:24:09.892 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sort-version 2023-06-22T08:24:09.892 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/groups-dash 2023-06-22T08:24:09.892 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/printenv 2023-06-22T08:24:09.892 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/seq 2023-06-22T08:24:09.893 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/fmt 2023-06-22T08:24:09.893 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/cat-buf 2023-06-22T08:24:09.893 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/dircolors 2023-06-22T08:24:09.893 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/fold 2023-06-22T08:24:09.893 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/selinux 2023-06-22T08:24:09.893 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/md5sum-parallel 2023-06-22T08:24:09.893 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/md5sum 2023-06-22T08:24:09.893 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/nice-fail 2023-06-22T08:24:09.894 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/groups-version 2023-06-22T08:24:09.894 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/date-sec 2023-06-22T08:24:09.894 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/split-l 2023-06-22T08:24:09.894 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/false-status 2023-06-22T08:24:09.894 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sha384sum 2023-06-22T08:24:09.894 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/truncate-dangling-symlink 2023-06-22T08:24:09.894 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/od-multiple-t 2023-06-22T08:24:09.894 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/wc-files0 2023-06-22T08:24:09.895 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/nohup 2023-06-22T08:24:09.895 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/truncate-no-create-missing 2023-06-22T08:24:09.895 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/head-pos 2023-06-22T08:24:09.895 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/uniq 2023-06-22T08:24:09.895 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/test-diag 2023-06-22T08:24:09.895 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/tac 2023-06-22T08:24:09.895 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sort-continue 2023-06-22T08:24:09.895 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/head 2023-06-22T08:24:09.896 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/ptx 2023-06-22T08:24:09.896 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/nproc-avail 2023-06-22T08:24:09.896 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/su-fail 2023-06-22T08:24:09.896 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/od-x8 2023-06-22T08:24:09.896 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/nice 2023-06-22T08:24:09.897 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/chcon 2023-06-22T08:24:09.897 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/head-c 2023-06-22T08:24:09.897 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sort-compress 2023-06-22T08:24:09.897 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/df 2023-06-22T08:24:09.897 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/chroot-fail 2023-06-22T08:24:09.897 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/date-next-dow 2023-06-22T08:24:09.897 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sha1sum 2023-06-22T08:24:09.898 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sha1sum-vec 2023-06-22T08:24:09.898 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/pathchk1 2023-06-22T08:24:09.901 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/arch 2023-06-22T08:24:09.901 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sort-month 2023-06-22T08:24:09.901 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/shred-passes 2023-06-22T08:24:09.901 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sha512sum 2023-06-22T08:24:09.901 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/paste 2023-06-22T08:24:09.902 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/wc 2023-06-22T08:24:09.902 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/ls-time 2023-06-22T08:24:09.902 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/pwd-unreadable-parent 2023-06-22T08:24:09.902 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/test 2023-06-22T08:24:09.902 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/truncate-parameters 2023-06-22T08:24:09.902 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sort-files0-from 2023-06-22T08:24:09.902 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/base64 2023-06-22T08:24:09.903 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/ptx-overrun 2023-06-22T08:24:09.903 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/printf-cov 2023-06-22T08:24:09.903 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/tee 2023-06-22T08:24:09.903 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/mknod 2023-06-22T08:24:09.903 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/printf-surprise 2023-06-22T08:24:09.903 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/ls-misc 2023-06-22T08:24:09.903 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/nproc-positive 2023-06-22T08:24:09.903 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/join 2023-06-22T08:24:09.904 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sort-merge 2023-06-22T08:24:09.904 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/factor 2023-06-22T08:24:09.904 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/sort 2023-06-22T08:24:09.904 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/truncate-fifo 2023-06-22T08:24:09.904 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/truncate-fail-diag 2023-06-22T08:24:09.904 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/stty-invalid 2023-06-22T08:24:09.904 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/env-null 2023-06-22T08:24:09.904 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/printf 2023-06-22T08:24:09.905 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/seq-long-double 2023-06-22T08:24:09.905 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/pwd-long 2023-06-22T08:24:09.905 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/tsort 2023-06-22T08:24:09.905 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/runcon-no-reorder 2023-06-22T08:24:09.905 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/date 2023-06-22T08:24:09.905 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/shred-exact 2023-06-22T08:24:09.905 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/shred-remove 2023-06-22T08:24:09.906 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/stdbuf 2023-06-22T08:24:09.906 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/truncate-overflow 2023-06-22T08:24:09.906 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/stty-row-col 2023-06-22T08:24:09.906 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/truncate-owned-by-other 2023-06-22T08:24:09.906 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/xstrtol 2023-06-22T08:24:09.906 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/invalid-opt 2023-06-22T08:24:09.907 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/readlink-fp-loop 2023-06-22T08:24:09.907 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/tr 2023-06-22T08:24:09.907 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/help-version 2023-06-22T08:24:09.907 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/id-groups 2023-06-22T08:24:09.907 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/misc/unexpand 2023-06-22T08:24:09.907 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chgrp/ 2023-06-22T08:24:09.967 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chgrp/basic 2023-06-22T08:24:10.052 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chgrp/default-no-deref 2023-06-22T08:24:10.053 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chgrp/deref 2023-06-22T08:24:10.053 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chgrp/posix-H 2023-06-22T08:24:10.053 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chgrp/recurse 2023-06-22T08:24:10.053 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/chgrp/no-x 2023-06-22T08:24:10.054 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/ 2023-06-22T08:24:10.057 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/big-4gb 2023-06-22T08:24:10.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/follow-stdin 2023-06-22T08:24:10.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/F-vs-rename 2023-06-22T08:24:10.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/inotify-rotate 2023-06-22T08:24:10.062 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/pid 2023-06-22T08:24:10.063 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/infloop-1 2023-06-22T08:24:10.063 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/assert-2 2023-06-22T08:24:10.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/pipe-f 2023-06-22T08:24:10.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/pipe-f2 2023-06-22T08:24:10.064 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/start-middle 2023-06-22T08:24:10.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/inotify-hash-abuse2 2023-06-22T08:24:10.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/inotify-hash-abuse 2023-06-22T08:24:10.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/inotify-race 2023-06-22T08:24:10.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/append-only 2023-06-22T08:24:10.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/tail-n0f 2023-06-22T08:24:10.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/flush-initial 2023-06-22T08:24:10.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/assert 2023-06-22T08:24:10.065 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/wait 2023-06-22T08:24:10.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/tail-2/proc-ksyms 2023-06-22T08:24:10.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/envvar-check 2023-06-22T08:24:10.066 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/readlink/ 2023-06-22T08:24:10.079 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/readlink/can-f 2023-06-22T08:24:10.092 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/readlink/can-m 2023-06-22T08:24:10.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/readlink/rl-1 2023-06-22T08:24:10.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/tests/readlink/can-e 2023-06-22T08:24:10.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_trailing_blank 2023-06-22T08:24:10.093 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/ 2023-06-22T08:24:10.098 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/coreutils.texi 2023-06-22T08:24:10.104 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/perm.texi 2023-06-22T08:24:10.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/version.texi 2023-06-22T08:24:10.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/Makefile.am 2023-06-22T08:24:10.105 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/coreutils.info 2023-06-22T08:24:10.109 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/stamp-vti 2023-06-22T08:24:10.109 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/fdl.texi 2023-06-22T08:24:10.109 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/Makefile.in 2023-06-22T08:24:10.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/getdate.texi 2023-06-22T08:24:10.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/constants.texi 2023-06-22T08:24:10.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/doc/ChangeLog-2007 2023-06-22T08:24:10.110 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/ 2023-06-22T08:24:10.116 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/split.c 2023-06-22T08:24:10.122 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/dircolors.hin 2023-06-22T08:24:10.122 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/test.c 2023-06-22T08:24:10.122 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/nproc.c 2023-06-22T08:24:10.122 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/cp-hash.c 2023-06-22T08:24:10.122 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/rm.c 2023-06-22T08:24:10.124 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/uname-arch.c 2023-06-22T08:24:10.124 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/chroot.c 2023-06-22T08:24:10.125 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/remove.h 2023-06-22T08:24:10.125 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/users.c 2023-06-22T08:24:10.125 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/dircolors.c 2023-06-22T08:24:10.125 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/join.c 2023-06-22T08:24:10.125 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/chown.c 2023-06-22T08:24:10.125 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/shuf.c 2023-06-22T08:24:10.126 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/sum.c 2023-06-22T08:24:10.126 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/csplit.c 2023-06-22T08:24:10.126 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/libstdbuf.c 2023-06-22T08:24:10.126 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/fold.c 2023-06-22T08:24:10.126 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/basename.c 2023-06-22T08:24:10.126 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/prog-fprintf.h 2023-06-22T08:24:10.126 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/sync.c 2023-06-22T08:24:10.127 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/lbracket.c 2023-06-22T08:24:10.127 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/ln.c 2023-06-22T08:24:10.127 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/tsort.c 2023-06-22T08:24:10.127 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/wc.c 2023-06-22T08:24:10.127 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/extract-magic 2023-06-22T08:24:10.127 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/hostid.c 2023-06-22T08:24:10.127 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/dd.c 2023-06-22T08:24:10.128 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/shred.c 2023-06-22T08:24:10.128 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/uname.h 2023-06-22T08:24:10.128 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/unlink.c 2023-06-22T08:24:10.128 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/Makefile.am 2023-06-22T08:24:10.130 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/seq.c 2023-06-22T08:24:10.130 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/uname-uname.c 2023-06-22T08:24:10.130 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/ptx.c 2023-06-22T08:24:10.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/wheel-size.h 2023-06-22T08:24:10.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/stty.c 2023-06-22T08:24:10.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/chcon.c 2023-06-22T08:24:10.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/dcgen 2023-06-22T08:24:10.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/uptime.c 2023-06-22T08:24:10.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/link.c 2023-06-22T08:24:10.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/install.c 2023-06-22T08:24:10.131 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/uname.c 2023-06-22T08:24:10.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/mknod.c 2023-06-22T08:24:10.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/su.c 2023-06-22T08:24:10.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/expand.c 2023-06-22T08:24:10.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/mv.c 2023-06-22T08:24:10.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/ls.c 2023-06-22T08:24:10.132 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/whoami.c 2023-06-22T08:24:10.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/ls-ls.c 2023-06-22T08:24:10.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/ls.h 2023-06-22T08:24:10.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/id.c 2023-06-22T08:24:10.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/Makefile.in 2023-06-22T08:24:10.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/du.c 2023-06-22T08:24:10.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/cut.c 2023-06-22T08:24:10.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/comm.c 2023-06-22T08:24:10.133 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/od.c 2023-06-22T08:24:10.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/expr.c 2023-06-22T08:24:10.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/nohup.c 2023-06-22T08:24:10.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/system.h 2023-06-22T08:24:10.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/cat.c 2023-06-22T08:24:10.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/mktemp.c 2023-06-22T08:24:10.134 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/tac.c 2023-06-22T08:24:10.139 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/ls-dir.c 2023-06-22T08:24:10.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/tac-pipe.c 2023-06-22T08:24:10.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/touch.c 2023-06-22T08:24:10.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/false.c 2023-06-22T08:24:10.140 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/echo.c 2023-06-22T08:24:10.141 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/printenv.c 2023-06-22T08:24:10.141 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/logname.c 2023-06-22T08:24:10.141 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/group-list.c 2023-06-22T08:24:10.141 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/pr.c 2023-06-22T08:24:10.141 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/chgrp.c 2023-06-22T08:24:10.142 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/cp-hash.h 2023-06-22T08:24:10.142 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/getlimits.c 2023-06-22T08:24:10.142 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/rmdir.c 2023-06-22T08:24:10.142 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/tail.c 2023-06-22T08:24:10.142 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/timeout.c 2023-06-22T08:24:10.143 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/sort.c 2023-06-22T08:24:10.143 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/operand2sig.c 2023-06-22T08:24:10.143 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/factor.c 2023-06-22T08:24:10.143 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/paste.c 2023-06-22T08:24:10.143 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/chmod.c 2023-06-22T08:24:10.144 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/pinky.c 2023-06-22T08:24:10.144 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/unexpand.c 2023-06-22T08:24:10.144 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/cksum.c 2023-06-22T08:24:10.144 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/date.c 2023-06-22T08:24:10.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/mkdir.c 2023-06-22T08:24:10.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/printf.c 2023-06-22T08:24:10.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/true.c 2023-06-22T08:24:10.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/c99-to-c89.diff 2023-06-22T08:24:10.145 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/groups.c 2023-06-22T08:24:10.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/dirname.c 2023-06-22T08:24:10.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/chown-core.h 2023-06-22T08:24:10.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/fs.h 2023-06-22T08:24:10.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/nl.c 2023-06-22T08:24:10.146 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/tee.c 2023-06-22T08:24:10.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/dircolors.h 2023-06-22T08:24:10.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/head.c 2023-06-22T08:24:10.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/uniq.c 2023-06-22T08:24:10.147 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/chown-core.c 2023-06-22T08:24:10.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/runcon.c 2023-06-22T08:24:10.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/sleep.c 2023-06-22T08:24:10.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/md5sum.c 2023-06-22T08:24:10.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/yes.c 2023-06-22T08:24:10.148 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/readlink.c 2023-06-22T08:24:10.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/env.c 2023-06-22T08:24:10.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/df.c 2023-06-22T08:24:10.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/nice.c 2023-06-22T08:24:10.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/fmt.c 2023-06-22T08:24:10.149 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/base64.c 2023-06-22T08:24:10.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/tty.c 2023-06-22T08:24:10.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/truncate.c 2023-06-22T08:24:10.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/mkfifo.c 2023-06-22T08:24:10.150 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/operand2sig.h 2023-06-22T08:24:10.151 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/copy.h 2023-06-22T08:24:10.151 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/kill.c 2023-06-22T08:24:10.151 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/who.c 2023-06-22T08:24:10.151 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/stat.c 2023-06-22T08:24:10.151 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/remove.c 2023-06-22T08:24:10.152 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/pwd.c 2023-06-22T08:24:10.152 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/prog-fprintf.c 2023-06-22T08:24:10.152 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/ls-vdir.c 2023-06-22T08:24:10.152 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/copy.c 2023-06-22T08:24:10.152 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/hostname.c 2023-06-22T08:24:10.152 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/wheel-gen.pl 2023-06-22T08:24:10.153 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/group-list.h 2023-06-22T08:24:10.153 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/cp.c 2023-06-22T08:24:10.153 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/setuidgid.c 2023-06-22T08:24:10.153 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/tr.c 2023-06-22T08:24:10.155 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/stdbuf.c 2023-06-22T08:24:10.155 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/pathchk.c 2023-06-22T08:24:10.155 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/src/wheel.h 2023-06-22T08:24:10.155 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/configure 2023-06-22T08:24:10.155 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/README 2023-06-22T08:24:10.155 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.vg-suppressions 2023-06-22T08:24:10.156 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_prohibit_atoi_atof 2023-06-22T08:24:10.156 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_prohibit_stat_st_blocks 2023-06-22T08:24:10.156 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/ChangeLog-2006 2023-06-22T08:24:10.156 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_program_name 2023-06-22T08:24:10.156 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/THANKStt.in 2023-06-22T08:24:10.156 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/ChangeLog-2008 2023-06-22T08:24:10.156 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/ChangeLog-2007 2023-06-22T08:24:10.157 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/ 2023-06-22T08:24:10.242 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-isnand.h 2023-06-22T08:24:10.282 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-ftello.sh 2023-06-22T08:24:10.282 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-snprintf.c 2023-06-22T08:24:10.282 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-openat.c 2023-06-22T08:24:10.282 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-dirent-safer.c 2023-06-22T08:24:10.283 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-cloexec.c 2023-06-22T08:24:10.283 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fstatat.c 2023-06-22T08:24:10.283 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fnmatch.c 2023-06-22T08:24:10.283 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getndelim2.c 2023-06-22T08:24:10.283 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-wcrtomb.sh 2023-06-22T08:24:10.283 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-filenamecat.c 2023-06-22T08:24:10.283 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fchownat.c 2023-06-22T08:24:10.283 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-stdlib.c 2023-06-22T08:24:10.284 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-read-file.c 2023-06-22T08:24:10.284 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-exclude7.sh 2023-06-22T08:24:10.284 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-closein.c 2023-06-22T08:24:10.284 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/connect.c 2023-06-22T08:24:10.284 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-rename.h 2023-06-22T08:24:10.284 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-memchr2.c 2023-06-22T08:24:10.284 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sys_time.c 2023-06-22T08:24:10.285 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/read-file.h 2023-06-22T08:24:10.285 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mkdirat.c 2023-06-22T08:24:10.285 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-isnand-nolibm.c 2023-06-22T08:24:10.285 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fopen-safer.c 2023-06-22T08:24:10.285 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-stdio.c 2023-06-22T08:24:10.285 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fseeko.c 2023-06-22T08:24:10.285 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-arpa_inet.c 2023-06-22T08:24:10.285 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freadahead.sh 2023-06-22T08:24:10.286 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-openat-safer.c 2023-06-22T08:24:10.286 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-unlink.c 2023-06-22T08:24:10.286 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsstr2.sh 2023-06-22T08:24:10.286 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-lchown.c 2023-06-22T08:24:10.286 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-nl_langinfo.c 2023-06-22T08:24:10.286 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-open.h 2023-06-22T08:24:10.286 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-memcmp.c 2023-06-22T08:24:10.287 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-vfprintf-posix.c 2023-06-22T08:24:10.287 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freadahead.c 2023-06-22T08:24:10.288 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/ioctl.c 2023-06-22T08:24:10.288 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-iconv-h.c 2023-06-22T08:24:10.288 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mkdir.c 2023-06-22T08:24:10.288 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-posixtm.c 2023-06-22T08:24:10.288 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/wctob.c 2023-06-22T08:24:10.288 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/glthread/ 2023-06-22T08:24:10.302 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/glthread/thread.c 2023-06-22T08:24:10.316 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/glthread/yield.h 2023-06-22T08:24:10.316 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/glthread/thread.h 2023-06-22T08:24:10.316 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/unistr/ 2023-06-22T08:24:10.318 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/unistr/test-u8-uctomb.c 2023-06-22T08:24:10.319 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-exclude5.sh 2023-06-22T08:24:10.319 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-rmdir.h 2023-06-22T08:24:10.320 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-symlink.h 2023-06-22T08:24:10.320 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-unsetenv.c 2023-06-22T08:24:10.320 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-exclude4.sh 2023-06-22T08:24:10.320 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/listen.c 2023-06-22T08:24:10.324 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-printf-posix.output 2023-06-22T08:24:10.325 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbrtowc3.sh 2023-06-22T08:24:10.325 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-dirent.c 2023-06-22T08:24:10.325 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-utimens-common.h 2023-06-22T08:24:10.325 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsinit.c 2023-06-22T08:24:10.325 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/w32sock.h 2023-06-22T08:24:10.325 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-printf-posix.h 2023-06-22T08:24:10.325 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fchdir.c 2023-06-22T08:24:10.326 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/read-file.c 2023-06-22T08:24:10.326 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-ctype.c 2023-06-22T08:24:10.326 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/binary-io.h 2023-06-22T08:24:10.326 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/nap.h 2023-06-22T08:24:10.326 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/sys_ioctl.in.h 2023-06-22T08:24:10.326 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-ftello2.sh 2023-06-22T08:24:10.326 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-memchr.c 2023-06-22T08:24:10.327 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsrtowcs3.sh 2023-06-22T08:24:10.327 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fseeko.sh 2023-06-22T08:24:10.327 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-select-out.sh 2023-06-22T08:24:10.327 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/inet_pton.c 2023-06-22T08:24:10.327 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-stdbool.c 2023-06-22T08:24:10.327 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-nanosleep.c 2023-06-22T08:24:10.327 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/bind.c 2023-06-22T08:24:10.327 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/unlinkdir.h 2023-06-22T08:24:10.328 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getline.c 2023-06-22T08:24:10.328 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-md5.c 2023-06-22T08:24:10.328 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/Makefile.am 2023-06-22T08:24:10.328 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-select-in.sh 2023-06-22T08:24:10.328 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-uname.c 2023-06-22T08:24:10.328 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-readlink.h 2023-06-22T08:24:10.328 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-string.c 2023-06-22T08:24:10.328 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-argmatch.c 2023-06-22T08:24:10.329 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-nl_langinfo.sh 2023-06-22T08:24:10.329 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sys_stat.c 2023-06-22T08:24:10.329 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-version-etc.c 2023-06-22T08:24:10.329 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-areadlink.h 2023-06-22T08:24:10.329 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-version-etc.sh 2023-06-22T08:24:10.329 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-binary-io.c 2023-06-22T08:24:10.329 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-exclude2.sh 2023-06-22T08:24:10.329 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xfprintf-posix.c 2023-06-22T08:24:10.330 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-set-mode-acl.c 2023-06-22T08:24:10.330 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-wchar.c 2023-06-22T08:24:10.330 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-vasprintf.c 2023-06-22T08:24:10.330 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xstrtoimax.sh 2023-06-22T08:24:10.330 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freadseek.sh 2023-06-22T08:24:10.330 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-isblank.c 2023-06-22T08:24:10.330 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fpending.sh 2023-06-22T08:24:10.330 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-lstat.h 2023-06-22T08:24:10.331 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-chown.c 2023-06-22T08:24:10.331 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xstrtoumax.sh 2023-06-22T08:24:10.331 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/nan.h 2023-06-22T08:24:10.331 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xprintf-posix.c 2023-06-22T08:24:10.331 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-inet_pton.c 2023-06-22T08:24:10.331 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-dup-safer.c 2023-06-22T08:24:10.331 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-closein.sh 2023-06-22T08:24:10.331 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-canonicalize.c 2023-06-22T08:24:10.332 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-isnanl-nolibm.c 2023-06-22T08:24:10.332 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xstrtol.c 2023-06-22T08:24:10.332 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-open.c 2023-06-22T08:24:10.332 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fprintf-posix.h 2023-06-22T08:24:10.332 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbrtowc1.sh 2023-06-22T08:24:10.332 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-errno.c 2023-06-22T08:24:10.332 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-netdb.c 2023-06-22T08:24:10.332 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-wctype.c 2023-06-22T08:24:10.333 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-strsignal.c 2023-06-22T08:24:10.333 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsstr3.sh 2023-06-22T08:24:10.333 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-yesno.c 2023-06-22T08:24:10.333 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-userspec.c 2023-06-22T08:24:10.333 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freopen.c 2023-06-22T08:24:10.333 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/uniwidth/ 2023-06-22T08:24:10.386 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/uniwidth/test-uc_width2.c 2023-06-22T08:24:10.510 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/uniwidth/test-uc_width.c 2023-06-22T08:24:10.511 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/uniwidth/test-uc_width2.sh 2023-06-22T08:24:10.511 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sockets.c 2023-06-22T08:24:10.511 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-link.h 2023-06-22T08:24:10.511 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-select-fd.c 2023-06-22T08:24:10.511 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xstrtoimax.c 2023-06-22T08:24:10.511 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xvasprintf.c 2023-06-22T08:24:10.512 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-yesno.sh 2023-06-22T08:24:10.512 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fflush.c 2023-06-22T08:24:10.512 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-vc-list-files-cvs.sh 2023-06-22T08:24:10.512 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbscasecmp.sh 2023-06-22T08:24:10.512 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-lseek.sh 2023-06-22T08:24:10.512 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/usleep.c 2023-06-22T08:24:10.512 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sleep.c 2023-06-22T08:24:10.513 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-link.c 2023-06-22T08:24:10.513 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-perror.c 2023-06-22T08:24:10.513 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsrtowcs.c 2023-06-22T08:24:10.514 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xalloc-die.sh 2023-06-22T08:24:10.515 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-i-ring.c 2023-06-22T08:24:10.515 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-priv-set.c 2023-06-22T08:24:10.515 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-btowc.c 2023-06-22T08:24:10.515 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sys_select.c 2023-06-22T08:24:10.515 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbrtowc.c 2023-06-22T08:24:10.515 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-strerror.c 2023-06-22T08:24:10.515 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-update-copyright.sh 2023-06-22T08:24:10.516 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-btowc2.sh 2023-06-22T08:24:10.516 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getcwd.c 2023-06-22T08:24:10.516 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/socket.c 2023-06-22T08:24:10.516 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-set-mode-acl.sh 2023-06-22T08:24:10.516 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-linkat.c 2023-06-22T08:24:10.516 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/perror.c 2023-06-22T08:24:10.516 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-wcrtomb.c 2023-06-22T08:24:10.517 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-vasnprintf.c 2023-06-22T08:24:10.517 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-memrchr.c 2023-06-22T08:24:10.517 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-ftello.c 2023-06-22T08:24:10.517 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-chown.h 2023-06-22T08:24:10.517 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-stat.h 2023-06-22T08:24:10.517 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/Makefile.in 2023-06-22T08:24:10.525 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getdelim.c 2023-06-22T08:24:10.525 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-vprintf-posix.sh 2023-06-22T08:24:10.525 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fseeko2.sh 2023-06-22T08:24:10.526 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-c-strcase.sh 2023-06-22T08:24:10.526 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsalign.c 2023-06-22T08:24:10.526 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fcntl-h.c 2023-06-22T08:24:10.526 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-vprintf-posix.c 2023-06-22T08:24:10.526 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-rename.c 2023-06-22T08:24:10.526 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-lstat.c 2023-06-22T08:24:10.526 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-malloca.c 2023-06-22T08:24:10.527 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xstrtoul.c 2023-06-22T08:24:10.527 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-perror.sh 2023-06-22T08:24:10.527 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-vasprintf-posix.c 2023-06-22T08:24:10.530 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-frexp.c 2023-06-22T08:24:10.530 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-environ.c 2023-06-22T08:24:10.530 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-stdint.c 2023-06-22T08:24:10.531 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-isnanl.h 2023-06-22T08:24:10.531 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fcntl-safer.c 2023-06-22T08:24:10.531 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-locale.c 2023-06-22T08:24:10.531 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbrtowc4.sh 2023-06-22T08:24:10.531 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-argv-iter.c 2023-06-22T08:24:10.531 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-utimens.h 2023-06-22T08:24:10.531 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xalloc-die.c 2023-06-22T08:24:10.532 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fcntl.c 2023-06-22T08:24:10.532 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-alloca-opt.c 2023-06-22T08:24:10.532 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-dirname.c 2023-06-22T08:24:10.532 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-setenv.c 2023-06-22T08:24:10.532 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-printf-frexpl.c 2023-06-22T08:24:10.532 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-atexit.sh 2023-06-22T08:24:10.532 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-stddef.c 2023-06-22T08:24:10.532 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-c-ctype.c 2023-06-22T08:24:10.533 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xstrtol.sh 2023-06-22T08:24:10.533 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getopt_long.h 2023-06-22T08:24:10.533 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-quotearg.sh 2023-06-22T08:24:10.533 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-langinfo.c 2023-06-22T08:24:10.534 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-inttypes.c 2023-06-22T08:24:10.534 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-filevercmp.c 2023-06-22T08:24:10.534 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-rmdir.c 2023-06-22T08:24:10.534 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-futimens.h 2023-06-22T08:24:10.534 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/setsockopt.c 2023-06-22T08:24:10.534 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-exclude1.sh 2023-06-22T08:24:10.534 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freopen-safer.c 2023-06-22T08:24:10.534 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getaddrinfo.c 2023-06-22T08:24:10.535 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-iconv.c 2023-06-22T08:24:10.535 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/zerosize-ptr.h 2023-06-22T08:24:10.535 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-lseek.c 2023-06-22T08:24:10.535 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-usleep.c 2023-06-22T08:24:10.535 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/gnulib.mk 2023-06-22T08:24:10.535 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-unlinkat.c 2023-06-22T08:24:10.535 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-quotearg.c 2023-06-22T08:24:10.566 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-base64.c 2023-06-22T08:24:10.566 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-frexpl.c 2023-06-22T08:24:10.567 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-gethostname.c 2023-06-22T08:24:10.567 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mkdir.h 2023-06-22T08:24:10.567 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-binary-io.sh 2023-06-22T08:24:10.567 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sys_ioctl.c 2023-06-22T08:24:10.567 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-time.c 2023-06-22T08:24:10.567 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/signature.h 2023-06-22T08:24:10.567 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-isnanf.h 2023-06-22T08:24:10.567 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-isnanf-nolibm.c 2023-06-22T08:24:10.568 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-wcwidth.c 2023-06-22T08:24:10.568 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freadseek.c 2023-06-22T08:24:10.568 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-areadlink-with-size.c 2023-06-22T08:24:10.568 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-select-stdin.c 2023-06-22T08:24:10.568 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsstr1.c 2023-06-22T08:24:10.568 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-exclude3.sh 2023-06-22T08:24:10.568 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fopen.c 2023-06-22T08:24:10.587 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-lock.c 2023-06-22T08:24:10.587 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/locale/ 2023-06-22T08:24:12.459 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/locale/fr/ 2023-06-22T08:24:12.462 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/ 2023-06-22T08:24:12.462 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/test-quotearg.mo 2023-06-22T08:24:12.464 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/test-quotearg.po 2023-06-22T08:24:12.464 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fpending.c 2023-06-22T08:24:12.466 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-netinet_in.c 2023-06-22T08:24:12.466 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-bitrotate.c 2023-06-22T08:24:12.467 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-c-strcasecmp.c 2023-06-22T08:24:12.467 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsstr3.c 2023-06-22T08:24:12.467 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-hash.c 2023-06-22T08:24:12.467 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-copy-acl.c 2023-06-22T08:24:12.468 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sigaction.c 2023-06-22T08:24:12.469 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xstrtoumax.c 2023-06-22T08:24:12.470 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fseterr.c 2023-06-22T08:24:12.470 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-areadlink.c 2023-06-22T08:24:12.470 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-signbit.c 2023-06-22T08:24:12.470 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-btowc1.sh 2023-06-22T08:24:12.470 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-strtod.c 2023-06-22T08:24:12.470 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-atexit.c 2023-06-22T08:24:12.471 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getdate.c 2023-06-22T08:24:12.471 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fpurge.c 2023-06-22T08:24:12.471 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-select.c 2023-06-22T08:24:12.471 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-tls.c 2023-06-22T08:24:12.471 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/sleep.c 2023-06-22T08:24:12.471 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbrtowc2.sh 2023-06-22T08:24:12.471 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-stat.c 2023-06-22T08:24:12.472 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freadptr.sh 2023-06-22T08:24:12.472 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/accept.c 2023-06-22T08:24:12.472 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-unistd.c 2023-06-22T08:24:12.472 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fopen.h 2023-06-22T08:24:12.472 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-math.c 2023-06-22T08:24:12.472 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-printf-frexp.c 2023-06-22T08:24:12.472 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-vfprintf-posix.sh 2023-06-22T08:24:12.472 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-exclude.c 2023-06-22T08:24:12.475 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freading.c 2023-06-22T08:24:12.476 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getdtablesize.c 2023-06-22T08:24:12.476 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/unlinkdir.c 2023-06-22T08:24:12.476 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fflush2.c 2023-06-22T08:24:12.477 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-stat-time.c 2023-06-22T08:24:12.477 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-inet_ntop.c 2023-06-22T08:24:12.477 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-vc-list-files-git.sh 2023-06-22T08:24:12.477 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getopt.c 2023-06-22T08:24:12.477 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-alignof.c 2023-06-22T08:24:12.477 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/macros.h 2023-06-22T08:24:12.478 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-symlinkat.c 2023-06-22T08:24:12.478 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getgroups.c 2023-06-22T08:24:12.478 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-xprintf-posix.sh 2023-06-22T08:24:12.478 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fdopendir.c 2023-06-22T08:24:12.478 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sys_socket.c 2023-06-22T08:24:12.478 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbscasecmp.c 2023-06-22T08:24:12.479 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-getopt.h 2023-06-22T08:24:12.479 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsrtowcs1.sh 2023-06-22T08:24:12.479 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsinit.sh 2023-06-22T08:24:12.479 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-gettimeofday.c 2023-06-22T08:24:12.479 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsstr2.c 2023-06-22T08:24:12.479 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/init.sh 2023-06-22T08:24:12.479 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-symlink.c 2023-06-22T08:24:12.479 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-utimens.c 2023-06-22T08:24:12.480 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-readlink.c 2023-06-22T08:24:12.480 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-lchown.h 2023-06-22T08:24:12.480 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-c-strncasecmp.c 2023-06-22T08:24:12.480 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-lutimens.h 2023-06-22T08:24:12.480 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-copy-acl.sh 2023-06-22T08:24:12.481 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-exclude6.sh 2023-06-22T08:24:12.481 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freadptr2.c 2023-06-22T08:24:12.481 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fflush2.sh 2023-06-22T08:24:12.481 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sys_utsname.c 2023-06-22T08:24:12.481 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsrtowcs2.sh 2023-06-22T08:24:12.481 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-file-has-acl.c 2023-06-22T08:24:12.481 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-signal.c 2023-06-22T08:24:12.481 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sha1.c 2023-06-22T08:24:12.482 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freadptr2.sh 2023-06-22T08:24:12.482 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-areadlinkat.c 2023-06-22T08:24:12.482 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-unlink.h 2023-06-22T08:24:12.482 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-dup2.c 2023-06-22T08:24:12.482 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-freadptr.c 2023-06-22T08:24:12.482 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-file-has-acl.sh 2023-06-22T08:24:12.483 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-u64.c 2023-06-22T08:24:12.483 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-striconv.c 2023-06-22T08:24:12.484 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-sameacls.c 2023-06-22T08:24:12.484 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-remove.c 2023-06-22T08:24:12.484 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-fsync.c 2023-06-22T08:24:12.484 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/gnulib-tests/test-mbsrtowcs4.sh 2023-06-22T08:24:12.484 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_require_config_h 2023-06-22T08:24:12.484 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_system_h_headers 2023-06-22T08:24:12.485 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_prohibit_fail_0 2023-06-22T08:24:12.485 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_prohibit_magic_number_exit 2023-06-22T08:24:12.485 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/THANKS-to-translators 2023-06-22T08:24:12.485 INFO:tasks.workunit.client.0.smithi023.stdout:coreutils-8.5/.x-sc_useless_cpp_parens 2023-06-22T08:24:12.485 INFO:tasks.workunit.client.0.smithi023.stderr:+ mkdir .snap/k 2023-06-22T08:24:12.671 INFO:tasks.workunit.client.0.smithi023.stderr:+ sync 2023-06-22T08:24:12.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~230s 2023-06-22T08:24:13.912 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -rv coreutils-8.5 2023-06-22T08:24:14.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_prohibit_strcmp' 2023-06-22T08:24:14.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_GPL_version' 2023-06-22T08:24:14.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.prev-version' 2023-06-22T08:24:14.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_program_name' 2023-06-22T08:24:14.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/expr.x' 2023-06-22T08:24:14.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/hostid.1' 2023-06-22T08:24:14.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/seq.x' 2023-06-22T08:24:14.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/yes.1' 2023-06-22T08:24:14.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tee.x' 2023-06-22T08:24:14.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/cat.x' 2023-06-22T08:24:14.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/true.x' 2023-06-22T08:24:14.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/dirname.1' 2023-06-22T08:24:14.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tail.1' 2023-06-22T08:24:14.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/timeout.1' 2023-06-22T08:24:14.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/uname.1' 2023-06-22T08:24:14.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/unlink.1' 2023-06-22T08:24:14.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/expr.1' 2023-06-22T08:24:14.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/printf.1' 2023-06-22T08:24:14.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/logname.x' 2023-06-22T08:24:14.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/expand.1' 2023-06-22T08:24:14.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sum.x' 2023-06-22T08:24:14.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/md5sum.x' 2023-06-22T08:24:14.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/env.x' 2023-06-22T08:24:14.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/nice.x' 2023-06-22T08:24:14.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/date.1' 2023-06-22T08:24:14.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tr.1' 2023-06-22T08:24:14.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/join.x' 2023-06-22T08:24:14.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/stty.1' 2023-06-22T08:24:14.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/seq.1' 2023-06-22T08:24:14.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/pinky.x' 2023-06-22T08:24:14.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/hostname.x' 2023-06-22T08:24:14.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/install.1' 2023-06-22T08:24:14.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sha512sum.x' 2023-06-22T08:24:14.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/install.x' 2023-06-22T08:24:14.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tail.x' 2023-06-22T08:24:14.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/dircolors.1' 2023-06-22T08:24:14.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/hostid.x' 2023-06-22T08:24:14.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/md5sum.1' 2023-06-22T08:24:14.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/split.x' 2023-06-22T08:24:14.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/mv.x' 2023-06-22T08:24:14.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/users.x' 2023-06-22T08:24:14.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/od.1' 2023-06-22T08:24:14.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/ln.x' 2023-06-22T08:24:14.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/kill.x' 2023-06-22T08:24:14.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/date.x' 2023-06-22T08:24:14.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/chown.1' 2023-06-22T08:24:14.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/paste.x' 2023-06-22T08:24:14.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/expand.x' 2023-06-22T08:24:14.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/rm.x' 2023-06-22T08:24:14.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sha384sum.1' 2023-06-22T08:24:14.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sleep.1' 2023-06-22T08:24:14.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sha512sum.1' 2023-06-22T08:24:14.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/shuf.x' 2023-06-22T08:24:14.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/join.1' 2023-06-22T08:24:14.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/cp.x' 2023-06-22T08:24:14.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/mktemp.1' 2023-06-22T08:24:14.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tsort.x' 2023-06-22T08:24:14.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/factor.1' 2023-06-22T08:24:14.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/ptx.x' 2023-06-22T08:24:14.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/who.x' 2023-06-22T08:24:14.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/touch.x' 2023-06-22T08:24:14.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/logname.1' 2023-06-22T08:24:14.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/dd.1' 2023-06-22T08:24:14.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sleep.x' 2023-06-22T08:24:14.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/base64.1' 2023-06-22T08:24:14.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/vdir.x' 2023-06-22T08:24:14.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/printenv.x' 2023-06-22T08:24:14.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sha224sum.1' 2023-06-22T08:24:14.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/chcon.1' 2023-06-22T08:24:14.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/whoami.1' 2023-06-22T08:24:14.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/unlink.x' 2023-06-22T08:24:14.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/runcon.1' 2023-06-22T08:24:14.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tty.x' 2023-06-22T08:24:14.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/id.1' 2023-06-22T08:24:14.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/printf.x' 2023-06-22T08:24:14.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/rmdir.1' 2023-06-22T08:24:14.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/od.x' 2023-06-22T08:24:14.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/rmdir.x' 2023-06-22T08:24:14.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/pathchk.x' 2023-06-22T08:24:14.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sort.1' 2023-06-22T08:24:14.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/chown.x' 2023-06-22T08:24:14.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/df.x' 2023-06-22T08:24:14.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/mkfifo.x' 2023-06-22T08:24:14.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/test.x' 2023-06-22T08:24:14.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/vdir.1' 2023-06-22T08:24:14.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/mknod.x' 2023-06-22T08:24:14.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/who.1' 2023-06-22T08:24:14.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/arch.x' 2023-06-22T08:24:14.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/stty.x' 2023-06-22T08:24:14.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/pr.1' 2023-06-22T08:24:14.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/test.1' 2023-06-22T08:24:14.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/uniq.x' 2023-06-22T08:24:14.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/stat.x' 2023-06-22T08:24:14.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/paste.1' 2023-06-22T08:24:14.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/cksum.x' 2023-06-22T08:24:14.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/ptx.1' 2023-06-22T08:24:14.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/Makefile.in' 2023-06-22T08:24:14.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/dircolors.x' 2023-06-22T08:24:14.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/uptime.x' 2023-06-22T08:24:14.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/chgrp.x' 2023-06-22T08:24:14.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/basename.1' 2023-06-22T08:24:14.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/mktemp.x' 2023-06-22T08:24:14.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sum.1' 2023-06-22T08:24:14.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sha1sum.x' 2023-06-22T08:24:14.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/groups.x' 2023-06-22T08:24:14.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/mknod.1' 2023-06-22T08:24:14.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/help2man' 2023-06-22T08:24:14.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/uname.x' 2023-06-22T08:24:14.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/ls.1' 2023-06-22T08:24:14.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/du.1' 2023-06-22T08:24:14.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/timeout.x' 2023-06-22T08:24:14.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/groups.1' 2023-06-22T08:24:14.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/dir.1' 2023-06-22T08:24:14.482 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sync.x' 2023-06-22T08:24:14.482 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/cat.1' 2023-06-22T08:24:14.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/comm.x' 2023-06-22T08:24:14.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sha256sum.1' 2023-06-22T08:24:14.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/comm.1' 2023-06-22T08:24:14.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/chroot.1' 2023-06-22T08:24:14.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/csplit.x' 2023-06-22T08:24:14.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sync.1' 2023-06-22T08:24:14.483 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sort.x' 2023-06-22T08:24:14.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/chmod.x' 2023-06-22T08:24:14.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/Makefile.am' 2023-06-22T08:24:14.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/readlink.1' 2023-06-22T08:24:14.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/uptime.1' 2023-06-22T08:24:14.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/pwd.x' 2023-06-22T08:24:14.484 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tac.1' 2023-06-22T08:24:14.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/dir.x' 2023-06-22T08:24:14.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/mkdir.x' 2023-06-22T08:24:14.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/split.1' 2023-06-22T08:24:14.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/chroot.x' 2023-06-22T08:24:14.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/false.x' 2023-06-22T08:24:14.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/df.1' 2023-06-22T08:24:14.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tac.x' 2023-06-22T08:24:14.485 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/du.x' 2023-06-22T08:24:14.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/rm.1' 2023-06-22T08:24:14.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/head.x' 2023-06-22T08:24:14.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/mv.1' 2023-06-22T08:24:14.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sha384sum.x' 2023-06-22T08:24:14.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/echo.1' 2023-06-22T08:24:14.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/su.x' 2023-06-22T08:24:14.486 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/pwd.1' 2023-06-22T08:24:14.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/shred.1' 2023-06-22T08:24:14.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/fold.x' 2023-06-22T08:24:14.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/touch.1' 2023-06-22T08:24:14.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/printenv.1' 2023-06-22T08:24:14.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/uniq.1' 2023-06-22T08:24:14.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/id.x' 2023-06-22T08:24:14.487 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/chmod.1' 2023-06-22T08:24:14.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/link.x' 2023-06-22T08:24:14.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/dirname.x' 2023-06-22T08:24:14.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/nproc.1' 2023-06-22T08:24:14.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/nice.1' 2023-06-22T08:24:14.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/shred.x' 2023-06-22T08:24:14.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/fold.1' 2023-06-22T08:24:14.488 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sha224sum.x' 2023-06-22T08:24:14.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/pinky.1' 2023-06-22T08:24:14.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/runcon.x' 2023-06-22T08:24:14.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/unexpand.x' 2023-06-22T08:24:14.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/readlink.x' 2023-06-22T08:24:14.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/stdbuf.x' 2023-06-22T08:24:14.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/pr.x' 2023-06-22T08:24:14.489 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/whoami.x' 2023-06-22T08:24:14.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/nl.x' 2023-06-22T08:24:14.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/base64.x' 2023-06-22T08:24:14.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tsort.1' 2023-06-22T08:24:14.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tty.1' 2023-06-22T08:24:14.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/kill.1' 2023-06-22T08:24:14.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/shuf.1' 2023-06-22T08:24:14.490 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/csplit.1' 2023-06-22T08:24:14.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/users.1' 2023-06-22T08:24:14.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/basename.x' 2023-06-22T08:24:14.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/stat.1' 2023-06-22T08:24:14.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sha1sum.1' 2023-06-22T08:24:14.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/fmt.1' 2023-06-22T08:24:14.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/unexpand.1' 2023-06-22T08:24:14.491 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/mkfifo.1' 2023-06-22T08:24:14.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/truncate.1' 2023-06-22T08:24:14.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/wc.1' 2023-06-22T08:24:14.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/dd.x' 2023-06-22T08:24:14.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/cp.1' 2023-06-22T08:24:14.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/chcon.x' 2023-06-22T08:24:14.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/sha256sum.x' 2023-06-22T08:24:14.492 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/nproc.x' 2023-06-22T08:24:14.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/wc.x' 2023-06-22T08:24:14.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/yes.x' 2023-06-22T08:24:14.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/true.1' 2023-06-22T08:24:14.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/mkdir.1' 2023-06-22T08:24:14.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/factor.x' 2023-06-22T08:24:14.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/cksum.1' 2023-06-22T08:24:14.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/pathchk.1' 2023-06-22T08:24:14.493 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/nohup.x' 2023-06-22T08:24:14.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/echo.x' 2023-06-22T08:24:14.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tee.1' 2023-06-22T08:24:14.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/nohup.1' 2023-06-22T08:24:14.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/false.1' 2023-06-22T08:24:14.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/chgrp.1' 2023-06-22T08:24:14.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/cut.1' 2023-06-22T08:24:14.494 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/cut.x' 2023-06-22T08:24:14.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/fmt.x' 2023-06-22T08:24:14.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/stdbuf.1' 2023-06-22T08:24:14.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/nl.1' 2023-06-22T08:24:14.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/env.1' 2023-06-22T08:24:14.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/truncate.x' 2023-06-22T08:24:14.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/ls.x' 2023-06-22T08:24:14.495 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/ln.1' 2023-06-22T08:24:14.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/link.1' 2023-06-22T08:24:14.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/head.1' 2023-06-22T08:24:14.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/man/tr.x' 2023-06-22T08:24:14.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/man' 2023-06-22T08:24:14.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/TODO' 2023-06-22T08:24:14.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_prohibit_magic_number_exit' 2023-06-22T08:24:14.496 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/who.c' 2023-06-22T08:24:14.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/chown-core.h' 2023-06-22T08:24:14.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/expr.c' 2023-06-22T08:24:14.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/ls-ls.c' 2023-06-22T08:24:14.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/cp-hash.h' 2023-06-22T08:24:14.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/chgrp.c' 2023-06-22T08:24:14.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/ptx.c' 2023-06-22T08:24:14.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/nohup.c' 2023-06-22T08:24:14.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/ls.h' 2023-06-22T08:24:14.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/chroot.c' 2023-06-22T08:24:14.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/tail.c' 2023-06-22T08:24:14.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/nice.c' 2023-06-22T08:24:14.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/fold.c' 2023-06-22T08:24:14.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/comm.c' 2023-06-22T08:24:14.673 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/chown-core.c' 2023-06-22T08:24:14.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/tac.c' 2023-06-22T08:24:14.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/prog-fprintf.c' 2023-06-22T08:24:14.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/tr.c' 2023-06-22T08:24:14.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/tac-pipe.c' 2023-06-22T08:24:14.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/groups.c' 2023-06-22T08:24:14.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/nproc.c' 2023-06-22T08:24:14.675 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/operand2sig.c' 2023-06-22T08:24:14.675 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/libstdbuf.c' 2023-06-22T08:24:14.675 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/nl.c' 2023-06-22T08:24:14.675 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/env.c' 2023-06-22T08:24:14.675 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/dcgen' 2023-06-22T08:24:14.675 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/stat.c' 2023-06-22T08:24:14.675 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/base64.c' 2023-06-22T08:24:14.676 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/chown.c' 2023-06-22T08:24:14.676 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/od.c' 2023-06-22T08:24:14.676 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/mkdir.c' 2023-06-22T08:24:14.676 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/logname.c' 2023-06-22T08:24:14.676 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/extract-magic' 2023-06-22T08:24:14.676 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/wheel-gen.pl' 2023-06-22T08:24:14.676 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/chcon.c' 2023-06-22T08:24:14.677 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/printenv.c' 2023-06-22T08:24:14.677 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/copy.c' 2023-06-22T08:24:14.677 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/ls-dir.c' 2023-06-22T08:24:14.677 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/cut.c' 2023-06-22T08:24:14.677 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/uptime.c' 2023-06-22T08:24:14.677 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/hostname.c' 2023-06-22T08:24:14.677 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/dirname.c' 2023-06-22T08:24:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/sleep.c' 2023-06-22T08:24:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/stdbuf.c' 2023-06-22T08:24:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/true.c' 2023-06-22T08:24:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/df.c' 2023-06-22T08:24:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/rmdir.c' 2023-06-22T08:24:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/copy.h' 2023-06-22T08:24:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/group-list.c' 2023-06-22T08:24:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/unexpand.c' 2023-06-22T08:24:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/stty.c' 2023-06-22T08:24:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/chmod.c' 2023-06-22T08:24:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/fmt.c' 2023-06-22T08:24:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/csplit.c' 2023-06-22T08:24:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/expand.c' 2023-06-22T08:24:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/kill.c' 2023-06-22T08:24:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/factor.c' 2023-06-22T08:24:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/wheel-size.h' 2023-06-22T08:24:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/pinky.c' 2023-06-22T08:24:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/yes.c' 2023-06-22T08:24:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/remove.c' 2023-06-22T08:24:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/Makefile.in' 2023-06-22T08:24:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/uname-uname.c' 2023-06-22T08:24:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/md5sum.c' 2023-06-22T08:24:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/ln.c' 2023-06-22T08:24:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/su.c' 2023-06-22T08:24:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/cksum.c' 2023-06-22T08:24:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/dircolors.h' 2023-06-22T08:24:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/group-list.h' 2023-06-22T08:24:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/false.c' 2023-06-22T08:24:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/operand2sig.h' 2023-06-22T08:24:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/id.c' 2023-06-22T08:24:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/rm.c' 2023-06-22T08:24:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/shred.c' 2023-06-22T08:24:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/printf.c' 2023-06-22T08:24:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/cp.c' 2023-06-22T08:24:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/uname-arch.c' 2023-06-22T08:24:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/dircolors.hin' 2023-06-22T08:24:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/setuidgid.c' 2023-06-22T08:24:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/cp-hash.c' 2023-06-22T08:24:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/tee.c' 2023-06-22T08:24:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/Makefile.am' 2023-06-22T08:24:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/head.c' 2023-06-22T08:24:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/users.c' 2023-06-22T08:24:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/runcon.c' 2023-06-22T08:24:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/du.c' 2023-06-22T08:24:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/date.c' 2023-06-22T08:24:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/unlink.c' 2023-06-22T08:24:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/echo.c' 2023-06-22T08:24:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/pathchk.c' 2023-06-22T08:24:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/tsort.c' 2023-06-22T08:24:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/touch.c' 2023-06-22T08:24:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/c99-to-c89.diff' 2023-06-22T08:24:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/join.c' 2023-06-22T08:24:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/cat.c' 2023-06-22T08:24:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/sync.c' 2023-06-22T08:24:14.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/shuf.c' 2023-06-22T08:24:14.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/truncate.c' 2023-06-22T08:24:14.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/ls-vdir.c' 2023-06-22T08:24:14.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/test.c' 2023-06-22T08:24:14.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/remove.h' 2023-06-22T08:24:14.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/seq.c' 2023-06-22T08:24:14.686 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/readlink.c' 2023-06-22T08:24:14.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/uname.c' 2023-06-22T08:24:14.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/sum.c' 2023-06-22T08:24:14.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/paste.c' 2023-06-22T08:24:14.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/uname.h' 2023-06-22T08:24:14.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/tty.c' 2023-06-22T08:24:14.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/fs.h' 2023-06-22T08:24:14.687 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/basename.c' 2023-06-22T08:24:14.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/dd.c' 2023-06-22T08:24:14.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/split.c' 2023-06-22T08:24:14.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/whoami.c' 2023-06-22T08:24:14.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/timeout.c' 2023-06-22T08:24:14.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/getlimits.c' 2023-06-22T08:24:14.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/dircolors.c' 2023-06-22T08:24:14.688 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/link.c' 2023-06-22T08:24:14.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/sort.c' 2023-06-22T08:24:14.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/wc.c' 2023-06-22T08:24:14.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/prog-fprintf.h' 2023-06-22T08:24:14.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/uniq.c' 2023-06-22T08:24:14.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/pr.c' 2023-06-22T08:24:14.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/system.h' 2023-06-22T08:24:14.689 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/pwd.c' 2023-06-22T08:24:14.690 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/hostid.c' 2023-06-22T08:24:14.690 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/install.c' 2023-06-22T08:24:15.135 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/wheel.h' 2023-06-22T08:24:15.136 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/mkfifo.c' 2023-06-22T08:24:15.136 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/mv.c' 2023-06-22T08:24:15.136 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/lbracket.c' 2023-06-22T08:24:15.136 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/mknod.c' 2023-06-22T08:24:15.136 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/mktemp.c' 2023-06-22T08:24:15.136 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/src/ls.c' 2023-06-22T08:24:15.137 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/src' 2023-06-22T08:24:15.137 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ru.gmo' 2023-06-22T08:24:15.137 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/zh_TW.gmo' 2023-06-22T08:24:15.137 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ko.gmo' 2023-06-22T08:24:15.137 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ro.po' 2023-06-22T08:24:15.137 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/vi.gmo' 2023-06-22T08:24:15.137 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/vi.po' 2023-06-22T08:24:15.137 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/da.gmo' 2023-06-22T08:24:15.138 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/fi.gmo' 2023-06-22T08:24:15.138 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/en.po' 2023-06-22T08:24:15.138 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/sl.po' 2023-06-22T08:24:15.138 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/fr.gmo' 2023-06-22T08:24:15.138 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/it.gmo' 2023-06-22T08:24:15.138 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/en@quot.header' 2023-06-22T08:24:15.138 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/el.po' 2023-06-22T08:24:15.138 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/tr.po' 2023-06-22T08:24:15.139 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/it.po' 2023-06-22T08:24:15.139 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/stamp-po' 2023-06-22T08:24:15.139 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/sk.gmo' 2023-06-22T08:24:15.139 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/tr.gmo' 2023-06-22T08:24:15.139 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ga.gmo' 2023-06-22T08:24:15.139 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ja.po' 2023-06-22T08:24:15.139 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/pl.po' 2023-06-22T08:24:15.140 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/pt.po' 2023-06-22T08:24:15.140 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/de.po' 2023-06-22T08:24:15.140 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/sv.gmo' 2023-06-22T08:24:15.140 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/Makevars' 2023-06-22T08:24:15.140 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/et.po' 2023-06-22T08:24:15.140 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ChangeLog-2007' 2023-06-22T08:24:15.140 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/fr.po' 2023-06-22T08:24:15.140 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/pt_BR.po' 2023-06-22T08:24:15.141 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ga.po' 2023-06-22T08:24:15.141 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/cs.po' 2023-06-22T08:24:15.141 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/pl.gmo' 2023-06-22T08:24:15.141 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/af.gmo' 2023-06-22T08:24:15.141 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/lt.po' 2023-06-22T08:24:15.141 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/cs.gmo' 2023-06-22T08:24:15.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/en.gmo' 2023-06-22T08:24:15.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ca.gmo' 2023-06-22T08:24:15.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/sl.gmo' 2023-06-22T08:24:15.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/eu.gmo' 2023-06-22T08:24:15.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/gl.gmo' 2023-06-22T08:24:15.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/uk.gmo' 2023-06-22T08:24:15.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/bg.gmo' 2023-06-22T08:24:15.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ro.gmo' 2023-06-22T08:24:15.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/hu.gmo' 2023-06-22T08:24:15.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/nl.po' 2023-06-22T08:24:15.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/el.gmo' 2023-06-22T08:24:15.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/LINGUAS' 2023-06-22T08:24:15.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/de.gmo' 2023-06-22T08:24:15.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ms.po' 2023-06-22T08:24:15.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/remove-potcdate.sin' 2023-06-22T08:24:15.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/da.po' 2023-06-22T08:24:15.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/en@boldquot.header' 2023-06-22T08:24:15.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/be.po' 2023-06-22T08:24:15.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/Makefile.in.in' 2023-06-22T08:24:15.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ca.po' 2023-06-22T08:24:15.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/id.po' 2023-06-22T08:24:15.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/hu.po' 2023-06-22T08:24:15.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/zh_CN.po' 2023-06-22T08:24:15.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/nb.gmo' 2023-06-22T08:24:15.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/nb.po' 2023-06-22T08:24:15.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/coreutils.pot' 2023-06-22T08:24:15.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/bg.po' 2023-06-22T08:24:15.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/id.gmo' 2023-06-22T08:24:15.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/sv.po' 2023-06-22T08:24:15.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/af.po' 2023-06-22T08:24:15.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/Rules-quot' 2023-06-22T08:24:15.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/eu.po' 2023-06-22T08:24:15.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/es.gmo' 2023-06-22T08:24:15.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/quot.sed' 2023-06-22T08:24:15.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ru.po' 2023-06-22T08:24:15.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ms.gmo' 2023-06-22T08:24:15.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/zh_TW.po' 2023-06-22T08:24:15.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/nl.gmo' 2023-06-22T08:24:15.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/zh_CN.gmo' 2023-06-22T08:24:15.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/pt.gmo' 2023-06-22T08:24:15.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/sk.po' 2023-06-22T08:24:15.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/lg.po' 2023-06-22T08:24:15.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ko.po' 2023-06-22T08:24:15.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/pt_BR.gmo' 2023-06-22T08:24:15.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/fi.po' 2023-06-22T08:24:15.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/es.po' 2023-06-22T08:24:15.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/lt.gmo' 2023-06-22T08:24:15.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/be.gmo' 2023-06-22T08:24:15.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/ja.gmo' 2023-06-22T08:24:15.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/lg.gmo' 2023-06-22T08:24:15.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/gl.po' 2023-06-22T08:24:15.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/boldquot.sed' 2023-06-22T08:24:15.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/et.gmo' 2023-06-22T08:24:15.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/insert-header.sin' 2023-06-22T08:24:15.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/POTFILES.in' 2023-06-22T08:24:15.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/po/uk.po' 2023-06-22T08:24:15.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/po' 2023-06-22T08:24:15.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_prohibit_empty_lines_at_EOF' 2023-06-22T08:24:15.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/GNUmakefile' 2023-06-22T08:24:15.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_useless_cpp_parens' 2023-06-22T08:24:15.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/ChangeLog' 2023-06-22T08:24:15.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_require_config_h' 2023-06-22T08:24:15.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/test-lib.sh' 2023-06-22T08:24:15.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/require-perl' 2023-06-22T08:24:15.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/octal' 2023-06-22T08:24:15.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/umask-x' 2023-06-22T08:24:15.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/equal-x' 2023-06-22T08:24:15.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/thru-dangling' 2023-06-22T08:24:15.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/equals' 2023-06-22T08:24:15.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/no-x' 2023-06-22T08:24:15.703 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/inaccessible' 2023-06-22T08:24:15.703 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/usage' 2023-06-22T08:24:15.704 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/c-option' 2023-06-22T08:24:15.704 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/silent' 2023-06-22T08:24:15.704 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chmod/setgid' 2023-06-22T08:24:15.704 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/chmod' 2023-06-22T08:24:15.704 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rmdir/t-slash' 2023-06-22T08:24:15.704 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rmdir/fail-perm' 2023-06-22T08:24:15.704 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rmdir/ignore' 2023-06-22T08:24:15.705 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/rmdir' 2023-06-22T08:24:15.705 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chgrp/posix-H' 2023-06-22T08:24:15.705 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chgrp/default-no-deref' 2023-06-22T08:24:15.705 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chgrp/no-x' 2023-06-22T08:24:15.705 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chgrp/basic' 2023-06-22T08:24:15.705 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chgrp/deref' 2023-06-22T08:24:15.705 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chgrp/recurse' 2023-06-22T08:24:15.706 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/chgrp' 2023-06-22T08:24:15.706 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/envvar-check' 2023-06-22T08:24:15.706 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chown/basic' 2023-06-22T08:24:15.706 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chown/deref' 2023-06-22T08:24:15.706 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chown/separator' 2023-06-22T08:24:15.706 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/chown/preserve-root' 2023-06-22T08:24:15.706 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/chown' 2023-06-22T08:24:15.706 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/acl' 2023-06-22T08:24:15.707 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/i-2' 2023-06-22T08:24:15.707 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/dup-source' 2023-06-22T08:24:15.707 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/to-symlink' 2023-06-22T08:24:15.707 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/childproof' 2023-06-22T08:24:15.707 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/i-5' 2023-06-22T08:24:15.707 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/mv-n' 2023-06-22T08:24:15.707 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/part-hardlink' 2023-06-22T08:24:15.707 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/dir2dir' 2023-06-22T08:24:15.708 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/part-rename' 2023-06-22T08:24:15.708 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/into-self-3' 2023-06-22T08:24:15.708 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/hard-2' 2023-06-22T08:24:15.708 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/trailing-slash' 2023-06-22T08:24:15.708 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/hard-verbose' 2023-06-22T08:24:15.708 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/force' 2023-06-22T08:24:15.708 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/i-link-no' 2023-06-22T08:24:15.708 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/i-3' 2023-06-22T08:24:15.709 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/backup-dir' 2023-06-22T08:24:15.709 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/i-4' 2023-06-22T08:24:15.709 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/into-self' 2023-06-22T08:24:15.709 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/diag' 2023-06-22T08:24:15.709 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/into-self-2' 2023-06-22T08:24:15.709 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/mv-special-1' 2023-06-22T08:24:15.709 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/update' 2023-06-22T08:24:15.709 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/atomic2' 2023-06-22T08:24:15.710 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/no-target-dir' 2023-06-22T08:24:15.710 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/hard-4' 2023-06-22T08:24:15.710 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/sticky-to-xpart' 2023-06-22T08:24:15.710 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/dir-file' 2023-06-22T08:24:15.710 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/i-1' 2023-06-22T08:24:15.710 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/hard-3' 2023-06-22T08:24:15.710 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/part-symlink' 2023-06-22T08:24:15.710 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/hard-link-1' 2023-06-22T08:24:15.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/backup-is-src' 2023-06-22T08:24:15.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/perm-1' 2023-06-22T08:24:15.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/partition-perm' 2023-06-22T08:24:15.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/part-fail' 2023-06-22T08:24:15.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/leak-fd' 2023-06-22T08:24:15.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/atomic' 2023-06-22T08:24:15.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mv/into-self-4' 2023-06-22T08:24:15.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/mv' 2023-06-22T08:24:15.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/multihardlink' 2023-06-22T08:24:15.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/stat-failed' 2023-06-22T08:24:15.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/stat-free-symlinks' 2023-06-22T08:24:15.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/color-clear-to-eol' 2023-06-22T08:24:15.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/rt-1' 2023-06-22T08:24:15.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/color-dtype-dir' 2023-06-22T08:24:15.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/symlink-slash' 2023-06-22T08:24:15.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/infloop' 2023-06-22T08:24:15.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/follow-slink' 2023-06-22T08:24:15.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/no-arg' 2023-06-22T08:24:15.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/inode' 2023-06-22T08:24:15.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/color-norm' 2023-06-22T08:24:15.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/no-cap' 2023-06-22T08:24:15.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/x-option' 2023-06-22T08:24:15.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/proc-selinux-segfault' 2023-06-22T08:24:15.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/abmon-align' 2023-06-22T08:24:15.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/file-type' 2023-06-22T08:24:15.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/stat-vs-dirent' 2023-06-22T08:24:15.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/nameless-uid' 2023-06-22T08:24:15.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/dangle' 2023-06-22T08:24:15.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/readdir-mountpoint-inode' 2023-06-22T08:24:15.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/stat-dtype' 2023-06-22T08:24:15.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/dired' 2023-06-22T08:24:15.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/capability' 2023-06-22T08:24:15.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/m-option' 2023-06-22T08:24:15.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ls/recursive' 2023-06-22T08:24:15.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/ls' 2023-06-22T08:24:15.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/p-slashdot' 2023-06-22T08:24:15.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/t-slash' 2023-06-22T08:24:16.403 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/perm' 2023-06-22T08:24:16.403 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/p-thru-slink' 2023-06-22T08:24:16.403 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/p-3' 2023-06-22T08:24:16.403 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/p-2' 2023-06-22T08:24:16.403 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/writable-under-readonly' 2023-06-22T08:24:16.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/selinux' 2023-06-22T08:24:16.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/p-1' 2023-06-22T08:24:16.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/p-v' 2023-06-22T08:24:16.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/parents' 2023-06-22T08:24:16.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/mkdir/special-1' 2023-06-22T08:24:16.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/mkdir' 2023-06-22T08:24:16.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/install/create-leading' 2023-06-22T08:24:16.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/install/install-C-root' 2023-06-22T08:24:16.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/install/install-C' 2023-06-22T08:24:16.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/install/basic-1' 2023-06-22T08:24:16.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/install/install-C-selinux' 2023-06-22T08:24:16.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/install/d-slashdot' 2023-06-22T08:24:16.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/install/strip-program' 2023-06-22T08:24:16.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/install/trap' 2023-06-22T08:24:16.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/install' 2023-06-22T08:24:16.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/readlink/rl-1' 2023-06-22T08:24:16.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/readlink/can-m' 2023-06-22T08:24:16.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/readlink/can-f' 2023-06-22T08:24:16.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/readlink/can-e' 2023-06-22T08:24:16.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/readlink' 2023-06-22T08:24:16.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/2g' 2023-06-22T08:24:16.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/8gb' 2023-06-22T08:24:16.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/restore-wd' 2023-06-22T08:24:16.407 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/long-from-unreadable' 2023-06-22T08:24:16.407 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/two-args' 2023-06-22T08:24:16.407 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/slink' 2023-06-22T08:24:16.407 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/inaccessible-cwd' 2023-06-22T08:24:16.407 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/no-x' 2023-06-22T08:24:16.407 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/fd-leak' 2023-06-22T08:24:16.407 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/basic' 2023-06-22T08:24:16.407 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/one-file-system' 2023-06-22T08:24:16.408 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/trailing-slash' 2023-06-22T08:24:16.408 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/no-deref' 2023-06-22T08:24:16.408 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/long-sloop' 2023-06-22T08:24:16.408 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/inacc-dir' 2023-06-22T08:24:16.408 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/deref' 2023-06-22T08:24:16.408 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/inacc-dest' 2023-06-22T08:24:16.409 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/files0-from' 2023-06-22T08:24:16.409 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/slash' 2023-06-22T08:24:16.409 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/exclude' 2023-06-22T08:24:16.409 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/hard-link' 2023-06-22T08:24:16.409 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/du/deref-args' 2023-06-22T08:24:16.409 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/du' 2023-06-22T08:24:16.409 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/id/no-context' 2023-06-22T08:24:16.410 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/id' 2023-06-22T08:24:16.410 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/not-owner' 2023-06-22T08:24:16.410 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/fifo' 2023-06-22T08:24:16.410 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/obsolescent' 2023-06-22T08:24:16.410 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/dangling-symlink' 2023-06-22T08:24:16.410 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/trailing-slash' 2023-06-22T08:24:16.411 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/no-create-missing' 2023-06-22T08:24:16.411 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/no-rights' 2023-06-22T08:24:16.411 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/relative' 2023-06-22T08:24:16.411 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/no-dereference' 2023-06-22T08:24:16.411 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/empty-file' 2023-06-22T08:24:16.411 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/dir-1' 2023-06-22T08:24:16.411 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/60-seconds' 2023-06-22T08:24:16.411 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/now-owned-by-other' 2023-06-22T08:24:16.412 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/fail-diag' 2023-06-22T08:24:16.412 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/touch/read-only' 2023-06-22T08:24:16.412 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/touch' 2023-06-22T08:24:16.412 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/flush-initial' 2023-06-22T08:24:16.412 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/F-vs-rename' 2023-06-22T08:24:16.412 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/assert-2' 2023-06-22T08:24:16.412 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/inotify-rotate' 2023-06-22T08:24:16.412 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/proc-ksyms' 2023-06-22T08:24:16.413 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/infloop-1' 2023-06-22T08:24:16.413 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/start-middle' 2023-06-22T08:24:16.413 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/inotify-hash-abuse' 2023-06-22T08:24:16.413 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/pipe-f2' 2023-06-22T08:24:16.413 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/tail-n0f' 2023-06-22T08:24:16.413 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/append-only' 2023-06-22T08:24:16.413 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/pipe-f' 2023-06-22T08:24:16.413 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/wait' 2023-06-22T08:24:16.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/assert' 2023-06-22T08:24:16.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/inotify-race' 2023-06-22T08:24:16.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/follow-stdin' 2023-06-22T08:24:16.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/inotify-hash-abuse2' 2023-06-22T08:24:16.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/big-4gb' 2023-06-22T08:24:16.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/tail-2/pid' 2023-06-22T08:24:16.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/tail-2' 2023-06-22T08:24:16.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/lang-default' 2023-06-22T08:24:16.415 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/Makefile.in' 2023-06-22T08:24:16.415 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/Coreutils.pm' 2023-06-22T08:24:16.415 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/Makefile.am' 2023-06-22T08:24:16.605 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:24:16.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:24:16.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/hash' 2023-06-22T08:24:16.770 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/rm1' 2023-06-22T08:24:16.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/empty-inacc' 2023-06-22T08:24:16.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/r-3' 2023-06-22T08:24:16.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/one-file-system2' 2023-06-22T08:24:16.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/readdir-bug' 2023-06-22T08:24:16.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/r-1' 2023-06-22T08:24:16.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/f-1' 2023-06-22T08:24:16.771 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/i-no-r' 2023-06-22T08:24:16.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/ir-1' 2023-06-22T08:24:16.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/fail-eacces' 2023-06-22T08:24:16.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/i-never' 2023-06-22T08:24:16.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/r-4' 2023-06-22T08:24:16.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/dir-no-w' 2023-06-22T08:24:16.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/no-give-up' 2023-06-22T08:24:16.772 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/dangling-symlink' 2023-06-22T08:24:16.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/unreadable' 2023-06-22T08:24:16.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/dot-rel' 2023-06-22T08:24:16.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/ignorable' 2023-06-22T08:24:16.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/one-file-system' 2023-06-22T08:24:16.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/interactive-once' 2023-06-22T08:24:16.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/unread2' 2023-06-22T08:24:16.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/inaccessible' 2023-06-22T08:24:16.773 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/isatty' 2023-06-22T08:24:16.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/rm5' 2023-06-22T08:24:16.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/interactive-always' 2023-06-22T08:24:16.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/unread3' 2023-06-22T08:24:16.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/fail-eperm' 2023-06-22T08:24:16.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/dir-nonrecur' 2023-06-22T08:24:16.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/empty-name' 2023-06-22T08:24:16.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/rm4' 2023-06-22T08:24:16.774 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/cycle' 2023-06-22T08:24:16.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/v-slash' 2023-06-22T08:24:16.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/deep-1' 2023-06-22T08:24:16.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/rm2' 2023-06-22T08:24:16.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/rm3' 2023-06-22T08:24:16.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/sunos-1' 2023-06-22T08:24:16.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/deep-2' 2023-06-22T08:24:16.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/i-1' 2023-06-22T08:24:16.775 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/r-2' 2023-06-22T08:24:16.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/ext3-perf' 2023-06-22T08:24:16.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/fail-2eperm' 2023-06-22T08:24:16.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/rm/read-only' 2023-06-22T08:24:16.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/rm' 2023-06-22T08:24:16.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/check.mk' 2023-06-22T08:24:16.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/pr-tests' 2023-06-22T08:24:16.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/o3ml24f-bl-tn' 2023-06-22T08:24:16.776 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+2l24f-bl' 2023-06-22T08:24:16.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/b2l17-FF' 2023-06-22T08:24:16.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+3ml24f-tn-bl' 2023-06-22T08:24:16.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/l24f-t' 2023-06-22T08:24:16.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3b3l15f-t' 2023-06-22T08:24:16.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/4-7l24-FF' 2023-06-22T08:24:16.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+6a2l17-FF' 2023-06-22T08:24:16.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nl24f-bl' 2023-06-22T08:24:16.777 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nSml24-bl-FF' 2023-06-22T08:24:16.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/a3l24f-lm' 2023-06-22T08:24:16.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+3ml20f-bl-FF' 2023-06-22T08:24:16.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3b3f-0F' 2023-06-22T08:24:16.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/FFn' 2023-06-22T08:24:16.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W27l24f-ll' 2023-06-22T08:24:16.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tne8o3-t_tab' 2023-06-22T08:24:16.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/a2l17-FF' 2023-06-22T08:24:16.778 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/a2l15-FF' 2023-06-22T08:24:16.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ml24f-0F' 2023-06-22T08:24:16.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W72Jl24f-ll' 2023-06-22T08:24:16.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/o3b3Snl24f-tn' 2023-06-22T08:24:16.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nSml24-t-tFFFF' 2023-06-22T08:24:16.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W26l24f-ll' 2023-06-22T08:24:16.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3b3l15-t' 2023-06-22T08:24:16.779 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ml24-t-FF' 2023-06-22T08:24:16.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ttb3-0FF' 2023-06-22T08:24:16.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+4-8a2l17-FF' 2023-06-22T08:24:16.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tFFt-ll' 2023-06-22T08:24:16.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/b3-0F' 2023-06-22T08:24:16.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+3-7l24-FF' 2023-06-22T08:24:16.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tt-0FF' 2023-06-22T08:24:16.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/Ja3l24f-lm' 2023-06-22T08:24:16.780 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/w72l24f-ll' 2023-06-22T08:24:16.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W35b3l24f-lm' 2023-06-22T08:24:16.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/0FF' 2023-06-22T08:24:16.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/0Ft' 2023-06-22T08:24:16.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/2-S_f-t_notab' 2023-06-22T08:24:16.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/2sw60f-t_notab' 2023-06-22T08:24:16.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3b2l17-FF' 2023-06-22T08:24:16.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tFFt-lm' 2023-06-22T08:24:16.781 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3-FF' 2023-06-22T08:24:16.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/t-t' 2023-06-22T08:24:16.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/Jml24f-lm-lo' 2023-06-22T08:24:16.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nN1+3l24f-bl' 2023-06-22T08:24:16.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/0FFnt' 2023-06-22T08:24:16.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tta3-0FF' 2023-06-22T08:24:16.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3-0F' 2023-06-22T08:24:16.782 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+6b3l13f-FF' 2023-06-22T08:24:16.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3ml24-t' 2023-06-22T08:24:16.783 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/t-0FF' 2023-06-22T08:24:16.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/o3Jml24f-lm-lo' 2023-06-22T08:24:16.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tn_2e8-t_tab' 2023-06-22T08:24:16.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+8l20-FF' 2023-06-22T08:24:16.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W35Ja3l24f-lm' 2023-06-22T08:24:16.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ml24-t' 2023-06-22T08:24:16.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W35ml24f-lm-lo' 2023-06-22T08:24:16.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W-72l24f-ll' 2023-06-22T08:24:16.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/o3mSl24f-bl-tn' 2023-06-22T08:24:16.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/t-FF' 2023-06-22T08:24:16.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3ml24-t-FF' 2023-06-22T08:24:16.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3b3f-FF' 2023-06-22T08:24:16.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tta3-FF' 2023-06-22T08:24:16.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W35Jml24f-lmlo' 2023-06-22T08:24:16.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tFnFt' 2023-06-22T08:24:16.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/0FnFnt' 2023-06-22T08:24:16.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/0FnFt' 2023-06-22T08:24:16.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nSml24-t-t-FF' 2023-06-22T08:24:16.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/b3f-0FF' 2023-06-22T08:24:16.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tn' 2023-06-22T08:24:16.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tt-FF' 2023-06-22T08:24:16.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W28l24f-ll' 2023-06-22T08:24:16.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tb3-FF' 2023-06-22T08:24:16.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tn2e8-t_tab' 2023-06-22T08:24:16.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/l24-FF' 2023-06-22T08:24:16.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+4b2l17f-0FF' 2023-06-22T08:24:16.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3a3f-0F' 2023-06-22T08:24:16.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/FFtn' 2023-06-22T08:24:16.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/a3f-FF' 2023-06-22T08:24:16.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+2l24f-0FF' 2023-06-22T08:24:16.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/2s_f-t_notab' 2023-06-22T08:24:16.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/a3l15-t' 2023-06-22T08:24:16.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3b3f-0FF' 2023-06-22T08:24:16.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nJml24f-lmlolm' 2023-06-22T08:24:16.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ttb3-FF' 2023-06-22T08:24:16.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/loli' 2023-06-22T08:24:16.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/b3l15-t' 2023-06-22T08:24:16.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3-5l24f-t' 2023-06-22T08:24:16.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ta3-0FF' 2023-06-22T08:24:16.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/0F' 2023-06-22T08:24:16.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/FF' 2023-06-22T08:24:16.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ml24f-t' 2023-06-22T08:24:16.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/t_tab_' 2023-06-22T08:24:16.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/o3b3l24f-tn' 2023-06-22T08:24:16.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/4l24-FF' 2023-06-22T08:24:16.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/o3mSnl24fbltn' 2023-06-22T08:24:16.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tFFn' 2023-06-22T08:24:16.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+5-8b3l17f-FF' 2023-06-22T08:24:16.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nSml20-bl-FF' 2023-06-22T08:24:16.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/o3a3Sl24f-tn' 2023-06-22T08:24:16.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nJml24f-lmlmlo' 2023-06-22T08:24:16.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W35a3l24f-lm' 2023-06-22T08:24:16.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nN15l24f-bl' 2023-06-22T08:24:16.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/a3f-0F' 2023-06-22T08:24:16.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/b3f-0F' 2023-06-22T08:24:16.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/t_notab' 2023-06-22T08:24:16.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+5a3l13f-0FF' 2023-06-22T08:24:16.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nSml20-t-t-FF' 2023-06-22T08:24:16.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tn2e5o3-t_tab' 2023-06-22T08:24:16.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ml24f-lm-lo' 2023-06-22T08:24:16.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3f-FF' 2023-06-22T08:24:16.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/t-bl' 2023-06-22T08:24:16.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3ml24f-t' 2023-06-22T08:24:16.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/b3f-FF' 2023-06-22T08:24:16.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/FnFn' 2023-06-22T08:24:16.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+3ml24f-bl-tn' 2023-06-22T08:24:16.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/0FFt' 2023-06-22T08:24:16.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tb3-0FF' 2023-06-22T08:24:16.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tt-t' 2023-06-22T08:24:16.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/2-Sf-t_notab' 2023-06-22T08:24:16.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ta3-FF' 2023-06-22T08:24:16.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W20l24f-ll' 2023-06-22T08:24:16.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tn_2e8S-t_tab' 2023-06-22T08:24:16.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/b2l15-FF' 2023-06-22T08:24:16.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/l24-t' 2023-06-22T08:24:16.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3l24f-t' 2023-06-22T08:24:16.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tn2e8o3-t_tab' 2023-06-22T08:24:16.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/a3f-0FF' 2023-06-22T08:24:16.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/a3l15f-t' 2023-06-22T08:24:16.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/Jb3l24f-lm' 2023-06-22T08:24:16.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3a2l17-FF' 2023-06-22T08:24:16.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3f-0F' 2023-06-22T08:24:16.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/t_tab' 2023-06-22T08:24:16.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3l24-t' 2023-06-22T08:24:16.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/W35Jb3l24f-lm' 2023-06-22T08:24:16.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tFFt-bl' 2023-06-22T08:24:16.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tFFt' 2023-06-22T08:24:16.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+2-5l24f-0FF' 2023-06-22T08:24:16.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+3l24f-bl' 2023-06-22T08:24:16.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+7l24-FF' 2023-06-22T08:24:16.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3a3l15-t' 2023-06-22T08:24:16.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ml24f-t-0F' 2023-06-22T08:24:16.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ml24-FF' 2023-06-22T08:24:16.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/0Fnt' 2023-06-22T08:24:16.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/2f-t_notab' 2023-06-22T08:24:16.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/2s_w60f-t_nota' 2023-06-22T08:24:16.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/ml20-FF-t' 2023-06-22T08:24:16.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tt-bl' 2023-06-22T08:24:17.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3a3l15f-t' 2023-06-22T08:24:17.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/o3a3l24f-tn' 2023-06-22T08:24:17.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/o3b3Sl24f-tn' 2023-06-22T08:24:17.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/tne8-t_tab' 2023-06-22T08:24:17.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/a3-0F' 2023-06-22T08:24:17.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/nSml20-t-tFFFF' 2023-06-22T08:24:17.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/b3l24f-lm' 2023-06-22T08:24:17.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/2w60f-t_notab' 2023-06-22T08:24:17.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/2sf-t_notab' 2023-06-22T08:24:17.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/b3l15f-t' 2023-06-22T08:24:17.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/3ml24-FF' 2023-06-22T08:24:17.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/n+3l24f-0FF' 2023-06-22T08:24:17.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/pr/o3a3Snl24f-tn' 2023-06-22T08:24:17.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/pr' 2023-06-22T08:24:17.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/CuTmpdir.pm' 2023-06-22T08:24:17.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/df/unreadable' 2023-06-22T08:24:17.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/df/total-verify' 2023-06-22T08:24:17.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/df' 2023-06-22T08:24:17.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ln/sf-1' 2023-06-22T08:24:17.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ln/hard-backup' 2023-06-22T08:24:17.368 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ln/slash-decorated-nonexistent-dest' 2023-06-22T08:24:17.368 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ln/hard-to-sym' 2023-06-22T08:24:17.368 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ln/target-1' 2023-06-22T08:24:17.369 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ln/backup-1' 2023-06-22T08:24:17.369 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/ln/misc' 2023-06-22T08:24:17.369 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/ln' 2023-06-22T08:24:17.369 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/cp-mv-backup' 2023-06-22T08:24:17.370 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/acl' 2023-06-22T08:24:17.370 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/perm' 2023-06-22T08:24:17.370 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/cp-deref' 2023-06-22T08:24:17.370 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/special-bits' 2023-06-22T08:24:17.371 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/link' 2023-06-22T08:24:17.371 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/preserve-slink-time' 2023-06-22T08:24:17.371 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/symlink-slash' 2023-06-22T08:24:17.371 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/cp-mv-enotsup-xattr' 2023-06-22T08:24:17.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/thru-dangling' 2023-06-22T08:24:17.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/reflink-perm' 2023-06-22T08:24:17.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/r-vs-symlink' 2023-06-22T08:24:17.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/preserve-2' 2023-06-22T08:24:17.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/special-f' 2023-06-22T08:24:17.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/file-perm-race' 2023-06-22T08:24:17.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/slink-2-slink' 2023-06-22T08:24:17.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/proc-zero-len' 2023-06-22T08:24:17.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/reflink-auto' 2023-06-22T08:24:17.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/src-base-dot' 2023-06-22T08:24:17.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/link-preserve' 2023-06-22T08:24:17.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/cp-i' 2023-06-22T08:24:17.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/cp-a-selinux' 2023-06-22T08:24:17.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/preserve-gid' 2023-06-22T08:24:17.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/no-deref-link3' 2023-06-22T08:24:17.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/backup-dir' 2023-06-22T08:24:17.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/into-self' 2023-06-22T08:24:17.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/same-file' 2023-06-22T08:24:17.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/cp-HL' 2023-06-22T08:24:17.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/link-no-deref' 2023-06-22T08:24:17.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/parent-perm-race' 2023-06-22T08:24:17.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/proc-short-read' 2023-06-22T08:24:17.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/no-deref-link1' 2023-06-22T08:24:17.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/parent-perm' 2023-06-22T08:24:17.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/dir-vs-file' 2023-06-22T08:24:17.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/sparse' 2023-06-22T08:24:17.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/link-heap' 2023-06-22T08:24:17.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/no-deref-link2' 2023-06-22T08:24:17.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/dir-rm-dest' 2023-06-22T08:24:17.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/dir-slash' 2023-06-22T08:24:17.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/fail-perm' 2023-06-22T08:24:17.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/deref-slink' 2023-06-22T08:24:17.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/existing-perm-race' 2023-06-22T08:24:17.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/abuse' 2023-06-22T08:24:17.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/cp-parents' 2023-06-22T08:24:17.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/backup-1' 2023-06-22T08:24:17.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/backup-is-src' 2023-06-22T08:24:17.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/cp/capability' 2023-06-22T08:24:17.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/cp' 2023-06-22T08:24:17.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/not-rewound' 2023-06-22T08:24:17.383 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/skip-seek2' 2023-06-22T08:24:17.383 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/unblock' 2023-06-22T08:24:17.383 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/skip-seek-past-dev' 2023-06-22T08:24:17.383 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/direct' 2023-06-22T08:24:17.384 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/unblock-sync' 2023-06-22T08:24:17.384 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/skip-seek-past-file' 2023-06-22T08:24:17.384 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/stderr' 2023-06-22T08:24:17.384 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/reblock' 2023-06-22T08:24:17.385 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/misc' 2023-06-22T08:24:17.385 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/dd/skip-seek' 2023-06-22T08:24:17.385 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/dd' 2023-06-22T08:24:17.385 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/fold' 2023-06-22T08:24:17.386 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/tty-eof' 2023-06-22T08:24:17.386 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sort-rand' 2023-06-22T08:24:17.386 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/seq-long-double' 2023-06-22T08:24:17.386 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/shred-remove' 2023-06-22T08:24:17.750 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/nice' 2023-06-22T08:24:17.750 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/tee' 2023-06-22T08:24:17.750 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/truncate-owned-by-other' 2023-06-22T08:24:17.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/mknod' 2023-06-22T08:24:17.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/expand' 2023-06-22T08:24:17.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/timeout-parameters' 2023-06-22T08:24:17.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sum' 2023-06-22T08:24:17.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/readlink-fp-loop' 2023-06-22T08:24:17.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/printenv' 2023-06-22T08:24:17.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/nohup' 2023-06-22T08:24:17.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/printf-hex' 2023-06-22T08:24:17.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/ptx' 2023-06-22T08:24:17.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/wc-files0' 2023-06-22T08:24:17.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/tac-continue' 2023-06-22T08:24:17.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sort-merge-fdlimit' 2023-06-22T08:24:17.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/truncate-dir-fail' 2023-06-22T08:24:17.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/env-null' 2023-06-22T08:24:17.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/xstrtol' 2023-06-22T08:24:17.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/tr' 2023-06-22T08:24:17.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/paste' 2023-06-22T08:24:17.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/printf' 2023-06-22T08:24:17.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/comm' 2023-06-22T08:24:17.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/id-context' 2023-06-22T08:24:17.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/chcon' 2023-06-22T08:24:17.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/shred-passes' 2023-06-22T08:24:17.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/stat-fmt' 2023-06-22T08:24:17.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/truncate-parameters' 2023-06-22T08:24:17.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/date-sec' 2023-06-22T08:24:17.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/tsort' 2023-06-22T08:24:17.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sha1sum-vec' 2023-06-22T08:24:17.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/stat-hyphen' 2023-06-22T08:24:17.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/od-multiple-t' 2023-06-22T08:24:17.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/xattr' 2023-06-22T08:24:17.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/od' 2023-06-22T08:24:17.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sum-sysv' 2023-06-22T08:24:17.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/date-next-dow' 2023-06-22T08:24:17.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/nproc-avail' 2023-06-22T08:24:17.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/split-a' 2023-06-22T08:24:17.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/unexpand' 2023-06-22T08:24:17.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/pathchk1' 2023-06-22T08:24:17.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/md5sum' 2023-06-22T08:24:17.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/groups-version' 2023-06-22T08:24:17.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/head' 2023-06-22T08:24:17.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/stty-invalid' 2023-06-22T08:24:17.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/factor' 2023-06-22T08:24:17.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/timeout' 2023-06-22T08:24:17.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/su-fail' 2023-06-22T08:24:17.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/wc-parallel' 2023-06-22T08:24:17.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/md5sum-newline' 2023-06-22T08:24:17.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/tee-dash' 2023-06-22T08:24:17.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/runcon-no-reorder' 2023-06-22T08:24:17.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/dircolors' 2023-06-22T08:24:17.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/basename' 2023-06-22T08:24:17.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/selinux' 2023-06-22T08:24:17.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/head-elide-tail' 2023-06-22T08:24:17.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/help-version' 2023-06-22T08:24:17.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/stdbuf' 2023-06-22T08:24:17.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/expr' 2023-06-22T08:24:17.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/groups-dash' 2023-06-22T08:24:17.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/shuf' 2023-06-22T08:24:17.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sort-version' 2023-06-22T08:24:17.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/pwd-long' 2023-06-22T08:24:17.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/env' 2023-06-22T08:24:17.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sort-continue' 2023-06-22T08:24:17.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/od-x8' 2023-06-22T08:24:17.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/arch' 2023-06-22T08:24:17.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/chcon-fail' 2023-06-22T08:24:17.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/tac' 2023-06-22T08:24:17.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/chroot-fail' 2023-06-22T08:24:17.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/cat-proc' 2023-06-22T08:24:17.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/truncate-fail-diag' 2023-06-22T08:24:17.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/ptx-overrun' 2023-06-22T08:24:17.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sort-month' 2023-06-22T08:24:17.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/date' 2023-06-22T08:24:17.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/ls-time' 2023-06-22T08:24:17.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/stat-printf' 2023-06-22T08:24:17.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/printf-surprise' 2023-06-22T08:24:17.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/pwd-option' 2023-06-22T08:24:17.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/id-groups' 2023-06-22T08:24:17.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/stat-slash' 2023-06-22T08:24:17.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/ls-misc' 2023-06-22T08:24:17.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/fmt-long-line' 2023-06-22T08:24:17.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/split-l' 2023-06-22T08:24:17.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/uniq' 2023-06-22T08:24:17.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/truncate-relative' 2023-06-22T08:24:17.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/mktemp' 2023-06-22T08:24:17.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/nproc-positive' 2023-06-22T08:24:17.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/truncate-dangling-symlink' 2023-06-22T08:24:17.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/dirname' 2023-06-22T08:24:17.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/wc' 2023-06-22T08:24:17.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/pr' 2023-06-22T08:24:18.614 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sort' 2023-06-22T08:24:18.614 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/chroot-credentials' 2023-06-22T08:24:18.614 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/base64' 2023-06-22T08:24:18.615 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/test-diag' 2023-06-22T08:24:18.615 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/od-N' 2023-06-22T08:24:18.615 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/invalid-opt' 2023-06-22T08:24:18.615 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sha224sum' 2023-06-22T08:24:18.616 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/split-fail' 2023-06-22T08:24:18.616 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/tail' 2023-06-22T08:24:18.616 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/shred-exact' 2023-06-22T08:24:18.617 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/df' 2023-06-22T08:24:18.617 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/seq' 2023-06-22T08:24:18.617 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/truncate-no-create-missing' 2023-06-22T08:24:18.617 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sha256sum' 2023-06-22T08:24:18.618 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sha1sum' 2023-06-22T08:24:18.618 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/nl' 2023-06-22T08:24:18.618 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/fmt' 2023-06-22T08:24:18.618 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/stty-row-col' 2023-06-22T08:24:18.618 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/close-stdout' 2023-06-22T08:24:18.619 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/nice-fail' 2023-06-22T08:24:18.619 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/truncate-overflow' 2023-06-22T08:24:18.619 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sort-compress' 2023-06-22T08:24:18.619 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/false-status' 2023-06-22T08:24:18.619 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/test' 2023-06-22T08:24:18.619 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sort-files0-from' 2023-06-22T08:24:18.619 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/df-P' 2023-06-22T08:24:18.620 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/md5sum-parallel' 2023-06-22T08:24:18.620 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/cut' 2023-06-22T08:24:18.620 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sort-merge' 2023-06-22T08:24:18.620 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/csplit' 2023-06-22T08:24:18.620 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/wc-files0-from' 2023-06-22T08:24:18.620 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/truncate-fifo' 2023-06-22T08:24:18.620 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sha512sum' 2023-06-22T08:24:18.621 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/head-c' 2023-06-22T08:24:18.621 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/join' 2023-06-22T08:24:18.621 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/sha384sum' 2023-06-22T08:24:18.621 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/cat-buf' 2023-06-22T08:24:18.621 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/printf-cov' 2023-06-22T08:24:18.621 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/head-pos' 2023-06-22T08:24:18.622 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/stty' 2023-06-22T08:24:18.622 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/misc/pwd-unreadable-parent' 2023-06-22T08:24:18.622 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests/misc' 2023-06-22T08:24:18.622 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/other-fs-tmpdir' 2023-06-22T08:24:18.622 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/tests/sample-test' 2023-06-22T08:24:18.622 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/tests' 2023-06-22T08:24:18.622 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/getdate.texi' 2023-06-22T08:24:18.622 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/perm.texi' 2023-06-22T08:24:18.623 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/coreutils.info' 2023-06-22T08:24:18.623 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/stamp-vti' 2023-06-22T08:24:18.623 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/ChangeLog-2007' 2023-06-22T08:24:18.623 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/Makefile.in' 2023-06-22T08:24:18.623 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/constants.texi' 2023-06-22T08:24:18.623 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/Makefile.am' 2023-06-22T08:24:18.623 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/fdl.texi' 2023-06-22T08:24:18.623 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/coreutils.texi' 2023-06-22T08:24:18.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/doc/version.texi' 2023-06-22T08:24:18.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/doc' 2023-06-22T08:24:18.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_prohibit_tab_based_indentation' 2023-06-22T08:24:18.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gl/modules/getloadavg.diff' 2023-06-22T08:24:18.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/gl/modules' 2023-06-22T08:24:18.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/gl' 2023-06-22T08:24:18.624 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_prohibit_stat_st_blocks' 2023-06-22T08:24:18.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/THANKS' 2023-06-22T08:24:18.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/hash.h' 2023-06-22T08:24:18.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fts.c' 2023-06-22T08:24:18.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/full-read.c' 2023-06-22T08:24:18.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/error.h' 2023-06-22T08:24:18.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/idcache.h' 2023-06-22T08:24:18.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/faccessat.c' 2023-06-22T08:24:18.625 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fcntl.in.h' 2023-06-22T08:24:18.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/lchown.c' 2023-06-22T08:24:18.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sys_stat.in.h' 2023-06-22T08:24:18.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/openat-safer.c' 2023-06-22T08:24:18.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrtol.c' 2023-06-22T08:24:18.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/verror.c' 2023-06-22T08:24:18.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/root-dev-ino.c' 2023-06-22T08:24:18.626 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrtod.c' 2023-06-22T08:24:18.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/printf-args.h' 2023-06-22T08:24:18.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/vasprintf.c' 2023-06-22T08:24:18.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/randint.c' 2023-06-22T08:24:18.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/progname.c' 2023-06-22T08:24:18.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/progname.h' 2023-06-22T08:24:18.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/se-context.in.h' 2023-06-22T08:24:18.627 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open.c' 2023-06-22T08:24:18.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/file-has-acl.c' 2023-06-22T08:24:18.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/trim.h' 2023-06-22T08:24:18.628 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbswidth.h' 2023-06-22T08:24:18.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/isnan.c' 2023-06-22T08:24:18.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/euidaccess.c' 2023-06-22T08:24:18.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/physmem.h' 2023-06-22T08:24:18.629 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/areadlink.c' 2023-06-22T08:24:18.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbuiter.h' 2023-06-22T08:24:18.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xmemxfrm.h' 2023-06-22T08:24:18.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fd-reopen.c' 2023-06-22T08:24:18.630 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/rand-isaac.h' 2023-06-22T08:24:18.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/savewd.c' 2023-06-22T08:24:18.631 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xfts.c' 2023-06-22T08:24:18.977 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getdate.c' 2023-06-22T08:24:18.977 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/diacrit.h' 2023-06-22T08:24:18.978 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xmemcoll.h' 2023-06-22T08:24:18.978 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/isnanf.c' 2023-06-22T08:24:18.978 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/savewd.h' 2023-06-22T08:24:18.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/gethostname.c' 2023-06-22T08:24:18.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/alloca.in.h' 2023-06-22T08:24:18.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getdelim.c' 2023-06-22T08:24:18.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mpsort.c' 2023-06-22T08:24:18.980 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/config.hin' 2023-06-22T08:24:18.980 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/close-stream.h' 2023-06-22T08:24:18.980 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/savedir.h' 2023-06-22T08:24:18.981 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/hash-triple.h' 2023-06-22T08:24:18.981 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/trim.c' 2023-06-22T08:24:18.981 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/physmem.c' 2023-06-22T08:24:18.981 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fchdir.c' 2023-06-22T08:24:18.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdarg.in.h' 2023-06-22T08:24:18.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strintcmp.c' 2023-06-22T08:24:18.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stpncpy.c' 2023-06-22T08:24:18.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xalloc.h' 2023-06-22T08:24:18.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/timespec.h' 2023-06-22T08:24:18.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/readtokens.h' 2023-06-22T08:24:18.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fsusage.h' 2023-06-22T08:24:18.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/linebuffer.c' 2023-06-22T08:24:18.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memcoll.h' 2023-06-22T08:24:18.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/string.in.h' 2023-06-22T08:24:18.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/quote.h' 2023-06-22T08:24:18.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strstr.c' 2023-06-22T08:24:18.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/realloc.c' 2023-06-22T08:24:18.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdlib-safer.h' 2023-06-22T08:24:18.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/userspec.c' 2023-06-22T08:24:18.986 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/siglist.h' 2023-06-22T08:24:18.987 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/safe-read.h' 2023-06-22T08:24:18.987 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/filenamecat.c' 2023-06-22T08:24:18.987 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbrtowc.c' 2023-06-22T08:24:18.988 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getndelim2.h' 2023-06-22T08:24:18.988 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/argv-iter.h' 2023-06-22T08:24:18.988 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/openat-proc.c' 2023-06-22T08:24:18.988 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdio-safer.h' 2023-06-22T08:24:18.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open-osf.gperf' 2023-06-22T08:24:18.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getugroups.h' 2023-06-22T08:24:18.989 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdio.in.h' 2023-06-22T08:24:18.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memcmp.c' 2023-06-22T08:24:18.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/tempname.c' 2023-06-22T08:24:18.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/inttostr.h' 2023-06-22T08:24:18.990 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/yesno.c' 2023-06-22T08:24:18.991 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/full-write.c' 2023-06-22T08:24:18.991 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strnumcmp.c' 2023-06-22T08:24:18.991 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sys_socket.in.h' 2023-06-22T08:24:18.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/arpa_inet.in.h' 2023-06-22T08:24:18.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unlocked-io.h' 2023-06-22T08:24:18.992 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unitypes.h' 2023-06-22T08:24:18.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/canon-host.h' 2023-06-22T08:24:18.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/euidaccess-stat.c' 2023-06-22T08:24:18.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/isapipe.c' 2023-06-22T08:24:18.993 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/quotearg.h' 2023-06-22T08:24:18.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/ref-add.sin' 2023-06-22T08:24:18.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstriconv.h' 2023-06-22T08:24:18.994 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fprintftime.h' 2023-06-22T08:24:18.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/c-ctype.c' 2023-06-22T08:24:18.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/float.in.h' 2023-06-22T08:24:18.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/cloexec.c' 2023-06-22T08:24:18.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/offtostr.c' 2023-06-22T08:24:18.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open-aix.gperf' 2023-06-22T08:24:18.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/wcwidth.c' 2023-06-22T08:24:18.995 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/nproc.c' 2023-06-22T08:24:18.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/gai_strerror.c' 2023-06-22T08:24:18.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fcntl.c' 2023-06-22T08:24:18.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getndelim2.c' 2023-06-22T08:24:18.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/w32sock.h' 2023-06-22T08:24:18.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/printf-frexpl.h' 2023-06-22T08:24:18.996 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getopt.in.h' 2023-06-22T08:24:18.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/asprintf.c' 2023-06-22T08:24:18.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/filemode.c' 2023-06-22T08:24:18.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strndup.c' 2023-06-22T08:24:18.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/uniwidth/cjk.h' 2023-06-22T08:24:18.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/uniwidth/width.c' 2023-06-22T08:24:18.997 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/lib/uniwidth' 2023-06-22T08:24:18.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fseterr.h' 2023-06-22T08:24:18.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/pipe-safer.c' 2023-06-22T08:24:18.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/freading.c' 2023-06-22T08:24:18.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/diacrit.c' 2023-06-22T08:24:18.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/quotearg.c' 2023-06-22T08:24:18.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/randint.h' 2023-06-22T08:24:18.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/inttypes.in.h' 2023-06-22T08:24:18.998 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/regex_internal.h' 2023-06-22T08:24:18.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dirent--.h' 2023-06-22T08:24:18.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdio-write.c' 2023-06-22T08:24:18.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/hash.c' 2023-06-22T08:24:18.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/atexit.c' 2023-06-22T08:24:18.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stat.c' 2023-06-22T08:24:18.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open-aix.h' 2023-06-22T08:24:18.999 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/posixtm.c' 2023-06-22T08:24:19.000 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/obstack.c' 2023-06-22T08:24:19.000 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/readutmp.h' 2023-06-22T08:24:19.000 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sig2str.h' 2023-06-22T08:24:19.000 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/str-two-way.h' 2023-06-22T08:24:19.000 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/copy-acl.c' 2023-06-22T08:24:19.000 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/canonicalize.c' 2023-06-22T08:24:19.001 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/userspec.h' 2023-06-22T08:24:19.001 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbsalign.h' 2023-06-22T08:24:19.001 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/wchar.in.h' 2023-06-22T08:24:19.001 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/filevercmp.h' 2023-06-22T08:24:19.305 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/regex_internal.c' 2023-06-22T08:24:19.305 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memcpy.c' 2023-06-22T08:24:19.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/base64.c' 2023-06-22T08:24:19.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/exclude.h' 2023-06-22T08:24:19.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/chown.c' 2023-06-22T08:24:19.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/alignof.h' 2023-06-22T08:24:19.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/signbitd.c' 2023-06-22T08:24:19.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/at-func.c' 2023-06-22T08:24:19.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/asnprintf.c' 2023-06-22T08:24:19.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sig2str.c' 2023-06-22T08:24:19.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mkdir.c' 2023-06-22T08:24:19.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/savedir.c' 2023-06-22T08:24:19.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strtoull.c' 2023-06-22T08:24:19.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/cycle-check.c' 2023-06-22T08:24:19.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/version-etc.h' 2023-06-22T08:24:19.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mkancesdirs.c' 2023-06-22T08:24:19.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unistr/u8-uctomb-aux.c' 2023-06-22T08:24:19.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unistr/u8-uctomb.c' 2023-06-22T08:24:19.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/lib/unistr' 2023-06-22T08:24:19.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/priv-set.h' 2023-06-22T08:24:19.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/time_r.c' 2023-06-22T08:24:19.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/at-func2.c' 2023-06-22T08:24:19.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open-solaris.h' 2023-06-22T08:24:19.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/freopen-safer.c' 2023-06-22T08:24:19.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xnanosleep.h' 2023-06-22T08:24:19.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/verify.h' 2023-06-22T08:24:19.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/vasnprintf.c' 2023-06-22T08:24:19.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getgroups.c' 2023-06-22T08:24:19.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/errno.in.h' 2023-06-22T08:24:19.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/long-options.h' 2023-06-22T08:24:19.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/posixver.c' 2023-06-22T08:24:19.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/netdb.in.h' 2023-06-22T08:24:19.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdlib--.h' 2023-06-22T08:24:19.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/same.h' 2023-06-22T08:24:19.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/gethrxtime.h' 2023-06-22T08:24:19.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/version-etc-fsf.c' 2023-06-22T08:24:19.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/printf-frexp.h' 2023-06-22T08:24:19.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/linebuffer.h' 2023-06-22T08:24:19.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xnanosleep.c' 2023-06-22T08:24:19.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/openat.c' 2023-06-22T08:24:19.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dup2.c' 2023-06-22T08:24:19.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/obstack.h' 2023-06-22T08:24:19.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/safe-read.c' 2023-06-22T08:24:19.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memchr2.valgrind' 2023-06-22T08:24:19.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/putenv.c' 2023-06-22T08:24:19.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/ctype.in.h' 2023-06-22T08:24:19.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unlinkat.c' 2023-06-22T08:24:19.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/gettime.c' 2023-06-22T08:24:19.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sha1.h' 2023-06-22T08:24:19.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/yesno.h' 2023-06-22T08:24:19.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/regexec.c' 2023-06-22T08:24:19.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/closeout.c' 2023-06-22T08:24:19.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/str-kmp.h' 2023-06-22T08:24:19.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getpass.h' 2023-06-22T08:24:19.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/cloexec.h' 2023-06-22T08:24:19.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open-hpux.h' 2023-06-22T08:24:19.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/close-hook.c' 2023-06-22T08:24:19.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fchmodat.c' 2023-06-22T08:24:19.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sig-handler.h' 2023-06-22T08:24:19.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/glthread/tls.h' 2023-06-22T08:24:19.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/glthread/tls.c' 2023-06-22T08:24:19.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/glthread/threadlib.c' 2023-06-22T08:24:19.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/glthread/lock.c' 2023-06-22T08:24:19.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/glthread/lock.h' 2023-06-22T08:24:19.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/lib/glthread' 2023-06-22T08:24:19.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getopt.c' 2023-06-22T08:24:19.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/readtokens0.c' 2023-06-22T08:24:19.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/gnulib.mk' 2023-06-22T08:24:19.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strtoimax.c' 2023-06-22T08:24:19.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/se-selinux.in.h' 2023-06-22T08:24:19.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/argv-iter.c' 2023-06-22T08:24:19.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/canonicalize.h' 2023-06-22T08:24:19.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fpending.c' 2023-06-22T08:24:19.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/openat-priv.h' 2023-06-22T08:24:19.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/closeout.h' 2023-06-22T08:24:19.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unistd--.h' 2023-06-22T08:24:19.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/close.c' 2023-06-22T08:24:19.316 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memchr.valgrind' 2023-06-22T08:24:19.316 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/ignore-value.h' 2023-06-22T08:24:19.316 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getcwd.c' 2023-06-22T08:24:19.316 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mkancesdirs.h' 2023-06-22T08:24:19.316 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbrlen.c' 2023-06-22T08:24:19.316 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/randread.c' 2023-06-22T08:24:19.316 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dirname.c' 2023-06-22T08:24:19.317 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/chdir-long.h' 2023-06-22T08:24:19.317 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/areadlink.h' 2023-06-22T08:24:19.317 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stat-time.h' 2023-06-22T08:24:19.317 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/save-cwd.h' 2023-06-22T08:24:19.317 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbsstr.c' 2023-06-22T08:24:19.317 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/open-safer.c' 2023-06-22T08:24:19.317 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mkdirat.c' 2023-06-22T08:24:19.318 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xgethostname.h' 2023-06-22T08:24:19.318 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/langinfo.in.h' 2023-06-22T08:24:19.318 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/rmdir.c' 2023-06-22T08:24:19.318 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fopen.c' 2023-06-22T08:24:19.318 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/close-stream.c' 2023-06-22T08:24:19.318 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unicodeio.c' 2023-06-22T08:24:19.318 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open-hpux.gperf' 2023-06-22T08:24:19.319 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/wctype.in.h' 2023-06-22T08:24:19.319 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbsinit.c' 2023-06-22T08:24:19.319 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/streq.h' 2023-06-22T08:24:19.319 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/open.c' 2023-06-22T08:24:19.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/freopen.c' 2023-06-22T08:24:19.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/posixtm.h' 2023-06-22T08:24:19.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dirfd.c' 2023-06-22T08:24:19.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fcntl-safer.h' 2023-06-22T08:24:19.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/isapipe.h' 2023-06-22T08:24:19.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrtold.c' 2023-06-22T08:24:19.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mgetgroups.h' 2023-06-22T08:24:19.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/quote.c' 2023-06-22T08:24:19.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strcspn.c' 2023-06-22T08:24:19.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/ChangeLog-2007' 2023-06-22T08:24:19.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mpsort.h' 2023-06-22T08:24:19.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/inttostr.c' 2023-06-22T08:24:19.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stpcpy.c' 2023-06-22T08:24:19.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/umaxtostr.c' 2023-06-22T08:24:19.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/printf-frexp.c' 2023-06-22T08:24:19.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/buffer-lcm.h' 2023-06-22T08:24:19.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/regcomp.c' 2023-06-22T08:24:19.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/openat-die.c' 2023-06-22T08:24:19.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/tempname.h' 2023-06-22T08:24:19.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getdtablesize.c' 2023-06-22T08:24:19.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strsignal.c' 2023-06-22T08:24:19.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getfilecon.c' 2023-06-22T08:24:19.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/utimens.h' 2023-06-22T08:24:19.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/snprintf.c' 2023-06-22T08:24:19.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fts-cycle.c' 2023-06-22T08:24:19.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fpending.h' 2023-06-22T08:24:19.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/group-member.h' 2023-06-22T08:24:19.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrtod.h' 2023-06-22T08:24:19.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getusershell.c' 2023-06-22T08:24:19.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/ref-del.sin' 2023-06-22T08:24:19.802 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/frexp.c' 2023-06-22T08:24:19.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getpagesize.c' 2023-06-22T08:24:19.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dirname.h' 2023-06-22T08:24:19.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sockets.c' 2023-06-22T08:24:19.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getopt_int.h' 2023-06-22T08:24:19.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/freadahead.h' 2023-06-22T08:24:19.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strftime.h' 2023-06-22T08:24:19.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/close-hook.h' 2023-06-22T08:24:19.803 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/raise.c' 2023-06-22T08:24:19.804 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fpucw.h' 2023-06-22T08:24:19.804 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/localcharset.c' 2023-06-22T08:24:19.804 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/lstat.c' 2023-06-22T08:24:19.804 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/rename.c' 2023-06-22T08:24:19.805 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/freadahead.c' 2023-06-22T08:24:19.805 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/argmatch.h' 2023-06-22T08:24:19.805 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fchown-stub.c' 2023-06-22T08:24:19.806 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/creat-safer.c' 2023-06-22T08:24:19.806 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strtoul.c' 2023-06-22T08:24:19.806 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xprintf.c' 2023-06-22T08:24:19.806 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mountlist.c' 2023-06-22T08:24:19.806 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mkdir-p.c' 2023-06-22T08:24:19.806 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/utimecmp.c' 2023-06-22T08:24:19.807 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mountlist.h' 2023-06-22T08:24:19.807 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/isnanl.c' 2023-06-22T08:24:19.807 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/randperm.h' 2023-06-22T08:24:19.807 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/remove.c' 2023-06-22T08:24:19.809 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dirent.in.h' 2023-06-22T08:24:19.809 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/malloc.c' 2023-06-22T08:24:19.809 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memrchr.c' 2023-06-22T08:24:19.810 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dirchownmod.h' 2023-06-22T08:24:19.810 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/argmatch.c' 2023-06-22T08:24:19.810 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/Makefile.in' 2023-06-22T08:24:19.810 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strnumcmp.h' 2023-06-22T08:24:19.810 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/c-strtold.c' 2023-06-22T08:24:19.811 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/freadptr.c' 2023-06-22T08:24:19.811 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/version-etc.c' 2023-06-22T08:24:19.811 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/file-type.h' 2023-06-22T08:24:19.811 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/same-inode.h' 2023-06-22T08:24:19.811 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xvasprintf.h' 2023-06-22T08:24:19.811 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/i-ring.c' 2023-06-22T08:24:19.811 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/c-strncasecmp.c' 2023-06-22T08:24:19.812 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/posixver.h' 2023-06-22T08:24:19.812 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/locale.in.h' 2023-06-22T08:24:19.812 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unistr.h' 2023-06-22T08:24:19.812 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xreadlink.c' 2023-06-22T08:24:19.812 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/human.c' 2023-06-22T08:24:19.812 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sha512.c' 2023-06-22T08:24:19.812 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open-solaris.gperf' 2023-06-22T08:24:19.813 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fdopendir.c' 2023-06-22T08:24:19.813 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/regex.c' 2023-06-22T08:24:19.813 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getaddrinfo.c' 2023-06-22T08:24:19.813 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/acl_entries.c' 2023-06-22T08:24:19.813 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getopt1.c' 2023-06-22T08:24:19.814 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/selinux-at.h' 2023-06-22T08:24:19.814 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strnlen1.c' 2023-06-22T08:24:19.814 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/readtokens0.h' 2023-06-22T08:24:19.814 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/symlink.c' 2023-06-22T08:24:19.814 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fclose.c' 2023-06-22T08:24:19.814 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/freadseek.h' 2023-06-22T08:24:19.814 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/safe-write.h' 2023-06-22T08:24:19.814 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fsusage.c' 2023-06-22T08:24:19.815 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/basename-lgpl.c' 2023-06-22T08:24:19.815 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/nproc.h' 2023-06-22T08:24:19.815 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stat-macros.h' 2023-06-22T08:24:19.815 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrtoimax.c' 2023-06-22T08:24:19.815 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xmemcoll.c' 2023-06-22T08:24:19.815 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dirent-safer.h' 2023-06-22T08:24:19.815 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbchar.c' 2023-06-22T08:24:19.815 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memchr2.h' 2023-06-22T08:24:19.816 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strdup.c' 2023-06-22T08:24:19.816 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strnlen.c' 2023-06-22T08:24:19.816 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/c-strtod.h' 2023-06-22T08:24:19.816 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getloadavg.c' 2023-06-22T08:24:19.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~237s 2023-06-22T08:24:20.024 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/file-type.c' 2023-06-22T08:24:20.025 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/i-ring.h' 2023-06-22T08:24:20.025 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fstatat.c' 2023-06-22T08:24:20.025 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sha512.h' 2023-06-22T08:24:20.025 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/gettimeofday.c' 2023-06-22T08:24:20.025 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbsrtowcs-state.c' 2023-06-22T08:24:20.025 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xmalloc.c' 2023-06-22T08:24:20.026 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memcoll.c' 2023-06-22T08:24:20.026 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrtoul.c' 2023-06-22T08:24:20.026 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fts_.h' 2023-06-22T08:24:20.026 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/c-ctype.h' 2023-06-22T08:24:20.026 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stripslash.c' 2023-06-22T08:24:20.026 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open-osf.h' 2023-06-22T08:24:20.026 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/wcrtomb.c' 2023-06-22T08:24:20.027 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/utimecmp.h' 2023-06-22T08:24:20.027 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdio--.h' 2023-06-22T08:24:20.027 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/randperm.c' 2023-06-22T08:24:20.027 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/areadlinkat.c' 2023-06-22T08:24:20.027 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/readutmp.c' 2023-06-22T08:24:20.027 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unicodeio.h' 2023-06-22T08:24:20.027 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/full-read.h' 2023-06-22T08:24:20.027 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fseterr.c' 2023-06-22T08:24:20.028 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/intprops.h' 2023-06-22T08:24:20.028 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/pathmax.h' 2023-06-22T08:24:20.028 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/utimens.c' 2023-06-22T08:24:20.028 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/propername.h' 2023-06-22T08:24:20.028 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/acl-internal.h' 2023-06-22T08:24:20.028 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/safe-write.c' 2023-06-22T08:24:20.028 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strtoll.c' 2023-06-22T08:24:20.029 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mkstemp.c' 2023-06-22T08:24:20.029 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/Makefile.am' 2023-06-22T08:24:20.029 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/file-set.h' 2023-06-22T08:24:20.029 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/md5.c' 2023-06-22T08:24:20.029 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/base64.h' 2023-06-22T08:24:20.029 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open-irix.gperf' 2023-06-22T08:24:20.030 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/vasnprintf.h' 2023-06-22T08:24:20.030 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrndup.h' 2023-06-22T08:24:20.030 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/hash-pjw.c' 2023-06-22T08:24:20.030 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sys_time.in.h' 2023-06-22T08:24:20.030 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strtol.c' 2023-06-22T08:24:20.030 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/backupfile.h' 2023-06-22T08:24:20.030 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/setenv.c' 2023-06-22T08:24:20.030 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sys_select.in.h' 2023-06-22T08:24:20.031 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/config.charset' 2023-06-22T08:24:20.031 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xgetcwd.h' 2023-06-22T08:24:20.031 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fnmatch_loop.c' 2023-06-22T08:24:20.031 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/hash-triple.c' 2023-06-22T08:24:20.031 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/hash-pjw.h' 2023-06-22T08:24:20.031 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memcmp2.c' 2023-06-22T08:24:20.032 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/write.c' 2023-06-22T08:24:20.032 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/modechange.h' 2023-06-22T08:24:20.032 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unlink.c' 2023-06-22T08:24:20.032 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdbool.in.h' 2023-06-22T08:24:20.032 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/time.in.h' 2023-06-22T08:24:20.032 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/freadptr.h' 2023-06-22T08:24:20.032 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/isblank.c' 2023-06-22T08:24:20.033 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getline.c' 2023-06-22T08:24:20.033 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/math.in.h' 2023-06-22T08:24:20.033 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/buffer-lcm.c' 2023-06-22T08:24:20.033 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/exclude.c' 2023-06-22T08:24:20.033 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/filevercmp.c' 2023-06-22T08:24:20.033 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/closein.c' 2023-06-22T08:24:20.033 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/closein.h' 2023-06-22T08:24:20.034 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xfts.h' 2023-06-22T08:24:20.034 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/file-set.c' 2023-06-22T08:24:20.034 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unistd-safer.h' 2023-06-22T08:24:20.034 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/uniwidth.h' 2023-06-22T08:24:20.034 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/symlinkat.c' 2023-06-22T08:24:20.034 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/readtokens.c' 2023-06-22T08:24:20.035 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/exitfail.h' 2023-06-22T08:24:20.035 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/malloca.c' 2023-06-22T08:24:20.035 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fd-safer.c' 2023-06-22T08:24:20.035 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xsize.h' 2023-06-22T08:24:20.035 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strnumcmp-in.h' 2023-06-22T08:24:20.035 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbswidth.c' 2023-06-22T08:24:20.035 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fopen-safer.c' 2023-06-22T08:24:20.035 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrndup.c' 2023-06-22T08:24:20.036 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/human.h' 2023-06-22T08:24:20.036 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/gettext.h' 2023-06-22T08:24:20.036 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mktime-internal.h' 2023-06-22T08:24:20.036 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fchownat.c' 2023-06-22T08:24:20.036 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dup-safer.c' 2023-06-22T08:24:20.037 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrtol-error.c' 2023-06-22T08:24:20.037 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/full-write.h' 2023-06-22T08:24:20.037 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dup-safer-flag.c' 2023-06-22T08:24:20.037 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/float+.h' 2023-06-22T08:24:20.037 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sigprocmask.c' 2023-06-22T08:24:20.037 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/printf-args.c' 2023-06-22T08:24:20.038 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/gethrxtime.c' 2023-06-22T08:24:20.038 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/randread.h' 2023-06-22T08:24:20.038 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strpbrk.c' 2023-06-22T08:24:20.038 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getugroups.c' 2023-06-22T08:24:20.038 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/chdir-long.c' 2023-06-22T08:24:20.038 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fcntl--.h' 2023-06-22T08:24:20.039 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mkstemp-safer.c' 2023-06-22T08:24:20.039 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbslen.c' 2023-06-22T08:24:20.039 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/bitrotate.h' 2023-06-22T08:24:20.039 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dev-ino.h' 2023-06-22T08:24:20.039 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xasprintf.c' 2023-06-22T08:24:20.039 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sockets.h' 2023-06-22T08:24:20.040 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/inet_ntop.c' 2023-06-22T08:24:20.040 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getdate.y' 2023-06-22T08:24:20.040 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stddef.in.h' 2023-06-22T08:24:20.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/isnand-nolibm.h' 2023-06-22T08:24:20.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbsrtowcs.c' 2023-06-22T08:24:20.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/cycle-check.h' 2023-06-22T08:24:20.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sha256.c' 2023-06-22T08:24:20.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/readlink.c' 2023-06-22T08:24:20.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/uname.c' 2023-06-22T08:24:20.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/printf-parse.h' 2023-06-22T08:24:20.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/printf-frexpl.c' 2023-06-22T08:24:20.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/exitfail.c' 2023-06-22T08:24:20.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/linkat.c' 2023-06-22T08:24:20.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/same.c' 2023-06-22T08:24:20.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/btowc.c' 2023-06-22T08:24:20.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memcasecmp.h' 2023-06-22T08:24:20.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sys_utsname.in.h' 2023-06-22T08:24:20.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/filenamecat-lgpl.c' 2023-06-22T08:24:20.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstriconv.c' 2023-06-22T08:24:20.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/rand-isaac.c' 2023-06-22T08:24:20.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrtol.h' 2023-06-22T08:24:20.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strtod.c' 2023-06-22T08:24:20.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/select.c' 2023-06-22T08:24:20.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/ftello.c' 2023-06-22T08:24:20.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strnlen1.h' 2023-06-22T08:24:20.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xtime.h' 2023-06-22T08:24:20.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/save-cwd.c' 2023-06-22T08:24:20.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sha256.h' 2023-06-22T08:24:20.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memxfrm.c' 2023-06-22T08:24:20.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/opendir-safer.c' 2023-06-22T08:24:20.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/size_max.h' 2023-06-22T08:24:20.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbscasecmp.c' 2023-06-22T08:24:20.262 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fflush.c' 2023-06-22T08:24:20.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mgetgroups.c' 2023-06-22T08:24:20.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getdate.h' 2023-06-22T08:24:20.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/malloca.valgrind' 2023-06-22T08:24:20.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv_open-irix.h' 2023-06-22T08:24:20.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xgetcwd.c' 2023-06-22T08:24:20.263 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xalloc-die.c' 2023-06-22T08:24:20.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/long-options.c' 2023-06-22T08:24:20.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/basename.c' 2023-06-22T08:24:20.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/error.c' 2023-06-22T08:24:20.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memcmp2.h' 2023-06-22T08:24:20.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/regex.h' 2023-06-22T08:24:20.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mkdir-p.h' 2023-06-22T08:24:20.264 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xstrtoumax.c' 2023-06-22T08:24:20.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dirchownmod.c' 2023-06-22T08:24:20.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/isnanl-nolibm.h' 2023-06-22T08:24:20.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/areadlink-with-size.c' 2023-06-22T08:24:20.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/printf-parse.c' 2023-06-22T08:24:20.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unistd.in.h' 2023-06-22T08:24:20.265 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strtoumax.c' 2023-06-22T08:24:20.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mempcpy.c' 2023-06-22T08:24:20.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xfreopen.c' 2023-06-22T08:24:20.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memmove.c' 2023-06-22T08:24:20.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xmemxfrm.c' 2023-06-22T08:24:20.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/link.c' 2023-06-22T08:24:20.266 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/priv-set.c' 2023-06-22T08:24:20.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/signbitl.c' 2023-06-22T08:24:20.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fnmatch.c' 2023-06-22T08:24:20.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/backupfile.c' 2023-06-22T08:24:20.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fd-safer-flag.c' 2023-06-22T08:24:20.267 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/signal.in.h' 2023-06-22T08:24:20.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/filenamecat.h' 2023-06-22T08:24:20.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xvasprintf.c' 2023-06-22T08:24:20.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/euidaccess-stat.h' 2023-06-22T08:24:20.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fsync.c' 2023-06-22T08:24:20.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/filemode.h' 2023-06-22T08:24:20.268 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/write-any-file.h' 2023-06-22T08:24:20.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdlib.in.h' 2023-06-22T08:24:20.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/selinux-at.c' 2023-06-22T08:24:20.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/netinet_in.in.h' 2023-06-22T08:24:20.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/u64.h' 2023-06-22T08:24:20.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/unsetenv.c' 2023-06-22T08:24:20.269 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memchr.c' 2023-06-22T08:24:20.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/verror.h' 2023-06-22T08:24:20.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/freading.h' 2023-06-22T08:24:20.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xgethostname.c' 2023-06-22T08:24:20.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/nanosleep.c' 2023-06-22T08:24:20.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xprintf.h' 2023-06-22T08:24:20.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/malloca.h' 2023-06-22T08:24:20.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/settime.c' 2023-06-22T08:24:20.270 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memchr2.c' 2023-06-22T08:24:20.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memcasecmp.c' 2023-06-22T08:24:20.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/frexpl.c' 2023-06-22T08:24:20.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/root-dev-ino.h' 2023-06-22T08:24:20.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/hard-locale.c' 2023-06-22T08:24:20.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/localcharset.h' 2023-06-22T08:24:20.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdio-impl.h' 2023-06-22T08:24:20.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/set-mode-acl.c' 2023-06-22T08:24:20.271 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/imaxtostr.c' 2023-06-22T08:24:20.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/freadseek.c' 2023-06-22T08:24:20.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/md5.h' 2023-06-22T08:24:20.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/idcache.c' 2023-06-22T08:24:20.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/calloc.c' 2023-06-22T08:24:20.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fseeko.c' 2023-06-22T08:24:20.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/propername.c' 2023-06-22T08:24:20.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/vprintf.c' 2023-06-22T08:24:20.272 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fileblocks.c' 2023-06-22T08:24:20.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memxfrm.h' 2023-06-22T08:24:20.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fprintftime.c' 2023-06-22T08:24:20.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/alloca.c' 2023-06-22T08:24:20.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/group-member.c' 2023-06-22T08:24:20.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/striconv.h' 2023-06-22T08:24:20.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbchar.h' 2023-06-22T08:24:20.273 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xreadlink.h' 2023-06-22T08:24:20.743 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/c-strcase.h' 2023-06-22T08:24:20.744 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/acl.h' 2023-06-22T08:24:20.744 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/modechange.c' 2023-06-22T08:24:20.744 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fpurge.c' 2023-06-22T08:24:20.744 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/openat.h' 2023-06-22T08:24:20.744 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/signbitf.c' 2023-06-22T08:24:20.745 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mktime.c' 2023-06-22T08:24:20.745 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbsalign.c' 2023-06-22T08:24:20.745 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/xfreopen.h' 2023-06-22T08:24:20.745 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/iconv.in.h' 2023-06-22T08:24:20.745 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/hard-locale.h' 2023-06-22T08:24:20.745 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/nl_langinfo.c' 2023-06-22T08:24:20.745 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/striconv.c' 2023-06-22T08:24:20.746 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sigaction.c' 2023-06-22T08:24:20.746 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/memset.c' 2023-06-22T08:24:20.746 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fd-reopen.h' 2023-06-22T08:24:20.746 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/mbiter.h' 2023-06-22T08:24:20.746 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strftime.c' 2023-06-22T08:24:20.746 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/uinttostr.c' 2023-06-22T08:24:20.746 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/canon-host.c' 2023-06-22T08:24:20.746 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/rpmatch.c' 2023-06-22T08:24:20.747 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/sha1.c' 2023-06-22T08:24:20.747 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/vfprintf.c' 2023-06-22T08:24:20.747 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/isnanf-nolibm.h' 2023-06-22T08:24:20.747 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/c-strtod.c' 2023-06-22T08:24:20.747 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/getpass.c' 2023-06-22T08:24:20.747 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/fnmatch.in.h' 2023-06-22T08:24:20.747 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/isnand.c' 2023-06-22T08:24:20.747 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/write-any-file.c' 2023-06-22T08:24:20.748 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/c-strcasecmp.c' 2023-06-22T08:24:20.748 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/stdint.in.h' 2023-06-22T08:24:20.748 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/dirname-lgpl.c' 2023-06-22T08:24:20.748 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/lseek.c' 2023-06-22T08:24:20.748 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/lib/strerror.c' 2023-06-22T08:24:20.748 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/lib' 2023-06-22T08:24:20.748 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.tarball-version' 2023-06-22T08:24:20.748 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_error_message_uppercase' 2023-06-22T08:24:20.749 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_space_tab' 2023-06-22T08:24:20.749 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/ChangeLog-2007' 2023-06-22T08:24:20.749 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_require_config_h_first' 2023-06-22T08:24:20.749 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/ABOUT-NLS' 2023-06-22T08:24:20.749 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/aclocal.m4' 2023-06-22T08:24:20.749 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/bootstrap' 2023-06-22T08:24:20.749 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/NEWS' 2023-06-22T08:24:20.749 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/ChangeLog-2006' 2023-06-22T08:24:20.750 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_unmarked_diagnostics' 2023-06-22T08:24:20.750 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/Makefile.in' 2023-06-22T08:24:20.750 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/ChangeLog-2005' 2023-06-22T08:24:20.750 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/timespec.m4' 2023-06-22T08:24:20.750 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/xfts.m4' 2023-06-22T08:24:20.750 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/jm-macros.m4' 2023-06-22T08:24:20.750 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getugroups.m4' 2023-06-22T08:24:20.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/unicodeio.m4' 2023-06-22T08:24:20.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/lib-link.m4' 2023-06-22T08:24:20.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stpcpy.m4' 2023-06-22T08:24:20.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/math_h.m4' 2023-06-22T08:24:20.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/printf.m4' 2023-06-22T08:24:20.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/locale-fr.m4' 2023-06-22T08:24:20.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sha1.m4' 2023-06-22T08:24:20.751 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/safe-write.m4' 2023-06-22T08:24:20.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/double-slash-root.m4' 2023-06-22T08:24:20.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/close-stream.m4' 2023-06-22T08:24:20.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/linkat.m4' 2023-06-22T08:24:20.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/nl_langinfo.m4' 2023-06-22T08:24:20.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/host-os.m4' 2023-06-22T08:24:20.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/priv-set.m4' 2023-06-22T08:24:20.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/write.m4' 2023-06-22T08:24:20.752 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/lchmod.m4' 2023-06-22T08:24:20.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/read-file.m4' 2023-06-22T08:24:20.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/netinet_in_h.m4' 2023-06-22T08:24:20.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/00gnulib.m4' 2023-06-22T08:24:20.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/utimbuf.m4' 2023-06-22T08:24:20.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/euidaccess.m4' 2023-06-22T08:24:20.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/cloexec.m4' 2023-06-22T08:24:20.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/argmatch.m4' 2023-06-22T08:24:20.753 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mountlist.m4' 2023-06-22T08:24:20.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/string_h.m4' 2023-06-22T08:24:20.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/utimens.m4' 2023-06-22T08:24:20.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/warn-on-use.m4' 2023-06-22T08:24:20.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/codeset.m4' 2023-06-22T08:24:20.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/unlink.m4' 2023-06-22T08:24:20.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fclose.m4' 2023-06-22T08:24:20.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/vfprintf-posix.m4' 2023-06-22T08:24:20.754 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/yield.m4' 2023-06-22T08:24:20.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/localcharset.m4' 2023-06-22T08:24:20.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/isnanf.m4' 2023-06-22T08:24:20.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/idcache.m4' 2023-06-22T08:24:20.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/readlink.m4' 2023-06-22T08:24:20.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mbrlen.m4' 2023-06-22T08:24:20.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getgroups.m4' 2023-06-22T08:24:20.755 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getopt.m4' 2023-06-22T08:24:20.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getndelim2.m4' 2023-06-22T08:24:20.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strnlen.m4' 2023-06-22T08:24:20.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mgetgroups.m4' 2023-06-22T08:24:20.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/signalblocking.m4' 2023-06-22T08:24:20.756 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stat.m4' 2023-06-22T08:24:20.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/gettext.m4' 2023-06-22T08:24:20.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/uname.m4' 2023-06-22T08:24:20.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/xnanosleep.m4' 2023-06-22T08:24:20.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strtoimax.m4' 2023-06-22T08:24:20.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/locale-zh.m4' 2023-06-22T08:24:20.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mode_t.m4' 2023-06-22T08:24:20.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/printf-frexp.m4' 2023-06-22T08:24:20.757 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strcspn.m4' 2023-06-22T08:24:20.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/wchar_h.m4' 2023-06-22T08:24:20.921 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/file-type.m4' 2023-06-22T08:24:20.922 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/usleep.m4' 2023-06-22T08:24:20.922 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stdlib-safer.m4' 2023-06-22T08:24:20.922 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/setenv.m4' 2023-06-22T08:24:20.922 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stat-prog.m4' 2023-06-22T08:24:20.922 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/write-any-file.m4' 2023-06-22T08:24:20.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/yesno.m4' 2023-06-22T08:24:20.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sys_ioctl_h.m4' 2023-06-22T08:24:20.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/lchown.m4' 2023-06-22T08:24:20.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/intlmacosx.m4' 2023-06-22T08:24:20.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strsignal.m4' 2023-06-22T08:24:20.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mbsinit.m4' 2023-06-22T08:24:20.923 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/memcasecmp.m4' 2023-06-22T08:24:20.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/freopen.m4' 2023-06-22T08:24:20.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/iconv.m4' 2023-06-22T08:24:20.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mbrtowc.m4' 2023-06-22T08:24:20.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/ioctl.m4' 2023-06-22T08:24:20.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sockpfaf.m4' 2023-06-22T08:24:20.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/thread.m4' 2023-06-22T08:24:20.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fflush.m4' 2023-06-22T08:24:20.924 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/assert.m4' 2023-06-22T08:24:20.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stpncpy.m4' 2023-06-22T08:24:20.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strftime.m4' 2023-06-22T08:24:20.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/wcrtomb.m4' 2023-06-22T08:24:20.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/langinfo_h.m4' 2023-06-22T08:24:20.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/safe-read.m4' 2023-06-22T08:24:20.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/clock_time.m4' 2023-06-22T08:24:20.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/tls.m4' 2023-06-22T08:24:20.925 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/memcoll.m4' 2023-06-22T08:24:20.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/readutmp.m4' 2023-06-22T08:24:20.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stdio_h.m4' 2023-06-22T08:24:20.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/ftello.m4' 2023-06-22T08:24:20.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/memcmp.m4' 2023-06-22T08:24:20.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/canon-host.m4' 2023-06-22T08:24:20.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/gmp.m4' 2023-06-22T08:24:20.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/environ.m4' 2023-06-22T08:24:20.926 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fdopendir.m4' 2023-06-22T08:24:20.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stdarg.m4' 2023-06-22T08:24:20.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stdbool.m4' 2023-06-22T08:24:20.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/intmax_t.m4' 2023-06-22T08:24:20.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/isapipe.m4' 2023-06-22T08:24:20.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/ldexpl.m4' 2023-06-22T08:24:20.927 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/remove.m4' 2023-06-22T08:24:20.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fpurge.m4' 2023-06-22T08:24:20.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getusershell.m4' 2023-06-22T08:24:20.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/memcpy.m4' 2023-06-22T08:24:20.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fcntl.m4' 2023-06-22T08:24:20.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/tempname.m4' 2023-06-22T08:24:20.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/xstrtol.m4' 2023-06-22T08:24:20.928 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fseeko.m4' 2023-06-22T08:24:20.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fnmatch.m4' 2023-06-22T08:24:20.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/save-cwd.m4' 2023-06-22T08:24:20.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/config-h.m4' 2023-06-22T08:24:20.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/filemode.m4' 2023-06-22T08:24:20.929 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/errno_h.m4' 2023-06-22T08:24:20.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/eealloc.m4' 2023-06-22T08:24:20.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strdup.m4' 2023-06-22T08:24:20.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/signal_h.m4' 2023-06-22T08:24:20.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/regex.m4' 2023-06-22T08:24:20.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mbchar.m4' 2023-06-22T08:24:20.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strtoull.m4' 2023-06-22T08:24:20.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stdint.m4' 2023-06-22T08:24:20.930 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/wctype_h.m4' 2023-06-22T08:24:20.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/realloc.m4' 2023-06-22T08:24:20.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/printf-frexpl.m4' 2023-06-22T08:24:20.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/open.m4' 2023-06-22T08:24:20.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/inline.m4' 2023-06-22T08:24:20.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/warnings.m4' 2023-06-22T08:24:20.931 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/vasprintf-posix.m4' 2023-06-22T08:24:20.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/xstrndup.m4' 2023-06-22T08:24:20.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sys_utsname_h.m4' 2023-06-22T08:24:20.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/locale-ja.m4' 2023-06-22T08:24:20.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/snprintf.m4' 2023-06-22T08:24:20.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/po.m4' 2023-06-22T08:24:20.932 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/hash.m4' 2023-06-22T08:24:20.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sig2str.m4' 2023-06-22T08:24:20.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/ctype.m4' 2023-06-22T08:24:20.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strnumcmp.m4' 2023-06-22T08:24:20.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/rpmatch.m4' 2023-06-22T08:24:20.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/md5.m4' 2023-06-22T08:24:20.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/boottime.m4' 2023-06-22T08:24:20.933 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/unistd_h.m4' 2023-06-22T08:24:20.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fd-reopen.m4' 2023-06-22T08:24:20.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/cycle-check.m4' 2023-06-22T08:24:20.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/ls-mntd-fs.m4' 2023-06-22T08:24:20.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mkancesdirs.m4' 2023-06-22T08:24:20.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/manywarnings.m4' 2023-06-22T08:24:20.934 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/ChangeLog-2007' 2023-06-22T08:24:20.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getcwd-path-max.m4' 2023-06-22T08:24:20.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/modechange.m4' 2023-06-22T08:24:20.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stdlib_h.m4' 2023-06-22T08:24:20.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/gethostname.m4' 2023-06-22T08:24:20.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/physmem.m4' 2023-06-22T08:24:20.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fileblocks.m4' 2023-06-22T08:24:20.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/socklen.m4' 2023-06-22T08:24:20.935 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/st_dm_mode.m4' 2023-06-22T08:24:20.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/ungetc.m4' 2023-06-22T08:24:20.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/posixver.m4' 2023-06-22T08:24:20.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getdtablesize.m4' 2023-06-22T08:24:20.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/check-decl.m4' 2023-06-22T08:24:20.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/threadlib.m4' 2023-06-22T08:24:20.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sys_socket_h.m4' 2023-06-22T08:24:20.936 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getaddrinfo.m4' 2023-06-22T08:24:20.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/float_h.m4' 2023-06-22T08:24:20.937 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/btowc.m4' 2023-06-22T08:24:21.104 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/userspec.m4' 2023-06-22T08:24:21.104 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/posixtm.m4' 2023-06-22T08:24:21.104 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/alloca.m4' 2023-06-22T08:24:21.105 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/lstat.m4' 2023-06-22T08:24:21.105 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/lock.m4' 2023-06-22T08:24:21.105 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/readtokens.m4' 2023-06-22T08:24:21.105 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/locale-tr.m4' 2023-06-22T08:24:21.105 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/wctob.m4' 2023-06-22T08:24:21.105 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/canonicalize.m4' 2023-06-22T08:24:21.106 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/servent.m4' 2023-06-22T08:24:21.106 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mbswidth.m4' 2023-06-22T08:24:21.106 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/memrchr.m4' 2023-06-22T08:24:21.106 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/autobuild.m4' 2023-06-22T08:24:21.106 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strtol.m4' 2023-06-22T08:24:21.106 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/closein.m4' 2023-06-22T08:24:21.107 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sha256.m4' 2023-06-22T08:24:21.107 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mbslen.m4' 2023-06-22T08:24:21.107 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/lseek.m4' 2023-06-22T08:24:21.107 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/base64.m4' 2023-06-22T08:24:21.107 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/iconv_h.m4' 2023-06-22T08:24:21.108 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/longlong.m4' 2023-06-22T08:24:21.108 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/dirfd.m4' 2023-06-22T08:24:21.108 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/d-ino.m4' 2023-06-22T08:24:21.109 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/ssize_t.m4' 2023-06-22T08:24:21.109 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/gnulib-comp.m4' 2023-06-22T08:24:21.109 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/memset.m4' 2023-06-22T08:24:21.109 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getcwd-abort-bug.m4' 2023-06-22T08:24:21.109 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/c-strtod.m4' 2023-06-22T08:24:21.109 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fts.m4' 2023-06-22T08:24:21.109 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/lib-prefix.m4' 2023-06-22T08:24:21.110 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mmap-anon.m4' 2023-06-22T08:24:21.110 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/close.m4' 2023-06-22T08:24:21.110 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/lib-ignore.m4' 2023-06-22T08:24:21.110 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/gnulib-common.m4' 2023-06-22T08:24:21.110 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/time_h.m4' 2023-06-22T08:24:21.110 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/nls.m4' 2023-06-22T08:24:21.110 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/dirname.m4' 2023-06-22T08:24:21.111 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fcntl_h.m4' 2023-06-22T08:24:21.111 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stddef_h.m4' 2023-06-22T08:24:21.111 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mkstemp.m4' 2023-06-22T08:24:21.111 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/link-follow.m4' 2023-06-22T08:24:21.111 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/wcwidth.m4' 2023-06-22T08:24:21.111 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/memchr.m4' 2023-06-22T08:24:21.111 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strerror.m4' 2023-06-22T08:24:21.112 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/iconv_open.m4' 2023-06-22T08:24:21.112 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/exponentf.m4' 2023-06-22T08:24:21.112 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/pathmax.m4' 2023-06-22T08:24:21.112 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stdio-safer.m4' 2023-06-22T08:24:21.112 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/acl.m4' 2023-06-22T08:24:21.112 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fpending.m4' 2023-06-22T08:24:21.112 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/putenv.m4' 2023-06-22T08:24:21.112 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/include-exclude-prog.m4' 2023-06-22T08:24:21.113 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mktime.m4' 2023-06-22T08:24:21.113 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getdate.m4' 2023-06-22T08:24:21.113 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fsync.m4' 2023-06-22T08:24:21.113 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/symlink.m4' 2023-06-22T08:24:21.113 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/quote.m4' 2023-06-22T08:24:21.114 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getloadavg.m4' 2023-06-22T08:24:21.114 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/gettime.m4' 2023-06-22T08:24:21.114 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/quotearg.m4' 2023-06-22T08:24:21.114 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/inttypes.m4' 2023-06-22T08:24:21.114 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/dos.m4' 2023-06-22T08:24:21.114 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/gettimeofday.m4' 2023-06-22T08:24:21.114 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/hostent.m4' 2023-06-22T08:24:21.114 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/chdir-long.m4' 2023-06-22T08:24:21.115 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/selinux-selinux-h.m4' 2023-06-22T08:24:21.115 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fcntl-safer.m4' 2023-06-22T08:24:21.115 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/inttostr.m4' 2023-06-22T08:24:21.115 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getline.m4' 2023-06-22T08:24:21.115 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/posix-shell.m4' 2023-06-22T08:24:21.115 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/d-type.m4' 2023-06-22T08:24:21.115 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/inet_pton.m4' 2023-06-22T08:24:21.115 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sigaction.m4' 2023-06-22T08:24:21.116 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mkdir.m4' 2023-06-22T08:24:21.116 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getdelim.m4' 2023-06-22T08:24:21.116 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/xsize.m4' 2023-06-22T08:24:21.116 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/symlinkat.m4' 2023-06-22T08:24:21.116 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/dirent-safer.m4' 2023-06-22T08:24:21.116 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/include_next.m4' 2023-06-22T08:24:21.116 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getcwd.m4' 2023-06-22T08:24:21.117 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sys_time_h.m4' 2023-06-22T08:24:21.117 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fprintftime.m4' 2023-06-22T08:24:21.117 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/prereq.m4' 2023-06-22T08:24:21.117 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/netdb_h.m4' 2023-06-22T08:24:21.117 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/perl.m4' 2023-06-22T08:24:21.117 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/extensions.m4' 2023-06-22T08:24:21.117 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/closeout.m4' 2023-06-22T08:24:21.117 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/jm-winsz2.m4' 2023-06-22T08:24:21.118 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/settime.m4' 2023-06-22T08:24:21.118 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/progtest.m4' 2023-06-22T08:24:21.118 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/atexit.m4' 2023-06-22T08:24:21.118 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/savedir.m4' 2023-06-22T08:24:21.118 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/isnand.m4' 2023-06-22T08:24:21.118 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/arpa_inet_h.m4' 2023-06-22T08:24:21.118 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/root-dev-ino.m4' 2023-06-22T08:24:21.118 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/hard-locale.m4' 2023-06-22T08:24:21.119 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/multiarch.m4' 2023-06-22T08:24:21.119 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/gnu-make.m4' 2023-06-22T08:24:21.119 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stdint_h.m4' 2023-06-22T08:24:21.119 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/openat.m4' 2023-06-22T08:24:21.119 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/nproc.m4' 2023-06-22T08:24:21.119 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/link.m4' 2023-06-22T08:24:21.119 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/faccessat.m4' 2023-06-22T08:24:21.119 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/uptime.m4' 2023-06-22T08:24:21.120 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fpieee.m4' 2023-06-22T08:24:21.497 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mkdir-p.m4' 2023-06-22T08:24:21.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/xvasprintf.m4' 2023-06-22T08:24:21.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strpbrk.m4' 2023-06-22T08:24:21.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/rename.m4' 2023-06-22T08:24:21.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/wchar_t.m4' 2023-06-22T08:24:21.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/glibc21.m4' 2023-06-22T08:24:21.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/perror.m4' 2023-06-22T08:24:21.498 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/backupfile.m4' 2023-06-22T08:24:21.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/xstrtod.m4' 2023-06-22T08:24:21.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/xattr.m4' 2023-06-22T08:24:21.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fsusage.m4' 2023-06-22T08:24:21.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/human.m4' 2023-06-22T08:24:21.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mbsrtowcs.m4' 2023-06-22T08:24:21.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/tm_gmtoff.m4' 2023-06-22T08:24:21.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strtod.m4' 2023-06-22T08:24:21.499 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/selinux-context-h.m4' 2023-06-22T08:24:21.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strtoumax.m4' 2023-06-22T08:24:21.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/inttypes-pri.m4' 2023-06-22T08:24:21.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/bison.m4' 2023-06-22T08:24:21.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/filenamecat.m4' 2023-06-22T08:24:21.500 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/i-ring.m4' 2023-06-22T08:24:21.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strstr.m4' 2023-06-22T08:24:21.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/calloc.m4' 2023-06-22T08:24:21.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getpass.m4' 2023-06-22T08:24:21.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/error.m4' 2023-06-22T08:24:21.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/vasnprintf.m4' 2023-06-22T08:24:21.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/exponentd.m4' 2023-06-22T08:24:21.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/unlocked-io.m4' 2023-06-22T08:24:21.501 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/nocrash.m4' 2023-06-22T08:24:21.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fstypename.m4' 2023-06-22T08:24:21.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/gethrxtime.m4' 2023-06-22T08:24:21.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/vasprintf.m4' 2023-06-22T08:24:21.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/isblank.m4' 2023-06-22T08:24:21.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/jm-winsz1.m4' 2023-06-22T08:24:21.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/memmove.m4' 2023-06-22T08:24:21.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/unlinkdir.m4' 2023-06-22T08:24:21.502 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sockets.m4' 2023-06-22T08:24:21.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/savewd.m4' 2023-06-22T08:24:21.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/frexpl.m4' 2023-06-22T08:24:21.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fopen.m4' 2023-06-22T08:24:21.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/unistd-safer.m4' 2023-06-22T08:24:21.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/utimecmp.m4' 2023-06-22T08:24:21.503 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mempcpy.m4' 2023-06-22T08:24:21.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mpsort.m4' 2023-06-22T08:24:21.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sys_stat_h.m4' 2023-06-22T08:24:21.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/exponentl.m4' 2023-06-22T08:24:21.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sha512.m4' 2023-06-22T08:24:21.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/time_r.m4' 2023-06-22T08:24:21.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sys_select_h.m4' 2023-06-22T08:24:21.504 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/vprintf-posix.m4' 2023-06-22T08:24:21.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/malloc.m4' 2023-06-22T08:24:21.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/tzset.m4' 2023-06-22T08:24:21.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/memxfrm.m4' 2023-06-22T08:24:21.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/flexmember.m4' 2023-06-22T08:24:21.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fcntl-o.m4' 2023-06-22T08:24:21.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/getpagesize.m4' 2023-06-22T08:24:21.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/unlink-busy.m4' 2023-06-22T08:24:21.505 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/fchdir.m4' 2023-06-22T08:24:21.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/malloca.m4' 2023-06-22T08:24:21.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/lib-ld.m4' 2023-06-22T08:24:21.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/euidaccess-stat.m4' 2023-06-22T08:24:21.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/xgetcwd.m4' 2023-06-22T08:24:21.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/dirent_h.m4' 2023-06-22T08:24:21.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/dup2.m4' 2023-06-22T08:24:21.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/locale_h.m4' 2023-06-22T08:24:21.506 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/stat-time.m4' 2023-06-22T08:24:21.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/select.m4' 2023-06-22T08:24:21.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/same.m4' 2023-06-22T08:24:21.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/freading.m4' 2023-06-22T08:24:21.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strtoll.m4' 2023-06-22T08:24:21.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/group-member.m4' 2023-06-22T08:24:21.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/inttypes_h.m4' 2023-06-22T08:24:21.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/long-options.m4' 2023-06-22T08:24:21.507 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/chown.m4' 2023-06-22T08:24:21.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/xalloc.m4' 2023-06-22T08:24:21.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/nanosleep.m4' 2023-06-22T08:24:21.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/utimes.m4' 2023-06-22T08:24:21.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/size_max.m4' 2023-06-22T08:24:21.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/frexp.m4' 2023-06-22T08:24:21.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/exclude.m4' 2023-06-22T08:24:21.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/wint_t.m4' 2023-06-22T08:24:21.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strndup.m4' 2023-06-22T08:24:21.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/lib-check.m4' 2023-06-22T08:24:21.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mbiter.m4' 2023-06-22T08:24:21.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/mbstate_t.m4' 2023-06-22T08:24:21.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/rmdir.m4' 2023-06-22T08:24:21.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/inet_ntop.m4' 2023-06-22T08:24:21.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/version-etc.m4' 2023-06-22T08:24:21.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/isnanl.m4' 2023-06-22T08:24:21.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/signbit.m4' 2023-06-22T08:24:21.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/strtoul.m4' 2023-06-22T08:24:21.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/m4/sleep.m4' 2023-06-22T08:24:21.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/m4' 2023-06-22T08:24:21.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.version' 2023-06-22T08:24:21.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_prohibit_fail_0' 2023-06-22T08:24:21.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/Makefile.am' 2023-06-22T08:24:21.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/INSTALL' 2023-06-22T08:24:21.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/maint.mk' 2023-06-22T08:24:21.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/bootstrap.conf' 2023-06-22T08:24:21.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xstrtoimax.c' 2023-06-22T08:24:21.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-update-copyright.sh' 2023-06-22T08:24:21.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/connect.c' 2023-06-22T08:24:21.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-signbit.c' 2023-06-22T08:24:21.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-stdlib.c' 2023-06-22T08:24:21.973 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-wcrtomb.c' 2023-06-22T08:24:21.973 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sameacls.c' 2023-06-22T08:24:21.974 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-isnanf-nolibm.c' 2023-06-22T08:24:21.974 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-wcwidth.c' 2023-06-22T08:24:21.974 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-argmatch.c' 2023-06-22T08:24:21.974 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-dirname.c' 2023-06-22T08:24:21.974 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fseeko.c' 2023-06-22T08:24:21.974 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sys_select.c' 2023-06-22T08:24:21.975 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freopen-safer.c' 2023-06-22T08:24:21.975 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbrtowc.c' 2023-06-22T08:24:21.975 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fseterr.c' 2023-06-22T08:24:21.975 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-vasprintf.c' 2023-06-22T08:24:21.975 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fopen.h' 2023-06-22T08:24:21.975 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-ftello.sh' 2023-06-22T08:24:21.975 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsinit.sh' 2023-06-22T08:24:21.976 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-dup-safer.c' 2023-06-22T08:24:21.976 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsstr3.c' 2023-06-22T08:24:21.976 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-vprintf-posix.sh' 2023-06-22T08:24:21.976 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-userspec.c' 2023-06-22T08:24:21.976 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-isnand.h' 2023-06-22T08:24:21.976 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freopen.c' 2023-06-22T08:24:21.976 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-file-has-acl.sh' 2023-06-22T08:24:21.976 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbrtowc4.sh' 2023-06-22T08:24:21.977 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsrtowcs2.sh' 2023-06-22T08:24:21.977 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fseeko.sh' 2023-06-22T08:24:21.977 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/nan.h' 2023-06-22T08:24:21.977 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sys_ioctl.c' 2023-06-22T08:24:21.977 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-btowc.c' 2023-06-22T08:24:21.977 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getopt.h' 2023-06-22T08:24:21.977 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mkdirat.c' 2023-06-22T08:24:21.977 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-stdint.c' 2023-06-22T08:24:21.978 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-c-strcasecmp.c' 2023-06-22T08:24:21.978 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsstr3.sh' 2023-06-22T08:24:21.978 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-readlink.h' 2023-06-22T08:24:21.978 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbrtowc1.sh' 2023-06-22T08:24:21.978 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-exclude.c' 2023-06-22T08:24:21.978 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freadptr.c' 2023-06-22T08:24:21.978 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fflush2.sh' 2023-06-22T08:24:21.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-inttypes.c' 2023-06-22T08:24:21.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sys_utsname.c' 2023-06-22T08:24:21.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-math.c' 2023-06-22T08:24:21.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/signature.h' 2023-06-22T08:24:21.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/w32sock.h' 2023-06-22T08:24:21.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-wchar.c' 2023-06-22T08:24:21.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/uniwidth/test-uc_width2.c' 2023-06-22T08:24:21.979 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/uniwidth/test-uc_width.c' 2023-06-22T08:24:21.980 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/uniwidth/test-uc_width2.sh' 2023-06-22T08:24:21.980 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/gnulib-tests/uniwidth' 2023-06-22T08:24:21.980 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-lstat.c' 2023-06-22T08:24:21.980 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/unlinkdir.h' 2023-06-22T08:24:21.980 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbscasecmp.c' 2023-06-22T08:24:21.980 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-netdb.c' 2023-06-22T08:24:21.981 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-stat.c' 2023-06-22T08:24:21.981 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-open.c' 2023-06-22T08:24:21.981 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-strsignal.c' 2023-06-22T08:24:21.981 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/accept.c' 2023-06-22T08:24:21.981 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xfprintf-posix.c' 2023-06-22T08:24:21.981 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sockets.c' 2023-06-22T08:24:21.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xalloc-die.c' 2023-06-22T08:24:21.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fchdir.c' 2023-06-22T08:24:21.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sigaction.c' 2023-06-22T08:24:21.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-i-ring.c' 2023-06-22T08:24:21.982 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-read-file.c' 2023-06-22T08:24:21.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-unlinkat.c' 2023-06-22T08:24:21.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-stddef.c' 2023-06-22T08:24:21.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-linkat.c' 2023-06-22T08:24:21.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-openat-safer.c' 2023-06-22T08:24:21.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-gettimeofday.c' 2023-06-22T08:24:21.983 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-remove.c' 2023-06-22T08:24:21.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-strtod.c' 2023-06-22T08:24:21.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-unlink.h' 2023-06-22T08:24:21.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-exclude6.sh' 2023-06-22T08:24:21.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-argv-iter.c' 2023-06-22T08:24:21.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-lstat.h' 2023-06-22T08:24:21.984 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/unistr/test-u8-uctomb.c' 2023-06-22T08:24:21.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/gnulib-tests/unistr' 2023-06-22T08:24:21.985 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/setsockopt.c' 2023-06-22T08:24:22.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-exclude3.sh' 2023-06-22T08:24:22.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/ioctl.c' 2023-06-22T08:24:22.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/test-quotearg.po' 2023-06-22T08:24:22.414 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/test-quotearg.mo' 2023-06-22T08:24:22.415 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES' 2023-06-22T08:24:22.415 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/gnulib-tests/locale/fr' 2023-06-22T08:24:22.415 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/gnulib-tests/locale' 2023-06-22T08:24:22.415 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-c-strncasecmp.c' 2023-06-22T08:24:22.415 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fpurge.c' 2023-06-22T08:24:22.415 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-exclude5.sh' 2023-06-22T08:24:22.415 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-arpa_inet.c' 2023-06-22T08:24:22.416 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fopen-safer.c' 2023-06-22T08:24:22.416 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-nl_langinfo.c' 2023-06-22T08:24:22.416 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-time.c' 2023-06-22T08:24:22.416 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-netinet_in.c' 2023-06-22T08:24:22.416 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fopen.c' 2023-06-22T08:24:22.416 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fnmatch.c' 2023-06-22T08:24:22.416 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/bind.c' 2023-06-22T08:24:22.417 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freadahead.sh' 2023-06-22T08:24:22.417 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-lutimens.h' 2023-06-22T08:24:22.417 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getndelim2.c' 2023-06-22T08:24:22.417 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-printf-posix.h' 2023-06-22T08:24:22.417 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xprintf-posix.c' 2023-06-22T08:24:22.417 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-environ.c' 2023-06-22T08:24:22.417 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-uname.c' 2023-06-22T08:24:22.418 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/glthread/thread.h' 2023-06-22T08:24:22.418 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/glthread/thread.c' 2023-06-22T08:24:22.418 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/glthread/yield.h' 2023-06-22T08:24:22.418 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/gnulib-tests/glthread' 2023-06-22T08:24:22.418 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/gnulib.mk' 2023-06-22T08:24:22.418 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-areadlink.c' 2023-06-22T08:24:22.418 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-link.h' 2023-06-22T08:24:22.419 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-nl_langinfo.sh' 2023-06-22T08:24:22.419 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-quotearg.c' 2023-06-22T08:24:22.419 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-vasnprintf.c' 2023-06-22T08:24:22.419 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-atexit.sh' 2023-06-22T08:24:22.419 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/sleep.c' 2023-06-22T08:24:22.419 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fpending.c' 2023-06-22T08:24:22.419 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-isblank.c' 2023-06-22T08:24:22.420 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xalloc-die.sh' 2023-06-22T08:24:22.420 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-wctype.c' 2023-06-22T08:24:22.420 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-utimens.h' 2023-06-22T08:24:22.420 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-file-has-acl.c' 2023-06-22T08:24:22.420 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-set-mode-acl.sh' 2023-06-22T08:24:22.420 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-btowc2.sh' 2023-06-22T08:24:22.421 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sys_time.c' 2023-06-22T08:24:22.421 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-areadlink.h' 2023-06-22T08:24:22.421 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freadptr2.sh' 2023-06-22T08:24:22.421 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsrtowcs3.sh' 2023-06-22T08:24:22.421 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-vfprintf-posix.c' 2023-06-22T08:24:22.421 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fsync.c' 2023-06-22T08:24:22.421 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-vc-list-files-cvs.sh' 2023-06-22T08:24:22.422 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-alloca-opt.c' 2023-06-22T08:24:22.422 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-yesno.sh' 2023-06-22T08:24:22.422 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-memrchr.c' 2023-06-22T08:24:22.422 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-c-strcase.sh' 2023-06-22T08:24:22.422 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-select-out.sh' 2023-06-22T08:24:22.422 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-binary-io.sh' 2023-06-22T08:24:22.422 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-wcrtomb.sh' 2023-06-22T08:24:22.423 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-ftello2.sh' 2023-06-22T08:24:22.423 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mkdir.h' 2023-06-22T08:24:22.423 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xstrtoul.c' 2023-06-22T08:24:22.423 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-printf-frexp.c' 2023-06-22T08:24:22.423 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-snprintf.c' 2023-06-22T08:24:22.423 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-isnand-nolibm.c' 2023-06-22T08:24:22.423 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-readlink.c' 2023-06-22T08:24:22.423 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sleep.c' 2023-06-22T08:24:22.424 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/usleep.c' 2023-06-22T08:24:22.424 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xstrtoimax.sh' 2023-06-22T08:24:22.424 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-stdbool.c' 2023-06-22T08:24:22.424 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-frexpl.c' 2023-06-22T08:24:22.424 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-alignof.c' 2023-06-22T08:24:22.424 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fflush.c' 2023-06-22T08:24:22.424 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-ctype.c' 2023-06-22T08:24:22.425 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/socket.c' 2023-06-22T08:24:22.425 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-yesno.c' 2023-06-22T08:24:22.758 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/Makefile.in' 2023-06-22T08:24:22.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-filenamecat.c' 2023-06-22T08:24:22.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-link.c' 2023-06-22T08:24:22.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-nanosleep.c' 2023-06-22T08:24:22.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sys_socket.c' 2023-06-22T08:24:22.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsrtowcs4.sh' 2023-06-22T08:24:22.759 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-copy-acl.c' 2023-06-22T08:24:22.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-rmdir.h' 2023-06-22T08:24:22.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-isnanl.h' 2023-06-22T08:24:22.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-signal.c' 2023-06-22T08:24:22.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbrtowc2.sh' 2023-06-22T08:24:22.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-iconv-h.c' 2023-06-22T08:24:22.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-exclude2.sh' 2023-06-22T08:24:22.760 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-perror.sh' 2023-06-22T08:24:22.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-langinfo.c' 2023-06-22T08:24:22.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-vprintf-posix.c' 2023-06-22T08:24:22.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-rmdir.c' 2023-06-22T08:24:22.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fcntl.c' 2023-06-22T08:24:22.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-dirent.c' 2023-06-22T08:24:22.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-chown.h' 2023-06-22T08:24:22.761 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/read-file.c' 2023-06-22T08:24:22.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-usleep.c' 2023-06-22T08:24:22.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-base64.c' 2023-06-22T08:24:22.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-atexit.c' 2023-06-22T08:24:22.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-memchr2.c' 2023-06-22T08:24:22.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fdopendir.c' 2023-06-22T08:24:22.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-stdio.c' 2023-06-22T08:24:22.762 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-vfprintf-posix.sh' 2023-06-22T08:24:22.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getdtablesize.c' 2023-06-22T08:24:22.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xstrtoumax.c' 2023-06-22T08:24:22.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fpending.sh' 2023-06-22T08:24:22.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-version-etc.c' 2023-06-22T08:24:22.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getgroups.c' 2023-06-22T08:24:22.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freadptr2.c' 2023-06-22T08:24:22.763 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-select-fd.c' 2023-06-22T08:24:22.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fcntl-safer.c' 2023-06-22T08:24:22.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fflush2.c' 2023-06-22T08:24:22.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-select-in.sh' 2023-06-22T08:24:22.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-dup2.c' 2023-06-22T08:24:22.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-frexp.c' 2023-06-22T08:24:22.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsstr2.sh' 2023-06-22T08:24:22.764 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/Makefile.am' 2023-06-22T08:24:22.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-select-stdin.c' 2023-06-22T08:24:22.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getdelim.c' 2023-06-22T08:24:22.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-btowc1.sh' 2023-06-22T08:24:22.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-locale.c' 2023-06-22T08:24:22.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-md5.c' 2023-06-22T08:24:22.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-rename.h' 2023-06-22T08:24:22.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-utimens.c' 2023-06-22T08:24:22.765 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-strerror.c' 2023-06-22T08:24:22.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-inet_pton.c' 2023-06-22T08:24:22.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-priv-set.c' 2023-06-22T08:24:22.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-open.h' 2023-06-22T08:24:22.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-unlink.c' 2023-06-22T08:24:22.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/macros.h' 2023-06-22T08:24:22.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-rename.c' 2023-06-22T08:24:22.766 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xstrtoumax.sh' 2023-06-22T08:24:22.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/inet_pton.c' 2023-06-22T08:24:22.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-memcmp.c' 2023-06-22T08:24:22.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xvasprintf.c' 2023-06-22T08:24:22.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freadahead.c' 2023-06-22T08:24:22.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-printf-posix.output' 2023-06-22T08:24:22.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-openat.c' 2023-06-22T08:24:22.767 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-c-ctype.c' 2023-06-22T08:24:22.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-chown.c' 2023-06-22T08:24:22.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sys_stat.c' 2023-06-22T08:24:22.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-closein.sh' 2023-06-22T08:24:22.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/init.sh' 2023-06-22T08:24:22.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-filevercmp.c' 2023-06-22T08:24:22.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getaddrinfo.c' 2023-06-22T08:24:22.768 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-areadlinkat.c' 2023-06-22T08:24:22.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-isnanl-nolibm.c' 2023-06-22T08:24:22.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-lchown.h' 2023-06-22T08:24:22.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getline.c' 2023-06-22T08:24:22.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-hash.c' 2023-06-22T08:24:22.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-canonicalize.c' 2023-06-22T08:24:22.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fchownat.c' 2023-06-22T08:24:22.769 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-striconv.c' 2023-06-22T08:24:23.371 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-binary-io.c' 2023-06-22T08:24:23.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-malloca.c' 2023-06-22T08:24:23.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freadseek.c' 2023-06-22T08:24:23.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-closein.c' 2023-06-22T08:24:23.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-unistd.c' 2023-06-22T08:24:23.372 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/listen.c' 2023-06-22T08:24:23.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-posixtm.c' 2023-06-22T08:24:23.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbrtowc3.sh' 2023-06-22T08:24:23.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsstr2.c' 2023-06-22T08:24:23.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/unlinkdir.c' 2023-06-22T08:24:23.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-version-etc.sh' 2023-06-22T08:24:23.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-lseek.sh' 2023-06-22T08:24:23.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/nap.h' 2023-06-22T08:24:23.373 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-gethostname.c' 2023-06-22T08:24:23.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freading.c' 2023-06-22T08:24:23.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-areadlink-with-size.c' 2023-06-22T08:24:23.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freadseek.sh' 2023-06-22T08:24:23.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-exclude7.sh' 2023-06-22T08:24:23.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsalign.c' 2023-06-22T08:24:23.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/wctob.c' 2023-06-22T08:24:23.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-quotearg.sh' 2023-06-22T08:24:23.374 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbscasecmp.sh' 2023-06-22T08:24:23.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/binary-io.h' 2023-06-22T08:24:23.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getopt.c' 2023-06-22T08:24:23.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-select.c' 2023-06-22T08:24:23.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-u64.c' 2023-06-22T08:24:23.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-stat.h' 2023-06-22T08:24:23.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-cloexec.c' 2023-06-22T08:24:23.375 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-freadptr.sh' 2023-06-22T08:24:23.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-lseek.c' 2023-06-22T08:24:23.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-perror.c' 2023-06-22T08:24:23.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-bitrotate.c' 2023-06-22T08:24:23.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-stat-time.c' 2023-06-22T08:24:23.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-ftello.c' 2023-06-22T08:24:23.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getcwd.c' 2023-06-22T08:24:23.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-symlink.h' 2023-06-22T08:24:23.376 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-inet_ntop.c' 2023-06-22T08:24:23.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/zerosize-ptr.h' 2023-06-22T08:24:23.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-sha1.c' 2023-06-22T08:24:23.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fcntl-h.c' 2023-06-22T08:24:23.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fseeko2.sh' 2023-06-22T08:24:23.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-printf-frexpl.c' 2023-06-22T08:24:23.377 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsrtowcs.c' 2023-06-22T08:24:23.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-futimens.h' 2023-06-22T08:24:23.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mkdir.c' 2023-06-22T08:24:23.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsrtowcs1.sh' 2023-06-22T08:24:23.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-exclude4.sh' 2023-06-22T08:24:23.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-exclude1.sh' 2023-06-22T08:24:23.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-symlink.c' 2023-06-22T08:24:23.378 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-unsetenv.c' 2023-06-22T08:24:23.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xstrtol.c' 2023-06-22T08:24:23.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsstr1.c' 2023-06-22T08:24:23.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-lchown.c' 2023-06-22T08:24:23.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-symlinkat.c' 2023-06-22T08:24:23.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-set-mode-acl.c' 2023-06-22T08:24:23.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xprintf-posix.sh' 2023-06-22T08:24:23.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-lock.c' 2023-06-22T08:24:23.379 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-copy-acl.sh' 2023-06-22T08:24:23.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/sys_ioctl.in.h' 2023-06-22T08:24:23.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/read-file.h' 2023-06-22T08:24:23.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fstatat.c' 2023-06-22T08:24:23.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-xstrtol.sh' 2023-06-22T08:24:23.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-isnanf.h' 2023-06-22T08:24:23.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getopt_long.h' 2023-06-22T08:24:23.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-memchr.c' 2023-06-22T08:24:23.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-fprintf-posix.h' 2023-06-22T08:24:23.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-setenv.c' 2023-06-22T08:24:23.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-iconv.c' 2023-06-22T08:24:23.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-getdate.c' 2023-06-22T08:24:23.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-dirent-safer.c' 2023-06-22T08:24:23.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-vasprintf-posix.c' 2023-06-22T08:24:23.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-errno.c' 2023-06-22T08:24:23.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-vc-list-files-git.sh' 2023-06-22T08:24:23.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-tls.c' 2023-06-22T08:24:23.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-mbsinit.c' 2023-06-22T08:24:23.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-string.c' 2023-06-22T08:24:23.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/perror.c' 2023-06-22T08:24:23.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/gnulib-tests/test-utimens-common.h' 2023-06-22T08:24:23.866 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/gnulib-tests' 2023-06-22T08:24:23.866 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/AUTHORS' 2023-06-22T08:24:23.867 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_po_check' 2023-06-22T08:24:23.867 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/dist-check.mk' 2023-06-22T08:24:23.867 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/ChangeLog-2008' 2023-06-22T08:24:23.867 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_trailing_blank' 2023-06-22T08:24:23.867 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_file_system' 2023-06-22T08:24:23.867 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/README' 2023-06-22T08:24:23.868 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/configure.ac' 2023-06-22T08:24:23.868 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_system_h_headers' 2023-06-22T08:24:23.868 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/THANKStt.in' 2023-06-22T08:24:23.868 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/old/textutils/ChangeLog' 2023-06-22T08:24:23.868 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/old/textutils/NEWS' 2023-06-22T08:24:23.868 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/old/textutils' 2023-06-22T08:24:23.868 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/old/sh-utils/ChangeLog' 2023-06-22T08:24:23.869 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/old/sh-utils/NEWS' 2023-06-22T08:24:23.869 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/old/sh-utils/ChangeLog.0' 2023-06-22T08:24:23.869 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/old/sh-utils' 2023-06-22T08:24:23.869 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/old/fileutils/ChangeLog' 2023-06-22T08:24:23.869 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/old/fileutils/NEWS' 2023-06-22T08:24:23.869 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/old/fileutils/ChangeLog-1997' 2023-06-22T08:24:23.869 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/old/fileutils' 2023-06-22T08:24:23.870 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/old' 2023-06-22T08:24:23.870 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_sun_os_names' 2023-06-22T08:24:23.870 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/THANKS-to-translators' 2023-06-22T08:24:23.870 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/gitlog-to-changelog' 2023-06-22T08:24:23.870 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/config.rpath' 2023-06-22T08:24:23.870 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/depcomp' 2023-06-22T08:24:23.870 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/gendocs.sh' 2023-06-22T08:24:23.871 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/compile' 2023-06-22T08:24:23.871 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/useless-if-before-free' 2023-06-22T08:24:23.871 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/do-release-commit-and-tag' 2023-06-22T08:24:23.871 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/vc-list-files' 2023-06-22T08:24:23.871 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/texinfo.tex' 2023-06-22T08:24:23.871 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/config.guess' 2023-06-22T08:24:23.871 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/update-copyright' 2023-06-22T08:24:23.872 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/announce-gen' 2023-06-22T08:24:23.872 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/ChangeLog-2007' 2023-06-22T08:24:23.872 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/cvsu' 2023-06-22T08:24:23.872 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/config.sub' 2023-06-22T08:24:23.872 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/install-sh' 2023-06-22T08:24:23.872 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/ylwrap' 2023-06-22T08:24:23.872 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/git-version-gen' 2023-06-22T08:24:23.873 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/arg-nonnull.h' 2023-06-22T08:24:23.873 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/mdate-sh' 2023-06-22T08:24:23.873 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/gnupload' 2023-06-22T08:24:23.873 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/unused-parameter.h' 2023-06-22T08:24:23.873 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/warn-on-use.h' 2023-06-22T08:24:23.873 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/missing' 2023-06-22T08:24:23.873 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/c++defs.h' 2023-06-22T08:24:23.874 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/build-aux/gnu-web-doc-update' 2023-06-22T08:24:23.874 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5/build-aux' 2023-06-22T08:24:23.874 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/configure' 2023-06-22T08:24:23.874 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.vg-suppressions' 2023-06-22T08:24:23.874 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_obsolete_symbols' 2023-06-22T08:24:23.874 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/.x-sc_prohibit_atoi_atof' 2023-06-22T08:24:23.874 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/COPYING' 2023-06-22T08:24:23.875 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'coreutils-8.5/cfg.mk' 2023-06-22T08:24:23.875 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'coreutils-8.5' 2023-06-22T08:24:23.875 INFO:tasks.workunit.client.0.smithi023.stderr:+ cp -av .snap/k . 2023-06-22T08:24:24.549 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k' -> './k' 2023-06-22T08:24:24.550 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils_8.5.orig.tar.gz' -> './k/coreutils_8.5.orig.tar.gz' 2023-06-22T08:24:24.550 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5' -> './k/coreutils-8.5' 2023-06-22T08:24:24.550 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/THANKS' -> './k/coreutils-8.5/THANKS' 2023-06-22T08:24:24.550 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/ABOUT-NLS' -> './k/coreutils-8.5/ABOUT-NLS' 2023-06-22T08:24:24.550 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_prohibit_strcmp' -> './k/coreutils-8.5/.x-sc_prohibit_strcmp' 2023-06-22T08:24:24.550 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_error_message_uppercase' -> './k/coreutils-8.5/.x-sc_error_message_uppercase' 2023-06-22T08:24:24.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_obsolete_symbols' -> './k/coreutils-8.5/.x-sc_obsolete_symbols' 2023-06-22T08:24:24.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/INSTALL' -> './k/coreutils-8.5/INSTALL' 2023-06-22T08:24:24.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/configure.ac' -> './k/coreutils-8.5/configure.ac' 2023-06-22T08:24:24.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_prohibit_tab_based_indentation' -> './k/coreutils-8.5/.x-sc_prohibit_tab_based_indentation' 2023-06-22T08:24:24.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/GNUmakefile' -> './k/coreutils-8.5/GNUmakefile' 2023-06-22T08:24:24.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/ChangeLog' -> './k/coreutils-8.5/ChangeLog' 2023-06-22T08:24:24.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/COPYING' -> './k/coreutils-8.5/COPYING' 2023-06-22T08:24:24.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/Makefile.am' -> './k/coreutils-8.5/Makefile.am' 2023-06-22T08:24:24.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_po_check' -> './k/coreutils-8.5/.x-sc_po_check' 2023-06-22T08:24:24.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/dist-check.mk' -> './k/coreutils-8.5/dist-check.mk' 2023-06-22T08:24:24.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_GPL_version' -> './k/coreutils-8.5/.x-sc_GPL_version' 2023-06-22T08:24:24.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_unmarked_diagnostics' -> './k/coreutils-8.5/.x-sc_unmarked_diagnostics' 2023-06-22T08:24:24.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.prev-version' -> './k/coreutils-8.5/.prev-version' 2023-06-22T08:24:24.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/bootstrap.conf' -> './k/coreutils-8.5/bootstrap.conf' 2023-06-22T08:24:24.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/ChangeLog-2005' -> './k/coreutils-8.5/ChangeLog-2005' 2023-06-22T08:24:24.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/aclocal.m4' -> './k/coreutils-8.5/aclocal.m4' 2023-06-22T08:24:24.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/AUTHORS' -> './k/coreutils-8.5/AUTHORS' 2023-06-22T08:24:24.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/cfg.mk' -> './k/coreutils-8.5/cfg.mk' 2023-06-22T08:24:24.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_file_system' -> './k/coreutils-8.5/.x-sc_file_system' 2023-06-22T08:24:24.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.version' -> './k/coreutils-8.5/.version' 2023-06-22T08:24:24.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/TODO' -> './k/coreutils-8.5/TODO' 2023-06-22T08:24:24.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_space_tab' -> './k/coreutils-8.5/.x-sc_space_tab' 2023-06-22T08:24:24.554 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_prohibit_empty_lines_at_EOF' -> './k/coreutils-8.5/.x-sc_prohibit_empty_lines_at_EOF' 2023-06-22T08:24:24.554 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/Makefile.in' -> './k/coreutils-8.5/Makefile.in' 2023-06-22T08:24:24.554 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/maint.mk' -> './k/coreutils-8.5/maint.mk' 2023-06-22T08:24:24.554 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old' -> './k/coreutils-8.5/old' 2023-06-22T08:24:24.554 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/textutils' -> './k/coreutils-8.5/old/textutils' 2023-06-22T08:24:24.554 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/textutils/NEWS' -> './k/coreutils-8.5/old/textutils/NEWS' 2023-06-22T08:24:24.554 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/textutils/ChangeLog' -> './k/coreutils-8.5/old/textutils/ChangeLog' 2023-06-22T08:24:24.554 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/fileutils' -> './k/coreutils-8.5/old/fileutils' 2023-06-22T08:24:24.555 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/fileutils/ChangeLog-1997' -> './k/coreutils-8.5/old/fileutils/ChangeLog-1997' 2023-06-22T08:24:24.555 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/fileutils/NEWS' -> './k/coreutils-8.5/old/fileutils/NEWS' 2023-06-22T08:24:24.555 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/fileutils/ChangeLog' -> './k/coreutils-8.5/old/fileutils/ChangeLog' 2023-06-22T08:24:24.555 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/sh-utils' -> './k/coreutils-8.5/old/sh-utils' 2023-06-22T08:24:24.555 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/sh-utils/NEWS' -> './k/coreutils-8.5/old/sh-utils/NEWS' 2023-06-22T08:24:24.555 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/sh-utils/ChangeLog.0' -> './k/coreutils-8.5/old/sh-utils/ChangeLog.0' 2023-06-22T08:24:24.555 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/old/sh-utils/ChangeLog' -> './k/coreutils-8.5/old/sh-utils/ChangeLog' 2023-06-22T08:24:24.555 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux' -> './k/coreutils-8.5/build-aux' 2023-06-22T08:24:24.556 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/cvsu' -> './k/coreutils-8.5/build-aux/cvsu' 2023-06-22T08:24:24.556 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/mdate-sh' -> './k/coreutils-8.5/build-aux/mdate-sh' 2023-06-22T08:24:24.556 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/gitlog-to-changelog' -> './k/coreutils-8.5/build-aux/gitlog-to-changelog' 2023-06-22T08:24:24.556 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/gnu-web-doc-update' -> './k/coreutils-8.5/build-aux/gnu-web-doc-update' 2023-06-22T08:24:24.556 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/useless-if-before-free' -> './k/coreutils-8.5/build-aux/useless-if-before-free' 2023-06-22T08:24:24.556 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/config.sub' -> './k/coreutils-8.5/build-aux/config.sub' 2023-06-22T08:24:24.887 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/missing' -> './k/coreutils-8.5/build-aux/missing' 2023-06-22T08:24:24.887 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/compile' -> './k/coreutils-8.5/build-aux/compile' 2023-06-22T08:24:24.887 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/install-sh' -> './k/coreutils-8.5/build-aux/install-sh' 2023-06-22T08:24:24.888 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/gendocs.sh' -> './k/coreutils-8.5/build-aux/gendocs.sh' 2023-06-22T08:24:24.888 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/update-copyright' -> './k/coreutils-8.5/build-aux/update-copyright' 2023-06-22T08:24:24.888 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/texinfo.tex' -> './k/coreutils-8.5/build-aux/texinfo.tex' 2023-06-22T08:24:24.888 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/unused-parameter.h' -> './k/coreutils-8.5/build-aux/unused-parameter.h' 2023-06-22T08:24:24.888 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/c++defs.h' -> './k/coreutils-8.5/build-aux/c++defs.h' 2023-06-22T08:24:24.888 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/config.guess' -> './k/coreutils-8.5/build-aux/config.guess' 2023-06-22T08:24:24.888 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/do-release-commit-and-tag' -> './k/coreutils-8.5/build-aux/do-release-commit-and-tag' 2023-06-22T08:24:24.889 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/git-version-gen' -> './k/coreutils-8.5/build-aux/git-version-gen' 2023-06-22T08:24:24.889 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/arg-nonnull.h' -> './k/coreutils-8.5/build-aux/arg-nonnull.h' 2023-06-22T08:24:24.889 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/gnupload' -> './k/coreutils-8.5/build-aux/gnupload' 2023-06-22T08:24:24.889 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/warn-on-use.h' -> './k/coreutils-8.5/build-aux/warn-on-use.h' 2023-06-22T08:24:24.889 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/vc-list-files' -> './k/coreutils-8.5/build-aux/vc-list-files' 2023-06-22T08:24:24.889 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/ylwrap' -> './k/coreutils-8.5/build-aux/ylwrap' 2023-06-22T08:24:24.889 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/ChangeLog-2007' -> './k/coreutils-8.5/build-aux/ChangeLog-2007' 2023-06-22T08:24:24.890 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/announce-gen' -> './k/coreutils-8.5/build-aux/announce-gen' 2023-06-22T08:24:24.890 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/depcomp' -> './k/coreutils-8.5/build-aux/depcomp' 2023-06-22T08:24:24.890 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/build-aux/config.rpath' -> './k/coreutils-8.5/build-aux/config.rpath' 2023-06-22T08:24:24.890 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gl' -> './k/coreutils-8.5/gl' 2023-06-22T08:24:24.890 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gl/modules' -> './k/coreutils-8.5/gl/modules' 2023-06-22T08:24:24.890 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gl/modules/getloadavg.diff' -> './k/coreutils-8.5/gl/modules/getloadavg.diff' 2023-06-22T08:24:24.890 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4' -> './k/coreutils-8.5/m4' 2023-06-22T08:24:24.891 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/dirent-safer.m4' -> './k/coreutils-8.5/m4/dirent-safer.m4' 2023-06-22T08:24:24.891 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/settime.m4' -> './k/coreutils-8.5/m4/settime.m4' 2023-06-22T08:24:24.891 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/readtokens.m4' -> './k/coreutils-8.5/m4/readtokens.m4' 2023-06-22T08:24:24.891 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/clock_time.m4' -> './k/coreutils-8.5/m4/clock_time.m4' 2023-06-22T08:24:24.891 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/config-h.m4' -> './k/coreutils-8.5/m4/config-h.m4' 2023-06-22T08:24:24.891 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/euidaccess.m4' -> './k/coreutils-8.5/m4/euidaccess.m4' 2023-06-22T08:24:24.891 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/printf-frexpl.m4' -> './k/coreutils-8.5/m4/printf-frexpl.m4' 2023-06-22T08:24:24.892 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/posix-shell.m4' -> './k/coreutils-8.5/m4/posix-shell.m4' 2023-06-22T08:24:24.892 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/servent.m4' -> './k/coreutils-8.5/m4/servent.m4' 2023-06-22T08:24:24.892 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/xstrtol.m4' -> './k/coreutils-8.5/m4/xstrtol.m4' 2023-06-22T08:24:24.892 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mkdir-p.m4' -> './k/coreutils-8.5/m4/mkdir-p.m4' 2023-06-22T08:24:24.892 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/00gnulib.m4' -> './k/coreutils-8.5/m4/00gnulib.m4' 2023-06-22T08:24:24.892 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mbsinit.m4' -> './k/coreutils-8.5/m4/mbsinit.m4' 2023-06-22T08:24:24.892 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/dup2.m4' -> './k/coreutils-8.5/m4/dup2.m4' 2023-06-22T08:24:24.892 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/eealloc.m4' -> './k/coreutils-8.5/m4/eealloc.m4' 2023-06-22T08:24:24.893 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/savewd.m4' -> './k/coreutils-8.5/m4/savewd.m4' 2023-06-22T08:24:24.893 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/memrchr.m4' -> './k/coreutils-8.5/m4/memrchr.m4' 2023-06-22T08:24:24.893 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/d-ino.m4' -> './k/coreutils-8.5/m4/d-ino.m4' 2023-06-22T08:24:24.893 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/bison.m4' -> './k/coreutils-8.5/m4/bison.m4' 2023-06-22T08:24:24.893 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/isnanl.m4' -> './k/coreutils-8.5/m4/isnanl.m4' 2023-06-22T08:24:24.893 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/lib-ld.m4' -> './k/coreutils-8.5/m4/lib-ld.m4' 2023-06-22T08:24:24.894 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stpcpy.m4' -> './k/coreutils-8.5/m4/stpcpy.m4' 2023-06-22T08:24:24.894 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/progtest.m4' -> './k/coreutils-8.5/m4/progtest.m4' 2023-06-22T08:24:24.894 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/unistd_h.m4' -> './k/coreutils-8.5/m4/unistd_h.m4' 2023-06-22T08:24:24.894 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/time_h.m4' -> './k/coreutils-8.5/m4/time_h.m4' 2023-06-22T08:24:24.894 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mkancesdirs.m4' -> './k/coreutils-8.5/m4/mkancesdirs.m4' 2023-06-22T08:24:24.946 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/signalblocking.m4' -> './k/coreutils-8.5/m4/signalblocking.m4' 2023-06-22T08:24:24.946 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/unlink.m4' -> './k/coreutils-8.5/m4/unlink.m4' 2023-06-22T08:24:24.946 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/backupfile.m4' -> './k/coreutils-8.5/m4/backupfile.m4' 2023-06-22T08:24:24.946 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/remove.m4' -> './k/coreutils-8.5/m4/remove.m4' 2023-06-22T08:24:24.946 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fts.m4' -> './k/coreutils-8.5/m4/fts.m4' 2023-06-22T08:24:24.946 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/xalloc.m4' -> './k/coreutils-8.5/m4/xalloc.m4' 2023-06-22T08:24:24.947 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/inet_ntop.m4' -> './k/coreutils-8.5/m4/inet_ntop.m4' 2023-06-22T08:24:24.947 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/iconv_open.m4' -> './k/coreutils-8.5/m4/iconv_open.m4' 2023-06-22T08:24:24.947 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/extensions.m4' -> './k/coreutils-8.5/m4/extensions.m4' 2023-06-22T08:24:24.947 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mbrtowc.m4' -> './k/coreutils-8.5/m4/mbrtowc.m4' 2023-06-22T08:24:24.947 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/check-decl.m4' -> './k/coreutils-8.5/m4/check-decl.m4' 2023-06-22T08:24:24.947 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/cycle-check.m4' -> './k/coreutils-8.5/m4/cycle-check.m4' 2023-06-22T08:24:24.947 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/signal_h.m4' -> './k/coreutils-8.5/m4/signal_h.m4' 2023-06-22T08:24:24.948 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sys_socket_h.m4' -> './k/coreutils-8.5/m4/sys_socket_h.m4' 2023-06-22T08:24:24.948 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fprintftime.m4' -> './k/coreutils-8.5/m4/fprintftime.m4' 2023-06-22T08:24:24.948 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/lib-check.m4' -> './k/coreutils-8.5/m4/lib-check.m4' 2023-06-22T08:24:24.948 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/isblank.m4' -> './k/coreutils-8.5/m4/isblank.m4' 2023-06-22T08:24:24.948 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/xstrtod.m4' -> './k/coreutils-8.5/m4/xstrtod.m4' 2023-06-22T08:24:24.948 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stdarg.m4' -> './k/coreutils-8.5/m4/stdarg.m4' 2023-06-22T08:24:24.948 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/selinux-selinux-h.m4' -> './k/coreutils-8.5/m4/selinux-selinux-h.m4' 2023-06-22T08:24:24.948 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fstypename.m4' -> './k/coreutils-8.5/m4/fstypename.m4' 2023-06-22T08:24:24.949 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/link.m4' -> './k/coreutils-8.5/m4/link.m4' 2023-06-22T08:24:24.949 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/md5.m4' -> './k/coreutils-8.5/m4/md5.m4' 2023-06-22T08:24:24.949 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/multiarch.m4' -> './k/coreutils-8.5/m4/multiarch.m4' 2023-06-22T08:24:24.949 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/ioctl.m4' -> './k/coreutils-8.5/m4/ioctl.m4' 2023-06-22T08:24:24.949 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/flexmember.m4' -> './k/coreutils-8.5/m4/flexmember.m4' 2023-06-22T08:24:24.949 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/locale-fr.m4' -> './k/coreutils-8.5/m4/locale-fr.m4' 2023-06-22T08:24:24.949 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fdopendir.m4' -> './k/coreutils-8.5/m4/fdopendir.m4' 2023-06-22T08:24:24.950 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/exponentf.m4' -> './k/coreutils-8.5/m4/exponentf.m4' 2023-06-22T08:24:24.950 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/utimecmp.m4' -> './k/coreutils-8.5/m4/utimecmp.m4' 2023-06-22T08:24:24.950 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/human.m4' -> './k/coreutils-8.5/m4/human.m4' 2023-06-22T08:24:24.950 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strerror.m4' -> './k/coreutils-8.5/m4/strerror.m4' 2023-06-22T08:24:24.950 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/socklen.m4' -> './k/coreutils-8.5/m4/socklen.m4' 2023-06-22T08:24:24.950 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/dirent_h.m4' -> './k/coreutils-8.5/m4/dirent_h.m4' 2023-06-22T08:24:24.950 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/locale-zh.m4' -> './k/coreutils-8.5/m4/locale-zh.m4' 2023-06-22T08:24:24.950 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getdate.m4' -> './k/coreutils-8.5/m4/getdate.m4' 2023-06-22T08:24:24.951 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/cloexec.m4' -> './k/coreutils-8.5/m4/cloexec.m4' 2023-06-22T08:24:24.951 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/i-ring.m4' -> './k/coreutils-8.5/m4/i-ring.m4' 2023-06-22T08:24:24.951 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/group-member.m4' -> './k/coreutils-8.5/m4/group-member.m4' 2023-06-22T08:24:24.951 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strtod.m4' -> './k/coreutils-8.5/m4/strtod.m4' 2023-06-22T08:24:24.951 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/chdir-long.m4' -> './k/coreutils-8.5/m4/chdir-long.m4' 2023-06-22T08:24:24.951 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strnlen.m4' -> './k/coreutils-8.5/m4/strnlen.m4' 2023-06-22T08:24:24.951 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strtoimax.m4' -> './k/coreutils-8.5/m4/strtoimax.m4' 2023-06-22T08:24:24.951 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getgroups.m4' -> './k/coreutils-8.5/m4/getgroups.m4' 2023-06-22T08:24:24.952 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/yield.m4' -> './k/coreutils-8.5/m4/yield.m4' 2023-06-22T08:24:24.952 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/write.m4' -> './k/coreutils-8.5/m4/write.m4' 2023-06-22T08:24:24.952 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/localcharset.m4' -> './k/coreutils-8.5/m4/localcharset.m4' 2023-06-22T08:24:24.952 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/usleep.m4' -> './k/coreutils-8.5/m4/usleep.m4' 2023-06-22T08:24:24.952 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/memmove.m4' -> './k/coreutils-8.5/m4/memmove.m4' 2023-06-22T08:24:24.952 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/close.m4' -> './k/coreutils-8.5/m4/close.m4' 2023-06-22T08:24:24.952 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/memcmp.m4' -> './k/coreutils-8.5/m4/memcmp.m4' 2023-06-22T08:24:24.952 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/autobuild.m4' -> './k/coreutils-8.5/m4/autobuild.m4' 2023-06-22T08:24:24.953 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fchdir.m4' -> './k/coreutils-8.5/m4/fchdir.m4' 2023-06-22T08:24:25.004 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fd-reopen.m4' -> './k/coreutils-8.5/m4/fd-reopen.m4' 2023-06-22T08:24:25.005 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/jm-winsz2.m4' -> './k/coreutils-8.5/m4/jm-winsz2.m4' 2023-06-22T08:24:25.005 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/posixver.m4' -> './k/coreutils-8.5/m4/posixver.m4' 2023-06-22T08:24:25.005 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/locale-ja.m4' -> './k/coreutils-8.5/m4/locale-ja.m4' 2023-06-22T08:24:25.005 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stat-prog.m4' -> './k/coreutils-8.5/m4/stat-prog.m4' 2023-06-22T08:24:25.005 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/wint_t.m4' -> './k/coreutils-8.5/m4/wint_t.m4' 2023-06-22T08:24:25.005 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/st_dm_mode.m4' -> './k/coreutils-8.5/m4/st_dm_mode.m4' 2023-06-22T08:24:25.006 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/tempname.m4' -> './k/coreutils-8.5/m4/tempname.m4' 2023-06-22T08:24:25.006 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/frexp.m4' -> './k/coreutils-8.5/m4/frexp.m4' 2023-06-22T08:24:25.006 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/dirfd.m4' -> './k/coreutils-8.5/m4/dirfd.m4' 2023-06-22T08:24:25.006 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sys_select_h.m4' -> './k/coreutils-8.5/m4/sys_select_h.m4' 2023-06-22T08:24:25.006 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/btowc.m4' -> './k/coreutils-8.5/m4/btowc.m4' 2023-06-22T08:24:25.006 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/filenamecat.m4' -> './k/coreutils-8.5/m4/filenamecat.m4' 2023-06-22T08:24:25.006 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/gmp.m4' -> './k/coreutils-8.5/m4/gmp.m4' 2023-06-22T08:24:25.006 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/isapipe.m4' -> './k/coreutils-8.5/m4/isapipe.m4' 2023-06-22T08:24:25.007 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/wctype_h.m4' -> './k/coreutils-8.5/m4/wctype_h.m4' 2023-06-22T08:24:25.007 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/ldexpl.m4' -> './k/coreutils-8.5/m4/ldexpl.m4' 2023-06-22T08:24:25.007 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getloadavg.m4' -> './k/coreutils-8.5/m4/getloadavg.m4' 2023-06-22T08:24:25.007 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mbchar.m4' -> './k/coreutils-8.5/m4/mbchar.m4' 2023-06-22T08:24:25.007 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/version-etc.m4' -> './k/coreutils-8.5/m4/version-etc.m4' 2023-06-22T08:24:25.007 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fcntl_h.m4' -> './k/coreutils-8.5/m4/fcntl_h.m4' 2023-06-22T08:24:25.007 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/gettime.m4' -> './k/coreutils-8.5/m4/gettime.m4' 2023-06-22T08:24:25.008 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/wchar_h.m4' -> './k/coreutils-8.5/m4/wchar_h.m4' 2023-06-22T08:24:25.008 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/vasprintf-posix.m4' -> './k/coreutils-8.5/m4/vasprintf-posix.m4' 2023-06-22T08:24:25.008 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/selinux-context-h.m4' -> './k/coreutils-8.5/m4/selinux-context-h.m4' 2023-06-22T08:24:25.008 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getpass.m4' -> './k/coreutils-8.5/m4/getpass.m4' 2023-06-22T08:24:25.008 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mempcpy.m4' -> './k/coreutils-8.5/m4/mempcpy.m4' 2023-06-22T08:24:25.008 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/save-cwd.m4' -> './k/coreutils-8.5/m4/save-cwd.m4' 2023-06-22T08:24:25.008 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fclose.m4' -> './k/coreutils-8.5/m4/fclose.m4' 2023-06-22T08:24:25.008 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/modechange.m4' -> './k/coreutils-8.5/m4/modechange.m4' 2023-06-22T08:24:25.009 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strstr.m4' -> './k/coreutils-8.5/m4/strstr.m4' 2023-06-22T08:24:25.009 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getdtablesize.m4' -> './k/coreutils-8.5/m4/getdtablesize.m4' 2023-06-22T08:24:25.009 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/wchar_t.m4' -> './k/coreutils-8.5/m4/wchar_t.m4' 2023-06-22T08:24:25.009 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mkdir.m4' -> './k/coreutils-8.5/m4/mkdir.m4' 2023-06-22T08:24:25.009 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/idcache.m4' -> './k/coreutils-8.5/m4/idcache.m4' 2023-06-22T08:24:25.009 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/tm_gmtoff.m4' -> './k/coreutils-8.5/m4/tm_gmtoff.m4' 2023-06-22T08:24:25.009 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/dirname.m4' -> './k/coreutils-8.5/m4/dirname.m4' 2023-06-22T08:24:25.009 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mbiter.m4' -> './k/coreutils-8.5/m4/mbiter.m4' 2023-06-22T08:24:25.010 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fcntl-safer.m4' -> './k/coreutils-8.5/m4/fcntl-safer.m4' 2023-06-22T08:24:25.010 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/manywarnings.m4' -> './k/coreutils-8.5/m4/manywarnings.m4' 2023-06-22T08:24:25.010 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/same.m4' -> './k/coreutils-8.5/m4/same.m4' 2023-06-22T08:24:25.010 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getdelim.m4' -> './k/coreutils-8.5/m4/getdelim.m4' 2023-06-22T08:24:25.010 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mountlist.m4' -> './k/coreutils-8.5/m4/mountlist.m4' 2023-06-22T08:24:25.010 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/lchmod.m4' -> './k/coreutils-8.5/m4/lchmod.m4' 2023-06-22T08:24:25.010 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/readlink.m4' -> './k/coreutils-8.5/m4/readlink.m4' 2023-06-22T08:24:25.010 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/gnu-make.m4' -> './k/coreutils-8.5/m4/gnu-make.m4' 2023-06-22T08:24:25.011 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/thread.m4' -> './k/coreutils-8.5/m4/thread.m4' 2023-06-22T08:24:25.011 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/hostent.m4' -> './k/coreutils-8.5/m4/hostent.m4' 2023-06-22T08:24:25.011 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/math_h.m4' -> './k/coreutils-8.5/m4/math_h.m4' 2023-06-22T08:24:25.011 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fpending.m4' -> './k/coreutils-8.5/m4/fpending.m4' 2023-06-22T08:24:25.011 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/xgetcwd.m4' -> './k/coreutils-8.5/m4/xgetcwd.m4' 2023-06-22T08:24:25.011 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/utimens.m4' -> './k/coreutils-8.5/m4/utimens.m4' 2023-06-22T08:24:25.011 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/exponentd.m4' -> './k/coreutils-8.5/m4/exponentd.m4' 2023-06-22T08:24:26.227 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getaddrinfo.m4' -> './k/coreutils-8.5/m4/getaddrinfo.m4' 2023-06-22T08:24:26.227 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sys_ioctl_h.m4' -> './k/coreutils-8.5/m4/sys_ioctl_h.m4' 2023-06-22T08:24:26.227 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stdlib_h.m4' -> './k/coreutils-8.5/m4/stdlib_h.m4' 2023-06-22T08:24:26.227 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mgetgroups.m4' -> './k/coreutils-8.5/m4/mgetgroups.m4' 2023-06-22T08:24:26.227 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/open.m4' -> './k/coreutils-8.5/m4/open.m4' 2023-06-22T08:24:26.228 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strpbrk.m4' -> './k/coreutils-8.5/m4/strpbrk.m4' 2023-06-22T08:24:26.228 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/ssize_t.m4' -> './k/coreutils-8.5/m4/ssize_t.m4' 2023-06-22T08:24:26.228 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/priv-set.m4' -> './k/coreutils-8.5/m4/priv-set.m4' 2023-06-22T08:24:26.228 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fcntl-o.m4' -> './k/coreutils-8.5/m4/fcntl-o.m4' 2023-06-22T08:24:26.228 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/inttostr.m4' -> './k/coreutils-8.5/m4/inttostr.m4' 2023-06-22T08:24:26.228 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/malloca.m4' -> './k/coreutils-8.5/m4/malloca.m4' 2023-06-22T08:24:26.228 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/unlocked-io.m4' -> './k/coreutils-8.5/m4/unlocked-io.m4' 2023-06-22T08:24:26.229 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getugroups.m4' -> './k/coreutils-8.5/m4/getugroups.m4' 2023-06-22T08:24:26.229 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sha1.m4' -> './k/coreutils-8.5/m4/sha1.m4' 2023-06-22T08:24:26.229 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fpurge.m4' -> './k/coreutils-8.5/m4/fpurge.m4' 2023-06-22T08:24:26.229 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/gnulib-comp.m4' -> './k/coreutils-8.5/m4/gnulib-comp.m4' 2023-06-22T08:24:26.229 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/wcrtomb.m4' -> './k/coreutils-8.5/m4/wcrtomb.m4' 2023-06-22T08:24:26.229 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/putenv.m4' -> './k/coreutils-8.5/m4/putenv.m4' 2023-06-22T08:24:26.229 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/lstat.m4' -> './k/coreutils-8.5/m4/lstat.m4' 2023-06-22T08:24:26.230 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strsignal.m4' -> './k/coreutils-8.5/m4/strsignal.m4' 2023-06-22T08:24:26.230 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mbslen.m4' -> './k/coreutils-8.5/m4/mbslen.m4' 2023-06-22T08:24:26.230 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/inttypes.m4' -> './k/coreutils-8.5/m4/inttypes.m4' 2023-06-22T08:24:26.230 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stdlib-safer.m4' -> './k/coreutils-8.5/m4/stdlib-safer.m4' 2023-06-22T08:24:26.230 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strtol.m4' -> './k/coreutils-8.5/m4/strtol.m4' 2023-06-22T08:24:26.230 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/lib-ignore.m4' -> './k/coreutils-8.5/m4/lib-ignore.m4' 2023-06-22T08:24:26.230 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/readutmp.m4' -> './k/coreutils-8.5/m4/readutmp.m4' 2023-06-22T08:24:26.230 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strtoll.m4' -> './k/coreutils-8.5/m4/strtoll.m4' 2023-06-22T08:24:26.231 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stdbool.m4' -> './k/coreutils-8.5/m4/stdbool.m4' 2023-06-22T08:24:26.231 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/setenv.m4' -> './k/coreutils-8.5/m4/setenv.m4' 2023-06-22T08:24:26.231 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/alloca.m4' -> './k/coreutils-8.5/m4/alloca.m4' 2023-06-22T08:24:26.231 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/size_max.m4' -> './k/coreutils-8.5/m4/size_max.m4' 2023-06-22T08:24:26.232 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/include_next.m4' -> './k/coreutils-8.5/m4/include_next.m4' 2023-06-22T08:24:26.232 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sys_stat_h.m4' -> './k/coreutils-8.5/m4/sys_stat_h.m4' 2023-06-22T08:24:26.232 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strndup.m4' -> './k/coreutils-8.5/m4/strndup.m4' 2023-06-22T08:24:26.232 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/link-follow.m4' -> './k/coreutils-8.5/m4/link-follow.m4' 2023-06-22T08:24:26.232 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/ctype.m4' -> './k/coreutils-8.5/m4/ctype.m4' 2023-06-22T08:24:26.232 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/timespec.m4' -> './k/coreutils-8.5/m4/timespec.m4' 2023-06-22T08:24:26.232 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getline.m4' -> './k/coreutils-8.5/m4/getline.m4' 2023-06-22T08:24:26.233 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/calloc.m4' -> './k/coreutils-8.5/m4/calloc.m4' 2023-06-22T08:24:26.233 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strftime.m4' -> './k/coreutils-8.5/m4/strftime.m4' 2023-06-22T08:24:26.233 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/arpa_inet_h.m4' -> './k/coreutils-8.5/m4/arpa_inet_h.m4' 2023-06-22T08:24:26.233 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sig2str.m4' -> './k/coreutils-8.5/m4/sig2str.m4' 2023-06-22T08:24:26.233 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/printf.m4' -> './k/coreutils-8.5/m4/printf.m4' 2023-06-22T08:24:26.233 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/environ.m4' -> './k/coreutils-8.5/m4/environ.m4' 2023-06-22T08:24:26.233 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/vprintf-posix.m4' -> './k/coreutils-8.5/m4/vprintf-posix.m4' 2023-06-22T08:24:26.233 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/intlmacosx.m4' -> './k/coreutils-8.5/m4/intlmacosx.m4' 2023-06-22T08:24:26.234 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/locale-tr.m4' -> './k/coreutils-8.5/m4/locale-tr.m4' 2023-06-22T08:24:26.234 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mktime.m4' -> './k/coreutils-8.5/m4/mktime.m4' 2023-06-22T08:24:26.234 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/iconv.m4' -> './k/coreutils-8.5/m4/iconv.m4' 2023-06-22T08:24:26.234 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/pathmax.m4' -> './k/coreutils-8.5/m4/pathmax.m4' 2023-06-22T08:24:26.234 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/printf-frexp.m4' -> './k/coreutils-8.5/m4/printf-frexp.m4' 2023-06-22T08:24:26.234 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/userspec.m4' -> './k/coreutils-8.5/m4/userspec.m4' 2023-06-22T08:24:26.234 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/host-os.m4' -> './k/coreutils-8.5/m4/host-os.m4' 2023-06-22T08:24:26.293 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/file-type.m4' -> './k/coreutils-8.5/m4/file-type.m4' 2023-06-22T08:24:26.293 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/ls-mntd-fs.m4' -> './k/coreutils-8.5/m4/ls-mntd-fs.m4' 2023-06-22T08:24:26.293 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sha512.m4' -> './k/coreutils-8.5/m4/sha512.m4' 2023-06-22T08:24:26.293 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stdio-safer.m4' -> './k/coreutils-8.5/m4/stdio-safer.m4' 2023-06-22T08:24:26.293 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/lock.m4' -> './k/coreutils-8.5/m4/lock.m4' 2023-06-22T08:24:26.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/c-strtod.m4' -> './k/coreutils-8.5/m4/c-strtod.m4' 2023-06-22T08:24:26.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/closein.m4' -> './k/coreutils-8.5/m4/closein.m4' 2023-06-22T08:24:26.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/frexpl.m4' -> './k/coreutils-8.5/m4/frexpl.m4' 2023-06-22T08:24:26.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/posixtm.m4' -> './k/coreutils-8.5/m4/posixtm.m4' 2023-06-22T08:24:26.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/rmdir.m4' -> './k/coreutils-8.5/m4/rmdir.m4' 2023-06-22T08:24:26.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/chown.m4' -> './k/coreutils-8.5/m4/chown.m4' 2023-06-22T08:24:26.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stdint.m4' -> './k/coreutils-8.5/m4/stdint.m4' 2023-06-22T08:24:26.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stddef_h.m4' -> './k/coreutils-8.5/m4/stddef_h.m4' 2023-06-22T08:24:26.295 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sigaction.m4' -> './k/coreutils-8.5/m4/sigaction.m4' 2023-06-22T08:24:26.295 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/error.m4' -> './k/coreutils-8.5/m4/error.m4' 2023-06-22T08:24:26.295 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/vasnprintf.m4' -> './k/coreutils-8.5/m4/vasnprintf.m4' 2023-06-22T08:24:26.295 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/utimbuf.m4' -> './k/coreutils-8.5/m4/utimbuf.m4' 2023-06-22T08:24:26.295 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sockets.m4' -> './k/coreutils-8.5/m4/sockets.m4' 2023-06-22T08:24:26.296 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/canonicalize.m4' -> './k/coreutils-8.5/m4/canonicalize.m4' 2023-06-22T08:24:26.296 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sys_time_h.m4' -> './k/coreutils-8.5/m4/sys_time_h.m4' 2023-06-22T08:24:26.296 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/unlink-busy.m4' -> './k/coreutils-8.5/m4/unlink-busy.m4' 2023-06-22T08:24:26.296 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/gettimeofday.m4' -> './k/coreutils-8.5/m4/gettimeofday.m4' 2023-06-22T08:24:26.296 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/write-any-file.m4' -> './k/coreutils-8.5/m4/write-any-file.m4' 2023-06-22T08:24:26.296 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strdup.m4' -> './k/coreutils-8.5/m4/strdup.m4' 2023-06-22T08:24:26.296 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/regex.m4' -> './k/coreutils-8.5/m4/regex.m4' 2023-06-22T08:24:26.297 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sleep.m4' -> './k/coreutils-8.5/m4/sleep.m4' 2023-06-22T08:24:26.297 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/float_h.m4' -> './k/coreutils-8.5/m4/float_h.m4' 2023-06-22T08:24:26.297 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/warn-on-use.m4' -> './k/coreutils-8.5/m4/warn-on-use.m4' 2023-06-22T08:24:26.297 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/atexit.m4' -> './k/coreutils-8.5/m4/atexit.m4' 2023-06-22T08:24:26.297 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getcwd-path-max.m4' -> './k/coreutils-8.5/m4/getcwd-path-max.m4' 2023-06-22T08:24:26.297 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fnmatch.m4' -> './k/coreutils-8.5/m4/fnmatch.m4' 2023-06-22T08:24:26.297 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strnumcmp.m4' -> './k/coreutils-8.5/m4/strnumcmp.m4' 2023-06-22T08:24:26.297 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/gnulib-common.m4' -> './k/coreutils-8.5/m4/gnulib-common.m4' 2023-06-22T08:24:26.298 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/codeset.m4' -> './k/coreutils-8.5/m4/codeset.m4' 2023-06-22T08:24:26.298 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/time_r.m4' -> './k/coreutils-8.5/m4/time_r.m4' 2023-06-22T08:24:26.298 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/memcoll.m4' -> './k/coreutils-8.5/m4/memcoll.m4' 2023-06-22T08:24:26.298 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/string_h.m4' -> './k/coreutils-8.5/m4/string_h.m4' 2023-06-22T08:24:26.298 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sys_utsname_h.m4' -> './k/coreutils-8.5/m4/sys_utsname_h.m4' 2023-06-22T08:24:26.298 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stdint_h.m4' -> './k/coreutils-8.5/m4/stdint_h.m4' 2023-06-22T08:24:26.298 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/nproc.m4' -> './k/coreutils-8.5/m4/nproc.m4' 2023-06-22T08:24:26.298 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/symlink.m4' -> './k/coreutils-8.5/m4/symlink.m4' 2023-06-22T08:24:26.299 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/rename.m4' -> './k/coreutils-8.5/m4/rename.m4' 2023-06-22T08:24:26.299 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/read-file.m4' -> './k/coreutils-8.5/m4/read-file.m4' 2023-06-22T08:24:26.299 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strtoumax.m4' -> './k/coreutils-8.5/m4/strtoumax.m4' 2023-06-22T08:24:26.299 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getopt.m4' -> './k/coreutils-8.5/m4/getopt.m4' 2023-06-22T08:24:26.299 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fopen.m4' -> './k/coreutils-8.5/m4/fopen.m4' 2023-06-22T08:24:26.299 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/ungetc.m4' -> './k/coreutils-8.5/m4/ungetc.m4' 2023-06-22T08:24:26.299 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/xsize.m4' -> './k/coreutils-8.5/m4/xsize.m4' 2023-06-22T08:24:26.300 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fflush.m4' -> './k/coreutils-8.5/m4/fflush.m4' 2023-06-22T08:24:26.300 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/acl.m4' -> './k/coreutils-8.5/m4/acl.m4' 2023-06-22T08:24:26.300 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fsusage.m4' -> './k/coreutils-8.5/m4/fsusage.m4' 2023-06-22T08:24:26.300 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/gethrxtime.m4' -> './k/coreutils-8.5/m4/gethrxtime.m4' 2023-06-22T08:24:26.300 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/nls.m4' -> './k/coreutils-8.5/m4/nls.m4' 2023-06-22T08:24:26.300 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/lib-link.m4' -> './k/coreutils-8.5/m4/lib-link.m4' 2023-06-22T08:24:26.354 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mpsort.m4' -> './k/coreutils-8.5/m4/mpsort.m4' 2023-06-22T08:24:26.354 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/netinet_in_h.m4' -> './k/coreutils-8.5/m4/netinet_in_h.m4' 2023-06-22T08:24:26.354 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fseeko.m4' -> './k/coreutils-8.5/m4/fseeko.m4' 2023-06-22T08:24:26.354 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/inet_pton.m4' -> './k/coreutils-8.5/m4/inet_pton.m4' 2023-06-22T08:24:26.355 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/yesno.m4' -> './k/coreutils-8.5/m4/yesno.m4' 2023-06-22T08:24:26.355 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/physmem.m4' -> './k/coreutils-8.5/m4/physmem.m4' 2023-06-22T08:24:26.355 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/intmax_t.m4' -> './k/coreutils-8.5/m4/intmax_t.m4' 2023-06-22T08:24:26.355 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/inttypes-pri.m4' -> './k/coreutils-8.5/m4/inttypes-pri.m4' 2023-06-22T08:24:26.355 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/malloc.m4' -> './k/coreutils-8.5/m4/malloc.m4' 2023-06-22T08:24:26.355 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/jm-winsz1.m4' -> './k/coreutils-8.5/m4/jm-winsz1.m4' 2023-06-22T08:24:26.355 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/close-stream.m4' -> './k/coreutils-8.5/m4/close-stream.m4' 2023-06-22T08:24:26.355 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/nocrash.m4' -> './k/coreutils-8.5/m4/nocrash.m4' 2023-06-22T08:24:26.356 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/realloc.m4' -> './k/coreutils-8.5/m4/realloc.m4' 2023-06-22T08:24:26.356 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/canon-host.m4' -> './k/coreutils-8.5/m4/canon-host.m4' 2023-06-22T08:24:26.356 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/memcpy.m4' -> './k/coreutils-8.5/m4/memcpy.m4' 2023-06-22T08:24:26.356 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/tls.m4' -> './k/coreutils-8.5/m4/tls.m4' 2023-06-22T08:24:26.356 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/perl.m4' -> './k/coreutils-8.5/m4/perl.m4' 2023-06-22T08:24:26.356 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/signbit.m4' -> './k/coreutils-8.5/m4/signbit.m4' 2023-06-22T08:24:26.356 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/memchr.m4' -> './k/coreutils-8.5/m4/memchr.m4' 2023-06-22T08:24:26.357 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/d-type.m4' -> './k/coreutils-8.5/m4/d-type.m4' 2023-06-22T08:24:26.357 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/lchown.m4' -> './k/coreutils-8.5/m4/lchown.m4' 2023-06-22T08:24:26.357 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/lib-prefix.m4' -> './k/coreutils-8.5/m4/lib-prefix.m4' 2023-06-22T08:24:26.357 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getcwd.m4' -> './k/coreutils-8.5/m4/getcwd.m4' 2023-06-22T08:24:26.357 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/locale_h.m4' -> './k/coreutils-8.5/m4/locale_h.m4' 2023-06-22T08:24:26.357 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/memxfrm.m4' -> './k/coreutils-8.5/m4/memxfrm.m4' 2023-06-22T08:24:26.357 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mbstate_t.m4' -> './k/coreutils-8.5/m4/mbstate_t.m4' 2023-06-22T08:24:26.357 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/filemode.m4' -> './k/coreutils-8.5/m4/filemode.m4' 2023-06-22T08:24:26.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/memset.m4' -> './k/coreutils-8.5/m4/memset.m4' 2023-06-22T08:24:26.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/gethostname.m4' -> './k/coreutils-8.5/m4/gethostname.m4' 2023-06-22T08:24:26.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/vasprintf.m4' -> './k/coreutils-8.5/m4/vasprintf.m4' 2023-06-22T08:24:26.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/unistd-safer.m4' -> './k/coreutils-8.5/m4/unistd-safer.m4' 2023-06-22T08:24:26.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/xfts.m4' -> './k/coreutils-8.5/m4/xfts.m4' 2023-06-22T08:24:26.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/uptime.m4' -> './k/coreutils-8.5/m4/uptime.m4' 2023-06-22T08:24:26.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/prereq.m4' -> './k/coreutils-8.5/m4/prereq.m4' 2023-06-22T08:24:26.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/symlinkat.m4' -> './k/coreutils-8.5/m4/symlinkat.m4' 2023-06-22T08:24:26.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getusershell.m4' -> './k/coreutils-8.5/m4/getusershell.m4' 2023-06-22T08:24:26.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/isnand.m4' -> './k/coreutils-8.5/m4/isnand.m4' 2023-06-22T08:24:26.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/select.m4' -> './k/coreutils-8.5/m4/select.m4' 2023-06-22T08:24:26.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/wctob.m4' -> './k/coreutils-8.5/m4/wctob.m4' 2023-06-22T08:24:26.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/double-slash-root.m4' -> './k/coreutils-8.5/m4/double-slash-root.m4' 2023-06-22T08:24:26.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/long-options.m4' -> './k/coreutils-8.5/m4/long-options.m4' 2023-06-22T08:24:26.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/glibc21.m4' -> './k/coreutils-8.5/m4/glibc21.m4' 2023-06-22T08:24:26.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stat-time.m4' -> './k/coreutils-8.5/m4/stat-time.m4' 2023-06-22T08:24:26.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/rpmatch.m4' -> './k/coreutils-8.5/m4/rpmatch.m4' 2023-06-22T08:24:26.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/xattr.m4' -> './k/coreutils-8.5/m4/xattr.m4' 2023-06-22T08:24:26.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/gettext.m4' -> './k/coreutils-8.5/m4/gettext.m4' 2023-06-22T08:24:26.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fsync.m4' -> './k/coreutils-8.5/m4/fsync.m4' 2023-06-22T08:24:26.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/exclude.m4' -> './k/coreutils-8.5/m4/exclude.m4' 2023-06-22T08:24:26.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/freopen.m4' -> './k/coreutils-8.5/m4/freopen.m4' 2023-06-22T08:24:26.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strtoul.m4' -> './k/coreutils-8.5/m4/strtoul.m4' 2023-06-22T08:24:26.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/netdb_h.m4' -> './k/coreutils-8.5/m4/netdb_h.m4' 2023-06-22T08:24:26.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/snprintf.m4' -> './k/coreutils-8.5/m4/snprintf.m4' 2023-06-22T08:24:26.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stpncpy.m4' -> './k/coreutils-8.5/m4/stpncpy.m4' 2023-06-22T08:24:26.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/xnanosleep.m4' -> './k/coreutils-8.5/m4/xnanosleep.m4' 2023-06-22T08:24:26.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/warnings.m4' -> './k/coreutils-8.5/m4/warnings.m4' 2023-06-22T08:24:26.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/lseek.m4' -> './k/coreutils-8.5/m4/lseek.m4' 2023-06-22T08:24:26.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/hard-locale.m4' -> './k/coreutils-8.5/m4/hard-locale.m4' 2023-06-22T08:24:26.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/include-exclude-prog.m4' -> './k/coreutils-8.5/m4/include-exclude-prog.m4' 2023-06-22T08:24:26.416 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mmap-anon.m4' -> './k/coreutils-8.5/m4/mmap-anon.m4' 2023-06-22T08:24:26.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mkstemp.m4' -> './k/coreutils-8.5/m4/mkstemp.m4' 2023-06-22T08:24:26.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/openat.m4' -> './k/coreutils-8.5/m4/openat.m4' 2023-06-22T08:24:26.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mbrlen.m4' -> './k/coreutils-8.5/m4/mbrlen.m4' 2023-06-22T08:24:26.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/jm-macros.m4' -> './k/coreutils-8.5/m4/jm-macros.m4' 2023-06-22T08:24:26.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/ftello.m4' -> './k/coreutils-8.5/m4/ftello.m4' 2023-06-22T08:24:26.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stdio_h.m4' -> './k/coreutils-8.5/m4/stdio_h.m4' 2023-06-22T08:24:26.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/exponentl.m4' -> './k/coreutils-8.5/m4/exponentl.m4' 2023-06-22T08:24:26.417 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getpagesize.m4' -> './k/coreutils-8.5/m4/getpagesize.m4' 2023-06-22T08:24:26.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/argmatch.m4' -> './k/coreutils-8.5/m4/argmatch.m4' 2023-06-22T08:24:26.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/perror.m4' -> './k/coreutils-8.5/m4/perror.m4' 2023-06-22T08:24:26.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/inline.m4' -> './k/coreutils-8.5/m4/inline.m4' 2023-06-22T08:24:26.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/inttypes_h.m4' -> './k/coreutils-8.5/m4/inttypes_h.m4' 2023-06-22T08:24:26.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/iconv_h.m4' -> './k/coreutils-8.5/m4/iconv_h.m4' 2023-06-22T08:24:26.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/base64.m4' -> './k/coreutils-8.5/m4/base64.m4' 2023-06-22T08:24:26.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/stat.m4' -> './k/coreutils-8.5/m4/stat.m4' 2023-06-22T08:24:26.418 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/tzset.m4' -> './k/coreutils-8.5/m4/tzset.m4' 2023-06-22T08:24:26.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/unicodeio.m4' -> './k/coreutils-8.5/m4/unicodeio.m4' 2023-06-22T08:24:26.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sha256.m4' -> './k/coreutils-8.5/m4/sha256.m4' 2023-06-22T08:24:26.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/errno_h.m4' -> './k/coreutils-8.5/m4/errno_h.m4' 2023-06-22T08:24:26.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/unlinkdir.m4' -> './k/coreutils-8.5/m4/unlinkdir.m4' 2023-06-22T08:24:26.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/boottime.m4' -> './k/coreutils-8.5/m4/boottime.m4' 2023-06-22T08:24:26.419 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/freading.m4' -> './k/coreutils-8.5/m4/freading.m4' 2023-06-22T08:24:26.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mode_t.m4' -> './k/coreutils-8.5/m4/mode_t.m4' 2023-06-22T08:24:26.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getndelim2.m4' -> './k/coreutils-8.5/m4/getndelim2.m4' 2023-06-22T08:24:26.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/xvasprintf.m4' -> './k/coreutils-8.5/m4/xvasprintf.m4' 2023-06-22T08:24:26.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mbsrtowcs.m4' -> './k/coreutils-8.5/m4/mbsrtowcs.m4' 2023-06-22T08:24:26.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/hash.m4' -> './k/coreutils-8.5/m4/hash.m4' 2023-06-22T08:24:26.420 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/safe-write.m4' -> './k/coreutils-8.5/m4/safe-write.m4' 2023-06-22T08:24:26.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/po.m4' -> './k/coreutils-8.5/m4/po.m4' 2023-06-22T08:24:26.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/isnanf.m4' -> './k/coreutils-8.5/m4/isnanf.m4' 2023-06-22T08:24:26.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/longlong.m4' -> './k/coreutils-8.5/m4/longlong.m4' 2023-06-22T08:24:26.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/xstrndup.m4' -> './k/coreutils-8.5/m4/xstrndup.m4' 2023-06-22T08:24:26.421 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/euidaccess-stat.m4' -> './k/coreutils-8.5/m4/euidaccess-stat.m4' 2023-06-22T08:24:26.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/getcwd-abort-bug.m4' -> './k/coreutils-8.5/m4/getcwd-abort-bug.m4' 2023-06-22T08:24:26.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/faccessat.m4' -> './k/coreutils-8.5/m4/faccessat.m4' 2023-06-22T08:24:26.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/nanosleep.m4' -> './k/coreutils-8.5/m4/nanosleep.m4' 2023-06-22T08:24:26.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/ChangeLog-2007' -> './k/coreutils-8.5/m4/ChangeLog-2007' 2023-06-22T08:24:26.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/utimes.m4' -> './k/coreutils-8.5/m4/utimes.m4' 2023-06-22T08:24:26.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fileblocks.m4' -> './k/coreutils-8.5/m4/fileblocks.m4' 2023-06-22T08:24:26.422 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/sockpfaf.m4' -> './k/coreutils-8.5/m4/sockpfaf.m4' 2023-06-22T08:24:26.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strtoull.m4' -> './k/coreutils-8.5/m4/strtoull.m4' 2023-06-22T08:24:26.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/assert.m4' -> './k/coreutils-8.5/m4/assert.m4' 2023-06-22T08:24:26.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/quotearg.m4' -> './k/coreutils-8.5/m4/quotearg.m4' 2023-06-22T08:24:26.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/uname.m4' -> './k/coreutils-8.5/m4/uname.m4' 2023-06-22T08:24:26.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/savedir.m4' -> './k/coreutils-8.5/m4/savedir.m4' 2023-06-22T08:24:26.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/threadlib.m4' -> './k/coreutils-8.5/m4/threadlib.m4' 2023-06-22T08:24:26.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/langinfo_h.m4' -> './k/coreutils-8.5/m4/langinfo_h.m4' 2023-06-22T08:24:26.423 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/linkat.m4' -> './k/coreutils-8.5/m4/linkat.m4' 2023-06-22T08:24:26.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fpieee.m4' -> './k/coreutils-8.5/m4/fpieee.m4' 2023-06-22T08:24:26.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/memcasecmp.m4' -> './k/coreutils-8.5/m4/memcasecmp.m4' 2023-06-22T08:24:26.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/vfprintf-posix.m4' -> './k/coreutils-8.5/m4/vfprintf-posix.m4' 2023-06-22T08:24:26.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/dos.m4' -> './k/coreutils-8.5/m4/dos.m4' 2023-06-22T08:24:26.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/root-dev-ino.m4' -> './k/coreutils-8.5/m4/root-dev-ino.m4' 2023-06-22T08:24:26.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/quote.m4' -> './k/coreutils-8.5/m4/quote.m4' 2023-06-22T08:24:26.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/mbswidth.m4' -> './k/coreutils-8.5/m4/mbswidth.m4' 2023-06-22T08:24:26.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/nl_langinfo.m4' -> './k/coreutils-8.5/m4/nl_langinfo.m4' 2023-06-22T08:24:26.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/fcntl.m4' -> './k/coreutils-8.5/m4/fcntl.m4' 2023-06-22T08:24:26.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/strcspn.m4' -> './k/coreutils-8.5/m4/strcspn.m4' 2023-06-22T08:24:26.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/safe-read.m4' -> './k/coreutils-8.5/m4/safe-read.m4' 2023-06-22T08:24:26.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/wcwidth.m4' -> './k/coreutils-8.5/m4/wcwidth.m4' 2023-06-22T08:24:26.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/m4/closeout.m4' -> './k/coreutils-8.5/m4/closeout.m4' 2023-06-22T08:24:26.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.tarball-version' -> './k/coreutils-8.5/.tarball-version' 2023-06-22T08:24:26.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/NEWS' -> './k/coreutils-8.5/NEWS' 2023-06-22T08:24:26.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_sun_os_names' -> './k/coreutils-8.5/.x-sc_sun_os_names' 2023-06-22T08:24:26.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/bootstrap' -> './k/coreutils-8.5/bootstrap' 2023-06-22T08:24:26.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man' -> './k/coreutils-8.5/man' 2023-06-22T08:24:26.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/date.x' -> './k/coreutils-8.5/man/date.x' 2023-06-22T08:24:26.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/basename.1' -> './k/coreutils-8.5/man/basename.1' 2023-06-22T08:24:26.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/pathchk.x' -> './k/coreutils-8.5/man/pathchk.x' 2023-06-22T08:24:26.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tail.x' -> './k/coreutils-8.5/man/tail.x' 2023-06-22T08:24:26.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/yes.x' -> './k/coreutils-8.5/man/yes.x' 2023-06-22T08:24:26.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/pinky.1' -> './k/coreutils-8.5/man/pinky.1' 2023-06-22T08:24:26.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/kill.x' -> './k/coreutils-8.5/man/kill.x' 2023-06-22T08:24:26.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/pathchk.1' -> './k/coreutils-8.5/man/pathchk.1' 2023-06-22T08:24:26.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sync.1' -> './k/coreutils-8.5/man/sync.1' 2023-06-22T08:24:26.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/dir.1' -> './k/coreutils-8.5/man/dir.1' 2023-06-22T08:24:26.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/mv.1' -> './k/coreutils-8.5/man/mv.1' 2023-06-22T08:24:26.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/rm.1' -> './k/coreutils-8.5/man/rm.1' 2023-06-22T08:24:26.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/dircolors.x' -> './k/coreutils-8.5/man/dircolors.x' 2023-06-22T08:24:26.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/rmdir.1' -> './k/coreutils-8.5/man/rmdir.1' 2023-06-22T08:24:26.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tty.1' -> './k/coreutils-8.5/man/tty.1' 2023-06-22T08:24:26.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/fold.x' -> './k/coreutils-8.5/man/fold.x' 2023-06-22T08:24:26.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/fold.1' -> './k/coreutils-8.5/man/fold.1' 2023-06-22T08:24:26.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/dir.x' -> './k/coreutils-8.5/man/dir.x' 2023-06-22T08:24:26.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/fmt.x' -> './k/coreutils-8.5/man/fmt.x' 2023-06-22T08:24:26.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/paste.1' -> './k/coreutils-8.5/man/paste.1' 2023-06-22T08:24:26.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/chmod.x' -> './k/coreutils-8.5/man/chmod.x' 2023-06-22T08:24:26.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/od.1' -> './k/coreutils-8.5/man/od.1' 2023-06-22T08:24:26.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/ptx.x' -> './k/coreutils-8.5/man/ptx.x' 2023-06-22T08:24:26.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/nice.1' -> './k/coreutils-8.5/man/nice.1' 2023-06-22T08:24:26.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/install.1' -> './k/coreutils-8.5/man/install.1' 2023-06-22T08:24:26.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/dd.x' -> './k/coreutils-8.5/man/dd.x' 2023-06-22T08:24:26.582 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/cksum.1' -> './k/coreutils-8.5/man/cksum.1' 2023-06-22T08:24:26.582 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/test.1' -> './k/coreutils-8.5/man/test.1' 2023-06-22T08:24:26.582 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/chgrp.1' -> './k/coreutils-8.5/man/chgrp.1' 2023-06-22T08:24:26.582 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/unlink.1' -> './k/coreutils-8.5/man/unlink.1' 2023-06-22T08:24:26.582 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/chmod.1' -> './k/coreutils-8.5/man/chmod.1' 2023-06-22T08:24:26.582 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/dd.1' -> './k/coreutils-8.5/man/dd.1' 2023-06-22T08:24:26.582 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/du.1' -> './k/coreutils-8.5/man/du.1' 2023-06-22T08:24:26.582 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tsort.1' -> './k/coreutils-8.5/man/tsort.1' 2023-06-22T08:24:26.583 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/ls.1' -> './k/coreutils-8.5/man/ls.1' 2023-06-22T08:24:26.583 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/split.x' -> './k/coreutils-8.5/man/split.x' 2023-06-22T08:24:26.583 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/nohup.x' -> './k/coreutils-8.5/man/nohup.x' 2023-06-22T08:24:26.583 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sha1sum.1' -> './k/coreutils-8.5/man/sha1sum.1' 2023-06-22T08:24:26.583 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tr.1' -> './k/coreutils-8.5/man/tr.1' 2023-06-22T08:24:26.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~244s 2023-06-22T08:24:26.645 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/head.x' -> './k/coreutils-8.5/man/head.x' 2023-06-22T08:24:26.645 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/false.1' -> './k/coreutils-8.5/man/false.1' 2023-06-22T08:24:26.645 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/pwd.x' -> './k/coreutils-8.5/man/pwd.x' 2023-06-22T08:24:26.645 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/logname.x' -> './k/coreutils-8.5/man/logname.x' 2023-06-22T08:24:26.645 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/paste.x' -> './k/coreutils-8.5/man/paste.x' 2023-06-22T08:24:26.645 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/nl.x' -> './k/coreutils-8.5/man/nl.x' 2023-06-22T08:24:26.646 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sha224sum.1' -> './k/coreutils-8.5/man/sha224sum.1' 2023-06-22T08:24:26.646 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/mkdir.1' -> './k/coreutils-8.5/man/mkdir.1' 2023-06-22T08:24:26.646 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/nl.1' -> './k/coreutils-8.5/man/nl.1' 2023-06-22T08:24:26.646 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sha256sum.1' -> './k/coreutils-8.5/man/sha256sum.1' 2023-06-22T08:24:26.646 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/printf.x' -> './k/coreutils-8.5/man/printf.x' 2023-06-22T08:24:26.646 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/md5sum.1' -> './k/coreutils-8.5/man/md5sum.1' 2023-06-22T08:24:26.646 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/Makefile.am' -> './k/coreutils-8.5/man/Makefile.am' 2023-06-22T08:24:26.646 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/dirname.1' -> './k/coreutils-8.5/man/dirname.1' 2023-06-22T08:24:26.647 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/shuf.x' -> './k/coreutils-8.5/man/shuf.x' 2023-06-22T08:24:26.647 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/echo.1' -> './k/coreutils-8.5/man/echo.1' 2023-06-22T08:24:26.647 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/chcon.x' -> './k/coreutils-8.5/man/chcon.x' 2023-06-22T08:24:26.647 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/chroot.1' -> './k/coreutils-8.5/man/chroot.1' 2023-06-22T08:24:26.647 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sha384sum.x' -> './k/coreutils-8.5/man/sha384sum.x' 2023-06-22T08:24:26.647 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/stdbuf.1' -> './k/coreutils-8.5/man/stdbuf.1' 2023-06-22T08:24:26.647 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/users.1' -> './k/coreutils-8.5/man/users.1' 2023-06-22T08:24:26.648 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/readlink.x' -> './k/coreutils-8.5/man/readlink.x' 2023-06-22T08:24:26.648 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/cut.x' -> './k/coreutils-8.5/man/cut.x' 2023-06-22T08:24:26.648 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/groups.1' -> './k/coreutils-8.5/man/groups.1' 2023-06-22T08:24:26.648 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/seq.1' -> './k/coreutils-8.5/man/seq.1' 2023-06-22T08:24:26.648 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/help2man' -> './k/coreutils-8.5/man/help2man' 2023-06-22T08:24:26.648 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/dirname.x' -> './k/coreutils-8.5/man/dirname.x' 2023-06-22T08:24:26.649 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/uname.x' -> './k/coreutils-8.5/man/uname.x' 2023-06-22T08:24:26.649 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/uptime.x' -> './k/coreutils-8.5/man/uptime.x' 2023-06-22T08:24:26.649 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/cut.1' -> './k/coreutils-8.5/man/cut.1' 2023-06-22T08:24:26.649 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/cksum.x' -> './k/coreutils-8.5/man/cksum.x' 2023-06-22T08:24:26.649 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/nohup.1' -> './k/coreutils-8.5/man/nohup.1' 2023-06-22T08:24:26.649 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/mv.x' -> './k/coreutils-8.5/man/mv.x' 2023-06-22T08:24:26.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sort.1' -> './k/coreutils-8.5/man/sort.1' 2023-06-22T08:24:26.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/chown.1' -> './k/coreutils-8.5/man/chown.1' 2023-06-22T08:24:26.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/pr.x' -> './k/coreutils-8.5/man/pr.x' 2023-06-22T08:24:26.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/true.x' -> './k/coreutils-8.5/man/true.x' 2023-06-22T08:24:26.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/base64.1' -> './k/coreutils-8.5/man/base64.1' 2023-06-22T08:24:26.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/factor.1' -> './k/coreutils-8.5/man/factor.1' 2023-06-22T08:24:26.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/ln.x' -> './k/coreutils-8.5/man/ln.x' 2023-06-22T08:24:26.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/head.1' -> './k/coreutils-8.5/man/head.1' 2023-06-22T08:24:26.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/unexpand.1' -> './k/coreutils-8.5/man/unexpand.1' 2023-06-22T08:24:26.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/df.x' -> './k/coreutils-8.5/man/df.x' 2023-06-22T08:24:26.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/join.x' -> './k/coreutils-8.5/man/join.x' 2023-06-22T08:24:26.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/du.x' -> './k/coreutils-8.5/man/du.x' 2023-06-22T08:24:26.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/dircolors.1' -> './k/coreutils-8.5/man/dircolors.1' 2023-06-22T08:24:26.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/uniq.x' -> './k/coreutils-8.5/man/uniq.x' 2023-06-22T08:24:26.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/split.1' -> './k/coreutils-8.5/man/split.1' 2023-06-22T08:24:26.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tty.x' -> './k/coreutils-8.5/man/tty.x' 2023-06-22T08:24:26.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/uname.1' -> './k/coreutils-8.5/man/uname.1' 2023-06-22T08:24:26.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/uptime.1' -> './k/coreutils-8.5/man/uptime.1' 2023-06-22T08:24:26.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sort.x' -> './k/coreutils-8.5/man/sort.x' 2023-06-22T08:24:26.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sha256sum.x' -> './k/coreutils-8.5/man/sha256sum.x' 2023-06-22T08:24:26.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sha512sum.x' -> './k/coreutils-8.5/man/sha512sum.x' 2023-06-22T08:24:26.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/nice.x' -> './k/coreutils-8.5/man/nice.x' 2023-06-22T08:24:26.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/pinky.x' -> './k/coreutils-8.5/man/pinky.x' 2023-06-22T08:24:26.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/fmt.1' -> './k/coreutils-8.5/man/fmt.1' 2023-06-22T08:24:26.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/expr.1' -> './k/coreutils-8.5/man/expr.1' 2023-06-22T08:24:27.188 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.389+0000 7f130d9ff700 1 Processor -- start 2023-06-22T08:24:27.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.389+0000 7f130d9ff700 1 -- start start 2023-06-22T08:24:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f130d9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f13080c97d0 0x7f1308159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f130d9ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f13080c45f0 0x7f13080c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f130d9ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13080c4fb0 con 0x7f1308152d70 2023-06-22T08:24:27.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f130d9ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13080c95e0 con 0x7f13080c45f0 2023-06-22T08:24:27.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f130d9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13080c9c00 con 0x7f13080c97d0 2023-06-22T08:24:27.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f13067fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f13080c97d0 0x7f1308159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f13077fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f13080c45f0 0x7f13080c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f13067fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f13080c97d0 0x7f1308159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:58722/0 (socket says 172.21.15.23:58722) 2023-06-22T08:24:27.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f13067fc700 1 -- 172.21.15.23:0/3603959864 learned_addr learned my addr 172.21.15.23:0/3603959864 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:27.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f13067fc700 1 -- 172.21.15.23:0/3603959864 >> v2:172.21.15.67:3300/0 conn(0x7f13080c45f0 msgr2=0x7f13080c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:27.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f13067fc700 1 --2- 172.21.15.23:0/3603959864 >> v2:172.21.15.67:3300/0 conn(0x7f13080c45f0 0x7f13080c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f13067fc700 1 -- 172.21.15.23:0/3603959864 >> v1:172.21.15.67:6789/0 conn(0x7f1308152d70 legacy=0x7f13080cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:27.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.391+0000 7f13067fc700 1 -- 172.21.15.23:0/3603959864 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130815bea0 con 0x7f13080c97d0 2023-06-22T08:24:27.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.392+0000 7f13067fc700 1 --2- 172.21.15.23:0/3603959864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f13080c97d0 0x7f1308159330 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0098f0 tx=0x7f12fc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b5658fa367f5c9c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:27.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.392+0000 7f1305ffb700 1 -- 172.21.15.23:0/3603959864 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc011830 con 0x7f13080c97d0 2023-06-22T08:24:27.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.392+0000 7f1305ffb700 1 -- 172.21.15.23:0/3603959864 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc011990 con 0x7f13080c97d0 2023-06-22T08:24:27.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.392+0000 7f1305ffb700 1 -- 172.21.15.23:0/3603959864 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc011d20 con 0x7f13080c97d0 2023-06-22T08:24:27.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.392+0000 7f130d9ff700 1 -- 172.21.15.23:0/3603959864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f13080c97d0 msgr2=0x7f1308159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:27.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.392+0000 7f130d9ff700 1 --2- 172.21.15.23:0/3603959864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f13080c97d0 0x7f1308159330 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0098f0 tx=0x7f12fc009c00 comp rx=0 tx=0).stop 2023-06-22T08:24:27.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.393+0000 7f130d9ff700 1 -- 172.21.15.23:0/3603959864 shutdown_connections 2023-06-22T08:24:27.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.393+0000 7f130d9ff700 1 --2- 172.21.15.23:0/3603959864 >> v2:172.21.15.67:3300/0 conn(0x7f13080c45f0 0x7f13080c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.393+0000 7f130d9ff700 1 --2- 172.21.15.23:0/3603959864 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f13080c97d0 0x7f1308159330 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.393+0000 7f130d9ff700 1 -- 172.21.15.23:0/3603959864 >> 172.21.15.23:0/3603959864 conn(0x7f130806ae30 msgr2=0x7f130806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:27.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.393+0000 7f130d9ff700 1 -- 172.21.15.23:0/3603959864 shutdown_connections 2023-06-22T08:24:27.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.393+0000 7f130d9ff700 1 -- 172.21.15.23:0/3603959864 wait complete. 2023-06-22T08:24:27.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.394+0000 7f130d9ff700 1 Processor -- start 2023-06-22T08:24:27.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.394+0000 7f130d9ff700 1 -- start start 2023-06-22T08:24:27.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.395+0000 7f130d9ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080c45f0 0x7f13081ee160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.395+0000 7f130d9ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13080c97d0 0x7f13081f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.395+0000 7f130d9ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1308152d70 0x7f1308148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.395+0000 7f130d9ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130814ee70 con 0x7f13080c97d0 2023-06-22T08:24:27.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.395+0000 7f130d9ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13081f16d0 con 0x7f13080c45f0 2023-06-22T08:24:27.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.395+0000 7f130d9ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1308150db0 con 0x7f1308152d70 2023-06-22T08:24:27.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f13067fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13080c97d0 0x7f13081f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f13067fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13080c97d0 0x7f13081f2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56502/0 (socket says 172.21.15.23:56502) 2023-06-22T08:24:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f13067fc700 1 -- 172.21.15.23:0/1909790250 learned_addr learned my addr 172.21.15.23:0/1909790250 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f1306ffd700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080c45f0 0x7f13081ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f13077fe700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1308152d70 0x7f1308148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f13067fc700 1 -- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1308152d70 msgr2=0x7f1308148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f13067fc700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1308152d70 0x7f1308148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f13067fc700 1 -- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080c45f0 msgr2=0x7f13081ee160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f13067fc700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080c45f0 0x7f13081ee160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f13067fc700 1 -- 172.21.15.23:0/1909790250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc009550 con 0x7f13080c97d0 2023-06-22T08:24:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f1306ffd700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080c45f0 0x7f13081ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:24:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.396+0000 7f13067fc700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13080c97d0 0x7f13081f2170 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f12fc000c00 tx=0x7f12fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.397+0000 7f12effff700 1 -- 172.21.15.23:0/1909790250 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc004480 con 0x7f13080c97d0 2023-06-22T08:24:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.397+0000 7f12effff700 1 -- 172.21.15.23:0/1909790250 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc01f070 con 0x7f13080c97d0 2023-06-22T08:24:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.397+0000 7f130d9ff700 1 -- 172.21.15.23:0/1909790250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1308151090 con 0x7f13080c97d0 2023-06-22T08:24:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.397+0000 7f12effff700 1 -- 172.21.15.23:0/1909790250 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0038c0 con 0x7f13080c97d0 2023-06-22T08:24:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.397+0000 7f130d9ff700 1 -- 172.21.15.23:0/1909790250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13081f1970 con 0x7f13080c97d0 2023-06-22T08:24:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.398+0000 7f130d9ff700 1 -- 172.21.15.23:0/1909790250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130804ea20 con 0x7f13080c97d0 2023-06-22T08:24:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.399+0000 7f12effff700 1 -- 172.21.15.23:0/1909790250 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f12fc021030 con 0x7f13080c97d0 2023-06-22T08:24:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.399+0000 7f12effff700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12f406ad10 0x7f12f406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.400+0000 7f12effff700 1 -- 172.21.15.23:0/1909790250 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f12fc08b060 con 0x7f13080c97d0 2023-06-22T08:24:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.400+0000 7f1306ffd700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12f406ad10 0x7f12f406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.401+0000 7f1306ffd700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12f406ad10 0x7f12f406d1c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f12f8007fd0 tx=0x7f12f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:27.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.405+0000 7f12effff700 1 -- 172.21.15.23:0/1909790250 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f12fc055920 con 0x7f13080c97d0 2023-06-22T08:24:27.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.662+0000 7f130d9ff700 1 -- 172.21.15.23:0/1909790250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13080c9e00 con 0x7f13080c97d0 2023-06-22T08:24:27.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.662+0000 7f12effff700 1 -- 172.21.15.23:0/1909790250 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5489 (secure 0 0 0) 0x7f12fc058f40 con 0x7f13080c97d0 2023-06-22T08:24:27.666 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:27.666 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6185,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:26.088347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:27.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 -- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12f406ad10 msgr2=0x7f12f406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:27.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12f406ad10 0x7f12f406d1c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f12f8007fd0 tx=0x7f12f8009040 comp rx=0 tx=0).stop 2023-06-22T08:24:27.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 -- 172.21.15.23:0/1909790250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13080c97d0 msgr2=0x7f13081f2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:27.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13080c97d0 0x7f13081f2170 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f12fc000c00 tx=0x7f12fc004750 comp rx=0 tx=0).stop 2023-06-22T08:24:27.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 -- 172.21.15.23:0/1909790250 shutdown_connections 2023-06-22T08:24:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1308152d70 0x7f1308148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f12f406ad10 0x7f12f406d1c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13080c45f0 0x7f13081ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 --2- 172.21.15.23:0/1909790250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13080c97d0 0x7f13081f2170 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 -- 172.21.15.23:0/1909790250 >> 172.21.15.23:0/1909790250 conn(0x7f130806ae30 msgr2=0x7f130814e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 -- 172.21.15.23:0/1909790250 shutdown_connections 2023-06-22T08:24:27.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.665+0000 7f130d9ff700 1 -- 172.21.15.23:0/1909790250 wait complete. 2023-06-22T08:24:27.671 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 191 2023-06-22T08:24:27.687 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=2) 2023-06-22T08:24:27.687 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-06-22T08:24:27.688 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:24:27.716 INFO:tasks.ceph.mds.e:Stopped 2023-06-22T08:24:27.716 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:27.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.952+0000 7f21ca3ff700 1 Processor -- start 2023-06-22T08:24:27.956 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.952+0000 7f21ca3ff700 1 -- start start 2023-06-22T08:24:27.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.953+0000 7f21ca3ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f21c40c9fd0 0x7f21c40cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.953+0000 7f21ca3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 0x7f21c40c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.953+0000 7f21ca3ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c415c370 con 0x7f21c4152ce0 2023-06-22T08:24:27.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.953+0000 7f21ca3ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c41530b0 con 0x7f21c40c9fd0 2023-06-22T08:24:27.957 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.953+0000 7f21ca3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c40cf6f0 con 0x7f21c40c40b0 2023-06-22T08:24:27.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c3fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f21c4152ce0 0x7f21c415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52848/0 (socket says 172.21.15.23:52848) 2023-06-22T08:24:27.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c3fff700 1 -- 172.21.15.23:0/4031725929 learned_addr learned my addr 172.21.15.23:0/4031725929 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:27.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c899c700 1 --2- 172.21.15.23:0/4031725929 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 0x7f21c40c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.958 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c37fe700 1 --2- 172.21.15.23:0/4031725929 >> v2:172.21.15.67:3300/0 conn(0x7f21c40c9fd0 0x7f21c40cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.959 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c2ffd700 1 -- 172.21.15.23:0/4031725929 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477799524 0 0) 0x7f21c415c370 con 0x7f21c4152ce0 2023-06-22T08:24:27.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c2ffd700 1 -- 172.21.15.23:0/4031725929 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21ac0035b0 con 0x7f21c4152ce0 2023-06-22T08:24:27.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c899c700 1 -- 172.21.15.23:0/4031725929 >> v2:172.21.15.67:3300/0 conn(0x7f21c40c9fd0 msgr2=0x7f21c40cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:27.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c899c700 1 --2- 172.21.15.23:0/4031725929 >> v2:172.21.15.67:3300/0 conn(0x7f21c40c9fd0 0x7f21c40cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c899c700 1 -- 172.21.15.23:0/4031725929 >> v1:172.21.15.23:6789/0 conn(0x7f21c4152ce0 legacy=0x7f21c415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:27.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c899c700 1 -- 172.21.15.23:0/4031725929 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c415bda0 con 0x7f21c40c40b0 2023-06-22T08:24:27.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.954+0000 7f21c37fe700 1 --2- 172.21.15.23:0/4031725929 >> v2:172.21.15.67:3300/0 conn(0x7f21c40c9fd0 0x7f21c40cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:24:27.960 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.955+0000 7f21c899c700 1 --2- 172.21.15.23:0/4031725929 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 0x7f21c40c4530 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009d90 tx=0x7f21bc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adb48e231aefbe7c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.955+0000 7f21c2ffd700 1 -- 172.21.15.23:0/4031725929 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc00e070 con 0x7f21c40c40b0 2023-06-22T08:24:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.955+0000 7f21c2ffd700 1 -- 172.21.15.23:0/4031725929 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc00bb20 con 0x7f21c40c40b0 2023-06-22T08:24:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.955+0000 7f21c2ffd700 1 -- 172.21.15.23:0/4031725929 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc012430 con 0x7f21c40c40b0 2023-06-22T08:24:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.955+0000 7f21ca3ff700 1 -- 172.21.15.23:0/4031725929 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 msgr2=0x7f21c40c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.955+0000 7f21ca3ff700 1 --2- 172.21.15.23:0/4031725929 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 0x7f21c40c4530 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009d90 tx=0x7f21bc00b330 comp rx=0 tx=0).stop 2023-06-22T08:24:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.956+0000 7f21ca3ff700 1 -- 172.21.15.23:0/4031725929 shutdown_connections 2023-06-22T08:24:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.956+0000 7f21ca3ff700 1 --2- 172.21.15.23:0/4031725929 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 0x7f21c40c4530 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.961 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.956+0000 7f21ca3ff700 1 --2- 172.21.15.23:0/4031725929 >> v2:172.21.15.67:3300/0 conn(0x7f21c40c9fd0 0x7f21c40cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.956+0000 7f21ca3ff700 1 -- 172.21.15.23:0/4031725929 >> 172.21.15.23:0/4031725929 conn(0x7f21c406ae30 msgr2=0x7f21c406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:27.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.956+0000 7f21ca3ff700 1 -- 172.21.15.23:0/4031725929 shutdown_connections 2023-06-22T08:24:27.962 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.956+0000 7f21ca3ff700 1 -- 172.21.15.23:0/4031725929 wait complete. 2023-06-22T08:24:27.963 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.957+0000 7f21ca3ff700 1 Processor -- start 2023-06-22T08:24:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.958+0000 7f21ca3ff700 1 -- start start 2023-06-22T08:24:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.958+0000 7f21ca3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 0x7f21c4150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21ca3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21c40c9fd0 0x7f21c40c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21ca3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c4152ce0 0x7f21c4150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21ca3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c4150f70 con 0x7f21c40c9fd0 2023-06-22T08:24:27.964 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21ca3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c4150d90 con 0x7f21c4152ce0 2023-06-22T08:24:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21ca3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c414ee90 con 0x7f21c40c40b0 2023-06-22T08:24:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21c37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21c40c9fd0 0x7f21c40c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21c3fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 0x7f21c4150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21c37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21c40c9fd0 0x7f21c40c5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56510/0 (socket says 172.21.15.23:56510) 2023-06-22T08:24:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21c37fe700 1 -- 172.21.15.23:0/585095060 learned_addr learned my addr 172.21.15.23:0/585095060 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:27.965 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21c899c700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c4152ce0 0x7f21c4150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21c37fe700 1 -- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 msgr2=0x7f21c4150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21c37fe700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 0x7f21c4150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21c37fe700 1 -- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c4152ce0 msgr2=0x7f21c4150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21c37fe700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c4152ce0 0x7f21c4150850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.959+0000 7f21c37fe700 1 -- 172.21.15.23:0/585095060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b80094a0 con 0x7f21c40c9fd0 2023-06-22T08:24:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.960+0000 7f21c37fe700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21c40c9fd0 0x7f21c40c5c10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f21b800ebe0 tx=0x7f21b800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:27.966 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.960+0000 7f21c0ff9700 1 -- 172.21.15.23:0/585095060 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b800c850 con 0x7f21c40c9fd0 2023-06-22T08:24:27.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.960+0000 7f21c0ff9700 1 -- 172.21.15.23:0/585095060 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b800c9b0 con 0x7f21c40c9fd0 2023-06-22T08:24:27.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.960+0000 7f21ca3ff700 1 -- 172.21.15.23:0/585095060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21bc009a70 con 0x7f21c40c9fd0 2023-06-22T08:24:27.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.960+0000 7f21c0ff9700 1 -- 172.21.15.23:0/585095060 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b800cd40 con 0x7f21c40c9fd0 2023-06-22T08:24:27.967 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.960+0000 7f21ca3ff700 1 -- 172.21.15.23:0/585095060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c414fcb0 con 0x7f21c40c9fd0 2023-06-22T08:24:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.961+0000 7f21ca3ff700 1 -- 172.21.15.23:0/585095060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c414fe40 con 0x7f21c40c9fd0 2023-06-22T08:24:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.962+0000 7f21c0ff9700 1 -- 172.21.15.23:0/585095060 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f21b8013470 con 0x7f21c40c9fd0 2023-06-22T08:24:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.963+0000 7f21c0ff9700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f21b006ad10 0x7f21b006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.963+0000 7f21c0ff9700 1 -- 172.21.15.23:0/585095060 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f21b808a080 con 0x7f21c40c9fd0 2023-06-22T08:24:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.963+0000 7f21c3fff700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f21b006ad10 0x7f21b006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:27.968 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.964+0000 7f21c3fff700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f21b006ad10 0x7f21b006d1c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f21b4007fd0 tx=0x7f21b4007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:27.967+0000 7f21c0ff9700 1 -- 172.21.15.23:0/585095060 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f21b80bc310 con 0x7f21c40c9fd0 2023-06-22T08:24:28.172 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/wc.1' -> './k/coreutils-8.5/man/wc.1' 2023-06-22T08:24:28.173 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/true.1' -> './k/coreutils-8.5/man/true.1' 2023-06-22T08:24:28.173 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/comm.x' -> './k/coreutils-8.5/man/comm.x' 2023-06-22T08:24:28.173 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/truncate.x' -> './k/coreutils-8.5/man/truncate.x' 2023-06-22T08:24:28.173 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/install.x' -> './k/coreutils-8.5/man/install.x' 2023-06-22T08:24:28.173 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/wc.x' -> './k/coreutils-8.5/man/wc.x' 2023-06-22T08:24:28.173 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/uniq.1' -> './k/coreutils-8.5/man/uniq.1' 2023-06-22T08:24:28.174 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/arch.x' -> './k/coreutils-8.5/man/arch.x' 2023-06-22T08:24:28.174 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sum.x' -> './k/coreutils-8.5/man/sum.x' 2023-06-22T08:24:28.174 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/timeout.x' -> './k/coreutils-8.5/man/timeout.x' 2023-06-22T08:24:28.174 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/link.x' -> './k/coreutils-8.5/man/link.x' 2023-06-22T08:24:28.174 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tac.1' -> './k/coreutils-8.5/man/tac.1' 2023-06-22T08:24:28.174 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sync.x' -> './k/coreutils-8.5/man/sync.x' 2023-06-22T08:24:28.174 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/Makefile.in' -> './k/coreutils-8.5/man/Makefile.in' 2023-06-22T08:24:28.175 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/mkfifo.1' -> './k/coreutils-8.5/man/mkfifo.1' 2023-06-22T08:24:28.175 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/od.x' -> './k/coreutils-8.5/man/od.x' 2023-06-22T08:24:28.175 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/vdir.1' -> './k/coreutils-8.5/man/vdir.1' 2023-06-22T08:24:28.175 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/timeout.1' -> './k/coreutils-8.5/man/timeout.1' 2023-06-22T08:24:28.175 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/mkfifo.x' -> './k/coreutils-8.5/man/mkfifo.x' 2023-06-22T08:24:28.175 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/basename.x' -> './k/coreutils-8.5/man/basename.x' 2023-06-22T08:24:28.175 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/expr.x' -> './k/coreutils-8.5/man/expr.x' 2023-06-22T08:24:28.175 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/mktemp.x' -> './k/coreutils-8.5/man/mktemp.x' 2023-06-22T08:24:28.176 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/vdir.x' -> './k/coreutils-8.5/man/vdir.x' 2023-06-22T08:24:28.176 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/su.x' -> './k/coreutils-8.5/man/su.x' 2023-06-22T08:24:28.176 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/env.1' -> './k/coreutils-8.5/man/env.1' 2023-06-22T08:24:28.176 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/who.x' -> './k/coreutils-8.5/man/who.x' 2023-06-22T08:24:28.176 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/hostid.1' -> './k/coreutils-8.5/man/hostid.1' 2023-06-22T08:24:28.176 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tee.x' -> './k/coreutils-8.5/man/tee.x' 2023-06-22T08:24:28.176 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/rmdir.x' -> './k/coreutils-8.5/man/rmdir.x' 2023-06-22T08:24:28.177 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/hostid.x' -> './k/coreutils-8.5/man/hostid.x' 2023-06-22T08:24:28.177 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/id.x' -> './k/coreutils-8.5/man/id.x' 2023-06-22T08:24:28.177 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/unexpand.x' -> './k/coreutils-8.5/man/unexpand.x' 2023-06-22T08:24:28.177 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/pr.1' -> './k/coreutils-8.5/man/pr.1' 2023-06-22T08:24:28.177 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/csplit.1' -> './k/coreutils-8.5/man/csplit.1' 2023-06-22T08:24:28.177 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/truncate.1' -> './k/coreutils-8.5/man/truncate.1' 2023-06-22T08:24:28.177 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tee.1' -> './k/coreutils-8.5/man/tee.1' 2023-06-22T08:24:28.177 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/env.x' -> './k/coreutils-8.5/man/env.x' 2023-06-22T08:24:28.178 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/chroot.x' -> './k/coreutils-8.5/man/chroot.x' 2023-06-22T08:24:28.178 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/cp.x' -> './k/coreutils-8.5/man/cp.x' 2023-06-22T08:24:28.178 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/whoami.x' -> './k/coreutils-8.5/man/whoami.x' 2023-06-22T08:24:28.178 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/logname.1' -> './k/coreutils-8.5/man/logname.1' 2023-06-22T08:24:28.178 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/mknod.1' -> './k/coreutils-8.5/man/mknod.1' 2023-06-22T08:24:28.178 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/groups.x' -> './k/coreutils-8.5/man/groups.x' 2023-06-22T08:24:28.178 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/cp.1' -> './k/coreutils-8.5/man/cp.1' 2023-06-22T08:24:28.179 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/runcon.x' -> './k/coreutils-8.5/man/runcon.x' 2023-06-22T08:24:28.179 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/yes.1' -> './k/coreutils-8.5/man/yes.1' 2023-06-22T08:24:28.179 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/stat.1' -> './k/coreutils-8.5/man/stat.1' 2023-06-22T08:24:28.179 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/nproc.x' -> './k/coreutils-8.5/man/nproc.x' 2023-06-22T08:24:28.179 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/unlink.x' -> './k/coreutils-8.5/man/unlink.x' 2023-06-22T08:24:28.179 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sha384sum.1' -> './k/coreutils-8.5/man/sha384sum.1' 2023-06-22T08:24:28.179 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/nproc.1' -> './k/coreutils-8.5/man/nproc.1' 2023-06-22T08:24:28.179 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/seq.x' -> './k/coreutils-8.5/man/seq.x' 2023-06-22T08:24:28.180 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sha512sum.1' -> './k/coreutils-8.5/man/sha512sum.1' 2023-06-22T08:24:28.180 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/stat.x' -> './k/coreutils-8.5/man/stat.x' 2023-06-22T08:24:28.180 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sleep.x' -> './k/coreutils-8.5/man/sleep.x' 2023-06-22T08:24:28.180 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/printenv.1' -> './k/coreutils-8.5/man/printenv.1' 2023-06-22T08:24:28.180 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tr.x' -> './k/coreutils-8.5/man/tr.x' 2023-06-22T08:24:28.180 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tail.1' -> './k/coreutils-8.5/man/tail.1' 2023-06-22T08:24:28.258 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.255+0000 7f21ca3ff700 1 -- 172.21.15.23:0/585095060 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21c404ea20 con 0x7f21c40c9fd0 2023-06-22T08:24:28.259 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.256+0000 7f21c0ff9700 1 -- 172.21.15.23:0/585095060 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5489 (secure 0 0 0) 0x7f21b8017090 con 0x7f21c40c9fd0 2023-06-22T08:24:28.259 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:28.260 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6185,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:26.088347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:28.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 -- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f21b006ad10 msgr2=0x7f21b006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:28.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f21b006ad10 0x7f21b006d1c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f21b4007fd0 tx=0x7f21b4007dc0 comp rx=0 tx=0).stop 2023-06-22T08:24:28.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 -- 172.21.15.23:0/585095060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21c40c9fd0 msgr2=0x7f21c40c5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:28.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21c40c9fd0 0x7f21c40c5c10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f21b800ebe0 tx=0x7f21b800eef0 comp rx=0 tx=0).stop 2023-06-22T08:24:28.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 -- 172.21.15.23:0/585095060 shutdown_connections 2023-06-22T08:24:28.262 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f21b006ad10 0x7f21b006d1c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:28.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21c4152ce0 0x7f21c4150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:28.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f21c40c40b0 0x7f21c4150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:28.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 --2- 172.21.15.23:0/585095060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21c40c9fd0 0x7f21c40c5c10 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:28.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 -- 172.21.15.23:0/585095060 >> 172.21.15.23:0/585095060 conn(0x7f21c406ae30 msgr2=0x7f21c4151c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:28.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 -- 172.21.15.23:0/585095060 shutdown_connections 2023-06-22T08:24:28.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:28.258+0000 7f21ca3ff700 1 -- 172.21.15.23:0/585095060 wait complete. 2023-06-22T08:24:28.264 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 191 2023-06-22T08:24:28.281 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:24:30.282 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:30.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.482+0000 7f1a0bbff700 1 Processor -- start 2023-06-22T08:24:30.486 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.482+0000 7f1a0bbff700 1 -- start start 2023-06-22T08:24:30.487 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.484+0000 7f1a0bbff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a04152dd0 0x7f1a040aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:30.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.484+0000 7f1a0bbff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a040c7400 con 0x7f1a040c8bd0 2023-06-22T08:24:30.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.484+0000 7f1a0bbff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a040c81c0 con 0x7f1a040cc640 2023-06-22T08:24:30.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.484+0000 7f1a0bbff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a040c84a0 con 0x7f1a04152dd0 2023-06-22T08:24:30.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.484+0000 7f1a0919a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1a040c8bd0 0x7f1a0406afc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:52850/0 (socket says 172.21.15.23:52850) 2023-06-22T08:24:30.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.484+0000 7f1a0919a700 1 -- 172.21.15.23:0/2922585583 learned_addr learned my addr 172.21.15.23:0/2922585583 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:30.488 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.484+0000 7f1a0999b700 1 --2- 172.21.15.23:0/2922585583 >> v2:172.21.15.67:3300/0 conn(0x7f1a04152dd0 0x7f1a040aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:30.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.485+0000 7f1a08999700 1 -- 172.21.15.23:0/2922585583 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477384656 0 0) 0x7f1a040c81c0 con 0x7f1a040cc640 2023-06-22T08:24:30.490 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.485+0000 7f1a08999700 1 -- 172.21.15.23:0/2922585583 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f00035b0 con 0x7f1a040cc640 2023-06-22T08:24:30.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.485+0000 7f1a0999b700 1 -- 172.21.15.23:0/2922585583 >> v1:172.21.15.67:6789/0 conn(0x7f1a040cc640 legacy=0x7f1a04163280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:30.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.485+0000 7f1a0999b700 1 -- 172.21.15.23:0/2922585583 >> v1:172.21.15.23:6789/0 conn(0x7f1a040c8bd0 legacy=0x7f1a0406afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:30.491 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.485+0000 7f1a0999b700 1 -- 172.21.15.23:0/2922585583 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a040cd370 con 0x7f1a04152dd0 2023-06-22T08:24:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.485+0000 7f1a0999b700 1 --2- 172.21.15.23:0/2922585583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04152dd0 0x7f1a040aedc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009820 tx=0x7f1a00009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9ebff4701dcf204 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.485+0000 7f1a08999700 1 -- 172.21.15.23:0/2922585583 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00005e30 con 0x7f1a04152dd0 2023-06-22T08:24:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.485+0000 7f1a08999700 1 -- 172.21.15.23:0/2922585583 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a00006030 con 0x7f1a04152dd0 2023-06-22T08:24:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.485+0000 7f1a08999700 1 -- 172.21.15.23:0/2922585583 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a000063c0 con 0x7f1a04152dd0 2023-06-22T08:24:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.486+0000 7f1a0bbff700 1 -- 172.21.15.23:0/2922585583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04152dd0 msgr2=0x7f1a040aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.486+0000 7f1a0bbff700 1 --2- 172.21.15.23:0/2922585583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04152dd0 0x7f1a040aedc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009820 tx=0x7f1a00009b30 comp rx=0 tx=0).stop 2023-06-22T08:24:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.486+0000 7f1a0bbff700 1 -- 172.21.15.23:0/2922585583 shutdown_connections 2023-06-22T08:24:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.486+0000 7f1a0bbff700 1 --2- 172.21.15.23:0/2922585583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a04152dd0 0x7f1a040aedc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.486+0000 7f1a0bbff700 1 -- 172.21.15.23:0/2922585583 >> 172.21.15.23:0/2922585583 conn(0x7f1a0406fb70 msgr2=0x7f1a0406ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.486+0000 7f1a0bbff700 1 -- 172.21.15.23:0/2922585583 shutdown_connections 2023-06-22T08:24:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.486+0000 7f1a0bbff700 1 -- 172.21.15.23:0/2922585583 wait complete. 2023-06-22T08:24:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.488+0000 7f1a0bbff700 1 Processor -- start 2023-06-22T08:24:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.489+0000 7f1a0bbff700 1 -- start start 2023-06-22T08:24:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0bbff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a040c8bd0 0x7f1a040ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0bbff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1a040cc640 0x7f1a040af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0bbff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a04152dd0 0x7f1a040b1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0bbff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a040affe0 con 0x7f1a04152dd0 2023-06-22T08:24:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0bbff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a040b05f0 con 0x7f1a040c8bd0 2023-06-22T08:24:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0bbff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a040afd40 con 0x7f1a040cc640 2023-06-22T08:24:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0999b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a040c8bd0 0x7f1a040ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0919a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1a040cc640 0x7f1a040af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:30.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0a19c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a04152dd0 0x7f1a040b1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:30.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0999b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a040c8bd0 0x7f1a040ae470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:34862/0 (socket says 172.21.15.23:34862) 2023-06-22T08:24:30.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.490+0000 7f1a0999b700 1 -- 172.21.15.23:0/1184642379 learned_addr learned my addr 172.21.15.23:0/1184642379 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:30.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f1a0999b700 1 -- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1a040cc640 msgr2=0x7f1a040af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:30.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f1a0999b700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1a040cc640 0x7f1a040af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:30.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f1a0999b700 1 -- 172.21.15.23:0/1184642379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a04152dd0 msgr2=0x7f1a040b1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:30.497 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f1a0999b700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a04152dd0 0x7f1a040b1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:30.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f1a0999b700 1 -- 172.21.15.23:0/1184642379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a00009430 con 0x7f1a040c8bd0 2023-06-22T08:24:30.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f1a0a19c700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a04152dd0 0x7f1a040b1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:24:30.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f1a0999b700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a040c8bd0 0x7f1a040ae470 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1a00000c00 tx=0x7f1a00004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:30.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f19fa7fc700 1 -- 172.21.15.23:0/1184642379 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00004040 con 0x7f1a040c8bd0 2023-06-22T08:24:30.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f19fa7fc700 1 -- 172.21.15.23:0/1184642379 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a000041a0 con 0x7f1a040c8bd0 2023-06-22T08:24:30.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f19fa7fc700 1 -- 172.21.15.23:0/1184642379 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00004550 con 0x7f1a040c8bd0 2023-06-22T08:24:30.498 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.491+0000 7f1a0bbff700 1 -- 172.21.15.23:0/1184642379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a041f1440 con 0x7f1a040c8bd0 2023-06-22T08:24:30.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.492+0000 7f1a0bbff700 1 -- 172.21.15.23:0/1184642379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a041f19e0 con 0x7f1a040c8bd0 2023-06-22T08:24:30.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.493+0000 7f1a0bbff700 1 -- 172.21.15.23:0/1184642379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0415b3c0 con 0x7f1a040c8bd0 2023-06-22T08:24:30.499 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.494+0000 7f19fa7fc700 1 -- 172.21.15.23:0/1184642379 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f1a00022070 con 0x7f1a040c8bd0 2023-06-22T08:24:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.495+0000 7f19fa7fc700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f19e806ace0 0x7f19e806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.495+0000 7f19fa7fc700 1 -- 172.21.15.23:0/1184642379 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f1a0008b3c0 con 0x7f1a040c8bd0 2023-06-22T08:24:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.495+0000 7f1a0919a700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f19e806ace0 0x7f19e806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.496+0000 7f1a0919a700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f19e806ace0 0x7f19e806d190 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1a040aeb70 tx=0x7f19f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.500+0000 7f19fa7fc700 1 -- 172.21.15.23:0/1184642379 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f1a00055d10 con 0x7f1a040c8bd0 2023-06-22T08:24:30.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.755+0000 7f1a0bbff700 1 -- 172.21.15.23:0/1184642379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a0404ea20 con 0x7f1a040c8bd0 2023-06-22T08:24:30.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.756+0000 7f19fa7fc700 1 -- 172.21.15.23:0/1184642379 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5489 (secure 0 0 0) 0x7f1a0002a020 con 0x7f1a040c8bd0 2023-06-22T08:24:30.760 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:30.760 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6185,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:26.088347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:30.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.758+0000 7f1a0bbff700 1 -- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f19e806ace0 msgr2=0x7f19e806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:30.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.758+0000 7f1a0bbff700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f19e806ace0 0x7f19e806d190 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1a040aeb70 tx=0x7f19f4009040 comp rx=0 tx=0).stop 2023-06-22T08:24:30.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.758+0000 7f1a0bbff700 1 -- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a040c8bd0 msgr2=0x7f1a040ae470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:30.762 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.758+0000 7f1a0bbff700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a040c8bd0 0x7f1a040ae470 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1a00000c00 tx=0x7f1a00004750 comp rx=0 tx=0).stop 2023-06-22T08:24:30.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.758+0000 7f1a0bbff700 1 -- 172.21.15.23:0/1184642379 shutdown_connections 2023-06-22T08:24:30.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.758+0000 7f1a0bbff700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a04152dd0 0x7f1a040b1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:30.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.758+0000 7f1a0bbff700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f19e806ace0 0x7f19e806d190 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:30.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.758+0000 7f1a0bbff700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a040c8bd0 0x7f1a040ae470 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:30.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.758+0000 7f1a0bbff700 1 --2- 172.21.15.23:0/1184642379 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f1a040cc640 0x7f1a040af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:30.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.758+0000 7f1a0bbff700 1 -- 172.21.15.23:0/1184642379 >> 172.21.15.23:0/1184642379 conn(0x7f1a0406fb70 msgr2=0x7f1a040cb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:30.763 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.759+0000 7f1a0bbff700 1 -- 172.21.15.23:0/1184642379 shutdown_connections 2023-06-22T08:24:30.764 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:30.759+0000 7f1a0bbff700 1 -- 172.21.15.23:0/1184642379 wait complete. 2023-06-22T08:24:30.764 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 191 2023-06-22T08:24:30.780 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:24:32.781 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:32.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.992+0000 7f334a1ff700 1 Processor -- start 2023-06-22T08:24:32.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.992+0000 7f334a1ff700 1 -- start start 2023-06-22T08:24:32.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.994+0000 7f334a1ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3344152d70 0x7f33440c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.994+0000 7f334a1ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f33440c45f0 0x7f33440c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.994+0000 7f334a1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3344159790 con 0x7f3344159330 2023-06-22T08:24:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.994+0000 7f334a1ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33440c95e0 con 0x7f33440c45f0 2023-06-22T08:24:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.994+0000 7f334a1ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334415bea0 con 0x7f3344152d70 2023-06-22T08:24:32.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.994+0000 7f3343fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f33440c45f0 0x7f33440c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:32.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.994+0000 7f33437fe700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3344152d70 0x7f33440c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:32.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.994+0000 7f3342ffd700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3344159330 0x7f33440cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:60550/0 (socket says 172.21.15.23:60550) 2023-06-22T08:24:32.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.994+0000 7f3342ffd700 1 -- 172.21.15.23:0/873083729 learned_addr learned my addr 172.21.15.23:0/873083729 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.995+0000 7f3343fff700 1 -- 172.21.15.23:0/873083729 >> v2:172.21.15.67:3300/0 conn(0x7f3344152d70 msgr2=0x7f33440c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.995+0000 7f3343fff700 1 --2- 172.21.15.23:0/873083729 >> v2:172.21.15.67:3300/0 conn(0x7f3344152d70 0x7f33440c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.995+0000 7f3343fff700 1 -- 172.21.15.23:0/873083729 >> v1:172.21.15.67:6789/0 conn(0x7f3344159330 legacy=0x7f33440cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:33.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.995+0000 7f3343fff700 1 -- 172.21.15.23:0/873083729 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334415c160 con 0x7f33440c45f0 2023-06-22T08:24:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.995+0000 7f3343fff700 1 --2- 172.21.15.23:0/873083729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33440c45f0 0x7f33440c4a10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f33440c8ef0 tx=0x7f333c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df761b1e880829ab server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.995+0000 7f33427fc700 1 -- 172.21.15.23:0/873083729 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c00e070 con 0x7f33440c45f0 2023-06-22T08:24:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.995+0000 7f33427fc700 1 -- 172.21.15.23:0/873083729 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c00bb20 con 0x7f33440c45f0 2023-06-22T08:24:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.995+0000 7f33427fc700 1 -- 172.21.15.23:0/873083729 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c012430 con 0x7f33440c45f0 2023-06-22T08:24:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.996+0000 7f334a1ff700 1 -- 172.21.15.23:0/873083729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33440c45f0 msgr2=0x7f33440c4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:33.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.996+0000 7f334a1ff700 1 --2- 172.21.15.23:0/873083729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33440c45f0 0x7f33440c4a10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f33440c8ef0 tx=0x7f333c00b330 comp rx=0 tx=0).stop 2023-06-22T08:24:33.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.996+0000 7f334a1ff700 1 -- 172.21.15.23:0/873083729 shutdown_connections 2023-06-22T08:24:33.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.996+0000 7f334a1ff700 1 --2- 172.21.15.23:0/873083729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33440c45f0 0x7f33440c4a10 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:33.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.996+0000 7f334a1ff700 1 --2- 172.21.15.23:0/873083729 >> v2:172.21.15.67:3300/0 conn(0x7f3344152d70 0x7f33440c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:33.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.996+0000 7f334a1ff700 1 -- 172.21.15.23:0/873083729 >> 172.21.15.23:0/873083729 conn(0x7f334406ae30 msgr2=0x7f334406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:33.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.996+0000 7f334a1ff700 1 -- 172.21.15.23:0/873083729 shutdown_connections 2023-06-22T08:24:33.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.996+0000 7f334a1ff700 1 -- 172.21.15.23:0/873083729 wait complete. 2023-06-22T08:24:33.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.997+0000 7f334a1ff700 1 Processor -- start 2023-06-22T08:24:33.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.998+0000 7f334a1ff700 1 -- start start 2023-06-22T08:24:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.998+0000 7f334a1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440c45f0 0x7f3344158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.998+0000 7f334a1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344152d70 0x7f3344151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f334a1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3344159330 0x7f334414c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f334a1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33440b2710 con 0x7f3344152d70 2023-06-22T08:24:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f334a1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3344158780 con 0x7f33440c45f0 2023-06-22T08:24:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f334a1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33440b2980 con 0x7f3344159330 2023-06-22T08:24:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f3342ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344152d70 0x7f3344151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f3342ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344152d70 0x7f3344151620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56530/0 (socket says 172.21.15.23:56530) 2023-06-22T08:24:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f3342ffd700 1 -- 172.21.15.23:0/3909825001 learned_addr learned my addr 172.21.15.23:0/3909825001 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f3343fff700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3344159330 0x7f334414c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f33437fe700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440c45f0 0x7f3344158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f3342ffd700 1 -- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3344159330 msgr2=0x7f334414c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f3342ffd700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3344159330 0x7f334414c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f3342ffd700 1 -- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440c45f0 msgr2=0x7f3344158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f3342ffd700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440c45f0 0x7f3344158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f3342ffd700 1 -- 172.21.15.23:0/3909825001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333c009a70 con 0x7f3344152d70 2023-06-22T08:24:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:32.999+0000 7f3342ffd700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344152d70 0x7f3344151620 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f333800c480 tx=0x7f333800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.000+0000 7f332bfff700 1 -- 172.21.15.23:0/3909825001 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338005d70 con 0x7f3344152d70 2023-06-22T08:24:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.000+0000 7f332bfff700 1 -- 172.21.15.23:0/3909825001 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333800ce90 con 0x7f3344152d70 2023-06-22T08:24:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.000+0000 7f334a1ff700 1 -- 172.21.15.23:0/3909825001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33440b2c60 con 0x7f3344152d70 2023-06-22T08:24:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.000+0000 7f332bfff700 1 -- 172.21.15.23:0/3909825001 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33380072d0 con 0x7f3344152d70 2023-06-22T08:24:33.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.000+0000 7f334a1ff700 1 -- 172.21.15.23:0/3909825001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3344157d20 con 0x7f3344152d70 2023-06-22T08:24:33.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.001+0000 7f334a1ff700 1 -- 172.21.15.23:0/3909825001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3344157b80 con 0x7f3344152d70 2023-06-22T08:24:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.002+0000 7f332bfff700 1 -- 172.21.15.23:0/3909825001 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f3338007430 con 0x7f3344152d70 2023-06-22T08:24:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.003+0000 7f332bfff700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f333006ad10 0x7f333006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.003+0000 7f332bfff700 1 -- 172.21.15.23:0/3909825001 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f3338088e70 con 0x7f3344152d70 2023-06-22T08:24:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.003+0000 7f33437fe700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f333006ad10 0x7f333006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.004+0000 7f33437fe700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f333006ad10 0x7f333006d1c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3334009be0 tx=0x7f3334009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:33.011 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.008+0000 7f332bfff700 1 -- 172.21.15.23:0/3909825001 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f3344157b80 con 0x7f3344152d70 2023-06-22T08:24:33.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.280+0000 7f334a1ff700 1 -- 172.21.15.23:0/3909825001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3344157b80 con 0x7f3344152d70 2023-06-22T08:24:33.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.280+0000 7f332bfff700 1 -- 172.21.15.23:0/3909825001 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5489 (secure 0 0 0) 0x7f3344157b80 con 0x7f3344152d70 2023-06-22T08:24:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:33.284 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6185,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:26.088347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:33.286 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 -- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f333006ad10 msgr2=0x7f333006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:33.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f333006ad10 0x7f333006d1c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3334009be0 tx=0x7f3334009210 comp rx=0 tx=0).stop 2023-06-22T08:24:33.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 -- 172.21.15.23:0/3909825001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344152d70 msgr2=0x7f3344151620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:33.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344152d70 0x7f3344151620 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f333800c480 tx=0x7f333800c790 comp rx=0 tx=0).stop 2023-06-22T08:24:33.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 -- 172.21.15.23:0/3909825001 shutdown_connections 2023-06-22T08:24:33.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3344159330 0x7f334414c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:33.287 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f333006ad10 0x7f333006d1c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:33.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33440c45f0 0x7f3344158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:33.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 --2- 172.21.15.23:0/3909825001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344152d70 0x7f3344151620 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:33.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 -- 172.21.15.23:0/3909825001 >> 172.21.15.23:0/3909825001 conn(0x7f334406ae30 msgr2=0x7f334406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:33.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 -- 172.21.15.23:0/3909825001 shutdown_connections 2023-06-22T08:24:33.288 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:33.283+0000 7f334a1ff700 1 -- 172.21.15.23:0/3909825001 wait complete. 2023-06-22T08:24:33.288 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 191 2023-06-22T08:24:33.304 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:24:33.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~251s 2023-06-22T08:24:35.306 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:35.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.518+0000 7f0cc29ff700 1 Processor -- start 2023-06-22T08:24:35.522 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.518+0000 7f0cc29ff700 1 -- start start 2023-06-22T08:24:35.523 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.520+0000 7f0cc29ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0cbc14f850 0x7f0cbc0c3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.520+0000 7f0cc29ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0cbc0c69e0 0x7f0cbc0c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.520+0000 7f0cc29ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc0c51a0 con 0x7f0cbc071070 2023-06-22T08:24:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.520+0000 7f0cc29ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cbc0c44e0 con 0x7f0cbc14f850 2023-06-22T08:24:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.520+0000 7f0cc29ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cbc0c5440 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.520+0000 7f0cbbfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0cbc14f850 0x7f0cbc0c3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.520+0000 7f0cbbfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0cbc14f850 0x7f0cbc0c3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56546/0 (socket says 172.21.15.23:56546) 2023-06-22T08:24:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.520+0000 7f0cbbfff700 1 -- 172.21.15.23:0/2277394950 learned_addr learned my addr 172.21.15.23:0/2277394950 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:35.525 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.520+0000 7f0cbb7fe700 1 --2- 172.21.15.23:0/2277394950 >> v2:172.21.15.67:3300/0 conn(0x7f0cbc0c69e0 0x7f0cbc0c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:35.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.521+0000 7f0cbbfff700 1 -- 172.21.15.23:0/2277394950 >> v2:172.21.15.67:3300/0 conn(0x7f0cbc0c69e0 msgr2=0x7f0cbc0c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:35.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.521+0000 7f0cbbfff700 1 --2- 172.21.15.23:0/2277394950 >> v2:172.21.15.67:3300/0 conn(0x7f0cbc0c69e0 0x7f0cbc0c35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.521+0000 7f0cbbfff700 1 -- 172.21.15.23:0/2277394950 >> v1:172.21.15.67:6789/0 conn(0x7f0cbc071070 legacy=0x7f0cbc155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:35.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.521+0000 7f0cbbfff700 1 -- 172.21.15.23:0/2277394950 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc0c5720 con 0x7f0cbc14f850 2023-06-22T08:24:35.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.521+0000 7f0cbbfff700 1 --2- 172.21.15.23:0/2277394950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc14f850 0x7f0cbc0c3080 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc06f9a0 tx=0x7f0cac009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b15679e283827e1 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:35.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.521+0000 7f0cbaffd700 1 -- 172.21.15.23:0/2277394950 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac011830 con 0x7f0cbc14f850 2023-06-22T08:24:35.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.521+0000 7f0cbaffd700 1 -- 172.21.15.23:0/2277394950 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac011990 con 0x7f0cbc14f850 2023-06-22T08:24:35.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.521+0000 7f0cbaffd700 1 -- 172.21.15.23:0/2277394950 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac011d20 con 0x7f0cbc14f850 2023-06-22T08:24:35.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.522+0000 7f0cc29ff700 1 -- 172.21.15.23:0/2277394950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc14f850 msgr2=0x7f0cbc0c3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:35.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.522+0000 7f0cc29ff700 1 --2- 172.21.15.23:0/2277394950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc14f850 0x7f0cbc0c3080 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc06f9a0 tx=0x7f0cac009bd0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.522+0000 7f0cc29ff700 1 -- 172.21.15.23:0/2277394950 shutdown_connections 2023-06-22T08:24:35.527 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.522+0000 7f0cc29ff700 1 --2- 172.21.15.23:0/2277394950 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc14f850 0x7f0cbc0c3080 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.522+0000 7f0cc29ff700 1 --2- 172.21.15.23:0/2277394950 >> v2:172.21.15.67:3300/0 conn(0x7f0cbc0c69e0 0x7f0cbc0c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.522+0000 7f0cc29ff700 1 -- 172.21.15.23:0/2277394950 >> 172.21.15.23:0/2277394950 conn(0x7f0cbc0ca3f0 msgr2=0x7f0cbc0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.522+0000 7f0cc29ff700 1 -- 172.21.15.23:0/2277394950 shutdown_connections 2023-06-22T08:24:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.522+0000 7f0cc29ff700 1 -- 172.21.15.23:0/2277394950 wait complete. 2023-06-22T08:24:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.524+0000 7f0cc29ff700 1 Processor -- start 2023-06-22T08:24:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.524+0000 7f0cc29ff700 1 -- start start 2023-06-22T08:24:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cc29ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071070 0x7f0cbc1e0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cc29ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc0c69e0 0x7f0cbc149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cc29ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cbc14f850 0x7f0cbc1430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cc29ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc150430 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cc29ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc149760 con 0x7f0cbc071070 2023-06-22T08:24:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cc29ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc1512b0 con 0x7f0cbc14f850 2023-06-22T08:24:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cbb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc0c69e0 0x7f0cbc149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cbb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc0c69e0 0x7f0cbc149ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56548/0 (socket says 172.21.15.23:56548) 2023-06-22T08:24:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cbbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071070 0x7f0cbc1e0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cbb7fe700 1 -- 172.21.15.23:0/3491833657 learned_addr learned my addr 172.21.15.23:0/3491833657 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cc0f9c700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cbc14f850 0x7f0cbc1430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cbb7fe700 1 -- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cbc14f850 msgr2=0x7f0cbc1430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cbb7fe700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cbc14f850 0x7f0cbc1430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cbb7fe700 1 -- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071070 msgr2=0x7f0cbc1e0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cbb7fe700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071070 0x7f0cbc1e0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.525+0000 7f0cbb7fe700 1 -- 172.21.15.23:0/3491833657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cac009550 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.526+0000 7f0cbb7fe700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc0c69e0 0x7f0cbc149ed0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000e9e0 tx=0x7f0cb000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.526+0000 7f0cb8ff9700 1 -- 172.21.15.23:0/3491833657 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0009710 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.526+0000 7f0cb8ff9700 1 -- 172.21.15.23:0/3491833657 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb000ee80 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.526+0000 7f0cb8ff9700 1 -- 172.21.15.23:0/3491833657 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb000ca50 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.526+0000 7f0cc29ff700 1 -- 172.21.15.23:0/3491833657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc153590 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.526+0000 7f0cc29ff700 1 -- 172.21.15.23:0/3491833657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc1ea6b0 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.527+0000 7f0cc29ff700 1 -- 172.21.15.23:0/3491833657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc0c2950 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.528+0000 7f0cb8ff9700 1 -- 172.21.15.23:0/3491833657 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0cb000cbb0 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.529+0000 7f0cb8ff9700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ca806ad10 0x7f0ca806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:35.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.529+0000 7f0cb8ff9700 1 -- 172.21.15.23:0/3491833657 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f0cb0089b80 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.529+0000 7f0cbbfff700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ca806ad10 0x7f0ca806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:35.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.530+0000 7f0cbbfff700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ca806ad10 0x7f0ca806d1c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0cac014040 tx=0x7f0cac008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:35.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.534+0000 7f0cb8ff9700 1 -- 172.21.15.23:0/3491833657 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0cb0057ed0 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.795+0000 7f0cc29ff700 1 -- 172.21.15.23:0/3491833657 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cbc04ea20 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.799 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.796+0000 7f0cb8ff9700 1 -- 172.21.15.23:0/3491833657 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5489 (secure 0 0 0) 0x7f0cb0021090 con 0x7f0cbc0c69e0 2023-06-22T08:24:35.800 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:35.800 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6185,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:26.088347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:35.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 -- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ca806ad10 msgr2=0x7f0ca806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:35.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ca806ad10 0x7f0ca806d1c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0cac014040 tx=0x7f0cac008030 comp rx=0 tx=0).stop 2023-06-22T08:24:35.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 -- 172.21.15.23:0/3491833657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc0c69e0 msgr2=0x7f0cbc149ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:35.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc0c69e0 0x7f0cbc149ed0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000e9e0 tx=0x7f0cb000ecf0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 -- 172.21.15.23:0/3491833657 shutdown_connections 2023-06-22T08:24:35.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0cbc14f850 0x7f0cbc1430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0ca806ad10 0x7f0ca806d1c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071070 0x7f0cbc1e0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 --2- 172.21.15.23:0/3491833657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0cbc0c69e0 0x7f0cbc149ed0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:35.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 -- 172.21.15.23:0/3491833657 >> 172.21.15.23:0/3491833657 conn(0x7f0cbc0ca3f0 msgr2=0x7f0cbc0702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:35.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 -- 172.21.15.23:0/3491833657 shutdown_connections 2023-06-22T08:24:35.804 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:35.799+0000 7f0cc29ff700 1 -- 172.21.15.23:0/3491833657 wait complete. 2023-06-22T08:24:35.804 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 191 2023-06-22T08:24:35.820 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:24:37.821 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:38.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.031+0000 7fc8a1bff700 1 Processor -- start 2023-06-22T08:24:38.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.031+0000 7fc8a1bff700 1 -- start start 2023-06-22T08:24:38.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc8a1bff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc89c15a320 0x7fc89c0c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:38.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc8a1bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 0x7fc89c160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:38.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc8a1bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89c0c6a60 con 0x7fc89c0c3a40 2023-06-22T08:24:38.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc8a1bff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc89c157570 con 0x7fc89c15a320 2023-06-22T08:24:38.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc8a1bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c0c4350 con 0x7fc89c06d050 2023-06-22T08:24:38.037 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc89affd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc89c15a320 0x7fc89c0c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:38.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc89affd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc89c15a320 0x7fc89c0c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38602/0 (socket says 172.21.15.23:38602) 2023-06-22T08:24:38.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc89bfff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 0x7fc89c160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:38.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc89affd700 1 -- 172.21.15.23:0/1383693191 learned_addr learned my addr 172.21.15.23:0/1383693191 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:38.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc89affd700 1 -- 172.21.15.23:0/1383693191 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 msgr2=0x7fc89c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:38.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc89affd700 1 --2- 172.21.15.23:0/1383693191 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 0x7fc89c160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:38.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc89affd700 1 -- 172.21.15.23:0/1383693191 >> v1:172.21.15.67:6789/0 conn(0x7fc89c0c3a40 legacy=0x7fc89c159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:38.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc89affd700 1 -- 172.21.15.23:0/1383693191 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c14d690 con 0x7fc89c15a320 2023-06-22T08:24:38.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.033+0000 7fc89a7fc700 1 -- 172.21.15.23:0/1383693191 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902427217 0 0) 0x7fc89c0c6a60 con 0x7fc89c0c3a40 2023-06-22T08:24:38.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.034+0000 7fc89affd700 1 --2- 172.21.15.23:0/1383693191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc89c15a320 0x7fc89c0c3e10 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc8900098f0 tx=0x7fc890009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55931a2082e58d45 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:38.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.034+0000 7fc89a7fc700 1 -- 172.21.15.23:0/1383693191 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc890011830 con 0x7fc89c15a320 2023-06-22T08:24:38.041 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.034+0000 7fc89a7fc700 1 -- 172.21.15.23:0/1383693191 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc890011990 con 0x7fc89c15a320 2023-06-22T08:24:38.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.034+0000 7fc89a7fc700 1 -- 172.21.15.23:0/1383693191 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc890011d20 con 0x7fc89c15a320 2023-06-22T08:24:38.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.034+0000 7fc8a1bff700 1 -- 172.21.15.23:0/1383693191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc89c15a320 msgr2=0x7fc89c0c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:38.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.034+0000 7fc8a1bff700 1 --2- 172.21.15.23:0/1383693191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc89c15a320 0x7fc89c0c3e10 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc8900098f0 tx=0x7fc890009c00 comp rx=0 tx=0).stop 2023-06-22T08:24:38.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.035+0000 7fc8a1bff700 1 -- 172.21.15.23:0/1383693191 shutdown_connections 2023-06-22T08:24:38.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.035+0000 7fc8a1bff700 1 --2- 172.21.15.23:0/1383693191 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 0x7fc89c160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:38.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.035+0000 7fc8a1bff700 1 --2- 172.21.15.23:0/1383693191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc89c15a320 0x7fc89c0c3e10 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:38.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.035+0000 7fc8a1bff700 1 -- 172.21.15.23:0/1383693191 >> 172.21.15.23:0/1383693191 conn(0x7fc89c0ca3f0 msgr2=0x7fc89c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:38.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.035+0000 7fc8a1bff700 1 -- 172.21.15.23:0/1383693191 shutdown_connections 2023-06-22T08:24:38.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.035+0000 7fc8a1bff700 1 -- 172.21.15.23:0/1383693191 wait complete. 2023-06-22T08:24:38.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.036+0000 7fc8a1bff700 1 Processor -- start 2023-06-22T08:24:38.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.037+0000 7fc8a1bff700 1 -- start start 2023-06-22T08:24:38.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.037+0000 7fc8a1bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 0x7fc89c150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:38.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc8a1bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc89c0c3a40 0x7fc89c1eebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc8a1bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc89c15a320 0x7fc89c1f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc8a1bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c1ef940 con 0x7fc89c15a320 2023-06-22T08:24:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc8a1bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c1ee4d0 con 0x7fc89c0c3a40 2023-06-22T08:24:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc8a1bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c1f1e80 con 0x7fc89c06d050 2023-06-22T08:24:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89b7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 0x7fc89c150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc89c0c3a40 0x7fc89c1eebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89b7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 0x7fc89c150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49726/0 (socket says 172.21.15.23:49726) 2023-06-22T08:24:38.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89b7fe700 1 -- 172.21.15.23:0/4017934862 learned_addr learned my addr 172.21.15.23:0/4017934862 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:38.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89bfff700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc89c15a320 0x7fc89c1f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:38.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89b7fe700 1 -- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc89c0c3a40 msgr2=0x7fc89c1eebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:38.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89b7fe700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc89c0c3a40 0x7fc89c1eebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:38.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89b7fe700 1 -- 172.21.15.23:0/4017934862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc89c15a320 msgr2=0x7fc89c1f0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89b7fe700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc89c15a320 0x7fc89c1f0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89b7fe700 1 -- 172.21.15.23:0/4017934862 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc890009550 con 0x7fc89c06d050 2023-06-22T08:24:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89affd700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc89c0c3a40 0x7fc89c1eebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:24:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.038+0000 7fc89bfff700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc89c15a320 0x7fc89c1f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:24:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.039+0000 7fc89b7fe700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 0x7fc89c150420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00c750 tx=0x7fc88c00ca60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.039+0000 7fc883fff700 1 -- 172.21.15.23:0/4017934862 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c0054e0 con 0x7fc89c06d050 2023-06-22T08:24:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.039+0000 7fc883fff700 1 -- 172.21.15.23:0/4017934862 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88c003680 con 0x7fc89c06d050 2023-06-22T08:24:38.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.039+0000 7fc883fff700 1 -- 172.21.15.23:0/4017934862 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c003a30 con 0x7fc89c06d050 2023-06-22T08:24:38.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.039+0000 7fc8a1bff700 1 -- 172.21.15.23:0/4017934862 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89c1f2160 con 0x7fc89c06d050 2023-06-22T08:24:38.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.039+0000 7fc8a1bff700 1 -- 172.21.15.23:0/4017934862 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89c1f2720 con 0x7fc89c06d050 2023-06-22T08:24:38.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.040+0000 7fc8a1bff700 1 -- 172.21.15.23:0/4017934862 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89c1496e0 con 0x7fc89c06d050 2023-06-22T08:24:38.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.041+0000 7fc883fff700 1 -- 172.21.15.23:0/4017934862 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc88c003b90 con 0x7fc89c06d050 2023-06-22T08:24:38.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.042+0000 7fc883fff700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc88806ace0 0x7fc88806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:38.048 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.042+0000 7fc883fff700 1 -- 172.21.15.23:0/4017934862 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7fc88c089160 con 0x7fc89c06d050 2023-06-22T08:24:38.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.043+0000 7fc89affd700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc88806ace0 0x7fc88806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:38.049 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.043+0000 7fc89affd700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc88806ace0 0x7fc88806d190 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc890009c40 tx=0x7fc890008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:38.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.048+0000 7fc883fff700 1 -- 172.21.15.23:0/4017934862 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc88c053a50 con 0x7fc89c06d050 2023-06-22T08:24:38.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.303+0000 7fc8a1bff700 1 -- 172.21.15.23:0/4017934862 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc89c04ea20 con 0x7fc89c06d050 2023-06-22T08:24:38.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.304+0000 7fc883fff700 1 -- 172.21.15.23:0/4017934862 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5489 (secure 0 0 0) 0x7fc88c017070 con 0x7fc89c06d050 2023-06-22T08:24:38.308 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:38.308 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6185,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:26.088347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:38.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 -- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc88806ace0 msgr2=0x7fc88806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:38.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc88806ace0 0x7fc88806d190 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc890009c40 tx=0x7fc890008030 comp rx=0 tx=0).stop 2023-06-22T08:24:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 -- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 msgr2=0x7fc89c150420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 0x7fc89c150420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00c750 tx=0x7fc88c00ca60 comp rx=0 tx=0).stop 2023-06-22T08:24:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 -- 172.21.15.23:0/4017934862 shutdown_connections 2023-06-22T08:24:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc89c15a320 0x7fc89c1f0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc89c06d050 0x7fc89c150420 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc88806ace0 0x7fc88806d190 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 --2- 172.21.15.23:0/4017934862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc89c0c3a40 0x7fc89c1eebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:38.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 -- 172.21.15.23:0/4017934862 >> 172.21.15.23:0/4017934862 conn(0x7fc89c0ca3f0 msgr2=0x7fc89c155340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:38.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 -- 172.21.15.23:0/4017934862 shutdown_connections 2023-06-22T08:24:38.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:38.307+0000 7fc8a1bff700 1 -- 172.21.15.23:0/4017934862 wait complete. 2023-06-22T08:24:38.312 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 191 2023-06-22T08:24:38.328 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:24:40.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~257s 2023-06-22T08:24:40.330 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:40.532 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.529+0000 7ff8b3dff700 1 Processor -- start 2023-06-22T08:24:40.533 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.529+0000 7ff8b3dff700 1 -- start start 2023-06-22T08:24:40.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.530+0000 7ff8b3dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c9fd0 0x7ff8ac0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:40.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.530+0000 7ff8b3dff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff8ac0c40b0 0x7ff8ac0c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:40.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.530+0000 7ff8b3dff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac15c370 con 0x7ff8ac152ce0 2023-06-22T08:24:40.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.530+0000 7ff8b3dff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ac1530b0 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.534 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.530+0000 7ff8b3dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0cf6f0 con 0x7ff8ac0c9fd0 2023-06-22T08:24:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b1b9b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff8ac152ce0 0x7ff8ac15c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:40238/0 (socket says 172.21.15.23:40238) 2023-06-22T08:24:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b139a700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c9fd0 0x7ff8ac0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b1b9b700 1 -- 172.21.15.23:0/2291691072 learned_addr learned my addr 172.21.15.23:0/2291691072 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b239c700 1 --2- 172.21.15.23:0/2291691072 >> v2:172.21.15.23:3300/0 conn(0x7ff8ac0c40b0 0x7ff8ac0c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:40.536 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b0b99700 1 -- 172.21.15.23:0/2291691072 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947463621 0 0) 0x7ff8ac15c370 con 0x7ff8ac152ce0 2023-06-22T08:24:40.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b0b99700 1 -- 172.21.15.23:0/2291691072 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8980035b0 con 0x7ff8ac152ce0 2023-06-22T08:24:40.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b239c700 1 -- 172.21.15.23:0/2291691072 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c9fd0 msgr2=0x7ff8ac0cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:40.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b239c700 1 --2- 172.21.15.23:0/2291691072 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c9fd0 0x7ff8ac0cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:40.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b239c700 1 -- 172.21.15.23:0/2291691072 >> v1:172.21.15.67:6789/0 conn(0x7ff8ac152ce0 legacy=0x7ff8ac15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:40.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b239c700 1 -- 172.21.15.23:0/2291691072 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac15bda0 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b139a700 1 --2- 172.21.15.23:0/2291691072 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c9fd0 0x7ff8ac0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:24:40.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.531+0000 7ff8b239c700 1 --2- 172.21.15.23:0/2291691072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8ac0c40b0 0x7ff8ac0c4530 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009d90 tx=0x7ff8a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb4e5ea07f49722 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:40.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b0b99700 1 -- 172.21.15.23:0/2291691072 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a400e070 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b0b99700 1 -- 172.21.15.23:0/2291691072 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a400bb20 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b0b99700 1 -- 172.21.15.23:0/2291691072 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4012430 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b3dff700 1 -- 172.21.15.23:0/2291691072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8ac0c40b0 msgr2=0x7ff8ac0c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:40.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b3dff700 1 --2- 172.21.15.23:0/2291691072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8ac0c40b0 0x7ff8ac0c4530 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009d90 tx=0x7ff8a400b330 comp rx=0 tx=0).stop 2023-06-22T08:24:40.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b3dff700 1 -- 172.21.15.23:0/2291691072 shutdown_connections 2023-06-22T08:24:40.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b3dff700 1 --2- 172.21.15.23:0/2291691072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8ac0c40b0 0x7ff8ac0c4530 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b3dff700 1 --2- 172.21.15.23:0/2291691072 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c9fd0 0x7ff8ac0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b3dff700 1 -- 172.21.15.23:0/2291691072 >> 172.21.15.23:0/2291691072 conn(0x7ff8ac06ae30 msgr2=0x7ff8ac06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b3dff700 1 -- 172.21.15.23:0/2291691072 shutdown_connections 2023-06-22T08:24:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.532+0000 7ff8b3dff700 1 -- 172.21.15.23:0/2291691072 wait complete. 2023-06-22T08:24:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.534+0000 7ff8b3dff700 1 Processor -- start 2023-06-22T08:24:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.534+0000 7ff8b3dff700 1 -- start start 2023-06-22T08:24:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.535+0000 7ff8b3dff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c40b0 0x7ff8ac0c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.535+0000 7ff8b3dff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8ac0c9fd0 0x7ff8ac0ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.535+0000 7ff8b3dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac152ce0 0x7ff8ac1e38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b3dff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1f1840 con 0x7ff8ac0c9fd0 2023-06-22T08:24:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b3dff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1f1ae0 con 0x7ff8ac152ce0 2023-06-22T08:24:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b3dff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1f1dc0 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b1b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c40b0 0x7ff8ac0c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b1b9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c40b0 0x7ff8ac0c7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:49752/0 (socket says 172.21.15.23:49752) 2023-06-22T08:24:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b1b9b700 1 -- 172.21.15.23:0/636533826 learned_addr learned my addr 172.21.15.23:0/636533826 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b139a700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8ac0c9fd0 0x7ff8ac0ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b239c700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac152ce0 0x7ff8ac1e38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b1b9b700 1 -- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac152ce0 msgr2=0x7ff8ac1e38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:40.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b1b9b700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac152ce0 0x7ff8ac1e38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:40.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b1b9b700 1 -- 172.21.15.23:0/636533826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8ac0c9fd0 msgr2=0x7ff8ac0ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:40.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b1b9b700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8ac0c9fd0 0x7ff8ac0ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:40.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b1b9b700 1 -- 172.21.15.23:0/636533826 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c0094a0 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.536+0000 7ff8b1b9b700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c40b0 0x7ff8ac0c7be0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800c700 tx=0x7ff8a800ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:40.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.537+0000 7ff8a27fc700 1 -- 172.21.15.23:0/636533826 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8005d70 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.537+0000 7ff8a27fc700 1 -- 172.21.15.23:0/636533826 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a80070d0 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.543 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.537+0000 7ff8b3dff700 1 -- 172.21.15.23:0/636533826 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a4009a70 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.537+0000 7ff8a27fc700 1 -- 172.21.15.23:0/636533826 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8007460 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.544 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.537+0000 7ff8b3dff700 1 -- 172.21.15.23:0/636533826 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac1f2380 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.538+0000 7ff8b3dff700 1 -- 172.21.15.23:0/636533826 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac04ea20 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.539+0000 7ff8a27fc700 1 -- 172.21.15.23:0/636533826 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7ff8a80075c0 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.540+0000 7ff8a27fc700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff89006ad10 0x7ff89006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.540+0000 7ff8a27fc700 1 -- 172.21.15.23:0/636533826 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7ff8a80890a0 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.540+0000 7ff8b139a700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff89006ad10 0x7ff89006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.541+0000 7ff8b139a700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff89006ad10 0x7ff89006d1c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0cb3d0 tx=0x7ff89c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:40.548 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.545+0000 7ff8a27fc700 1 -- 172.21.15.23:0/636533826 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7ff8a8053960 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.802 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.799+0000 7ff8b3dff700 1 -- 172.21.15.23:0/636533826 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8ac1f1f00 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.803 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.801+0000 7ff8a27fc700 1 -- 172.21.15.23:0/636533826 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5489 (secure 0 0 0) 0x7ff8ac1f1f00 con 0x7ff8ac0c40b0 2023-06-22T08:24:40.803 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:40.804 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6185,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:26.088347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 -- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff89006ad10 msgr2=0x7ff89006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff89006ad10 0x7ff89006d1c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0cb3d0 tx=0x7ff89c009210 comp rx=0 tx=0).stop 2023-06-22T08:24:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 -- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c40b0 msgr2=0x7ff8ac0c7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c40b0 0x7ff8ac0c7be0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800c700 tx=0x7ff8a800ca10 comp rx=0 tx=0).stop 2023-06-22T08:24:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 -- 172.21.15.23:0/636533826 shutdown_connections 2023-06-22T08:24:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7ff89006ad10 0x7ff89006d1c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8ac152ce0 0x7ff8ac1e38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff8ac0c40b0 0x7ff8ac0c7be0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 --2- 172.21.15.23:0/636533826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8ac0c9fd0 0x7ff8ac0ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 -- 172.21.15.23:0/636533826 >> 172.21.15.23:0/636533826 conn(0x7ff8ac06ae30 msgr2=0x7ff8ac14c430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 -- 172.21.15.23:0/636533826 shutdown_connections 2023-06-22T08:24:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:40.803+0000 7ff8b3dff700 1 -- 172.21.15.23:0/636533826 wait complete. 2023-06-22T08:24:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 191 2023-06-22T08:24:40.824 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:24:42.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.031+0000 7f23f0fff700 1 Processor -- start 2023-06-22T08:24:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.031+0000 7f23f0fff700 1 -- start start 2023-06-22T08:24:43.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23f0fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec152d70 0x7f23ec0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:43.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23f0fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f23ec159330 0x7f23ec0cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:43.035 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23f0fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ec15bea0 con 0x7f23ec0cb800 2023-06-22T08:24:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23f0fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ec0c95e0 con 0x7f23ec159330 2023-06-22T08:24:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23f0fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec15c160 con 0x7f23ec152d70 2023-06-22T08:24:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23e3fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f23ec159330 0x7f23ec0cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23ead9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f23ec0cb800 0x7f23ec0c45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41674/0 (socket says 172.21.15.23:41674) 2023-06-22T08:24:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23ead9d700 1 -- 172.21.15.23:0/2424799315 learned_addr learned my addr 172.21.15.23:0/2424799315 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23ea59c700 1 --2- 172.21.15.23:0/2424799315 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec152d70 0x7f23ec0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:43.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23e3fff700 1 -- 172.21.15.23:0/2424799315 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec152d70 msgr2=0x7f23ec0c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:43.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23e3fff700 1 --2- 172.21.15.23:0/2424799315 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec152d70 0x7f23ec0c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:43.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23e3fff700 1 -- 172.21.15.23:0/2424799315 >> v1:172.21.15.23:6789/0 conn(0x7f23ec0cb800 legacy=0x7f23ec0c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:43.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23e3fff700 1 -- 172.21.15.23:0/2424799315 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ec15c440 con 0x7f23ec159330 2023-06-22T08:24:43.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.033+0000 7f23e9d9b700 1 -- 172.21.15.23:0/2424799315 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742436456 0 0) 0x7f23ec15bea0 con 0x7f23ec0cb800 2023-06-22T08:24:43.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.034+0000 7f23e3fff700 1 --2- 172.21.15.23:0/2424799315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec159330 0x7f23ec0cb2c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009840 tx=0x7f23d4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0b602af78195f0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:43.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.034+0000 7f23e9d9b700 1 -- 172.21.15.23:0/2424799315 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d4011830 con 0x7f23ec159330 2023-06-22T08:24:43.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.034+0000 7f23e9d9b700 1 -- 172.21.15.23:0/2424799315 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d4011990 con 0x7f23ec159330 2023-06-22T08:24:43.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.034+0000 7f23e9d9b700 1 -- 172.21.15.23:0/2424799315 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d4011d20 con 0x7f23ec159330 2023-06-22T08:24:43.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.034+0000 7f23f0fff700 1 -- 172.21.15.23:0/2424799315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec159330 msgr2=0x7f23ec0cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:43.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.034+0000 7f23f0fff700 1 --2- 172.21.15.23:0/2424799315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec159330 0x7f23ec0cb2c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009840 tx=0x7f23d4009b50 comp rx=0 tx=0).stop 2023-06-22T08:24:43.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.035+0000 7f23f0fff700 1 -- 172.21.15.23:0/2424799315 shutdown_connections 2023-06-22T08:24:43.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.035+0000 7f23f0fff700 1 --2- 172.21.15.23:0/2424799315 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec152d70 0x7f23ec0c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:43.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.035+0000 7f23f0fff700 1 --2- 172.21.15.23:0/2424799315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec159330 0x7f23ec0cb2c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:43.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.035+0000 7f23f0fff700 1 -- 172.21.15.23:0/2424799315 >> 172.21.15.23:0/2424799315 conn(0x7f23ec06ae30 msgr2=0x7f23ec06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:43.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.035+0000 7f23f0fff700 1 -- 172.21.15.23:0/2424799315 shutdown_connections 2023-06-22T08:24:43.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.035+0000 7f23f0fff700 1 -- 172.21.15.23:0/2424799315 wait complete. 2023-06-22T08:24:43.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.037+0000 7f23f0fff700 1 Processor -- start 2023-06-22T08:24:43.040 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.037+0000 7f23f0fff700 1 -- start start 2023-06-22T08:24:43.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.038+0000 7f23f0fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec0cb800 0x7f23ec1578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:43.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.038+0000 7f23f0fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec152d70 0x7f23ec159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:43.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.038+0000 7f23f0fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec159330 0x7f23ec1584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:43.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.038+0000 7f23f0fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec1589e0 con 0x7f23ec0cb800 2023-06-22T08:24:43.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.038+0000 7f23f0fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec1f1600 con 0x7f23ec152d70 2023-06-22T08:24:43.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.038+0000 7f23f0fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec157460 con 0x7f23ec159330 2023-06-22T08:24:43.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23ea59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec0cb800 0x7f23ec1578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:43.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23ea59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec0cb800 0x7f23ec1578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38636/0 (socket says 172.21.15.23:38636) 2023-06-22T08:24:43.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23ea59c700 1 -- 172.21.15.23:0/2914591269 learned_addr learned my addr 172.21.15.23:0/2914591269 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:43.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23e3fff700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec152d70 0x7f23ec159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:43.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23ead9d700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec159330 0x7f23ec1584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:43.043 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23ea59c700 1 -- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec159330 msgr2=0x7f23ec1584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23ea59c700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec159330 0x7f23ec1584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23ea59c700 1 -- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec152d70 msgr2=0x7f23ec159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23ea59c700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec152d70 0x7f23ec159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23ea59c700 1 -- 172.21.15.23:0/2914591269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d40094a0 con 0x7f23ec0cb800 2023-06-22T08:24:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.039+0000 7f23ead9d700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec159330 0x7f23ec1584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:24:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.040+0000 7f23ea59c700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec0cb800 0x7f23ec1578f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00e9e0 tx=0x7f23dc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.040+0000 7f23e2ffd700 1 -- 172.21.15.23:0/2914591269 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc009710 con 0x7f23ec0cb800 2023-06-22T08:24:43.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.040+0000 7f23e2ffd700 1 -- 172.21.15.23:0/2914591269 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23dc00c850 con 0x7f23ec0cb800 2023-06-22T08:24:43.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.040+0000 7f23e2ffd700 1 -- 172.21.15.23:0/2914591269 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23dc00cbc0 con 0x7f23ec0cb800 2023-06-22T08:24:43.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.040+0000 7f23f0fff700 1 -- 172.21.15.23:0/2914591269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ec151430 con 0x7f23ec0cb800 2023-06-22T08:24:43.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.040+0000 7f23f0fff700 1 -- 172.21.15.23:0/2914591269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ec156b30 con 0x7f23ec0cb800 2023-06-22T08:24:43.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.041+0000 7f23f0fff700 1 -- 172.21.15.23:0/2914591269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ec04ea20 con 0x7f23ec0cb800 2023-06-22T08:24:43.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.042+0000 7f23e2ffd700 1 -- 172.21.15.23:0/2914591269 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f23dc00cd20 con 0x7f23ec0cb800 2023-06-22T08:24:43.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.043+0000 7f23e2ffd700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23cc06ace0 0x7f23cc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:43.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.043+0000 7f23e2ffd700 1 -- 172.21.15.23:0/2914591269 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 7931+0+0 (secure 0 0 0) 0x7f23dc08a0b0 con 0x7f23ec0cb800 2023-06-22T08:24:43.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.043+0000 7f23e3fff700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23cc06ace0 0x7f23cc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:43.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.044+0000 7f23e3fff700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23cc06ace0 0x7f23cc06d190 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f23d40090b0 tx=0x7f23d4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.048+0000 7f23e2ffd700 1 -- 172.21.15.23:0/2914591269 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f23dc054970 con 0x7f23ec0cb800 2023-06-22T08:24:43.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.303+0000 7f23f0fff700 1 -- 172.21.15.23:0/2914591269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23ec0c9e00 con 0x7f23ec0cb800 2023-06-22T08:24:43.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.304+0000 7f23e2ffd700 1 -- 172.21.15.23:0/2914591269 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5489 (secure 0 0 0) 0x7f23dc057f90 con 0x7f23ec0cb800 2023-06-22T08:24:43.306 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:43.306 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6185,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:26.088347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6313,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6313":{"gid":6313,"name":"e","rank":2,"incarnation":132,"state":"up:active","state_seq":36,"addr":"172.21.15.23:6837/1502668188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":1502668188},{"type":"v1","addr":"172.21.15.23:6837","nonce":1502668188}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:43.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.306+0000 7f23f0fff700 1 -- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23cc06ace0 msgr2=0x7f23cc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:43.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.306+0000 7f23f0fff700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23cc06ace0 0x7f23cc06d190 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f23d40090b0 tx=0x7f23d4008030 comp rx=0 tx=0).stop 2023-06-22T08:24:43.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.306+0000 7f23f0fff700 1 -- 172.21.15.23:0/2914591269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec0cb800 msgr2=0x7f23ec1578f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:43.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.306+0000 7f23f0fff700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec0cb800 0x7f23ec1578f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00e9e0 tx=0x7f23dc00ecf0 comp rx=0 tx=0).stop 2023-06-22T08:24:43.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.306+0000 7f23f0fff700 1 -- 172.21.15.23:0/2914591269 shutdown_connections 2023-06-22T08:24:43.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.306+0000 7f23f0fff700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f23ec159330 0x7f23ec1584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:43.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.306+0000 7f23f0fff700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ec0cb800 0x7f23ec1578f0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:43.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.306+0000 7f23f0fff700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23cc06ace0 0x7f23cc06d190 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:43.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.306+0000 7f23f0fff700 1 --2- 172.21.15.23:0/2914591269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23ec152d70 0x7f23ec159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:43.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.306+0000 7f23f0fff700 1 -- 172.21.15.23:0/2914591269 >> 172.21.15.23:0/2914591269 conn(0x7f23ec06ae30 msgr2=0x7f23ec151bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:43.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.307+0000 7f23f0fff700 1 -- 172.21.15.23:0/2914591269 shutdown_connections 2023-06-22T08:24:43.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:43.307+0000 7f23f0fff700 1 -- 172.21.15.23:0/2914591269 wait complete. 2023-06-22T08:24:43.311 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 191 2023-06-22T08:24:43.328 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-06-22T08:24:45.329 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:45.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.536+0000 7f8953fff700 1 Processor -- start 2023-06-22T08:24:45.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.536+0000 7f8953fff700 1 -- start start 2023-06-22T08:24:45.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.537+0000 7f8953fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c6f30 0x7f8954070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:45.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.537+0000 7f8953fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f895406fbb0 con 0x7f89540c4d10 2023-06-22T08:24:45.611 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.537+0000 7f8953fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f895406a740 con 0x7f89540c7300 2023-06-22T08:24:45.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.537+0000 7f8953fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895406aa20 con 0x7f89540c6f30 2023-06-22T08:24:45.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.537+0000 7f89537fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f89540c4d10 0x7f8954165b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:41690/0 (socket says 172.21.15.23:41690) 2023-06-22T08:24:45.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.537+0000 7f89537fe700 1 -- 172.21.15.23:0/2792490269 learned_addr learned my addr 172.21.15.23:0/2792490269 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:45.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.537+0000 7f8952ffd700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c6f30 0x7f8954070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:45.612 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.538+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190919663 0 0) 0x7f895406fbb0 con 0x7f89540c4d10 2023-06-22T08:24:45.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f893c003590 con 0x7f89540c4d10 2023-06-22T08:24:45.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306492886 0 0) 0x7f895406a740 con 0x7f89540c7300 2023-06-22T08:24:45.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f895406fbb0 con 0x7f89540c7300 2023-06-22T08:24:45.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1475466990 0 0) 0x7f893c003590 con 0x7f89540c4d10 2023-06-22T08:24:45.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c6f30 msgr2=0x7f8954070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:45.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c6f30 0x7f8954070fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:45.613 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 >> v1:172.21.15.67:6789/0 conn(0x7f89540c7300 legacy=0x7f89541694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:45.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895406fe00 con 0x7f89540c4d10 2023-06-22T08:24:45.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7f894c0019f0 con 0x7f89540c4d10 2023-06-22T08:24:45.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 >> v1:172.21.15.23:6789/0 conn(0x7f89540c4d10 legacy=0x7f8954165b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:45.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f893c003d60 0x7f893c004150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:45.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f893c004690 0x7f893c00d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:45.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c6f30 0x7f893c00f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:45.614 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895406aa20 con 0x7f893c003d60 2023-06-22T08:24:45.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c00ff30 con 0x7f893c004690 2023-06-22T08:24:45.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.607+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0100a0 con 0x7f89540c6f30 2023-06-22T08:24:45.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.608+0000 7f8952ffd700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f893c003d60 0x7f893c004150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:45.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.608+0000 7f89537fe700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c6f30 0x7f893c00f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:45.615 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.608+0000 7f89527fc700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f893c004690 0x7f893c00d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:45.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.608+0000 7f8952ffd700 1 -- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c6f30 msgr2=0x7f893c00f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:45.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.608+0000 7f8952ffd700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c6f30 0x7f893c00f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:45.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.608+0000 7f8952ffd700 1 -- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f893c004690 msgr2=0x7f893c00d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:45.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.608+0000 7f8952ffd700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f893c004690 0x7f893c00d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:45.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.608+0000 7f8952ffd700 1 -- 172.21.15.23:0/2792490269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c003590 con 0x7f893c003d60 2023-06-22T08:24:45.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.608+0000 7f89527fc700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f893c004690 0x7f893c00d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:24:45.616 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.609+0000 7f8952ffd700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f893c003d60 0x7f893c004150 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8944009a80 tx=0x7f893c004de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bacf085ff1276c6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:45.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.609+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89440092d0 con 0x7f893c003d60 2023-06-22T08:24:45.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.609+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894400f040 con 0x7f893c003d60 2023-06-22T08:24:45.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.609+0000 7f8951ffb700 1 -- 172.21.15.23:0/2792490269 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894400e830 con 0x7f893c003d60 2023-06-22T08:24:45.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.609+0000 7f8953fff700 1 -- 172.21.15.23:0/2792490269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f893c003d60 msgr2=0x7f893c004150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:45.619 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.609+0000 7f8953fff700 1 --2- 172.21.15.23:0/2792490269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f893c003d60 0x7f893c004150 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8944009a80 tx=0x7f893c004de0 comp rx=0 tx=0).stop 2023-06-22T08:24:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.609+0000 7f89537fe700 1 -- 172.21.15.23:0/2792490269 reap_dead start 2023-06-22T08:24:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.610+0000 7f8953fff700 1 -- 172.21.15.23:0/2792490269 shutdown_connections 2023-06-22T08:24:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.610+0000 7f8953fff700 1 -- 172.21.15.23:0/2792490269 >> 172.21.15.23:0/2792490269 conn(0x7f895406e3b0 msgr2=0x7f8954069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.610+0000 7f8953fff700 1 -- 172.21.15.23:0/2792490269 shutdown_connections 2023-06-22T08:24:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.610+0000 7f8953fff700 1 -- 172.21.15.23:0/2792490269 wait complete. 2023-06-22T08:24:45.620 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.612+0000 7f8953fff700 1 Processor -- start 2023-06-22T08:24:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.612+0000 7f8953fff700 1 -- start start 2023-06-22T08:24:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f8953fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89540c3090 0x7f89541f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f8953fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c3460 0x7f89541f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f8953fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8954169710 0x7f8954166b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f8953fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895415f3c0 con 0x7f8954169710 2023-06-22T08:24:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f8953fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89541fb580 con 0x7f89540c3090 2023-06-22T08:24:45.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f8953fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89541fb860 con 0x7f89540c3460 2023-06-22T08:24:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f8952ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89540c3090 0x7f89541f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f8952ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89540c3090 0x7f89541f2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:51250/0 (socket says 172.21.15.23:51250) 2023-06-22T08:24:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f8952ffd700 1 -- 172.21.15.23:0/215092350 learned_addr learned my addr 172.21.15.23:0/215092350 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f89527fc700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c3460 0x7f89541f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f89537fe700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8954169710 0x7f8954166b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f89527fc700 1 -- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89540c3090 msgr2=0x7f89541f2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f89527fc700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89540c3090 0x7f89541f2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:45.622 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f89527fc700 1 -- 172.21.15.23:0/215092350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8954169710 msgr2=0x7f8954166b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f89527fc700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8954169710 0x7f8954166b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.613+0000 7f89527fc700 1 -- 172.21.15.23:0/215092350 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89541fbb40 con 0x7f89540c3460 2023-06-22T08:24:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.614+0000 7f89527fc700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c3460 0x7f89541f1dc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f894800f810 tx=0x7f894800fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.614+0000 7f893b7fe700 1 -- 172.21.15.23:0/215092350 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89480052e0 con 0x7f89540c3460 2023-06-22T08:24:45.623 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.614+0000 7f893b7fe700 1 -- 172.21.15.23:0/215092350 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8948005440 con 0x7f89540c3460 2023-06-22T08:24:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.614+0000 7f893b7fe700 1 -- 172.21.15.23:0/215092350 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89480038c0 con 0x7f89540c3460 2023-06-22T08:24:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.614+0000 7f8953fff700 1 -- 172.21.15.23:0/215092350 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89541fbe20 con 0x7f89540c3460 2023-06-22T08:24:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.614+0000 7f8953fff700 1 -- 172.21.15.23:0/215092350 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8954167040 con 0x7f89540c3460 2023-06-22T08:24:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.615+0000 7f8953fff700 1 -- 172.21.15.23:0/215092350 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89540c86c0 con 0x7f89540c3460 2023-06-22T08:24:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.616+0000 7f893b7fe700 1 -- 172.21.15.23:0/215092350 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f8948003a20 con 0x7f89540c3460 2023-06-22T08:24:45.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.617+0000 7f893b7fe700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f894006ad10 0x7f894006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:45.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.617+0000 7f893b7fe700 1 -- 172.21.15.23:0/215092350 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7f8948089b10 con 0x7f89540c3460 2023-06-22T08:24:45.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.617+0000 7f8952ffd700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f894006ad10 0x7f894006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.618+0000 7f8952ffd700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f894006ad10 0x7f894006d1c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8944000c00 tx=0x7f8944013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:45.626 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.622+0000 7f893b7fe700 1 -- 172.21.15.23:0/215092350 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f89480542e0 con 0x7f89540c3460 2023-06-22T08:24:45.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.895+0000 7f8953fff700 1 -- 172.21.15.23:0/215092350 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8954071680 con 0x7f89540c3460 2023-06-22T08:24:45.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.897+0000 7f893b7fe700 1 -- 172.21.15.23:0/215092350 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4730 (secure 0 0 0) 0x7f8948057900 con 0x7f89540c3460 2023-06-22T08:24:45.899 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:45.899 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:44.732071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.899+0000 7f8953fff700 1 -- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f894006ad10 msgr2=0x7f894006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.899+0000 7f8953fff700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f894006ad10 0x7f894006d1c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8944000c00 tx=0x7f8944013000 comp rx=0 tx=0).stop 2023-06-22T08:24:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.899+0000 7f8953fff700 1 -- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c3460 msgr2=0x7f89541f1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.899+0000 7f8953fff700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c3460 0x7f89541f1dc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f894800f810 tx=0x7f894800fb20 comp rx=0 tx=0).stop 2023-06-22T08:24:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.899+0000 7f8953fff700 1 -- 172.21.15.23:0/215092350 shutdown_connections 2023-06-22T08:24:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.900+0000 7f8953fff700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8954169710 0x7f8954166b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.900+0000 7f8953fff700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f894006ad10 0x7f894006d1c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.900+0000 7f8953fff700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89540c3090 0x7f89541f2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.900+0000 7f8953fff700 1 --2- 172.21.15.23:0/215092350 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f89540c3460 0x7f89541f1dc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.900+0000 7f8953fff700 1 -- 172.21.15.23:0/215092350 >> 172.21.15.23:0/215092350 conn(0x7f893c003d60 msgr2=0x7f8954165d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.900+0000 7f8953fff700 1 -- 172.21.15.23:0/215092350 shutdown_connections 2023-06-22T08:24:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:45.900+0000 7f8953fff700 1 -- 172.21.15.23:0/215092350 wait complete. 2023-06-22T08:24:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 192 2023-06-22T08:24:45.920 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-06-22T08:24:45.921 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-06-22T08:24:45.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.122+0000 7f3374fff700 1 Processor -- start 2023-06-22T08:24:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.122+0000 7f3374fff700 1 -- start start 2023-06-22T08:24:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.124+0000 7f3374fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3370166380 0x7f337006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.124+0000 7f3374fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3370069eb0 0x7f337006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.124+0000 7f3374fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33700c6b20 con 0x7f337006d530 2023-06-22T08:24:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.124+0000 7f3374fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337006d050 con 0x7f3370166380 2023-06-22T08:24:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.124+0000 7f3374fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3370071420 con 0x7f3370069eb0 2023-06-22T08:24:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.124+0000 7f336e59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3370166380 0x7f337006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.124+0000 7f336ed9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f337006d530 0x7f337006d990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:46694/0 (socket says 172.21.15.23:46694) 2023-06-22T08:24:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.124+0000 7f336ed9d700 1 -- 172.21.15.23:0/1456664545 learned_addr learned my addr 172.21.15.23:0/1456664545 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.124+0000 7f336dd9b700 1 --2- 172.21.15.23:0/1456664545 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3370069eb0 0x7f337006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.124+0000 7f336d59a700 1 -- 172.21.15.23:0/1456664545 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677481294 0 0) 0x7f33700c6b20 con 0x7f337006d530 2023-06-22T08:24:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.342+0000 7f336d59a700 1 -- 172.21.15.23:0/1456664545 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3358003590 con 0x7f337006d530 2023-06-22T08:24:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.342+0000 7f336e59c700 1 -- 172.21.15.23:0/1456664545 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3370069eb0 msgr2=0x7f337006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.342+0000 7f336e59c700 1 --2- 172.21.15.23:0/1456664545 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3370069eb0 0x7f337006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.342+0000 7f336e59c700 1 -- 172.21.15.23:0/1456664545 >> v1:172.21.15.23:6789/0 conn(0x7f337006d530 legacy=0x7f337006d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:46.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.342+0000 7f336e59c700 1 -- 172.21.15.23:0/1456664545 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3370071700 con 0x7f3370166380 2023-06-22T08:24:46.348 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.343+0000 7f336e59c700 1 --2- 172.21.15.23:0/1456664545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370166380 0x7f337006fcb0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3360014b30 tx=0x7f3360014e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77647047e87f0f86 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.343+0000 7f336d59a700 1 -- 172.21.15.23:0/1456664545 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360008d70 con 0x7f3370166380 2023-06-22T08:24:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.343+0000 7f336d59a700 1 -- 172.21.15.23:0/1456664545 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336001d440 con 0x7f3370166380 2023-06-22T08:24:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.343+0000 7f336d59a700 1 -- 172.21.15.23:0/1456664545 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336001d7d0 con 0x7f3370166380 2023-06-22T08:24:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.344+0000 7f3374fff700 1 -- 172.21.15.23:0/1456664545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370166380 msgr2=0x7f337006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.344+0000 7f3374fff700 1 --2- 172.21.15.23:0/1456664545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370166380 0x7f337006fcb0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3360014b30 tx=0x7f3360014e40 comp rx=0 tx=0).stop 2023-06-22T08:24:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.344+0000 7f3374fff700 1 -- 172.21.15.23:0/1456664545 shutdown_connections 2023-06-22T08:24:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.344+0000 7f3374fff700 1 --2- 172.21.15.23:0/1456664545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370166380 0x7f337006fcb0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.344+0000 7f3374fff700 1 --2- 172.21.15.23:0/1456664545 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3370069eb0 0x7f337006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.344+0000 7f3374fff700 1 -- 172.21.15.23:0/1456664545 >> 172.21.15.23:0/1456664545 conn(0x7f33700ca3f0 msgr2=0x7f33700ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.344+0000 7f3374fff700 1 -- 172.21.15.23:0/1456664545 shutdown_connections 2023-06-22T08:24:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.344+0000 7f3374fff700 1 -- 172.21.15.23:0/1456664545 wait complete. 2023-06-22T08:24:46.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.347+0000 7f3374fff700 1 Processor -- start 2023-06-22T08:24:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.347+0000 7f3374fff700 1 -- start start 2023-06-22T08:24:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.348+0000 7f3374fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370069eb0 0x7f33700c3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.348+0000 7f3374fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337006d530 0x7f337015c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.348+0000 7f3374fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3370166380 0x7f337015cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.348+0000 7f3374fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33700c3790 con 0x7f3370069eb0 2023-06-22T08:24:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.348+0000 7f3374fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33700c2dc0 con 0x7f337006d530 2023-06-22T08:24:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.348+0000 7f3374fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33700c1f40 con 0x7f3370166380 2023-06-22T08:24:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.349+0000 7f336e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370069eb0 0x7f33700c3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.349+0000 7f336e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370069eb0 0x7f33700c3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42994/0 (socket says 172.21.15.23:42994) 2023-06-22T08:24:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.349+0000 7f336e59c700 1 -- 172.21.15.23:0/641567743 learned_addr learned my addr 172.21.15.23:0/641567743 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.349+0000 7f336ed9d700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3370166380 0x7f337015cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.349+0000 7f336dd9b700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337006d530 0x7f337015c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.349+0000 7f336e59c700 1 -- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3370166380 msgr2=0x7f337015cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.349+0000 7f336e59c700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3370166380 0x7f337015cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:46.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.349+0000 7f336e59c700 1 -- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337006d530 msgr2=0x7f337015c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:46.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.349+0000 7f336e59c700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337006d530 0x7f337015c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:46.354 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.349+0000 7f336e59c700 1 -- 172.21.15.23:0/641567743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360014790 con 0x7f3370069eb0 2023-06-22T08:24:46.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.350+0000 7f336e59c700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370069eb0 0x7f33700c3250 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f336001ebc0 tx=0x7f3360004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:46.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.350+0000 7f335effd700 1 -- 172.21.15.23:0/641567743 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336002c070 con 0x7f3370069eb0 2023-06-22T08:24:46.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.350+0000 7f335effd700 1 -- 172.21.15.23:0/641567743 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3360004040 con 0x7f3370069eb0 2023-06-22T08:24:46.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.350+0000 7f335effd700 1 -- 172.21.15.23:0/641567743 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33600041b0 con 0x7f3370069eb0 2023-06-22T08:24:46.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.350+0000 7f3374fff700 1 -- 172.21.15.23:0/641567743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33700c9250 con 0x7f3370069eb0 2023-06-22T08:24:46.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.350+0000 7f3374fff700 1 -- 172.21.15.23:0/641567743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33700c2490 con 0x7f3370069eb0 2023-06-22T08:24:46.355 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.352+0000 7f3374fff700 1 -- 172.21.15.23:0/641567743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33700c4870 con 0x7f3370069eb0 2023-06-22T08:24:46.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.353+0000 7f335effd700 1 -- 172.21.15.23:0/641567743 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f33600036a0 con 0x7f3370069eb0 2023-06-22T08:24:46.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.354+0000 7f335effd700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f335006ace0 0x7f335006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:46.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.354+0000 7f335effd700 1 -- 172.21.15.23:0/641567743 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7f3360093570 con 0x7f3370069eb0 2023-06-22T08:24:46.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.354+0000 7f336dd9b700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f335006ace0 0x7f335006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:46.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.355+0000 7f336dd9b700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f335006ace0 0x7f335006d190 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f337015d5a0 tx=0x7f336400f040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:46.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.358+0000 7f335effd700 1 -- 172.21.15.23:0/641567743 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f336005de70 con 0x7f3370069eb0 2023-06-22T08:24:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.618+0000 7f3374fff700 1 -- 172.21.15.23:0/641567743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f337004ea20 con 0x7f3370069eb0 2023-06-22T08:24:46.621 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.619+0000 7f335effd700 1 -- 172.21.15.23:0/641567743 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4730 (secure 0 0 0) 0x7f3360061490 con 0x7f3370069eb0 2023-06-22T08:24:46.622 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:46.622 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:44.732071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 -- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f335006ace0 msgr2=0x7f335006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f335006ace0 0x7f335006d190 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f337015d5a0 tx=0x7f336400f040 comp rx=0 tx=0).stop 2023-06-22T08:24:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 -- 172.21.15.23:0/641567743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370069eb0 msgr2=0x7f33700c3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370069eb0 0x7f33700c3250 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f336001ebc0 tx=0x7f3360004750 comp rx=0 tx=0).stop 2023-06-22T08:24:46.624 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 -- 172.21.15.23:0/641567743 shutdown_connections 2023-06-22T08:24:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3370166380 0x7f337015cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3370069eb0 0x7f33700c3250 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f335006ace0 0x7f335006d190 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 --2- 172.21.15.23:0/641567743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337006d530 0x7f337015c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 -- 172.21.15.23:0/641567743 >> 172.21.15.23:0/641567743 conn(0x7f33700ca3f0 msgr2=0x7f337006a280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 -- 172.21.15.23:0/641567743 shutdown_connections 2023-06-22T08:24:46.625 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:46.621+0000 7f3374fff700 1 -- 172.21.15.23:0/641567743 wait complete. 2023-06-22T08:24:46.626 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 192 2023-06-22T08:24:46.643 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 192, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 192, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_6185": { "addr": "172.21.15.67:6836/2287946240", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 2287946240, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 2287946240, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6185, "incarnation": 192, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_6616": { "addr": "172.21.15.67:6837/1855579717", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1855579717, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1855579717, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6616, "incarnation": 162, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_6705": { "addr": "172.21.15.23:6835/3148174349", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 3148174349, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 3148174349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6705, "incarnation": 184, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 24 }, "gid_6730": { "addr": "172.21.15.23:6834/456927633", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 456927633, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 456927633, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6730, "incarnation": 172, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 111, "max_file_size": 1099511627776, "max_mds": 4, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:24:44.732071+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 6616, "mds_1": 6730, "mds_2": 6185, "mds_3": 6705 } } } ], "standbys": [] } 2023-06-22T08:24:46.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~264s 2023-06-22T08:24:46.802 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:24:46.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:24:48.644 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:48.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.852+0000 7fbeb63ff700 1 Processor -- start 2023-06-22T08:24:48.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.852+0000 7fbeb63ff700 1 -- start start 2023-06-22T08:24:48.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.853+0000 7fbeb63ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbeb015a320 0x7fbeb00c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:48.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbeb63ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb006d050 0x7fbeb0160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:48.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbeb63ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb00c6a60 con 0x7fbeb00c3a40 2023-06-22T08:24:48.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbeb63ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb0157570 con 0x7fbeb015a320 2023-06-22T08:24:48.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbeb63ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb00c4350 con 0x7fbeb006d050 2023-06-22T08:24:48.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbeb499c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb006d050 0x7fbeb0160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:48.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbea7fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbeb015a320 0x7fbeb00c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:48.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbea7fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbeb015a320 0x7fbeb00c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:44598/0 (socket says 172.21.15.23:44598) 2023-06-22T08:24:48.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbeaffff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbeb00c3a40 0x7fbeb0159c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:44926/0 (socket says 172.21.15.23:44926) 2023-06-22T08:24:48.858 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbea7fff700 1 -- 172.21.15.23:0/2641986304 learned_addr learned my addr 172.21.15.23:0/2641986304 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:48.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbeaf7fe700 1 -- 172.21.15.23:0/2641986304 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406651544 0 0) 0x7fbeb00c6a60 con 0x7fbeb00c3a40 2023-06-22T08:24:48.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbea7fff700 1 -- 172.21.15.23:0/2641986304 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb006d050 msgr2=0x7fbeb0160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:48.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbea7fff700 1 --2- 172.21.15.23:0/2641986304 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb006d050 0x7fbeb0160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:48.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbea7fff700 1 -- 172.21.15.23:0/2641986304 >> v1:172.21.15.67:6789/0 conn(0x7fbeb00c3a40 legacy=0x7fbeb0159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:48.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbea7fff700 1 -- 172.21.15.23:0/2641986304 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb014d690 con 0x7fbeb015a320 2023-06-22T08:24:48.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.854+0000 7fbeb499c700 1 --2- 172.21.15.23:0/2641986304 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb006d050 0x7fbeb0160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:24:48.859 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.855+0000 7fbea7fff700 1 --2- 172.21.15.23:0/2641986304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbeb015a320 0x7fbeb00c3e10 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fbe980098f0 tx=0x7fbe98009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d15a0851164b41a5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:48.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.855+0000 7fbeaf7fe700 1 -- 172.21.15.23:0/2641986304 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe98011830 con 0x7fbeb015a320 2023-06-22T08:24:48.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.855+0000 7fbeaf7fe700 1 -- 172.21.15.23:0/2641986304 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe98011990 con 0x7fbeb015a320 2023-06-22T08:24:48.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.855+0000 7fbeaf7fe700 1 -- 172.21.15.23:0/2641986304 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe98011d20 con 0x7fbeb015a320 2023-06-22T08:24:48.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.855+0000 7fbeb63ff700 1 -- 172.21.15.23:0/2641986304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbeb015a320 msgr2=0x7fbeb00c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:48.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.855+0000 7fbeb63ff700 1 --2- 172.21.15.23:0/2641986304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbeb015a320 0x7fbeb00c3e10 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fbe980098f0 tx=0x7fbe98009c00 comp rx=0 tx=0).stop 2023-06-22T08:24:48.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.856+0000 7fbeb63ff700 1 -- 172.21.15.23:0/2641986304 shutdown_connections 2023-06-22T08:24:48.860 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.856+0000 7fbeb63ff700 1 --2- 172.21.15.23:0/2641986304 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb006d050 0x7fbeb0160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:48.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.856+0000 7fbeb63ff700 1 --2- 172.21.15.23:0/2641986304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbeb015a320 0x7fbeb00c3e10 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:48.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.856+0000 7fbeb63ff700 1 -- 172.21.15.23:0/2641986304 >> 172.21.15.23:0/2641986304 conn(0x7fbeb00ca3f0 msgr2=0x7fbeb00ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:48.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.856+0000 7fbeb63ff700 1 -- 172.21.15.23:0/2641986304 shutdown_connections 2023-06-22T08:24:48.861 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.856+0000 7fbeb63ff700 1 -- 172.21.15.23:0/2641986304 wait complete. 2023-06-22T08:24:48.862 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.857+0000 7fbeb63ff700 1 Processor -- start 2023-06-22T08:24:48.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.858+0000 7fbeb63ff700 1 -- start start 2023-06-22T08:24:48.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.858+0000 7fbeb63ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbeb006d050 0x7fbeb0141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:48.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.858+0000 7fbeb63ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb00c3a40 0x7fbeb01eac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:48.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb63ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb015a320 0x7fbeb01476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:48.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb63ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb00c9640 con 0x7fbeb015a320 2023-06-22T08:24:48.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb63ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb01ea7b0 con 0x7fbeb006d050 2023-06-22T08:24:48.863 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb63ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb01edf20 con 0x7fbeb00c3a40 2023-06-22T08:24:48.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb499c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb015a320 0x7fbeb01476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:48.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb499c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb015a320 0x7fbeb01476f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42998/0 (socket says 172.21.15.23:42998) 2023-06-22T08:24:48.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb499c700 1 -- 172.21.15.23:0/289277160 learned_addr learned my addr 172.21.15.23:0/289277160 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:48.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeaffff700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbeb006d050 0x7fbeb0141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:48.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbea7fff700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb00c3a40 0x7fbeb01eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:48.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb499c700 1 -- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb00c3a40 msgr2=0x7fbeb01eac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:48.864 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb499c700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb00c3a40 0x7fbeb01eac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:48.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb499c700 1 -- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbeb006d050 msgr2=0x7fbeb0141220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:48.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb499c700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbeb006d050 0x7fbeb0141220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:48.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.859+0000 7fbeb499c700 1 -- 172.21.15.23:0/289277160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea8009a70 con 0x7fbeb015a320 2023-06-22T08:24:48.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.860+0000 7fbeb499c700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb015a320 0x7fbeb01476f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fbea800efc0 tx=0x7fbea800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:48.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.860+0000 7fbead7fa700 1 -- 172.21.15.23:0/289277160 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea800ca90 con 0x7fbeb015a320 2023-06-22T08:24:48.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.860+0000 7fbead7fa700 1 -- 172.21.15.23:0/289277160 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea800f040 con 0x7fbeb015a320 2023-06-22T08:24:48.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.860+0000 7fbeb63ff700 1 -- 172.21.15.23:0/289277160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe98009550 con 0x7fbeb015a320 2023-06-22T08:24:48.865 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.860+0000 7fbead7fa700 1 -- 172.21.15.23:0/289277160 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea800ce30 con 0x7fbeb015a320 2023-06-22T08:24:48.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.860+0000 7fbeb63ff700 1 -- 172.21.15.23:0/289277160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb01ee4e0 con 0x7fbeb015a320 2023-06-22T08:24:48.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.861+0000 7fbeb63ff700 1 -- 172.21.15.23:0/289277160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb01ea920 con 0x7fbeb015a320 2023-06-22T08:24:48.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.862+0000 7fbead7fa700 1 -- 172.21.15.23:0/289277160 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fbea8014480 con 0x7fbeb015a320 2023-06-22T08:24:48.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.863+0000 7fbead7fa700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbe9006ace0 0x7fbe9006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:48.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.863+0000 7fbead7fa700 1 -- 172.21.15.23:0/289277160 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7fbea808a3a0 con 0x7fbeb015a320 2023-06-22T08:24:48.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.863+0000 7fbeaffff700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbe9006ace0 0x7fbe9006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:48.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.864+0000 7fbeaffff700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbe9006ace0 0x7fbe9006d190 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fbea0007fd0 tx=0x7fbea0007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:48.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:48.867+0000 7fbead7fa700 1 -- 172.21.15.23:0/289277160 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fbea8054b70 con 0x7fbeb015a320 2023-06-22T08:24:49.126 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.124+0000 7fbeb63ff700 1 -- 172.21.15.23:0/289277160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbeb004ea20 con 0x7fbeb015a320 2023-06-22T08:24:49.127 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.124+0000 7fbead7fa700 1 -- 172.21.15.23:0/289277160 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4730 (secure 0 0 0) 0x7fbea8018090 con 0x7fbeb015a320 2023-06-22T08:24:49.127 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:49.127 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:44.732071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:49.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 -- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbe9006ace0 msgr2=0x7fbe9006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:49.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbe9006ace0 0x7fbe9006d190 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fbea0007fd0 tx=0x7fbea0007de0 comp rx=0 tx=0).stop 2023-06-22T08:24:49.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 -- 172.21.15.23:0/289277160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb015a320 msgr2=0x7fbeb01476f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:49.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb015a320 0x7fbeb01476f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fbea800efc0 tx=0x7fbea800c3a0 comp rx=0 tx=0).stop 2023-06-22T08:24:49.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 -- 172.21.15.23:0/289277160 shutdown_connections 2023-06-22T08:24:49.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb015a320 0x7fbeb01476f0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:49.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fbe9006ace0 0x7fbe9006d190 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:49.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbeb006d050 0x7fbeb0141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:49.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 --2- 172.21.15.23:0/289277160 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbeb00c3a40 0x7fbeb01eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:49.131 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 -- 172.21.15.23:0/289277160 >> 172.21.15.23:0/289277160 conn(0x7fbeb00ca3f0 msgr2=0x7fbeb00c6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:49.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 -- 172.21.15.23:0/289277160 shutdown_connections 2023-06-22T08:24:49.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:49.127+0000 7fbeb63ff700 1 -- 172.21.15.23:0/289277160 wait complete. 2023-06-22T08:24:49.132 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 192 2023-06-22T08:24:49.148 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:24:51.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:51.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.367+0000 7fb0c21ff700 1 Processor -- start 2023-06-22T08:24:51.369 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.367+0000 7fb0c21ff700 1 -- start start 2023-06-22T08:24:51.370 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.368+0000 7fb0c21ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb0bc0c9200 0x7fb0bc0c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:51.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.368+0000 7fb0c21ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb0bc15c550 0x7fb0bc0cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:51.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.368+0000 7fb0c21ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0bc0cbd60 con 0x7fb0bc0cb260 2023-06-22T08:24:51.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.368+0000 7fb0c21ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0bc0cbb70 con 0x7fb0bc0c9200 2023-06-22T08:24:51.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.368+0000 7fb0c21ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0bc158980 con 0x7fb0bc15c550 2023-06-22T08:24:51.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.368+0000 7fb0bb7fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb0bc0cb260 0x7fb0bc15be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:44940/0 (socket says 172.21.15.23:44940) 2023-06-22T08:24:51.371 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.368+0000 7fb0bb7fe700 1 -- 172.21.15.23:0/1017204607 learned_addr learned my addr 172.21.15.23:0/1017204607 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:51.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.368+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/1017204607 >> v2:172.21.15.67:3300/0 conn(0x7fb0bc15c550 0x7fb0bc0cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:51.372 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.368+0000 7fb0baffd700 1 --2- 172.21.15.23:0/1017204607 >> v2:172.21.15.23:3300/0 conn(0x7fb0bc0c9200 0x7fb0bc0c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:51.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0ba7fc700 1 -- 172.21.15.23:0/1017204607 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587335205 0 0) 0x7fb0bc0cbd60 con 0x7fb0bc0cb260 2023-06-22T08:24:51.373 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0ba7fc700 1 -- 172.21.15.23:0/1017204607 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a40035b0 con 0x7fb0bc0cb260 2023-06-22T08:24:51.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0baffd700 1 -- 172.21.15.23:0/1017204607 >> v2:172.21.15.67:3300/0 conn(0x7fb0bc15c550 msgr2=0x7fb0bc0cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:51.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0baffd700 1 --2- 172.21.15.23:0/1017204607 >> v2:172.21.15.67:3300/0 conn(0x7fb0bc15c550 0x7fb0bc0cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:51.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0baffd700 1 -- 172.21.15.23:0/1017204607 >> v1:172.21.15.67:6789/0 conn(0x7fb0bc0cb260 legacy=0x7fb0bc15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:51.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0baffd700 1 -- 172.21.15.23:0/1017204607 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0bc159550 con 0x7fb0bc0c9200 2023-06-22T08:24:51.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/1017204607 >> v2:172.21.15.67:3300/0 conn(0x7fb0bc15c550 0x7fb0bc0cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:24:51.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0baffd700 1 --2- 172.21.15.23:0/1017204607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc0c9200 0x7fb0bc0c4a50 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc08c8f0 tx=0x7fb0b0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8251c81985fcc6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:51.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0ba7fc700 1 -- 172.21.15.23:0/1017204607 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0011830 con 0x7fb0bc0c9200 2023-06-22T08:24:51.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0ba7fc700 1 -- 172.21.15.23:0/1017204607 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b0011990 con 0x7fb0bc0c9200 2023-06-22T08:24:51.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.369+0000 7fb0ba7fc700 1 -- 172.21.15.23:0/1017204607 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0011d20 con 0x7fb0bc0c9200 2023-06-22T08:24:51.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.370+0000 7fb0c21ff700 1 -- 172.21.15.23:0/1017204607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc0c9200 msgr2=0x7fb0bc0c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:51.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.370+0000 7fb0c21ff700 1 --2- 172.21.15.23:0/1017204607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc0c9200 0x7fb0bc0c4a50 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc08c8f0 tx=0x7fb0b0009b20 comp rx=0 tx=0).stop 2023-06-22T08:24:51.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.370+0000 7fb0c21ff700 1 -- 172.21.15.23:0/1017204607 shutdown_connections 2023-06-22T08:24:51.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.370+0000 7fb0c21ff700 1 --2- 172.21.15.23:0/1017204607 >> v2:172.21.15.67:3300/0 conn(0x7fb0bc15c550 0x7fb0bc0cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:51.375 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.370+0000 7fb0c21ff700 1 --2- 172.21.15.23:0/1017204607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc0c9200 0x7fb0bc0c4a50 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:51.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.370+0000 7fb0c21ff700 1 -- 172.21.15.23:0/1017204607 >> 172.21.15.23:0/1017204607 conn(0x7fb0bc065b90 msgr2=0x7fb0bc065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:51.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.370+0000 7fb0c21ff700 1 -- 172.21.15.23:0/1017204607 shutdown_connections 2023-06-22T08:24:51.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.370+0000 7fb0c21ff700 1 -- 172.21.15.23:0/1017204607 wait complete. 2023-06-22T08:24:51.377 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.372+0000 7fb0c21ff700 1 Processor -- start 2023-06-22T08:24:51.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.373+0000 7fb0c21ff700 1 -- start start 2023-06-22T08:24:51.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0c21ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc0c9200 0x7fb0bc0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:51.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0c21ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc0cb260 0x7fb0bc0b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:51.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0c21ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0bc15c550 0x7fb0bc14c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:51.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0c21ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc158240 con 0x7fb0bc0cb260 2023-06-22T08:24:51.378 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0c21ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc15a7a0 con 0x7fb0bc0c9200 2023-06-22T08:24:51.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0c21ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0bc157750 con 0x7fb0bc15c550 2023-06-22T08:24:51.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0bb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc0c9200 0x7fb0bc0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:51.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0bb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc0c9200 0x7fb0bc0b2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:44620/0 (socket says 172.21.15.23:44620) 2023-06-22T08:24:51.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0bb7fe700 1 -- 172.21.15.23:0/2742068658 learned_addr learned my addr 172.21.15.23:0/2742068658 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:51.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0baffd700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc0cb260 0x7fb0bc0b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:51.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.374+0000 7fb0bbfff700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0bc15c550 0x7fb0bc14c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:51.379 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.375+0000 7fb0bb7fe700 1 -- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0bc15c550 msgr2=0x7fb0bc14c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:51.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.375+0000 7fb0bb7fe700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0bc15c550 0x7fb0bc14c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:51.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.375+0000 7fb0bb7fe700 1 -- 172.21.15.23:0/2742068658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc0cb260 msgr2=0x7fb0bc0b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:51.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.375+0000 7fb0bb7fe700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc0cb260 0x7fb0bc0b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:51.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.375+0000 7fb0bb7fe700 1 -- 172.21.15.23:0/2742068658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b4009a70 con 0x7fb0bc0c9200 2023-06-22T08:24:51.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.375+0000 7fb0baffd700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc0cb260 0x7fb0bc0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:24:51.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.375+0000 7fb0bb7fe700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc0c9200 0x7fb0bc0b2c40 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00c730 tx=0x7fb0ac00ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:51.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.376+0000 7fb0a3fff700 1 -- 172.21.15.23:0/2742068658 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac005d70 con 0x7fb0bc0c9200 2023-06-22T08:24:51.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.376+0000 7fb0a3fff700 1 -- 172.21.15.23:0/2742068658 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ac0070d0 con 0x7fb0bc0c9200 2023-06-22T08:24:51.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.376+0000 7fb0c21ff700 1 -- 172.21.15.23:0/2742068658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b00094a0 con 0x7fb0bc0c9200 2023-06-22T08:24:51.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.376+0000 7fb0a3fff700 1 -- 172.21.15.23:0/2742068658 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac007460 con 0x7fb0bc0c9200 2023-06-22T08:24:51.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.376+0000 7fb0c21ff700 1 -- 172.21.15.23:0/2742068658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0bc0b21f0 con 0x7fb0bc0c9200 2023-06-22T08:24:51.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.377+0000 7fb0c21ff700 1 -- 172.21.15.23:0/2742068658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0bc04ea20 con 0x7fb0bc0c9200 2023-06-22T08:24:51.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.379+0000 7fb0a3fff700 1 -- 172.21.15.23:0/2742068658 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb0ac0075c0 con 0x7fb0bc0c9200 2023-06-22T08:24:51.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.380+0000 7fb0a3fff700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0a806acb0 0x7fb0a806d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:51.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.380+0000 7fb0a3fff700 1 -- 172.21.15.23:0/2742068658 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7fb0ac089570 con 0x7fb0bc0c9200 2023-06-22T08:24:51.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.380+0000 7fb0baffd700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0a806acb0 0x7fb0a806d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:51.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.381+0000 7fb0baffd700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0a806acb0 0x7fb0a806d160 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00090b0 tx=0x7fb0b0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:51.389 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.387+0000 7fb0a3fff700 1 -- 172.21.15.23:0/2742068658 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb0ac053dd0 con 0x7fb0bc0c9200 2023-06-22T08:24:51.653 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.651+0000 7fb0c21ff700 1 -- 172.21.15.23:0/2742068658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0bc0c3c10 con 0x7fb0bc0c9200 2023-06-22T08:24:51.654 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.652+0000 7fb0a3fff700 1 -- 172.21.15.23:0/2742068658 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4730 (secure 0 0 0) 0x7fb0ac017090 con 0x7fb0bc0c9200 2023-06-22T08:24:51.655 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:51.655 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:44.732071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:replay","state_seq":1,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:51.657 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 -- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0a806acb0 msgr2=0x7fb0a806d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0a806acb0 0x7fb0a806d160 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00090b0 tx=0x7fb0b0008030 comp rx=0 tx=0).stop 2023-06-22T08:24:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 -- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc0c9200 msgr2=0x7fb0bc0b2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc0c9200 0x7fb0bc0b2c40 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00c730 tx=0x7fb0ac00ca40 comp rx=0 tx=0).stop 2023-06-22T08:24:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 -- 172.21.15.23:0/2742068658 shutdown_connections 2023-06-22T08:24:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb0bc15c550 0x7fb0bc14c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb0a806acb0 0x7fb0a806d160 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:51.658 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0bc0c9200 0x7fb0bc0b2c40 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 --2- 172.21.15.23:0/2742068658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0bc0cb260 0x7fb0bc0b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 -- 172.21.15.23:0/2742068658 >> 172.21.15.23:0/2742068658 conn(0x7fb0bc065b90 msgr2=0x7fb0bc150e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 -- 172.21.15.23:0/2742068658 shutdown_connections 2023-06-22T08:24:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:51.655+0000 7fb0c21ff700 1 -- 172.21.15.23:0/2742068658 wait complete. 2023-06-22T08:24:51.659 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 192 2023-06-22T08:24:51.676 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:24:53.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~271s 2023-06-22T08:24:53.678 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.886+0000 7fb9bd3ff700 1 Processor -- start 2023-06-22T08:24:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.886+0000 7fb9bd3ff700 1 -- start start 2023-06-22T08:24:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.887+0000 7fb9bd3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb9b80cf1b0 0x7fb9b80c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.887+0000 7fb9bd3ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b815c370 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.887+0000 7fb9bd3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b815bda0 con 0x7fb9b8152ce0 2023-06-22T08:24:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.887+0000 7fb9bd3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b81530b0 con 0x7fb9b80cf1b0 2023-06-22T08:24:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.888+0000 7fb9b77fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb9b80cf1b0 0x7fb9b80c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:53.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.888+0000 7fb9b67fc700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb9b80c9fd0 0x7fb9b80c40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:46698/0 (socket says 172.21.15.23:46698) 2023-06-22T08:24:53.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.888+0000 7fb9b67fc700 1 -- 172.21.15.23:0/1648799316 learned_addr learned my addr 172.21.15.23:0/1648799316 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:53.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.888+0000 7fb9b77fe700 1 -- 172.21.15.23:0/1648799316 >> v1:172.21.15.67:6789/0 conn(0x7fb9b8152ce0 legacy=0x7fb9b815c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:53.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.888+0000 7fb9b77fe700 1 -- 172.21.15.23:0/1648799316 >> v1:172.21.15.23:6789/0 conn(0x7fb9b80c9fd0 legacy=0x7fb9b80c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:53.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.888+0000 7fb9b77fe700 1 -- 172.21.15.23:0/1648799316 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b80cf630 con 0x7fb9b80cf1b0 2023-06-22T08:24:53.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.888+0000 7fb9b77fe700 1 --2- 172.21.15.23:0/1648799316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80cf1b0 0x7fb9b80c47c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009d90 tx=0x7fb9b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6113a66d1e9066 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:53.892 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.889+0000 7fb9b5ffb700 1 -- 172.21.15.23:0/1648799316 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b000e070 con 0x7fb9b80cf1b0 2023-06-22T08:24:53.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.889+0000 7fb9b5ffb700 1 -- 172.21.15.23:0/1648799316 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b000bbd0 con 0x7fb9b80cf1b0 2023-06-22T08:24:53.893 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.889+0000 7fb9b5ffb700 1 -- 172.21.15.23:0/1648799316 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0012430 con 0x7fb9b80cf1b0 2023-06-22T08:24:53.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.889+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/1648799316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80cf1b0 msgr2=0x7fb9b80c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:53.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.889+0000 7fb9bd3ff700 1 --2- 172.21.15.23:0/1648799316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80cf1b0 0x7fb9b80c47c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009d90 tx=0x7fb9b000b330 comp rx=0 tx=0).stop 2023-06-22T08:24:53.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.889+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/1648799316 shutdown_connections 2023-06-22T08:24:53.894 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.889+0000 7fb9bd3ff700 1 --2- 172.21.15.23:0/1648799316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80cf1b0 0x7fb9b80c47c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.889+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/1648799316 >> 172.21.15.23:0/1648799316 conn(0x7fb9b806ae30 msgr2=0x7fb9b806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.889+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/1648799316 shutdown_connections 2023-06-22T08:24:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.889+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/1648799316 wait complete. 2023-06-22T08:24:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.892+0000 7fb9bd3ff700 1 Processor -- start 2023-06-22T08:24:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.892+0000 7fb9bd3ff700 1 -- start start 2023-06-22T08:24:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.893+0000 7fb9bd3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80c9fd0 0x7fb9b8155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.893+0000 7fb9bd3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b80cf1b0 0x7fb9b81f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.893+0000 7fb9bd3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b8152ce0 0x7fb9b8153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.893+0000 7fb9bd3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b8155080 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.893+0000 7fb9bd3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b81558a0 con 0x7fb9b8152ce0 2023-06-22T08:24:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.893+0000 7fb9bd3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b8153740 con 0x7fb9b80cf1b0 2023-06-22T08:24:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.893+0000 7fb9b6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80c9fd0 0x7fb9b8155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.893+0000 7fb9b6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80c9fd0 0x7fb9b8155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43038/0 (socket says 172.21.15.23:43038) 2023-06-22T08:24:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.893+0000 7fb9b6ffd700 1 -- 172.21.15.23:0/2435921707 learned_addr learned my addr 172.21.15.23:0/2435921707 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb9b67fc700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b80cf1b0 0x7fb9b81f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb9b77fe700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b8152ce0 0x7fb9b8153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb9b6ffd700 1 -- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b80cf1b0 msgr2=0x7fb9b81f19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb9b6ffd700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b80cf1b0 0x7fb9b81f19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb9b6ffd700 1 -- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b8152ce0 msgr2=0x7fb9b8153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb9b6ffd700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b8152ce0 0x7fb9b8153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb9b6ffd700 1 -- 172.21.15.23:0/2435921707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b0009a70 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb9b67fc700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b80cf1b0 0x7fb9b81f19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:24:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb9b6ffd700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80c9fd0 0x7fb9b8155360 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800c450 tx=0x7fb9a800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb99f7fe700 1 -- 172.21.15.23:0/2435921707 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a80052d0 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb99f7fe700 1 -- 172.21.15.23:0/2435921707 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a8005430 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/2435921707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b81f24b0 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.894+0000 7fb99f7fe700 1 -- 172.21.15.23:0/2435921707 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a80038c0 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.895+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/2435921707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b80ce470 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.895+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/2435921707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b804ea20 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.897+0000 7fb99f7fe700 1 -- 172.21.15.23:0/2435921707 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fb9a8003a20 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.897+0000 7fb99f7fe700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb9a406ace0 0x7fb9a406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.898+0000 7fb99f7fe700 1 -- 172.21.15.23:0/2435921707 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7fb9a8088ff0 con 0x7fb9b80c9fd0 2023-06-22T08:24:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.898+0000 7fb9b67fc700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb9a406ace0 0x7fb9a406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.899+0000 7fb9b67fc700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb9a406ace0 0x7fb9a406d190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8150410 tx=0x7fb9ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:53.902+0000 7fb99f7fe700 1 -- 172.21.15.23:0/2435921707 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fb9a80bb310 con 0x7fb9b80c9fd0 2023-06-22T08:24:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.156+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/2435921707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9b80b13d0 con 0x7fb9b80c9fd0 2023-06-22T08:24:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.156+0000 7fb99f7fe700 1 -- 172.21.15.23:0/2435921707 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+4732 (secure 0 0 0) 0x7fb9a8017090 con 0x7fb9b80c9fd0 2023-06-22T08:24:54.159 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:54.159 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:52.087011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:resolve","state_seq":13,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb9a406ace0 msgr2=0x7fb9a406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb9a406ace0 0x7fb9a406d190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8150410 tx=0x7fb9ac009040 comp rx=0 tx=0).stop 2023-06-22T08:24:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/2435921707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80c9fd0 msgr2=0x7fb9b8155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:54.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80c9fd0 0x7fb9b8155360 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800c450 tx=0x7fb9a800c760 comp rx=0 tx=0).stop 2023-06-22T08:24:54.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/2435921707 shutdown_connections 2023-06-22T08:24:54.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fb9a406ace0 0x7fb9a406d190 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:54.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9b8152ce0 0x7fb9b8153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:54.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9b80c9fd0 0x7fb9b8155360 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:54.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 --2- 172.21.15.23:0/2435921707 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb9b80cf1b0 0x7fb9b81f19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:54.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/2435921707 >> 172.21.15.23:0/2435921707 conn(0x7fb9b806ae30 msgr2=0x7fb9b814d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:54.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/2435921707 shutdown_connections 2023-06-22T08:24:54.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:54.159+0000 7fb9bd3ff700 1 -- 172.21.15.23:0/2435921707 wait complete. 2023-06-22T08:24:54.163 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 193 2023-06-22T08:24:54.180 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:24:56.181 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:56.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.389+0000 7f0d23fff700 1 Processor -- start 2023-06-22T08:24:56.392 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.389+0000 7f0d23fff700 1 -- start start 2023-06-22T08:24:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.390+0000 7f0d23fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0d1c0cb6b0 0x7f0d1c069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d23fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0d1c0c5740 0x7f0d1c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d23fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0d1c0cba80 0x7f0d1c159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d23fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c06bd20 con 0x7f0d1c0cb6b0 2023-06-22T08:24:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d23fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c158f60 con 0x7f0d1c0cba80 2023-06-22T08:24:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d23fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0cc450 con 0x7f0d1c0c5740 2023-06-22T08:24:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d21d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0d1c0cb6b0 0x7f0d1c069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d21d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0d1c0cb6b0 0x7f0d1c069eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60038/0 (socket says 172.21.15.23:60038) 2023-06-22T08:24:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d21d9b700 1 -- 172.21.15.23:0/1523928615 learned_addr learned my addr 172.21.15.23:0/1523928615 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d2259c700 1 --2- 172.21.15.23:0/1523928615 >> v2:172.21.15.67:3300/0 conn(0x7f0d1c0cba80 0x7f0d1c159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d2159a700 1 --2- 172.21.15.23:0/1523928615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0d1c0c5740 0x7f0d1c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d21d9b700 1 -- 172.21.15.23:0/1523928615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0d1c0c5740 msgr2=0x7f0d1c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d21d9b700 1 --2- 172.21.15.23:0/1523928615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0d1c0c5740 0x7f0d1c1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d21d9b700 1 -- 172.21.15.23:0/1523928615 >> v2:172.21.15.67:3300/0 conn(0x7f0d1c0cba80 msgr2=0x7f0d1c159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:24:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d21d9b700 1 --2- 172.21.15.23:0/1523928615 >> v2:172.21.15.67:3300/0 conn(0x7f0d1c0cba80 0x7f0d1c159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.391+0000 7f0d21d9b700 1 -- 172.21.15.23:0/1523928615 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c158980 con 0x7f0d1c0cb6b0 2023-06-22T08:24:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.392+0000 7f0d21d9b700 1 --2- 172.21.15.23:0/1523928615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0cb6b0 0x7f0d1c069eb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0c5180 tx=0x7f0d18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ee38836bcb17b9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.392+0000 7f0d20d99700 1 -- 172.21.15.23:0/1523928615 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18011830 con 0x7f0d1c0cb6b0 2023-06-22T08:24:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.392+0000 7f0d20d99700 1 -- 172.21.15.23:0/1523928615 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d18011990 con 0x7f0d1c0cb6b0 2023-06-22T08:24:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.392+0000 7f0d20d99700 1 -- 172.21.15.23:0/1523928615 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d18011d20 con 0x7f0d1c0cb6b0 2023-06-22T08:24:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.392+0000 7f0d23fff700 1 -- 172.21.15.23:0/1523928615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0cb6b0 msgr2=0x7f0d1c069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.392+0000 7f0d23fff700 1 --2- 172.21.15.23:0/1523928615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0cb6b0 0x7f0d1c069eb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0c5180 tx=0x7f0d18009bd0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.393+0000 7f0d23fff700 1 -- 172.21.15.23:0/1523928615 shutdown_connections 2023-06-22T08:24:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.393+0000 7f0d23fff700 1 --2- 172.21.15.23:0/1523928615 >> v2:172.21.15.67:3300/0 conn(0x7f0d1c0cba80 0x7f0d1c159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.393+0000 7f0d23fff700 1 --2- 172.21.15.23:0/1523928615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0cb6b0 0x7f0d1c069eb0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.393+0000 7f0d23fff700 1 --2- 172.21.15.23:0/1523928615 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0d1c0c5740 0x7f0d1c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.393+0000 7f0d23fff700 1 -- 172.21.15.23:0/1523928615 >> 172.21.15.23:0/1523928615 conn(0x7f0d1c070d80 msgr2=0x7f0d1c071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.393+0000 7f0d23fff700 1 -- 172.21.15.23:0/1523928615 shutdown_connections 2023-06-22T08:24:56.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.393+0000 7f0d23fff700 1 -- 172.21.15.23:0/1523928615 wait complete. 2023-06-22T08:24:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.394+0000 7f0d23fff700 1 Processor -- start 2023-06-22T08:24:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.395+0000 7f0d23fff700 1 -- start start 2023-06-22T08:24:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.395+0000 7f0d23fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0cb6b0 0x7f0d1c157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.395+0000 7f0d23fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d1c0cba80 0x7f0d1c0c3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.395+0000 7f0d23fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0d1c1f25e0 0x7f0d1c14aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.395+0000 7f0d23fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c14d110 con 0x7f0d1c0cb6b0 2023-06-22T08:24:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.395+0000 7f0d23fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c158240 con 0x7f0d1c0cba80 2023-06-22T08:24:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.395+0000 7f0d23fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c1567f0 con 0x7f0d1c1f25e0 2023-06-22T08:24:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d21d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0cb6b0 0x7f0d1c157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d2159a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d1c0cba80 0x7f0d1c0c3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d2159a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d1c0cba80 0x7f0d1c0c3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:37380/0 (socket says 172.21.15.23:37380) 2023-06-22T08:24:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d2159a700 1 -- 172.21.15.23:0/2212551975 learned_addr learned my addr 172.21.15.23:0/2212551975 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d2259c700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0d1c1f25e0 0x7f0d1c14aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d2159a700 1 -- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0d1c1f25e0 msgr2=0x7f0d1c14aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d2159a700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0d1c1f25e0 0x7f0d1c14aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d2159a700 1 -- 172.21.15.23:0/2212551975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0cb6b0 msgr2=0x7f0d1c157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d2159a700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0cb6b0 0x7f0d1c157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d2159a700 1 -- 172.21.15.23:0/2212551975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d18009550 con 0x7f0d1c0cba80 2023-06-22T08:24:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.396+0000 7f0d2159a700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d1c0cba80 0x7f0d1c0c3430 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e9b0 tx=0x7f0d0c00ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.397+0000 7f0d127fc700 1 -- 172.21.15.23:0/2212551975 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c009710 con 0x7f0d1c0cba80 2023-06-22T08:24:56.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.397+0000 7f0d127fc700 1 -- 172.21.15.23:0/2212551975 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c00ee50 con 0x7f0d1c0cba80 2023-06-22T08:24:56.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.397+0000 7f0d127fc700 1 -- 172.21.15.23:0/2212551975 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c00ca50 con 0x7f0d1c0cba80 2023-06-22T08:24:56.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.397+0000 7f0d23fff700 1 -- 172.21.15.23:0/2212551975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c156a90 con 0x7f0d1c0cba80 2023-06-22T08:24:56.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.397+0000 7f0d23fff700 1 -- 172.21.15.23:0/2212551975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c1f1de0 con 0x7f0d1c0cba80 2023-06-22T08:24:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.398+0000 7f0d23fff700 1 -- 172.21.15.23:0/2212551975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c1f2430 con 0x7f0d1c0cba80 2023-06-22T08:24:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.399+0000 7f0d127fc700 1 -- 172.21.15.23:0/2212551975 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0d0c00cbb0 con 0x7f0d1c0cba80 2023-06-22T08:24:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.400+0000 7f0d127fc700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0d0006ace0 0x7f0d0006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.400+0000 7f0d127fc700 1 -- 172.21.15.23:0/2212551975 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7f0d0c089d30 con 0x7f0d1c0cba80 2023-06-22T08:24:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.400+0000 7f0d21d9b700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0d0006ace0 0x7f0d0006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.401+0000 7f0d21d9b700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0d0006ace0 0x7f0d0006d190 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009c10 tx=0x7f0d18008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.405+0000 7f0d127fc700 1 -- 172.21.15.23:0/2212551975 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0d0c057f90 con 0x7f0d1c0cba80 2023-06-22T08:24:56.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.658+0000 7f0d23fff700 1 -- 172.21.15.23:0/2212551975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d1c1f1960 con 0x7f0d1c0cba80 2023-06-22T08:24:56.661 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.659+0000 7f0d127fc700 1 -- 172.21.15.23:0/2212551975 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+4732 (secure 0 0 0) 0x7f0d1c1f1960 con 0x7f0d1c0cba80 2023-06-22T08:24:56.662 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:56.662 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:52.087011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:resolve","state_seq":13,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:56.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 -- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0d0006ace0 msgr2=0x7f0d0006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:56.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0d0006ace0 0x7f0d0006d190 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009c10 tx=0x7f0d18008030 comp rx=0 tx=0).stop 2023-06-22T08:24:56.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 -- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d1c0cba80 msgr2=0x7f0d1c0c3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:56.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d1c0cba80 0x7f0d1c0c3430 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e9b0 tx=0x7f0d0c00ecc0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.665 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 -- 172.21.15.23:0/2212551975 shutdown_connections 2023-06-22T08:24:56.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0d1c1f25e0 0x7f0d1c14aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d1c0cb6b0 0x7f0d1c157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f0d0006ace0 0x7f0d0006d190 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 --2- 172.21.15.23:0/2212551975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d1c0cba80 0x7f0d1c0c3430 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:56.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 -- 172.21.15.23:0/2212551975 >> 172.21.15.23:0/2212551975 conn(0x7f0d1c070d80 msgr2=0x7f0d1c0c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:56.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 -- 172.21.15.23:0/2212551975 shutdown_connections 2023-06-22T08:24:56.666 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:56.662+0000 7f0d23fff700 1 -- 172.21.15.23:0/2212551975 wait complete. 2023-06-22T08:24:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 193 2023-06-22T08:24:56.683 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:24:58.684 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:24:58.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.893+0000 7f4bdc3ff700 1 Processor -- start 2023-06-22T08:24:58.896 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.894+0000 7f4bdc3ff700 1 -- start start 2023-06-22T08:24:58.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.895+0000 7f4bdc3ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4bd4152d70 0x7f4bd40c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:58.897 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.895+0000 7f4bdc3ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd4159790 con 0x7f4bd40c45f0 2023-06-22T08:24:58.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.895+0000 7f4bdc3ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd415bea0 con 0x7f4bd4159330 2023-06-22T08:24:58.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.895+0000 7f4bdc3ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd40c95e0 con 0x7f4bd4152d70 2023-06-22T08:24:58.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.895+0000 7f4bda19b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4bd4152d70 0x7f4bd40c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:58.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.895+0000 7f4bda99c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4bd40c45f0 0x7f4bd40c4a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34560/0 (socket says 172.21.15.23:34560) 2023-06-22T08:24:58.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.895+0000 7f4bda99c700 1 -- 172.21.15.23:0/3688957134 learned_addr learned my addr 172.21.15.23:0/3688957134 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:24:58.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.896+0000 7f4bd9199700 1 -- 172.21.15.23:0/3688957134 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758531784 0 0) 0x7f4bd4159790 con 0x7f4bd40c45f0 2023-06-22T08:24:58.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.896+0000 7f4bd9199700 1 -- 172.21.15.23:0/3688957134 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc00035b0 con 0x7f4bd40c45f0 2023-06-22T08:24:58.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.896+0000 7f4bd9199700 1 -- 172.21.15.23:0/3688957134 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970362754 0 0) 0x7f4bd415bea0 con 0x7f4bd4159330 2023-06-22T08:24:58.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.896+0000 7f4bda19b700 1 -- 172.21.15.23:0/3688957134 >> v1:172.21.15.67:6789/0 conn(0x7f4bd4159330 legacy=0x7f4bd40cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:58.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.896+0000 7f4bda19b700 1 -- 172.21.15.23:0/3688957134 >> v1:172.21.15.23:6789/0 conn(0x7f4bd40c45f0 legacy=0x7f4bd40c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:58.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.896+0000 7f4bda19b700 1 -- 172.21.15.23:0/3688957134 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd415c160 con 0x7f4bd4152d70 2023-06-22T08:24:58.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.896+0000 7f4bda19b700 1 --2- 172.21.15.23:0/3688957134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd4152d70 0x7f4bd40c97d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009840 tx=0x7f4bd0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38138fa7b82b3b62 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:58.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.896+0000 7f4bd9199700 1 -- 172.21.15.23:0/3688957134 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0011830 con 0x7f4bd4152d70 2023-06-22T08:24:58.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.896+0000 7f4bd9199700 1 -- 172.21.15.23:0/3688957134 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd0011990 con 0x7f4bd4152d70 2023-06-22T08:24:58.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.896+0000 7f4bd9199700 1 -- 172.21.15.23:0/3688957134 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0011d20 con 0x7f4bd4152d70 2023-06-22T08:24:58.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.897+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/3688957134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd4152d70 msgr2=0x7f4bd40c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:58.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.897+0000 7f4bdc3ff700 1 --2- 172.21.15.23:0/3688957134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd4152d70 0x7f4bd40c97d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009840 tx=0x7f4bd0009b50 comp rx=0 tx=0).stop 2023-06-22T08:24:58.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.897+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/3688957134 shutdown_connections 2023-06-22T08:24:58.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.897+0000 7f4bdc3ff700 1 --2- 172.21.15.23:0/3688957134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd4152d70 0x7f4bd40c97d0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:58.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.897+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/3688957134 >> 172.21.15.23:0/3688957134 conn(0x7f4bd406ae30 msgr2=0x7f4bd406b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:58.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.897+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/3688957134 shutdown_connections 2023-06-22T08:24:58.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.897+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/3688957134 wait complete. 2023-06-22T08:24:58.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.899+0000 7f4bdc3ff700 1 Processor -- start 2023-06-22T08:24:58.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.899+0000 7f4bdc3ff700 1 -- start start 2023-06-22T08:24:58.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bdc3ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd40c45f0 0x7f4bd414b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:58.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bdc3ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4bd4152d70 0x7f4bd4151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:58.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bdc3ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd4159330 0x7f4bd40b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:58.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bdc3ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd4148030 con 0x7f4bd40c45f0 2023-06-22T08:24:58.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bdc3ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd4151900 con 0x7f4bd4159330 2023-06-22T08:24:58.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bdc3ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd4147380 con 0x7f4bd4152d70 2023-06-22T08:24:58.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bda19b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd40c45f0 0x7f4bd414b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:58.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bda19b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd40c45f0 0x7f4bd414b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60068/0 (socket says 172.21.15.23:60068) 2023-06-22T08:24:58.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bda19b700 1 -- 172.21.15.23:0/4189644922 learned_addr learned my addr 172.21.15.23:0/4189644922 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:24:58.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bda99c700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd4159330 0x7f4bd40b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:58.905 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bd999a700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4bd4152d70 0x7f4bd4151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:58.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bda19b700 1 -- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4bd4152d70 msgr2=0x7f4bd4151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:58.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bda19b700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4bd4152d70 0x7f4bd4151d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:58.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bda19b700 1 -- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd4159330 msgr2=0x7f4bd40b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:58.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bda19b700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd4159330 0x7f4bd40b28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:58.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bda19b700 1 -- 172.21.15.23:0/4189644922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd00094a0 con 0x7f4bd40c45f0 2023-06-22T08:24:58.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bd999a700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4bd4152d70 0x7f4bd4151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:24:58.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.900+0000 7f4bda99c700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd4159330 0x7f4bd40b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T08:24:58.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.901+0000 7f4bda19b700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd40c45f0 0x7f4bd414b6b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00090b0 tx=0x7f4bd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:58.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.901+0000 7f4bcaffd700 1 -- 172.21.15.23:0/4189644922 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd001a910 con 0x7f4bd40c45f0 2023-06-22T08:24:58.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.901+0000 7f4bcaffd700 1 -- 172.21.15.23:0/4189644922 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd001aa70 con 0x7f4bd40c45f0 2023-06-22T08:24:58.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.901+0000 7f4bcaffd700 1 -- 172.21.15.23:0/4189644922 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd001ade0 con 0x7f4bd40c45f0 2023-06-22T08:24:58.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.901+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/4189644922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd41f1820 con 0x7f4bd40c45f0 2023-06-22T08:24:58.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.901+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/4189644922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd40b3e60 con 0x7f4bd40c45f0 2023-06-22T08:24:58.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.902+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/4189644922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd4151be0 con 0x7f4bd40c45f0 2023-06-22T08:24:58.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.903+0000 7f4bcaffd700 1 -- 172.21.15.23:0/4189644922 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f4bd0022070 con 0x7f4bd40c45f0 2023-06-22T08:24:58.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.904+0000 7f4bcaffd700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4bb806ace0 0x7f4bb806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:24:58.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.904+0000 7f4bcaffd700 1 -- 172.21.15.23:0/4189644922 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7f4bd008aae0 con 0x7f4bd40c45f0 2023-06-22T08:24:58.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.905+0000 7f4bd999a700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4bb806ace0 0x7f4bb806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:24:58.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.905+0000 7f4bd999a700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4bb806ace0 0x7f4bb806d190 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4bd414c170 tx=0x7f4bc4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:24:58.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:58.908+0000 7f4bcaffd700 1 -- 172.21.15.23:0/4189644922 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f4bd00553f0 con 0x7f4bd40c45f0 2023-06-22T08:24:59.160 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.158+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/4189644922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bd4158660 con 0x7f4bd40c45f0 2023-06-22T08:24:59.161 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.158+0000 7f4bcaffd700 1 -- 172.21.15.23:0/4189644922 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+4734 (secure 0 0 0) 0x7f4bd0029070 con 0x7f4bd40c45f0 2023-06-22T08:24:59.161 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:24:59.161 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:57.751683+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:24:59.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.160+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4bb806ace0 msgr2=0x7f4bb806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:59.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.160+0000 7f4bdc3ff700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4bb806ace0 0x7f4bb806d190 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4bd414c170 tx=0x7f4bc4009040 comp rx=0 tx=0).stop 2023-06-22T08:24:59.163 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.160+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/4189644922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd40c45f0 msgr2=0x7f4bd414b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:24:59.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.160+0000 7f4bdc3ff700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd40c45f0 0x7f4bd414b6b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4bd00090b0 tx=0x7f4bd0004750 comp rx=0 tx=0).stop 2023-06-22T08:24:59.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.161+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/4189644922 shutdown_connections 2023-06-22T08:24:59.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.161+0000 7f4bdc3ff700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f4bb806ace0 0x7f4bb806d190 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:59.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.161+0000 7f4bdc3ff700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bd4159330 0x7f4bd40b28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:59.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.161+0000 7f4bdc3ff700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bd40c45f0 0x7f4bd414b6b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:59.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.161+0000 7f4bdc3ff700 1 --2- 172.21.15.23:0/4189644922 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4bd4152d70 0x7f4bd4151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:24:59.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.161+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/4189644922 >> 172.21.15.23:0/4189644922 conn(0x7f4bd406ae30 msgr2=0x7f4bd40cc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:24:59.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.161+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/4189644922 shutdown_connections 2023-06-22T08:24:59.165 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:24:59.161+0000 7f4bdc3ff700 1 -- 172.21.15.23:0/4189644922 wait complete. 2023-06-22T08:24:59.165 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 194 2023-06-22T08:24:59.182 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 194, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-06-22T07:59:55.139239+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 194, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3 ], "info": { "gid_6185": { "addr": "172.21.15.67:6836/2287946240", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6834", "nonce": 2287946240, "type": "v2" }, { "addr": "172.21.15.67:6836", "nonce": 2287946240, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6185, "incarnation": 192, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:reconnect", "state_seq": 15 }, "gid_6616": { "addr": "172.21.15.67:6837/1855579717", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6835", "nonce": 1855579717, "type": "v2" }, { "addr": "172.21.15.67:6837", "nonce": 1855579717, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 6616, "incarnation": 162, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 26 }, "gid_6705": { "addr": "172.21.15.23:6835/3148174349", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6833", "nonce": 3148174349, "type": "v2" }, { "addr": "172.21.15.23:6835", "nonce": 3148174349, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6705, "incarnation": 184, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 24 }, "gid_6730": { "addr": "172.21.15.23:6834/456927633", "addrs": { "addrvec": [ { "addr": "172.21.15.23:6832", "nonce": 456927633, "type": "v2" }, { "addr": "172.21.15.23:6834", "nonce": 456927633, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6730, "incarnation": 172, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 111, "max_file_size": 1099511627776, "max_mds": 4, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-06-22T08:24:57.751683+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 4 ], "tableserver": 0, "up": { "mds_0": 6616, "mds_1": 6730, "mds_2": 6185, "mds_3": 6705 } } } ], "standbys": [] } 2023-06-22T08:25:00.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~278s 2023-06-22T08:25:01.183 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:25:01.394 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.391+0000 7f980edff700 1 Processor -- start 2023-06-22T08:25:01.395 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.391+0000 7f980edff700 1 -- start start 2023-06-22T08:25:01.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980edff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9808152d70 0x7f98080c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:01.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980edff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f98080c45f0 0x7f98080c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:01.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980edff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98080c4f90 con 0x7f9808159330 2023-06-22T08:25:01.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980edff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98080c95e0 con 0x7f9808152d70 2023-06-22T08:25:01.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980edff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9808159790 con 0x7f98080c45f0 2023-06-22T08:25:01.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980cb9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9808152d70 0x7f98080c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:01.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f9807fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9808159330 0x7f98080cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34568/0 (socket says 172.21.15.23:34568) 2023-06-22T08:25:01.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f9807fff700 1 -- 172.21.15.23:0/4267357105 learned_addr learned my addr 172.21.15.23:0/4267357105 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:25:01.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980d39c700 1 --2- 172.21.15.23:0/4267357105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f98080c45f0 0x7f98080c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:01.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980cb9b700 1 -- 172.21.15.23:0/4267357105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f98080c45f0 msgr2=0x7f98080c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:01.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980cb9b700 1 --2- 172.21.15.23:0/4267357105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f98080c45f0 0x7f98080c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:01.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980cb9b700 1 -- 172.21.15.23:0/4267357105 >> v1:172.21.15.23:6789/0 conn(0x7f9808159330 legacy=0x7f98080cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:01.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f980cb9b700 1 -- 172.21.15.23:0/4267357105 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980815bea0 con 0x7f9808152d70 2023-06-22T08:25:01.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.393+0000 7f98077fe700 1 -- 172.21.15.23:0/4267357105 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398211672 0 0) 0x7f98080c4f90 con 0x7f9808159330 2023-06-22T08:25:01.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.394+0000 7f980cb9b700 1 --2- 172.21.15.23:0/4267357105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9808152d70 0x7f98080c97d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009840 tx=0x7f97fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55297ee059ecec59 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:01.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.394+0000 7f98077fe700 1 -- 172.21.15.23:0/4267357105 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc011830 con 0x7f9808152d70 2023-06-22T08:25:01.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.394+0000 7f98077fe700 1 -- 172.21.15.23:0/4267357105 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc011990 con 0x7f9808152d70 2023-06-22T08:25:01.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.394+0000 7f98077fe700 1 -- 172.21.15.23:0/4267357105 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc011d20 con 0x7f9808152d70 2023-06-22T08:25:01.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.394+0000 7f980edff700 1 -- 172.21.15.23:0/4267357105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9808152d70 msgr2=0x7f98080c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:01.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.394+0000 7f980edff700 1 --2- 172.21.15.23:0/4267357105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9808152d70 0x7f98080c97d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009840 tx=0x7f97fc009b50 comp rx=0 tx=0).stop 2023-06-22T08:25:01.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.395+0000 7f980edff700 1 -- 172.21.15.23:0/4267357105 shutdown_connections 2023-06-22T08:25:01.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.395+0000 7f980edff700 1 --2- 172.21.15.23:0/4267357105 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f98080c45f0 0x7f98080c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:01.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.395+0000 7f980edff700 1 --2- 172.21.15.23:0/4267357105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9808152d70 0x7f98080c97d0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:01.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.395+0000 7f980edff700 1 -- 172.21.15.23:0/4267357105 >> 172.21.15.23:0/4267357105 conn(0x7f980806ae30 msgr2=0x7f980806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:01.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.395+0000 7f980edff700 1 -- 172.21.15.23:0/4267357105 shutdown_connections 2023-06-22T08:25:01.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.395+0000 7f980edff700 1 -- 172.21.15.23:0/4267357105 wait complete. 2023-06-22T08:25:01.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.396+0000 7f980edff700 1 Processor -- start 2023-06-22T08:25:01.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.397+0000 7f980edff700 1 -- start start 2023-06-22T08:25:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.397+0000 7f980edff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98080c45f0 0x7f98080b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.397+0000 7f980edff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9808152d70 0x7f98081f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.397+0000 7f980edff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9808159330 0x7f98081f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.397+0000 7f980edff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98081566d0 con 0x7f98080c45f0 2023-06-22T08:25:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.397+0000 7f980edff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98081f1540 con 0x7f9808159330 2023-06-22T08:25:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.397+0000 7f980edff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9808156900 con 0x7f9808152d70 2023-06-22T08:25:01.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f980cb9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98080c45f0 0x7f98080b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:01.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f980cb9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98080c45f0 0x7f98080b37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60084/0 (socket says 172.21.15.23:60084) 2023-06-22T08:25:01.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f980cb9b700 1 -- 172.21.15.23:0/768297364 learned_addr learned my addr 172.21.15.23:0/768297364 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:01.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f9807fff700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9808152d70 0x7f98081f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:01.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f980d39c700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9808159330 0x7f98081f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:01.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f980cb9b700 1 -- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9808152d70 msgr2=0x7f98081f19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:01.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f980cb9b700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9808152d70 0x7f98081f19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:01.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f980cb9b700 1 -- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9808159330 msgr2=0x7f98081f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:01.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f980cb9b700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9808159330 0x7f98081f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:01.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f980cb9b700 1 -- 172.21.15.23:0/768297364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc0094a0 con 0x7f98080c45f0 2023-06-22T08:25:01.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.398+0000 7f980cb9b700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98080c45f0 0x7f98080b37e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009fd0 tx=0x7f97fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:01.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.399+0000 7f98057fa700 1 -- 172.21.15.23:0/768297364 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0038a0 con 0x7f98080c45f0 2023-06-22T08:25:01.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.399+0000 7f98057fa700 1 -- 172.21.15.23:0/768297364 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc01f070 con 0x7f98080c45f0 2023-06-22T08:25:01.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.399+0000 7f980edff700 1 -- 172.21.15.23:0/768297364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9808156be0 con 0x7f98080c45f0 2023-06-22T08:25:01.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.399+0000 7f98057fa700 1 -- 172.21.15.23:0/768297364 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc003c40 con 0x7f98080c45f0 2023-06-22T08:25:01.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.399+0000 7f980edff700 1 -- 172.21.15.23:0/768297364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98081571a0 con 0x7f98080c45f0 2023-06-22T08:25:01.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.400+0000 7f980edff700 1 -- 172.21.15.23:0/768297364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98081f1820 con 0x7f98080c45f0 2023-06-22T08:25:01.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.401+0000 7f98057fa700 1 -- 172.21.15.23:0/768297364 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f97fc026480 con 0x7f98080c45f0 2023-06-22T08:25:01.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.402+0000 7f98057fa700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f97ec06ace0 0x7f97ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:01.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.402+0000 7f98057fa700 1 -- 172.21.15.23:0/768297364 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7f97fc021070 con 0x7f98080c45f0 2023-06-22T08:25:01.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.402+0000 7f9807fff700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f97ec06ace0 0x7f97ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:01.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.403+0000 7f9807fff700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f97ec06ace0 0x7f97ec06d190 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f98080b4930 tx=0x7f97f4007c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:01.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.406+0000 7f98057fa700 1 -- 172.21.15.23:0/768297364 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f97fc055ed0 con 0x7f98080c45f0 2023-06-22T08:25:01.663 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.660+0000 7f980edff700 1 -- 172.21.15.23:0/768297364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f980804ea20 con 0x7f98080c45f0 2023-06-22T08:25:01.664 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.661+0000 7f98057fa700 1 -- 172.21.15.23:0/768297364 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+4734 (secure 0 0 0) 0x7f97fc0594f0 con 0x7f98080c45f0 2023-06-22T08:25:01.664 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:25:01.664 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:24:57.751683+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:25:01.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.663+0000 7f980edff700 1 -- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f97ec06ace0 msgr2=0x7f97ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:01.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.663+0000 7f980edff700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f97ec06ace0 0x7f97ec06d190 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f98080b4930 tx=0x7f97f4007c90 comp rx=0 tx=0).stop 2023-06-22T08:25:01.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.663+0000 7f980edff700 1 -- 172.21.15.23:0/768297364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98080c45f0 msgr2=0x7f98080b37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:01.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.663+0000 7f980edff700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98080c45f0 0x7f98080b37e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009fd0 tx=0x7f97fc004750 comp rx=0 tx=0).stop 2023-06-22T08:25:01.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.663+0000 7f980edff700 1 -- 172.21.15.23:0/768297364 shutdown_connections 2023-06-22T08:25:01.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.663+0000 7f980edff700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f97ec06ace0 0x7f97ec06d190 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:01.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.663+0000 7f980edff700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9808159330 0x7f98081f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:01.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.663+0000 7f980edff700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98080c45f0 0x7f98080b37e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:01.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.663+0000 7f980edff700 1 --2- 172.21.15.23:0/768297364 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f9808152d70 0x7f98081f19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:01.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.663+0000 7f980edff700 1 -- 172.21.15.23:0/768297364 >> 172.21.15.23:0/768297364 conn(0x7f980806ae30 msgr2=0x7f9808151a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:01.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.664+0000 7f980edff700 1 -- 172.21.15.23:0/768297364 shutdown_connections 2023-06-22T08:25:01.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:01.664+0000 7f980edff700 1 -- 172.21.15.23:0/768297364 wait complete. 2023-06-22T08:25:01.669 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 194 2023-06-22T08:25:01.687 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:25:03.688 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:25:03.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.895+0000 7f202d1ff700 1 Processor -- start 2023-06-22T08:25:03.898 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.896+0000 7f202d1ff700 1 -- start start 2023-06-22T08:25:03.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.897+0000 7f202d1ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f20280c3a40 0x7f20280c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:03.899 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.897+0000 7f202d1ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20280c6a60 con 0x7f202815a320 2023-06-22T08:25:03.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.897+0000 7f202d1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f202814d690 con 0x7f202806d050 2023-06-22T08:25:03.900 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.897+0000 7f202d1ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2028157570 con 0x7f20280c3a40 2023-06-22T08:25:03.901 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.897+0000 7f202759e700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f202815a320 0x7f2028160ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:34574/0 (socket says 172.21.15.23:34574) 2023-06-22T08:25:03.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.897+0000 7f202759e700 1 -- 172.21.15.23:0/1559149838 learned_addr learned my addr 172.21.15.23:0/1559149838 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:25:03.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.897+0000 7f2026d9d700 1 --2- 172.21.15.23:0/1559149838 >> v2:172.21.15.23:3300/0 conn(0x7f20280c3a40 0x7f20280c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:03.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.898+0000 7f202659c700 1 -- 172.21.15.23:0/1559149838 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118370580 0 0) 0x7f202814d690 con 0x7f202806d050 2023-06-22T08:25:03.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.898+0000 7f202659c700 1 -- 172.21.15.23:0/1559149838 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20140035b0 con 0x7f202806d050 2023-06-22T08:25:03.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.898+0000 7f2026d9d700 1 -- 172.21.15.23:0/1559149838 >> v1:172.21.15.67:6789/0 conn(0x7f202806d050 legacy=0x7f2028159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:03.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.898+0000 7f2026d9d700 1 -- 172.21.15.23:0/1559149838 >> v1:172.21.15.23:6789/0 conn(0x7f202815a320 legacy=0x7f2028160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:03.902 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.898+0000 7f202659c700 1 -- 172.21.15.23:0/1559149838 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052558084 0 0) 0x7f20280c6a60 con 0x7f202815a320 2023-06-22T08:25:03.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.898+0000 7f2026d9d700 1 -- 172.21.15.23:0/1559149838 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20280c6430 con 0x7f20280c3a40 2023-06-22T08:25:03.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.898+0000 7f2026d9d700 1 --2- 172.21.15.23:0/1559149838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 0x7f20280c3e10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2018009840 tx=0x7f2018009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f192b789e874698 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:03.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.898+0000 7f202659c700 1 -- 172.21.15.23:0/1559149838 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2018011830 con 0x7f20280c3a40 2023-06-22T08:25:03.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.898+0000 7f202659c700 1 -- 172.21.15.23:0/1559149838 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2018011990 con 0x7f20280c3a40 2023-06-22T08:25:03.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.898+0000 7f202659c700 1 -- 172.21.15.23:0/1559149838 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2018011d00 con 0x7f20280c3a40 2023-06-22T08:25:03.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.899+0000 7f202d1ff700 1 -- 172.21.15.23:0/1559149838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 msgr2=0x7f20280c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:03.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.899+0000 7f202d1ff700 1 --2- 172.21.15.23:0/1559149838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 0x7f20280c3e10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2018009840 tx=0x7f2018009b50 comp rx=0 tx=0).stop 2023-06-22T08:25:03.903 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.899+0000 7f202d1ff700 1 -- 172.21.15.23:0/1559149838 shutdown_connections 2023-06-22T08:25:03.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.899+0000 7f202d1ff700 1 --2- 172.21.15.23:0/1559149838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 0x7f20280c3e10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:03.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.899+0000 7f202d1ff700 1 -- 172.21.15.23:0/1559149838 >> 172.21.15.23:0/1559149838 conn(0x7f20280ca3f0 msgr2=0x7f20280ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:03.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.899+0000 7f202d1ff700 1 -- 172.21.15.23:0/1559149838 shutdown_connections 2023-06-22T08:25:03.904 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.899+0000 7f202d1ff700 1 -- 172.21.15.23:0/1559149838 wait complete. 2023-06-22T08:25:03.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.901+0000 7f202d1ff700 1 Processor -- start 2023-06-22T08:25:03.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.901+0000 7f202d1ff700 1 -- start start 2023-06-22T08:25:03.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f202d1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f202806d050 0x7f20281554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:03.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f202d1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 0x7f20281ee050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:03.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f202d1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f202815a320 0x7f20281f15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:03.906 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f202d1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20280c8a50 con 0x7f20280c3a40 2023-06-22T08:25:03.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f202d1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2028157e60 con 0x7f202815a320 2023-06-22T08:25:03.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f202d1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20280c8ee0 con 0x7f202806d050 2023-06-22T08:25:03.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f201ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 0x7f20281ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:03.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f201ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 0x7f20281ee050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60088/0 (socket says 172.21.15.23:60088) 2023-06-22T08:25:03.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f201ffff700 1 -- 172.21.15.23:0/3382756636 learned_addr learned my addr 172.21.15.23:0/3382756636 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:03.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f2026d9d700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f202806d050 0x7f20281554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:03.907 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.902+0000 7f202759e700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f202815a320 0x7f20281f15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:03.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f201ffff700 1 -- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f202806d050 msgr2=0x7f20281554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:03.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f201ffff700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f202806d050 0x7f20281554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:03.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f201ffff700 1 -- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f202815a320 msgr2=0x7f20281f15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:03.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f201ffff700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f202815a320 0x7f20281f15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:03.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f201ffff700 1 -- 172.21.15.23:0/3382756636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20180094a0 con 0x7f20280c3a40 2023-06-22T08:25:03.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f201ffff700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 0x7f20281ee050 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f201000c520 tx=0x7f201000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:03.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f201f7fe700 1 -- 172.21.15.23:0/3382756636 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20100052d0 con 0x7f20280c3a40 2023-06-22T08:25:03.908 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f201f7fe700 1 -- 172.21.15.23:0/3382756636 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2010005430 con 0x7f20280c3a40 2023-06-22T08:25:03.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f202d1ff700 1 -- 172.21.15.23:0/3382756636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20281597b0 con 0x7f20280c3a40 2023-06-22T08:25:03.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f201f7fe700 1 -- 172.21.15.23:0/3382756636 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20100038a0 con 0x7f20280c3a40 2023-06-22T08:25:03.909 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.903+0000 7f202d1ff700 1 -- 172.21.15.23:0/3382756636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2028157b80 con 0x7f20280c3a40 2023-06-22T08:25:03.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.904+0000 7f202d1ff700 1 -- 172.21.15.23:0/3382756636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20280c8c00 con 0x7f20280c3a40 2023-06-22T08:25:03.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.905+0000 7f201f7fe700 1 -- 172.21.15.23:0/3382756636 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f2010003a00 con 0x7f20280c3a40 2023-06-22T08:25:03.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.906+0000 7f201f7fe700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f200806ad10 0x7f200806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:03.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.906+0000 7f201f7fe700 1 -- 172.21.15.23:0/3382756636 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7f20100891f0 con 0x7f20280c3a40 2023-06-22T08:25:03.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.906+0000 7f2026d9d700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f200806ad10 0x7f200806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:03.910 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.907+0000 7f2026d9d700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f200806ad10 0x7f200806d1c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f20180090b0 tx=0x7f2018008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:03.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:03.911+0000 7f201f7fe700 1 -- 172.21.15.23:0/3382756636 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f20280c8c00 con 0x7f20280c3a40 2023-06-22T08:25:04.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.161+0000 7f202d1ff700 1 -- 172.21.15.23:0/3382756636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f202804ea20 con 0x7f20280c3a40 2023-06-22T08:25:04.164 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.162+0000 7f201f7fe700 1 -- 172.21.15.23:0/3382756636 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4731 (secure 0 0 0) 0x7f2010017020 con 0x7f20280c3a40 2023-06-22T08:25:04.165 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:25:04.165 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:25:03.079226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:25:04.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 -- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f200806ad10 msgr2=0x7f200806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:04.167 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f200806ad10 0x7f200806d1c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f20180090b0 tx=0x7f2018008030 comp rx=0 tx=0).stop 2023-06-22T08:25:04.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 -- 172.21.15.23:0/3382756636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 msgr2=0x7f20281ee050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:04.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 0x7f20281ee050 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f201000c520 tx=0x7f201000c830 comp rx=0 tx=0).stop 2023-06-22T08:25:04.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 -- 172.21.15.23:0/3382756636 shutdown_connections 2023-06-22T08:25:04.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f200806ad10 0x7f200806d1c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:04.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f202815a320 0x7f20281f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:04.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f202806d050 0x7f20281554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:04.168 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 --2- 172.21.15.23:0/3382756636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20280c3a40 0x7f20281ee050 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:04.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 -- 172.21.15.23:0/3382756636 >> 172.21.15.23:0/3382756636 conn(0x7f20280ca3f0 msgr2=0x7f202814c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:04.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 -- 172.21.15.23:0/3382756636 shutdown_connections 2023-06-22T08:25:04.169 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:04.165+0000 7f202d1ff700 1 -- 172.21.15.23:0/3382756636 wait complete. 2023-06-22T08:25:04.169 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 195 2023-06-22T08:25:04.185 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:25:06.187 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:25:06.397 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.395+0000 7f77609ff700 1 Processor -- start 2023-06-22T08:25:06.398 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.395+0000 7f77609ff700 1 -- start start 2023-06-22T08:25:06.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.396+0000 7f77609ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f775c0cb6b0 0x7f775c069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:06.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.396+0000 7f77609ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f775c152e10 0x7f775c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:06.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.396+0000 7f77609ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775c158f60 con 0x7f775c0c5740 2023-06-22T08:25:06.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.396+0000 7f77609ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775c06bd20 con 0x7f775c152e10 2023-06-22T08:25:06.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.396+0000 7f77609ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f775c0cc450 con 0x7f775c0cb6b0 2023-06-22T08:25:06.399 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775ad9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f775c152e10 0x7f775c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:06.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775ad9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f775c152e10 0x7f775c1593f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39764/0 (socket says 172.21.15.23:39764) 2023-06-22T08:25:06.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775ad9d700 1 -- 172.21.15.23:0/2919280982 learned_addr learned my addr 172.21.15.23:0/2919280982 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:06.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775a59c700 1 --2- 172.21.15.23:0/2919280982 >> v2:172.21.15.67:3300/0 conn(0x7f775c0cb6b0 0x7f775c069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:06.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775ad9d700 1 -- 172.21.15.23:0/2919280982 >> v2:172.21.15.67:3300/0 conn(0x7f775c0cb6b0 msgr2=0x7f775c069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:06.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775ad9d700 1 --2- 172.21.15.23:0/2919280982 >> v2:172.21.15.67:3300/0 conn(0x7f775c0cb6b0 0x7f775c069eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:06.401 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775ad9d700 1 -- 172.21.15.23:0/2919280982 >> v1:172.21.15.67:6789/0 conn(0x7f775c0c5740 legacy=0x7f775c152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:06.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775ad9d700 1 -- 172.21.15.23:0/2919280982 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775c158980 con 0x7f775c152e10 2023-06-22T08:25:06.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775ad9d700 1 --2- 172.21.15.23:0/2919280982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c152e10 0x7f775c1593f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f775c0cc230 tx=0x7f775400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1a768908510efe server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:06.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775959a700 1 -- 172.21.15.23:0/2919280982 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775400e070 con 0x7f775c152e10 2023-06-22T08:25:06.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.397+0000 7f775959a700 1 -- 172.21.15.23:0/2919280982 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775400b910 con 0x7f775c152e10 2023-06-22T08:25:06.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.398+0000 7f775959a700 1 -- 172.21.15.23:0/2919280982 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775400bcc0 con 0x7f775c152e10 2023-06-22T08:25:06.402 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.398+0000 7f77609ff700 1 -- 172.21.15.23:0/2919280982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c152e10 msgr2=0x7f775c1593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:06.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.398+0000 7f77609ff700 1 --2- 172.21.15.23:0/2919280982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c152e10 0x7f775c1593f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f775c0cc230 tx=0x7f775400b330 comp rx=0 tx=0).stop 2023-06-22T08:25:06.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.398+0000 7f77609ff700 1 -- 172.21.15.23:0/2919280982 shutdown_connections 2023-06-22T08:25:06.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.398+0000 7f77609ff700 1 --2- 172.21.15.23:0/2919280982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c152e10 0x7f775c1593f0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:06.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.398+0000 7f77609ff700 1 --2- 172.21.15.23:0/2919280982 >> v2:172.21.15.67:3300/0 conn(0x7f775c0cb6b0 0x7f775c069eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:06.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.398+0000 7f77609ff700 1 -- 172.21.15.23:0/2919280982 >> 172.21.15.23:0/2919280982 conn(0x7f775c070d80 msgr2=0x7f775c071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:06.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.398+0000 7f77609ff700 1 -- 172.21.15.23:0/2919280982 shutdown_connections 2023-06-22T08:25:06.403 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.398+0000 7f77609ff700 1 -- 172.21.15.23:0/2919280982 wait complete. 2023-06-22T08:25:06.405 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.400+0000 7f77609ff700 1 Processor -- start 2023-06-22T08:25:06.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.400+0000 7f77609ff700 1 -- start start 2023-06-22T08:25:06.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f77609ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f775c0cb6b0 0x7f775c1f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:06.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f77609ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f775c152e10 0x7f775c0ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:06.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f77609ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c0c6bf0 0x7f775c143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:06.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f77609ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c1e4250 con 0x7f775c0c6bf0 2023-06-22T08:25:06.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f77609ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775c1e49f0 con 0x7f775c152e10 2023-06-22T08:25:06.406 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f77609ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775c1f1a60 con 0x7f775c0cb6b0 2023-06-22T08:25:06.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f775ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c0c6bf0 0x7f775c143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:06.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f775ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c0c6bf0 0x7f775c143610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39772/0 (socket says 172.21.15.23:39772) 2023-06-22T08:25:06.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f775ad9d700 1 -- 172.21.15.23:0/4175760537 learned_addr learned my addr 172.21.15.23:0/4175760537 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:06.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f775a59c700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f775c0cb6b0 0x7f775c1f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:06.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.401+0000 7f7759d9b700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f775c152e10 0x7f775c0ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:06.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f775ad9d700 1 -- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f775c0cb6b0 msgr2=0x7f775c1f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:06.407 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f775ad9d700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f775c0cb6b0 0x7f775c1f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:06.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f775ad9d700 1 -- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f775c152e10 msgr2=0x7f775c0ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:06.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f775ad9d700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f775c152e10 0x7f775c0ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:06.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f775ad9d700 1 -- 172.21.15.23:0/4175760537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7754009b30 con 0x7f775c0c6bf0 2023-06-22T08:25:06.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f775ad9d700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c0c6bf0 0x7f775c143610 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7754012000 tx=0x7f7754004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:06.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f774affd700 1 -- 172.21.15.23:0/4175760537 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7754004880 con 0x7f775c0c6bf0 2023-06-22T08:25:06.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f774affd700 1 -- 172.21.15.23:0/4175760537 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77540092e0 con 0x7f775c0c6bf0 2023-06-22T08:25:06.408 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f77609ff700 1 -- 172.21.15.23:0/4175760537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f775c1f2430 con 0x7f775c0c6bf0 2023-06-22T08:25:06.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f774affd700 1 -- 172.21.15.23:0/4175760537 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775401b650 con 0x7f775c0c6bf0 2023-06-22T08:25:06.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.402+0000 7f77609ff700 1 -- 172.21.15.23:0/4175760537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775c1e4710 con 0x7f775c0c6bf0 2023-06-22T08:25:06.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.403+0000 7f77609ff700 1 -- 172.21.15.23:0/4175760537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775c1f1bd0 con 0x7f775c0c6bf0 2023-06-22T08:25:06.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.404+0000 7f774affd700 1 -- 172.21.15.23:0/4175760537 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f775401b7b0 con 0x7f775c0c6bf0 2023-06-22T08:25:06.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.405+0000 7f774affd700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f773c06ace0 0x7f773c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:06.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.405+0000 7f774affd700 1 -- 172.21.15.23:0/4175760537 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7f775408adf0 con 0x7f775c0c6bf0 2023-06-22T08:25:06.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.406+0000 7f775a59c700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f773c06ace0 0x7f773c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:06.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.406+0000 7f775a59c700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f773c06ace0 0x7f773c06d190 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f774c009be0 tx=0x7f774c009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:06.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.410+0000 7f774affd700 1 -- 172.21.15.23:0/4175760537 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f7754055650 con 0x7f775c0c6bf0 2023-06-22T08:25:06.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.666+0000 7f77609ff700 1 -- 172.21.15.23:0/4175760537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f775c04ea20 con 0x7f775c0c6bf0 2023-06-22T08:25:06.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.667+0000 7f774affd700 1 -- 172.21.15.23:0/4175760537 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4731 (secure 0 0 0) 0x7f7754058c70 con 0x7f775c0c6bf0 2023-06-22T08:25:06.670 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:25:06.670 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:25:03.079226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:25:06.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.669+0000 7f77609ff700 1 -- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f773c06ace0 msgr2=0x7f773c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:06.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.669+0000 7f77609ff700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f773c06ace0 0x7f773c06d190 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f774c009be0 tx=0x7f774c009210 comp rx=0 tx=0).stop 2023-06-22T08:25:06.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.669+0000 7f77609ff700 1 -- 172.21.15.23:0/4175760537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c0c6bf0 msgr2=0x7f775c143610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:06.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.669+0000 7f77609ff700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c0c6bf0 0x7f775c143610 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7754012000 tx=0x7f7754004ad0 comp rx=0 tx=0).stop 2023-06-22T08:25:06.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.669+0000 7f77609ff700 1 -- 172.21.15.23:0/4175760537 shutdown_connections 2023-06-22T08:25:06.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.669+0000 7f77609ff700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f775c0c6bf0 0x7f775c143610 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:06.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.669+0000 7f77609ff700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f775c0cb6b0 0x7f775c1f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:06.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.669+0000 7f77609ff700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f773c06ace0 0x7f773c06d190 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:06.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.669+0000 7f77609ff700 1 --2- 172.21.15.23:0/4175760537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f775c152e10 0x7f775c0ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:06.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.669+0000 7f77609ff700 1 -- 172.21.15.23:0/4175760537 >> 172.21.15.23:0/4175760537 conn(0x7f775c070d80 msgr2=0x7f775c157ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:06.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.670+0000 7f77609ff700 1 -- 172.21.15.23:0/4175760537 shutdown_connections 2023-06-22T08:25:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:06.670+0000 7f77609ff700 1 -- 172.21.15.23:0/4175760537 wait complete. 2023-06-22T08:25:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 195 2023-06-22T08:25:06.690 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-06-22T08:25:06.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~284s 2023-06-22T08:25:08.691 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:25:08.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.908+0000 7f905edff700 1 Processor -- start 2023-06-22T08:25:08.911 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.909+0000 7f905edff700 1 -- start start 2023-06-22T08:25:08.912 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.910+0000 7f905edff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f90580c3a40 0x7f90580c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:08.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.910+0000 7f905edff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f905806d050 0x7f9058159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:08.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.910+0000 7f905edff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90580c6a60 con 0x7f9058156fc0 2023-06-22T08:25:08.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.910+0000 7f905edff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9058157570 con 0x7f90580c3a40 2023-06-22T08:25:08.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.910+0000 7f905edff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90580c4350 con 0x7f905806d050 2023-06-22T08:25:08.913 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.910+0000 7f905cb9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f90580c3a40 0x7f90580c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:08.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.910+0000 7f905d39c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9058156fc0 0x7f9058069eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:47590/0 (socket says 172.21.15.23:47590) 2023-06-22T08:25:08.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.910+0000 7f905d39c700 1 -- 172.21.15.23:0/4162173808 learned_addr learned my addr 172.21.15.23:0/4162173808 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:25:08.920 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.910+0000 7f9057fff700 1 --2- 172.21.15.23:0/4162173808 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f905806d050 0x7f9058159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:08.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.911+0000 7f9057fff700 1 -- 172.21.15.23:0/4162173808 >> v2:172.21.15.23:3300/0 conn(0x7f90580c3a40 msgr2=0x7f90580c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:08.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.911+0000 7f9057fff700 1 --2- 172.21.15.23:0/4162173808 >> v2:172.21.15.23:3300/0 conn(0x7f90580c3a40 0x7f90580c3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:08.923 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.911+0000 7f90577fe700 1 -- 172.21.15.23:0/4162173808 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630064305 0 0) 0x7f90580c6a60 con 0x7f9058156fc0 2023-06-22T08:25:08.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.911+0000 7f9057fff700 1 -- 172.21.15.23:0/4162173808 >> v1:172.21.15.23:6789/0 conn(0x7f9058156fc0 legacy=0x7f9058069eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:08.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.911+0000 7f9057fff700 1 -- 172.21.15.23:0/4162173808 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905814d690 con 0x7f905806d050 2023-06-22T08:25:08.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.911+0000 7f905cb9b700 1 --2- 172.21.15.23:0/4162173808 >> v2:172.21.15.23:3300/0 conn(0x7f90580c3a40 0x7f90580c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:25:08.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.911+0000 7f9057fff700 1 --2- 172.21.15.23:0/4162173808 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f905806d050 0x7f9058159c10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9044009840 tx=0x7f9044009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=199b68a66d79cf8c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:08.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.911+0000 7f90577fe700 1 -- 172.21.15.23:0/4162173808 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044011830 con 0x7f905806d050 2023-06-22T08:25:08.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.911+0000 7f90577fe700 1 -- 172.21.15.23:0/4162173808 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9044011990 con 0x7f905806d050 2023-06-22T08:25:08.924 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.911+0000 7f90577fe700 1 -- 172.21.15.23:0/4162173808 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044011d20 con 0x7f905806d050 2023-06-22T08:25:08.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.912+0000 7f905edff700 1 -- 172.21.15.23:0/4162173808 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f905806d050 msgr2=0x7f9058159c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:08.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.912+0000 7f905edff700 1 --2- 172.21.15.23:0/4162173808 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f905806d050 0x7f9058159c10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9044009840 tx=0x7f9044009b50 comp rx=0 tx=0).stop 2023-06-22T08:25:08.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.912+0000 7f905edff700 1 -- 172.21.15.23:0/4162173808 shutdown_connections 2023-06-22T08:25:08.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.912+0000 7f905edff700 1 --2- 172.21.15.23:0/4162173808 >> v2:172.21.15.23:3300/0 conn(0x7f90580c3a40 0x7f90580c3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:08.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.912+0000 7f905edff700 1 --2- 172.21.15.23:0/4162173808 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f905806d050 0x7f9058159c10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:08.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.912+0000 7f905edff700 1 -- 172.21.15.23:0/4162173808 >> 172.21.15.23:0/4162173808 conn(0x7f90580ca3f0 msgr2=0x7f90580ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:08.925 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.912+0000 7f905edff700 1 -- 172.21.15.23:0/4162173808 shutdown_connections 2023-06-22T08:25:08.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.912+0000 7f905edff700 1 -- 172.21.15.23:0/4162173808 wait complete. 2023-06-22T08:25:08.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.914+0000 7f905edff700 1 Processor -- start 2023-06-22T08:25:08.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.914+0000 7f905edff700 1 -- start start 2023-06-22T08:25:08.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905edff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f905806d050 0x7f9058158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:08.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905edff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90580c3a40 0x7f90581ee940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:08.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905edff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9058156fc0 0x7f90581eee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:08.926 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905edff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90581e1a60 con 0x7f905806d050 2023-06-22T08:25:08.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905edff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90581f2700 con 0x7f9058156fc0 2023-06-22T08:25:08.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905edff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90581ee440 con 0x7f90580c3a40 2023-06-22T08:25:08.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905cb9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f905806d050 0x7f9058158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:08.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905cb9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f905806d050 0x7f9058158d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39794/0 (socket says 172.21.15.23:39794) 2023-06-22T08:25:08.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905cb9b700 1 -- 172.21.15.23:0/1733694208 learned_addr learned my addr 172.21.15.23:0/1733694208 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:08.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f9057fff700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90580c3a40 0x7f90581ee940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:08.927 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905d39c700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9058156fc0 0x7f90581eee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:08.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905cb9b700 1 -- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90580c3a40 msgr2=0x7f90581ee940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:08.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905cb9b700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90580c3a40 0x7f90581ee940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:08.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905cb9b700 1 -- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9058156fc0 msgr2=0x7f90581eee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:08.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905cb9b700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9058156fc0 0x7f90581eee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:08.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.915+0000 7f905cb9b700 1 -- 172.21.15.23:0/1733694208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c0094a0 con 0x7f905806d050 2023-06-22T08:25:08.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.916+0000 7f905cb9b700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f905806d050 0x7f9058158d00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f904c00ebe0 tx=0x7f904c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:08.928 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.916+0000 7f90557fa700 1 -- 172.21.15.23:0/1733694208 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c00c850 con 0x7f905806d050 2023-06-22T08:25:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.916+0000 7f90557fa700 1 -- 172.21.15.23:0/1733694208 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904c00c9b0 con 0x7f905806d050 2023-06-22T08:25:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.916+0000 7f905edff700 1 -- 172.21.15.23:0/1733694208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90440094a0 con 0x7f905806d050 2023-06-22T08:25:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.916+0000 7f90557fa700 1 -- 172.21.15.23:0/1733694208 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c00cd40 con 0x7f905806d050 2023-06-22T08:25:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.916+0000 7f905edff700 1 -- 172.21.15.23:0/1733694208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f905814b0e0 con 0x7f905806d050 2023-06-22T08:25:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.917+0000 7f905edff700 1 -- 172.21.15.23:0/1733694208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90581ee660 con 0x7f905806d050 2023-06-22T08:25:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.918+0000 7f90557fa700 1 -- 172.21.15.23:0/1733694208 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f904c013470 con 0x7f905806d050 2023-06-22T08:25:08.929 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.919+0000 7f90557fa700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f903c06ace0 0x7f903c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:08.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.919+0000 7f90557fa700 1 -- 172.21.15.23:0/1733694208 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7f904c089c70 con 0x7f905806d050 2023-06-22T08:25:08.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.919+0000 7f9057fff700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f903c06ace0 0x7f903c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:08.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.920+0000 7f9057fff700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f903c06ace0 0x7f903c06d190 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f90440090b0 tx=0x7f904401c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:08.930 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:08.923+0000 7f90557fa700 1 -- 172.21.15.23:0/1733694208 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f90581ee660 con 0x7f905806d050 2023-06-22T08:25:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.176+0000 7f905edff700 1 -- 172.21.15.23:0/1733694208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90581ee600 con 0x7f905806d050 2023-06-22T08:25:09.179 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.176+0000 7f90557fa700 1 -- 172.21.15.23:0/1733694208 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+4731 (secure 0 0 0) 0x7f904c017090 con 0x7f905806d050 2023-06-22T08:25:09.179 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:25:09.179 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:25:08.751242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:active","state_seq":19,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:25:09.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 -- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f903c06ace0 msgr2=0x7f903c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:09.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f903c06ace0 0x7f903c06d190 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f90440090b0 tx=0x7f904401c040 comp rx=0 tx=0).stop 2023-06-22T08:25:09.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 -- 172.21.15.23:0/1733694208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f905806d050 msgr2=0x7f9058158d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:09.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f905806d050 0x7f9058158d00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f904c00ebe0 tx=0x7f904c00eef0 comp rx=0 tx=0).stop 2023-06-22T08:25:09.182 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 -- 172.21.15.23:0/1733694208 shutdown_connections 2023-06-22T08:25:09.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f903c06ace0 0x7f903c06d190 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:09.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9058156fc0 0x7f90581eee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:09.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f905806d050 0x7f9058158d00 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:09.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 --2- 172.21.15.23:0/1733694208 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f90580c3a40 0x7f90581ee940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:09.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 -- 172.21.15.23:0/1733694208 >> 172.21.15.23:0/1733694208 conn(0x7f90580ca3f0 msgr2=0x7f90580c9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:09.183 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 -- 172.21.15.23:0/1733694208 shutdown_connections 2023-06-22T08:25:09.184 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:09.179+0000 7f905edff700 1 -- 172.21.15.23:0/1733694208 wait complete. 2023-06-22T08:25:09.184 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 196 2023-06-22T08:25:09.199 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=2, replacing gid=6313 2023-06-22T08:25:09.200 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.e 2023-06-22T08:25:12.996 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T08:25:12.993+0000 7f6838c81700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-06-22T08:25:13.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~291s 2023-06-22T08:25:13.859 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/md5sum.x' -> './k/coreutils-8.5/man/md5sum.x' 2023-06-22T08:25:13.859 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/link.1' -> './k/coreutils-8.5/man/link.1' 2023-06-22T08:25:13.859 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/echo.x' -> './k/coreutils-8.5/man/echo.x' 2023-06-22T08:25:13.860 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/ptx.1' -> './k/coreutils-8.5/man/ptx.1' 2023-06-22T08:25:13.860 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/factor.x' -> './k/coreutils-8.5/man/factor.x' 2023-06-22T08:25:13.860 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/who.1' -> './k/coreutils-8.5/man/who.1' 2023-06-22T08:25:13.860 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/mktemp.1' -> './k/coreutils-8.5/man/mktemp.1' 2023-06-22T08:25:13.860 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/shred.1' -> './k/coreutils-8.5/man/shred.1' 2023-06-22T08:25:13.860 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/runcon.1' -> './k/coreutils-8.5/man/runcon.1' 2023-06-22T08:25:13.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/stty.x' -> './k/coreutils-8.5/man/stty.x' 2023-06-22T08:25:13.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/comm.1' -> './k/coreutils-8.5/man/comm.1' 2023-06-22T08:25:13.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/shuf.1' -> './k/coreutils-8.5/man/shuf.1' 2023-06-22T08:25:13.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/stty.1' -> './k/coreutils-8.5/man/stty.1' 2023-06-22T08:25:13.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/printenv.x' -> './k/coreutils-8.5/man/printenv.x' 2023-06-22T08:25:13.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/join.1' -> './k/coreutils-8.5/man/join.1' 2023-06-22T08:25:13.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/hostname.x' -> './k/coreutils-8.5/man/hostname.x' 2023-06-22T08:25:13.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/ln.1' -> './k/coreutils-8.5/man/ln.1' 2023-06-22T08:25:13.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/false.x' -> './k/coreutils-8.5/man/false.x' 2023-06-22T08:25:13.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/cat.1' -> './k/coreutils-8.5/man/cat.1' 2023-06-22T08:25:13.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/users.x' -> './k/coreutils-8.5/man/users.x' 2023-06-22T08:25:13.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/printf.1' -> './k/coreutils-8.5/man/printf.1' 2023-06-22T08:25:13.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/ls.x' -> './k/coreutils-8.5/man/ls.x' 2023-06-22T08:25:13.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sha224sum.x' -> './k/coreutils-8.5/man/sha224sum.x' 2023-06-22T08:25:13.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/whoami.1' -> './k/coreutils-8.5/man/whoami.1' 2023-06-22T08:25:13.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/expand.1' -> './k/coreutils-8.5/man/expand.1' 2023-06-22T08:25:13.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/chgrp.x' -> './k/coreutils-8.5/man/chgrp.x' 2023-06-22T08:25:13.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sleep.1' -> './k/coreutils-8.5/man/sleep.1' 2023-06-22T08:25:13.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/csplit.x' -> './k/coreutils-8.5/man/csplit.x' 2023-06-22T08:25:13.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/pwd.1' -> './k/coreutils-8.5/man/pwd.1' 2023-06-22T08:25:13.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/kill.1' -> './k/coreutils-8.5/man/kill.1' 2023-06-22T08:25:13.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/touch.x' -> './k/coreutils-8.5/man/touch.x' 2023-06-22T08:25:13.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/df.1' -> './k/coreutils-8.5/man/df.1' 2023-06-22T08:25:13.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/stdbuf.x' -> './k/coreutils-8.5/man/stdbuf.x' 2023-06-22T08:25:13.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/mkdir.x' -> './k/coreutils-8.5/man/mkdir.x' 2023-06-22T08:25:13.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/id.1' -> './k/coreutils-8.5/man/id.1' 2023-06-22T08:25:13.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/mknod.x' -> './k/coreutils-8.5/man/mknod.x' 2023-06-22T08:25:13.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sum.1' -> './k/coreutils-8.5/man/sum.1' 2023-06-22T08:25:13.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/rm.x' -> './k/coreutils-8.5/man/rm.x' 2023-06-22T08:25:13.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/date.1' -> './k/coreutils-8.5/man/date.1' 2023-06-22T08:25:13.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/chown.x' -> './k/coreutils-8.5/man/chown.x' 2023-06-22T08:25:13.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/touch.1' -> './k/coreutils-8.5/man/touch.1' 2023-06-22T08:25:13.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tac.x' -> './k/coreutils-8.5/man/tac.x' 2023-06-22T08:25:13.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/base64.x' -> './k/coreutils-8.5/man/base64.x' 2023-06-22T08:25:13.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/tsort.x' -> './k/coreutils-8.5/man/tsort.x' 2023-06-22T08:25:13.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/shred.x' -> './k/coreutils-8.5/man/shred.x' 2023-06-22T08:25:13.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/readlink.1' -> './k/coreutils-8.5/man/readlink.1' 2023-06-22T08:25:13.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/sha1sum.x' -> './k/coreutils-8.5/man/sha1sum.x' 2023-06-22T08:25:13.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/expand.x' -> './k/coreutils-8.5/man/expand.x' 2023-06-22T08:25:13.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/test.x' -> './k/coreutils-8.5/man/test.x' 2023-06-22T08:25:13.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/chcon.1' -> './k/coreutils-8.5/man/chcon.1' 2023-06-22T08:25:13.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/man/cat.x' -> './k/coreutils-8.5/man/cat.x' 2023-06-22T08:25:13.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib' -> './k/coreutils-8.5/lib' 2023-06-22T08:25:13.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/streq.h' -> './k/coreutils-8.5/lib/streq.h' 2023-06-22T08:25:13.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/asprintf.c' -> './k/coreutils-8.5/lib/asprintf.c' 2023-06-22T08:25:13.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strnlen1.c' -> './k/coreutils-8.5/lib/strnlen1.c' 2023-06-22T08:25:13.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dirfd.c' -> './k/coreutils-8.5/lib/dirfd.c' 2023-06-22T08:25:13.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/safe-write.h' -> './k/coreutils-8.5/lib/safe-write.h' 2023-06-22T08:25:13.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/snprintf.c' -> './k/coreutils-8.5/lib/snprintf.c' 2023-06-22T08:25:13.973 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbiter.h' -> './k/coreutils-8.5/lib/mbiter.h' 2023-06-22T08:25:13.973 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mountlist.c' -> './k/coreutils-8.5/lib/mountlist.c' 2023-06-22T08:25:13.973 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/propername.h' -> './k/coreutils-8.5/lib/propername.h' 2023-06-22T08:25:13.973 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memset.c' -> './k/coreutils-8.5/lib/memset.c' 2023-06-22T08:25:13.973 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/nproc.c' -> './k/coreutils-8.5/lib/nproc.c' 2023-06-22T08:25:13.974 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getdelim.c' -> './k/coreutils-8.5/lib/getdelim.c' 2023-06-22T08:25:13.974 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/signbitl.c' -> './k/coreutils-8.5/lib/signbitl.c' 2023-06-22T08:25:13.974 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/group-member.h' -> './k/coreutils-8.5/lib/group-member.h' 2023-06-22T08:25:13.974 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/long-options.c' -> './k/coreutils-8.5/lib/long-options.c' 2023-06-22T08:25:13.974 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fts_.h' -> './k/coreutils-8.5/lib/fts_.h' 2023-06-22T08:25:13.974 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fts.c' -> './k/coreutils-8.5/lib/fts.c' 2023-06-22T08:25:13.974 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/verify.h' -> './k/coreutils-8.5/lib/verify.h' 2023-06-22T08:25:13.974 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/str-two-way.h' -> './k/coreutils-8.5/lib/str-two-way.h' 2023-06-22T08:25:13.975 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/areadlink.c' -> './k/coreutils-8.5/lib/areadlink.c' 2023-06-22T08:25:13.975 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/c-strtod.h' -> './k/coreutils-8.5/lib/c-strtod.h' 2023-06-22T08:25:13.975 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sha256.h' -> './k/coreutils-8.5/lib/sha256.h' 2023-06-22T08:25:13.975 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stddef.in.h' -> './k/coreutils-8.5/lib/stddef.in.h' 2023-06-22T08:25:13.975 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/randint.c' -> './k/coreutils-8.5/lib/randint.c' 2023-06-22T08:25:13.975 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dup-safer.c' -> './k/coreutils-8.5/lib/dup-safer.c' 2023-06-22T08:25:13.975 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getpass.h' -> './k/coreutils-8.5/lib/getpass.h' 2023-06-22T08:25:13.976 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/gettext.h' -> './k/coreutils-8.5/lib/gettext.h' 2023-06-22T08:25:13.976 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/randint.h' -> './k/coreutils-8.5/lib/randint.h' 2023-06-22T08:25:13.976 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdio-write.c' -> './k/coreutils-8.5/lib/stdio-write.c' 2023-06-22T08:25:13.976 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xgethostname.h' -> './k/coreutils-8.5/lib/xgethostname.h' 2023-06-22T08:25:13.976 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/asnprintf.c' -> './k/coreutils-8.5/lib/asnprintf.c' 2023-06-22T08:25:13.976 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open-solaris.gperf' -> './k/coreutils-8.5/lib/iconv_open-solaris.gperf' 2023-06-22T08:25:13.976 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdio-impl.h' -> './k/coreutils-8.5/lib/stdio-impl.h' 2023-06-22T08:25:13.976 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mpsort.h' -> './k/coreutils-8.5/lib/mpsort.h' 2023-06-22T08:25:13.977 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/closein.c' -> './k/coreutils-8.5/lib/closein.c' 2023-06-22T08:25:13.977 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/hard-locale.c' -> './k/coreutils-8.5/lib/hard-locale.c' 2023-06-22T08:25:13.977 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/utimecmp.c' -> './k/coreutils-8.5/lib/utimecmp.c' 2023-06-22T08:25:13.977 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/vasnprintf.c' -> './k/coreutils-8.5/lib/vasnprintf.c' 2023-06-22T08:25:13.977 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/chown.c' -> './k/coreutils-8.5/lib/chown.c' 2023-06-22T08:25:13.977 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/printf-args.c' -> './k/coreutils-8.5/lib/printf-args.c' 2023-06-22T08:25:13.977 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/netdb.in.h' -> './k/coreutils-8.5/lib/netdb.in.h' 2023-06-22T08:25:13.977 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strnlen.c' -> './k/coreutils-8.5/lib/strnlen.c' 2023-06-22T08:25:13.978 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/errno.in.h' -> './k/coreutils-8.5/lib/errno.in.h' 2023-06-22T08:25:13.978 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbslen.c' -> './k/coreutils-8.5/lib/mbslen.c' 2023-06-22T08:25:13.978 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/obstack.h' -> './k/coreutils-8.5/lib/obstack.h' 2023-06-22T08:25:13.978 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/file-type.h' -> './k/coreutils-8.5/lib/file-type.h' 2023-06-22T08:25:13.978 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbsrtowcs.c' -> './k/coreutils-8.5/lib/mbsrtowcs.c' 2023-06-22T08:25:13.978 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fnmatch_loop.c' -> './k/coreutils-8.5/lib/fnmatch_loop.c' 2023-06-22T08:25:13.978 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/safe-write.c' -> './k/coreutils-8.5/lib/safe-write.c' 2023-06-22T08:25:13.978 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mpsort.c' -> './k/coreutils-8.5/lib/mpsort.c' 2023-06-22T08:25:13.979 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/netinet_in.in.h' -> './k/coreutils-8.5/lib/netinet_in.in.h' 2023-06-22T08:25:13.979 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xfreopen.h' -> './k/coreutils-8.5/lib/xfreopen.h' 2023-06-22T08:25:13.979 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/wctype.in.h' -> './k/coreutils-8.5/lib/wctype.in.h' 2023-06-22T08:25:13.979 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stripslash.c' -> './k/coreutils-8.5/lib/stripslash.c' 2023-06-22T08:25:13.979 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/closein.h' -> './k/coreutils-8.5/lib/closein.h' 2023-06-22T08:25:13.979 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memchr2.c' -> './k/coreutils-8.5/lib/memchr2.c' 2023-06-22T08:25:13.979 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memcoll.c' -> './k/coreutils-8.5/lib/memcoll.c' 2023-06-22T08:25:13.979 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/malloca.h' -> './k/coreutils-8.5/lib/malloca.h' 2023-06-22T08:25:14.085 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stpcpy.c' -> './k/coreutils-8.5/lib/stpcpy.c' 2023-06-22T08:25:14.085 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/striconv.c' -> './k/coreutils-8.5/lib/striconv.c' 2023-06-22T08:25:14.085 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/basename.c' -> './k/coreutils-8.5/lib/basename.c' 2023-06-22T08:25:14.085 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sha256.c' -> './k/coreutils-8.5/lib/sha256.c' 2023-06-22T08:25:14.086 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdint.in.h' -> './k/coreutils-8.5/lib/stdint.in.h' 2023-06-22T08:25:14.086 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/acl-internal.h' -> './k/coreutils-8.5/lib/acl-internal.h' 2023-06-22T08:25:14.086 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/atexit.c' -> './k/coreutils-8.5/lib/atexit.c' 2023-06-22T08:25:14.086 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/size_max.h' -> './k/coreutils-8.5/lib/size_max.h' 2023-06-22T08:25:14.086 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/buffer-lcm.h' -> './k/coreutils-8.5/lib/buffer-lcm.h' 2023-06-22T08:25:14.086 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/idcache.h' -> './k/coreutils-8.5/lib/idcache.h' 2023-06-22T08:25:14.087 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/root-dev-ino.c' -> './k/coreutils-8.5/lib/root-dev-ino.c' 2023-06-22T08:25:14.087 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/physmem.c' -> './k/coreutils-8.5/lib/physmem.c' 2023-06-22T08:25:14.087 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mkancesdirs.c' -> './k/coreutils-8.5/lib/mkancesdirs.c' 2023-06-22T08:25:14.087 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/readtokens0.c' -> './k/coreutils-8.5/lib/readtokens0.c' 2023-06-22T08:25:14.087 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fnmatch.in.h' -> './k/coreutils-8.5/lib/fnmatch.in.h' 2023-06-22T08:25:14.087 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dirname.h' -> './k/coreutils-8.5/lib/dirname.h' 2023-06-22T08:25:14.087 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/regex_internal.c' -> './k/coreutils-8.5/lib/regex_internal.c' 2023-06-22T08:25:14.088 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strcspn.c' -> './k/coreutils-8.5/lib/strcspn.c' 2023-06-22T08:25:14.088 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getugroups.h' -> './k/coreutils-8.5/lib/getugroups.h' 2023-06-22T08:25:14.088 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/file-set.c' -> './k/coreutils-8.5/lib/file-set.c' 2023-06-22T08:25:14.088 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sys_socket.in.h' -> './k/coreutils-8.5/lib/sys_socket.in.h' 2023-06-22T08:25:14.088 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/inttostr.h' -> './k/coreutils-8.5/lib/inttostr.h' 2023-06-22T08:25:14.088 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mkdirat.c' -> './k/coreutils-8.5/lib/mkdirat.c' 2023-06-22T08:25:14.089 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/readtokens.h' -> './k/coreutils-8.5/lib/readtokens.h' 2023-06-22T08:25:14.089 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fd-reopen.h' -> './k/coreutils-8.5/lib/fd-reopen.h' 2023-06-22T08:25:14.089 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/w32sock.h' -> './k/coreutils-8.5/lib/w32sock.h' 2023-06-22T08:25:14.089 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memcasecmp.c' -> './k/coreutils-8.5/lib/memcasecmp.c' 2023-06-22T08:25:14.089 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/at-func.c' -> './k/coreutils-8.5/lib/at-func.c' 2023-06-22T08:25:14.089 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unistd-safer.h' -> './k/coreutils-8.5/lib/unistd-safer.h' 2023-06-22T08:25:14.089 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/filevercmp.h' -> './k/coreutils-8.5/lib/filevercmp.h' 2023-06-22T08:25:14.090 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/c-strcasecmp.c' -> './k/coreutils-8.5/lib/c-strcasecmp.c' 2023-06-22T08:25:14.090 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fopen-safer.c' -> './k/coreutils-8.5/lib/fopen-safer.c' 2023-06-22T08:25:14.090 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/modechange.h' -> './k/coreutils-8.5/lib/modechange.h' 2023-06-22T08:25:14.090 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sys_stat.in.h' -> './k/coreutils-8.5/lib/sys_stat.in.h' 2023-06-22T08:25:14.090 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/freading.c' -> './k/coreutils-8.5/lib/freading.c' 2023-06-22T08:25:14.090 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/utimens.h' -> './k/coreutils-8.5/lib/utimens.h' 2023-06-22T08:25:14.090 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/md5.h' -> './k/coreutils-8.5/lib/md5.h' 2023-06-22T08:25:14.091 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/isnanf.c' -> './k/coreutils-8.5/lib/isnanf.c' 2023-06-22T08:25:14.091 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sha1.c' -> './k/coreutils-8.5/lib/sha1.c' 2023-06-22T08:25:14.091 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/filenamecat-lgpl.c' -> './k/coreutils-8.5/lib/filenamecat-lgpl.c' 2023-06-22T08:25:14.091 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/config.hin' -> './k/coreutils-8.5/lib/config.hin' 2023-06-22T08:25:14.091 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/ignore-value.h' -> './k/coreutils-8.5/lib/ignore-value.h' 2023-06-22T08:25:14.091 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strstr.c' -> './k/coreutils-8.5/lib/strstr.c' 2023-06-22T08:25:14.092 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/verror.c' -> './k/coreutils-8.5/lib/verror.c' 2023-06-22T08:25:14.092 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/randperm.h' -> './k/coreutils-8.5/lib/randperm.h' 2023-06-22T08:25:14.092 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/vasprintf.c' -> './k/coreutils-8.5/lib/vasprintf.c' 2023-06-22T08:25:14.092 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/c-strncasecmp.c' -> './k/coreutils-8.5/lib/c-strncasecmp.c' 2023-06-22T08:25:14.092 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/basename-lgpl.c' -> './k/coreutils-8.5/lib/basename-lgpl.c' 2023-06-22T08:25:14.092 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/human.h' -> './k/coreutils-8.5/lib/human.h' 2023-06-22T08:25:14.093 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/regex.h' -> './k/coreutils-8.5/lib/regex.h' 2023-06-22T08:25:14.093 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/regcomp.c' -> './k/coreutils-8.5/lib/regcomp.c' 2023-06-22T08:25:14.093 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/freadahead.c' -> './k/coreutils-8.5/lib/freadahead.c' 2023-06-22T08:25:14.255 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/trim.h' -> './k/coreutils-8.5/lib/trim.h' 2023-06-22T08:25:14.255 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getgroups.c' -> './k/coreutils-8.5/lib/getgroups.c' 2023-06-22T08:25:14.256 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xprintf.c' -> './k/coreutils-8.5/lib/xprintf.c' 2023-06-22T08:25:14.256 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/euidaccess-stat.h' -> './k/coreutils-8.5/lib/euidaccess-stat.h' 2023-06-22T08:25:14.256 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/rand-isaac.h' -> './k/coreutils-8.5/lib/rand-isaac.h' 2023-06-22T08:25:14.256 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mgetgroups.h' -> './k/coreutils-8.5/lib/mgetgroups.h' 2023-06-22T08:25:14.256 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/selinux-at.h' -> './k/coreutils-8.5/lib/selinux-at.h' 2023-06-22T08:25:14.256 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open-solaris.h' -> './k/coreutils-8.5/lib/iconv_open-solaris.h' 2023-06-22T08:25:14.257 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/setenv.c' -> './k/coreutils-8.5/lib/setenv.c' 2023-06-22T08:25:14.257 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/locale.in.h' -> './k/coreutils-8.5/lib/locale.in.h' 2023-06-22T08:25:14.257 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/base64.h' -> './k/coreutils-8.5/lib/base64.h' 2023-06-22T08:25:14.257 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/open-safer.c' -> './k/coreutils-8.5/lib/open-safer.c' 2023-06-22T08:25:14.257 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/exitfail.c' -> './k/coreutils-8.5/lib/exitfail.c' 2023-06-22T08:25:14.257 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/closeout.h' -> './k/coreutils-8.5/lib/closeout.h' 2023-06-22T08:25:14.257 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/idcache.c' -> './k/coreutils-8.5/lib/idcache.c' 2023-06-22T08:25:14.258 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unlink.c' -> './k/coreutils-8.5/lib/unlink.c' 2023-06-22T08:25:14.258 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xfts.c' -> './k/coreutils-8.5/lib/xfts.c' 2023-06-22T08:25:14.258 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memrchr.c' -> './k/coreutils-8.5/lib/memrchr.c' 2023-06-22T08:25:14.258 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/Makefile.am' -> './k/coreutils-8.5/lib/Makefile.am' 2023-06-22T08:25:14.258 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrtol.h' -> './k/coreutils-8.5/lib/xstrtol.h' 2023-06-22T08:25:14.258 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/gettimeofday.c' -> './k/coreutils-8.5/lib/gettimeofday.c' 2023-06-22T08:25:14.259 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/isnanf-nolibm.h' -> './k/coreutils-8.5/lib/isnanf-nolibm.h' 2023-06-22T08:25:14.259 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/chdir-long.c' -> './k/coreutils-8.5/lib/chdir-long.c' 2023-06-22T08:25:14.259 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbchar.c' -> './k/coreutils-8.5/lib/mbchar.c' 2023-06-22T08:25:14.259 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/canon-host.h' -> './k/coreutils-8.5/lib/canon-host.h' 2023-06-22T08:25:14.259 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unsetenv.c' -> './k/coreutils-8.5/lib/unsetenv.c' 2023-06-22T08:25:14.259 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/freadseek.c' -> './k/coreutils-8.5/lib/freadseek.c' 2023-06-22T08:25:14.259 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xnanosleep.c' -> './k/coreutils-8.5/lib/xnanosleep.c' 2023-06-22T08:25:14.260 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/nanosleep.c' -> './k/coreutils-8.5/lib/nanosleep.c' 2023-06-22T08:25:14.260 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/ctype.in.h' -> './k/coreutils-8.5/lib/ctype.in.h' 2023-06-22T08:25:14.260 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xmalloc.c' -> './k/coreutils-8.5/lib/xmalloc.c' 2023-06-22T08:25:14.260 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strtoul.c' -> './k/coreutils-8.5/lib/strtoul.c' 2023-06-22T08:25:14.260 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sha512.h' -> './k/coreutils-8.5/lib/sha512.h' 2023-06-22T08:25:14.260 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dirchownmod.h' -> './k/coreutils-8.5/lib/dirchownmod.h' 2023-06-22T08:25:14.260 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/freadptr.h' -> './k/coreutils-8.5/lib/freadptr.h' 2023-06-22T08:25:14.261 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getfilecon.c' -> './k/coreutils-8.5/lib/getfilecon.c' 2023-06-22T08:25:14.261 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/alignof.h' -> './k/coreutils-8.5/lib/alignof.h' 2023-06-22T08:25:14.261 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mempcpy.c' -> './k/coreutils-8.5/lib/mempcpy.c' 2023-06-22T08:25:14.261 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/savedir.h' -> './k/coreutils-8.5/lib/savedir.h' 2023-06-22T08:25:14.261 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/areadlink-with-size.c' -> './k/coreutils-8.5/lib/areadlink-with-size.c' 2023-06-22T08:25:14.261 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/inttypes.in.h' -> './k/coreutils-8.5/lib/inttypes.in.h' 2023-06-22T08:25:14.261 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/config.charset' -> './k/coreutils-8.5/lib/config.charset' 2023-06-22T08:25:14.262 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrtold.c' -> './k/coreutils-8.5/lib/xstrtold.c' 2023-06-22T08:25:14.262 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fcntl-safer.h' -> './k/coreutils-8.5/lib/fcntl-safer.h' 2023-06-22T08:25:14.262 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrndup.c' -> './k/coreutils-8.5/lib/xstrndup.c' 2023-06-22T08:25:14.262 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/write.c' -> './k/coreutils-8.5/lib/write.c' 2023-06-22T08:25:14.262 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/math.in.h' -> './k/coreutils-8.5/lib/math.in.h' 2023-06-22T08:25:14.262 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/openat-priv.h' -> './k/coreutils-8.5/lib/openat-priv.h' 2023-06-22T08:25:14.262 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/vasnprintf.h' -> './k/coreutils-8.5/lib/vasnprintf.h' 2023-06-22T08:25:14.262 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getcwd.c' -> './k/coreutils-8.5/lib/getcwd.c' 2023-06-22T08:25:14.263 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdbool.in.h' -> './k/coreutils-8.5/lib/stdbool.in.h' 2023-06-22T08:25:14.263 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strtol.c' -> './k/coreutils-8.5/lib/strtol.c' 2023-06-22T08:25:14.362 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/argmatch.c' -> './k/coreutils-8.5/lib/argmatch.c' 2023-06-22T08:25:14.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/acl_entries.c' -> './k/coreutils-8.5/lib/acl_entries.c' 2023-06-22T08:25:14.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/progname.c' -> './k/coreutils-8.5/lib/progname.c' 2023-06-22T08:25:14.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mktime-internal.h' -> './k/coreutils-8.5/lib/mktime-internal.h' 2023-06-22T08:25:14.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/printf-frexpl.c' -> './k/coreutils-8.5/lib/printf-frexpl.c' 2023-06-22T08:25:14.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/offtostr.c' -> './k/coreutils-8.5/lib/offtostr.c' 2023-06-22T08:25:14.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memchr2.h' -> './k/coreutils-8.5/lib/memchr2.h' 2023-06-22T08:25:14.364 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrtoumax.c' -> './k/coreutils-8.5/lib/xstrtoumax.c' 2023-06-22T08:25:14.364 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getdate.y' -> './k/coreutils-8.5/lib/getdate.y' 2023-06-22T08:25:14.364 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/close-hook.h' -> './k/coreutils-8.5/lib/close-hook.h' 2023-06-22T08:25:14.364 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/printf-parse.c' -> './k/coreutils-8.5/lib/printf-parse.c' 2023-06-22T08:25:14.364 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/isnand.c' -> './k/coreutils-8.5/lib/isnand.c' 2023-06-22T08:25:14.365 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/timespec.h' -> './k/coreutils-8.5/lib/timespec.h' 2023-06-22T08:25:14.365 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getusershell.c' -> './k/coreutils-8.5/lib/getusershell.c' 2023-06-22T08:25:14.365 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrtod.c' -> './k/coreutils-8.5/lib/xstrtod.c' 2023-06-22T08:25:14.365 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/verror.h' -> './k/coreutils-8.5/lib/verror.h' 2023-06-22T08:25:14.365 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/same.c' -> './k/coreutils-8.5/lib/same.c' 2023-06-22T08:25:14.365 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/quote.c' -> './k/coreutils-8.5/lib/quote.c' 2023-06-22T08:25:14.366 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/raise.c' -> './k/coreutils-8.5/lib/raise.c' 2023-06-22T08:25:14.366 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/link.c' -> './k/coreutils-8.5/lib/link.c' 2023-06-22T08:25:14.366 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/cloexec.h' -> './k/coreutils-8.5/lib/cloexec.h' 2023-06-22T08:25:14.366 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sys_utsname.in.h' -> './k/coreutils-8.5/lib/sys_utsname.in.h' 2023-06-22T08:25:14.366 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/cloexec.c' -> './k/coreutils-8.5/lib/cloexec.c' 2023-06-22T08:25:14.367 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/vfprintf.c' -> './k/coreutils-8.5/lib/vfprintf.c' 2023-06-22T08:25:14.367 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/opendir-safer.c' -> './k/coreutils-8.5/lib/opendir-safer.c' 2023-06-22T08:25:14.367 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/hash-pjw.h' -> './k/coreutils-8.5/lib/hash-pjw.h' 2023-06-22T08:25:14.367 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/uname.c' -> './k/coreutils-8.5/lib/uname.c' 2023-06-22T08:25:14.367 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/full-read.c' -> './k/coreutils-8.5/lib/full-read.c' 2023-06-22T08:25:14.367 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/priv-set.h' -> './k/coreutils-8.5/lib/priv-set.h' 2023-06-22T08:25:14.368 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/close.c' -> './k/coreutils-8.5/lib/close.c' 2023-06-22T08:25:14.368 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fpending.h' -> './k/coreutils-8.5/lib/fpending.h' 2023-06-22T08:25:14.368 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/i-ring.h' -> './k/coreutils-8.5/lib/i-ring.h' 2023-06-22T08:25:14.368 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/nl_langinfo.c' -> './k/coreutils-8.5/lib/nl_langinfo.c' 2023-06-22T08:25:14.368 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xmemxfrm.h' -> './k/coreutils-8.5/lib/xmemxfrm.h' 2023-06-22T08:25:14.368 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/c-strcase.h' -> './k/coreutils-8.5/lib/c-strcase.h' 2023-06-22T08:25:14.368 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fchdir.c' -> './k/coreutils-8.5/lib/fchdir.c' 2023-06-22T08:25:14.369 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/canon-host.c' -> './k/coreutils-8.5/lib/canon-host.c' 2023-06-22T08:25:14.369 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/signbitf.c' -> './k/coreutils-8.5/lib/signbitf.c' 2023-06-22T08:25:14.369 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xnanosleep.h' -> './k/coreutils-8.5/lib/xnanosleep.h' 2023-06-22T08:25:14.369 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/tempname.c' -> './k/coreutils-8.5/lib/tempname.c' 2023-06-22T08:25:14.369 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xreadlink.c' -> './k/coreutils-8.5/lib/xreadlink.c' 2023-06-22T08:25:14.369 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xgetcwd.h' -> './k/coreutils-8.5/lib/xgetcwd.h' 2023-06-22T08:25:14.369 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/regex.c' -> './k/coreutils-8.5/lib/regex.c' 2023-06-22T08:25:14.369 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/diacrit.h' -> './k/coreutils-8.5/lib/diacrit.h' 2023-06-22T08:25:14.370 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/posixver.c' -> './k/coreutils-8.5/lib/posixver.c' 2023-06-22T08:25:14.370 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getugroups.c' -> './k/coreutils-8.5/lib/getugroups.c' 2023-06-22T08:25:14.370 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/readutmp.h' -> './k/coreutils-8.5/lib/readutmp.h' 2023-06-22T08:25:14.370 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/quote.h' -> './k/coreutils-8.5/lib/quote.h' 2023-06-22T08:25:14.370 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/quotearg.c' -> './k/coreutils-8.5/lib/quotearg.c' 2023-06-22T08:25:14.370 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/filevercmp.c' -> './k/coreutils-8.5/lib/filevercmp.c' 2023-06-22T08:25:14.371 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/printf-args.h' -> './k/coreutils-8.5/lib/printf-args.h' 2023-06-22T08:25:14.371 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mountlist.h' -> './k/coreutils-8.5/lib/mountlist.h' 2023-06-22T08:25:14.371 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrtol-error.c' -> './k/coreutils-8.5/lib/xstrtol-error.c' 2023-06-22T08:25:14.469 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/rename.c' -> './k/coreutils-8.5/lib/rename.c' 2023-06-22T08:25:14.469 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sha1.h' -> './k/coreutils-8.5/lib/sha1.h' 2023-06-22T08:25:14.469 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strndup.c' -> './k/coreutils-8.5/lib/strndup.c' 2023-06-22T08:25:14.470 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/priv-set.c' -> './k/coreutils-8.5/lib/priv-set.c' 2023-06-22T08:25:14.470 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xfts.h' -> './k/coreutils-8.5/lib/xfts.h' 2023-06-22T08:25:14.470 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/argv-iter.c' -> './k/coreutils-8.5/lib/argv-iter.c' 2023-06-22T08:25:14.470 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdio--.h' -> './k/coreutils-8.5/lib/stdio--.h' 2023-06-22T08:25:14.470 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/yesno.c' -> './k/coreutils-8.5/lib/yesno.c' 2023-06-22T08:25:14.470 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getopt.c' -> './k/coreutils-8.5/lib/getopt.c' 2023-06-22T08:25:14.471 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strpbrk.c' -> './k/coreutils-8.5/lib/strpbrk.c' 2023-06-22T08:25:14.471 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dirname-lgpl.c' -> './k/coreutils-8.5/lib/dirname-lgpl.c' 2023-06-22T08:25:14.471 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/filemode.h' -> './k/coreutils-8.5/lib/filemode.h' 2023-06-22T08:25:14.471 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbrtowc.c' -> './k/coreutils-8.5/lib/mbrtowc.c' 2023-06-22T08:25:14.471 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open.c' -> './k/coreutils-8.5/lib/iconv_open.c' 2023-06-22T08:25:14.471 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/euidaccess-stat.c' -> './k/coreutils-8.5/lib/euidaccess-stat.c' 2023-06-22T08:25:14.472 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stat-time.h' -> './k/coreutils-8.5/lib/stat-time.h' 2023-06-22T08:25:14.472 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/frexpl.c' -> './k/coreutils-8.5/lib/frexpl.c' 2023-06-22T08:25:14.472 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sys_select.in.h' -> './k/coreutils-8.5/lib/sys_select.in.h' 2023-06-22T08:25:14.472 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strftime.h' -> './k/coreutils-8.5/lib/strftime.h' 2023-06-22T08:25:14.472 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strftime.c' -> './k/coreutils-8.5/lib/strftime.c' 2023-06-22T08:25:14.472 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open-aix.gperf' -> './k/coreutils-8.5/lib/iconv_open-aix.gperf' 2023-06-22T08:25:14.473 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/buffer-lcm.c' -> './k/coreutils-8.5/lib/buffer-lcm.c' 2023-06-22T08:25:14.473 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unistr.h' -> './k/coreutils-8.5/lib/unistr.h' 2023-06-22T08:25:14.473 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/modechange.c' -> './k/coreutils-8.5/lib/modechange.c' 2023-06-22T08:25:14.473 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/tempname.h' -> './k/coreutils-8.5/lib/tempname.h' 2023-06-22T08:25:14.473 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/faccessat.c' -> './k/coreutils-8.5/lib/faccessat.c' 2023-06-22T08:25:14.473 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unicodeio.h' -> './k/coreutils-8.5/lib/unicodeio.h' 2023-06-22T08:25:14.473 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/filemode.c' -> './k/coreutils-8.5/lib/filemode.c' 2023-06-22T08:25:14.474 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strdup.c' -> './k/coreutils-8.5/lib/strdup.c' 2023-06-22T08:25:14.474 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strsignal.c' -> './k/coreutils-8.5/lib/strsignal.c' 2023-06-22T08:25:14.474 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/euidaccess.c' -> './k/coreutils-8.5/lib/euidaccess.c' 2023-06-22T08:25:14.474 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/select.c' -> './k/coreutils-8.5/lib/select.c' 2023-06-22T08:25:14.474 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memcoll.h' -> './k/coreutils-8.5/lib/memcoll.h' 2023-06-22T08:25:14.474 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fseeko.c' -> './k/coreutils-8.5/lib/fseeko.c' 2023-06-22T08:25:14.475 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/langinfo.in.h' -> './k/coreutils-8.5/lib/langinfo.in.h' 2023-06-22T08:25:14.475 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getopt.in.h' -> './k/coreutils-8.5/lib/getopt.in.h' 2023-06-22T08:25:14.475 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/localcharset.c' -> './k/coreutils-8.5/lib/localcharset.c' 2023-06-22T08:25:14.475 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/isapipe.c' -> './k/coreutils-8.5/lib/isapipe.c' 2023-06-22T08:25:14.475 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/settime.c' -> './k/coreutils-8.5/lib/settime.c' 2023-06-22T08:25:14.475 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/bitrotate.h' -> './k/coreutils-8.5/lib/bitrotate.h' 2023-06-22T08:25:14.475 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/isnanl-nolibm.h' -> './k/coreutils-8.5/lib/isnanl-nolibm.h' 2023-06-22T08:25:14.475 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fcntl.c' -> './k/coreutils-8.5/lib/fcntl.c' 2023-06-22T08:25:14.476 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/exclude.h' -> './k/coreutils-8.5/lib/exclude.h' 2023-06-22T08:25:14.476 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/symlinkat.c' -> './k/coreutils-8.5/lib/symlinkat.c' 2023-06-22T08:25:14.476 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fpending.c' -> './k/coreutils-8.5/lib/fpending.c' 2023-06-22T08:25:14.477 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/savewd.c' -> './k/coreutils-8.5/lib/savewd.c' 2023-06-22T08:25:14.477 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fchownat.c' -> './k/coreutils-8.5/lib/fchownat.c' 2023-06-22T08:25:14.477 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/root-dev-ino.h' -> './k/coreutils-8.5/lib/root-dev-ino.h' 2023-06-22T08:25:14.477 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/randread.h' -> './k/coreutils-8.5/lib/randread.h' 2023-06-22T08:25:14.477 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/freadseek.h' -> './k/coreutils-8.5/lib/freadseek.h' 2023-06-22T08:25:14.477 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strtoll.c' -> './k/coreutils-8.5/lib/strtoll.c' 2023-06-22T08:25:14.477 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdio-safer.h' -> './k/coreutils-8.5/lib/stdio-safer.h' 2023-06-22T08:25:14.478 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strtoumax.c' -> './k/coreutils-8.5/lib/strtoumax.c' 2023-06-22T08:25:14.573 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/argv-iter.h' -> './k/coreutils-8.5/lib/argv-iter.h' 2023-06-22T08:25:14.573 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dirent-safer.h' -> './k/coreutils-8.5/lib/dirent-safer.h' 2023-06-22T08:25:14.573 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dirent.in.h' -> './k/coreutils-8.5/lib/dirent.in.h' 2023-06-22T08:25:14.574 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/malloca.valgrind' -> './k/coreutils-8.5/lib/malloca.valgrind' 2023-06-22T08:25:14.574 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/savewd.h' -> './k/coreutils-8.5/lib/savewd.h' 2023-06-22T08:25:14.574 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/uinttostr.c' -> './k/coreutils-8.5/lib/uinttostr.c' 2023-06-22T08:25:14.574 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getline.c' -> './k/coreutils-8.5/lib/getline.c' 2023-06-22T08:25:14.574 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/chdir-long.h' -> './k/coreutils-8.5/lib/chdir-long.h' 2023-06-22T08:25:14.574 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/cycle-check.h' -> './k/coreutils-8.5/lib/cycle-check.h' 2023-06-22T08:25:14.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fts-cycle.c' -> './k/coreutils-8.5/lib/fts-cycle.c' 2023-06-22T08:25:14.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/file-has-acl.c' -> './k/coreutils-8.5/lib/file-has-acl.c' 2023-06-22T08:25:14.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/areadlink.h' -> './k/coreutils-8.5/lib/areadlink.h' 2023-06-22T08:25:14.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sig2str.h' -> './k/coreutils-8.5/lib/sig2str.h' 2023-06-22T08:25:14.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memcpy.c' -> './k/coreutils-8.5/lib/memcpy.c' 2023-06-22T08:25:14.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/freading.h' -> './k/coreutils-8.5/lib/freading.h' 2023-06-22T08:25:14.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/full-write.h' -> './k/coreutils-8.5/lib/full-write.h' 2023-06-22T08:25:14.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unistd.in.h' -> './k/coreutils-8.5/lib/unistd.in.h' 2023-06-22T08:25:14.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/symlink.c' -> './k/coreutils-8.5/lib/symlink.c' 2023-06-22T08:25:14.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/se-selinux.in.h' -> './k/coreutils-8.5/lib/se-selinux.in.h' 2023-06-22T08:25:14.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/write-any-file.h' -> './k/coreutils-8.5/lib/write-any-file.h' 2023-06-22T08:25:14.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/printf-frexpl.h' -> './k/coreutils-8.5/lib/printf-frexpl.h' 2023-06-22T08:25:14.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/Makefile.in' -> './k/coreutils-8.5/lib/Makefile.in' 2023-06-22T08:25:14.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/filenamecat.h' -> './k/coreutils-8.5/lib/filenamecat.h' 2023-06-22T08:25:14.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/malloca.c' -> './k/coreutils-8.5/lib/malloca.c' 2023-06-22T08:25:14.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/utimecmp.h' -> './k/coreutils-8.5/lib/utimecmp.h' 2023-06-22T08:25:14.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdlib.in.h' -> './k/coreutils-8.5/lib/stdlib.in.h' 2023-06-22T08:25:14.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdio.in.h' -> './k/coreutils-8.5/lib/stdio.in.h' 2023-06-22T08:25:14.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbsinit.c' -> './k/coreutils-8.5/lib/mbsinit.c' 2023-06-22T08:25:14.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/diacrit.c' -> './k/coreutils-8.5/lib/diacrit.c' 2023-06-22T08:25:14.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/se-context.in.h' -> './k/coreutils-8.5/lib/se-context.in.h' 2023-06-22T08:25:14.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xvasprintf.c' -> './k/coreutils-8.5/lib/xvasprintf.c' 2023-06-22T08:25:14.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xgetcwd.c' -> './k/coreutils-8.5/lib/xgetcwd.c' 2023-06-22T08:25:14.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sockets.h' -> './k/coreutils-8.5/lib/sockets.h' 2023-06-22T08:25:14.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/striconv.h' -> './k/coreutils-8.5/lib/striconv.h' 2023-06-22T08:25:14.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mkstemp-safer.c' -> './k/coreutils-8.5/lib/mkstemp-safer.c' 2023-06-22T08:25:14.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/u64.h' -> './k/coreutils-8.5/lib/u64.h' 2023-06-22T08:25:14.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrtoimax.c' -> './k/coreutils-8.5/lib/xstrtoimax.c' 2023-06-22T08:25:14.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/time_r.c' -> './k/coreutils-8.5/lib/time_r.c' 2023-06-22T08:25:14.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbchar.h' -> './k/coreutils-8.5/lib/mbchar.h' 2023-06-22T08:25:14.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xtime.h' -> './k/coreutils-8.5/lib/xtime.h' 2023-06-22T08:25:14.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/set-mode-acl.c' -> './k/coreutils-8.5/lib/set-mode-acl.c' 2023-06-22T08:25:14.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/version-etc.c' -> './k/coreutils-8.5/lib/version-etc.c' 2023-06-22T08:25:14.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dup2.c' -> './k/coreutils-8.5/lib/dup2.c' 2023-06-22T08:25:14.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unlinkat.c' -> './k/coreutils-8.5/lib/unlinkat.c' 2023-06-22T08:25:14.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sigprocmask.c' -> './k/coreutils-8.5/lib/sigprocmask.c' 2023-06-22T08:25:14.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/hash-triple.h' -> './k/coreutils-8.5/lib/hash-triple.h' 2023-06-22T08:25:14.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fprintftime.c' -> './k/coreutils-8.5/lib/fprintftime.c' 2023-06-22T08:25:14.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unistd--.h' -> './k/coreutils-8.5/lib/unistd--.h' 2023-06-22T08:25:14.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/umaxtostr.c' -> './k/coreutils-8.5/lib/umaxtostr.c' 2023-06-22T08:25:14.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/i-ring.c' -> './k/coreutils-8.5/lib/i-ring.c' 2023-06-22T08:25:14.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/signbitd.c' -> './k/coreutils-8.5/lib/signbitd.c' 2023-06-22T08:25:14.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/ftello.c' -> './k/coreutils-8.5/lib/ftello.c' 2023-06-22T08:25:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/lchown.c' -> './k/coreutils-8.5/lib/lchown.c' 2023-06-22T08:25:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/lstat.c' -> './k/coreutils-8.5/lib/lstat.c' 2023-06-22T08:25:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fpurge.c' -> './k/coreutils-8.5/lib/fpurge.c' 2023-06-22T08:25:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fstatat.c' -> './k/coreutils-8.5/lib/fstatat.c' 2023-06-22T08:25:14.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/lseek.c' -> './k/coreutils-8.5/lib/lseek.c' 2023-06-22T08:25:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fchmodat.c' -> './k/coreutils-8.5/lib/fchmodat.c' 2023-06-22T08:25:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/alloca.c' -> './k/coreutils-8.5/lib/alloca.c' 2023-06-22T08:25:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memcmp.c' -> './k/coreutils-8.5/lib/memcmp.c' 2023-06-22T08:25:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrndup.h' -> './k/coreutils-8.5/lib/xstrndup.h' 2023-06-22T08:25:14.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/frexp.c' -> './k/coreutils-8.5/lib/frexp.c' 2023-06-22T08:25:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/nproc.h' -> './k/coreutils-8.5/lib/nproc.h' 2023-06-22T08:25:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fsusage.h' -> './k/coreutils-8.5/lib/fsusage.h' 2023-06-22T08:25:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fd-safer-flag.c' -> './k/coreutils-8.5/lib/fd-safer-flag.c' 2023-06-22T08:25:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getdtablesize.c' -> './k/coreutils-8.5/lib/getdtablesize.c' 2023-06-22T08:25:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/savedir.c' -> './k/coreutils-8.5/lib/savedir.c' 2023-06-22T08:25:14.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/wcwidth.c' -> './k/coreutils-8.5/lib/wcwidth.c' 2023-06-22T08:25:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbsstr.c' -> './k/coreutils-8.5/lib/mbsstr.c' 2023-06-22T08:25:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fpucw.h' -> './k/coreutils-8.5/lib/fpucw.h' 2023-06-22T08:25:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open-hpux.h' -> './k/coreutils-8.5/lib/iconv_open-hpux.h' 2023-06-22T08:25:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/readtokens0.h' -> './k/coreutils-8.5/lib/readtokens0.h' 2023-06-22T08:25:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getdate.c' -> './k/coreutils-8.5/lib/getdate.c' 2023-06-22T08:25:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/utimens.c' -> './k/coreutils-8.5/lib/utimens.c' 2023-06-22T08:25:14.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getndelim2.c' -> './k/coreutils-8.5/lib/getndelim2.c' 2023-06-22T08:25:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fflush.c' -> './k/coreutils-8.5/lib/fflush.c' 2023-06-22T08:25:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/version-etc-fsf.c' -> './k/coreutils-8.5/lib/version-etc-fsf.c' 2023-06-22T08:25:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xsize.h' -> './k/coreutils-8.5/lib/xsize.h' 2023-06-22T08:25:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xgethostname.c' -> './k/coreutils-8.5/lib/xgethostname.c' 2023-06-22T08:25:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strnlen1.h' -> './k/coreutils-8.5/lib/strnlen1.h' 2023-06-22T08:25:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/isnanl.c' -> './k/coreutils-8.5/lib/isnanl.c' 2023-06-22T08:25:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/same-inode.h' -> './k/coreutils-8.5/lib/same-inode.h' 2023-06-22T08:25:14.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/regex_internal.h' -> './k/coreutils-8.5/lib/regex_internal.h' 2023-06-22T08:25:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/full-read.h' -> './k/coreutils-8.5/lib/full-read.h' 2023-06-22T08:25:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/intprops.h' -> './k/coreutils-8.5/lib/intprops.h' 2023-06-22T08:25:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fsusage.c' -> './k/coreutils-8.5/lib/fsusage.c' 2023-06-22T08:25:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/save-cwd.c' -> './k/coreutils-8.5/lib/save-cwd.c' 2023-06-22T08:25:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memchr.valgrind' -> './k/coreutils-8.5/lib/memchr.valgrind' 2023-06-22T08:25:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/exitfail.h' -> './k/coreutils-8.5/lib/exitfail.h' 2023-06-22T08:25:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/printf-frexp.h' -> './k/coreutils-8.5/lib/printf-frexp.h' 2023-06-22T08:25:14.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/isblank.c' -> './k/coreutils-8.5/lib/isblank.c' 2023-06-22T08:25:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/arpa_inet.in.h' -> './k/coreutils-8.5/lib/arpa_inet.in.h' 2023-06-22T08:25:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strerror.c' -> './k/coreutils-8.5/lib/strerror.c' 2023-06-22T08:25:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/freadptr.c' -> './k/coreutils-8.5/lib/freadptr.c' 2023-06-22T08:25:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/openat.h' -> './k/coreutils-8.5/lib/openat.h' 2023-06-22T08:25:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open-irix.h' -> './k/coreutils-8.5/lib/iconv_open-irix.h' 2023-06-22T08:25:14.684 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/open.c' -> './k/coreutils-8.5/lib/open.c' 2023-06-22T08:25:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/vprintf.c' -> './k/coreutils-8.5/lib/vprintf.c' 2023-06-22T08:25:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/same.h' -> './k/coreutils-8.5/lib/same.h' 2023-06-22T08:25:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/gai_strerror.c' -> './k/coreutils-8.5/lib/gai_strerror.c' 2023-06-22T08:25:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/imaxtostr.c' -> './k/coreutils-8.5/lib/imaxtostr.c' 2023-06-22T08:25:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbsalign.h' -> './k/coreutils-8.5/lib/mbsalign.h' 2023-06-22T08:25:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/glthread' -> './k/coreutils-8.5/lib/glthread' 2023-06-22T08:25:14.685 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/glthread/threadlib.c' -> './k/coreutils-8.5/lib/glthread/threadlib.c' 2023-06-22T08:25:14.855 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/glthread/lock.c' -> './k/coreutils-8.5/lib/glthread/lock.c' 2023-06-22T08:25:14.856 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/glthread/tls.h' -> './k/coreutils-8.5/lib/glthread/tls.h' 2023-06-22T08:25:14.856 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/glthread/tls.c' -> './k/coreutils-8.5/lib/glthread/tls.c' 2023-06-22T08:25:14.856 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/glthread/lock.h' -> './k/coreutils-8.5/lib/glthread/lock.h' 2023-06-22T08:25:14.856 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unistr' -> './k/coreutils-8.5/lib/unistr' 2023-06-22T08:25:14.856 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unistr/u8-uctomb-aux.c' -> './k/coreutils-8.5/lib/unistr/u8-uctomb-aux.c' 2023-06-22T08:25:14.856 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unistr/u8-uctomb.c' -> './k/coreutils-8.5/lib/unistr/u8-uctomb.c' 2023-06-22T08:25:14.857 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memcmp2.h' -> './k/coreutils-8.5/lib/memcmp2.h' 2023-06-22T08:25:14.857 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xfreopen.c' -> './k/coreutils-8.5/lib/xfreopen.c' 2023-06-22T08:25:14.857 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/rand-isaac.c' -> './k/coreutils-8.5/lib/rand-isaac.c' 2023-06-22T08:25:14.857 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/full-write.c' -> './k/coreutils-8.5/lib/full-write.c' 2023-06-22T08:25:14.857 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/rmdir.c' -> './k/coreutils-8.5/lib/rmdir.c' 2023-06-22T08:25:14.858 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/save-cwd.h' -> './k/coreutils-8.5/lib/save-cwd.h' 2023-06-22T08:25:14.858 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/at-func2.c' -> './k/coreutils-8.5/lib/at-func2.c' 2023-06-22T08:25:14.858 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/gethrxtime.c' -> './k/coreutils-8.5/lib/gethrxtime.c' 2023-06-22T08:25:14.858 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strtod.c' -> './k/coreutils-8.5/lib/strtod.c' 2023-06-22T08:25:14.858 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbsalign.c' -> './k/coreutils-8.5/lib/mbsalign.c' 2023-06-22T08:25:14.858 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open-osf.gperf' -> './k/coreutils-8.5/lib/iconv_open-osf.gperf' 2023-06-22T08:25:14.858 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/ref-add.sin' -> './k/coreutils-8.5/lib/ref-add.sin' 2023-06-22T08:25:14.858 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memcmp2.c' -> './k/coreutils-8.5/lib/memcmp2.c' 2023-06-22T08:25:14.859 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/gnulib.mk' -> './k/coreutils-8.5/lib/gnulib.mk' 2023-06-22T08:25:14.859 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/ref-del.sin' -> './k/coreutils-8.5/lib/ref-del.sin' 2023-06-22T08:25:14.859 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stpncpy.c' -> './k/coreutils-8.5/lib/stpncpy.c' 2023-06-22T08:25:14.859 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv.in.h' -> './k/coreutils-8.5/lib/iconv.in.h' 2023-06-22T08:25:14.859 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/posixtm.h' -> './k/coreutils-8.5/lib/posixtm.h' 2023-06-22T08:25:14.860 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrtol.c' -> './k/coreutils-8.5/lib/xstrtol.c' 2023-06-22T08:25:14.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getopt1.c' -> './k/coreutils-8.5/lib/getopt1.c' 2023-06-22T08:25:14.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/quotearg.h' -> './k/coreutils-8.5/lib/quotearg.h' 2023-06-22T08:25:14.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/cycle-check.c' -> './k/coreutils-8.5/lib/cycle-check.c' 2023-06-22T08:25:14.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/calloc.c' -> './k/coreutils-8.5/lib/calloc.c' 2023-06-22T08:25:14.861 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/areadlinkat.c' -> './k/coreutils-8.5/lib/areadlinkat.c' 2023-06-22T08:25:14.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/hash-triple.c' -> './k/coreutils-8.5/lib/hash-triple.c' 2023-06-22T08:25:14.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getpass.c' -> './k/coreutils-8.5/lib/getpass.c' 2023-06-22T08:25:14.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstriconv.c' -> './k/coreutils-8.5/lib/xstriconv.c' 2023-06-22T08:25:14.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dirent--.h' -> './k/coreutils-8.5/lib/dirent--.h' 2023-06-22T08:25:14.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sigaction.c' -> './k/coreutils-8.5/lib/sigaction.c' 2023-06-22T08:25:14.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/linkat.c' -> './k/coreutils-8.5/lib/linkat.c' 2023-06-22T08:25:14.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unitypes.h' -> './k/coreutils-8.5/lib/unitypes.h' 2023-06-22T08:25:14.862 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/linebuffer.c' -> './k/coreutils-8.5/lib/linebuffer.c' 2023-06-22T08:25:14.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fnmatch.c' -> './k/coreutils-8.5/lib/fnmatch.c' 2023-06-22T08:25:14.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xasprintf.c' -> './k/coreutils-8.5/lib/xasprintf.c' 2023-06-22T08:25:14.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/printf-parse.h' -> './k/coreutils-8.5/lib/printf-parse.h' 2023-06-22T08:25:14.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/string.in.h' -> './k/coreutils-8.5/lib/string.in.h' 2023-06-22T08:25:14.863 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mkdir.c' -> './k/coreutils-8.5/lib/mkdir.c' 2023-06-22T08:25:14.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fchown-stub.c' -> './k/coreutils-8.5/lib/fchown-stub.c' 2023-06-22T08:25:14.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/human.c' -> './k/coreutils-8.5/lib/human.c' 2023-06-22T08:25:14.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sockets.c' -> './k/coreutils-8.5/lib/sockets.c' 2023-06-22T08:25:14.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/canonicalize.h' -> './k/coreutils-8.5/lib/canonicalize.h' 2023-06-22T08:25:14.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getopt_int.h' -> './k/coreutils-8.5/lib/getopt_int.h' 2023-06-22T08:25:14.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/version-etc.h' -> './k/coreutils-8.5/lib/version-etc.h' 2023-06-22T08:25:14.864 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdlib--.h' -> './k/coreutils-8.5/lib/stdlib--.h' 2023-06-22T08:25:14.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/backupfile.h' -> './k/coreutils-8.5/lib/backupfile.h' 2023-06-22T08:25:14.996 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/file-type.c' -> './k/coreutils-8.5/lib/file-type.c' 2023-06-22T08:25:14.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/file-set.h' -> './k/coreutils-8.5/lib/file-set.h' 2023-06-22T08:25:14.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbuiter.h' -> './k/coreutils-8.5/lib/mbuiter.h' 2023-06-22T08:25:14.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fcntl.in.h' -> './k/coreutils-8.5/lib/fcntl.in.h' 2023-06-22T08:25:14.997 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/inet_ntop.c' -> './k/coreutils-8.5/lib/inet_ntop.c' 2023-06-22T08:25:14.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fseterr.c' -> './k/coreutils-8.5/lib/fseterr.c' 2023-06-22T08:25:14.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/close-stream.h' -> './k/coreutils-8.5/lib/close-stream.h' 2023-06-22T08:25:14.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/freopen-safer.c' -> './k/coreutils-8.5/lib/freopen-safer.c' 2023-06-22T08:25:14.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stat-macros.h' -> './k/coreutils-8.5/lib/stat-macros.h' 2023-06-22T08:25:14.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dirname.c' -> './k/coreutils-8.5/lib/dirname.c' 2023-06-22T08:25:14.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dev-ino.h' -> './k/coreutils-8.5/lib/dev-ino.h' 2023-06-22T08:25:14.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xalloc.h' -> './k/coreutils-8.5/lib/xalloc.h' 2023-06-22T08:25:14.998 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memcasecmp.h' -> './k/coreutils-8.5/lib/memcasecmp.h' 2023-06-22T08:25:14.999 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/hash.c' -> './k/coreutils-8.5/lib/hash.c' 2023-06-22T08:25:14.999 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/userspec.c' -> './k/coreutils-8.5/lib/userspec.c' 2023-06-22T08:25:14.999 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/hard-locale.h' -> './k/coreutils-8.5/lib/hard-locale.h' 2023-06-22T08:25:14.999 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xvasprintf.h' -> './k/coreutils-8.5/lib/xvasprintf.h' 2023-06-22T08:25:14.999 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/wcrtomb.c' -> './k/coreutils-8.5/lib/wcrtomb.c' 2023-06-22T08:25:14.999 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/gethrxtime.h' -> './k/coreutils-8.5/lib/gethrxtime.h' 2023-06-22T08:25:14.999 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fprintftime.h' -> './k/coreutils-8.5/lib/fprintftime.h' 2023-06-22T08:25:14.999 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mgetgroups.c' -> './k/coreutils-8.5/lib/mgetgroups.c' 2023-06-22T08:25:15.000 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/randread.c' -> './k/coreutils-8.5/lib/randread.c' 2023-06-22T08:25:15.000 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstriconv.h' -> './k/coreutils-8.5/lib/xstriconv.h' 2023-06-22T08:25:15.000 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dup-safer-flag.c' -> './k/coreutils-8.5/lib/dup-safer-flag.c' 2023-06-22T08:25:15.000 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/yesno.h' -> './k/coreutils-8.5/lib/yesno.h' 2023-06-22T08:25:15.000 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbscasecmp.c' -> './k/coreutils-8.5/lib/mbscasecmp.c' 2023-06-22T08:25:15.000 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strtoimax.c' -> './k/coreutils-8.5/lib/strtoimax.c' 2023-06-22T08:25:15.000 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbsrtowcs-state.c' -> './k/coreutils-8.5/lib/mbsrtowcs-state.c' 2023-06-22T08:25:15.001 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mkstemp.c' -> './k/coreutils-8.5/lib/mkstemp.c' 2023-06-22T08:25:15.001 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xmemcoll.c' -> './k/coreutils-8.5/lib/xmemcoll.c' 2023-06-22T08:25:15.001 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strintcmp.c' -> './k/coreutils-8.5/lib/strintcmp.c' 2023-06-22T08:25:15.001 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fclose.c' -> './k/coreutils-8.5/lib/fclose.c' 2023-06-22T08:25:15.001 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fd-reopen.c' -> './k/coreutils-8.5/lib/fd-reopen.c' 2023-06-22T08:25:15.001 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/inttostr.c' -> './k/coreutils-8.5/lib/inttostr.c' 2023-06-22T08:25:15.001 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/c-ctype.c' -> './k/coreutils-8.5/lib/c-ctype.c' 2023-06-22T08:25:15.001 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/openat-die.c' -> './k/coreutils-8.5/lib/openat-die.c' 2023-06-22T08:25:15.002 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/error.c' -> './k/coreutils-8.5/lib/error.c' 2023-06-22T08:25:15.002 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sig2str.c' -> './k/coreutils-8.5/lib/sig2str.c' 2023-06-22T08:25:15.002 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unlocked-io.h' -> './k/coreutils-8.5/lib/unlocked-io.h' 2023-06-22T08:25:15.002 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memxfrm.c' -> './k/coreutils-8.5/lib/memxfrm.c' 2023-06-22T08:25:15.002 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/pipe-safer.c' -> './k/coreutils-8.5/lib/pipe-safer.c' 2023-06-22T08:25:15.002 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/uniwidth' -> './k/coreutils-8.5/lib/uniwidth' 2023-06-22T08:25:15.002 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/uniwidth/width.c' -> './k/coreutils-8.5/lib/uniwidth/width.c' 2023-06-22T08:25:15.002 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/uniwidth/cjk.h' -> './k/coreutils-8.5/lib/uniwidth/cjk.h' 2023-06-22T08:25:15.003 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strtoull.c' -> './k/coreutils-8.5/lib/strtoull.c' 2023-06-22T08:25:15.003 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/backupfile.c' -> './k/coreutils-8.5/lib/backupfile.c' 2023-06-22T08:25:15.003 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open-osf.h' -> './k/coreutils-8.5/lib/iconv_open-osf.h' 2023-06-22T08:25:15.003 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/linebuffer.h' -> './k/coreutils-8.5/lib/linebuffer.h' 2023-06-22T08:25:15.003 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xprintf.h' -> './k/coreutils-8.5/lib/xprintf.h' 2023-06-22T08:25:15.003 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/exclude.c' -> './k/coreutils-8.5/lib/exclude.c' 2023-06-22T08:25:15.004 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/userspec.h' -> './k/coreutils-8.5/lib/userspec.h' 2023-06-22T08:25:15.004 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mkancesdirs.h' -> './k/coreutils-8.5/lib/mkancesdirs.h' 2023-06-22T08:25:15.103 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/hash-pjw.c' -> './k/coreutils-8.5/lib/hash-pjw.c' 2023-06-22T08:25:15.104 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fd-safer.c' -> './k/coreutils-8.5/lib/fd-safer.c' 2023-06-22T08:25:15.104 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/long-options.h' -> './k/coreutils-8.5/lib/long-options.h' 2023-06-22T08:25:15.104 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/freadahead.h' -> './k/coreutils-8.5/lib/freadahead.h' 2023-06-22T08:25:15.104 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/gethostname.c' -> './k/coreutils-8.5/lib/gethostname.c' 2023-06-22T08:25:15.104 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/putenv.c' -> './k/coreutils-8.5/lib/putenv.c' 2023-06-22T08:25:15.104 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbrlen.c' -> './k/coreutils-8.5/lib/mbrlen.c' 2023-06-22T08:25:15.105 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/group-member.c' -> './k/coreutils-8.5/lib/group-member.c' 2023-06-22T08:25:15.105 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sha512.c' -> './k/coreutils-8.5/lib/sha512.c' 2023-06-22T08:25:15.105 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/signal.in.h' -> './k/coreutils-8.5/lib/signal.in.h' 2023-06-22T08:25:15.105 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mktime.c' -> './k/coreutils-8.5/lib/mktime.c' 2023-06-22T08:25:15.105 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/readlink.c' -> './k/coreutils-8.5/lib/readlink.c' 2023-06-22T08:25:15.105 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mkdir-p.c' -> './k/coreutils-8.5/lib/mkdir-p.c' 2023-06-22T08:25:15.105 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getndelim2.h' -> './k/coreutils-8.5/lib/getndelim2.h' 2023-06-22T08:25:15.106 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memxfrm.h' -> './k/coreutils-8.5/lib/memxfrm.h' 2023-06-22T08:25:15.106 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/c-strtod.c' -> './k/coreutils-8.5/lib/c-strtod.c' 2023-06-22T08:25:15.106 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fseterr.h' -> './k/coreutils-8.5/lib/fseterr.h' 2023-06-22T08:25:15.106 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open-hpux.gperf' -> './k/coreutils-8.5/lib/iconv_open-hpux.gperf' 2023-06-22T08:25:15.106 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/filenamecat.c' -> './k/coreutils-8.5/lib/filenamecat.c' 2023-06-22T08:25:15.106 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memmove.c' -> './k/coreutils-8.5/lib/memmove.c' 2023-06-22T08:25:15.106 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/str-kmp.h' -> './k/coreutils-8.5/lib/str-kmp.h' 2023-06-22T08:25:15.107 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/base64.c' -> './k/coreutils-8.5/lib/base64.c' 2023-06-22T08:25:15.107 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/close-hook.c' -> './k/coreutils-8.5/lib/close-hook.c' 2023-06-22T08:25:15.107 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/dirchownmod.c' -> './k/coreutils-8.5/lib/dirchownmod.c' 2023-06-22T08:25:15.107 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/openat-safer.c' -> './k/coreutils-8.5/lib/openat-safer.c' 2023-06-22T08:25:15.107 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/posixtm.c' -> './k/coreutils-8.5/lib/posixtm.c' 2023-06-22T08:25:15.107 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/printf-frexp.c' -> './k/coreutils-8.5/lib/printf-frexp.c' 2023-06-22T08:25:15.107 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/safe-read.h' -> './k/coreutils-8.5/lib/safe-read.h' 2023-06-22T08:25:15.107 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/regexec.c' -> './k/coreutils-8.5/lib/regexec.c' 2023-06-22T08:25:15.108 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/uniwidth.h' -> './k/coreutils-8.5/lib/uniwidth.h' 2023-06-22T08:25:15.108 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/write-any-file.c' -> './k/coreutils-8.5/lib/write-any-file.c' 2023-06-22T08:25:15.108 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open-aix.h' -> './k/coreutils-8.5/lib/iconv_open-aix.h' 2023-06-22T08:25:15.108 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/unicodeio.c' -> './k/coreutils-8.5/lib/unicodeio.c' 2023-06-22T08:25:15.108 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/gettime.c' -> './k/coreutils-8.5/lib/gettime.c' 2023-06-22T08:25:15.108 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strnumcmp.h' -> './k/coreutils-8.5/lib/strnumcmp.h' 2023-06-22T08:25:15.108 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/btowc.c' -> './k/coreutils-8.5/lib/btowc.c' 2023-06-22T08:25:15.109 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sys_time.in.h' -> './k/coreutils-8.5/lib/sys_time.in.h' 2023-06-22T08:25:15.109 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/isnan.c' -> './k/coreutils-8.5/lib/isnan.c' 2023-06-22T08:25:15.109 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getaddrinfo.c' -> './k/coreutils-8.5/lib/getaddrinfo.c' 2023-06-22T08:25:15.109 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getpagesize.c' -> './k/coreutils-8.5/lib/getpagesize.c' 2023-06-22T08:25:15.109 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/posixver.h' -> './k/coreutils-8.5/lib/posixver.h' 2023-06-22T08:25:15.109 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/openat.c' -> './k/coreutils-8.5/lib/openat.c' 2023-06-22T08:25:15.109 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/closeout.c' -> './k/coreutils-8.5/lib/closeout.c' 2023-06-22T08:25:15.109 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fopen.c' -> './k/coreutils-8.5/lib/fopen.c' 2023-06-22T08:25:15.110 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memchr2.valgrind' -> './k/coreutils-8.5/lib/memchr2.valgrind' 2023-06-22T08:25:15.110 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/time.in.h' -> './k/coreutils-8.5/lib/time.in.h' 2023-06-22T08:25:15.110 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/physmem.h' -> './k/coreutils-8.5/lib/physmem.h' 2023-06-22T08:25:15.110 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/iconv_open-irix.gperf' -> './k/coreutils-8.5/lib/iconv_open-irix.gperf' 2023-06-22T08:25:15.110 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stat.c' -> './k/coreutils-8.5/lib/stat.c' 2023-06-22T08:25:15.110 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/siglist.h' -> './k/coreutils-8.5/lib/siglist.h' 2023-06-22T08:25:15.110 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbswidth.h' -> './k/coreutils-8.5/lib/mbswidth.h' 2023-06-22T08:25:15.111 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrtod.h' -> './k/coreutils-8.5/lib/xstrtod.h' 2023-06-22T08:25:15.201 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-06-22T08:25:15.201 INFO:tasks.ceph.mds.e:Restarting daemon 2023-06-22T08:25:15.202 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-06-22T08:25:15.207 INFO:tasks.ceph.mds.e:Started 2023-06-22T08:25:15.208 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T08:25:15.213 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/md5.c' -> './k/coreutils-8.5/lib/md5.c' 2023-06-22T08:25:15.214 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getloadavg.c' -> './k/coreutils-8.5/lib/getloadavg.c' 2023-06-22T08:25:15.214 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/getdate.h' -> './k/coreutils-8.5/lib/getdate.h' 2023-06-22T08:25:15.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdlib-safer.h' -> './k/coreutils-8.5/lib/stdlib-safer.h' 2023-06-22T08:25:15.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/remove.c' -> './k/coreutils-8.5/lib/remove.c' 2023-06-22T08:25:15.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/malloc.c' -> './k/coreutils-8.5/lib/malloc.c' 2023-06-22T08:25:15.215 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/progname.h' -> './k/coreutils-8.5/lib/progname.h' 2023-06-22T08:25:15.216 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/argmatch.h' -> './k/coreutils-8.5/lib/argmatch.h' 2023-06-22T08:25:15.216 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/ChangeLog-2007' -> './k/coreutils-8.5/lib/ChangeLog-2007' 2023-06-22T08:25:15.216 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mbswidth.c' -> './k/coreutils-8.5/lib/mbswidth.c' 2023-06-22T08:25:15.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/error.h' -> './k/coreutils-8.5/lib/error.h' 2023-06-22T08:25:15.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/obstack.c' -> './k/coreutils-8.5/lib/obstack.c' 2023-06-22T08:25:15.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/acl.h' -> './k/coreutils-8.5/lib/acl.h' 2023-06-22T08:25:15.217 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strnumcmp-in.h' -> './k/coreutils-8.5/lib/strnumcmp-in.h' 2023-06-22T08:25:15.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xstrtoul.c' -> './k/coreutils-8.5/lib/xstrtoul.c' 2023-06-22T08:25:15.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/float+.h' -> './k/coreutils-8.5/lib/float+.h' 2023-06-22T08:25:15.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/canonicalize.c' -> './k/coreutils-8.5/lib/canonicalize.c' 2023-06-22T08:25:15.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fcntl--.h' -> './k/coreutils-8.5/lib/fcntl--.h' 2023-06-22T08:25:15.218 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/wchar.in.h' -> './k/coreutils-8.5/lib/wchar.in.h' 2023-06-22T08:25:15.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/trim.c' -> './k/coreutils-8.5/lib/trim.c' 2023-06-22T08:25:15.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/rpmatch.c' -> './k/coreutils-8.5/lib/rpmatch.c' 2023-06-22T08:25:15.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/isapipe.h' -> './k/coreutils-8.5/lib/isapipe.h' 2023-06-22T08:25:15.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/memchr.c' -> './k/coreutils-8.5/lib/memchr.c' 2023-06-22T08:25:15.219 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/localcharset.h' -> './k/coreutils-8.5/lib/localcharset.h' 2023-06-22T08:25:15.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/sig-handler.h' -> './k/coreutils-8.5/lib/sig-handler.h' 2023-06-22T08:25:15.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/openat-proc.c' -> './k/coreutils-8.5/lib/openat-proc.c' 2023-06-22T08:25:15.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/pathmax.h' -> './k/coreutils-8.5/lib/pathmax.h' 2023-06-22T08:25:15.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/close-stream.c' -> './k/coreutils-8.5/lib/close-stream.c' 2023-06-22T08:25:15.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/readutmp.c' -> './k/coreutils-8.5/lib/readutmp.c' 2023-06-22T08:25:15.220 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/float.in.h' -> './k/coreutils-8.5/lib/float.in.h' 2023-06-22T08:25:15.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/creat-safer.c' -> './k/coreutils-8.5/lib/creat-safer.c' 2023-06-22T08:25:15.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/randperm.c' -> './k/coreutils-8.5/lib/randperm.c' 2023-06-22T08:25:15.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/readtokens.c' -> './k/coreutils-8.5/lib/readtokens.c' 2023-06-22T08:25:15.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/safe-read.c' -> './k/coreutils-8.5/lib/safe-read.c' 2023-06-22T08:25:15.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xreadlink.h' -> './k/coreutils-8.5/lib/xreadlink.h' 2023-06-22T08:25:15.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/stdarg.in.h' -> './k/coreutils-8.5/lib/stdarg.in.h' 2023-06-22T08:25:15.221 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xalloc-die.c' -> './k/coreutils-8.5/lib/xalloc-die.c' 2023-06-22T08:25:15.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/copy-acl.c' -> './k/coreutils-8.5/lib/copy-acl.c' 2023-06-22T08:25:15.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fsync.c' -> './k/coreutils-8.5/lib/fsync.c' 2023-06-22T08:25:15.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/strnumcmp.c' -> './k/coreutils-8.5/lib/strnumcmp.c' 2023-06-22T08:25:15.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/selinux-at.c' -> './k/coreutils-8.5/lib/selinux-at.c' 2023-06-22T08:25:15.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/realloc.c' -> './k/coreutils-8.5/lib/realloc.c' 2023-06-22T08:25:15.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fdopendir.c' -> './k/coreutils-8.5/lib/fdopendir.c' 2023-06-22T08:25:15.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/alloca.in.h' -> './k/coreutils-8.5/lib/alloca.in.h' 2023-06-22T08:25:15.222 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/freopen.c' -> './k/coreutils-8.5/lib/freopen.c' 2023-06-22T08:25:15.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/mkdir-p.h' -> './k/coreutils-8.5/lib/mkdir-p.h' 2023-06-22T08:25:15.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/isnand-nolibm.h' -> './k/coreutils-8.5/lib/isnand-nolibm.h' 2023-06-22T08:25:15.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/propername.c' -> './k/coreutils-8.5/lib/propername.c' 2023-06-22T08:25:15.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/fileblocks.c' -> './k/coreutils-8.5/lib/fileblocks.c' 2023-06-22T08:25:15.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xmemcoll.h' -> './k/coreutils-8.5/lib/xmemcoll.h' 2023-06-22T08:25:15.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/hash.h' -> './k/coreutils-8.5/lib/hash.h' 2023-06-22T08:25:15.223 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/c-strtold.c' -> './k/coreutils-8.5/lib/c-strtold.c' 2023-06-22T08:25:15.368 INFO:tasks.ceph.mds.e.smithi023.stdout:starting mds.e at 2023-06-22T08:25:15.435 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/c-ctype.h' -> './k/coreutils-8.5/lib/c-ctype.h' 2023-06-22T08:25:15.435 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/lib/xmemxfrm.c' -> './k/coreutils-8.5/lib/xmemxfrm.c' 2023-06-22T08:25:15.435 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_require_config_h_first' -> './k/coreutils-8.5/.x-sc_require_config_h_first' 2023-06-22T08:25:15.435 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po' -> './k/coreutils-8.5/po' 2023-06-22T08:25:15.436 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/sl.po' -> './k/coreutils-8.5/po/sl.po' 2023-06-22T08:25:15.436 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/zh_TW.po' -> './k/coreutils-8.5/po/zh_TW.po' 2023-06-22T08:25:15.436 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/sk.po' -> './k/coreutils-8.5/po/sk.po' 2023-06-22T08:25:15.436 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/el.gmo' -> './k/coreutils-8.5/po/el.gmo' 2023-06-22T08:25:15.436 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/en.gmo' -> './k/coreutils-8.5/po/en.gmo' 2023-06-22T08:25:15.436 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/lg.po' -> './k/coreutils-8.5/po/lg.po' 2023-06-22T08:25:15.437 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/sv.gmo' -> './k/coreutils-8.5/po/sv.gmo' 2023-06-22T08:25:15.437 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/pl.gmo' -> './k/coreutils-8.5/po/pl.gmo' 2023-06-22T08:25:15.437 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/insert-header.sin' -> './k/coreutils-8.5/po/insert-header.sin' 2023-06-22T08:25:15.437 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/sk.gmo' -> './k/coreutils-8.5/po/sk.gmo' 2023-06-22T08:25:15.437 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ja.gmo' -> './k/coreutils-8.5/po/ja.gmo' 2023-06-22T08:25:15.437 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ko.po' -> './k/coreutils-8.5/po/ko.po' 2023-06-22T08:25:15.437 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/da.gmo' -> './k/coreutils-8.5/po/da.gmo' 2023-06-22T08:25:15.438 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/LINGUAS' -> './k/coreutils-8.5/po/LINGUAS' 2023-06-22T08:25:15.438 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/zh_TW.gmo' -> './k/coreutils-8.5/po/zh_TW.gmo' 2023-06-22T08:25:15.438 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/tr.gmo' -> './k/coreutils-8.5/po/tr.gmo' 2023-06-22T08:25:15.438 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/gl.gmo' -> './k/coreutils-8.5/po/gl.gmo' 2023-06-22T08:25:15.438 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/fi.po' -> './k/coreutils-8.5/po/fi.po' 2023-06-22T08:25:15.438 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/da.po' -> './k/coreutils-8.5/po/da.po' 2023-06-22T08:25:15.438 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/fi.gmo' -> './k/coreutils-8.5/po/fi.gmo' 2023-06-22T08:25:15.438 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/quot.sed' -> './k/coreutils-8.5/po/quot.sed' 2023-06-22T08:25:15.439 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/stamp-po' -> './k/coreutils-8.5/po/stamp-po' 2023-06-22T08:25:15.439 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ms.gmo' -> './k/coreutils-8.5/po/ms.gmo' 2023-06-22T08:25:15.439 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/pt_BR.po' -> './k/coreutils-8.5/po/pt_BR.po' 2023-06-22T08:25:15.439 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/Makevars' -> './k/coreutils-8.5/po/Makevars' 2023-06-22T08:25:15.439 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/boldquot.sed' -> './k/coreutils-8.5/po/boldquot.sed' 2023-06-22T08:25:15.439 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/coreutils.pot' -> './k/coreutils-8.5/po/coreutils.pot' 2023-06-22T08:25:15.439 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/de.po' -> './k/coreutils-8.5/po/de.po' 2023-06-22T08:25:15.440 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/pt.gmo' -> './k/coreutils-8.5/po/pt.gmo' 2023-06-22T08:25:15.440 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/Makefile.in.in' -> './k/coreutils-8.5/po/Makefile.in.in' 2023-06-22T08:25:15.440 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/nl.gmo' -> './k/coreutils-8.5/po/nl.gmo' 2023-06-22T08:25:15.440 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/fr.gmo' -> './k/coreutils-8.5/po/fr.gmo' 2023-06-22T08:25:15.440 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ga.gmo' -> './k/coreutils-8.5/po/ga.gmo' 2023-06-22T08:25:15.440 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/cs.po' -> './k/coreutils-8.5/po/cs.po' 2023-06-22T08:25:15.441 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ru.po' -> './k/coreutils-8.5/po/ru.po' 2023-06-22T08:25:15.441 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/hu.po' -> './k/coreutils-8.5/po/hu.po' 2023-06-22T08:25:15.441 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/id.po' -> './k/coreutils-8.5/po/id.po' 2023-06-22T08:25:15.441 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ru.gmo' -> './k/coreutils-8.5/po/ru.gmo' 2023-06-22T08:25:15.441 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/sl.gmo' -> './k/coreutils-8.5/po/sl.gmo' 2023-06-22T08:25:15.441 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/remove-potcdate.sin' -> './k/coreutils-8.5/po/remove-potcdate.sin' 2023-06-22T08:25:15.441 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/hu.gmo' -> './k/coreutils-8.5/po/hu.gmo' 2023-06-22T08:25:15.441 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/et.po' -> './k/coreutils-8.5/po/et.po' 2023-06-22T08:25:15.442 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/gl.po' -> './k/coreutils-8.5/po/gl.po' 2023-06-22T08:25:15.442 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/vi.gmo' -> './k/coreutils-8.5/po/vi.gmo' 2023-06-22T08:25:15.442 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/bg.po' -> './k/coreutils-8.5/po/bg.po' 2023-06-22T08:25:15.442 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/pl.po' -> './k/coreutils-8.5/po/pl.po' 2023-06-22T08:25:15.442 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/nl.po' -> './k/coreutils-8.5/po/nl.po' 2023-06-22T08:25:15.442 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/pt_BR.gmo' -> './k/coreutils-8.5/po/pt_BR.gmo' 2023-06-22T08:25:15.442 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ko.gmo' -> './k/coreutils-8.5/po/ko.gmo' 2023-06-22T08:25:15.442 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/de.gmo' -> './k/coreutils-8.5/po/de.gmo' 2023-06-22T08:25:15.443 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/POTFILES.in' -> './k/coreutils-8.5/po/POTFILES.in' 2023-06-22T08:25:15.443 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/sv.po' -> './k/coreutils-8.5/po/sv.po' 2023-06-22T08:25:15.443 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/it.po' -> './k/coreutils-8.5/po/it.po' 2023-06-22T08:25:15.443 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/eu.gmo' -> './k/coreutils-8.5/po/eu.gmo' 2023-06-22T08:25:15.443 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/en.po' -> './k/coreutils-8.5/po/en.po' 2023-06-22T08:25:15.443 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/it.gmo' -> './k/coreutils-8.5/po/it.gmo' 2023-06-22T08:25:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.445+0000 7f01d1fff700 1 Processor -- start 2023-06-22T08:25:15.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.446+0000 7f01d1fff700 1 -- start start 2023-06-22T08:25:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.447+0000 7f01d1fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c9fd0 0x7f01cc0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:15.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.447+0000 7f01d1fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f01cc0c40b0 0x7f01cc0c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:15.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.447+0000 7f01d1fff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01cc15c370 con 0x7f01cc152ce0 2023-06-22T08:25:15.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.447+0000 7f01d1fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01cc1530b0 con 0x7f01cc0c40b0 2023-06-22T08:25:15.450 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.447+0000 7f01d1fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc0cf6f0 con 0x7f01cc0c9fd0 2023-06-22T08:25:15.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.447+0000 7f01cb7fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f01cc152ce0 0x7f01cc15c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:42398/0 (socket says 172.21.15.23:42398) 2023-06-22T08:25:15.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.447+0000 7f01cb7fe700 1 -- 172.21.15.23:0/2245947495 learned_addr learned my addr 172.21.15.23:0/2245947495 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:25:15.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.447+0000 7f01cbfff700 1 --2- 172.21.15.23:0/2245947495 >> v2:172.21.15.67:3300/0 conn(0x7f01cc0c40b0 0x7f01cc0c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:15.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.447+0000 7f01caffd700 1 --2- 172.21.15.23:0/2245947495 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c9fd0 0x7f01cc0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:15.451 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01ca7fc700 1 -- 172.21.15.23:0/2245947495 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864725837 0 0) 0x7f01cc15c370 con 0x7f01cc152ce0 2023-06-22T08:25:15.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01ca7fc700 1 -- 172.21.15.23:0/2245947495 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b40035b0 con 0x7f01cc152ce0 2023-06-22T08:25:15.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01cbfff700 1 -- 172.21.15.23:0/2245947495 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c9fd0 msgr2=0x7f01cc0cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:15.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01cbfff700 1 --2- 172.21.15.23:0/2245947495 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c9fd0 0x7f01cc0cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01cbfff700 1 -- 172.21.15.23:0/2245947495 >> v1:172.21.15.67:6789/0 conn(0x7f01cc152ce0 legacy=0x7f01cc15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:15.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01cbfff700 1 -- 172.21.15.23:0/2245947495 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc15bda0 con 0x7f01cc0c40b0 2023-06-22T08:25:15.452 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01caffd700 1 --2- 172.21.15.23:0/2245947495 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c9fd0 0x7f01cc0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:25:15.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01cbfff700 1 --2- 172.21.15.23:0/2245947495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc0c40b0 0x7f01cc0c4530 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009d90 tx=0x7f01c400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0f944fb165d13dd server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:15.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01ca7fc700 1 -- 172.21.15.23:0/2245947495 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c400e070 con 0x7f01cc0c40b0 2023-06-22T08:25:15.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01ca7fc700 1 -- 172.21.15.23:0/2245947495 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c400bb20 con 0x7f01cc0c40b0 2023-06-22T08:25:15.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.448+0000 7f01ca7fc700 1 -- 172.21.15.23:0/2245947495 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4012430 con 0x7f01cc0c40b0 2023-06-22T08:25:15.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.449+0000 7f01d1fff700 1 -- 172.21.15.23:0/2245947495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc0c40b0 msgr2=0x7f01cc0c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:15.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.449+0000 7f01d1fff700 1 --2- 172.21.15.23:0/2245947495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc0c40b0 0x7f01cc0c4530 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009d90 tx=0x7f01c400b330 comp rx=0 tx=0).stop 2023-06-22T08:25:15.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.449+0000 7f01d1fff700 1 -- 172.21.15.23:0/2245947495 shutdown_connections 2023-06-22T08:25:15.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.449+0000 7f01d1fff700 1 --2- 172.21.15.23:0/2245947495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc0c40b0 0x7f01cc0c4530 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.454 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.449+0000 7f01d1fff700 1 --2- 172.21.15.23:0/2245947495 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c9fd0 0x7f01cc0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.449+0000 7f01d1fff700 1 -- 172.21.15.23:0/2245947495 >> 172.21.15.23:0/2245947495 conn(0x7f01cc06ae30 msgr2=0x7f01cc06b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:15.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.449+0000 7f01d1fff700 1 -- 172.21.15.23:0/2245947495 shutdown_connections 2023-06-22T08:25:15.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.449+0000 7f01d1fff700 1 -- 172.21.15.23:0/2245947495 wait complete. 2023-06-22T08:25:15.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.451+0000 7f01d1fff700 1 Processor -- start 2023-06-22T08:25:15.455 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.452+0000 7f01d1fff700 1 -- start start 2023-06-22T08:25:15.456 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01d1fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c40b0 0x7f01cc0c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:15.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01d1fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01cc0c9fd0 0x7f01cc0ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:15.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01d1fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc152ce0 0x7f01cc1e38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:15.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01d1fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc1f1840 con 0x7f01cc0c9fd0 2023-06-22T08:25:15.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01d1fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc1f1ae0 con 0x7f01cc152ce0 2023-06-22T08:25:15.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01d1fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc1f1dc0 con 0x7f01cc0c40b0 2023-06-22T08:25:15.457 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01cb7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c40b0 0x7f01cc0c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:15.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01cbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc152ce0 0x7f01cc1e38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:15.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01cbfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc152ce0 0x7f01cc1e38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:33876/0 (socket says 172.21.15.23:33876) 2023-06-22T08:25:15.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01caffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01cc0c9fd0 0x7f01cc0ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:15.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01cbfff700 1 -- 172.21.15.23:0/555173810 learned_addr learned my addr 172.21.15.23:0/555173810 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:15.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01cbfff700 1 -- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c40b0 msgr2=0x7f01cc0c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:15.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.453+0000 7f01cbfff700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c40b0 0x7f01cc0c7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.454+0000 7f01cbfff700 1 -- 172.21.15.23:0/555173810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01cc0c9fd0 msgr2=0x7f01cc0ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:15.458 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.454+0000 7f01cbfff700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01cc0c9fd0 0x7f01cc0ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.454+0000 7f01cbfff700 1 -- 172.21.15.23:0/555173810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00094a0 con 0x7f01cc152ce0 2023-06-22T08:25:15.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.454+0000 7f01cbfff700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc152ce0 0x7f01cc1e38f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f01c4014040 tx=0x7f01c4004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:15.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.454+0000 7f01b3fff700 1 -- 172.21.15.23:0/555173810 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c40129f0 con 0x7f01cc152ce0 2023-06-22T08:25:15.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.454+0000 7f01b3fff700 1 -- 172.21.15.23:0/555173810 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c4020070 con 0x7f01cc152ce0 2023-06-22T08:25:15.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.454+0000 7f01b3fff700 1 -- 172.21.15.23:0/555173810 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4012dd0 con 0x7f01cc152ce0 2023-06-22T08:25:15.459 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.454+0000 7f01d1fff700 1 -- 172.21.15.23:0/555173810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c4009a70 con 0x7f01cc152ce0 2023-06-22T08:25:15.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.455+0000 7f01d1fff700 1 -- 172.21.15.23:0/555173810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01cc1f2380 con 0x7f01cc152ce0 2023-06-22T08:25:15.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.456+0000 7f01d1fff700 1 -- 172.21.15.23:0/555173810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01cc04ea20 con 0x7f01cc152ce0 2023-06-22T08:25:15.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.457+0000 7f01b3fff700 1 -- 172.21.15.23:0/555173810 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f01c4004040 con 0x7f01cc152ce0 2023-06-22T08:25:15.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.458+0000 7f01b3fff700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01b806ad10 0x7f01b806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:15.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.458+0000 7f01b3fff700 1 -- 172.21.15.23:0/555173810 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7f01c4022070 con 0x7f01cc152ce0 2023-06-22T08:25:15.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.458+0000 7f01cb7fe700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01b806ad10 0x7f01b806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:15.461 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.459+0000 7f01cb7fe700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01b806ad10 0x7f01b806d1c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f01bc007fd0 tx=0x7f01bc007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:15.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.463+0000 7f01b3fff700 1 -- 172.21.15.23:0/555173810 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f01c4056160 con 0x7f01cc152ce0 2023-06-22T08:25:15.595 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/en@boldquot.header' -> './k/coreutils-8.5/po/en@boldquot.header' 2023-06-22T08:25:15.595 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/et.gmo' -> './k/coreutils-8.5/po/et.gmo' 2023-06-22T08:25:15.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ja.po' -> './k/coreutils-8.5/po/ja.po' 2023-06-22T08:25:15.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/zh_CN.gmo' -> './k/coreutils-8.5/po/zh_CN.gmo' 2023-06-22T08:25:15.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/uk.po' -> './k/coreutils-8.5/po/uk.po' 2023-06-22T08:25:15.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ca.gmo' -> './k/coreutils-8.5/po/ca.gmo' 2023-06-22T08:25:15.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/Rules-quot' -> './k/coreutils-8.5/po/Rules-quot' 2023-06-22T08:25:15.596 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ro.po' -> './k/coreutils-8.5/po/ro.po' 2023-06-22T08:25:15.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/nb.gmo' -> './k/coreutils-8.5/po/nb.gmo' 2023-06-22T08:25:15.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/bg.gmo' -> './k/coreutils-8.5/po/bg.gmo' 2023-06-22T08:25:15.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/be.po' -> './k/coreutils-8.5/po/be.po' 2023-06-22T08:25:15.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/vi.po' -> './k/coreutils-8.5/po/vi.po' 2023-06-22T08:25:15.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/lt.gmo' -> './k/coreutils-8.5/po/lt.gmo' 2023-06-22T08:25:15.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/uk.gmo' -> './k/coreutils-8.5/po/uk.gmo' 2023-06-22T08:25:15.597 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/id.gmo' -> './k/coreutils-8.5/po/id.gmo' 2023-06-22T08:25:15.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/af.po' -> './k/coreutils-8.5/po/af.po' 2023-06-22T08:25:15.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/eu.po' -> './k/coreutils-8.5/po/eu.po' 2023-06-22T08:25:15.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ms.po' -> './k/coreutils-8.5/po/ms.po' 2023-06-22T08:25:15.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/en@quot.header' -> './k/coreutils-8.5/po/en@quot.header' 2023-06-22T08:25:15.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/af.gmo' -> './k/coreutils-8.5/po/af.gmo' 2023-06-22T08:25:15.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/es.gmo' -> './k/coreutils-8.5/po/es.gmo' 2023-06-22T08:25:15.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ga.po' -> './k/coreutils-8.5/po/ga.po' 2023-06-22T08:25:15.598 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ro.gmo' -> './k/coreutils-8.5/po/ro.gmo' 2023-06-22T08:25:15.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ca.po' -> './k/coreutils-8.5/po/ca.po' 2023-06-22T08:25:15.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/lg.gmo' -> './k/coreutils-8.5/po/lg.gmo' 2023-06-22T08:25:15.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/zh_CN.po' -> './k/coreutils-8.5/po/zh_CN.po' 2023-06-22T08:25:15.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/cs.gmo' -> './k/coreutils-8.5/po/cs.gmo' 2023-06-22T08:25:15.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/nb.po' -> './k/coreutils-8.5/po/nb.po' 2023-06-22T08:25:15.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/ChangeLog-2007' -> './k/coreutils-8.5/po/ChangeLog-2007' 2023-06-22T08:25:15.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/tr.po' -> './k/coreutils-8.5/po/tr.po' 2023-06-22T08:25:15.599 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/be.gmo' -> './k/coreutils-8.5/po/be.gmo' 2023-06-22T08:25:15.600 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/fr.po' -> './k/coreutils-8.5/po/fr.po' 2023-06-22T08:25:15.600 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/el.po' -> './k/coreutils-8.5/po/el.po' 2023-06-22T08:25:15.600 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/lt.po' -> './k/coreutils-8.5/po/lt.po' 2023-06-22T08:25:15.600 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/es.po' -> './k/coreutils-8.5/po/es.po' 2023-06-22T08:25:15.600 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/po/pt.po' -> './k/coreutils-8.5/po/pt.po' 2023-06-22T08:25:15.600 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests' -> './k/coreutils-8.5/tests' 2023-06-22T08:25:15.600 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/test-lib.sh' -> './k/coreutils-8.5/tests/test-lib.sh' 2023-06-22T08:25:15.601 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/sample-test' -> './k/coreutils-8.5/tests/sample-test' 2023-06-22T08:25:15.601 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/CuTmpdir.pm' -> './k/coreutils-8.5/tests/CuTmpdir.pm' 2023-06-22T08:25:15.601 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/Makefile.am' -> './k/coreutils-8.5/tests/Makefile.am' 2023-06-22T08:25:15.601 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/require-perl' -> './k/coreutils-8.5/tests/require-perl' 2023-06-22T08:25:15.601 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/lang-default' -> './k/coreutils-8.5/tests/lang-default' 2023-06-22T08:25:15.601 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/check.mk' -> './k/coreutils-8.5/tests/check.mk' 2023-06-22T08:25:15.601 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/Makefile.in' -> './k/coreutils-8.5/tests/Makefile.in' 2023-06-22T08:25:15.601 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd' -> './k/coreutils-8.5/tests/dd' 2023-06-22T08:25:15.602 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/unblock-sync' -> './k/coreutils-8.5/tests/dd/unblock-sync' 2023-06-22T08:25:15.602 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/skip-seek2' -> './k/coreutils-8.5/tests/dd/skip-seek2' 2023-06-22T08:25:15.602 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/unblock' -> './k/coreutils-8.5/tests/dd/unblock' 2023-06-22T08:25:15.602 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/stderr' -> './k/coreutils-8.5/tests/dd/stderr' 2023-06-22T08:25:15.602 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/direct' -> './k/coreutils-8.5/tests/dd/direct' 2023-06-22T08:25:15.602 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/skip-seek-past-file' -> './k/coreutils-8.5/tests/dd/skip-seek-past-file' 2023-06-22T08:25:15.602 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/not-rewound' -> './k/coreutils-8.5/tests/dd/not-rewound' 2023-06-22T08:25:15.602 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/skip-seek-past-dev' -> './k/coreutils-8.5/tests/dd/skip-seek-past-dev' 2023-06-22T08:25:15.603 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/misc' -> './k/coreutils-8.5/tests/dd/misc' 2023-06-22T08:25:15.676 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/reblock' -> './k/coreutils-8.5/tests/dd/reblock' 2023-06-22T08:25:15.676 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/dd/skip-seek' -> './k/coreutils-8.5/tests/dd/skip-seek' 2023-06-22T08:25:15.676 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr' -> './k/coreutils-8.5/tests/pr' 2023-06-22T08:25:15.677 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/t_tab_' -> './k/coreutils-8.5/tests/pr/t_tab_' 2023-06-22T08:25:15.677 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/FF' -> './k/coreutils-8.5/tests/pr/FF' 2023-06-22T08:25:15.677 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/FnFn' -> './k/coreutils-8.5/tests/pr/FnFn' 2023-06-22T08:25:15.677 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/o3ml24f-bl-tn' -> './k/coreutils-8.5/tests/pr/o3ml24f-bl-tn' 2023-06-22T08:25:15.677 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+5-8b3l17f-FF' -> './k/coreutils-8.5/tests/pr/n+5-8b3l17f-FF' 2023-06-22T08:25:15.677 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+3ml24f-bl-tn' -> './k/coreutils-8.5/tests/pr/n+3ml24f-bl-tn' 2023-06-22T08:25:15.677 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/FFtn' -> './k/coreutils-8.5/tests/pr/FFtn' 2023-06-22T08:25:15.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/a2l15-FF' -> './k/coreutils-8.5/tests/pr/a2l15-FF' 2023-06-22T08:25:15.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tFnFt' -> './k/coreutils-8.5/tests/pr/tFnFt' 2023-06-22T08:25:15.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tn2e5o3-t_tab' -> './k/coreutils-8.5/tests/pr/tn2e5o3-t_tab' 2023-06-22T08:25:15.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3ml24f-t' -> './k/coreutils-8.5/tests/pr/3ml24f-t' 2023-06-22T08:25:15.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nSml20-t-tFFFF' -> './k/coreutils-8.5/tests/pr/nSml20-t-tFFFF' 2023-06-22T08:25:15.678 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tn2e8-t_tab' -> './k/coreutils-8.5/tests/pr/tn2e8-t_tab' 2023-06-22T08:25:15.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/b3f-FF' -> './k/coreutils-8.5/tests/pr/b3f-FF' 2023-06-22T08:25:15.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/2w60f-t_notab' -> './k/coreutils-8.5/tests/pr/2w60f-t_notab' 2023-06-22T08:25:15.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/FFn' -> './k/coreutils-8.5/tests/pr/FFn' 2023-06-22T08:25:15.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/t_tab' -> './k/coreutils-8.5/tests/pr/t_tab' 2023-06-22T08:25:15.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/b3-0F' -> './k/coreutils-8.5/tests/pr/b3-0F' 2023-06-22T08:25:15.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+4-8a2l17-FF' -> './k/coreutils-8.5/tests/pr/n+4-8a2l17-FF' 2023-06-22T08:25:15.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+4b2l17f-0FF' -> './k/coreutils-8.5/tests/pr/n+4b2l17f-0FF' 2023-06-22T08:25:15.679 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3a3l15-t' -> './k/coreutils-8.5/tests/pr/3a3l15-t' 2023-06-22T08:25:15.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tFFn' -> './k/coreutils-8.5/tests/pr/tFFn' 2023-06-22T08:25:15.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3l24f-t' -> './k/coreutils-8.5/tests/pr/3l24f-t' 2023-06-22T08:25:15.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/0FnFt' -> './k/coreutils-8.5/tests/pr/0FnFt' 2023-06-22T08:25:15.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3-FF' -> './k/coreutils-8.5/tests/pr/3-FF' 2023-06-22T08:25:15.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/t-FF' -> './k/coreutils-8.5/tests/pr/t-FF' 2023-06-22T08:25:15.680 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nSml24-bl-FF' -> './k/coreutils-8.5/tests/pr/nSml24-bl-FF' 2023-06-22T08:25:15.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+3l24f-bl' -> './k/coreutils-8.5/tests/pr/n+3l24f-bl' 2023-06-22T08:25:15.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/b3l15-t' -> './k/coreutils-8.5/tests/pr/b3l15-t' 2023-06-22T08:25:15.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3b3f-FF' -> './k/coreutils-8.5/tests/pr/3b3f-FF' 2023-06-22T08:25:15.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3b3f-0F' -> './k/coreutils-8.5/tests/pr/3b3f-0F' 2023-06-22T08:25:15.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W35b3l24f-lm' -> './k/coreutils-8.5/tests/pr/W35b3l24f-lm' 2023-06-22T08:25:15.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/0Ft' -> './k/coreutils-8.5/tests/pr/0Ft' 2023-06-22T08:25:15.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tt-bl' -> './k/coreutils-8.5/tests/pr/tt-bl' 2023-06-22T08:25:15.681 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3ml24-t-FF' -> './k/coreutils-8.5/tests/pr/3ml24-t-FF' 2023-06-22T08:25:15.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tFFt-lm' -> './k/coreutils-8.5/tests/pr/tFFt-lm' 2023-06-22T08:25:15.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nSml20-bl-FF' -> './k/coreutils-8.5/tests/pr/nSml20-bl-FF' 2023-06-22T08:25:15.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tn2e8o3-t_tab' -> './k/coreutils-8.5/tests/pr/tn2e8o3-t_tab' 2023-06-22T08:25:15.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W28l24f-ll' -> './k/coreutils-8.5/tests/pr/W28l24f-ll' 2023-06-22T08:25:15.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nSml24-t-tFFFF' -> './k/coreutils-8.5/tests/pr/nSml24-t-tFFFF' 2023-06-22T08:25:15.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/t-0FF' -> './k/coreutils-8.5/tests/pr/t-0FF' 2023-06-22T08:25:15.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tb3-FF' -> './k/coreutils-8.5/tests/pr/tb3-FF' 2023-06-22T08:25:15.682 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/o3b3Sl24f-tn' -> './k/coreutils-8.5/tests/pr/o3b3Sl24f-tn' 2023-06-22T08:25:15.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/o3mSl24f-bl-tn' -> './k/coreutils-8.5/tests/pr/o3mSl24f-bl-tn' 2023-06-22T08:25:15.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3b2l17-FF' -> './k/coreutils-8.5/tests/pr/3b2l17-FF' 2023-06-22T08:25:15.683 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/2f-t_notab' -> './k/coreutils-8.5/tests/pr/2f-t_notab' 2023-06-22T08:25:15.734 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/b2l15-FF' -> './k/coreutils-8.5/tests/pr/b2l15-FF' 2023-06-22T08:25:15.735 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3b3f-0FF' -> './k/coreutils-8.5/tests/pr/3b3f-0FF' 2023-06-22T08:25:15.735 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/a3l15-t' -> './k/coreutils-8.5/tests/pr/a3l15-t' 2023-06-22T08:25:15.735 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/o3Jml24f-lm-lo' -> './k/coreutils-8.5/tests/pr/o3Jml24f-lm-lo' 2023-06-22T08:25:15.735 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tb3-0FF' -> './k/coreutils-8.5/tests/pr/tb3-0FF' 2023-06-22T08:25:15.735 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/l24-t' -> './k/coreutils-8.5/tests/pr/l24-t' 2023-06-22T08:25:15.735 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W35ml24f-lm-lo' -> './k/coreutils-8.5/tests/pr/W35ml24f-lm-lo' 2023-06-22T08:25:15.735 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/a3-0F' -> './k/coreutils-8.5/tests/pr/a3-0F' 2023-06-22T08:25:15.736 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ml24f-0F' -> './k/coreutils-8.5/tests/pr/ml24f-0F' 2023-06-22T08:25:15.736 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/loli' -> './k/coreutils-8.5/tests/pr/loli' 2023-06-22T08:25:15.736 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3ml24-t' -> './k/coreutils-8.5/tests/pr/3ml24-t' 2023-06-22T08:25:15.736 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/a3l24f-lm' -> './k/coreutils-8.5/tests/pr/a3l24f-lm' 2023-06-22T08:25:15.736 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/Jb3l24f-lm' -> './k/coreutils-8.5/tests/pr/Jb3l24f-lm' 2023-06-22T08:25:15.736 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/t-t' -> './k/coreutils-8.5/tests/pr/t-t' 2023-06-22T08:25:15.736 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/0F' -> './k/coreutils-8.5/tests/pr/0F' 2023-06-22T08:25:15.737 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/w72l24f-ll' -> './k/coreutils-8.5/tests/pr/w72l24f-ll' 2023-06-22T08:25:15.737 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ml24f-t' -> './k/coreutils-8.5/tests/pr/ml24f-t' 2023-06-22T08:25:15.737 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/b3l15f-t' -> './k/coreutils-8.5/tests/pr/b3l15f-t' 2023-06-22T08:25:15.737 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/0FnFnt' -> './k/coreutils-8.5/tests/pr/0FnFnt' 2023-06-22T08:25:15.737 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/b3l24f-lm' -> './k/coreutils-8.5/tests/pr/b3l24f-lm' 2023-06-22T08:25:15.737 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/o3mSnl24fbltn' -> './k/coreutils-8.5/tests/pr/o3mSnl24fbltn' 2023-06-22T08:25:15.737 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nJml24f-lmlolm' -> './k/coreutils-8.5/tests/pr/nJml24f-lmlolm' 2023-06-22T08:25:15.737 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/o3a3Sl24f-tn' -> './k/coreutils-8.5/tests/pr/o3a3Sl24f-tn' 2023-06-22T08:25:15.738 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tne8-t_tab' -> './k/coreutils-8.5/tests/pr/tne8-t_tab' 2023-06-22T08:25:15.738 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3ml24-FF' -> './k/coreutils-8.5/tests/pr/3ml24-FF' 2023-06-22T08:25:15.738 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/a3f-0FF' -> './k/coreutils-8.5/tests/pr/a3f-0FF' 2023-06-22T08:25:15.738 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ml20-FF-t' -> './k/coreutils-8.5/tests/pr/ml20-FF-t' 2023-06-22T08:25:15.738 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/pr-tests' -> './k/coreutils-8.5/tests/pr/pr-tests' 2023-06-22T08:25:15.738 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W35Jb3l24f-lm' -> './k/coreutils-8.5/tests/pr/W35Jb3l24f-lm' 2023-06-22T08:25:15.738 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ttb3-0FF' -> './k/coreutils-8.5/tests/pr/ttb3-0FF' 2023-06-22T08:25:15.739 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nl24f-bl' -> './k/coreutils-8.5/tests/pr/nl24f-bl' 2023-06-22T08:25:15.739 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nSml24-t-t-FF' -> './k/coreutils-8.5/tests/pr/nSml24-t-t-FF' 2023-06-22T08:25:15.739 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3b3l15f-t' -> './k/coreutils-8.5/tests/pr/3b3l15f-t' 2023-06-22T08:25:15.739 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/0FF' -> './k/coreutils-8.5/tests/pr/0FF' 2023-06-22T08:25:15.739 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3f-0F' -> './k/coreutils-8.5/tests/pr/3f-0F' 2023-06-22T08:25:15.739 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/Ja3l24f-lm' -> './k/coreutils-8.5/tests/pr/Ja3l24f-lm' 2023-06-22T08:25:15.739 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3a2l17-FF' -> './k/coreutils-8.5/tests/pr/3a2l17-FF' 2023-06-22T08:25:15.739 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ml24f-t-0F' -> './k/coreutils-8.5/tests/pr/ml24f-t-0F' 2023-06-22T08:25:15.740 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tn_2e8-t_tab' -> './k/coreutils-8.5/tests/pr/tn_2e8-t_tab' 2023-06-22T08:25:15.740 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/o3a3Snl24f-tn' -> './k/coreutils-8.5/tests/pr/o3a3Snl24f-tn' 2023-06-22T08:25:15.740 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/2s_w60f-t_nota' -> './k/coreutils-8.5/tests/pr/2s_w60f-t_nota' 2023-06-22T08:25:15.740 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/t-bl' -> './k/coreutils-8.5/tests/pr/t-bl' 2023-06-22T08:25:15.740 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ml24-t-FF' -> './k/coreutils-8.5/tests/pr/ml24-t-FF' 2023-06-22T08:25:15.740 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+3-7l24-FF' -> './k/coreutils-8.5/tests/pr/n+3-7l24-FF' 2023-06-22T08:25:15.740 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/t_notab' -> './k/coreutils-8.5/tests/pr/t_notab' 2023-06-22T08:25:15.741 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W72Jl24f-ll' -> './k/coreutils-8.5/tests/pr/W72Jl24f-ll' 2023-06-22T08:25:15.741 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/a2l17-FF' -> './k/coreutils-8.5/tests/pr/a2l17-FF' 2023-06-22T08:25:15.741 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ml24-FF' -> './k/coreutils-8.5/tests/pr/ml24-FF' 2023-06-22T08:25:15.741 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/b3f-0F' -> './k/coreutils-8.5/tests/pr/b3f-0F' 2023-06-22T08:25:15.746 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.744+0000 7f01d1fff700 1 -- 172.21.15.23:0/555173810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01cc0c8c40 con 0x7f01cc152ce0 2023-06-22T08:25:15.753 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.751+0000 7f01b3fff700 1 -- 172.21.15.23:0/555173810 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+5488 (secure 0 0 0) 0x7f01c402c020 con 0x7f01cc152ce0 2023-06-22T08:25:15.754 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:25:15.754 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6284,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/4201059730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":4201059730},{"type":"v1","addr":"172.21.15.23:6837","nonce":4201059730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-22T07:59:55.139239+0000","modified":"2023-06-22T08:25:08.751242+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":4,"in":[0,1,2,3],"up":{"mds_0":6616,"mds_1":6730,"mds_2":6185,"mds_3":6705},"failed":[],"damaged":[],"stopped":[4],"info":{"gid_6185":{"gid":6185,"name":"f","rank":2,"incarnation":192,"state":"up:active","state_seq":19,"addr":"172.21.15.67:6836/2287946240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2287946240},{"type":"v1","addr":"172.21.15.67:6836","nonce":2287946240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6616":{"gid":6616,"name":"d","rank":0,"incarnation":162,"state":"up:active","state_seq":26,"addr":"172.21.15.67:6837/1855579717","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6835","nonce":1855579717},{"type":"v1","addr":"172.21.15.67:6837","nonce":1855579717}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6705":{"gid":6705,"name":"a","rank":3,"incarnation":184,"state":"up:active","state_seq":24,"addr":"172.21.15.23:6835/3148174349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6833","nonce":3148174349},{"type":"v1","addr":"172.21.15.23:6835","nonce":3148174349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_6730":{"gid":6730,"name":"c","rank":1,"incarnation":172,"state":"up:active","state_seq":13,"addr":"172.21.15.23:6834/456927633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":456927633},{"type":"v1","addr":"172.21.15.23:6834","nonce":456927633}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-06-22T08:25:15.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.753+0000 7f01d1fff700 1 -- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01b806ad10 msgr2=0x7f01b806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:15.756 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.753+0000 7f01d1fff700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01b806ad10 0x7f01b806d1c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f01bc007fd0 tx=0x7f01bc007dc0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.753+0000 7f01d1fff700 1 -- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc152ce0 msgr2=0x7f01cc1e38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:15.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.753+0000 7f01d1fff700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc152ce0 0x7f01cc1e38f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f01c4014040 tx=0x7f01c4004ad0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.757 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.754+0000 7f01d1fff700 1 -- 172.21.15.23:0/555173810 shutdown_connections 2023-06-22T08:25:15.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.754+0000 7f01d1fff700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f01b806ad10 0x7f01b806d1c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.754+0000 7f01d1fff700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01cc152ce0 0x7f01cc1e38f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.754+0000 7f01d1fff700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f01cc0c40b0 0x7f01cc0c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.758 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.754+0000 7f01d1fff700 1 --2- 172.21.15.23:0/555173810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01cc0c9fd0 0x7f01cc0ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:15.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.754+0000 7f01d1fff700 1 -- 172.21.15.23:0/555173810 >> 172.21.15.23:0/555173810 conn(0x7f01cc06ae30 msgr2=0x7f01cc14c430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:15.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.754+0000 7f01d1fff700 1 -- 172.21.15.23:0/555173810 shutdown_connections 2023-06-22T08:25:15.759 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:15.754+0000 7f01d1fff700 1 -- 172.21.15.23:0/555173810 wait complete. 2023-06-22T08:25:15.760 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 197 2023-06-22T08:25:15.775 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-06-22T08:25:15.776 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 111 secs before thrashing 2023-06-22T08:25:15.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+3l24f-0FF' -> './k/coreutils-8.5/tests/pr/n+3l24f-0FF' 2023-06-22T08:25:15.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+3ml20f-bl-FF' -> './k/coreutils-8.5/tests/pr/n+3ml20f-bl-FF' 2023-06-22T08:25:15.794 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/a3f-0F' -> './k/coreutils-8.5/tests/pr/a3f-0F' 2023-06-22T08:25:15.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W35Jml24f-lmlo' -> './k/coreutils-8.5/tests/pr/W35Jml24f-lmlo' 2023-06-22T08:25:15.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W27l24f-ll' -> './k/coreutils-8.5/tests/pr/W27l24f-ll' 2023-06-22T08:25:15.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ml24-t' -> './k/coreutils-8.5/tests/pr/ml24-t' 2023-06-22T08:25:15.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/a3f-FF' -> './k/coreutils-8.5/tests/pr/a3f-FF' 2023-06-22T08:25:15.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tFFt-bl' -> './k/coreutils-8.5/tests/pr/tFFt-bl' 2023-06-22T08:25:15.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tta3-FF' -> './k/coreutils-8.5/tests/pr/tta3-FF' 2023-06-22T08:25:15.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3-0F' -> './k/coreutils-8.5/tests/pr/3-0F' 2023-06-22T08:25:15.795 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/2sf-t_notab' -> './k/coreutils-8.5/tests/pr/2sf-t_notab' 2023-06-22T08:25:15.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3a3l15f-t' -> './k/coreutils-8.5/tests/pr/3a3l15f-t' 2023-06-22T08:25:15.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W26l24f-ll' -> './k/coreutils-8.5/tests/pr/W26l24f-ll' 2023-06-22T08:25:15.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+7l24-FF' -> './k/coreutils-8.5/tests/pr/n+7l24-FF' 2023-06-22T08:25:15.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ml24f-lm-lo' -> './k/coreutils-8.5/tests/pr/ml24f-lm-lo' 2023-06-22T08:25:15.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/l24-FF' -> './k/coreutils-8.5/tests/pr/l24-FF' 2023-06-22T08:25:15.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+2-5l24f-0FF' -> './k/coreutils-8.5/tests/pr/n+2-5l24f-0FF' 2023-06-22T08:25:15.796 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3f-FF' -> './k/coreutils-8.5/tests/pr/3f-FF' 2023-06-22T08:25:15.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+3ml24f-tn-bl' -> './k/coreutils-8.5/tests/pr/n+3ml24f-tn-bl' 2023-06-22T08:25:15.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ta3-0FF' -> './k/coreutils-8.5/tests/pr/ta3-0FF' 2023-06-22T08:25:15.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3b3l15-t' -> './k/coreutils-8.5/tests/pr/3b3l15-t' 2023-06-22T08:25:15.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+2l24f-0FF' -> './k/coreutils-8.5/tests/pr/n+2l24f-0FF' 2023-06-22T08:25:15.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tn_2e8S-t_tab' -> './k/coreutils-8.5/tests/pr/tn_2e8S-t_tab' 2023-06-22T08:25:15.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ta3-FF' -> './k/coreutils-8.5/tests/pr/ta3-FF' 2023-06-22T08:25:15.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tFFt-ll' -> './k/coreutils-8.5/tests/pr/tFFt-ll' 2023-06-22T08:25:15.797 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+6b3l13f-FF' -> './k/coreutils-8.5/tests/pr/n+6b3l13f-FF' 2023-06-22T08:25:15.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/2-Sf-t_notab' -> './k/coreutils-8.5/tests/pr/2-Sf-t_notab' 2023-06-22T08:25:15.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/0FFnt' -> './k/coreutils-8.5/tests/pr/0FFnt' 2023-06-22T08:25:15.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W20l24f-ll' -> './k/coreutils-8.5/tests/pr/W20l24f-ll' 2023-06-22T08:25:15.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/Jml24f-lm-lo' -> './k/coreutils-8.5/tests/pr/Jml24f-lm-lo' 2023-06-22T08:25:15.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3l24-t' -> './k/coreutils-8.5/tests/pr/3l24-t' 2023-06-22T08:25:15.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/o3b3l24f-tn' -> './k/coreutils-8.5/tests/pr/o3b3l24f-tn' 2023-06-22T08:25:15.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W35a3l24f-lm' -> './k/coreutils-8.5/tests/pr/W35a3l24f-lm' 2023-06-22T08:25:15.798 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/2-S_f-t_notab' -> './k/coreutils-8.5/tests/pr/2-S_f-t_notab' 2023-06-22T08:25:15.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3-5l24f-t' -> './k/coreutils-8.5/tests/pr/3-5l24f-t' 2023-06-22T08:25:15.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/o3b3Snl24f-tn' -> './k/coreutils-8.5/tests/pr/o3b3Snl24f-tn' 2023-06-22T08:25:15.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+2l24f-bl' -> './k/coreutils-8.5/tests/pr/n+2l24f-bl' 2023-06-22T08:25:15.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W35Ja3l24f-lm' -> './k/coreutils-8.5/tests/pr/W35Ja3l24f-lm' 2023-06-22T08:25:15.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+8l20-FF' -> './k/coreutils-8.5/tests/pr/n+8l20-FF' 2023-06-22T08:25:15.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/0FFt' -> './k/coreutils-8.5/tests/pr/0FFt' 2023-06-22T08:25:15.799 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/o3a3l24f-tn' -> './k/coreutils-8.5/tests/pr/o3a3l24f-tn' 2023-06-22T08:25:15.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/2s_f-t_notab' -> './k/coreutils-8.5/tests/pr/2s_f-t_notab' 2023-06-22T08:25:15.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nN15l24f-bl' -> './k/coreutils-8.5/tests/pr/nN15l24f-bl' 2023-06-22T08:25:15.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/0Fnt' -> './k/coreutils-8.5/tests/pr/0Fnt' 2023-06-22T08:25:15.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/4-7l24-FF' -> './k/coreutils-8.5/tests/pr/4-7l24-FF' 2023-06-22T08:25:15.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tn' -> './k/coreutils-8.5/tests/pr/tn' 2023-06-22T08:25:15.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/b2l17-FF' -> './k/coreutils-8.5/tests/pr/b2l17-FF' 2023-06-22T08:25:15.800 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nJml24f-lmlmlo' -> './k/coreutils-8.5/tests/pr/nJml24f-lmlmlo' 2023-06-22T08:25:16.902 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:25:16.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:25:18.268 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tta3-0FF' -> './k/coreutils-8.5/tests/pr/tta3-0FF' 2023-06-22T08:25:18.269 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+5a3l13f-0FF' -> './k/coreutils-8.5/tests/pr/n+5a3l13f-0FF' 2023-06-22T08:25:18.269 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nN1+3l24f-bl' -> './k/coreutils-8.5/tests/pr/nN1+3l24f-bl' 2023-06-22T08:25:18.269 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/2sw60f-t_notab' -> './k/coreutils-8.5/tests/pr/2sw60f-t_notab' 2023-06-22T08:25:18.269 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/W-72l24f-ll' -> './k/coreutils-8.5/tests/pr/W-72l24f-ll' 2023-06-22T08:25:18.269 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tt-0FF' -> './k/coreutils-8.5/tests/pr/tt-0FF' 2023-06-22T08:25:18.269 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/4l24-FF' -> './k/coreutils-8.5/tests/pr/4l24-FF' 2023-06-22T08:25:18.269 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/3a3f-0F' -> './k/coreutils-8.5/tests/pr/3a3f-0F' 2023-06-22T08:25:18.270 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tFFt' -> './k/coreutils-8.5/tests/pr/tFFt' 2023-06-22T08:25:18.270 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/n+6a2l17-FF' -> './k/coreutils-8.5/tests/pr/n+6a2l17-FF' 2023-06-22T08:25:18.270 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/a3l15f-t' -> './k/coreutils-8.5/tests/pr/a3l15f-t' 2023-06-22T08:25:18.270 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/l24f-t' -> './k/coreutils-8.5/tests/pr/l24f-t' 2023-06-22T08:25:18.270 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/b3f-0FF' -> './k/coreutils-8.5/tests/pr/b3f-0FF' 2023-06-22T08:25:18.270 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tt-t' -> './k/coreutils-8.5/tests/pr/tt-t' 2023-06-22T08:25:18.270 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/ttb3-FF' -> './k/coreutils-8.5/tests/pr/ttb3-FF' 2023-06-22T08:25:18.271 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/nSml20-t-t-FF' -> './k/coreutils-8.5/tests/pr/nSml20-t-t-FF' 2023-06-22T08:25:18.271 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tne8o3-t_tab' -> './k/coreutils-8.5/tests/pr/tne8o3-t_tab' 2023-06-22T08:25:18.271 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/pr/tt-FF' -> './k/coreutils-8.5/tests/pr/tt-FF' 2023-06-22T08:25:18.271 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/other-fs-tmpdir' -> './k/coreutils-8.5/tests/other-fs-tmpdir' 2023-06-22T08:25:18.271 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rmdir' -> './k/coreutils-8.5/tests/rmdir' 2023-06-22T08:25:18.271 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rmdir/ignore' -> './k/coreutils-8.5/tests/rmdir/ignore' 2023-06-22T08:25:18.271 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rmdir/fail-perm' -> './k/coreutils-8.5/tests/rmdir/fail-perm' 2023-06-22T08:25:18.271 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rmdir/t-slash' -> './k/coreutils-8.5/tests/rmdir/t-slash' 2023-06-22T08:25:18.272 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm' -> './k/coreutils-8.5/tests/rm' 2023-06-22T08:25:18.272 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/r-2' -> './k/coreutils-8.5/tests/rm/r-2' 2023-06-22T08:25:18.272 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/fail-2eperm' -> './k/coreutils-8.5/tests/rm/fail-2eperm' 2023-06-22T08:25:18.272 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/empty-inacc' -> './k/coreutils-8.5/tests/rm/empty-inacc' 2023-06-22T08:25:18.272 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/cycle' -> './k/coreutils-8.5/tests/rm/cycle' 2023-06-22T08:25:18.272 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/sunos-1' -> './k/coreutils-8.5/tests/rm/sunos-1' 2023-06-22T08:25:18.272 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/v-slash' -> './k/coreutils-8.5/tests/rm/v-slash' 2023-06-22T08:25:18.273 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/no-give-up' -> './k/coreutils-8.5/tests/rm/no-give-up' 2023-06-22T08:25:18.273 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/dot-rel' -> './k/coreutils-8.5/tests/rm/dot-rel' 2023-06-22T08:25:18.273 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/one-file-system' -> './k/coreutils-8.5/tests/rm/one-file-system' 2023-06-22T08:25:18.273 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/i-never' -> './k/coreutils-8.5/tests/rm/i-never' 2023-06-22T08:25:18.273 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/rm1' -> './k/coreutils-8.5/tests/rm/rm1' 2023-06-22T08:25:18.273 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/unread2' -> './k/coreutils-8.5/tests/rm/unread2' 2023-06-22T08:25:18.273 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/read-only' -> './k/coreutils-8.5/tests/rm/read-only' 2023-06-22T08:25:18.273 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/deep-2' -> './k/coreutils-8.5/tests/rm/deep-2' 2023-06-22T08:25:18.274 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/r-4' -> './k/coreutils-8.5/tests/rm/r-4' 2023-06-22T08:25:18.274 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/dir-no-w' -> './k/coreutils-8.5/tests/rm/dir-no-w' 2023-06-22T08:25:18.274 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/interactive-always' -> './k/coreutils-8.5/tests/rm/interactive-always' 2023-06-22T08:25:18.274 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/one-file-system2' -> './k/coreutils-8.5/tests/rm/one-file-system2' 2023-06-22T08:25:18.274 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/unreadable' -> './k/coreutils-8.5/tests/rm/unreadable' 2023-06-22T08:25:18.274 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/r-1' -> './k/coreutils-8.5/tests/rm/r-1' 2023-06-22T08:25:18.274 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/fail-eacces' -> './k/coreutils-8.5/tests/rm/fail-eacces' 2023-06-22T08:25:18.274 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/inaccessible' -> './k/coreutils-8.5/tests/rm/inaccessible' 2023-06-22T08:25:18.275 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/unread3' -> './k/coreutils-8.5/tests/rm/unread3' 2023-06-22T08:25:18.275 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/ir-1' -> './k/coreutils-8.5/tests/rm/ir-1' 2023-06-22T08:25:18.275 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/rm3' -> './k/coreutils-8.5/tests/rm/rm3' 2023-06-22T08:25:18.370 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/rm5' -> './k/coreutils-8.5/tests/rm/rm5' 2023-06-22T08:25:18.370 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/deep-1' -> './k/coreutils-8.5/tests/rm/deep-1' 2023-06-22T08:25:18.370 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/ext3-perf' -> './k/coreutils-8.5/tests/rm/ext3-perf' 2023-06-22T08:25:18.371 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/dangling-symlink' -> './k/coreutils-8.5/tests/rm/dangling-symlink' 2023-06-22T08:25:18.371 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/isatty' -> './k/coreutils-8.5/tests/rm/isatty' 2023-06-22T08:25:18.371 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/i-1' -> './k/coreutils-8.5/tests/rm/i-1' 2023-06-22T08:25:18.371 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/dir-nonrecur' -> './k/coreutils-8.5/tests/rm/dir-nonrecur' 2023-06-22T08:25:18.371 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/interactive-once' -> './k/coreutils-8.5/tests/rm/interactive-once' 2023-06-22T08:25:18.371 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/f-1' -> './k/coreutils-8.5/tests/rm/f-1' 2023-06-22T08:25:18.371 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/fail-eperm' -> './k/coreutils-8.5/tests/rm/fail-eperm' 2023-06-22T08:25:18.372 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/empty-name' -> './k/coreutils-8.5/tests/rm/empty-name' 2023-06-22T08:25:18.372 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/hash' -> './k/coreutils-8.5/tests/rm/hash' 2023-06-22T08:25:18.372 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/i-no-r' -> './k/coreutils-8.5/tests/rm/i-no-r' 2023-06-22T08:25:18.372 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/r-3' -> './k/coreutils-8.5/tests/rm/r-3' 2023-06-22T08:25:18.372 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/readdir-bug' -> './k/coreutils-8.5/tests/rm/readdir-bug' 2023-06-22T08:25:18.372 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/rm2' -> './k/coreutils-8.5/tests/rm/rm2' 2023-06-22T08:25:18.372 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/rm4' -> './k/coreutils-8.5/tests/rm/rm4' 2023-06-22T08:25:18.373 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/rm/ignorable' -> './k/coreutils-8.5/tests/rm/ignorable' 2023-06-22T08:25:18.373 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/envvar-check' -> './k/coreutils-8.5/tests/envvar-check' 2023-06-22T08:25:18.373 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chown' -> './k/coreutils-8.5/tests/chown' 2023-06-22T08:25:18.373 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chown/separator' -> './k/coreutils-8.5/tests/chown/separator' 2023-06-22T08:25:18.373 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chown/deref' -> './k/coreutils-8.5/tests/chown/deref' 2023-06-22T08:25:18.373 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chown/preserve-root' -> './k/coreutils-8.5/tests/chown/preserve-root' 2023-06-22T08:25:18.373 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chown/basic' -> './k/coreutils-8.5/tests/chown/basic' 2023-06-22T08:25:18.373 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/id' -> './k/coreutils-8.5/tests/id' 2023-06-22T08:25:18.374 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/id/no-context' -> './k/coreutils-8.5/tests/id/no-context' 2023-06-22T08:25:18.374 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp' -> './k/coreutils-8.5/tests/cp' 2023-06-22T08:25:18.374 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/preserve-2' -> './k/coreutils-8.5/tests/cp/preserve-2' 2023-06-22T08:25:18.374 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/dir-vs-file' -> './k/coreutils-8.5/tests/cp/dir-vs-file' 2023-06-22T08:25:18.374 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/preserve-slink-time' -> './k/coreutils-8.5/tests/cp/preserve-slink-time' 2023-06-22T08:25:18.374 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/same-file' -> './k/coreutils-8.5/tests/cp/same-file' 2023-06-22T08:25:18.374 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/src-base-dot' -> './k/coreutils-8.5/tests/cp/src-base-dot' 2023-06-22T08:25:18.374 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/cp-a-selinux' -> './k/coreutils-8.5/tests/cp/cp-a-selinux' 2023-06-22T08:25:18.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/r-vs-symlink' -> './k/coreutils-8.5/tests/cp/r-vs-symlink' 2023-06-22T08:25:18.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/capability' -> './k/coreutils-8.5/tests/cp/capability' 2023-06-22T08:25:18.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/existing-perm-race' -> './k/coreutils-8.5/tests/cp/existing-perm-race' 2023-06-22T08:25:18.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/parent-perm-race' -> './k/coreutils-8.5/tests/cp/parent-perm-race' 2023-06-22T08:25:18.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/reflink-auto' -> './k/coreutils-8.5/tests/cp/reflink-auto' 2023-06-22T08:25:18.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/cp-mv-enotsup-xattr' -> './k/coreutils-8.5/tests/cp/cp-mv-enotsup-xattr' 2023-06-22T08:25:18.375 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/no-deref-link2' -> './k/coreutils-8.5/tests/cp/no-deref-link2' 2023-06-22T08:25:18.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/cp-mv-backup' -> './k/coreutils-8.5/tests/cp/cp-mv-backup' 2023-06-22T08:25:18.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/link-no-deref' -> './k/coreutils-8.5/tests/cp/link-no-deref' 2023-06-22T08:25:18.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/file-perm-race' -> './k/coreutils-8.5/tests/cp/file-perm-race' 2023-06-22T08:25:18.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/abuse' -> './k/coreutils-8.5/tests/cp/abuse' 2023-06-22T08:25:18.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/link-heap' -> './k/coreutils-8.5/tests/cp/link-heap' 2023-06-22T08:25:18.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/backup-dir' -> './k/coreutils-8.5/tests/cp/backup-dir' 2023-06-22T08:25:18.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/preserve-gid' -> './k/coreutils-8.5/tests/cp/preserve-gid' 2023-06-22T08:25:18.376 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/link' -> './k/coreutils-8.5/tests/cp/link' 2023-06-22T08:25:18.479 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/cp-parents' -> './k/coreutils-8.5/tests/cp/cp-parents' 2023-06-22T08:25:18.479 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/no-deref-link1' -> './k/coreutils-8.5/tests/cp/no-deref-link1' 2023-06-22T08:25:18.480 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/into-self' -> './k/coreutils-8.5/tests/cp/into-self' 2023-06-22T08:25:18.480 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/parent-perm' -> './k/coreutils-8.5/tests/cp/parent-perm' 2023-06-22T08:25:18.480 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/cp-deref' -> './k/coreutils-8.5/tests/cp/cp-deref' 2023-06-22T08:25:18.480 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/proc-short-read' -> './k/coreutils-8.5/tests/cp/proc-short-read' 2023-06-22T08:25:18.480 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/backup-is-src' -> './k/coreutils-8.5/tests/cp/backup-is-src' 2023-06-22T08:25:18.480 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/cp-i' -> './k/coreutils-8.5/tests/cp/cp-i' 2023-06-22T08:25:18.480 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/special-f' -> './k/coreutils-8.5/tests/cp/special-f' 2023-06-22T08:25:18.481 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/proc-zero-len' -> './k/coreutils-8.5/tests/cp/proc-zero-len' 2023-06-22T08:25:18.481 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/slink-2-slink' -> './k/coreutils-8.5/tests/cp/slink-2-slink' 2023-06-22T08:25:18.481 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/special-bits' -> './k/coreutils-8.5/tests/cp/special-bits' 2023-06-22T08:25:18.481 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/fail-perm' -> './k/coreutils-8.5/tests/cp/fail-perm' 2023-06-22T08:25:18.481 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/link-preserve' -> './k/coreutils-8.5/tests/cp/link-preserve' 2023-06-22T08:25:18.481 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/dir-rm-dest' -> './k/coreutils-8.5/tests/cp/dir-rm-dest' 2023-06-22T08:25:18.481 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/backup-1' -> './k/coreutils-8.5/tests/cp/backup-1' 2023-06-22T08:25:18.481 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/cp-HL' -> './k/coreutils-8.5/tests/cp/cp-HL' 2023-06-22T08:25:18.482 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/thru-dangling' -> './k/coreutils-8.5/tests/cp/thru-dangling' 2023-06-22T08:25:18.482 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/sparse' -> './k/coreutils-8.5/tests/cp/sparse' 2023-06-22T08:25:18.482 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/deref-slink' -> './k/coreutils-8.5/tests/cp/deref-slink' 2023-06-22T08:25:18.482 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/symlink-slash' -> './k/coreutils-8.5/tests/cp/symlink-slash' 2023-06-22T08:25:18.482 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/acl' -> './k/coreutils-8.5/tests/cp/acl' 2023-06-22T08:25:18.482 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/perm' -> './k/coreutils-8.5/tests/cp/perm' 2023-06-22T08:25:18.482 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/reflink-perm' -> './k/coreutils-8.5/tests/cp/reflink-perm' 2023-06-22T08:25:18.483 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/no-deref-link3' -> './k/coreutils-8.5/tests/cp/no-deref-link3' 2023-06-22T08:25:18.483 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/cp/dir-slash' -> './k/coreutils-8.5/tests/cp/dir-slash' 2023-06-22T08:25:18.483 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir' -> './k/coreutils-8.5/tests/mkdir' 2023-06-22T08:25:18.483 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/p-v' -> './k/coreutils-8.5/tests/mkdir/p-v' 2023-06-22T08:25:18.483 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/p-3' -> './k/coreutils-8.5/tests/mkdir/p-3' 2023-06-22T08:25:18.483 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/writable-under-readonly' -> './k/coreutils-8.5/tests/mkdir/writable-under-readonly' 2023-06-22T08:25:18.483 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/p-slashdot' -> './k/coreutils-8.5/tests/mkdir/p-slashdot' 2023-06-22T08:25:18.483 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/selinux' -> './k/coreutils-8.5/tests/mkdir/selinux' 2023-06-22T08:25:18.484 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/p-1' -> './k/coreutils-8.5/tests/mkdir/p-1' 2023-06-22T08:25:18.484 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/p-2' -> './k/coreutils-8.5/tests/mkdir/p-2' 2023-06-22T08:25:18.484 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/parents' -> './k/coreutils-8.5/tests/mkdir/parents' 2023-06-22T08:25:18.484 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/special-1' -> './k/coreutils-8.5/tests/mkdir/special-1' 2023-06-22T08:25:18.484 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/perm' -> './k/coreutils-8.5/tests/mkdir/perm' 2023-06-22T08:25:18.484 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/p-thru-slink' -> './k/coreutils-8.5/tests/mkdir/p-thru-slink' 2023-06-22T08:25:18.484 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mkdir/t-slash' -> './k/coreutils-8.5/tests/mkdir/t-slash' 2023-06-22T08:25:18.485 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ln' -> './k/coreutils-8.5/tests/ln' 2023-06-22T08:25:18.485 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ln/hard-to-sym' -> './k/coreutils-8.5/tests/ln/hard-to-sym' 2023-06-22T08:25:18.485 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ln/hard-backup' -> './k/coreutils-8.5/tests/ln/hard-backup' 2023-06-22T08:25:18.485 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ln/sf-1' -> './k/coreutils-8.5/tests/ln/sf-1' 2023-06-22T08:25:18.485 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ln/backup-1' -> './k/coreutils-8.5/tests/ln/backup-1' 2023-06-22T08:25:18.485 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ln/slash-decorated-nonexistent-dest' -> './k/coreutils-8.5/tests/ln/slash-decorated-nonexistent-dest' 2023-06-22T08:25:18.485 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ln/misc' -> './k/coreutils-8.5/tests/ln/misc' 2023-06-22T08:25:18.485 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ln/target-1' -> './k/coreutils-8.5/tests/ln/target-1' 2023-06-22T08:25:18.586 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/install' -> './k/coreutils-8.5/tests/install' 2023-06-22T08:25:18.586 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/install/install-C-selinux' -> './k/coreutils-8.5/tests/install/install-C-selinux' 2023-06-22T08:25:18.586 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/install/trap' -> './k/coreutils-8.5/tests/install/trap' 2023-06-22T08:25:18.586 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/install/basic-1' -> './k/coreutils-8.5/tests/install/basic-1' 2023-06-22T08:25:18.586 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/install/strip-program' -> './k/coreutils-8.5/tests/install/strip-program' 2023-06-22T08:25:18.586 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/install/d-slashdot' -> './k/coreutils-8.5/tests/install/d-slashdot' 2023-06-22T08:25:18.587 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/install/install-C' -> './k/coreutils-8.5/tests/install/install-C' 2023-06-22T08:25:18.587 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/install/install-C-root' -> './k/coreutils-8.5/tests/install/install-C-root' 2023-06-22T08:25:18.587 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/install/create-leading' -> './k/coreutils-8.5/tests/install/create-leading' 2023-06-22T08:25:18.587 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod' -> './k/coreutils-8.5/tests/chmod' 2023-06-22T08:25:18.587 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/octal' -> './k/coreutils-8.5/tests/chmod/octal' 2023-06-22T08:25:18.587 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/umask-x' -> './k/coreutils-8.5/tests/chmod/umask-x' 2023-06-22T08:25:18.587 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/inaccessible' -> './k/coreutils-8.5/tests/chmod/inaccessible' 2023-06-22T08:25:18.588 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/equal-x' -> './k/coreutils-8.5/tests/chmod/equal-x' 2023-06-22T08:25:18.588 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/silent' -> './k/coreutils-8.5/tests/chmod/silent' 2023-06-22T08:25:18.588 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/equals' -> './k/coreutils-8.5/tests/chmod/equals' 2023-06-22T08:25:18.588 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/usage' -> './k/coreutils-8.5/tests/chmod/usage' 2023-06-22T08:25:18.588 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/thru-dangling' -> './k/coreutils-8.5/tests/chmod/thru-dangling' 2023-06-22T08:25:18.588 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/setgid' -> './k/coreutils-8.5/tests/chmod/setgid' 2023-06-22T08:25:18.588 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/no-x' -> './k/coreutils-8.5/tests/chmod/no-x' 2023-06-22T08:25:18.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chmod/c-option' -> './k/coreutils-8.5/tests/chmod/c-option' 2023-06-22T08:25:18.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/df' -> './k/coreutils-8.5/tests/df' 2023-06-22T08:25:18.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/df/total-verify' -> './k/coreutils-8.5/tests/df/total-verify' 2023-06-22T08:25:18.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/df/unreadable' -> './k/coreutils-8.5/tests/df/unreadable' 2023-06-22T08:25:18.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du' -> './k/coreutils-8.5/tests/du' 2023-06-22T08:25:18.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/exclude' -> './k/coreutils-8.5/tests/du/exclude' 2023-06-22T08:25:18.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/one-file-system' -> './k/coreutils-8.5/tests/du/one-file-system' 2023-06-22T08:25:18.589 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/long-sloop' -> './k/coreutils-8.5/tests/du/long-sloop' 2023-06-22T08:25:18.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/slink' -> './k/coreutils-8.5/tests/du/slink' 2023-06-22T08:25:18.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/inacc-dir' -> './k/coreutils-8.5/tests/du/inacc-dir' 2023-06-22T08:25:18.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/inaccessible-cwd' -> './k/coreutils-8.5/tests/du/inaccessible-cwd' 2023-06-22T08:25:18.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/inacc-dest' -> './k/coreutils-8.5/tests/du/inacc-dest' 2023-06-22T08:25:18.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/deref-args' -> './k/coreutils-8.5/tests/du/deref-args' 2023-06-22T08:25:18.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/deref' -> './k/coreutils-8.5/tests/du/deref' 2023-06-22T08:25:18.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/trailing-slash' -> './k/coreutils-8.5/tests/du/trailing-slash' 2023-06-22T08:25:18.590 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/files0-from' -> './k/coreutils-8.5/tests/du/files0-from' 2023-06-22T08:25:18.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/two-args' -> './k/coreutils-8.5/tests/du/two-args' 2023-06-22T08:25:18.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/hard-link' -> './k/coreutils-8.5/tests/du/hard-link' 2023-06-22T08:25:18.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/long-from-unreadable' -> './k/coreutils-8.5/tests/du/long-from-unreadable' 2023-06-22T08:25:18.591 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/8gb' -> './k/coreutils-8.5/tests/du/8gb' 2023-06-22T08:25:18.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/2g' -> './k/coreutils-8.5/tests/du/2g' 2023-06-22T08:25:18.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/restore-wd' -> './k/coreutils-8.5/tests/du/restore-wd' 2023-06-22T08:25:18.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/slash' -> './k/coreutils-8.5/tests/du/slash' 2023-06-22T08:25:18.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/no-x' -> './k/coreutils-8.5/tests/du/no-x' 2023-06-22T08:25:18.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/fd-leak' -> './k/coreutils-8.5/tests/du/fd-leak' 2023-06-22T08:25:18.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/no-deref' -> './k/coreutils-8.5/tests/du/no-deref' 2023-06-22T08:25:18.592 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/du/basic' -> './k/coreutils-8.5/tests/du/basic' 2023-06-22T08:25:18.666 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls' -> './k/coreutils-8.5/tests/ls' 2023-06-22T08:25:18.667 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/readdir-mountpoint-inode' -> './k/coreutils-8.5/tests/ls/readdir-mountpoint-inode' 2023-06-22T08:25:18.667 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/multihardlink' -> './k/coreutils-8.5/tests/ls/multihardlink' 2023-06-22T08:25:18.667 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/color-dtype-dir' -> './k/coreutils-8.5/tests/ls/color-dtype-dir' 2023-06-22T08:25:18.667 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/file-type' -> './k/coreutils-8.5/tests/ls/file-type' 2023-06-22T08:25:18.667 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/rt-1' -> './k/coreutils-8.5/tests/ls/rt-1' 2023-06-22T08:25:18.667 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/stat-free-symlinks' -> './k/coreutils-8.5/tests/ls/stat-free-symlinks' 2023-06-22T08:25:18.668 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/no-cap' -> './k/coreutils-8.5/tests/ls/no-cap' 2023-06-22T08:25:18.668 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/capability' -> './k/coreutils-8.5/tests/ls/capability' 2023-06-22T08:25:18.668 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/inode' -> './k/coreutils-8.5/tests/ls/inode' 2023-06-22T08:25:18.668 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/color-norm' -> './k/coreutils-8.5/tests/ls/color-norm' 2023-06-22T08:25:18.668 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/abmon-align' -> './k/coreutils-8.5/tests/ls/abmon-align' 2023-06-22T08:25:18.668 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/dangle' -> './k/coreutils-8.5/tests/ls/dangle' 2023-06-22T08:25:18.668 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/dired' -> './k/coreutils-8.5/tests/ls/dired' 2023-06-22T08:25:18.668 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/m-option' -> './k/coreutils-8.5/tests/ls/m-option' 2023-06-22T08:25:18.669 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/nameless-uid' -> './k/coreutils-8.5/tests/ls/nameless-uid' 2023-06-22T08:25:18.669 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/proc-selinux-segfault' -> './k/coreutils-8.5/tests/ls/proc-selinux-segfault' 2023-06-22T08:25:18.669 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/stat-vs-dirent' -> './k/coreutils-8.5/tests/ls/stat-vs-dirent' 2023-06-22T08:25:18.669 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/color-clear-to-eol' -> './k/coreutils-8.5/tests/ls/color-clear-to-eol' 2023-06-22T08:25:18.669 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/no-arg' -> './k/coreutils-8.5/tests/ls/no-arg' 2023-06-22T08:25:18.669 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/follow-slink' -> './k/coreutils-8.5/tests/ls/follow-slink' 2023-06-22T08:25:18.669 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/symlink-slash' -> './k/coreutils-8.5/tests/ls/symlink-slash' 2023-06-22T08:25:18.670 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/stat-dtype' -> './k/coreutils-8.5/tests/ls/stat-dtype' 2023-06-22T08:25:18.670 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/x-option' -> './k/coreutils-8.5/tests/ls/x-option' 2023-06-22T08:25:18.670 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/infloop' -> './k/coreutils-8.5/tests/ls/infloop' 2023-06-22T08:25:18.670 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/stat-failed' -> './k/coreutils-8.5/tests/ls/stat-failed' 2023-06-22T08:25:18.670 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/ls/recursive' -> './k/coreutils-8.5/tests/ls/recursive' 2023-06-22T08:25:18.670 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv' -> './k/coreutils-8.5/tests/mv' 2023-06-22T08:25:18.670 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/mv-n' -> './k/coreutils-8.5/tests/mv/mv-n' 2023-06-22T08:25:18.670 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/to-symlink' -> './k/coreutils-8.5/tests/mv/to-symlink' 2023-06-22T08:25:18.671 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/part-fail' -> './k/coreutils-8.5/tests/mv/part-fail' 2023-06-22T08:25:18.671 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/hard-4' -> './k/coreutils-8.5/tests/mv/hard-4' 2023-06-22T08:25:18.671 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/leak-fd' -> './k/coreutils-8.5/tests/mv/leak-fd' 2023-06-22T08:25:18.671 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/no-target-dir' -> './k/coreutils-8.5/tests/mv/no-target-dir' 2023-06-22T08:25:18.671 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/part-rename' -> './k/coreutils-8.5/tests/mv/part-rename' 2023-06-22T08:25:18.671 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/dir-file' -> './k/coreutils-8.5/tests/mv/dir-file' 2023-06-22T08:25:18.671 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/hard-2' -> './k/coreutils-8.5/tests/mv/hard-2' 2023-06-22T08:25:18.671 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/part-hardlink' -> './k/coreutils-8.5/tests/mv/part-hardlink' 2023-06-22T08:25:18.672 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/sticky-to-xpart' -> './k/coreutils-8.5/tests/mv/sticky-to-xpart' 2023-06-22T08:25:18.672 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/mv-special-1' -> './k/coreutils-8.5/tests/mv/mv-special-1' 2023-06-22T08:25:18.672 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/trailing-slash' -> './k/coreutils-8.5/tests/mv/trailing-slash' 2023-06-22T08:25:18.672 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/dir2dir' -> './k/coreutils-8.5/tests/mv/dir2dir' 2023-06-22T08:25:18.672 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/into-self-2' -> './k/coreutils-8.5/tests/mv/into-self-2' 2023-06-22T08:25:18.672 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/backup-dir' -> './k/coreutils-8.5/tests/mv/backup-dir' 2023-06-22T08:25:18.672 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/diag' -> './k/coreutils-8.5/tests/mv/diag' 2023-06-22T08:25:18.672 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/atomic' -> './k/coreutils-8.5/tests/mv/atomic' 2023-06-22T08:25:18.673 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/force' -> './k/coreutils-8.5/tests/mv/force' 2023-06-22T08:25:18.673 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/into-self-3' -> './k/coreutils-8.5/tests/mv/into-self-3' 2023-06-22T08:25:18.766 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/into-self' -> './k/coreutils-8.5/tests/mv/into-self' 2023-06-22T08:25:18.766 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/i-2' -> './k/coreutils-8.5/tests/mv/i-2' 2023-06-22T08:25:18.767 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/childproof' -> './k/coreutils-8.5/tests/mv/childproof' 2023-06-22T08:25:18.767 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/perm-1' -> './k/coreutils-8.5/tests/mv/perm-1' 2023-06-22T08:25:18.767 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/i-3' -> './k/coreutils-8.5/tests/mv/i-3' 2023-06-22T08:25:18.767 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/partition-perm' -> './k/coreutils-8.5/tests/mv/partition-perm' 2023-06-22T08:25:18.767 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/part-symlink' -> './k/coreutils-8.5/tests/mv/part-symlink' 2023-06-22T08:25:18.767 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/backup-is-src' -> './k/coreutils-8.5/tests/mv/backup-is-src' 2023-06-22T08:25:18.767 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/i-1' -> './k/coreutils-8.5/tests/mv/i-1' 2023-06-22T08:25:18.768 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/update' -> './k/coreutils-8.5/tests/mv/update' 2023-06-22T08:25:18.768 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/i-4' -> './k/coreutils-8.5/tests/mv/i-4' 2023-06-22T08:25:18.768 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/hard-verbose' -> './k/coreutils-8.5/tests/mv/hard-verbose' 2023-06-22T08:25:18.768 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/dup-source' -> './k/coreutils-8.5/tests/mv/dup-source' 2023-06-22T08:25:18.768 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/i-link-no' -> './k/coreutils-8.5/tests/mv/i-link-no' 2023-06-22T08:25:18.768 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/atomic2' -> './k/coreutils-8.5/tests/mv/atomic2' 2023-06-22T08:25:18.768 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/into-self-4' -> './k/coreutils-8.5/tests/mv/into-self-4' 2023-06-22T08:25:18.769 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/hard-link-1' -> './k/coreutils-8.5/tests/mv/hard-link-1' 2023-06-22T08:25:18.769 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/i-5' -> './k/coreutils-8.5/tests/mv/i-5' 2023-06-22T08:25:18.769 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/acl' -> './k/coreutils-8.5/tests/mv/acl' 2023-06-22T08:25:18.769 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/mv/hard-3' -> './k/coreutils-8.5/tests/mv/hard-3' 2023-06-22T08:25:18.769 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch' -> './k/coreutils-8.5/tests/touch' 2023-06-22T08:25:18.769 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/empty-file' -> './k/coreutils-8.5/tests/touch/empty-file' 2023-06-22T08:25:18.769 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/read-only' -> './k/coreutils-8.5/tests/touch/read-only' 2023-06-22T08:25:18.769 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/fail-diag' -> './k/coreutils-8.5/tests/touch/fail-diag' 2023-06-22T08:25:18.770 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/trailing-slash' -> './k/coreutils-8.5/tests/touch/trailing-slash' 2023-06-22T08:25:18.770 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/no-dereference' -> './k/coreutils-8.5/tests/touch/no-dereference' 2023-06-22T08:25:18.770 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/dir-1' -> './k/coreutils-8.5/tests/touch/dir-1' 2023-06-22T08:25:18.770 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/dangling-symlink' -> './k/coreutils-8.5/tests/touch/dangling-symlink' 2023-06-22T08:25:18.770 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/now-owned-by-other' -> './k/coreutils-8.5/tests/touch/now-owned-by-other' 2023-06-22T08:25:18.770 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/no-create-missing' -> './k/coreutils-8.5/tests/touch/no-create-missing' 2023-06-22T08:25:18.770 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/no-rights' -> './k/coreutils-8.5/tests/touch/no-rights' 2023-06-22T08:25:18.770 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/obsolescent' -> './k/coreutils-8.5/tests/touch/obsolescent' 2023-06-22T08:25:18.771 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/60-seconds' -> './k/coreutils-8.5/tests/touch/60-seconds' 2023-06-22T08:25:18.771 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/not-owner' -> './k/coreutils-8.5/tests/touch/not-owner' 2023-06-22T08:25:18.771 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/relative' -> './k/coreutils-8.5/tests/touch/relative' 2023-06-22T08:25:18.771 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/touch/fifo' -> './k/coreutils-8.5/tests/touch/fifo' 2023-06-22T08:25:18.771 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc' -> './k/coreutils-8.5/tests/misc' 2023-06-22T08:25:18.771 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/basename' -> './k/coreutils-8.5/tests/misc/basename' 2023-06-22T08:25:18.771 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/stat-fmt' -> './k/coreutils-8.5/tests/misc/stat-fmt' 2023-06-22T08:25:18.771 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/pr' -> './k/coreutils-8.5/tests/misc/pr' 2023-06-22T08:25:18.772 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/dirname' -> './k/coreutils-8.5/tests/misc/dirname' 2023-06-22T08:25:18.772 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/chcon-fail' -> './k/coreutils-8.5/tests/misc/chcon-fail' 2023-06-22T08:25:18.772 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/head-elide-tail' -> './k/coreutils-8.5/tests/misc/head-elide-tail' 2023-06-22T08:25:18.772 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/chroot-credentials' -> './k/coreutils-8.5/tests/misc/chroot-credentials' 2023-06-22T08:25:18.772 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/nl' -> './k/coreutils-8.5/tests/misc/nl' 2023-06-22T08:25:18.772 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/tee-dash' -> './k/coreutils-8.5/tests/misc/tee-dash' 2023-06-22T08:25:18.772 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/stat-hyphen' -> './k/coreutils-8.5/tests/misc/stat-hyphen' 2023-06-22T08:25:18.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sort-rand' -> './k/coreutils-8.5/tests/misc/sort-rand' 2023-06-22T08:25:18.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/expr' -> './k/coreutils-8.5/tests/misc/expr' 2023-06-22T08:25:18.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/fmt-long-line' -> './k/coreutils-8.5/tests/misc/fmt-long-line' 2023-06-22T08:25:18.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/split-fail' -> './k/coreutils-8.5/tests/misc/split-fail' 2023-06-22T08:25:18.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/truncate-relative' -> './k/coreutils-8.5/tests/misc/truncate-relative' 2023-06-22T08:25:18.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/truncate-dir-fail' -> './k/coreutils-8.5/tests/misc/truncate-dir-fail' 2023-06-22T08:25:18.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/csplit' -> './k/coreutils-8.5/tests/misc/csplit' 2023-06-22T08:25:18.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/od-N' -> './k/coreutils-8.5/tests/misc/od-N' 2023-06-22T08:25:18.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/xattr' -> './k/coreutils-8.5/tests/misc/xattr' 2023-06-22T08:25:18.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/mktemp' -> './k/coreutils-8.5/tests/misc/mktemp' 2023-06-22T08:25:18.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/cut' -> './k/coreutils-8.5/tests/misc/cut' 2023-06-22T08:25:18.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/cat-proc' -> './k/coreutils-8.5/tests/misc/cat-proc' 2023-06-22T08:25:18.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/comm' -> './k/coreutils-8.5/tests/misc/comm' 2023-06-22T08:25:18.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/timeout-parameters' -> './k/coreutils-8.5/tests/misc/timeout-parameters' 2023-06-22T08:25:18.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/tac-continue' -> './k/coreutils-8.5/tests/misc/tac-continue' 2023-06-22T08:25:18.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/tty-eof' -> './k/coreutils-8.5/tests/misc/tty-eof' 2023-06-22T08:25:18.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/df-P' -> './k/coreutils-8.5/tests/misc/df-P' 2023-06-22T08:25:18.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/shuf' -> './k/coreutils-8.5/tests/misc/shuf' 2023-06-22T08:25:18.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/env' -> './k/coreutils-8.5/tests/misc/env' 2023-06-22T08:25:18.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/od' -> './k/coreutils-8.5/tests/misc/od' 2023-06-22T08:25:18.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/split-a' -> './k/coreutils-8.5/tests/misc/split-a' 2023-06-22T08:25:18.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/tail' -> './k/coreutils-8.5/tests/misc/tail' 2023-06-22T08:25:18.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/stat-slash' -> './k/coreutils-8.5/tests/misc/stat-slash' 2023-06-22T08:25:18.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/timeout' -> './k/coreutils-8.5/tests/misc/timeout' 2023-06-22T08:25:18.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/close-stdout' -> './k/coreutils-8.5/tests/misc/close-stdout' 2023-06-22T08:25:18.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/wc-files0-from' -> './k/coreutils-8.5/tests/misc/wc-files0-from' 2023-06-22T08:25:18.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sum' -> './k/coreutils-8.5/tests/misc/sum' 2023-06-22T08:25:18.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/expand' -> './k/coreutils-8.5/tests/misc/expand' 2023-06-22T08:25:18.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sum-sysv' -> './k/coreutils-8.5/tests/misc/sum-sysv' 2023-06-22T08:25:18.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/wc-parallel' -> './k/coreutils-8.5/tests/misc/wc-parallel' 2023-06-22T08:25:18.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/md5sum-newline' -> './k/coreutils-8.5/tests/misc/md5sum-newline' 2023-06-22T08:25:18.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sha224sum' -> './k/coreutils-8.5/tests/misc/sha224sum' 2023-06-22T08:25:18.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/stty' -> './k/coreutils-8.5/tests/misc/stty' 2023-06-22T08:25:18.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/printf-hex' -> './k/coreutils-8.5/tests/misc/printf-hex' 2023-06-22T08:25:18.828 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/id-context' -> './k/coreutils-8.5/tests/misc/id-context' 2023-06-22T08:25:18.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/stat-printf' -> './k/coreutils-8.5/tests/misc/stat-printf' 2023-06-22T08:25:18.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/pwd-option' -> './k/coreutils-8.5/tests/misc/pwd-option' 2023-06-22T08:25:18.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sha256sum' -> './k/coreutils-8.5/tests/misc/sha256sum' 2023-06-22T08:25:18.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sort-version' -> './k/coreutils-8.5/tests/misc/sort-version' 2023-06-22T08:25:18.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/groups-dash' -> './k/coreutils-8.5/tests/misc/groups-dash' 2023-06-22T08:25:18.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/printenv' -> './k/coreutils-8.5/tests/misc/printenv' 2023-06-22T08:25:18.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/seq' -> './k/coreutils-8.5/tests/misc/seq' 2023-06-22T08:25:18.829 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/fmt' -> './k/coreutils-8.5/tests/misc/fmt' 2023-06-22T08:25:18.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/cat-buf' -> './k/coreutils-8.5/tests/misc/cat-buf' 2023-06-22T08:25:18.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/dircolors' -> './k/coreutils-8.5/tests/misc/dircolors' 2023-06-22T08:25:18.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/fold' -> './k/coreutils-8.5/tests/misc/fold' 2023-06-22T08:25:18.830 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/selinux' -> './k/coreutils-8.5/tests/misc/selinux' 2023-06-22T08:25:18.880 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/md5sum-parallel' -> './k/coreutils-8.5/tests/misc/md5sum-parallel' 2023-06-22T08:25:18.880 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/md5sum' -> './k/coreutils-8.5/tests/misc/md5sum' 2023-06-22T08:25:18.880 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/nice-fail' -> './k/coreutils-8.5/tests/misc/nice-fail' 2023-06-22T08:25:18.880 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/groups-version' -> './k/coreutils-8.5/tests/misc/groups-version' 2023-06-22T08:25:18.880 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/date-sec' -> './k/coreutils-8.5/tests/misc/date-sec' 2023-06-22T08:25:18.880 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/split-l' -> './k/coreutils-8.5/tests/misc/split-l' 2023-06-22T08:25:18.881 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/false-status' -> './k/coreutils-8.5/tests/misc/false-status' 2023-06-22T08:25:18.881 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sha384sum' -> './k/coreutils-8.5/tests/misc/sha384sum' 2023-06-22T08:25:18.881 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/truncate-dangling-symlink' -> './k/coreutils-8.5/tests/misc/truncate-dangling-symlink' 2023-06-22T08:25:18.881 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/od-multiple-t' -> './k/coreutils-8.5/tests/misc/od-multiple-t' 2023-06-22T08:25:18.881 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/wc-files0' -> './k/coreutils-8.5/tests/misc/wc-files0' 2023-06-22T08:25:18.881 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/nohup' -> './k/coreutils-8.5/tests/misc/nohup' 2023-06-22T08:25:18.881 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/truncate-no-create-missing' -> './k/coreutils-8.5/tests/misc/truncate-no-create-missing' 2023-06-22T08:25:18.882 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/head-pos' -> './k/coreutils-8.5/tests/misc/head-pos' 2023-06-22T08:25:18.882 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/uniq' -> './k/coreutils-8.5/tests/misc/uniq' 2023-06-22T08:25:18.882 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/test-diag' -> './k/coreutils-8.5/tests/misc/test-diag' 2023-06-22T08:25:18.882 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/tac' -> './k/coreutils-8.5/tests/misc/tac' 2023-06-22T08:25:18.882 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sort-continue' -> './k/coreutils-8.5/tests/misc/sort-continue' 2023-06-22T08:25:18.882 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/head' -> './k/coreutils-8.5/tests/misc/head' 2023-06-22T08:25:18.882 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/ptx' -> './k/coreutils-8.5/tests/misc/ptx' 2023-06-22T08:25:18.882 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/nproc-avail' -> './k/coreutils-8.5/tests/misc/nproc-avail' 2023-06-22T08:25:18.883 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/su-fail' -> './k/coreutils-8.5/tests/misc/su-fail' 2023-06-22T08:25:18.883 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/od-x8' -> './k/coreutils-8.5/tests/misc/od-x8' 2023-06-22T08:25:18.883 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/nice' -> './k/coreutils-8.5/tests/misc/nice' 2023-06-22T08:25:18.883 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/chcon' -> './k/coreutils-8.5/tests/misc/chcon' 2023-06-22T08:25:18.883 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/head-c' -> './k/coreutils-8.5/tests/misc/head-c' 2023-06-22T08:25:18.883 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sort-compress' -> './k/coreutils-8.5/tests/misc/sort-compress' 2023-06-22T08:25:18.883 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/df' -> './k/coreutils-8.5/tests/misc/df' 2023-06-22T08:25:18.883 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/chroot-fail' -> './k/coreutils-8.5/tests/misc/chroot-fail' 2023-06-22T08:25:18.884 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/date-next-dow' -> './k/coreutils-8.5/tests/misc/date-next-dow' 2023-06-22T08:25:18.884 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sha1sum' -> './k/coreutils-8.5/tests/misc/sha1sum' 2023-06-22T08:25:18.884 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sha1sum-vec' -> './k/coreutils-8.5/tests/misc/sha1sum-vec' 2023-06-22T08:25:18.884 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/pathchk1' -> './k/coreutils-8.5/tests/misc/pathchk1' 2023-06-22T08:25:18.884 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/arch' -> './k/coreutils-8.5/tests/misc/arch' 2023-06-22T08:25:18.884 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sort-month' -> './k/coreutils-8.5/tests/misc/sort-month' 2023-06-22T08:25:18.884 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/shred-passes' -> './k/coreutils-8.5/tests/misc/shred-passes' 2023-06-22T08:25:18.885 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sha512sum' -> './k/coreutils-8.5/tests/misc/sha512sum' 2023-06-22T08:25:18.885 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/paste' -> './k/coreutils-8.5/tests/misc/paste' 2023-06-22T08:25:18.885 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/wc' -> './k/coreutils-8.5/tests/misc/wc' 2023-06-22T08:25:18.885 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/ls-time' -> './k/coreutils-8.5/tests/misc/ls-time' 2023-06-22T08:25:18.885 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/pwd-unreadable-parent' -> './k/coreutils-8.5/tests/misc/pwd-unreadable-parent' 2023-06-22T08:25:18.885 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/test' -> './k/coreutils-8.5/tests/misc/test' 2023-06-22T08:25:18.885 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/truncate-parameters' -> './k/coreutils-8.5/tests/misc/truncate-parameters' 2023-06-22T08:25:18.885 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sort-files0-from' -> './k/coreutils-8.5/tests/misc/sort-files0-from' 2023-06-22T08:25:18.886 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/base64' -> './k/coreutils-8.5/tests/misc/base64' 2023-06-22T08:25:20.357 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/ptx-overrun' -> './k/coreutils-8.5/tests/misc/ptx-overrun' 2023-06-22T08:25:20.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/printf-cov' -> './k/coreutils-8.5/tests/misc/printf-cov' 2023-06-22T08:25:20.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/tee' -> './k/coreutils-8.5/tests/misc/tee' 2023-06-22T08:25:20.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/mknod' -> './k/coreutils-8.5/tests/misc/mknod' 2023-06-22T08:25:20.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/printf-surprise' -> './k/coreutils-8.5/tests/misc/printf-surprise' 2023-06-22T08:25:20.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/ls-misc' -> './k/coreutils-8.5/tests/misc/ls-misc' 2023-06-22T08:25:20.358 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/nproc-positive' -> './k/coreutils-8.5/tests/misc/nproc-positive' 2023-06-22T08:25:20.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/join' -> './k/coreutils-8.5/tests/misc/join' 2023-06-22T08:25:20.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sort-merge' -> './k/coreutils-8.5/tests/misc/sort-merge' 2023-06-22T08:25:20.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/factor' -> './k/coreutils-8.5/tests/misc/factor' 2023-06-22T08:25:20.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sort' -> './k/coreutils-8.5/tests/misc/sort' 2023-06-22T08:25:20.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/truncate-fifo' -> './k/coreutils-8.5/tests/misc/truncate-fifo' 2023-06-22T08:25:20.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/truncate-fail-diag' -> './k/coreutils-8.5/tests/misc/truncate-fail-diag' 2023-06-22T08:25:20.359 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/stty-invalid' -> './k/coreutils-8.5/tests/misc/stty-invalid' 2023-06-22T08:25:20.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/env-null' -> './k/coreutils-8.5/tests/misc/env-null' 2023-06-22T08:25:20.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/printf' -> './k/coreutils-8.5/tests/misc/printf' 2023-06-22T08:25:20.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/seq-long-double' -> './k/coreutils-8.5/tests/misc/seq-long-double' 2023-06-22T08:25:20.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/pwd-long' -> './k/coreutils-8.5/tests/misc/pwd-long' 2023-06-22T08:25:20.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/tsort' -> './k/coreutils-8.5/tests/misc/tsort' 2023-06-22T08:25:20.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/runcon-no-reorder' -> './k/coreutils-8.5/tests/misc/runcon-no-reorder' 2023-06-22T08:25:20.360 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/date' -> './k/coreutils-8.5/tests/misc/date' 2023-06-22T08:25:20.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/shred-exact' -> './k/coreutils-8.5/tests/misc/shred-exact' 2023-06-22T08:25:20.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/shred-remove' -> './k/coreutils-8.5/tests/misc/shred-remove' 2023-06-22T08:25:20.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/stdbuf' -> './k/coreutils-8.5/tests/misc/stdbuf' 2023-06-22T08:25:20.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/truncate-overflow' -> './k/coreutils-8.5/tests/misc/truncate-overflow' 2023-06-22T08:25:20.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/stty-row-col' -> './k/coreutils-8.5/tests/misc/stty-row-col' 2023-06-22T08:25:20.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/truncate-owned-by-other' -> './k/coreutils-8.5/tests/misc/truncate-owned-by-other' 2023-06-22T08:25:20.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/xstrtol' -> './k/coreutils-8.5/tests/misc/xstrtol' 2023-06-22T08:25:20.361 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/invalid-opt' -> './k/coreutils-8.5/tests/misc/invalid-opt' 2023-06-22T08:25:20.362 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/readlink-fp-loop' -> './k/coreutils-8.5/tests/misc/readlink-fp-loop' 2023-06-22T08:25:20.362 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/tr' -> './k/coreutils-8.5/tests/misc/tr' 2023-06-22T08:25:20.362 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/help-version' -> './k/coreutils-8.5/tests/misc/help-version' 2023-06-22T08:25:20.362 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/id-groups' -> './k/coreutils-8.5/tests/misc/id-groups' 2023-06-22T08:25:20.362 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/unexpand' -> './k/coreutils-8.5/tests/misc/unexpand' 2023-06-22T08:25:20.362 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/misc/sort-merge-fdlimit' -> './k/coreutils-8.5/tests/misc/sort-merge-fdlimit' 2023-06-22T08:25:20.362 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chgrp' -> './k/coreutils-8.5/tests/chgrp' 2023-06-22T08:25:20.362 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chgrp/default-no-deref' -> './k/coreutils-8.5/tests/chgrp/default-no-deref' 2023-06-22T08:25:20.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chgrp/deref' -> './k/coreutils-8.5/tests/chgrp/deref' 2023-06-22T08:25:20.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chgrp/posix-H' -> './k/coreutils-8.5/tests/chgrp/posix-H' 2023-06-22T08:25:20.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chgrp/recurse' -> './k/coreutils-8.5/tests/chgrp/recurse' 2023-06-22T08:25:20.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chgrp/no-x' -> './k/coreutils-8.5/tests/chgrp/no-x' 2023-06-22T08:25:20.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/chgrp/basic' -> './k/coreutils-8.5/tests/chgrp/basic' 2023-06-22T08:25:20.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2' -> './k/coreutils-8.5/tests/tail-2' 2023-06-22T08:25:20.363 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/follow-stdin' -> './k/coreutils-8.5/tests/tail-2/follow-stdin' 2023-06-22T08:25:20.364 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/F-vs-rename' -> './k/coreutils-8.5/tests/tail-2/F-vs-rename' 2023-06-22T08:25:20.438 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~298s 2023-06-22T08:25:20.501 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/inotify-rotate' -> './k/coreutils-8.5/tests/tail-2/inotify-rotate' 2023-06-22T08:25:20.501 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/pid' -> './k/coreutils-8.5/tests/tail-2/pid' 2023-06-22T08:25:20.501 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/infloop-1' -> './k/coreutils-8.5/tests/tail-2/infloop-1' 2023-06-22T08:25:20.502 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/assert-2' -> './k/coreutils-8.5/tests/tail-2/assert-2' 2023-06-22T08:25:20.502 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/pipe-f' -> './k/coreutils-8.5/tests/tail-2/pipe-f' 2023-06-22T08:25:20.502 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/pipe-f2' -> './k/coreutils-8.5/tests/tail-2/pipe-f2' 2023-06-22T08:25:20.502 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/start-middle' -> './k/coreutils-8.5/tests/tail-2/start-middle' 2023-06-22T08:25:20.502 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/inotify-hash-abuse2' -> './k/coreutils-8.5/tests/tail-2/inotify-hash-abuse2' 2023-06-22T08:25:20.502 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/inotify-hash-abuse' -> './k/coreutils-8.5/tests/tail-2/inotify-hash-abuse' 2023-06-22T08:25:20.502 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/inotify-race' -> './k/coreutils-8.5/tests/tail-2/inotify-race' 2023-06-22T08:25:20.503 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/append-only' -> './k/coreutils-8.5/tests/tail-2/append-only' 2023-06-22T08:25:20.503 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/tail-n0f' -> './k/coreutils-8.5/tests/tail-2/tail-n0f' 2023-06-22T08:25:20.503 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/flush-initial' -> './k/coreutils-8.5/tests/tail-2/flush-initial' 2023-06-22T08:25:20.503 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/assert' -> './k/coreutils-8.5/tests/tail-2/assert' 2023-06-22T08:25:20.503 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/wait' -> './k/coreutils-8.5/tests/tail-2/wait' 2023-06-22T08:25:20.503 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/proc-ksyms' -> './k/coreutils-8.5/tests/tail-2/proc-ksyms' 2023-06-22T08:25:20.503 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/tail-2/big-4gb' -> './k/coreutils-8.5/tests/tail-2/big-4gb' 2023-06-22T08:25:20.503 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/readlink' -> './k/coreutils-8.5/tests/readlink' 2023-06-22T08:25:20.504 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/readlink/can-m' -> './k/coreutils-8.5/tests/readlink/can-m' 2023-06-22T08:25:20.504 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/readlink/rl-1' -> './k/coreutils-8.5/tests/readlink/rl-1' 2023-06-22T08:25:20.504 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/readlink/can-e' -> './k/coreutils-8.5/tests/readlink/can-e' 2023-06-22T08:25:20.504 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/readlink/can-f' -> './k/coreutils-8.5/tests/readlink/can-f' 2023-06-22T08:25:20.504 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/tests/Coreutils.pm' -> './k/coreutils-8.5/tests/Coreutils.pm' 2023-06-22T08:25:20.504 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_trailing_blank' -> './k/coreutils-8.5/.x-sc_trailing_blank' 2023-06-22T08:25:20.504 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/configure' -> './k/coreutils-8.5/configure' 2023-06-22T08:25:20.505 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/README' -> './k/coreutils-8.5/README' 2023-06-22T08:25:20.505 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.vg-suppressions' -> './k/coreutils-8.5/.vg-suppressions' 2023-06-22T08:25:20.505 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_prohibit_atoi_atof' -> './k/coreutils-8.5/.x-sc_prohibit_atoi_atof' 2023-06-22T08:25:20.505 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_prohibit_stat_st_blocks' -> './k/coreutils-8.5/.x-sc_prohibit_stat_st_blocks' 2023-06-22T08:25:20.505 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/ChangeLog-2006' -> './k/coreutils-8.5/ChangeLog-2006' 2023-06-22T08:25:20.505 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_program_name' -> './k/coreutils-8.5/.x-sc_program_name' 2023-06-22T08:25:20.505 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/THANKStt.in' -> './k/coreutils-8.5/THANKStt.in' 2023-06-22T08:25:20.505 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/ChangeLog-2008' -> './k/coreutils-8.5/ChangeLog-2008' 2023-06-22T08:25:20.506 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/ChangeLog-2007' -> './k/coreutils-8.5/ChangeLog-2007' 2023-06-22T08:25:20.506 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc' -> './k/coreutils-8.5/doc' 2023-06-22T08:25:20.506 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/perm.texi' -> './k/coreutils-8.5/doc/perm.texi' 2023-06-22T08:25:20.506 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/version.texi' -> './k/coreutils-8.5/doc/version.texi' 2023-06-22T08:25:20.506 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/Makefile.am' -> './k/coreutils-8.5/doc/Makefile.am' 2023-06-22T08:25:20.506 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/coreutils.info' -> './k/coreutils-8.5/doc/coreutils.info' 2023-06-22T08:25:20.506 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/stamp-vti' -> './k/coreutils-8.5/doc/stamp-vti' 2023-06-22T08:25:20.506 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/fdl.texi' -> './k/coreutils-8.5/doc/fdl.texi' 2023-06-22T08:25:20.507 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/Makefile.in' -> './k/coreutils-8.5/doc/Makefile.in' 2023-06-22T08:25:20.507 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/getdate.texi' -> './k/coreutils-8.5/doc/getdate.texi' 2023-06-22T08:25:20.507 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/constants.texi' -> './k/coreutils-8.5/doc/constants.texi' 2023-06-22T08:25:20.507 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/ChangeLog-2007' -> './k/coreutils-8.5/doc/ChangeLog-2007' 2023-06-22T08:25:20.507 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/doc/coreutils.texi' -> './k/coreutils-8.5/doc/coreutils.texi' 2023-06-22T08:25:20.507 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src' -> './k/coreutils-8.5/src' 2023-06-22T08:25:20.507 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/dircolors.hin' -> './k/coreutils-8.5/src/dircolors.hin' 2023-06-22T08:25:20.507 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/test.c' -> './k/coreutils-8.5/src/test.c' 2023-06-22T08:25:20.574 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/nproc.c' -> './k/coreutils-8.5/src/nproc.c' 2023-06-22T08:25:20.574 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/cp-hash.c' -> './k/coreutils-8.5/src/cp-hash.c' 2023-06-22T08:25:20.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/rm.c' -> './k/coreutils-8.5/src/rm.c' 2023-06-22T08:25:20.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/uname-arch.c' -> './k/coreutils-8.5/src/uname-arch.c' 2023-06-22T08:25:20.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/chroot.c' -> './k/coreutils-8.5/src/chroot.c' 2023-06-22T08:25:20.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/remove.h' -> './k/coreutils-8.5/src/remove.h' 2023-06-22T08:25:20.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/users.c' -> './k/coreutils-8.5/src/users.c' 2023-06-22T08:25:20.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/dircolors.c' -> './k/coreutils-8.5/src/dircolors.c' 2023-06-22T08:25:20.575 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/join.c' -> './k/coreutils-8.5/src/join.c' 2023-06-22T08:25:20.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/chown.c' -> './k/coreutils-8.5/src/chown.c' 2023-06-22T08:25:20.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/shuf.c' -> './k/coreutils-8.5/src/shuf.c' 2023-06-22T08:25:20.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/sum.c' -> './k/coreutils-8.5/src/sum.c' 2023-06-22T08:25:20.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/csplit.c' -> './k/coreutils-8.5/src/csplit.c' 2023-06-22T08:25:20.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/libstdbuf.c' -> './k/coreutils-8.5/src/libstdbuf.c' 2023-06-22T08:25:20.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/fold.c' -> './k/coreutils-8.5/src/fold.c' 2023-06-22T08:25:20.576 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/basename.c' -> './k/coreutils-8.5/src/basename.c' 2023-06-22T08:25:20.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/prog-fprintf.h' -> './k/coreutils-8.5/src/prog-fprintf.h' 2023-06-22T08:25:20.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/sync.c' -> './k/coreutils-8.5/src/sync.c' 2023-06-22T08:25:20.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/lbracket.c' -> './k/coreutils-8.5/src/lbracket.c' 2023-06-22T08:25:20.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/ln.c' -> './k/coreutils-8.5/src/ln.c' 2023-06-22T08:25:20.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/tsort.c' -> './k/coreutils-8.5/src/tsort.c' 2023-06-22T08:25:20.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/wc.c' -> './k/coreutils-8.5/src/wc.c' 2023-06-22T08:25:20.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/extract-magic' -> './k/coreutils-8.5/src/extract-magic' 2023-06-22T08:25:20.577 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/hostid.c' -> './k/coreutils-8.5/src/hostid.c' 2023-06-22T08:25:20.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/dd.c' -> './k/coreutils-8.5/src/dd.c' 2023-06-22T08:25:20.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/shred.c' -> './k/coreutils-8.5/src/shred.c' 2023-06-22T08:25:20.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/uname.h' -> './k/coreutils-8.5/src/uname.h' 2023-06-22T08:25:20.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/unlink.c' -> './k/coreutils-8.5/src/unlink.c' 2023-06-22T08:25:20.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/Makefile.am' -> './k/coreutils-8.5/src/Makefile.am' 2023-06-22T08:25:20.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/seq.c' -> './k/coreutils-8.5/src/seq.c' 2023-06-22T08:25:20.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/uname-uname.c' -> './k/coreutils-8.5/src/uname-uname.c' 2023-06-22T08:25:20.578 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/ptx.c' -> './k/coreutils-8.5/src/ptx.c' 2023-06-22T08:25:20.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/wheel-size.h' -> './k/coreutils-8.5/src/wheel-size.h' 2023-06-22T08:25:20.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/stty.c' -> './k/coreutils-8.5/src/stty.c' 2023-06-22T08:25:20.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/chcon.c' -> './k/coreutils-8.5/src/chcon.c' 2023-06-22T08:25:20.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/dcgen' -> './k/coreutils-8.5/src/dcgen' 2023-06-22T08:25:20.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/uptime.c' -> './k/coreutils-8.5/src/uptime.c' 2023-06-22T08:25:20.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/link.c' -> './k/coreutils-8.5/src/link.c' 2023-06-22T08:25:20.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/install.c' -> './k/coreutils-8.5/src/install.c' 2023-06-22T08:25:20.579 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/uname.c' -> './k/coreutils-8.5/src/uname.c' 2023-06-22T08:25:20.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/mknod.c' -> './k/coreutils-8.5/src/mknod.c' 2023-06-22T08:25:20.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/su.c' -> './k/coreutils-8.5/src/su.c' 2023-06-22T08:25:20.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/expand.c' -> './k/coreutils-8.5/src/expand.c' 2023-06-22T08:25:20.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/mv.c' -> './k/coreutils-8.5/src/mv.c' 2023-06-22T08:25:20.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/ls.c' -> './k/coreutils-8.5/src/ls.c' 2023-06-22T08:25:20.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/whoami.c' -> './k/coreutils-8.5/src/whoami.c' 2023-06-22T08:25:20.580 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/ls-ls.c' -> './k/coreutils-8.5/src/ls-ls.c' 2023-06-22T08:25:20.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/ls.h' -> './k/coreutils-8.5/src/ls.h' 2023-06-22T08:25:20.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/id.c' -> './k/coreutils-8.5/src/id.c' 2023-06-22T08:25:20.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/Makefile.in' -> './k/coreutils-8.5/src/Makefile.in' 2023-06-22T08:25:20.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/du.c' -> './k/coreutils-8.5/src/du.c' 2023-06-22T08:25:20.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/cut.c' -> './k/coreutils-8.5/src/cut.c' 2023-06-22T08:25:20.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/comm.c' -> './k/coreutils-8.5/src/comm.c' 2023-06-22T08:25:20.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/od.c' -> './k/coreutils-8.5/src/od.c' 2023-06-22T08:25:20.581 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/expr.c' -> './k/coreutils-8.5/src/expr.c' 2023-06-22T08:25:20.582 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/nohup.c' -> './k/coreutils-8.5/src/nohup.c' 2023-06-22T08:25:20.582 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/system.h' -> './k/coreutils-8.5/src/system.h' 2023-06-22T08:25:20.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/cat.c' -> './k/coreutils-8.5/src/cat.c' 2023-06-22T08:25:20.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/mktemp.c' -> './k/coreutils-8.5/src/mktemp.c' 2023-06-22T08:25:20.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/tac.c' -> './k/coreutils-8.5/src/tac.c' 2023-06-22T08:25:20.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/ls-dir.c' -> './k/coreutils-8.5/src/ls-dir.c' 2023-06-22T08:25:20.650 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/tac-pipe.c' -> './k/coreutils-8.5/src/tac-pipe.c' 2023-06-22T08:25:20.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/touch.c' -> './k/coreutils-8.5/src/touch.c' 2023-06-22T08:25:20.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/false.c' -> './k/coreutils-8.5/src/false.c' 2023-06-22T08:25:20.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/echo.c' -> './k/coreutils-8.5/src/echo.c' 2023-06-22T08:25:20.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/printenv.c' -> './k/coreutils-8.5/src/printenv.c' 2023-06-22T08:25:20.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/logname.c' -> './k/coreutils-8.5/src/logname.c' 2023-06-22T08:25:20.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/group-list.c' -> './k/coreutils-8.5/src/group-list.c' 2023-06-22T08:25:20.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/pr.c' -> './k/coreutils-8.5/src/pr.c' 2023-06-22T08:25:20.651 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/chgrp.c' -> './k/coreutils-8.5/src/chgrp.c' 2023-06-22T08:25:20.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/cp-hash.h' -> './k/coreutils-8.5/src/cp-hash.h' 2023-06-22T08:25:20.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/getlimits.c' -> './k/coreutils-8.5/src/getlimits.c' 2023-06-22T08:25:20.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/rmdir.c' -> './k/coreutils-8.5/src/rmdir.c' 2023-06-22T08:25:20.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/tail.c' -> './k/coreutils-8.5/src/tail.c' 2023-06-22T08:25:20.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/timeout.c' -> './k/coreutils-8.5/src/timeout.c' 2023-06-22T08:25:20.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/sort.c' -> './k/coreutils-8.5/src/sort.c' 2023-06-22T08:25:20.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/operand2sig.c' -> './k/coreutils-8.5/src/operand2sig.c' 2023-06-22T08:25:20.652 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/factor.c' -> './k/coreutils-8.5/src/factor.c' 2023-06-22T08:25:20.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/paste.c' -> './k/coreutils-8.5/src/paste.c' 2023-06-22T08:25:20.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/chmod.c' -> './k/coreutils-8.5/src/chmod.c' 2023-06-22T08:25:20.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/pinky.c' -> './k/coreutils-8.5/src/pinky.c' 2023-06-22T08:25:20.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/unexpand.c' -> './k/coreutils-8.5/src/unexpand.c' 2023-06-22T08:25:20.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/cksum.c' -> './k/coreutils-8.5/src/cksum.c' 2023-06-22T08:25:20.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/date.c' -> './k/coreutils-8.5/src/date.c' 2023-06-22T08:25:20.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/mkdir.c' -> './k/coreutils-8.5/src/mkdir.c' 2023-06-22T08:25:20.653 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/printf.c' -> './k/coreutils-8.5/src/printf.c' 2023-06-22T08:25:20.654 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/true.c' -> './k/coreutils-8.5/src/true.c' 2023-06-22T08:25:20.654 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/c99-to-c89.diff' -> './k/coreutils-8.5/src/c99-to-c89.diff' 2023-06-22T08:25:20.654 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/groups.c' -> './k/coreutils-8.5/src/groups.c' 2023-06-22T08:25:20.654 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/dirname.c' -> './k/coreutils-8.5/src/dirname.c' 2023-06-22T08:25:20.654 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/chown-core.h' -> './k/coreutils-8.5/src/chown-core.h' 2023-06-22T08:25:20.654 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/fs.h' -> './k/coreutils-8.5/src/fs.h' 2023-06-22T08:25:20.654 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/nl.c' -> './k/coreutils-8.5/src/nl.c' 2023-06-22T08:25:20.655 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/tee.c' -> './k/coreutils-8.5/src/tee.c' 2023-06-22T08:25:20.655 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/dircolors.h' -> './k/coreutils-8.5/src/dircolors.h' 2023-06-22T08:25:20.655 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/head.c' -> './k/coreutils-8.5/src/head.c' 2023-06-22T08:25:20.655 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/uniq.c' -> './k/coreutils-8.5/src/uniq.c' 2023-06-22T08:25:20.655 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/chown-core.c' -> './k/coreutils-8.5/src/chown-core.c' 2023-06-22T08:25:20.655 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/runcon.c' -> './k/coreutils-8.5/src/runcon.c' 2023-06-22T08:25:20.655 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/sleep.c' -> './k/coreutils-8.5/src/sleep.c' 2023-06-22T08:25:20.655 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/md5sum.c' -> './k/coreutils-8.5/src/md5sum.c' 2023-06-22T08:25:20.656 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/yes.c' -> './k/coreutils-8.5/src/yes.c' 2023-06-22T08:25:20.656 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/readlink.c' -> './k/coreutils-8.5/src/readlink.c' 2023-06-22T08:25:20.656 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/env.c' -> './k/coreutils-8.5/src/env.c' 2023-06-22T08:25:20.656 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/df.c' -> './k/coreutils-8.5/src/df.c' 2023-06-22T08:25:20.656 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/nice.c' -> './k/coreutils-8.5/src/nice.c' 2023-06-22T08:25:20.656 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/fmt.c' -> './k/coreutils-8.5/src/fmt.c' 2023-06-22T08:25:20.656 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/base64.c' -> './k/coreutils-8.5/src/base64.c' 2023-06-22T08:25:20.656 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/tty.c' -> './k/coreutils-8.5/src/tty.c' 2023-06-22T08:25:20.657 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/truncate.c' -> './k/coreutils-8.5/src/truncate.c' 2023-06-22T08:25:20.657 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/mkfifo.c' -> './k/coreutils-8.5/src/mkfifo.c' 2023-06-22T08:25:20.657 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/operand2sig.h' -> './k/coreutils-8.5/src/operand2sig.h' 2023-06-22T08:25:20.657 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/copy.h' -> './k/coreutils-8.5/src/copy.h' 2023-06-22T08:25:20.657 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/kill.c' -> './k/coreutils-8.5/src/kill.c' 2023-06-22T08:25:20.773 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/who.c' -> './k/coreutils-8.5/src/who.c' 2023-06-22T08:25:20.774 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/stat.c' -> './k/coreutils-8.5/src/stat.c' 2023-06-22T08:25:20.774 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/remove.c' -> './k/coreutils-8.5/src/remove.c' 2023-06-22T08:25:20.774 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/pwd.c' -> './k/coreutils-8.5/src/pwd.c' 2023-06-22T08:25:20.774 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/prog-fprintf.c' -> './k/coreutils-8.5/src/prog-fprintf.c' 2023-06-22T08:25:20.774 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/ls-vdir.c' -> './k/coreutils-8.5/src/ls-vdir.c' 2023-06-22T08:25:20.774 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/copy.c' -> './k/coreutils-8.5/src/copy.c' 2023-06-22T08:25:20.775 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/hostname.c' -> './k/coreutils-8.5/src/hostname.c' 2023-06-22T08:25:20.775 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/wheel-gen.pl' -> './k/coreutils-8.5/src/wheel-gen.pl' 2023-06-22T08:25:20.775 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/group-list.h' -> './k/coreutils-8.5/src/group-list.h' 2023-06-22T08:25:20.775 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/cp.c' -> './k/coreutils-8.5/src/cp.c' 2023-06-22T08:25:20.775 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/setuidgid.c' -> './k/coreutils-8.5/src/setuidgid.c' 2023-06-22T08:25:20.775 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/tr.c' -> './k/coreutils-8.5/src/tr.c' 2023-06-22T08:25:20.775 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/stdbuf.c' -> './k/coreutils-8.5/src/stdbuf.c' 2023-06-22T08:25:20.776 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/pathchk.c' -> './k/coreutils-8.5/src/pathchk.c' 2023-06-22T08:25:20.776 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/wheel.h' -> './k/coreutils-8.5/src/wheel.h' 2023-06-22T08:25:20.776 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/src/split.c' -> './k/coreutils-8.5/src/split.c' 2023-06-22T08:25:20.776 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_require_config_h' -> './k/coreutils-8.5/.x-sc_require_config_h' 2023-06-22T08:25:20.776 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_system_h_headers' -> './k/coreutils-8.5/.x-sc_system_h_headers' 2023-06-22T08:25:20.776 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_prohibit_fail_0' -> './k/coreutils-8.5/.x-sc_prohibit_fail_0' 2023-06-22T08:25:20.776 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_prohibit_magic_number_exit' -> './k/coreutils-8.5/.x-sc_prohibit_magic_number_exit' 2023-06-22T08:25:20.776 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/THANKS-to-translators' -> './k/coreutils-8.5/THANKS-to-translators' 2023-06-22T08:25:20.777 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/.x-sc_useless_cpp_parens' -> './k/coreutils-8.5/.x-sc_useless_cpp_parens' 2023-06-22T08:25:20.777 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests' -> './k/coreutils-8.5/gnulib-tests' 2023-06-22T08:25:20.777 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-ftello.sh' -> './k/coreutils-8.5/gnulib-tests/test-ftello.sh' 2023-06-22T08:25:20.777 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-snprintf.c' -> './k/coreutils-8.5/gnulib-tests/test-snprintf.c' 2023-06-22T08:25:20.777 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-openat.c' -> './k/coreutils-8.5/gnulib-tests/test-openat.c' 2023-06-22T08:25:20.777 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-dirent-safer.c' -> './k/coreutils-8.5/gnulib-tests/test-dirent-safer.c' 2023-06-22T08:25:20.777 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-cloexec.c' -> './k/coreutils-8.5/gnulib-tests/test-cloexec.c' 2023-06-22T08:25:20.777 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fstatat.c' -> './k/coreutils-8.5/gnulib-tests/test-fstatat.c' 2023-06-22T08:25:20.778 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fnmatch.c' -> './k/coreutils-8.5/gnulib-tests/test-fnmatch.c' 2023-06-22T08:25:20.778 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getndelim2.c' -> './k/coreutils-8.5/gnulib-tests/test-getndelim2.c' 2023-06-22T08:25:20.778 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-wcrtomb.sh' -> './k/coreutils-8.5/gnulib-tests/test-wcrtomb.sh' 2023-06-22T08:25:20.778 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-filenamecat.c' -> './k/coreutils-8.5/gnulib-tests/test-filenamecat.c' 2023-06-22T08:25:20.778 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fchownat.c' -> './k/coreutils-8.5/gnulib-tests/test-fchownat.c' 2023-06-22T08:25:20.778 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-stdlib.c' -> './k/coreutils-8.5/gnulib-tests/test-stdlib.c' 2023-06-22T08:25:20.778 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-read-file.c' -> './k/coreutils-8.5/gnulib-tests/test-read-file.c' 2023-06-22T08:25:20.779 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-exclude7.sh' -> './k/coreutils-8.5/gnulib-tests/test-exclude7.sh' 2023-06-22T08:25:20.779 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-closein.c' -> './k/coreutils-8.5/gnulib-tests/test-closein.c' 2023-06-22T08:25:20.779 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/connect.c' -> './k/coreutils-8.5/gnulib-tests/connect.c' 2023-06-22T08:25:20.779 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-rename.h' -> './k/coreutils-8.5/gnulib-tests/test-rename.h' 2023-06-22T08:25:20.779 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-memchr2.c' -> './k/coreutils-8.5/gnulib-tests/test-memchr2.c' 2023-06-22T08:25:20.779 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sys_time.c' -> './k/coreutils-8.5/gnulib-tests/test-sys_time.c' 2023-06-22T08:25:20.779 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/read-file.h' -> './k/coreutils-8.5/gnulib-tests/read-file.h' 2023-06-22T08:25:20.779 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mkdirat.c' -> './k/coreutils-8.5/gnulib-tests/test-mkdirat.c' 2023-06-22T08:25:20.821 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-isnand-nolibm.c' -> './k/coreutils-8.5/gnulib-tests/test-isnand-nolibm.c' 2023-06-22T08:25:20.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fopen-safer.c' -> './k/coreutils-8.5/gnulib-tests/test-fopen-safer.c' 2023-06-22T08:25:20.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-stdio.c' -> './k/coreutils-8.5/gnulib-tests/test-stdio.c' 2023-06-22T08:25:20.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fseeko.c' -> './k/coreutils-8.5/gnulib-tests/test-fseeko.c' 2023-06-22T08:25:20.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-arpa_inet.c' -> './k/coreutils-8.5/gnulib-tests/test-arpa_inet.c' 2023-06-22T08:25:20.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freadahead.sh' -> './k/coreutils-8.5/gnulib-tests/test-freadahead.sh' 2023-06-22T08:25:20.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-openat-safer.c' -> './k/coreutils-8.5/gnulib-tests/test-openat-safer.c' 2023-06-22T08:25:20.822 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-unlink.c' -> './k/coreutils-8.5/gnulib-tests/test-unlink.c' 2023-06-22T08:25:20.823 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsstr2.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbsstr2.sh' 2023-06-22T08:25:20.823 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-lchown.c' -> './k/coreutils-8.5/gnulib-tests/test-lchown.c' 2023-06-22T08:25:20.823 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-nl_langinfo.c' -> './k/coreutils-8.5/gnulib-tests/test-nl_langinfo.c' 2023-06-22T08:25:20.823 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-open.h' -> './k/coreutils-8.5/gnulib-tests/test-open.h' 2023-06-22T08:25:20.823 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-memcmp.c' -> './k/coreutils-8.5/gnulib-tests/test-memcmp.c' 2023-06-22T08:25:20.823 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-vfprintf-posix.c' -> './k/coreutils-8.5/gnulib-tests/test-vfprintf-posix.c' 2023-06-22T08:25:20.823 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freadahead.c' -> './k/coreutils-8.5/gnulib-tests/test-freadahead.c' 2023-06-22T08:25:20.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/ioctl.c' -> './k/coreutils-8.5/gnulib-tests/ioctl.c' 2023-06-22T08:25:20.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-iconv-h.c' -> './k/coreutils-8.5/gnulib-tests/test-iconv-h.c' 2023-06-22T08:25:20.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mkdir.c' -> './k/coreutils-8.5/gnulib-tests/test-mkdir.c' 2023-06-22T08:25:20.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-posixtm.c' -> './k/coreutils-8.5/gnulib-tests/test-posixtm.c' 2023-06-22T08:25:20.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/wctob.c' -> './k/coreutils-8.5/gnulib-tests/wctob.c' 2023-06-22T08:25:20.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-exclude5.sh' -> './k/coreutils-8.5/gnulib-tests/test-exclude5.sh' 2023-06-22T08:25:20.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-rmdir.h' -> './k/coreutils-8.5/gnulib-tests/test-rmdir.h' 2023-06-22T08:25:20.824 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-symlink.h' -> './k/coreutils-8.5/gnulib-tests/test-symlink.h' 2023-06-22T08:25:20.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-unsetenv.c' -> './k/coreutils-8.5/gnulib-tests/test-unsetenv.c' 2023-06-22T08:25:20.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-exclude4.sh' -> './k/coreutils-8.5/gnulib-tests/test-exclude4.sh' 2023-06-22T08:25:20.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/listen.c' -> './k/coreutils-8.5/gnulib-tests/listen.c' 2023-06-22T08:25:20.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-printf-posix.output' -> './k/coreutils-8.5/gnulib-tests/test-printf-posix.output' 2023-06-22T08:25:20.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbrtowc3.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbrtowc3.sh' 2023-06-22T08:25:20.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-dirent.c' -> './k/coreutils-8.5/gnulib-tests/test-dirent.c' 2023-06-22T08:25:20.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-utimens-common.h' -> './k/coreutils-8.5/gnulib-tests/test-utimens-common.h' 2023-06-22T08:25:20.825 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsinit.c' -> './k/coreutils-8.5/gnulib-tests/test-mbsinit.c' 2023-06-22T08:25:20.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/w32sock.h' -> './k/coreutils-8.5/gnulib-tests/w32sock.h' 2023-06-22T08:25:20.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-printf-posix.h' -> './k/coreutils-8.5/gnulib-tests/test-printf-posix.h' 2023-06-22T08:25:20.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fchdir.c' -> './k/coreutils-8.5/gnulib-tests/test-fchdir.c' 2023-06-22T08:25:20.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/read-file.c' -> './k/coreutils-8.5/gnulib-tests/read-file.c' 2023-06-22T08:25:20.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-ctype.c' -> './k/coreutils-8.5/gnulib-tests/test-ctype.c' 2023-06-22T08:25:20.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/binary-io.h' -> './k/coreutils-8.5/gnulib-tests/binary-io.h' 2023-06-22T08:25:20.826 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/nap.h' -> './k/coreutils-8.5/gnulib-tests/nap.h' 2023-06-22T08:25:20.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/sys_ioctl.in.h' -> './k/coreutils-8.5/gnulib-tests/sys_ioctl.in.h' 2023-06-22T08:25:20.827 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-ftello2.sh' -> './k/coreutils-8.5/gnulib-tests/test-ftello2.sh' 2023-06-22T08:25:20.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-memchr.c' -> './k/coreutils-8.5/gnulib-tests/test-memchr.c' 2023-06-22T08:25:20.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsrtowcs3.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbsrtowcs3.sh' 2023-06-22T08:25:20.865 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fseeko.sh' -> './k/coreutils-8.5/gnulib-tests/test-fseeko.sh' 2023-06-22T08:25:20.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-select-out.sh' -> './k/coreutils-8.5/gnulib-tests/test-select-out.sh' 2023-06-22T08:25:20.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/inet_pton.c' -> './k/coreutils-8.5/gnulib-tests/inet_pton.c' 2023-06-22T08:25:20.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-stdbool.c' -> './k/coreutils-8.5/gnulib-tests/test-stdbool.c' 2023-06-22T08:25:20.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-nanosleep.c' -> './k/coreutils-8.5/gnulib-tests/test-nanosleep.c' 2023-06-22T08:25:20.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/bind.c' -> './k/coreutils-8.5/gnulib-tests/bind.c' 2023-06-22T08:25:20.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/unlinkdir.h' -> './k/coreutils-8.5/gnulib-tests/unlinkdir.h' 2023-06-22T08:25:20.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getline.c' -> './k/coreutils-8.5/gnulib-tests/test-getline.c' 2023-06-22T08:25:20.866 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-md5.c' -> './k/coreutils-8.5/gnulib-tests/test-md5.c' 2023-06-22T08:25:20.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/Makefile.am' -> './k/coreutils-8.5/gnulib-tests/Makefile.am' 2023-06-22T08:25:20.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-select-in.sh' -> './k/coreutils-8.5/gnulib-tests/test-select-in.sh' 2023-06-22T08:25:20.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-uname.c' -> './k/coreutils-8.5/gnulib-tests/test-uname.c' 2023-06-22T08:25:20.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-readlink.h' -> './k/coreutils-8.5/gnulib-tests/test-readlink.h' 2023-06-22T08:25:20.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-string.c' -> './k/coreutils-8.5/gnulib-tests/test-string.c' 2023-06-22T08:25:20.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-argmatch.c' -> './k/coreutils-8.5/gnulib-tests/test-argmatch.c' 2023-06-22T08:25:20.867 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-nl_langinfo.sh' -> './k/coreutils-8.5/gnulib-tests/test-nl_langinfo.sh' 2023-06-22T08:25:20.868 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sys_stat.c' -> './k/coreutils-8.5/gnulib-tests/test-sys_stat.c' 2023-06-22T08:25:20.868 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-version-etc.c' -> './k/coreutils-8.5/gnulib-tests/test-version-etc.c' 2023-06-22T08:25:20.868 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-areadlink.h' -> './k/coreutils-8.5/gnulib-tests/test-areadlink.h' 2023-06-22T08:25:20.868 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-version-etc.sh' -> './k/coreutils-8.5/gnulib-tests/test-version-etc.sh' 2023-06-22T08:25:20.868 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-binary-io.c' -> './k/coreutils-8.5/gnulib-tests/test-binary-io.c' 2023-06-22T08:25:20.868 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-exclude2.sh' -> './k/coreutils-8.5/gnulib-tests/test-exclude2.sh' 2023-06-22T08:25:20.868 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xfprintf-posix.c' -> './k/coreutils-8.5/gnulib-tests/test-xfprintf-posix.c' 2023-06-22T08:25:20.868 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-set-mode-acl.c' -> './k/coreutils-8.5/gnulib-tests/test-set-mode-acl.c' 2023-06-22T08:25:20.869 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-wchar.c' -> './k/coreutils-8.5/gnulib-tests/test-wchar.c' 2023-06-22T08:25:20.869 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-vasprintf.c' -> './k/coreutils-8.5/gnulib-tests/test-vasprintf.c' 2023-06-22T08:25:20.869 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xstrtoimax.sh' -> './k/coreutils-8.5/gnulib-tests/test-xstrtoimax.sh' 2023-06-22T08:25:20.869 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freadseek.sh' -> './k/coreutils-8.5/gnulib-tests/test-freadseek.sh' 2023-06-22T08:25:20.869 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-isblank.c' -> './k/coreutils-8.5/gnulib-tests/test-isblank.c' 2023-06-22T08:25:20.869 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fpending.sh' -> './k/coreutils-8.5/gnulib-tests/test-fpending.sh' 2023-06-22T08:25:20.869 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-lstat.h' -> './k/coreutils-8.5/gnulib-tests/test-lstat.h' 2023-06-22T08:25:20.870 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-chown.c' -> './k/coreutils-8.5/gnulib-tests/test-chown.c' 2023-06-22T08:25:20.870 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xstrtoumax.sh' -> './k/coreutils-8.5/gnulib-tests/test-xstrtoumax.sh' 2023-06-22T08:25:20.870 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/nan.h' -> './k/coreutils-8.5/gnulib-tests/nan.h' 2023-06-22T08:25:20.870 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xprintf-posix.c' -> './k/coreutils-8.5/gnulib-tests/test-xprintf-posix.c' 2023-06-22T08:25:20.870 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-inet_pton.c' -> './k/coreutils-8.5/gnulib-tests/test-inet_pton.c' 2023-06-22T08:25:20.870 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-dup-safer.c' -> './k/coreutils-8.5/gnulib-tests/test-dup-safer.c' 2023-06-22T08:25:20.912 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-closein.sh' -> './k/coreutils-8.5/gnulib-tests/test-closein.sh' 2023-06-22T08:25:20.912 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-canonicalize.c' -> './k/coreutils-8.5/gnulib-tests/test-canonicalize.c' 2023-06-22T08:25:20.912 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-isnanl-nolibm.c' -> './k/coreutils-8.5/gnulib-tests/test-isnanl-nolibm.c' 2023-06-22T08:25:20.912 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xstrtol.c' -> './k/coreutils-8.5/gnulib-tests/test-xstrtol.c' 2023-06-22T08:25:20.912 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-open.c' -> './k/coreutils-8.5/gnulib-tests/test-open.c' 2023-06-22T08:25:20.913 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fprintf-posix.h' -> './k/coreutils-8.5/gnulib-tests/test-fprintf-posix.h' 2023-06-22T08:25:20.913 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbrtowc1.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbrtowc1.sh' 2023-06-22T08:25:20.913 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-errno.c' -> './k/coreutils-8.5/gnulib-tests/test-errno.c' 2023-06-22T08:25:20.913 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-netdb.c' -> './k/coreutils-8.5/gnulib-tests/test-netdb.c' 2023-06-22T08:25:20.913 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-wctype.c' -> './k/coreutils-8.5/gnulib-tests/test-wctype.c' 2023-06-22T08:25:20.914 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-strsignal.c' -> './k/coreutils-8.5/gnulib-tests/test-strsignal.c' 2023-06-22T08:25:20.914 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsstr3.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbsstr3.sh' 2023-06-22T08:25:20.914 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-yesno.c' -> './k/coreutils-8.5/gnulib-tests/test-yesno.c' 2023-06-22T08:25:20.914 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-userspec.c' -> './k/coreutils-8.5/gnulib-tests/test-userspec.c' 2023-06-22T08:25:20.914 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freopen.c' -> './k/coreutils-8.5/gnulib-tests/test-freopen.c' 2023-06-22T08:25:20.914 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sockets.c' -> './k/coreutils-8.5/gnulib-tests/test-sockets.c' 2023-06-22T08:25:20.914 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-link.h' -> './k/coreutils-8.5/gnulib-tests/test-link.h' 2023-06-22T08:25:20.915 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-select-fd.c' -> './k/coreutils-8.5/gnulib-tests/test-select-fd.c' 2023-06-22T08:25:20.915 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xstrtoimax.c' -> './k/coreutils-8.5/gnulib-tests/test-xstrtoimax.c' 2023-06-22T08:25:20.915 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xvasprintf.c' -> './k/coreutils-8.5/gnulib-tests/test-xvasprintf.c' 2023-06-22T08:25:20.915 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-yesno.sh' -> './k/coreutils-8.5/gnulib-tests/test-yesno.sh' 2023-06-22T08:25:20.915 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fflush.c' -> './k/coreutils-8.5/gnulib-tests/test-fflush.c' 2023-06-22T08:25:20.915 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-vc-list-files-cvs.sh' -> './k/coreutils-8.5/gnulib-tests/test-vc-list-files-cvs.sh' 2023-06-22T08:25:20.916 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbscasecmp.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbscasecmp.sh' 2023-06-22T08:25:20.916 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-lseek.sh' -> './k/coreutils-8.5/gnulib-tests/test-lseek.sh' 2023-06-22T08:25:20.916 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/usleep.c' -> './k/coreutils-8.5/gnulib-tests/usleep.c' 2023-06-22T08:25:20.916 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sleep.c' -> './k/coreutils-8.5/gnulib-tests/test-sleep.c' 2023-06-22T08:25:20.916 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-link.c' -> './k/coreutils-8.5/gnulib-tests/test-link.c' 2023-06-22T08:25:20.916 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-perror.c' -> './k/coreutils-8.5/gnulib-tests/test-perror.c' 2023-06-22T08:25:20.916 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsrtowcs.c' -> './k/coreutils-8.5/gnulib-tests/test-mbsrtowcs.c' 2023-06-22T08:25:20.917 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xalloc-die.sh' -> './k/coreutils-8.5/gnulib-tests/test-xalloc-die.sh' 2023-06-22T08:25:20.917 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-i-ring.c' -> './k/coreutils-8.5/gnulib-tests/test-i-ring.c' 2023-06-22T08:25:20.917 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-priv-set.c' -> './k/coreutils-8.5/gnulib-tests/test-priv-set.c' 2023-06-22T08:25:20.917 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-btowc.c' -> './k/coreutils-8.5/gnulib-tests/test-btowc.c' 2023-06-22T08:25:20.917 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sys_select.c' -> './k/coreutils-8.5/gnulib-tests/test-sys_select.c' 2023-06-22T08:25:20.917 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbrtowc.c' -> './k/coreutils-8.5/gnulib-tests/test-mbrtowc.c' 2023-06-22T08:25:20.917 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-strerror.c' -> './k/coreutils-8.5/gnulib-tests/test-strerror.c' 2023-06-22T08:25:20.917 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-update-copyright.sh' -> './k/coreutils-8.5/gnulib-tests/test-update-copyright.sh' 2023-06-22T08:25:20.918 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-btowc2.sh' -> './k/coreutils-8.5/gnulib-tests/test-btowc2.sh' 2023-06-22T08:25:23.238 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getcwd.c' -> './k/coreutils-8.5/gnulib-tests/test-getcwd.c' 2023-06-22T08:25:23.239 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/socket.c' -> './k/coreutils-8.5/gnulib-tests/socket.c' 2023-06-22T08:25:23.239 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-set-mode-acl.sh' -> './k/coreutils-8.5/gnulib-tests/test-set-mode-acl.sh' 2023-06-22T08:25:23.239 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-linkat.c' -> './k/coreutils-8.5/gnulib-tests/test-linkat.c' 2023-06-22T08:25:23.239 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/perror.c' -> './k/coreutils-8.5/gnulib-tests/perror.c' 2023-06-22T08:25:23.240 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-wcrtomb.c' -> './k/coreutils-8.5/gnulib-tests/test-wcrtomb.c' 2023-06-22T08:25:23.240 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-vasnprintf.c' -> './k/coreutils-8.5/gnulib-tests/test-vasnprintf.c' 2023-06-22T08:25:23.240 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-memrchr.c' -> './k/coreutils-8.5/gnulib-tests/test-memrchr.c' 2023-06-22T08:25:23.240 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-ftello.c' -> './k/coreutils-8.5/gnulib-tests/test-ftello.c' 2023-06-22T08:25:23.241 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-chown.h' -> './k/coreutils-8.5/gnulib-tests/test-chown.h' 2023-06-22T08:25:23.241 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-stat.h' -> './k/coreutils-8.5/gnulib-tests/test-stat.h' 2023-06-22T08:25:23.241 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/Makefile.in' -> './k/coreutils-8.5/gnulib-tests/Makefile.in' 2023-06-22T08:25:23.241 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getdelim.c' -> './k/coreutils-8.5/gnulib-tests/test-getdelim.c' 2023-06-22T08:25:23.241 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-vprintf-posix.sh' -> './k/coreutils-8.5/gnulib-tests/test-vprintf-posix.sh' 2023-06-22T08:25:23.241 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fseeko2.sh' -> './k/coreutils-8.5/gnulib-tests/test-fseeko2.sh' 2023-06-22T08:25:23.241 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-c-strcase.sh' -> './k/coreutils-8.5/gnulib-tests/test-c-strcase.sh' 2023-06-22T08:25:23.242 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsalign.c' -> './k/coreutils-8.5/gnulib-tests/test-mbsalign.c' 2023-06-22T08:25:23.242 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fcntl-h.c' -> './k/coreutils-8.5/gnulib-tests/test-fcntl-h.c' 2023-06-22T08:25:23.242 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-vprintf-posix.c' -> './k/coreutils-8.5/gnulib-tests/test-vprintf-posix.c' 2023-06-22T08:25:23.242 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-rename.c' -> './k/coreutils-8.5/gnulib-tests/test-rename.c' 2023-06-22T08:25:23.242 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-lstat.c' -> './k/coreutils-8.5/gnulib-tests/test-lstat.c' 2023-06-22T08:25:23.242 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-malloca.c' -> './k/coreutils-8.5/gnulib-tests/test-malloca.c' 2023-06-22T08:25:23.242 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xstrtoul.c' -> './k/coreutils-8.5/gnulib-tests/test-xstrtoul.c' 2023-06-22T08:25:23.242 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-perror.sh' -> './k/coreutils-8.5/gnulib-tests/test-perror.sh' 2023-06-22T08:25:23.243 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-vasprintf-posix.c' -> './k/coreutils-8.5/gnulib-tests/test-vasprintf-posix.c' 2023-06-22T08:25:23.243 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-frexp.c' -> './k/coreutils-8.5/gnulib-tests/test-frexp.c' 2023-06-22T08:25:23.243 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-environ.c' -> './k/coreutils-8.5/gnulib-tests/test-environ.c' 2023-06-22T08:25:23.243 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-stdint.c' -> './k/coreutils-8.5/gnulib-tests/test-stdint.c' 2023-06-22T08:25:23.243 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-isnanl.h' -> './k/coreutils-8.5/gnulib-tests/test-isnanl.h' 2023-06-22T08:25:23.243 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fcntl-safer.c' -> './k/coreutils-8.5/gnulib-tests/test-fcntl-safer.c' 2023-06-22T08:25:23.243 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-locale.c' -> './k/coreutils-8.5/gnulib-tests/test-locale.c' 2023-06-22T08:25:23.243 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbrtowc4.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbrtowc4.sh' 2023-06-22T08:25:23.244 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-argv-iter.c' -> './k/coreutils-8.5/gnulib-tests/test-argv-iter.c' 2023-06-22T08:25:23.244 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-utimens.h' -> './k/coreutils-8.5/gnulib-tests/test-utimens.h' 2023-06-22T08:25:23.244 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xalloc-die.c' -> './k/coreutils-8.5/gnulib-tests/test-xalloc-die.c' 2023-06-22T08:25:23.244 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fcntl.c' -> './k/coreutils-8.5/gnulib-tests/test-fcntl.c' 2023-06-22T08:25:23.244 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-alloca-opt.c' -> './k/coreutils-8.5/gnulib-tests/test-alloca-opt.c' 2023-06-22T08:25:23.244 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-dirname.c' -> './k/coreutils-8.5/gnulib-tests/test-dirname.c' 2023-06-22T08:25:23.244 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-setenv.c' -> './k/coreutils-8.5/gnulib-tests/test-setenv.c' 2023-06-22T08:25:23.288 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-printf-frexpl.c' -> './k/coreutils-8.5/gnulib-tests/test-printf-frexpl.c' 2023-06-22T08:25:23.288 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-atexit.sh' -> './k/coreutils-8.5/gnulib-tests/test-atexit.sh' 2023-06-22T08:25:23.288 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-stddef.c' -> './k/coreutils-8.5/gnulib-tests/test-stddef.c' 2023-06-22T08:25:23.288 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-c-ctype.c' -> './k/coreutils-8.5/gnulib-tests/test-c-ctype.c' 2023-06-22T08:25:23.289 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xstrtol.sh' -> './k/coreutils-8.5/gnulib-tests/test-xstrtol.sh' 2023-06-22T08:25:23.289 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getopt_long.h' -> './k/coreutils-8.5/gnulib-tests/test-getopt_long.h' 2023-06-22T08:25:23.289 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-quotearg.sh' -> './k/coreutils-8.5/gnulib-tests/test-quotearg.sh' 2023-06-22T08:25:23.289 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-langinfo.c' -> './k/coreutils-8.5/gnulib-tests/test-langinfo.c' 2023-06-22T08:25:23.289 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-inttypes.c' -> './k/coreutils-8.5/gnulib-tests/test-inttypes.c' 2023-06-22T08:25:23.289 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-filevercmp.c' -> './k/coreutils-8.5/gnulib-tests/test-filevercmp.c' 2023-06-22T08:25:23.289 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-rmdir.c' -> './k/coreutils-8.5/gnulib-tests/test-rmdir.c' 2023-06-22T08:25:23.289 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-futimens.h' -> './k/coreutils-8.5/gnulib-tests/test-futimens.h' 2023-06-22T08:25:23.290 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/setsockopt.c' -> './k/coreutils-8.5/gnulib-tests/setsockopt.c' 2023-06-22T08:25:23.290 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-exclude1.sh' -> './k/coreutils-8.5/gnulib-tests/test-exclude1.sh' 2023-06-22T08:25:23.290 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freopen-safer.c' -> './k/coreutils-8.5/gnulib-tests/test-freopen-safer.c' 2023-06-22T08:25:23.290 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getaddrinfo.c' -> './k/coreutils-8.5/gnulib-tests/test-getaddrinfo.c' 2023-06-22T08:25:23.290 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-iconv.c' -> './k/coreutils-8.5/gnulib-tests/test-iconv.c' 2023-06-22T08:25:23.290 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/zerosize-ptr.h' -> './k/coreutils-8.5/gnulib-tests/zerosize-ptr.h' 2023-06-22T08:25:23.290 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-lseek.c' -> './k/coreutils-8.5/gnulib-tests/test-lseek.c' 2023-06-22T08:25:23.290 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-usleep.c' -> './k/coreutils-8.5/gnulib-tests/test-usleep.c' 2023-06-22T08:25:23.291 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/gnulib.mk' -> './k/coreutils-8.5/gnulib-tests/gnulib.mk' 2023-06-22T08:25:23.291 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-unlinkat.c' -> './k/coreutils-8.5/gnulib-tests/test-unlinkat.c' 2023-06-22T08:25:23.291 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-quotearg.c' -> './k/coreutils-8.5/gnulib-tests/test-quotearg.c' 2023-06-22T08:25:23.291 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-base64.c' -> './k/coreutils-8.5/gnulib-tests/test-base64.c' 2023-06-22T08:25:23.291 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-frexpl.c' -> './k/coreutils-8.5/gnulib-tests/test-frexpl.c' 2023-06-22T08:25:23.291 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-gethostname.c' -> './k/coreutils-8.5/gnulib-tests/test-gethostname.c' 2023-06-22T08:25:23.291 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mkdir.h' -> './k/coreutils-8.5/gnulib-tests/test-mkdir.h' 2023-06-22T08:25:23.292 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-binary-io.sh' -> './k/coreutils-8.5/gnulib-tests/test-binary-io.sh' 2023-06-22T08:25:23.292 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sys_ioctl.c' -> './k/coreutils-8.5/gnulib-tests/test-sys_ioctl.c' 2023-06-22T08:25:23.292 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-time.c' -> './k/coreutils-8.5/gnulib-tests/test-time.c' 2023-06-22T08:25:23.293 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/signature.h' -> './k/coreutils-8.5/gnulib-tests/signature.h' 2023-06-22T08:25:23.293 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-isnanf.h' -> './k/coreutils-8.5/gnulib-tests/test-isnanf.h' 2023-06-22T08:25:23.293 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-isnanf-nolibm.c' -> './k/coreutils-8.5/gnulib-tests/test-isnanf-nolibm.c' 2023-06-22T08:25:23.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-wcwidth.c' -> './k/coreutils-8.5/gnulib-tests/test-wcwidth.c' 2023-06-22T08:25:23.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freadseek.c' -> './k/coreutils-8.5/gnulib-tests/test-freadseek.c' 2023-06-22T08:25:23.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-areadlink-with-size.c' -> './k/coreutils-8.5/gnulib-tests/test-areadlink-with-size.c' 2023-06-22T08:25:23.294 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-select-stdin.c' -> './k/coreutils-8.5/gnulib-tests/test-select-stdin.c' 2023-06-22T08:25:23.295 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsstr1.c' -> './k/coreutils-8.5/gnulib-tests/test-mbsstr1.c' 2023-06-22T08:25:23.295 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-exclude3.sh' -> './k/coreutils-8.5/gnulib-tests/test-exclude3.sh' 2023-06-22T08:25:23.336 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fopen.c' -> './k/coreutils-8.5/gnulib-tests/test-fopen.c' 2023-06-22T08:25:23.336 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-lock.c' -> './k/coreutils-8.5/gnulib-tests/test-lock.c' 2023-06-22T08:25:23.336 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-netinet_in.c' -> './k/coreutils-8.5/gnulib-tests/test-netinet_in.c' 2023-06-22T08:25:23.336 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-bitrotate.c' -> './k/coreutils-8.5/gnulib-tests/test-bitrotate.c' 2023-06-22T08:25:23.337 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-c-strcasecmp.c' -> './k/coreutils-8.5/gnulib-tests/test-c-strcasecmp.c' 2023-06-22T08:25:23.337 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsstr3.c' -> './k/coreutils-8.5/gnulib-tests/test-mbsstr3.c' 2023-06-22T08:25:23.337 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-hash.c' -> './k/coreutils-8.5/gnulib-tests/test-hash.c' 2023-06-22T08:25:23.337 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-copy-acl.c' -> './k/coreutils-8.5/gnulib-tests/test-copy-acl.c' 2023-06-22T08:25:23.337 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sigaction.c' -> './k/coreutils-8.5/gnulib-tests/test-sigaction.c' 2023-06-22T08:25:23.337 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xstrtoumax.c' -> './k/coreutils-8.5/gnulib-tests/test-xstrtoumax.c' 2023-06-22T08:25:23.337 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fseterr.c' -> './k/coreutils-8.5/gnulib-tests/test-fseterr.c' 2023-06-22T08:25:23.337 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-areadlink.c' -> './k/coreutils-8.5/gnulib-tests/test-areadlink.c' 2023-06-22T08:25:23.338 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-signbit.c' -> './k/coreutils-8.5/gnulib-tests/test-signbit.c' 2023-06-22T08:25:23.338 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-btowc1.sh' -> './k/coreutils-8.5/gnulib-tests/test-btowc1.sh' 2023-06-22T08:25:23.338 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-strtod.c' -> './k/coreutils-8.5/gnulib-tests/test-strtod.c' 2023-06-22T08:25:23.338 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-atexit.c' -> './k/coreutils-8.5/gnulib-tests/test-atexit.c' 2023-06-22T08:25:23.338 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getdate.c' -> './k/coreutils-8.5/gnulib-tests/test-getdate.c' 2023-06-22T08:25:23.338 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fpurge.c' -> './k/coreutils-8.5/gnulib-tests/test-fpurge.c' 2023-06-22T08:25:23.338 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-select.c' -> './k/coreutils-8.5/gnulib-tests/test-select.c' 2023-06-22T08:25:23.338 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-tls.c' -> './k/coreutils-8.5/gnulib-tests/test-tls.c' 2023-06-22T08:25:23.339 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/sleep.c' -> './k/coreutils-8.5/gnulib-tests/sleep.c' 2023-06-22T08:25:23.339 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbrtowc2.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbrtowc2.sh' 2023-06-22T08:25:23.339 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-stat.c' -> './k/coreutils-8.5/gnulib-tests/test-stat.c' 2023-06-22T08:25:23.339 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freadptr.sh' -> './k/coreutils-8.5/gnulib-tests/test-freadptr.sh' 2023-06-22T08:25:23.339 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/accept.c' -> './k/coreutils-8.5/gnulib-tests/accept.c' 2023-06-22T08:25:23.339 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-unistd.c' -> './k/coreutils-8.5/gnulib-tests/test-unistd.c' 2023-06-22T08:25:23.339 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fopen.h' -> './k/coreutils-8.5/gnulib-tests/test-fopen.h' 2023-06-22T08:25:23.340 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-math.c' -> './k/coreutils-8.5/gnulib-tests/test-math.c' 2023-06-22T08:25:23.340 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-printf-frexp.c' -> './k/coreutils-8.5/gnulib-tests/test-printf-frexp.c' 2023-06-22T08:25:23.340 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-vfprintf-posix.sh' -> './k/coreutils-8.5/gnulib-tests/test-vfprintf-posix.sh' 2023-06-22T08:25:23.340 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-exclude.c' -> './k/coreutils-8.5/gnulib-tests/test-exclude.c' 2023-06-22T08:25:23.340 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freading.c' -> './k/coreutils-8.5/gnulib-tests/test-freading.c' 2023-06-22T08:25:23.340 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getdtablesize.c' -> './k/coreutils-8.5/gnulib-tests/test-getdtablesize.c' 2023-06-22T08:25:23.340 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/unlinkdir.c' -> './k/coreutils-8.5/gnulib-tests/unlinkdir.c' 2023-06-22T08:25:23.340 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fflush2.c' -> './k/coreutils-8.5/gnulib-tests/test-fflush2.c' 2023-06-22T08:25:23.341 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-stat-time.c' -> './k/coreutils-8.5/gnulib-tests/test-stat-time.c' 2023-06-22T08:25:23.341 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-inet_ntop.c' -> './k/coreutils-8.5/gnulib-tests/test-inet_ntop.c' 2023-06-22T08:25:23.341 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-vc-list-files-git.sh' -> './k/coreutils-8.5/gnulib-tests/test-vc-list-files-git.sh' 2023-06-22T08:25:23.341 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getopt.c' -> './k/coreutils-8.5/gnulib-tests/test-getopt.c' 2023-06-22T08:25:23.341 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-alignof.c' -> './k/coreutils-8.5/gnulib-tests/test-alignof.c' 2023-06-22T08:25:23.385 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/macros.h' -> './k/coreutils-8.5/gnulib-tests/macros.h' 2023-06-22T08:25:23.386 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-symlinkat.c' -> './k/coreutils-8.5/gnulib-tests/test-symlinkat.c' 2023-06-22T08:25:23.386 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getgroups.c' -> './k/coreutils-8.5/gnulib-tests/test-getgroups.c' 2023-06-22T08:25:23.386 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-xprintf-posix.sh' -> './k/coreutils-8.5/gnulib-tests/test-xprintf-posix.sh' 2023-06-22T08:25:23.386 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fdopendir.c' -> './k/coreutils-8.5/gnulib-tests/test-fdopendir.c' 2023-06-22T08:25:23.386 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sys_socket.c' -> './k/coreutils-8.5/gnulib-tests/test-sys_socket.c' 2023-06-22T08:25:23.386 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbscasecmp.c' -> './k/coreutils-8.5/gnulib-tests/test-mbscasecmp.c' 2023-06-22T08:25:23.386 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-getopt.h' -> './k/coreutils-8.5/gnulib-tests/test-getopt.h' 2023-06-22T08:25:23.387 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsrtowcs1.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbsrtowcs1.sh' 2023-06-22T08:25:23.387 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsinit.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbsinit.sh' 2023-06-22T08:25:23.387 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-gettimeofday.c' -> './k/coreutils-8.5/gnulib-tests/test-gettimeofday.c' 2023-06-22T08:25:23.387 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsstr2.c' -> './k/coreutils-8.5/gnulib-tests/test-mbsstr2.c' 2023-06-22T08:25:23.387 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/init.sh' -> './k/coreutils-8.5/gnulib-tests/init.sh' 2023-06-22T08:25:23.387 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-symlink.c' -> './k/coreutils-8.5/gnulib-tests/test-symlink.c' 2023-06-22T08:25:23.387 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-utimens.c' -> './k/coreutils-8.5/gnulib-tests/test-utimens.c' 2023-06-22T08:25:23.388 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-readlink.c' -> './k/coreutils-8.5/gnulib-tests/test-readlink.c' 2023-06-22T08:25:23.388 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-lchown.h' -> './k/coreutils-8.5/gnulib-tests/test-lchown.h' 2023-06-22T08:25:23.388 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-c-strncasecmp.c' -> './k/coreutils-8.5/gnulib-tests/test-c-strncasecmp.c' 2023-06-22T08:25:23.388 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-lutimens.h' -> './k/coreutils-8.5/gnulib-tests/test-lutimens.h' 2023-06-22T08:25:23.388 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-copy-acl.sh' -> './k/coreutils-8.5/gnulib-tests/test-copy-acl.sh' 2023-06-22T08:25:23.388 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-exclude6.sh' -> './k/coreutils-8.5/gnulib-tests/test-exclude6.sh' 2023-06-22T08:25:23.388 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freadptr2.c' -> './k/coreutils-8.5/gnulib-tests/test-freadptr2.c' 2023-06-22T08:25:23.388 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fflush2.sh' -> './k/coreutils-8.5/gnulib-tests/test-fflush2.sh' 2023-06-22T08:25:23.389 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sys_utsname.c' -> './k/coreutils-8.5/gnulib-tests/test-sys_utsname.c' 2023-06-22T08:25:23.389 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsrtowcs2.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbsrtowcs2.sh' 2023-06-22T08:25:23.389 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-file-has-acl.c' -> './k/coreutils-8.5/gnulib-tests/test-file-has-acl.c' 2023-06-22T08:25:23.389 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-signal.c' -> './k/coreutils-8.5/gnulib-tests/test-signal.c' 2023-06-22T08:25:23.389 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sha1.c' -> './k/coreutils-8.5/gnulib-tests/test-sha1.c' 2023-06-22T08:25:23.389 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freadptr2.sh' -> './k/coreutils-8.5/gnulib-tests/test-freadptr2.sh' 2023-06-22T08:25:23.389 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-areadlinkat.c' -> './k/coreutils-8.5/gnulib-tests/test-areadlinkat.c' 2023-06-22T08:25:23.390 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-unlink.h' -> './k/coreutils-8.5/gnulib-tests/test-unlink.h' 2023-06-22T08:25:23.390 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-dup2.c' -> './k/coreutils-8.5/gnulib-tests/test-dup2.c' 2023-06-22T08:25:23.390 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-freadptr.c' -> './k/coreutils-8.5/gnulib-tests/test-freadptr.c' 2023-06-22T08:25:23.390 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-file-has-acl.sh' -> './k/coreutils-8.5/gnulib-tests/test-file-has-acl.sh' 2023-06-22T08:25:23.390 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-u64.c' -> './k/coreutils-8.5/gnulib-tests/test-u64.c' 2023-06-22T08:25:23.390 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-striconv.c' -> './k/coreutils-8.5/gnulib-tests/test-striconv.c' 2023-06-22T08:25:23.390 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-sameacls.c' -> './k/coreutils-8.5/gnulib-tests/test-sameacls.c' 2023-06-22T08:25:23.390 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-remove.c' -> './k/coreutils-8.5/gnulib-tests/test-remove.c' 2023-06-22T08:25:23.391 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fsync.c' -> './k/coreutils-8.5/gnulib-tests/test-fsync.c' 2023-06-22T08:25:23.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-mbsrtowcs4.sh' -> './k/coreutils-8.5/gnulib-tests/test-mbsrtowcs4.sh' 2023-06-22T08:25:23.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-isnand.h' -> './k/coreutils-8.5/gnulib-tests/test-isnand.h' 2023-06-22T08:25:23.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/glthread' -> './k/coreutils-8.5/gnulib-tests/glthread' 2023-06-22T08:25:23.551 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/glthread/yield.h' -> './k/coreutils-8.5/gnulib-tests/glthread/yield.h' 2023-06-22T08:25:23.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/glthread/thread.h' -> './k/coreutils-8.5/gnulib-tests/glthread/thread.h' 2023-06-22T08:25:23.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/glthread/thread.c' -> './k/coreutils-8.5/gnulib-tests/glthread/thread.c' 2023-06-22T08:25:23.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/unistr' -> './k/coreutils-8.5/gnulib-tests/unistr' 2023-06-22T08:25:23.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/unistr/test-u8-uctomb.c' -> './k/coreutils-8.5/gnulib-tests/unistr/test-u8-uctomb.c' 2023-06-22T08:25:23.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/uniwidth' -> './k/coreutils-8.5/gnulib-tests/uniwidth' 2023-06-22T08:25:23.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/uniwidth/test-uc_width.c' -> './k/coreutils-8.5/gnulib-tests/uniwidth/test-uc_width.c' 2023-06-22T08:25:23.552 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/uniwidth/test-uc_width2.sh' -> './k/coreutils-8.5/gnulib-tests/uniwidth/test-uc_width2.sh' 2023-06-22T08:25:23.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/uniwidth/test-uc_width2.c' -> './k/coreutils-8.5/gnulib-tests/uniwidth/test-uc_width2.c' 2023-06-22T08:25:23.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/locale' -> './k/coreutils-8.5/gnulib-tests/locale' 2023-06-22T08:25:23.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/locale/fr' -> './k/coreutils-8.5/gnulib-tests/locale/fr' 2023-06-22T08:25:23.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES' -> './k/coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES' 2023-06-22T08:25:23.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/test-quotearg.po' -> './k/coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/test-quotearg.po' 2023-06-22T08:25:23.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/test-quotearg.mo' -> './k/coreutils-8.5/gnulib-tests/locale/fr/LC_MESSAGES/test-quotearg.mo' 2023-06-22T08:25:23.553 INFO:tasks.workunit.client.0.smithi023.stdout:'.snap/k/coreutils-8.5/gnulib-tests/test-fpending.c' -> './k/coreutils-8.5/gnulib-tests/test-fpending.c' 2023-06-22T08:25:23.554 INFO:tasks.workunit.client.0.smithi023.stderr:+ rmdir .snap/k 2023-06-22T08:25:23.585 INFO:tasks.workunit.client.0.smithi023.stderr:+ rm -rv k 2023-06-22T08:25:23.831 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils_8.5.orig.tar.gz' 2023-06-22T08:25:23.832 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_prohibit_strcmp' 2023-06-22T08:25:23.832 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_GPL_version' 2023-06-22T08:25:23.832 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.prev-version' 2023-06-22T08:25:23.832 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_program_name' 2023-06-22T08:25:23.832 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/expr.x' 2023-06-22T08:25:23.832 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/hostid.1' 2023-06-22T08:25:23.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/seq.x' 2023-06-22T08:25:23.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/yes.1' 2023-06-22T08:25:23.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tee.x' 2023-06-22T08:25:23.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/cat.x' 2023-06-22T08:25:23.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/true.x' 2023-06-22T08:25:23.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/dirname.1' 2023-06-22T08:25:23.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tail.1' 2023-06-22T08:25:23.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/timeout.1' 2023-06-22T08:25:23.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/uname.1' 2023-06-22T08:25:23.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/unlink.1' 2023-06-22T08:25:23.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/expr.1' 2023-06-22T08:25:23.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/printf.1' 2023-06-22T08:25:23.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/logname.x' 2023-06-22T08:25:23.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/expand.1' 2023-06-22T08:25:23.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sum.x' 2023-06-22T08:25:23.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/md5sum.x' 2023-06-22T08:25:23.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/env.x' 2023-06-22T08:25:23.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/nice.x' 2023-06-22T08:25:23.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/date.1' 2023-06-22T08:25:23.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tr.1' 2023-06-22T08:25:23.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/join.x' 2023-06-22T08:25:23.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/stty.1' 2023-06-22T08:25:23.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/seq.1' 2023-06-22T08:25:23.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/pinky.x' 2023-06-22T08:25:23.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/hostname.x' 2023-06-22T08:25:23.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/install.1' 2023-06-22T08:25:23.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sha512sum.x' 2023-06-22T08:25:23.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/install.x' 2023-06-22T08:25:23.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tail.x' 2023-06-22T08:25:23.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/dircolors.1' 2023-06-22T08:25:23.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/hostid.x' 2023-06-22T08:25:23.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/md5sum.1' 2023-06-22T08:25:23.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/split.x' 2023-06-22T08:25:23.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/mv.x' 2023-06-22T08:25:23.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/users.x' 2023-06-22T08:25:23.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/od.1' 2023-06-22T08:25:23.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/ln.x' 2023-06-22T08:25:23.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/kill.x' 2023-06-22T08:25:23.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/date.x' 2023-06-22T08:25:23.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/chown.1' 2023-06-22T08:25:23.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/paste.x' 2023-06-22T08:25:23.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/expand.x' 2023-06-22T08:25:23.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/rm.x' 2023-06-22T08:25:23.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sha384sum.1' 2023-06-22T08:25:23.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sleep.1' 2023-06-22T08:25:23.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sha512sum.1' 2023-06-22T08:25:23.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/shuf.x' 2023-06-22T08:25:23.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/join.1' 2023-06-22T08:25:23.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/cp.x' 2023-06-22T08:25:23.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/mktemp.1' 2023-06-22T08:25:23.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tsort.x' 2023-06-22T08:25:23.840 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/factor.1' 2023-06-22T08:25:23.840 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/ptx.x' 2023-06-22T08:25:23.840 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/who.x' 2023-06-22T08:25:23.840 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/touch.x' 2023-06-22T08:25:23.840 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/logname.1' 2023-06-22T08:25:23.840 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/dd.1' 2023-06-22T08:25:23.840 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sleep.x' 2023-06-22T08:25:23.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/base64.1' 2023-06-22T08:25:23.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/vdir.x' 2023-06-22T08:25:23.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/printenv.x' 2023-06-22T08:25:23.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sha224sum.1' 2023-06-22T08:25:23.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/chcon.1' 2023-06-22T08:25:23.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/whoami.1' 2023-06-22T08:25:23.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/unlink.x' 2023-06-22T08:25:23.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/runcon.1' 2023-06-22T08:25:23.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tty.x' 2023-06-22T08:25:23.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/id.1' 2023-06-22T08:25:23.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/printf.x' 2023-06-22T08:25:23.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/rmdir.1' 2023-06-22T08:25:23.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/od.x' 2023-06-22T08:25:23.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/rmdir.x' 2023-06-22T08:25:23.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/pathchk.x' 2023-06-22T08:25:23.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sort.1' 2023-06-22T08:25:23.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/chown.x' 2023-06-22T08:25:23.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/df.x' 2023-06-22T08:25:23.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/mkfifo.x' 2023-06-22T08:25:23.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/test.x' 2023-06-22T08:25:23.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/vdir.1' 2023-06-22T08:25:23.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/mknod.x' 2023-06-22T08:25:23.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/who.1' 2023-06-22T08:25:23.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/arch.x' 2023-06-22T08:25:23.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/stty.x' 2023-06-22T08:25:23.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/pr.1' 2023-06-22T08:25:23.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/test.1' 2023-06-22T08:25:23.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/uniq.x' 2023-06-22T08:25:23.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/stat.x' 2023-06-22T08:25:23.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/paste.1' 2023-06-22T08:25:23.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/cksum.x' 2023-06-22T08:25:23.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/ptx.1' 2023-06-22T08:25:23.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/Makefile.in' 2023-06-22T08:25:23.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/dircolors.x' 2023-06-22T08:25:23.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/uptime.x' 2023-06-22T08:25:23.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/chgrp.x' 2023-06-22T08:25:23.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/basename.1' 2023-06-22T08:25:23.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/mktemp.x' 2023-06-22T08:25:23.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sum.1' 2023-06-22T08:25:23.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sha1sum.x' 2023-06-22T08:25:23.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/groups.x' 2023-06-22T08:25:23.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/mknod.1' 2023-06-22T08:25:24.224 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/help2man' 2023-06-22T08:25:24.224 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/uname.x' 2023-06-22T08:25:24.225 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/ls.1' 2023-06-22T08:25:24.225 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/du.1' 2023-06-22T08:25:24.225 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/timeout.x' 2023-06-22T08:25:24.225 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/groups.1' 2023-06-22T08:25:24.225 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/dir.1' 2023-06-22T08:25:24.225 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sync.x' 2023-06-22T08:25:24.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/cat.1' 2023-06-22T08:25:24.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/comm.x' 2023-06-22T08:25:24.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sha256sum.1' 2023-06-22T08:25:24.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/comm.1' 2023-06-22T08:25:24.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/chroot.1' 2023-06-22T08:25:24.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/csplit.x' 2023-06-22T08:25:24.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sync.1' 2023-06-22T08:25:24.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sort.x' 2023-06-22T08:25:24.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/chmod.x' 2023-06-22T08:25:24.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/Makefile.am' 2023-06-22T08:25:24.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/readlink.1' 2023-06-22T08:25:24.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/uptime.1' 2023-06-22T08:25:24.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/pwd.x' 2023-06-22T08:25:24.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tac.1' 2023-06-22T08:25:24.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/dir.x' 2023-06-22T08:25:24.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/mkdir.x' 2023-06-22T08:25:24.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/split.1' 2023-06-22T08:25:24.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/chroot.x' 2023-06-22T08:25:24.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/false.x' 2023-06-22T08:25:24.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/df.1' 2023-06-22T08:25:24.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tac.x' 2023-06-22T08:25:24.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/du.x' 2023-06-22T08:25:24.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/rm.1' 2023-06-22T08:25:24.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/head.x' 2023-06-22T08:25:24.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/mv.1' 2023-06-22T08:25:24.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sha384sum.x' 2023-06-22T08:25:24.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/echo.1' 2023-06-22T08:25:24.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/su.x' 2023-06-22T08:25:24.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/pwd.1' 2023-06-22T08:25:24.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/shred.1' 2023-06-22T08:25:24.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/fold.x' 2023-06-22T08:25:24.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/touch.1' 2023-06-22T08:25:24.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/printenv.1' 2023-06-22T08:25:24.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/uniq.1' 2023-06-22T08:25:24.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/id.x' 2023-06-22T08:25:24.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/chmod.1' 2023-06-22T08:25:24.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/link.x' 2023-06-22T08:25:24.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/dirname.x' 2023-06-22T08:25:24.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/nproc.1' 2023-06-22T08:25:24.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/nice.1' 2023-06-22T08:25:24.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/shred.x' 2023-06-22T08:25:24.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/fold.1' 2023-06-22T08:25:24.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sha224sum.x' 2023-06-22T08:25:24.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/pinky.1' 2023-06-22T08:25:24.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/runcon.x' 2023-06-22T08:25:24.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/unexpand.x' 2023-06-22T08:25:24.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/readlink.x' 2023-06-22T08:25:24.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/stdbuf.x' 2023-06-22T08:25:24.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/pr.x' 2023-06-22T08:25:24.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/whoami.x' 2023-06-22T08:25:24.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/nl.x' 2023-06-22T08:25:24.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/base64.x' 2023-06-22T08:25:24.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tsort.1' 2023-06-22T08:25:24.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tty.1' 2023-06-22T08:25:24.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/kill.1' 2023-06-22T08:25:24.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/shuf.1' 2023-06-22T08:25:24.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/csplit.1' 2023-06-22T08:25:24.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/users.1' 2023-06-22T08:25:24.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/basename.x' 2023-06-22T08:25:24.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/stat.1' 2023-06-22T08:25:24.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sha1sum.1' 2023-06-22T08:25:24.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/fmt.1' 2023-06-22T08:25:24.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/unexpand.1' 2023-06-22T08:25:24.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/mkfifo.1' 2023-06-22T08:25:24.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/truncate.1' 2023-06-22T08:25:24.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/wc.1' 2023-06-22T08:25:24.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/dd.x' 2023-06-22T08:25:24.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/cp.1' 2023-06-22T08:25:24.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/chcon.x' 2023-06-22T08:25:24.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/sha256sum.x' 2023-06-22T08:25:24.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/nproc.x' 2023-06-22T08:25:24.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/wc.x' 2023-06-22T08:25:24.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/yes.x' 2023-06-22T08:25:24.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/true.1' 2023-06-22T08:25:24.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/mkdir.1' 2023-06-22T08:25:24.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/factor.x' 2023-06-22T08:25:24.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/cksum.1' 2023-06-22T08:25:24.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/pathchk.1' 2023-06-22T08:25:24.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/nohup.x' 2023-06-22T08:25:24.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/echo.x' 2023-06-22T08:25:24.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tee.1' 2023-06-22T08:25:24.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/nohup.1' 2023-06-22T08:25:24.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/false.1' 2023-06-22T08:25:24.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/chgrp.1' 2023-06-22T08:25:24.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/cut.1' 2023-06-22T08:25:24.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/cut.x' 2023-06-22T08:25:24.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/fmt.x' 2023-06-22T08:25:24.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/stdbuf.1' 2023-06-22T08:25:24.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/nl.1' 2023-06-22T08:25:24.238 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/env.1' 2023-06-22T08:25:24.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/truncate.x' 2023-06-22T08:25:24.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/ls.x' 2023-06-22T08:25:24.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/ln.1' 2023-06-22T08:25:24.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/link.1' 2023-06-22T08:25:24.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/head.1' 2023-06-22T08:25:24.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/man/tr.x' 2023-06-22T08:25:24.239 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/man' 2023-06-22T08:25:24.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/TODO' 2023-06-22T08:25:24.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_prohibit_magic_number_exit' 2023-06-22T08:25:24.240 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/who.c' 2023-06-22T08:25:24.380 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/chown-core.h' 2023-06-22T08:25:24.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/expr.c' 2023-06-22T08:25:24.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/ls-ls.c' 2023-06-22T08:25:24.381 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/cp-hash.h' 2023-06-22T08:25:24.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/chgrp.c' 2023-06-22T08:25:24.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/ptx.c' 2023-06-22T08:25:24.382 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/nohup.c' 2023-06-22T08:25:24.383 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/ls.h' 2023-06-22T08:25:24.383 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/chroot.c' 2023-06-22T08:25:24.383 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/tail.c' 2023-06-22T08:25:24.383 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/nice.c' 2023-06-22T08:25:24.384 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/fold.c' 2023-06-22T08:25:24.384 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/comm.c' 2023-06-22T08:25:24.384 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/chown-core.c' 2023-06-22T08:25:24.384 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/tac.c' 2023-06-22T08:25:24.385 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/prog-fprintf.c' 2023-06-22T08:25:24.385 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/tr.c' 2023-06-22T08:25:24.385 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/tac-pipe.c' 2023-06-22T08:25:24.385 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/groups.c' 2023-06-22T08:25:24.386 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/nproc.c' 2023-06-22T08:25:24.386 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/operand2sig.c' 2023-06-22T08:25:24.386 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/libstdbuf.c' 2023-06-22T08:25:24.386 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/nl.c' 2023-06-22T08:25:24.387 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/env.c' 2023-06-22T08:25:24.387 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/dcgen' 2023-06-22T08:25:24.387 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/stat.c' 2023-06-22T08:25:24.388 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/base64.c' 2023-06-22T08:25:24.388 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/chown.c' 2023-06-22T08:25:24.388 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/od.c' 2023-06-22T08:25:24.388 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/mkdir.c' 2023-06-22T08:25:24.389 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/logname.c' 2023-06-22T08:25:24.389 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/extract-magic' 2023-06-22T08:25:24.389 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/wheel-gen.pl' 2023-06-22T08:25:24.389 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/chcon.c' 2023-06-22T08:25:24.389 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/printenv.c' 2023-06-22T08:25:24.390 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/copy.c' 2023-06-22T08:25:24.390 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/ls-dir.c' 2023-06-22T08:25:24.390 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/cut.c' 2023-06-22T08:25:24.390 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/uptime.c' 2023-06-22T08:25:24.391 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/hostname.c' 2023-06-22T08:25:24.391 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/dirname.c' 2023-06-22T08:25:24.391 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/sleep.c' 2023-06-22T08:25:24.391 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/stdbuf.c' 2023-06-22T08:25:24.392 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/true.c' 2023-06-22T08:25:24.392 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/df.c' 2023-06-22T08:25:24.392 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/rmdir.c' 2023-06-22T08:25:24.392 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/copy.h' 2023-06-22T08:25:24.393 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/group-list.c' 2023-06-22T08:25:24.393 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/unexpand.c' 2023-06-22T08:25:24.393 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/stty.c' 2023-06-22T08:25:24.393 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/chmod.c' 2023-06-22T08:25:24.394 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/fmt.c' 2023-06-22T08:25:24.394 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/csplit.c' 2023-06-22T08:25:24.394 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/expand.c' 2023-06-22T08:25:24.394 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/kill.c' 2023-06-22T08:25:24.394 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/factor.c' 2023-06-22T08:25:24.395 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/wheel-size.h' 2023-06-22T08:25:24.395 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/pinky.c' 2023-06-22T08:25:24.395 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/yes.c' 2023-06-22T08:25:24.395 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/remove.c' 2023-06-22T08:25:24.396 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/Makefile.in' 2023-06-22T08:25:24.396 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/uname-uname.c' 2023-06-22T08:25:24.396 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/md5sum.c' 2023-06-22T08:25:24.396 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/ln.c' 2023-06-22T08:25:24.397 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/su.c' 2023-06-22T08:25:24.397 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/cksum.c' 2023-06-22T08:25:24.397 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/dircolors.h' 2023-06-22T08:25:24.397 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/group-list.h' 2023-06-22T08:25:24.398 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/false.c' 2023-06-22T08:25:24.398 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/operand2sig.h' 2023-06-22T08:25:24.398 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/id.c' 2023-06-22T08:25:24.398 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/rm.c' 2023-06-22T08:25:24.399 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/shred.c' 2023-06-22T08:25:24.399 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/printf.c' 2023-06-22T08:25:24.399 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/cp.c' 2023-06-22T08:25:24.399 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/uname-arch.c' 2023-06-22T08:25:24.400 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/dircolors.hin' 2023-06-22T08:25:24.400 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/setuidgid.c' 2023-06-22T08:25:24.400 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/cp-hash.c' 2023-06-22T08:25:24.400 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/tee.c' 2023-06-22T08:25:24.401 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/Makefile.am' 2023-06-22T08:25:24.401 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/head.c' 2023-06-22T08:25:24.401 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/users.c' 2023-06-22T08:25:24.401 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/runcon.c' 2023-06-22T08:25:24.402 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/du.c' 2023-06-22T08:25:24.402 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/date.c' 2023-06-22T08:25:24.402 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/unlink.c' 2023-06-22T08:25:24.402 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/echo.c' 2023-06-22T08:25:24.403 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/pathchk.c' 2023-06-22T08:25:24.403 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/tsort.c' 2023-06-22T08:25:24.403 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/touch.c' 2023-06-22T08:25:24.403 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/c99-to-c89.diff' 2023-06-22T08:25:24.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/join.c' 2023-06-22T08:25:24.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/cat.c' 2023-06-22T08:25:24.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/sync.c' 2023-06-22T08:25:24.404 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/shuf.c' 2023-06-22T08:25:24.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/truncate.c' 2023-06-22T08:25:24.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/ls-vdir.c' 2023-06-22T08:25:24.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/test.c' 2023-06-22T08:25:24.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/remove.h' 2023-06-22T08:25:24.405 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/seq.c' 2023-06-22T08:25:24.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/readlink.c' 2023-06-22T08:25:24.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/uname.c' 2023-06-22T08:25:24.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/sum.c' 2023-06-22T08:25:24.406 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/paste.c' 2023-06-22T08:25:24.407 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/uname.h' 2023-06-22T08:25:24.827 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/tty.c' 2023-06-22T08:25:24.827 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/fs.h' 2023-06-22T08:25:24.827 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/basename.c' 2023-06-22T08:25:24.827 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/dd.c' 2023-06-22T08:25:24.827 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/split.c' 2023-06-22T08:25:24.828 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/whoami.c' 2023-06-22T08:25:24.828 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/timeout.c' 2023-06-22T08:25:24.828 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/getlimits.c' 2023-06-22T08:25:24.828 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/dircolors.c' 2023-06-22T08:25:24.828 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/link.c' 2023-06-22T08:25:24.828 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/sort.c' 2023-06-22T08:25:24.829 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/wc.c' 2023-06-22T08:25:24.829 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/prog-fprintf.h' 2023-06-22T08:25:24.829 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/uniq.c' 2023-06-22T08:25:24.829 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/pr.c' 2023-06-22T08:25:24.829 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/system.h' 2023-06-22T08:25:24.829 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/pwd.c' 2023-06-22T08:25:24.829 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/hostid.c' 2023-06-22T08:25:24.830 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/install.c' 2023-06-22T08:25:24.830 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/wheel.h' 2023-06-22T08:25:24.830 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/mkfifo.c' 2023-06-22T08:25:24.831 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/mv.c' 2023-06-22T08:25:24.831 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/lbracket.c' 2023-06-22T08:25:24.831 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/mknod.c' 2023-06-22T08:25:24.831 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/mktemp.c' 2023-06-22T08:25:24.832 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/src/ls.c' 2023-06-22T08:25:24.832 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/src' 2023-06-22T08:25:24.832 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ru.gmo' 2023-06-22T08:25:24.832 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/zh_TW.gmo' 2023-06-22T08:25:24.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ko.gmo' 2023-06-22T08:25:24.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ro.po' 2023-06-22T08:25:24.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/vi.gmo' 2023-06-22T08:25:24.833 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/vi.po' 2023-06-22T08:25:24.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/da.gmo' 2023-06-22T08:25:24.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/fi.gmo' 2023-06-22T08:25:24.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/en.po' 2023-06-22T08:25:24.834 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/sl.po' 2023-06-22T08:25:24.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/fr.gmo' 2023-06-22T08:25:24.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/it.gmo' 2023-06-22T08:25:24.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/en@quot.header' 2023-06-22T08:25:24.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/el.po' 2023-06-22T08:25:24.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/tr.po' 2023-06-22T08:25:24.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/it.po' 2023-06-22T08:25:24.835 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/stamp-po' 2023-06-22T08:25:24.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/sk.gmo' 2023-06-22T08:25:24.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/tr.gmo' 2023-06-22T08:25:24.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ga.gmo' 2023-06-22T08:25:24.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ja.po' 2023-06-22T08:25:24.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/pl.po' 2023-06-22T08:25:24.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/pt.po' 2023-06-22T08:25:24.836 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/de.po' 2023-06-22T08:25:24.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/sv.gmo' 2023-06-22T08:25:24.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/Makevars' 2023-06-22T08:25:24.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/et.po' 2023-06-22T08:25:24.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ChangeLog-2007' 2023-06-22T08:25:24.837 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/fr.po' 2023-06-22T08:25:24.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/pt_BR.po' 2023-06-22T08:25:24.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ga.po' 2023-06-22T08:25:24.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/cs.po' 2023-06-22T08:25:24.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/pl.gmo' 2023-06-22T08:25:24.838 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/af.gmo' 2023-06-22T08:25:24.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/lt.po' 2023-06-22T08:25:24.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/cs.gmo' 2023-06-22T08:25:24.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/en.gmo' 2023-06-22T08:25:24.839 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ca.gmo' 2023-06-22T08:25:24.840 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/sl.gmo' 2023-06-22T08:25:24.840 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/eu.gmo' 2023-06-22T08:25:24.840 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/gl.gmo' 2023-06-22T08:25:24.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/uk.gmo' 2023-06-22T08:25:24.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/bg.gmo' 2023-06-22T08:25:24.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ro.gmo' 2023-06-22T08:25:24.841 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/hu.gmo' 2023-06-22T08:25:24.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/nl.po' 2023-06-22T08:25:24.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/el.gmo' 2023-06-22T08:25:24.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/LINGUAS' 2023-06-22T08:25:24.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/de.gmo' 2023-06-22T08:25:24.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ms.po' 2023-06-22T08:25:24.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/remove-potcdate.sin' 2023-06-22T08:25:24.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/da.po' 2023-06-22T08:25:24.842 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/en@boldquot.header' 2023-06-22T08:25:24.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/be.po' 2023-06-22T08:25:24.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/Makefile.in.in' 2023-06-22T08:25:24.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ca.po' 2023-06-22T08:25:24.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/id.po' 2023-06-22T08:25:24.843 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/hu.po' 2023-06-22T08:25:24.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/zh_CN.po' 2023-06-22T08:25:24.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/nb.gmo' 2023-06-22T08:25:24.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/nb.po' 2023-06-22T08:25:24.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/coreutils.pot' 2023-06-22T08:25:24.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/bg.po' 2023-06-22T08:25:24.844 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/id.gmo' 2023-06-22T08:25:24.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/sv.po' 2023-06-22T08:25:24.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/af.po' 2023-06-22T08:25:24.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/Rules-quot' 2023-06-22T08:25:24.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/eu.po' 2023-06-22T08:25:24.845 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/es.gmo' 2023-06-22T08:25:24.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/quot.sed' 2023-06-22T08:25:24.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ru.po' 2023-06-22T08:25:24.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ms.gmo' 2023-06-22T08:25:24.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/zh_TW.po' 2023-06-22T08:25:24.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/nl.gmo' 2023-06-22T08:25:24.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/zh_CN.gmo' 2023-06-22T08:25:24.846 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/pt.gmo' 2023-06-22T08:25:24.847 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/sk.po' 2023-06-22T08:25:24.847 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/lg.po' 2023-06-22T08:25:24.847 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ko.po' 2023-06-22T08:25:24.847 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/pt_BR.gmo' 2023-06-22T08:25:24.847 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/fi.po' 2023-06-22T08:25:24.847 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/es.po' 2023-06-22T08:25:24.848 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/lt.gmo' 2023-06-22T08:25:24.848 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/be.gmo' 2023-06-22T08:25:25.193 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/ja.gmo' 2023-06-22T08:25:25.193 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/lg.gmo' 2023-06-22T08:25:25.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/gl.po' 2023-06-22T08:25:25.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/boldquot.sed' 2023-06-22T08:25:25.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/et.gmo' 2023-06-22T08:25:25.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/insert-header.sin' 2023-06-22T08:25:25.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/POTFILES.in' 2023-06-22T08:25:25.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/po/uk.po' 2023-06-22T08:25:25.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/po' 2023-06-22T08:25:25.194 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_prohibit_empty_lines_at_EOF' 2023-06-22T08:25:25.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/GNUmakefile' 2023-06-22T08:25:25.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_useless_cpp_parens' 2023-06-22T08:25:25.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/ChangeLog' 2023-06-22T08:25:25.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_require_config_h' 2023-06-22T08:25:25.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/test-lib.sh' 2023-06-22T08:25:25.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/require-perl' 2023-06-22T08:25:25.195 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/octal' 2023-06-22T08:25:25.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/umask-x' 2023-06-22T08:25:25.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/equal-x' 2023-06-22T08:25:25.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/thru-dangling' 2023-06-22T08:25:25.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/equals' 2023-06-22T08:25:25.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/no-x' 2023-06-22T08:25:25.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/inaccessible' 2023-06-22T08:25:25.196 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/usage' 2023-06-22T08:25:25.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/c-option' 2023-06-22T08:25:25.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/silent' 2023-06-22T08:25:25.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chmod/setgid' 2023-06-22T08:25:25.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/chmod' 2023-06-22T08:25:25.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rmdir/t-slash' 2023-06-22T08:25:25.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rmdir/fail-perm' 2023-06-22T08:25:25.197 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rmdir/ignore' 2023-06-22T08:25:25.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/rmdir' 2023-06-22T08:25:25.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chgrp/posix-H' 2023-06-22T08:25:25.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chgrp/default-no-deref' 2023-06-22T08:25:25.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chgrp/no-x' 2023-06-22T08:25:25.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chgrp/basic' 2023-06-22T08:25:25.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chgrp/deref' 2023-06-22T08:25:25.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chgrp/recurse' 2023-06-22T08:25:25.198 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/chgrp' 2023-06-22T08:25:25.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/envvar-check' 2023-06-22T08:25:25.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chown/basic' 2023-06-22T08:25:25.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chown/deref' 2023-06-22T08:25:25.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chown/separator' 2023-06-22T08:25:25.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/chown/preserve-root' 2023-06-22T08:25:25.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/chown' 2023-06-22T08:25:25.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/acl' 2023-06-22T08:25:25.199 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/i-2' 2023-06-22T08:25:25.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/dup-source' 2023-06-22T08:25:25.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/to-symlink' 2023-06-22T08:25:25.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/childproof' 2023-06-22T08:25:25.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/i-5' 2023-06-22T08:25:25.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/mv-n' 2023-06-22T08:25:25.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/part-hardlink' 2023-06-22T08:25:25.200 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/dir2dir' 2023-06-22T08:25:25.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/part-rename' 2023-06-22T08:25:25.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/into-self-3' 2023-06-22T08:25:25.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/hard-2' 2023-06-22T08:25:25.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/trailing-slash' 2023-06-22T08:25:25.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/hard-verbose' 2023-06-22T08:25:25.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/force' 2023-06-22T08:25:25.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/i-link-no' 2023-06-22T08:25:25.201 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/i-3' 2023-06-22T08:25:25.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/backup-dir' 2023-06-22T08:25:25.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/i-4' 2023-06-22T08:25:25.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/into-self' 2023-06-22T08:25:25.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/diag' 2023-06-22T08:25:25.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/into-self-2' 2023-06-22T08:25:25.202 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/mv-special-1' 2023-06-22T08:25:25.203 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/update' 2023-06-22T08:25:25.203 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/atomic2' 2023-06-22T08:25:25.203 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/no-target-dir' 2023-06-22T08:25:25.204 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/hard-4' 2023-06-22T08:25:25.204 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/sticky-to-xpart' 2023-06-22T08:25:25.204 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/dir-file' 2023-06-22T08:25:25.204 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/i-1' 2023-06-22T08:25:25.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/hard-3' 2023-06-22T08:25:25.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/part-symlink' 2023-06-22T08:25:25.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/hard-link-1' 2023-06-22T08:25:25.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/backup-is-src' 2023-06-22T08:25:25.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/perm-1' 2023-06-22T08:25:25.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/partition-perm' 2023-06-22T08:25:25.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/part-fail' 2023-06-22T08:25:25.205 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/leak-fd' 2023-06-22T08:25:25.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/atomic' 2023-06-22T08:25:25.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mv/into-self-4' 2023-06-22T08:25:25.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/mv' 2023-06-22T08:25:25.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/multihardlink' 2023-06-22T08:25:25.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/stat-failed' 2023-06-22T08:25:25.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/stat-free-symlinks' 2023-06-22T08:25:25.206 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/color-clear-to-eol' 2023-06-22T08:25:25.207 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/rt-1' 2023-06-22T08:25:25.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/color-dtype-dir' 2023-06-22T08:25:25.508 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/symlink-slash' 2023-06-22T08:25:25.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/infloop' 2023-06-22T08:25:25.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/follow-slink' 2023-06-22T08:25:25.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/no-arg' 2023-06-22T08:25:25.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/inode' 2023-06-22T08:25:25.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/color-norm' 2023-06-22T08:25:25.509 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/no-cap' 2023-06-22T08:25:25.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/x-option' 2023-06-22T08:25:25.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/proc-selinux-segfault' 2023-06-22T08:25:25.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/abmon-align' 2023-06-22T08:25:25.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/file-type' 2023-06-22T08:25:25.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/stat-vs-dirent' 2023-06-22T08:25:25.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/nameless-uid' 2023-06-22T08:25:25.510 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/dangle' 2023-06-22T08:25:25.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/readdir-mountpoint-inode' 2023-06-22T08:25:25.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/stat-dtype' 2023-06-22T08:25:25.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/dired' 2023-06-22T08:25:25.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/capability' 2023-06-22T08:25:25.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/m-option' 2023-06-22T08:25:25.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ls/recursive' 2023-06-22T08:25:25.511 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/ls' 2023-06-22T08:25:25.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/p-slashdot' 2023-06-22T08:25:25.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/t-slash' 2023-06-22T08:25:25.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/perm' 2023-06-22T08:25:25.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/p-thru-slink' 2023-06-22T08:25:25.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/p-3' 2023-06-22T08:25:25.512 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/p-2' 2023-06-22T08:25:25.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/writable-under-readonly' 2023-06-22T08:25:25.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/selinux' 2023-06-22T08:25:25.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/p-1' 2023-06-22T08:25:25.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/p-v' 2023-06-22T08:25:25.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/parents' 2023-06-22T08:25:25.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/mkdir/special-1' 2023-06-22T08:25:25.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/mkdir' 2023-06-22T08:25:25.513 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/install/create-leading' 2023-06-22T08:25:25.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/install/install-C-root' 2023-06-22T08:25:25.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/install/install-C' 2023-06-22T08:25:25.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/install/basic-1' 2023-06-22T08:25:25.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/install/install-C-selinux' 2023-06-22T08:25:25.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/install/d-slashdot' 2023-06-22T08:25:25.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/install/strip-program' 2023-06-22T08:25:25.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/install/trap' 2023-06-22T08:25:25.514 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/install' 2023-06-22T08:25:25.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/readlink/rl-1' 2023-06-22T08:25:25.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/readlink/can-m' 2023-06-22T08:25:25.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/readlink/can-f' 2023-06-22T08:25:25.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/readlink/can-e' 2023-06-22T08:25:25.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/readlink' 2023-06-22T08:25:25.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/2g' 2023-06-22T08:25:25.515 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/8gb' 2023-06-22T08:25:25.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/restore-wd' 2023-06-22T08:25:25.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/long-from-unreadable' 2023-06-22T08:25:25.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/two-args' 2023-06-22T08:25:25.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/slink' 2023-06-22T08:25:25.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/inaccessible-cwd' 2023-06-22T08:25:25.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/no-x' 2023-06-22T08:25:25.516 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/fd-leak' 2023-06-22T08:25:25.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/basic' 2023-06-22T08:25:25.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/one-file-system' 2023-06-22T08:25:25.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/trailing-slash' 2023-06-22T08:25:25.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/no-deref' 2023-06-22T08:25:25.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/long-sloop' 2023-06-22T08:25:25.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/inacc-dir' 2023-06-22T08:25:25.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/deref' 2023-06-22T08:25:25.517 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/inacc-dest' 2023-06-22T08:25:25.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/files0-from' 2023-06-22T08:25:25.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/slash' 2023-06-22T08:25:25.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/exclude' 2023-06-22T08:25:25.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/hard-link' 2023-06-22T08:25:25.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/du/deref-args' 2023-06-22T08:25:25.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/du' 2023-06-22T08:25:25.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/id/no-context' 2023-06-22T08:25:25.518 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/id' 2023-06-22T08:25:25.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/not-owner' 2023-06-22T08:25:25.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/fifo' 2023-06-22T08:25:25.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/obsolescent' 2023-06-22T08:25:25.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/dangling-symlink' 2023-06-22T08:25:25.519 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/trailing-slash' 2023-06-22T08:25:25.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/no-create-missing' 2023-06-22T08:25:25.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/no-rights' 2023-06-22T08:25:25.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/relative' 2023-06-22T08:25:25.520 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/no-dereference' 2023-06-22T08:25:25.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/empty-file' 2023-06-22T08:25:25.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/dir-1' 2023-06-22T08:25:25.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/60-seconds' 2023-06-22T08:25:25.521 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/now-owned-by-other' 2023-06-22T08:25:26.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/fail-diag' 2023-06-22T08:25:26.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/touch/read-only' 2023-06-22T08:25:26.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/touch' 2023-06-22T08:25:26.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/flush-initial' 2023-06-22T08:25:26.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/F-vs-rename' 2023-06-22T08:25:26.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/assert-2' 2023-06-22T08:25:26.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/inotify-rotate' 2023-06-22T08:25:26.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/proc-ksyms' 2023-06-22T08:25:26.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/infloop-1' 2023-06-22T08:25:26.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/start-middle' 2023-06-22T08:25:26.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/inotify-hash-abuse' 2023-06-22T08:25:26.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/pipe-f2' 2023-06-22T08:25:26.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/tail-n0f' 2023-06-22T08:25:26.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/append-only' 2023-06-22T08:25:26.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/pipe-f' 2023-06-22T08:25:26.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/wait' 2023-06-22T08:25:26.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/assert' 2023-06-22T08:25:26.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/inotify-race' 2023-06-22T08:25:26.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/follow-stdin' 2023-06-22T08:25:26.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/inotify-hash-abuse2' 2023-06-22T08:25:26.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/big-4gb' 2023-06-22T08:25:26.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/tail-2/pid' 2023-06-22T08:25:26.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/tail-2' 2023-06-22T08:25:26.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/lang-default' 2023-06-22T08:25:26.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/Makefile.in' 2023-06-22T08:25:26.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/Coreutils.pm' 2023-06-22T08:25:26.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/Makefile.am' 2023-06-22T08:25:26.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/hash' 2023-06-22T08:25:26.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/rm1' 2023-06-22T08:25:26.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/empty-inacc' 2023-06-22T08:25:26.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/r-3' 2023-06-22T08:25:26.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/one-file-system2' 2023-06-22T08:25:26.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/readdir-bug' 2023-06-22T08:25:26.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/r-1' 2023-06-22T08:25:26.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/f-1' 2023-06-22T08:25:26.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/i-no-r' 2023-06-22T08:25:26.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/ir-1' 2023-06-22T08:25:26.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/fail-eacces' 2023-06-22T08:25:26.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/i-never' 2023-06-22T08:25:26.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/r-4' 2023-06-22T08:25:26.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/dir-no-w' 2023-06-22T08:25:26.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/no-give-up' 2023-06-22T08:25:26.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/dangling-symlink' 2023-06-22T08:25:26.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/unreadable' 2023-06-22T08:25:26.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/dot-rel' 2023-06-22T08:25:26.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/ignorable' 2023-06-22T08:25:26.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/one-file-system' 2023-06-22T08:25:26.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/interactive-once' 2023-06-22T08:25:26.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/unread2' 2023-06-22T08:25:26.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/inaccessible' 2023-06-22T08:25:26.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/isatty' 2023-06-22T08:25:26.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/rm5' 2023-06-22T08:25:26.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/interactive-always' 2023-06-22T08:25:26.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/unread3' 2023-06-22T08:25:26.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/fail-eperm' 2023-06-22T08:25:26.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/dir-nonrecur' 2023-06-22T08:25:26.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/empty-name' 2023-06-22T08:25:26.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/rm4' 2023-06-22T08:25:26.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/cycle' 2023-06-22T08:25:26.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/v-slash' 2023-06-22T08:25:26.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/deep-1' 2023-06-22T08:25:26.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/rm2' 2023-06-22T08:25:26.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/rm3' 2023-06-22T08:25:26.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/sunos-1' 2023-06-22T08:25:26.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/deep-2' 2023-06-22T08:25:26.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/i-1' 2023-06-22T08:25:26.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/r-2' 2023-06-22T08:25:26.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/ext3-perf' 2023-06-22T08:25:26.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/fail-2eperm' 2023-06-22T08:25:26.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/rm/read-only' 2023-06-22T08:25:26.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/rm' 2023-06-22T08:25:26.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/check.mk' 2023-06-22T08:25:26.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/pr-tests' 2023-06-22T08:25:26.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/o3ml24f-bl-tn' 2023-06-22T08:25:26.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+2l24f-bl' 2023-06-22T08:25:26.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/b2l17-FF' 2023-06-22T08:25:26.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+3ml24f-tn-bl' 2023-06-22T08:25:26.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/l24f-t' 2023-06-22T08:25:26.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3b3l15f-t' 2023-06-22T08:25:26.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/4-7l24-FF' 2023-06-22T08:25:26.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+6a2l17-FF' 2023-06-22T08:25:26.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nl24f-bl' 2023-06-22T08:25:26.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nSml24-bl-FF' 2023-06-22T08:25:26.019 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/a3l24f-lm' 2023-06-22T08:25:26.019 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+3ml20f-bl-FF' 2023-06-22T08:25:26.019 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3b3f-0F' 2023-06-22T08:25:26.019 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/FFn' 2023-06-22T08:25:26.019 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W27l24f-ll' 2023-06-22T08:25:26.019 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tne8o3-t_tab' 2023-06-22T08:25:26.019 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/a2l17-FF' 2023-06-22T08:25:26.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/a2l15-FF' 2023-06-22T08:25:26.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ml24f-0F' 2023-06-22T08:25:26.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W72Jl24f-ll' 2023-06-22T08:25:26.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/o3b3Snl24f-tn' 2023-06-22T08:25:26.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nSml24-t-tFFFF' 2023-06-22T08:25:26.142 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W26l24f-ll' 2023-06-22T08:25:26.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3b3l15-t' 2023-06-22T08:25:26.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ml24-t-FF' 2023-06-22T08:25:26.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ttb3-0FF' 2023-06-22T08:25:26.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+4-8a2l17-FF' 2023-06-22T08:25:26.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tFFt-ll' 2023-06-22T08:25:26.143 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/b3-0F' 2023-06-22T08:25:26.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+3-7l24-FF' 2023-06-22T08:25:26.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tt-0FF' 2023-06-22T08:25:26.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/Ja3l24f-lm' 2023-06-22T08:25:26.144 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/w72l24f-ll' 2023-06-22T08:25:26.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W35b3l24f-lm' 2023-06-22T08:25:26.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/0FF' 2023-06-22T08:25:26.145 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/0Ft' 2023-06-22T08:25:26.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/2-S_f-t_notab' 2023-06-22T08:25:26.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/2sw60f-t_notab' 2023-06-22T08:25:26.146 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3b2l17-FF' 2023-06-22T08:25:26.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tFFt-lm' 2023-06-22T08:25:26.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3-FF' 2023-06-22T08:25:26.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/t-t' 2023-06-22T08:25:26.147 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/Jml24f-lm-lo' 2023-06-22T08:25:26.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nN1+3l24f-bl' 2023-06-22T08:25:26.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/0FFnt' 2023-06-22T08:25:26.148 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tta3-0FF' 2023-06-22T08:25:26.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3-0F' 2023-06-22T08:25:26.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+6b3l13f-FF' 2023-06-22T08:25:26.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3ml24-t' 2023-06-22T08:25:26.149 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/t-0FF' 2023-06-22T08:25:26.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/o3Jml24f-lm-lo' 2023-06-22T08:25:26.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tn_2e8-t_tab' 2023-06-22T08:25:26.150 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+8l20-FF' 2023-06-22T08:25:26.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W35Ja3l24f-lm' 2023-06-22T08:25:26.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ml24-t' 2023-06-22T08:25:26.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W35ml24f-lm-lo' 2023-06-22T08:25:26.151 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W-72l24f-ll' 2023-06-22T08:25:26.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/o3mSl24f-bl-tn' 2023-06-22T08:25:26.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/t-FF' 2023-06-22T08:25:26.152 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3ml24-t-FF' 2023-06-22T08:25:26.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3b3f-FF' 2023-06-22T08:25:26.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tta3-FF' 2023-06-22T08:25:26.153 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W35Jml24f-lmlo' 2023-06-22T08:25:26.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tFnFt' 2023-06-22T08:25:26.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/0FnFnt' 2023-06-22T08:25:26.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/0FnFt' 2023-06-22T08:25:26.154 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nSml24-t-t-FF' 2023-06-22T08:25:26.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/b3f-0FF' 2023-06-22T08:25:26.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tn' 2023-06-22T08:25:26.155 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tt-FF' 2023-06-22T08:25:26.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W28l24f-ll' 2023-06-22T08:25:26.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tb3-FF' 2023-06-22T08:25:26.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tn2e8-t_tab' 2023-06-22T08:25:26.156 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/l24-FF' 2023-06-22T08:25:26.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+4b2l17f-0FF' 2023-06-22T08:25:26.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3a3f-0F' 2023-06-22T08:25:26.157 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/FFtn' 2023-06-22T08:25:26.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/a3f-FF' 2023-06-22T08:25:26.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+2l24f-0FF' 2023-06-22T08:25:26.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/2s_f-t_notab' 2023-06-22T08:25:26.158 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/a3l15-t' 2023-06-22T08:25:26.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3b3f-0FF' 2023-06-22T08:25:26.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nJml24f-lmlolm' 2023-06-22T08:25:26.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ttb3-FF' 2023-06-22T08:25:26.159 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/loli' 2023-06-22T08:25:26.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/b3l15-t' 2023-06-22T08:25:26.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3-5l24f-t' 2023-06-22T08:25:26.160 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ta3-0FF' 2023-06-22T08:25:26.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/0F' 2023-06-22T08:25:26.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/FF' 2023-06-22T08:25:26.161 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ml24f-t' 2023-06-22T08:25:26.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/t_tab_' 2023-06-22T08:25:26.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/o3b3l24f-tn' 2023-06-22T08:25:26.162 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/4l24-FF' 2023-06-22T08:25:26.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/o3mSnl24fbltn' 2023-06-22T08:25:26.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tFFn' 2023-06-22T08:25:26.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+5-8b3l17f-FF' 2023-06-22T08:25:26.163 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nSml20-bl-FF' 2023-06-22T08:25:26.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/o3a3Sl24f-tn' 2023-06-22T08:25:26.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nJml24f-lmlmlo' 2023-06-22T08:25:26.164 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W35a3l24f-lm' 2023-06-22T08:25:26.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nN15l24f-bl' 2023-06-22T08:25:26.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/a3f-0F' 2023-06-22T08:25:26.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/b3f-0F' 2023-06-22T08:25:26.165 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/t_notab' 2023-06-22T08:25:26.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+5a3l13f-0FF' 2023-06-22T08:25:26.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nSml20-t-t-FF' 2023-06-22T08:25:26.166 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tn2e5o3-t_tab' 2023-06-22T08:25:26.462 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ml24f-lm-lo' 2023-06-22T08:25:26.462 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3f-FF' 2023-06-22T08:25:26.462 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/t-bl' 2023-06-22T08:25:26.462 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3ml24f-t' 2023-06-22T08:25:26.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/b3f-FF' 2023-06-22T08:25:26.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/FnFn' 2023-06-22T08:25:26.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+3ml24f-bl-tn' 2023-06-22T08:25:26.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/0FFt' 2023-06-22T08:25:26.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tb3-0FF' 2023-06-22T08:25:26.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tt-t' 2023-06-22T08:25:26.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/2-Sf-t_notab' 2023-06-22T08:25:26.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ta3-FF' 2023-06-22T08:25:26.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W20l24f-ll' 2023-06-22T08:25:26.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tn_2e8S-t_tab' 2023-06-22T08:25:26.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/b2l15-FF' 2023-06-22T08:25:26.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/l24-t' 2023-06-22T08:25:26.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3l24f-t' 2023-06-22T08:25:26.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tn2e8o3-t_tab' 2023-06-22T08:25:26.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/a3f-0FF' 2023-06-22T08:25:26.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/a3l15f-t' 2023-06-22T08:25:26.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/Jb3l24f-lm' 2023-06-22T08:25:26.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3a2l17-FF' 2023-06-22T08:25:26.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3f-0F' 2023-06-22T08:25:26.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/t_tab' 2023-06-22T08:25:26.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3l24-t' 2023-06-22T08:25:26.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/W35Jb3l24f-lm' 2023-06-22T08:25:26.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tFFt-bl' 2023-06-22T08:25:26.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tFFt' 2023-06-22T08:25:26.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+2-5l24f-0FF' 2023-06-22T08:25:26.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+3l24f-bl' 2023-06-22T08:25:26.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+7l24-FF' 2023-06-22T08:25:26.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3a3l15-t' 2023-06-22T08:25:26.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ml24f-t-0F' 2023-06-22T08:25:26.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ml24-FF' 2023-06-22T08:25:26.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/0Fnt' 2023-06-22T08:25:26.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/2f-t_notab' 2023-06-22T08:25:26.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/2s_w60f-t_nota' 2023-06-22T08:25:26.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/ml20-FF-t' 2023-06-22T08:25:26.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tt-bl' 2023-06-22T08:25:26.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3a3l15f-t' 2023-06-22T08:25:26.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/o3a3l24f-tn' 2023-06-22T08:25:26.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/o3b3Sl24f-tn' 2023-06-22T08:25:26.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/tne8-t_tab' 2023-06-22T08:25:26.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/a3-0F' 2023-06-22T08:25:26.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/nSml20-t-tFFFF' 2023-06-22T08:25:26.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/b3l24f-lm' 2023-06-22T08:25:26.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/2w60f-t_notab' 2023-06-22T08:25:26.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/2sf-t_notab' 2023-06-22T08:25:26.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/b3l15f-t' 2023-06-22T08:25:26.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/3ml24-FF' 2023-06-22T08:25:26.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/n+3l24f-0FF' 2023-06-22T08:25:26.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/pr/o3a3Snl24f-tn' 2023-06-22T08:25:26.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/pr' 2023-06-22T08:25:26.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/CuTmpdir.pm' 2023-06-22T08:25:26.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/df/unreadable' 2023-06-22T08:25:26.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/df/total-verify' 2023-06-22T08:25:26.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/df' 2023-06-22T08:25:26.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ln/sf-1' 2023-06-22T08:25:26.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ln/hard-backup' 2023-06-22T08:25:26.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ln/slash-decorated-nonexistent-dest' 2023-06-22T08:25:26.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ln/hard-to-sym' 2023-06-22T08:25:26.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ln/target-1' 2023-06-22T08:25:26.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ln/backup-1' 2023-06-22T08:25:26.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/ln/misc' 2023-06-22T08:25:26.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/ln' 2023-06-22T08:25:26.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/cp-mv-backup' 2023-06-22T08:25:26.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/acl' 2023-06-22T08:25:26.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/perm' 2023-06-22T08:25:26.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/cp-deref' 2023-06-22T08:25:26.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/special-bits' 2023-06-22T08:25:26.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/link' 2023-06-22T08:25:26.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/preserve-slink-time' 2023-06-22T08:25:26.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/symlink-slash' 2023-06-22T08:25:26.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/cp-mv-enotsup-xattr' 2023-06-22T08:25:26.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/thru-dangling' 2023-06-22T08:25:26.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/reflink-perm' 2023-06-22T08:25:26.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/r-vs-symlink' 2023-06-22T08:25:26.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/preserve-2' 2023-06-22T08:25:26.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/special-f' 2023-06-22T08:25:26.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/file-perm-race' 2023-06-22T08:25:26.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/slink-2-slink' 2023-06-22T08:25:26.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/proc-zero-len' 2023-06-22T08:25:26.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/reflink-auto' 2023-06-22T08:25:26.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/src-base-dot' 2023-06-22T08:25:26.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/link-preserve' 2023-06-22T08:25:26.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/cp-i' 2023-06-22T08:25:26.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/cp-a-selinux' 2023-06-22T08:25:26.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/preserve-gid' 2023-06-22T08:25:26.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/no-deref-link3' 2023-06-22T08:25:26.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/backup-dir' 2023-06-22T08:25:26.890 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/into-self' 2023-06-22T08:25:26.890 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/same-file' 2023-06-22T08:25:26.890 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/cp-HL' 2023-06-22T08:25:26.890 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/link-no-deref' 2023-06-22T08:25:26.891 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/parent-perm-race' 2023-06-22T08:25:26.891 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/proc-short-read' 2023-06-22T08:25:26.891 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/no-deref-link1' 2023-06-22T08:25:26.891 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/parent-perm' 2023-06-22T08:25:26.891 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/dir-vs-file' 2023-06-22T08:25:26.891 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/sparse' 2023-06-22T08:25:26.892 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/link-heap' 2023-06-22T08:25:26.892 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/no-deref-link2' 2023-06-22T08:25:26.892 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/dir-rm-dest' 2023-06-22T08:25:26.892 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/dir-slash' 2023-06-22T08:25:26.892 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/fail-perm' 2023-06-22T08:25:26.892 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/deref-slink' 2023-06-22T08:25:26.892 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/existing-perm-race' 2023-06-22T08:25:26.893 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/abuse' 2023-06-22T08:25:26.893 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/cp-parents' 2023-06-22T08:25:26.893 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/backup-1' 2023-06-22T08:25:26.893 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/backup-is-src' 2023-06-22T08:25:26.893 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/cp/capability' 2023-06-22T08:25:26.893 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/cp' 2023-06-22T08:25:26.893 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/not-rewound' 2023-06-22T08:25:26.894 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/skip-seek2' 2023-06-22T08:25:26.894 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/unblock' 2023-06-22T08:25:26.894 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/skip-seek-past-dev' 2023-06-22T08:25:26.894 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/direct' 2023-06-22T08:25:26.894 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/unblock-sync' 2023-06-22T08:25:26.894 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/skip-seek-past-file' 2023-06-22T08:25:26.895 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/stderr' 2023-06-22T08:25:26.895 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/reblock' 2023-06-22T08:25:26.895 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/misc' 2023-06-22T08:25:26.895 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/dd/skip-seek' 2023-06-22T08:25:26.895 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/dd' 2023-06-22T08:25:26.895 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/fold' 2023-06-22T08:25:26.895 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/tty-eof' 2023-06-22T08:25:26.895 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sort-rand' 2023-06-22T08:25:26.896 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/seq-long-double' 2023-06-22T08:25:26.896 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/shred-remove' 2023-06-22T08:25:26.896 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/nice' 2023-06-22T08:25:26.896 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/tee' 2023-06-22T08:25:26.896 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/truncate-owned-by-other' 2023-06-22T08:25:26.896 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/mknod' 2023-06-22T08:25:26.896 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/expand' 2023-06-22T08:25:26.897 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/timeout-parameters' 2023-06-22T08:25:26.897 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sum' 2023-06-22T08:25:26.897 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/readlink-fp-loop' 2023-06-22T08:25:26.898 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/printenv' 2023-06-22T08:25:26.898 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/nohup' 2023-06-22T08:25:26.898 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/printf-hex' 2023-06-22T08:25:26.898 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/ptx' 2023-06-22T08:25:26.898 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/wc-files0' 2023-06-22T08:25:26.898 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/tac-continue' 2023-06-22T08:25:26.898 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sort-merge-fdlimit' 2023-06-22T08:25:26.899 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/truncate-dir-fail' 2023-06-22T08:25:26.899 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/env-null' 2023-06-22T08:25:26.899 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/xstrtol' 2023-06-22T08:25:26.899 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/tr' 2023-06-22T08:25:26.899 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/paste' 2023-06-22T08:25:26.899 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/printf' 2023-06-22T08:25:26.899 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/comm' 2023-06-22T08:25:26.899 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/id-context' 2023-06-22T08:25:26.900 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/chcon' 2023-06-22T08:25:26.900 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/shred-passes' 2023-06-22T08:25:26.900 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/stat-fmt' 2023-06-22T08:25:26.900 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/truncate-parameters' 2023-06-22T08:25:26.900 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/date-sec' 2023-06-22T08:25:26.900 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/tsort' 2023-06-22T08:25:26.900 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sha1sum-vec' 2023-06-22T08:25:26.901 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/stat-hyphen' 2023-06-22T08:25:26.901 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/od-multiple-t' 2023-06-22T08:25:26.901 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/xattr' 2023-06-22T08:25:26.901 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/od' 2023-06-22T08:25:26.901 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sum-sysv' 2023-06-22T08:25:26.901 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/date-next-dow' 2023-06-22T08:25:26.902 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/nproc-avail' 2023-06-22T08:25:26.902 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/split-a' 2023-06-22T08:25:26.902 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/unexpand' 2023-06-22T08:25:26.902 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/pathchk1' 2023-06-22T08:25:26.902 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/md5sum' 2023-06-22T08:25:26.903 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/groups-version' 2023-06-22T08:25:26.903 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/head' 2023-06-22T08:25:26.903 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/stty-invalid' 2023-06-22T08:25:26.904 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/factor' 2023-06-22T08:25:26.904 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/timeout' 2023-06-22T08:25:26.904 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/su-fail' 2023-06-22T08:25:26.905 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/wc-parallel' 2023-06-22T08:25:27.225 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/md5sum-newline' 2023-06-22T08:25:27.225 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/tee-dash' 2023-06-22T08:25:27.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/runcon-no-reorder' 2023-06-22T08:25:27.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/dircolors' 2023-06-22T08:25:27.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/basename' 2023-06-22T08:25:27.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/selinux' 2023-06-22T08:25:27.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/head-elide-tail' 2023-06-22T08:25:27.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/help-version' 2023-06-22T08:25:27.226 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/stdbuf' 2023-06-22T08:25:27.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/expr' 2023-06-22T08:25:27.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/groups-dash' 2023-06-22T08:25:27.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/shuf' 2023-06-22T08:25:27.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sort-version' 2023-06-22T08:25:27.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/pwd-long' 2023-06-22T08:25:27.227 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/env' 2023-06-22T08:25:27.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sort-continue' 2023-06-22T08:25:27.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/od-x8' 2023-06-22T08:25:27.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/arch' 2023-06-22T08:25:27.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/chcon-fail' 2023-06-22T08:25:27.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/tac' 2023-06-22T08:25:27.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/chroot-fail' 2023-06-22T08:25:27.228 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/cat-proc' 2023-06-22T08:25:27.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/truncate-fail-diag' 2023-06-22T08:25:27.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/ptx-overrun' 2023-06-22T08:25:27.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sort-month' 2023-06-22T08:25:27.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/date' 2023-06-22T08:25:27.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/ls-time' 2023-06-22T08:25:27.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/stat-printf' 2023-06-22T08:25:27.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/printf-surprise' 2023-06-22T08:25:27.229 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/pwd-option' 2023-06-22T08:25:27.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/id-groups' 2023-06-22T08:25:27.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/stat-slash' 2023-06-22T08:25:27.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/ls-misc' 2023-06-22T08:25:27.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/fmt-long-line' 2023-06-22T08:25:27.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/split-l' 2023-06-22T08:25:27.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/uniq' 2023-06-22T08:25:27.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/truncate-relative' 2023-06-22T08:25:27.230 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/mktemp' 2023-06-22T08:25:27.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/nproc-positive' 2023-06-22T08:25:27.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/truncate-dangling-symlink' 2023-06-22T08:25:27.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/dirname' 2023-06-22T08:25:27.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/wc' 2023-06-22T08:25:27.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/pr' 2023-06-22T08:25:27.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sort' 2023-06-22T08:25:27.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/chroot-credentials' 2023-06-22T08:25:27.231 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/base64' 2023-06-22T08:25:27.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/test-diag' 2023-06-22T08:25:27.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/od-N' 2023-06-22T08:25:27.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/invalid-opt' 2023-06-22T08:25:27.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sha224sum' 2023-06-22T08:25:27.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/split-fail' 2023-06-22T08:25:27.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/tail' 2023-06-22T08:25:27.232 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/shred-exact' 2023-06-22T08:25:27.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/df' 2023-06-22T08:25:27.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/seq' 2023-06-22T08:25:27.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/truncate-no-create-missing' 2023-06-22T08:25:27.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sha256sum' 2023-06-22T08:25:27.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sha1sum' 2023-06-22T08:25:27.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/nl' 2023-06-22T08:25:27.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/fmt' 2023-06-22T08:25:27.233 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/stty-row-col' 2023-06-22T08:25:27.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/close-stdout' 2023-06-22T08:25:27.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/nice-fail' 2023-06-22T08:25:27.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/truncate-overflow' 2023-06-22T08:25:27.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sort-compress' 2023-06-22T08:25:27.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/false-status' 2023-06-22T08:25:27.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/test' 2023-06-22T08:25:27.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sort-files0-from' 2023-06-22T08:25:27.234 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/df-P' 2023-06-22T08:25:27.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/md5sum-parallel' 2023-06-22T08:25:27.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/cut' 2023-06-22T08:25:27.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sort-merge' 2023-06-22T08:25:27.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/csplit' 2023-06-22T08:25:27.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/wc-files0-from' 2023-06-22T08:25:27.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/truncate-fifo' 2023-06-22T08:25:27.235 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sha512sum' 2023-06-22T08:25:27.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/head-c' 2023-06-22T08:25:27.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/join' 2023-06-22T08:25:27.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/sha384sum' 2023-06-22T08:25:27.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/cat-buf' 2023-06-22T08:25:27.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/printf-cov' 2023-06-22T08:25:27.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/head-pos' 2023-06-22T08:25:27.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/stty' 2023-06-22T08:25:27.236 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/misc/pwd-unreadable-parent' 2023-06-22T08:25:27.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests/misc' 2023-06-22T08:25:27.237 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/other-fs-tmpdir' 2023-06-22T08:25:27.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~305s 2023-06-22T08:25:27.280 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-06-22T08:25:27.281 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T08:25:27.281 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-22T08:25:27.327 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T08:25:27.327 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-22T08:25:27.416 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T08:25:27.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/tests/sample-test' 2023-06-22T08:25:27.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/tests' 2023-06-22T08:25:27.524 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/getdate.texi' 2023-06-22T08:25:27.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/perm.texi' 2023-06-22T08:25:27.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/coreutils.info' 2023-06-22T08:25:27.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/stamp-vti' 2023-06-22T08:25:27.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/ChangeLog-2007' 2023-06-22T08:25:27.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/Makefile.in' 2023-06-22T08:25:27.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/constants.texi' 2023-06-22T08:25:27.525 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/Makefile.am' 2023-06-22T08:25:27.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/fdl.texi' 2023-06-22T08:25:27.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/coreutils.texi' 2023-06-22T08:25:27.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/doc/version.texi' 2023-06-22T08:25:27.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/doc' 2023-06-22T08:25:27.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_prohibit_tab_based_indentation' 2023-06-22T08:25:27.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/gl/modules/getloadavg.diff' 2023-06-22T08:25:27.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/gl/modules' 2023-06-22T08:25:27.526 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/gl' 2023-06-22T08:25:27.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_prohibit_stat_st_blocks' 2023-06-22T08:25:27.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/THANKS' 2023-06-22T08:25:27.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/hash.h' 2023-06-22T08:25:27.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fts.c' 2023-06-22T08:25:27.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/full-read.c' 2023-06-22T08:25:27.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/error.h' 2023-06-22T08:25:27.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/idcache.h' 2023-06-22T08:25:27.527 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/faccessat.c' 2023-06-22T08:25:27.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fcntl.in.h' 2023-06-22T08:25:27.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/lchown.c' 2023-06-22T08:25:27.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sys_stat.in.h' 2023-06-22T08:25:27.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/openat-safer.c' 2023-06-22T08:25:27.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrtol.c' 2023-06-22T08:25:27.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/verror.c' 2023-06-22T08:25:27.528 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/root-dev-ino.c' 2023-06-22T08:25:27.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrtod.c' 2023-06-22T08:25:27.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/printf-args.h' 2023-06-22T08:25:27.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/vasprintf.c' 2023-06-22T08:25:27.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/randint.c' 2023-06-22T08:25:27.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/progname.c' 2023-06-22T08:25:27.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/progname.h' 2023-06-22T08:25:27.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/se-context.in.h' 2023-06-22T08:25:27.529 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open.c' 2023-06-22T08:25:27.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/file-has-acl.c' 2023-06-22T08:25:27.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/trim.h' 2023-06-22T08:25:27.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbswidth.h' 2023-06-22T08:25:27.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/isnan.c' 2023-06-22T08:25:27.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/euidaccess.c' 2023-06-22T08:25:27.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/physmem.h' 2023-06-22T08:25:27.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/areadlink.c' 2023-06-22T08:25:27.530 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbuiter.h' 2023-06-22T08:25:27.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xmemxfrm.h' 2023-06-22T08:25:27.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fd-reopen.c' 2023-06-22T08:25:27.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/rand-isaac.h' 2023-06-22T08:25:27.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/savewd.c' 2023-06-22T08:25:27.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xfts.c' 2023-06-22T08:25:27.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getdate.c' 2023-06-22T08:25:27.531 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/diacrit.h' 2023-06-22T08:25:27.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xmemcoll.h' 2023-06-22T08:25:27.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/isnanf.c' 2023-06-22T08:25:27.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/savewd.h' 2023-06-22T08:25:27.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/gethostname.c' 2023-06-22T08:25:27.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/alloca.in.h' 2023-06-22T08:25:27.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getdelim.c' 2023-06-22T08:25:27.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mpsort.c' 2023-06-22T08:25:27.532 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/config.hin' 2023-06-22T08:25:27.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/close-stream.h' 2023-06-22T08:25:27.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/savedir.h' 2023-06-22T08:25:27.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/hash-triple.h' 2023-06-22T08:25:27.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/trim.c' 2023-06-22T08:25:27.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/physmem.c' 2023-06-22T08:25:27.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fchdir.c' 2023-06-22T08:25:27.533 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdarg.in.h' 2023-06-22T08:25:27.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strintcmp.c' 2023-06-22T08:25:27.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stpncpy.c' 2023-06-22T08:25:27.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xalloc.h' 2023-06-22T08:25:27.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/timespec.h' 2023-06-22T08:25:27.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/readtokens.h' 2023-06-22T08:25:27.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fsusage.h' 2023-06-22T08:25:27.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/linebuffer.c' 2023-06-22T08:25:27.534 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memcoll.h' 2023-06-22T08:25:27.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/string.in.h' 2023-06-22T08:25:27.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/quote.h' 2023-06-22T08:25:27.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strstr.c' 2023-06-22T08:25:27.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/realloc.c' 2023-06-22T08:25:27.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdlib-safer.h' 2023-06-22T08:25:27.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/userspec.c' 2023-06-22T08:25:27.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/siglist.h' 2023-06-22T08:25:27.535 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/safe-read.h' 2023-06-22T08:25:27.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/filenamecat.c' 2023-06-22T08:25:27.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbrtowc.c' 2023-06-22T08:25:27.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getndelim2.h' 2023-06-22T08:25:27.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/argv-iter.h' 2023-06-22T08:25:27.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/openat-proc.c' 2023-06-22T08:25:27.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdio-safer.h' 2023-06-22T08:25:27.536 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open-osf.gperf' 2023-06-22T08:25:27.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getugroups.h' 2023-06-22T08:25:27.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdio.in.h' 2023-06-22T08:25:27.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memcmp.c' 2023-06-22T08:25:27.537 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/tempname.c' 2023-06-22T08:25:27.835 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.833+0000 7fd6933ff700 1 Processor -- start 2023-06-22T08:25:27.836 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.833+0000 7fd6933ff700 1 -- start start 2023-06-22T08:25:27.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.835+0000 7fd6933ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd68c069eb0 0x7fd68c06d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:27.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.835+0000 7fd6933ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 0x7fd68c06a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:27.838 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.836+0000 7fd6933ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68c0c4d70 con 0x7fd68c0c5aa0 2023-06-22T08:25:27.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.836+0000 7fd6933ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68c0703c0 con 0x7fd68c069eb0 2023-06-22T08:25:27.839 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.836+0000 7fd6933ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c06a7c0 con 0x7fd68c15d9b0 2023-06-22T08:25:27.840 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.836+0000 7fd69199c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd68c0c5aa0 0x7fd68c06fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:50268/0 (socket says 172.21.15.23:50268) 2023-06-22T08:25:27.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.836+0000 7fd69199c700 1 -- 172.21.15.23:0/2361672625 learned_addr learned my addr 172.21.15.23:0/2361672625 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:25:27.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.836+0000 7fd69099a700 1 --2- 172.21.15.23:0/2361672625 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 0x7fd68c06a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:27.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.837+0000 7fd69119b700 1 --2- 172.21.15.23:0/2361672625 >> v2:172.21.15.67:3300/0 conn(0x7fd68c069eb0 0x7fd68c06d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:27.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.837+0000 7fd69099a700 1 -- 172.21.15.23:0/2361672625 >> v2:172.21.15.67:3300/0 conn(0x7fd68c069eb0 msgr2=0x7fd68c06d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:27.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.837+0000 7fd69099a700 1 --2- 172.21.15.23:0/2361672625 >> v2:172.21.15.67:3300/0 conn(0x7fd68c069eb0 0x7fd68c06d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:27.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.837+0000 7fd69099a700 1 -- 172.21.15.23:0/2361672625 >> v1:172.21.15.67:6789/0 conn(0x7fd68c0c5aa0 legacy=0x7fd68c06fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:27.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.837+0000 7fd69099a700 1 -- 172.21.15.23:0/2361672625 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c0c5610 con 0x7fd68c15d9b0 2023-06-22T08:25:27.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.838+0000 7fd69099a700 1 --2- 172.21.15.23:0/2361672625 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 0x7fd68c06a280 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009840 tx=0x7fd67c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3b4df116070e306 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:27.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.839+0000 7fd683fff700 1 -- 172.21.15.23:0/2361672625 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c004230 con 0x7fd68c15d9b0 2023-06-22T08:25:27.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.840+0000 7fd683fff700 1 -- 172.21.15.23:0/2361672625 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67c004390 con 0x7fd68c15d9b0 2023-06-22T08:25:27.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.840+0000 7fd683fff700 1 -- 172.21.15.23:0/2361672625 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c0036a0 con 0x7fd68c15d9b0 2023-06-22T08:25:27.844 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.841+0000 7fd6933ff700 1 -- 172.21.15.23:0/2361672625 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 msgr2=0x7fd68c06a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:27.845 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.841+0000 7fd6933ff700 1 --2- 172.21.15.23:0/2361672625 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 0x7fd68c06a280 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009840 tx=0x7fd67c009b50 comp rx=0 tx=0).stop 2023-06-22T08:25:27.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.841+0000 7fd6933ff700 1 -- 172.21.15.23:0/2361672625 shutdown_connections 2023-06-22T08:25:27.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.841+0000 7fd6933ff700 1 --2- 172.21.15.23:0/2361672625 >> v2:172.21.15.67:3300/0 conn(0x7fd68c069eb0 0x7fd68c06d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:27.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.841+0000 7fd6933ff700 1 --2- 172.21.15.23:0/2361672625 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 0x7fd68c06a280 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:27.846 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.841+0000 7fd6933ff700 1 -- 172.21.15.23:0/2361672625 >> 172.21.15.23:0/2361672625 conn(0x7fd68c0ca3f0 msgr2=0x7fd68c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:27.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.841+0000 7fd6933ff700 1 -- 172.21.15.23:0/2361672625 shutdown_connections 2023-06-22T08:25:27.847 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.841+0000 7fd6933ff700 1 -- 172.21.15.23:0/2361672625 wait complete. 2023-06-22T08:25:27.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.844+0000 7fd6933ff700 1 Processor -- start 2023-06-22T08:25:27.848 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.844+0000 7fd6933ff700 1 -- start start 2023-06-22T08:25:27.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.846+0000 7fd6933ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd68c069eb0 0x7fd68c1f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:27.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.846+0000 7fd6933ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd68c0c5aa0 0x7fd68c152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:27.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.846+0000 7fd6933ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 0x7fd68c0c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:27.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.846+0000 7fd6933ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c0c8960 con 0x7fd68c0c5aa0 2023-06-22T08:25:27.849 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.846+0000 7fd6933ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c1f2810 con 0x7fd68c069eb0 2023-06-22T08:25:27.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.846+0000 7fd6933ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c1f37e0 con 0x7fd68c15d9b0 2023-06-22T08:25:27.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.846+0000 7fd69199c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 0x7fd68c0c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:27.850 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.846+0000 7fd69199c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 0x7fd68c0c8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:46748/0 (socket says 172.21.15.23:46748) 2023-06-22T08:25:27.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.847+0000 7fd69199c700 1 -- 172.21.15.23:0/3098985616 learned_addr learned my addr 172.21.15.23:0/3098985616 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:27.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.847+0000 7fd69119b700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd68c069eb0 0x7fd68c1f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:27.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.847+0000 7fd69099a700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd68c0c5aa0 0x7fd68c152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:27.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.847+0000 7fd69119b700 1 -- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 msgr2=0x7fd68c0c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:27.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.847+0000 7fd69119b700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 0x7fd68c0c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:27.851 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.847+0000 7fd69119b700 1 -- 172.21.15.23:0/3098985616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd68c0c5aa0 msgr2=0x7fd68c152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:27.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.847+0000 7fd69119b700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd68c0c5aa0 0x7fd68c152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:27.852 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.847+0000 7fd69119b700 1 -- 172.21.15.23:0/3098985616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c0094a0 con 0x7fd68c069eb0 2023-06-22T08:25:27.854 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.850+0000 7fd69119b700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd68c069eb0 0x7fd68c1f6790 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd68800c820 tx=0x7fd68800cb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:27.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.851+0000 7fd681ffb700 1 -- 172.21.15.23:0/3098985616 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688004220 con 0x7fd68c069eb0 2023-06-22T08:25:27.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.851+0000 7fd681ffb700 1 -- 172.21.15.23:0/3098985616 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd688004380 con 0x7fd68c069eb0 2023-06-22T08:25:27.855 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.851+0000 7fd681ffb700 1 -- 172.21.15.23:0/3098985616 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6880036a0 con 0x7fd68c069eb0 2023-06-22T08:25:27.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.851+0000 7fd6933ff700 1 -- 172.21.15.23:0/3098985616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c1f6cd0 con 0x7fd68c069eb0 2023-06-22T08:25:27.856 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.851+0000 7fd6933ff700 1 -- 172.21.15.23:0/3098985616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c1f3350 con 0x7fd68c069eb0 2023-06-22T08:25:27.857 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.853+0000 7fd6933ff700 1 -- 172.21.15.23:0/3098985616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68c04ea20 con 0x7fd68c069eb0 2023-06-22T08:25:27.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.865+0000 7fd681ffb700 1 -- 172.21.15.23:0/3098985616 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fd688003800 con 0x7fd68c069eb0 2023-06-22T08:25:27.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.867+0000 7fd681ffb700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd67006ad30 0x7fd67006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:27.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.868+0000 7fd681ffb700 1 -- 172.21.15.23:0/3098985616 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8017+0+0 (secure 0 0 0) 0x7fd688089a50 con 0x7fd68c069eb0 2023-06-22T08:25:27.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.868+0000 7fd681ffb700 1 -- 172.21.15.23:0/3098985616 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fd6880b53b0 con 0x7fd68c069eb0 2023-06-22T08:25:27.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.869+0000 7fd69099a700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd67006ad30 0x7fd67006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:27.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:27.873+0000 7fd69099a700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd67006ad30 0x7fd67006d1e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd67c000c00 tx=0x7fd67c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:28.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/inttostr.h' 2023-06-22T08:25:28.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/yesno.c' 2023-06-22T08:25:28.248 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/full-write.c' 2023-06-22T08:25:28.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strnumcmp.c' 2023-06-22T08:25:28.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sys_socket.in.h' 2023-06-22T08:25:28.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/arpa_inet.in.h' 2023-06-22T08:25:28.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unlocked-io.h' 2023-06-22T08:25:28.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unitypes.h' 2023-06-22T08:25:28.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/canon-host.h' 2023-06-22T08:25:28.249 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/euidaccess-stat.c' 2023-06-22T08:25:28.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/isapipe.c' 2023-06-22T08:25:28.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/quotearg.h' 2023-06-22T08:25:28.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/ref-add.sin' 2023-06-22T08:25:28.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstriconv.h' 2023-06-22T08:25:28.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fprintftime.h' 2023-06-22T08:25:28.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/c-ctype.c' 2023-06-22T08:25:28.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/float.in.h' 2023-06-22T08:25:28.250 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/cloexec.c' 2023-06-22T08:25:28.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/offtostr.c' 2023-06-22T08:25:28.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open-aix.gperf' 2023-06-22T08:25:28.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/wcwidth.c' 2023-06-22T08:25:28.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/nproc.c' 2023-06-22T08:25:28.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/gai_strerror.c' 2023-06-22T08:25:28.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fcntl.c' 2023-06-22T08:25:28.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getndelim2.c' 2023-06-22T08:25:28.251 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/w32sock.h' 2023-06-22T08:25:28.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/printf-frexpl.h' 2023-06-22T08:25:28.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getopt.in.h' 2023-06-22T08:25:28.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/asprintf.c' 2023-06-22T08:25:28.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/filemode.c' 2023-06-22T08:25:28.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strndup.c' 2023-06-22T08:25:28.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/uniwidth/cjk.h' 2023-06-22T08:25:28.252 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/uniwidth/width.c' 2023-06-22T08:25:28.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/lib/uniwidth' 2023-06-22T08:25:28.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fseterr.h' 2023-06-22T08:25:28.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/pipe-safer.c' 2023-06-22T08:25:28.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/freading.c' 2023-06-22T08:25:28.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/diacrit.c' 2023-06-22T08:25:28.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/quotearg.c' 2023-06-22T08:25:28.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/randint.h' 2023-06-22T08:25:28.253 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/inttypes.in.h' 2023-06-22T08:25:28.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/regex_internal.h' 2023-06-22T08:25:28.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dirent--.h' 2023-06-22T08:25:28.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdio-write.c' 2023-06-22T08:25:28.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/hash.c' 2023-06-22T08:25:28.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/atexit.c' 2023-06-22T08:25:28.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stat.c' 2023-06-22T08:25:28.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open-aix.h' 2023-06-22T08:25:28.254 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/posixtm.c' 2023-06-22T08:25:28.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/obstack.c' 2023-06-22T08:25:28.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/readutmp.h' 2023-06-22T08:25:28.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sig2str.h' 2023-06-22T08:25:28.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/str-two-way.h' 2023-06-22T08:25:28.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/copy-acl.c' 2023-06-22T08:25:28.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/canonicalize.c' 2023-06-22T08:25:28.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/userspec.h' 2023-06-22T08:25:28.255 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbsalign.h' 2023-06-22T08:25:28.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/wchar.in.h' 2023-06-22T08:25:28.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/filevercmp.h' 2023-06-22T08:25:28.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/regex_internal.c' 2023-06-22T08:25:28.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memcpy.c' 2023-06-22T08:25:28.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/base64.c' 2023-06-22T08:25:28.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/exclude.h' 2023-06-22T08:25:28.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/chown.c' 2023-06-22T08:25:28.256 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/alignof.h' 2023-06-22T08:25:28.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/signbitd.c' 2023-06-22T08:25:28.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/at-func.c' 2023-06-22T08:25:28.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/asnprintf.c' 2023-06-22T08:25:28.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sig2str.c' 2023-06-22T08:25:28.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mkdir.c' 2023-06-22T08:25:28.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/savedir.c' 2023-06-22T08:25:28.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strtoull.c' 2023-06-22T08:25:28.257 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/cycle-check.c' 2023-06-22T08:25:28.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/version-etc.h' 2023-06-22T08:25:28.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mkancesdirs.c' 2023-06-22T08:25:28.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unistr/u8-uctomb-aux.c' 2023-06-22T08:25:28.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unistr/u8-uctomb.c' 2023-06-22T08:25:28.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/lib/unistr' 2023-06-22T08:25:28.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/priv-set.h' 2023-06-22T08:25:28.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/time_r.c' 2023-06-22T08:25:28.258 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/at-func2.c' 2023-06-22T08:25:28.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open-solaris.h' 2023-06-22T08:25:28.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/freopen-safer.c' 2023-06-22T08:25:28.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xnanosleep.h' 2023-06-22T08:25:28.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/verify.h' 2023-06-22T08:25:28.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/vasnprintf.c' 2023-06-22T08:25:28.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getgroups.c' 2023-06-22T08:25:28.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/errno.in.h' 2023-06-22T08:25:28.259 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/long-options.h' 2023-06-22T08:25:28.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/posixver.c' 2023-06-22T08:25:28.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/netdb.in.h' 2023-06-22T08:25:28.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdlib--.h' 2023-06-22T08:25:28.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/same.h' 2023-06-22T08:25:28.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/gethrxtime.h' 2023-06-22T08:25:28.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/version-etc-fsf.c' 2023-06-22T08:25:28.260 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/printf-frexp.h' 2023-06-22T08:25:28.261 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/linebuffer.h' 2023-06-22T08:25:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.320+0000 7fd6933ff700 1 -- 172.21.15.23:0/3098985616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd68c1f6cd0 con 0x7fd68c069eb0 2023-06-22T08:25:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.322+0000 7fd681ffb700 1 -- 172.21.15.23:0/3098985616 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+22194 (secure 0 0 0) 0x7fd68c1f6cd0 con 0x7fd68c069eb0 2023-06-22T08:25:28.329 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-06-22T08:25:28.329 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":112,"fsid":"2f053100-95a5-44db-a4cd-e7710a2fac9d","created":"2023-06-22T07:59:28.703327+0000","modified":"2023-06-22T08:25:28.097512+0000","last_up_change":"2023-06-22T07:59:43.393254+0000","last_in_change":"2023-06-22T07:59:31.491476+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-22T07:59:40.617747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-22T07:59:48.190528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T07:59:53.576494+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T07:59:54.105472+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":767,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-22T07:59:57.198896+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":767,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df8cf946-2149-44cf-9e40-d646b8aca813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6801","nonce":458708400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6803","nonce":458708400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6807","nonce":458708400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":458708400},{"type":"v1","addr":"172.21.15.23:6805","nonce":458708400}]},"public_addr":"172.21.15.23:6801/458708400","cluster_addr":"172.21.15.23:6803/458708400","heartbeat_back_addr":"172.21.15.23:6807/458708400","heartbeat_front_addr":"172.21.15.23:6805/458708400","state":["exists","up"]},{"osd":1,"uuid":"078652c0-df39-43b4-af0f-79560de2d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6817","nonce":3464394342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6819","nonce":3464394342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6823","nonce":3464394342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":3464394342},{"type":"v1","addr":"172.21.15.23:6821","nonce":3464394342}]},"public_addr":"172.21.15.23:6817/3464394342","cluster_addr":"172.21.15.23:6819/3464394342","heartbeat_back_addr":"172.21.15.23:6823/3464394342","heartbeat_front_addr":"172.21.15.23:6821/3464394342","state":["exists","up"]},{"osd":2,"uuid":"16b7f36c-0516-41a2-ab01-0f32e4e263c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6809","nonce":1093903508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6811","nonce":1093903508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6815","nonce":1093903508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1093903508},{"type":"v1","addr":"172.21.15.23:6813","nonce":1093903508}]},"public_addr":"172.21.15.23:6809/1093903508","cluster_addr":"172.21.15.23:6811/1093903508","heartbeat_back_addr":"172.21.15.23:6815/1093903508","heartbeat_front_addr":"172.21.15.23:6813/1093903508","state":["exists","up"]},{"osd":3,"uuid":"82c3c1dc-3259-4a66-984e-6696d109a377","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6825","nonce":236939976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6827","nonce":236939976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6831","nonce":236939976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":236939976},{"type":"v1","addr":"172.21.15.23:6829","nonce":236939976}]},"public_addr":"172.21.15.23:6825/236939976","cluster_addr":"172.21.15.23:6827/236939976","heartbeat_back_addr":"172.21.15.23:6831/236939976","heartbeat_front_addr":"172.21.15.23:6829/236939976","state":["exists","up"]},{"osd":4,"uuid":"481e2a90-480b-459d-82b1-62b4f4758233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6811","nonce":1534166692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6815","nonce":1534166692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6821","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6823","nonce":1534166692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":1534166692},{"type":"v1","addr":"172.21.15.67:6819","nonce":1534166692}]},"public_addr":"172.21.15.67:6811/1534166692","cluster_addr":"172.21.15.67:6815/1534166692","heartbeat_back_addr":"172.21.15.67:6823/1534166692","heartbeat_front_addr":"172.21.15.67:6819/1534166692","state":["exists","up"]},{"osd":5,"uuid":"a845ba15-d7a0-4542-8367-caf4a9ac4bdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6810","nonce":1884490442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6814","nonce":1884490442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6822","nonce":1884490442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":1884490442},{"type":"v1","addr":"172.21.15.67:6818","nonce":1884490442}]},"public_addr":"172.21.15.67:6810/1884490442","cluster_addr":"172.21.15.67:6814/1884490442","heartbeat_back_addr":"172.21.15.67:6822/1884490442","heartbeat_front_addr":"172.21.15.67:6818/1884490442","state":["exists","up"]},{"osd":6,"uuid":"688057f7-a7ba-44bc-9913-0adcd507e06d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6825","nonce":3723802251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6827","nonce":3723802251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6831","nonce":3723802251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3723802251},{"type":"v1","addr":"172.21.15.67:6829","nonce":3723802251}]},"public_addr":"172.21.15.67:6825/3723802251","cluster_addr":"172.21.15.67:6827/3723802251","heartbeat_back_addr":"172.21.15.67:6831/3723802251","heartbeat_front_addr":"172.21.15.67:6829/3723802251","state":["exists","up"]},{"osd":7,"uuid":"ee654631-ffe8-4952-9160-21a40babb1f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6801","nonce":471678163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6803","nonce":471678163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6807","nonce":471678163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":471678163},{"type":"v1","addr":"172.21.15.67:6805","nonce":471678163}]},"public_addr":"172.21.15.67:6801/471678163","cluster_addr":"172.21.15.67:6803/471678163","heartbeat_back_addr":"172.21.15.67:6807/471678163","heartbeat_front_addr":"172.21.15.67:6805/471678163","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.834301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.880229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:37.084094+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.847231+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.679880+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.675796+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.720327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T07:59:36.644390+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6837/1502668188":"2023-06-23T08:24:44.731195+0000","172.21.15.23:6836/1502668188":"2023-06-23T08:24:44.731195+0000","172.21.15.67:6839/298860192":"2023-06-23T08:23:34.705291+0000","172.21.15.67:6838/298860192":"2023-06-23T08:23:34.705291+0000","172.21.15.67:6834/3080866617":"2023-06-23T08:20:29.589258+0000","172.21.15.67:6835/2064462140":"2023-06-23T08:18:54.558433+0000","172.21.15.23:6834/1707022233":"2023-06-23T08:16:24.518029+0000","172.21.15.23:6834/3365097525":"2023-06-23T08:05:56.351872+0000","172.21.15.23:6835/934828076":"2023-06-23T08:22:04.631962+0000","172.21.15.23:6832/3365097525":"2023-06-23T08:05:56.351872+0000","172.21.15.67:6835/3798178226":"2023-06-23T08:03:59.299349+0000","172.21.15.67:6839/567663545":"2023-06-23T08:11:04.427068+0000","172.21.15.23:6833/934828076":"2023-06-23T08:22:04.631962+0000","172.21.15.67:6837/2064462140":"2023-06-23T08:18:54.558433+0000","172.21.15.23:6836/2058668008":"2023-06-23T08:14:04.481860+0000","172.21.15.23:6832/1707022233":"2023-06-23T08:16:24.518029+0000","172.21.15.23:6836/1762110734":"2023-06-23T08:03:09.287156+0000","172.21.15.23:6835/3894174037":"2023-06-23T08:07:19.357578+0000","172.21.15.23:6837/1762110734":"2023-06-23T08:03:09.287156+0000","172.21.15.23:6833/3894174037":"2023-06-23T08:07:19.357578+0000","172.21.15.67:6834/1962314516":"2023-06-23T08:08:34.377921+0000","172.21.15.23:6837/2058668008":"2023-06-23T08:14:04.481860+0000","172.21.15.67:6836/1962314516":"2023-06-23T08:08:34.377921+0000","172.21.15.67:6837/3798178226":"2023-06-23T08:03:59.299349+0000","172.21.15.67:6838/567663545":"2023-06-23T08:11:04.427068+0000","172.21.15.67:6835/1551460358":"2023-06-23T08:13:09.469013+0000","172.21.15.67:6836/3080866617":"2023-06-23T08:20:29.589258+0000","172.21.15.67:6837/1551460358":"2023-06-23T08:13:09.469013+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":766,"length":2}]},{"pool":5,"snaps":[{"begin":766,"length":2}]}],"new_removed_snaps":[{"pool":4,"snaps":[{"begin":766,"length":2}]},{"pool":5,"snaps":[{"begin":766,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T08:25:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.331+0000 7fd6933ff700 1 -- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd67006ad30 msgr2=0x7fd67006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.331+0000 7fd6933ff700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd67006ad30 0x7fd67006d1e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd67c000c00 tx=0x7fd67c008030 comp rx=0 tx=0).stop 2023-06-22T08:25:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.331+0000 7fd6933ff700 1 -- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd68c069eb0 msgr2=0x7fd68c1f6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:28.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.331+0000 7fd6933ff700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd68c069eb0 0x7fd68c1f6790 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd68800c820 tx=0x7fd68800cb30 comp rx=0 tx=0).stop 2023-06-22T08:25:28.349 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.347+0000 7fd6933ff700 1 -- 172.21.15.23:0/3098985616 shutdown_connections 2023-06-22T08:25:28.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.347+0000 7fd6933ff700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd68c15d9b0 0x7fd68c0c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:28.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.347+0000 7fd6933ff700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fd67006ad30 0x7fd67006d1e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:28.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.347+0000 7fd6933ff700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd68c069eb0 0x7fd68c1f6790 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:28.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.347+0000 7fd6933ff700 1 --2- 172.21.15.23:0/3098985616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd68c0c5aa0 0x7fd68c152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:28.350 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.347+0000 7fd6933ff700 1 -- 172.21.15.23:0/3098985616 >> 172.21.15.23:0/3098985616 conn(0x7fd68c0ca3f0 msgr2=0x7fd68c0705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:28.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.347+0000 7fd6933ff700 1 -- 172.21.15.23:0/3098985616 shutdown_connections 2023-06-22T08:25:28.351 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:28.348+0000 7fd6933ff700 1 -- 172.21.15.23:0/3098985616 wait complete. 2023-06-22T08:25:28.387 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-22T07:59:40.617747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-22T07:59:48.190528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-06-22T07:59:53.576494+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-06-22T07:59:54.105472+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 767, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-06-22T07:59:57.198896+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 767, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-06-22T08:25:28.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T08:25:28.785 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xnanosleep.c' 2023-06-22T08:25:28.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/openat.c' 2023-06-22T08:25:28.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dup2.c' 2023-06-22T08:25:28.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/obstack.h' 2023-06-22T08:25:28.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/safe-read.c' 2023-06-22T08:25:28.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memchr2.valgrind' 2023-06-22T08:25:28.786 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/putenv.c' 2023-06-22T08:25:28.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/ctype.in.h' 2023-06-22T08:25:28.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unlinkat.c' 2023-06-22T08:25:28.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/gettime.c' 2023-06-22T08:25:28.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sha1.h' 2023-06-22T08:25:28.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/yesno.h' 2023-06-22T08:25:28.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/regexec.c' 2023-06-22T08:25:28.787 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/closeout.c' 2023-06-22T08:25:28.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/str-kmp.h' 2023-06-22T08:25:28.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getpass.h' 2023-06-22T08:25:28.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/cloexec.h' 2023-06-22T08:25:28.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open-hpux.h' 2023-06-22T08:25:28.788 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/close-hook.c' 2023-06-22T08:25:28.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fchmodat.c' 2023-06-22T08:25:28.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sig-handler.h' 2023-06-22T08:25:28.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/glthread/tls.h' 2023-06-22T08:25:28.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/glthread/tls.c' 2023-06-22T08:25:28.789 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/glthread/threadlib.c' 2023-06-22T08:25:28.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/glthread/lock.c' 2023-06-22T08:25:28.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/glthread/lock.h' 2023-06-22T08:25:28.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/lib/glthread' 2023-06-22T08:25:28.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getopt.c' 2023-06-22T08:25:28.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/readtokens0.c' 2023-06-22T08:25:28.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/gnulib.mk' 2023-06-22T08:25:28.790 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strtoimax.c' 2023-06-22T08:25:28.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/se-selinux.in.h' 2023-06-22T08:25:28.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/argv-iter.c' 2023-06-22T08:25:28.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/canonicalize.h' 2023-06-22T08:25:28.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fpending.c' 2023-06-22T08:25:28.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/openat-priv.h' 2023-06-22T08:25:28.791 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/closeout.h' 2023-06-22T08:25:28.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unistd--.h' 2023-06-22T08:25:28.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/close.c' 2023-06-22T08:25:28.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memchr.valgrind' 2023-06-22T08:25:28.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/ignore-value.h' 2023-06-22T08:25:28.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getcwd.c' 2023-06-22T08:25:28.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mkancesdirs.h' 2023-06-22T08:25:28.792 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbrlen.c' 2023-06-22T08:25:28.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/randread.c' 2023-06-22T08:25:28.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dirname.c' 2023-06-22T08:25:28.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/chdir-long.h' 2023-06-22T08:25:28.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/areadlink.h' 2023-06-22T08:25:28.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stat-time.h' 2023-06-22T08:25:28.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/save-cwd.h' 2023-06-22T08:25:28.793 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbsstr.c' 2023-06-22T08:25:28.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/open-safer.c' 2023-06-22T08:25:28.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mkdirat.c' 2023-06-22T08:25:28.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xgethostname.h' 2023-06-22T08:25:28.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/langinfo.in.h' 2023-06-22T08:25:28.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/rmdir.c' 2023-06-22T08:25:28.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fopen.c' 2023-06-22T08:25:28.794 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/close-stream.c' 2023-06-22T08:25:28.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unicodeio.c' 2023-06-22T08:25:28.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open-hpux.gperf' 2023-06-22T08:25:28.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/wctype.in.h' 2023-06-22T08:25:28.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbsinit.c' 2023-06-22T08:25:28.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/streq.h' 2023-06-22T08:25:28.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/open.c' 2023-06-22T08:25:28.795 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/freopen.c' 2023-06-22T08:25:28.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/posixtm.h' 2023-06-22T08:25:28.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dirfd.c' 2023-06-22T08:25:28.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fcntl-safer.h' 2023-06-22T08:25:28.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/isapipe.h' 2023-06-22T08:25:28.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrtold.c' 2023-06-22T08:25:28.796 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mgetgroups.h' 2023-06-22T08:25:28.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/quote.c' 2023-06-22T08:25:28.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strcspn.c' 2023-06-22T08:25:28.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/ChangeLog-2007' 2023-06-22T08:25:28.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mpsort.h' 2023-06-22T08:25:28.797 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/inttostr.c' 2023-06-22T08:25:28.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stpcpy.c' 2023-06-22T08:25:28.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/umaxtostr.c' 2023-06-22T08:25:28.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/printf-frexp.c' 2023-06-22T08:25:28.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/buffer-lcm.h' 2023-06-22T08:25:28.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/regcomp.c' 2023-06-22T08:25:28.798 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/openat-die.c' 2023-06-22T08:25:28.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/tempname.h' 2023-06-22T08:25:28.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getdtablesize.c' 2023-06-22T08:25:28.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strsignal.c' 2023-06-22T08:25:28.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getfilecon.c' 2023-06-22T08:25:28.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/utimens.h' 2023-06-22T08:25:28.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/snprintf.c' 2023-06-22T08:25:28.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fts-cycle.c' 2023-06-22T08:25:28.799 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fpending.h' 2023-06-22T08:25:28.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/group-member.h' 2023-06-22T08:25:28.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrtod.h' 2023-06-22T08:25:28.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getusershell.c' 2023-06-22T08:25:28.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/ref-del.sin' 2023-06-22T08:25:28.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/frexp.c' 2023-06-22T08:25:28.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getpagesize.c' 2023-06-22T08:25:28.800 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dirname.h' 2023-06-22T08:25:28.801 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sockets.c' 2023-06-22T08:25:29.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.017+0000 7fc6ee1ff700 1 Processor -- start 2023-06-22T08:25:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.025+0000 7fc6ee1ff700 1 -- start start 2023-06-22T08:25:29.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.031+0000 7fc6ee1ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc6e80bd670 0x7fc6e815f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:29.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.031+0000 7fc6ee1ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e815cdc0 con 0x7fc6e806d200 2023-06-22T08:25:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.031+0000 7fc6ee1ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e80c5c20 con 0x7fc6e806d5d0 2023-06-22T08:25:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.031+0000 7fc6ee1ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e815cb40 con 0x7fc6e80bd670 2023-06-22T08:25:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.035+0000 7fc6e7fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc6e80bd670 0x7fc6e815f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.036+0000 7fc6e7fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc6e80bd670 0x7fc6e815f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:34238/0 (socket says 172.21.15.23:34238) 2023-06-22T08:25:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.036+0000 7fc6e7fff700 1 -- 172.21.15.23:0/1737296133 learned_addr learned my addr 172.21.15.23:0/1737296133 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.041+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812710710 0 0) 0x7fc6e80c5c20 con 0x7fc6e806d5d0 2023-06-22T08:25:29.044 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.041+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d00035b0 con 0x7fc6e806d5d0 2023-06-22T08:25:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.042+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 <== mon.1 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3283349281 0 0) 0x7fc6d00035b0 con 0x7fc6e806d5d0 2023-06-22T08:25:29.045 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.042+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 >> v2:172.21.15.67:3300/0 conn(0x7fc6e80bd670 msgr2=0x7fc6e815f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.042+0000 7fc6e67fc700 1 --2- 172.21.15.23:0/1737296133 >> v2:172.21.15.67:3300/0 conn(0x7fc6e80bd670 0x7fc6e815f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.042+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 >> v1:172.21.15.23:6789/0 conn(0x7fc6e806d200 legacy=0x7fc6e806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.042+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e80c50a0 con 0x7fc6e806d5d0 2023-06-22T08:25:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.042+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 <== mon.1 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3131165067 0 0) 0x7fc6dc001670 con 0x7fc6e806d5d0 2023-06-22T08:25:29.047 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.043+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 >> v1:172.21.15.67:6789/0 conn(0x7fc6e806d5d0 legacy=0x7fc6e815d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:29.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.043+0000 7fc6e67fc700 1 --2- 172.21.15.23:0/1737296133 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6d0003a50 0x7fc6d0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:29.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.043+0000 7fc6e67fc700 1 --2- 172.21.15.23:0/1737296133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d0004850 0x7fc6d0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:29.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.043+0000 7fc6e67fc700 1 --2- 172.21.15.23:0/1737296133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d0005200 0x7fc6d000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:29.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.043+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d000c090 con 0x7fc6d0005200 2023-06-22T08:25:29.050 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.043+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d000c200 con 0x7fc6d0004850 2023-06-22T08:25:29.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.043+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d000c3c0 con 0x7fc6d0003a50 2023-06-22T08:25:29.051 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.044+0000 7fc6e77fe700 1 --2- 172.21.15.23:0/1737296133 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6d0003a50 0x7fc6d0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:29.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.045+0000 7fc6e77fe700 1 -- 172.21.15.23:0/1737296133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d0004850 msgr2=0x7fc6d0004cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-06-22T08:25:29.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.045+0000 7fc6e77fe700 1 --2- 172.21.15.23:0/1737296133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d0004850 0x7fc6d0004cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:29.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.045+0000 7fc6e77fe700 1 -- 172.21.15.23:0/1737296133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d0005200 msgr2=0x7fc6d000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:29.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.045+0000 7fc6e77fe700 1 --2- 172.21.15.23:0/1737296133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6d0005200 0x7fc6d000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:29.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.045+0000 7fc6e77fe700 1 -- 172.21.15.23:0/1737296133 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d00035b0 con 0x7fc6d0003a50 2023-06-22T08:25:29.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.045+0000 7fc6e7fff700 1 -- 172.21.15.23:0/1737296133 reap_dead start 2023-06-22T08:25:29.052 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.046+0000 7fc6e77fe700 1 --2- 172.21.15.23:0/1737296133 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6d0003a50 0x7fc6d0003e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80017c0 tx=0x7fc6d0005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c4b4bc1358a564c server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:29.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.050+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d80052d0 con 0x7fc6d0003a50 2023-06-22T08:25:29.053 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.050+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d8005430 con 0x7fc6d0003a50 2023-06-22T08:25:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.050+0000 7fc6e67fc700 1 -- 172.21.15.23:0/1737296133 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d80038c0 con 0x7fc6d0003a50 2023-06-22T08:25:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.051+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1737296133 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6d0003a50 msgr2=0x7fc6d0003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.051+0000 7fc6ee1ff700 1 --2- 172.21.15.23:0/1737296133 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6d0003a50 0x7fc6d0003e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80017c0 tx=0x7fc6d0005880 comp rx=0 tx=0).stop 2023-06-22T08:25:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.052+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1737296133 shutdown_connections 2023-06-22T08:25:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.052+0000 7fc6ee1ff700 1 --2- 172.21.15.23:0/1737296133 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6d0003a50 0x7fc6d0003e60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.052+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1737296133 >> 172.21.15.23:0/1737296133 conn(0x7fc6e80cb070 msgr2=0x7fc6e80c6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.052+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1737296133 shutdown_connections 2023-06-22T08:25:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.052+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1737296133 wait complete. 2023-06-22T08:25:29.059 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.056+0000 7fc6ee1ff700 1 Processor -- start 2023-06-22T08:25:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.058+0000 7fc6ee1ff700 1 -- start start 2023-06-22T08:25:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.059+0000 7fc6ee1ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d0003a50 0x7fc6e81f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.059+0000 7fc6ee1ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e81f3e70 0x7fc6e81f3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.059+0000 7fc6ee1ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6e815c1b0 0x7fc6e81e6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.059+0000 7fc6ee1ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e81f2a00 con 0x7fc6e81f3e70 2023-06-22T08:25:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.060+0000 7fc6ee1ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e80c3640 con 0x7fc6d0003a50 2023-06-22T08:25:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.060+0000 7fc6ee1ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e81f6e90 con 0x7fc6e815c1b0 2023-06-22T08:25:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.061+0000 7fc6e6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e81f3e70 0x7fc6e81f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.061+0000 7fc6e6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e81f3e70 0x7fc6e81f3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38706/0 (socket says 172.21.15.23:38706) 2023-06-22T08:25:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.061+0000 7fc6e6ffd700 1 -- 172.21.15.23:0/1696551529 learned_addr learned my addr 172.21.15.23:0/1696551529 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.062+0000 7fc6e6ffd700 1 -- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6e815c1b0 msgr2=0x7fc6e81e6570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T08:25:29.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.062+0000 7fc6e6ffd700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6e815c1b0 0x7fc6e81e6570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:29.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.062+0000 7fc6e6ffd700 1 -- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d0003a50 msgr2=0x7fc6e81f6950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T08:25:29.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.062+0000 7fc6e6ffd700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d0003a50 0x7fc6e81f6950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:29.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.062+0000 7fc6e6ffd700 1 -- 172.21.15.23:0/1696551529 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e81f3950 con 0x7fc6e81f3e70 2023-06-22T08:25:29.066 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.062+0000 7fc6e6ffd700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e81f3e70 0x7fc6e81f3410 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00c760 tx=0x7fc6dc00cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.063+0000 7fc6cffff700 1 -- 172.21.15.23:0/1696551529 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc005480 con 0x7fc6e81f3e70 2023-06-22T08:25:29.068 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.064+0000 7fc6cffff700 1 -- 172.21.15.23:0/1696551529 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc003680 con 0x7fc6e81f3e70 2023-06-22T08:25:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.064+0000 7fc6cffff700 1 -- 172.21.15.23:0/1696551529 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc003a30 con 0x7fc6e81f3e70 2023-06-22T08:25:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.064+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1696551529 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e80c3890 con 0x7fc6e81f3e70 2023-06-22T08:25:29.069 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.064+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1696551529 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e80c3e50 con 0x7fc6e81f3e70 2023-06-22T08:25:29.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.070+0000 7fc6cffff700 1 -- 172.21.15.23:0/1696551529 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fc6dc003b90 con 0x7fc6e81f3e70 2023-06-22T08:25:29.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.072+0000 7fc6cffff700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6d406ade0 0x7fc6d406d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:29.080 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.072+0000 7fc6cffff700 1 -- 172.21.15.23:0/1696551529 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8129+0+0 (secure 0 0 0) 0x7fc6dc089380 con 0x7fc6e81f3e70 2023-06-22T08:25:29.085 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.076+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1696551529 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c80052f0 con 0x7fc6e81f3e70 2023-06-22T08:25:29.085 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.077+0000 7fc6e77fe700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6d406ade0 0x7fc6d406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:29.092 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.087+0000 7fc6e77fe700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6d406ade0 0x7fc6d406d290 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8005040 tx=0x7fc6d8009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:29.096 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.090+0000 7fc6cffff700 1 -- 172.21.15.23:0/1696551529 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fc6dc017020 con 0x7fc6e81f3e70 2023-06-22T08:25:29.463 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getopt_int.h' 2023-06-22T08:25:29.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/freadahead.h' 2023-06-22T08:25:29.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strftime.h' 2023-06-22T08:25:29.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/close-hook.h' 2023-06-22T08:25:29.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/raise.c' 2023-06-22T08:25:29.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fpucw.h' 2023-06-22T08:25:29.464 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/localcharset.c' 2023-06-22T08:25:29.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/lstat.c' 2023-06-22T08:25:29.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/rename.c' 2023-06-22T08:25:29.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/freadahead.c' 2023-06-22T08:25:29.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/argmatch.h' 2023-06-22T08:25:29.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fchown-stub.c' 2023-06-22T08:25:29.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/creat-safer.c' 2023-06-22T08:25:29.465 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strtoul.c' 2023-06-22T08:25:29.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xprintf.c' 2023-06-22T08:25:29.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mountlist.c' 2023-06-22T08:25:29.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mkdir-p.c' 2023-06-22T08:25:29.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/utimecmp.c' 2023-06-22T08:25:29.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mountlist.h' 2023-06-22T08:25:29.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/isnanl.c' 2023-06-22T08:25:29.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/randperm.h' 2023-06-22T08:25:29.466 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/remove.c' 2023-06-22T08:25:29.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dirent.in.h' 2023-06-22T08:25:29.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/malloc.c' 2023-06-22T08:25:29.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memrchr.c' 2023-06-22T08:25:29.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dirchownmod.h' 2023-06-22T08:25:29.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/argmatch.c' 2023-06-22T08:25:29.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/Makefile.in' 2023-06-22T08:25:29.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strnumcmp.h' 2023-06-22T08:25:29.467 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/c-strtold.c' 2023-06-22T08:25:29.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/freadptr.c' 2023-06-22T08:25:29.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/version-etc.c' 2023-06-22T08:25:29.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/file-type.h' 2023-06-22T08:25:29.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/same-inode.h' 2023-06-22T08:25:29.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xvasprintf.h' 2023-06-22T08:25:29.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/i-ring.c' 2023-06-22T08:25:29.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/c-strncasecmp.c' 2023-06-22T08:25:29.468 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/posixver.h' 2023-06-22T08:25:29.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/locale.in.h' 2023-06-22T08:25:29.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unistr.h' 2023-06-22T08:25:29.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xreadlink.c' 2023-06-22T08:25:29.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/human.c' 2023-06-22T08:25:29.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sha512.c' 2023-06-22T08:25:29.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open-solaris.gperf' 2023-06-22T08:25:29.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fdopendir.c' 2023-06-22T08:25:29.469 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/regex.c' 2023-06-22T08:25:29.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getaddrinfo.c' 2023-06-22T08:25:29.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/acl_entries.c' 2023-06-22T08:25:29.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getopt1.c' 2023-06-22T08:25:29.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/selinux-at.h' 2023-06-22T08:25:29.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strnlen1.c' 2023-06-22T08:25:29.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/readtokens0.h' 2023-06-22T08:25:29.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/symlink.c' 2023-06-22T08:25:29.470 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fclose.c' 2023-06-22T08:25:29.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/freadseek.h' 2023-06-22T08:25:29.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/safe-write.h' 2023-06-22T08:25:29.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fsusage.c' 2023-06-22T08:25:29.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/basename-lgpl.c' 2023-06-22T08:25:29.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/nproc.h' 2023-06-22T08:25:29.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stat-macros.h' 2023-06-22T08:25:29.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrtoimax.c' 2023-06-22T08:25:29.471 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xmemcoll.c' 2023-06-22T08:25:29.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dirent-safer.h' 2023-06-22T08:25:29.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbchar.c' 2023-06-22T08:25:29.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memchr2.h' 2023-06-22T08:25:29.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strdup.c' 2023-06-22T08:25:29.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strnlen.c' 2023-06-22T08:25:29.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/c-strtod.h' 2023-06-22T08:25:29.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getloadavg.c' 2023-06-22T08:25:29.472 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/file-type.c' 2023-06-22T08:25:29.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/i-ring.h' 2023-06-22T08:25:29.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fstatat.c' 2023-06-22T08:25:29.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sha512.h' 2023-06-22T08:25:29.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/gettimeofday.c' 2023-06-22T08:25:29.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbsrtowcs-state.c' 2023-06-22T08:25:29.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xmalloc.c' 2023-06-22T08:25:29.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memcoll.c' 2023-06-22T08:25:29.473 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrtoul.c' 2023-06-22T08:25:29.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fts_.h' 2023-06-22T08:25:29.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/c-ctype.h' 2023-06-22T08:25:29.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stripslash.c' 2023-06-22T08:25:29.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open-osf.h' 2023-06-22T08:25:29.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/wcrtomb.c' 2023-06-22T08:25:29.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/utimecmp.h' 2023-06-22T08:25:29.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdio--.h' 2023-06-22T08:25:29.474 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/randperm.c' 2023-06-22T08:25:29.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/areadlinkat.c' 2023-06-22T08:25:29.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/readutmp.c' 2023-06-22T08:25:29.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unicodeio.h' 2023-06-22T08:25:29.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/full-read.h' 2023-06-22T08:25:29.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fseterr.c' 2023-06-22T08:25:29.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/intprops.h' 2023-06-22T08:25:29.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/pathmax.h' 2023-06-22T08:25:29.475 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/utimens.c' 2023-06-22T08:25:29.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/propername.h' 2023-06-22T08:25:29.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/acl-internal.h' 2023-06-22T08:25:29.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/safe-write.c' 2023-06-22T08:25:29.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strtoll.c' 2023-06-22T08:25:29.476 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mkstemp.c' 2023-06-22T08:25:29.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.586+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1696551529 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc6c8005b20 con 0x7fc6e81f3e70 2023-06-22T08:25:29.589 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.587+0000 7fc6cffff700 1 -- 172.21.15.23:0/1696551529 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6dc053b60 con 0x7fc6e81f3e70 2023-06-22T08:25:29.591 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2023-06-22T08:25:29.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.591+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6d406ade0 msgr2=0x7fc6d406d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:29.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.592+0000 7fc6ee1ff700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6d406ade0 0x7fc6d406d290 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8005040 tx=0x7fc6d8009030 comp rx=0 tx=0).stop 2023-06-22T08:25:29.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.592+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1696551529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e81f3e70 msgr2=0x7fc6e81f3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:29.594 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.592+0000 7fc6ee1ff700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e81f3e70 0x7fc6e81f3410 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00c760 tx=0x7fc6dc00cb20 comp rx=0 tx=0).stop 2023-06-22T08:25:29.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.592+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1696551529 shutdown_connections 2023-06-22T08:25:29.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.592+0000 7fc6ee1ff700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6e815c1b0 0x7fc6e81e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:29.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.592+0000 7fc6ee1ff700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fc6d406ade0 0x7fc6d406d290 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:29.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.592+0000 7fc6ee1ff700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d0003a50 0x7fc6e81f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:29.595 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.592+0000 7fc6ee1ff700 1 --2- 172.21.15.23:0/1696551529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6e81f3e70 0x7fc6e81f3410 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:29.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.592+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1696551529 >> 172.21.15.23:0/1696551529 conn(0x7fc6e80cb070 msgr2=0x7fc6e815aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:29.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.593+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1696551529 shutdown_connections 2023-06-22T08:25:29.596 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:29.593+0000 7fc6ee1ff700 1 -- 172.21.15.23:0/1696551529 wait complete. 2023-06-22T08:25:29.636 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T08:25:30.303 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/Makefile.am' 2023-06-22T08:25:30.303 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/file-set.h' 2023-06-22T08:25:30.303 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/md5.c' 2023-06-22T08:25:30.303 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/base64.h' 2023-06-22T08:25:30.304 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open-irix.gperf' 2023-06-22T08:25:30.304 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/vasnprintf.h' 2023-06-22T08:25:30.304 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrndup.h' 2023-06-22T08:25:30.304 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/hash-pjw.c' 2023-06-22T08:25:30.304 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sys_time.in.h' 2023-06-22T08:25:30.304 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strtol.c' 2023-06-22T08:25:30.304 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/backupfile.h' 2023-06-22T08:25:30.305 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/setenv.c' 2023-06-22T08:25:30.305 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sys_select.in.h' 2023-06-22T08:25:30.305 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/config.charset' 2023-06-22T08:25:30.305 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xgetcwd.h' 2023-06-22T08:25:30.305 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fnmatch_loop.c' 2023-06-22T08:25:30.305 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/hash-triple.c' 2023-06-22T08:25:30.305 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/hash-pjw.h' 2023-06-22T08:25:30.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memcmp2.c' 2023-06-22T08:25:30.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/write.c' 2023-06-22T08:25:30.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/modechange.h' 2023-06-22T08:25:30.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unlink.c' 2023-06-22T08:25:30.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdbool.in.h' 2023-06-22T08:25:30.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/time.in.h' 2023-06-22T08:25:30.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/freadptr.h' 2023-06-22T08:25:30.306 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/isblank.c' 2023-06-22T08:25:30.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getline.c' 2023-06-22T08:25:30.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/math.in.h' 2023-06-22T08:25:30.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/buffer-lcm.c' 2023-06-22T08:25:30.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/exclude.c' 2023-06-22T08:25:30.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/filevercmp.c' 2023-06-22T08:25:30.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/closein.c' 2023-06-22T08:25:30.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/closein.h' 2023-06-22T08:25:30.307 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xfts.h' 2023-06-22T08:25:30.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/file-set.c' 2023-06-22T08:25:30.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unistd-safer.h' 2023-06-22T08:25:30.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/uniwidth.h' 2023-06-22T08:25:30.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/symlinkat.c' 2023-06-22T08:25:30.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/readtokens.c' 2023-06-22T08:25:30.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/exitfail.h' 2023-06-22T08:25:30.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/malloca.c' 2023-06-22T08:25:30.308 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fd-safer.c' 2023-06-22T08:25:30.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xsize.h' 2023-06-22T08:25:30.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strnumcmp-in.h' 2023-06-22T08:25:30.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbswidth.c' 2023-06-22T08:25:30.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fopen-safer.c' 2023-06-22T08:25:30.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrndup.c' 2023-06-22T08:25:30.309 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/human.h' 2023-06-22T08:25:30.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/gettext.h' 2023-06-22T08:25:30.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mktime-internal.h' 2023-06-22T08:25:30.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fchownat.c' 2023-06-22T08:25:30.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dup-safer.c' 2023-06-22T08:25:30.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrtol-error.c' 2023-06-22T08:25:30.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/full-write.h' 2023-06-22T08:25:30.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dup-safer-flag.c' 2023-06-22T08:25:30.310 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/float+.h' 2023-06-22T08:25:30.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sigprocmask.c' 2023-06-22T08:25:30.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/printf-args.c' 2023-06-22T08:25:30.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/gethrxtime.c' 2023-06-22T08:25:30.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/randread.h' 2023-06-22T08:25:30.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strpbrk.c' 2023-06-22T08:25:30.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getugroups.c' 2023-06-22T08:25:30.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/chdir-long.c' 2023-06-22T08:25:30.311 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fcntl--.h' 2023-06-22T08:25:30.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mkstemp-safer.c' 2023-06-22T08:25:30.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbslen.c' 2023-06-22T08:25:30.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/bitrotate.h' 2023-06-22T08:25:30.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dev-ino.h' 2023-06-22T08:25:30.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xasprintf.c' 2023-06-22T08:25:30.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sockets.h' 2023-06-22T08:25:30.312 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/inet_ntop.c' 2023-06-22T08:25:30.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getdate.y' 2023-06-22T08:25:30.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stddef.in.h' 2023-06-22T08:25:30.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/isnand-nolibm.h' 2023-06-22T08:25:30.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbsrtowcs.c' 2023-06-22T08:25:30.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/cycle-check.h' 2023-06-22T08:25:30.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sha256.c' 2023-06-22T08:25:30.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/readlink.c' 2023-06-22T08:25:30.313 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/uname.c' 2023-06-22T08:25:30.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/printf-parse.h' 2023-06-22T08:25:30.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/printf-frexpl.c' 2023-06-22T08:25:30.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/exitfail.c' 2023-06-22T08:25:30.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/linkat.c' 2023-06-22T08:25:30.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/same.c' 2023-06-22T08:25:30.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/btowc.c' 2023-06-22T08:25:30.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memcasecmp.h' 2023-06-22T08:25:30.314 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sys_utsname.in.h' 2023-06-22T08:25:30.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/filenamecat-lgpl.c' 2023-06-22T08:25:30.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstriconv.c' 2023-06-22T08:25:30.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/rand-isaac.c' 2023-06-22T08:25:30.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrtol.h' 2023-06-22T08:25:30.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strtod.c' 2023-06-22T08:25:30.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/select.c' 2023-06-22T08:25:30.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/ftello.c' 2023-06-22T08:25:30.315 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strnlen1.h' 2023-06-22T08:25:30.316 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xtime.h' 2023-06-22T08:25:30.316 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/save-cwd.c' 2023-06-22T08:25:30.316 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sha256.h' 2023-06-22T08:25:30.374 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.372+0000 7f04507ff700 1 Processor -- start 2023-06-22T08:25:30.376 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.373+0000 7f04507ff700 1 -- start start 2023-06-22T08:25:30.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.375+0000 7f04507ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04480c9fd0 0x7f044815c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:30.380 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.375+0000 7f04507ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0448159300 0x7f04480c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:30.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.375+0000 7f04507ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04480c94f0 con 0x7f0448152e00 2023-06-22T08:25:30.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.375+0000 7f04507ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04480ca400 con 0x7f0448159300 2023-06-22T08:25:30.381 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.375+0000 7f04507ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04480c9790 con 0x7f04480c9fd0 2023-06-22T08:25:30.382 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.379+0000 7f044e59b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f0448152e00 0x7f04480c40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:38474/0 (socket says 172.21.15.23:38474) 2023-06-22T08:25:30.383 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.379+0000 7f044e59b700 1 -- 172.21.15.23:0/1441154844 learned_addr learned my addr 172.21.15.23:0/1441154844 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:25:30.384 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.382+0000 7f044ed9c700 1 --2- 172.21.15.23:0/1441154844 >> v2:172.21.15.23:3300/0 conn(0x7f0448159300 0x7f04480c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:30.385 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.382+0000 7f044dd9a700 1 --2- 172.21.15.23:0/1441154844 >> v2:172.21.15.67:3300/0 conn(0x7f04480c9fd0 0x7f044815c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:30.390 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.386+0000 7f044ed9c700 1 -- 172.21.15.23:0/1441154844 >> v2:172.21.15.67:3300/0 conn(0x7f04480c9fd0 msgr2=0x7f044815c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.386+0000 7f044ed9c700 1 --2- 172.21.15.23:0/1441154844 >> v2:172.21.15.67:3300/0 conn(0x7f04480c9fd0 0x7f044815c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.386+0000 7f044ed9c700 1 -- 172.21.15.23:0/1441154844 >> v1:172.21.15.23:6789/0 conn(0x7f0448152e00 legacy=0x7f04480c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:30.391 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.386+0000 7f044ed9c700 1 -- 172.21.15.23:0/1441154844 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04480c9a70 con 0x7f0448159300 2023-06-22T08:25:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.391+0000 7f044ed9c700 1 --2- 172.21.15.23:0/1441154844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0448159300 0x7f04480c47c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f044808c120 tx=0x7f0438009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c41c4a052a25d20 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:30.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.408+0000 7f044d599700 1 -- 172.21.15.23:0/1441154844 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438011830 con 0x7f0448159300 2023-06-22T08:25:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.408+0000 7f044d599700 1 -- 172.21.15.23:0/1441154844 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0438011990 con 0x7f0448159300 2023-06-22T08:25:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.408+0000 7f044d599700 1 -- 172.21.15.23:0/1441154844 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438011d20 con 0x7f0448159300 2023-06-22T08:25:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.409+0000 7f04507ff700 1 -- 172.21.15.23:0/1441154844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0448159300 msgr2=0x7f04480c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.409+0000 7f04507ff700 1 --2- 172.21.15.23:0/1441154844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0448159300 0x7f04480c47c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f044808c120 tx=0x7f0438009b20 comp rx=0 tx=0).stop 2023-06-22T08:25:30.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.412+0000 7f04507ff700 1 -- 172.21.15.23:0/1441154844 shutdown_connections 2023-06-22T08:25:30.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.412+0000 7f04507ff700 1 --2- 172.21.15.23:0/1441154844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0448159300 0x7f04480c47c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:30.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.412+0000 7f04507ff700 1 --2- 172.21.15.23:0/1441154844 >> v2:172.21.15.67:3300/0 conn(0x7f04480c9fd0 0x7f044815c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:30.416 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.412+0000 7f04507ff700 1 -- 172.21.15.23:0/1441154844 >> 172.21.15.23:0/1441154844 conn(0x7f044806ae30 msgr2=0x7f044806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.423+0000 7f04507ff700 1 -- 172.21.15.23:0/1441154844 shutdown_connections 2023-06-22T08:25:30.430 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.426+0000 7f04507ff700 1 -- 172.21.15.23:0/1441154844 wait complete. 2023-06-22T08:25:30.434 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.428+0000 7f04507ff700 1 Processor -- start 2023-06-22T08:25:30.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.432+0000 7f04507ff700 1 -- start start 2023-06-22T08:25:30.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.433+0000 7f04507ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04480c9fd0 0x7f044806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:30.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.433+0000 7f04507ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0448152e00 0x7f04480c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:30.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.433+0000 7f04507ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0448159300 0x7f044815b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:30.444 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.433+0000 7f04507ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480c5c30 con 0x7f0448152e00 2023-06-22T08:25:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.433+0000 7f04507ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480c8ed0 con 0x7f04480c9fd0 2023-06-22T08:25:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.433+0000 7f04507ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044806dc20 con 0x7f0448159300 2023-06-22T08:25:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.436+0000 7f044e59b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04480c9fd0 0x7f044806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.436+0000 7f044e59b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04480c9fd0 0x7f044806d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:34280/0 (socket says 172.21.15.23:34280) 2023-06-22T08:25:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.436+0000 7f044e59b700 1 -- 172.21.15.23:0/1454919382 learned_addr learned my addr 172.21.15.23:0/1454919382 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.437+0000 7f044ed9c700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0448159300 0x7f044815b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.437+0000 7f044e59b700 1 -- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0448159300 msgr2=0x7f044815b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:30.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.437+0000 7f044e59b700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0448159300 0x7f044815b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:30.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.437+0000 7f044e59b700 1 -- 172.21.15.23:0/1454919382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0448152e00 msgr2=0x7f04480c8990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T08:25:30.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.437+0000 7f044e59b700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0448152e00 0x7f04480c8990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:30.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.437+0000 7f044e59b700 1 -- 172.21.15.23:0/1454919382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04380094a0 con 0x7f04480c9fd0 2023-06-22T08:25:30.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.437+0000 7f044e59b700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04480c9fd0 0x7f044806d380 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f044000cbd0 tx=0x7f044000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:30.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.438+0000 7f043effd700 1 -- 172.21.15.23:0/1454919382 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044000ab10 con 0x7f04480c9fd0 2023-06-22T08:25:30.446 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.438+0000 7f043effd700 1 -- 172.21.15.23:0/1454919382 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044000ac70 con 0x7f04480c9fd0 2023-06-22T08:25:30.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.438+0000 7f043effd700 1 -- 172.21.15.23:0/1454919382 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440012400 con 0x7f04480c9fd0 2023-06-22T08:25:30.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.439+0000 7f04507ff700 1 -- 172.21.15.23:0/1454919382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04480c7760 con 0x7f04480c9fd0 2023-06-22T08:25:30.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.439+0000 7f04507ff700 1 -- 172.21.15.23:0/1454919382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04480c6120 con 0x7f04480c9fd0 2023-06-22T08:25:30.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.441+0000 7f04507ff700 1 -- 172.21.15.23:0/1454919382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044804ea20 con 0x7f04480c9fd0 2023-06-22T08:25:30.447 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.444+0000 7f043effd700 1 -- 172.21.15.23:0/1454919382 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f0440012560 con 0x7f04480c9fd0 2023-06-22T08:25:30.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.445+0000 7f043effd700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f042c06adb0 0x7f042c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:30.448 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.445+0000 7f043effd700 1 -- 172.21.15.23:0/1454919382 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8129+0+0 (secure 0 0 0) 0x7f044008a150 con 0x7f04480c9fd0 2023-06-22T08:25:30.449 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.446+0000 7f044dd9a700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f042c06adb0 0x7f042c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:30.453 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.451+0000 7f043effd700 1 -- 172.21.15.23:0/1454919382 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f0440054b00 con 0x7f04480c9fd0 2023-06-22T08:25:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.458+0000 7f044dd9a700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f042c06adb0 0x7f042c06d260 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f04480b1180 tx=0x7f0444008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:30.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.870+0000 7f04507ff700 1 -- 172.21.15.23:0/1454919382 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f04480b2360 con 0x7f04480c9fd0 2023-06-22T08:25:30.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.871+0000 7f043effd700 1 -- 172.21.15.23:0/1454919382 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v112) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0440017020 con 0x7f04480c9fd0 2023-06-22T08:25:30.873 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2023-06-22T08:25:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.875+0000 7f04507ff700 1 -- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f042c06adb0 msgr2=0x7f042c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.875+0000 7f04507ff700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f042c06adb0 0x7f042c06d260 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f04480b1180 tx=0x7f0444008040 comp rx=0 tx=0).stop 2023-06-22T08:25:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.875+0000 7f04507ff700 1 -- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04480c9fd0 msgr2=0x7f044806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:30.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.875+0000 7f04507ff700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04480c9fd0 0x7f044806d380 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f044000cbd0 tx=0x7f044000cf90 comp rx=0 tx=0).stop 2023-06-22T08:25:30.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.875+0000 7f04507ff700 1 -- 172.21.15.23:0/1454919382 shutdown_connections 2023-06-22T08:25:30.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.875+0000 7f04507ff700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0448159300 0x7f044815b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:30.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.875+0000 7f04507ff700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f042c06adb0 0x7f042c06d260 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:30.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.875+0000 7f04507ff700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04480c9fd0 0x7f044806d380 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:30.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.875+0000 7f04507ff700 1 --2- 172.21.15.23:0/1454919382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0448152e00 0x7f04480c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:30.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.875+0000 7f04507ff700 1 -- 172.21.15.23:0/1454919382 >> 172.21.15.23:0/1454919382 conn(0x7f044806ae30 msgr2=0x7f04480b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:30.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.878+0000 7f04507ff700 1 -- 172.21.15.23:0/1454919382 shutdown_connections 2023-06-22T08:25:30.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:30.878+0000 7f04507ff700 1 -- 172.21.15.23:0/1454919382 wait complete. 2023-06-22T08:25:30.912 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-06-22T08:25:31.075 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memxfrm.c' 2023-06-22T08:25:31.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/opendir-safer.c' 2023-06-22T08:25:31.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/size_max.h' 2023-06-22T08:25:31.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbscasecmp.c' 2023-06-22T08:25:31.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fflush.c' 2023-06-22T08:25:31.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mgetgroups.c' 2023-06-22T08:25:31.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getdate.h' 2023-06-22T08:25:31.076 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/malloca.valgrind' 2023-06-22T08:25:31.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv_open-irix.h' 2023-06-22T08:25:31.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xgetcwd.c' 2023-06-22T08:25:31.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xalloc-die.c' 2023-06-22T08:25:31.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/long-options.c' 2023-06-22T08:25:31.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/basename.c' 2023-06-22T08:25:31.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/error.c' 2023-06-22T08:25:31.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memcmp2.h' 2023-06-22T08:25:31.077 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/regex.h' 2023-06-22T08:25:31.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mkdir-p.h' 2023-06-22T08:25:31.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xstrtoumax.c' 2023-06-22T08:25:31.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dirchownmod.c' 2023-06-22T08:25:31.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/isnanl-nolibm.h' 2023-06-22T08:25:31.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/areadlink-with-size.c' 2023-06-22T08:25:31.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/printf-parse.c' 2023-06-22T08:25:31.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unistd.in.h' 2023-06-22T08:25:31.078 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strtoumax.c' 2023-06-22T08:25:31.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mempcpy.c' 2023-06-22T08:25:31.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xfreopen.c' 2023-06-22T08:25:31.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memmove.c' 2023-06-22T08:25:31.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xmemxfrm.c' 2023-06-22T08:25:31.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/link.c' 2023-06-22T08:25:31.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/priv-set.c' 2023-06-22T08:25:31.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/signbitl.c' 2023-06-22T08:25:31.079 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fnmatch.c' 2023-06-22T08:25:31.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/backupfile.c' 2023-06-22T08:25:31.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fd-safer-flag.c' 2023-06-22T08:25:31.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/signal.in.h' 2023-06-22T08:25:31.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/filenamecat.h' 2023-06-22T08:25:31.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xvasprintf.c' 2023-06-22T08:25:31.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/euidaccess-stat.h' 2023-06-22T08:25:31.080 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fsync.c' 2023-06-22T08:25:31.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/filemode.h' 2023-06-22T08:25:31.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/write-any-file.h' 2023-06-22T08:25:31.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdlib.in.h' 2023-06-22T08:25:31.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/selinux-at.c' 2023-06-22T08:25:31.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/netinet_in.in.h' 2023-06-22T08:25:31.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/u64.h' 2023-06-22T08:25:31.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/unsetenv.c' 2023-06-22T08:25:31.081 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memchr.c' 2023-06-22T08:25:31.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/verror.h' 2023-06-22T08:25:31.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/freading.h' 2023-06-22T08:25:31.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xgethostname.c' 2023-06-22T08:25:31.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/nanosleep.c' 2023-06-22T08:25:31.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xprintf.h' 2023-06-22T08:25:31.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/malloca.h' 2023-06-22T08:25:31.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/settime.c' 2023-06-22T08:25:31.082 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memchr2.c' 2023-06-22T08:25:31.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memcasecmp.c' 2023-06-22T08:25:31.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/frexpl.c' 2023-06-22T08:25:31.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/root-dev-ino.h' 2023-06-22T08:25:31.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/hard-locale.c' 2023-06-22T08:25:31.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/localcharset.h' 2023-06-22T08:25:31.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdio-impl.h' 2023-06-22T08:25:31.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/set-mode-acl.c' 2023-06-22T08:25:31.083 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/imaxtostr.c' 2023-06-22T08:25:31.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/freadseek.c' 2023-06-22T08:25:31.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/md5.h' 2023-06-22T08:25:31.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/idcache.c' 2023-06-22T08:25:31.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/calloc.c' 2023-06-22T08:25:31.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fseeko.c' 2023-06-22T08:25:31.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/propername.c' 2023-06-22T08:25:31.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/vprintf.c' 2023-06-22T08:25:31.084 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fileblocks.c' 2023-06-22T08:25:31.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memxfrm.h' 2023-06-22T08:25:31.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fprintftime.c' 2023-06-22T08:25:31.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/alloca.c' 2023-06-22T08:25:31.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/group-member.c' 2023-06-22T08:25:31.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/striconv.h' 2023-06-22T08:25:31.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbchar.h' 2023-06-22T08:25:31.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xreadlink.h' 2023-06-22T08:25:31.085 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/c-strcase.h' 2023-06-22T08:25:31.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/acl.h' 2023-06-22T08:25:31.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/modechange.c' 2023-06-22T08:25:31.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fpurge.c' 2023-06-22T08:25:31.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/openat.h' 2023-06-22T08:25:31.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/signbitf.c' 2023-06-22T08:25:31.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mktime.c' 2023-06-22T08:25:31.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbsalign.c' 2023-06-22T08:25:31.086 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/xfreopen.h' 2023-06-22T08:25:31.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/iconv.in.h' 2023-06-22T08:25:31.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/hard-locale.h' 2023-06-22T08:25:31.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/nl_langinfo.c' 2023-06-22T08:25:31.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/striconv.c' 2023-06-22T08:25:31.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sigaction.c' 2023-06-22T08:25:31.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/memset.c' 2023-06-22T08:25:31.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fd-reopen.h' 2023-06-22T08:25:31.087 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/mbiter.h' 2023-06-22T08:25:31.088 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strftime.c' 2023-06-22T08:25:31.088 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/uinttostr.c' 2023-06-22T08:25:31.088 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/canon-host.c' 2023-06-22T08:25:31.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.302+0000 7f2407bff700 1 Processor -- start 2023-06-22T08:25:31.305 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.302+0000 7f2407bff700 1 -- start start 2023-06-22T08:25:31.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.304+0000 7f2407bff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f240015c050 0x7f24000c7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:31.306 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.304+0000 7f2407bff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240015cb80 con 0x7f240015c420 2023-06-22T08:25:31.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.304+0000 7f2407bff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240006cf90 con 0x7f24000c5620 2023-06-22T08:25:31.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.304+0000 7f2407bff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f240006e140 con 0x7f240015c050 2023-06-22T08:25:31.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.304+0000 7f240619c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f24000c5620 0x7f2400152600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:50288/0 (socket says 172.21.15.23:50288) 2023-06-22T08:25:31.307 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.304+0000 7f240619c700 1 -- 172.21.15.23:0/2668848277 learned_addr learned my addr 172.21.15.23:0/2668848277 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:25:31.308 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.305+0000 7f240599b700 1 --2- 172.21.15.23:0/2668848277 >> v2:172.21.15.67:3300/0 conn(0x7f240015c050 0x7f24000c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:31.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.305+0000 7f2404999700 1 -- 172.21.15.23:0/2668848277 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139330307 0 0) 0x7f240006cf90 con 0x7f24000c5620 2023-06-22T08:25:31.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.305+0000 7f2404999700 1 -- 172.21.15.23:0/2668848277 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ec0035b0 con 0x7f24000c5620 2023-06-22T08:25:31.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.305+0000 7f2404999700 1 -- 172.21.15.23:0/2668848277 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836330491 0 0) 0x7f23f0001110 con 0x7f240015c420 2023-06-22T08:25:31.309 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.305+0000 7f2404999700 1 -- 172.21.15.23:0/2668848277 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240006cf90 con 0x7f240015c420 2023-06-22T08:25:31.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.305+0000 7f240599b700 1 -- 172.21.15.23:0/2668848277 >> v1:172.21.15.67:6789/0 conn(0x7f24000c5620 legacy=0x7f2400152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:31.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.305+0000 7f240599b700 1 -- 172.21.15.23:0/2668848277 >> v1:172.21.15.23:6789/0 conn(0x7f240015c420 legacy=0x7f24000c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:31.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.305+0000 7f240599b700 1 -- 172.21.15.23:0/2668848277 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240006d1e0 con 0x7f240015c050 2023-06-22T08:25:31.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.306+0000 7f240599b700 1 --2- 172.21.15.23:0/2668848277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f240015c050 0x7f24000c7900 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009840 tx=0x7f23fc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f73699b1e672532b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:31.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.306+0000 7f2404999700 1 -- 172.21.15.23:0/2668848277 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc011830 con 0x7f240015c050 2023-06-22T08:25:31.310 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.306+0000 7f2404999700 1 -- 172.21.15.23:0/2668848277 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc011990 con 0x7f240015c050 2023-06-22T08:25:31.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.306+0000 7f2404999700 1 -- 172.21.15.23:0/2668848277 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc011d20 con 0x7f240015c050 2023-06-22T08:25:31.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.307+0000 7f2407bff700 1 -- 172.21.15.23:0/2668848277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f240015c050 msgr2=0x7f24000c7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:31.311 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.307+0000 7f2407bff700 1 --2- 172.21.15.23:0/2668848277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f240015c050 0x7f24000c7900 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009840 tx=0x7f23fc009b50 comp rx=0 tx=0).stop 2023-06-22T08:25:31.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.307+0000 7f2407bff700 1 -- 172.21.15.23:0/2668848277 shutdown_connections 2023-06-22T08:25:31.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.307+0000 7f2407bff700 1 --2- 172.21.15.23:0/2668848277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f240015c050 0x7f24000c7900 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:31.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.307+0000 7f2407bff700 1 -- 172.21.15.23:0/2668848277 >> 172.21.15.23:0/2668848277 conn(0x7f24000cb070 msgr2=0x7f24000c6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:31.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.307+0000 7f2407bff700 1 -- 172.21.15.23:0/2668848277 shutdown_connections 2023-06-22T08:25:31.312 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.307+0000 7f2407bff700 1 -- 172.21.15.23:0/2668848277 wait complete. 2023-06-22T08:25:31.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.309+0000 7f2407bff700 1 Processor -- start 2023-06-22T08:25:31.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.310+0000 7f2407bff700 1 -- start start 2023-06-22T08:25:31.313 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.311+0000 7f2407bff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24000c5620 0x7f24001eea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:31.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.311+0000 7f2407bff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f240015c050 0x7f240015aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:31.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.311+0000 7f2407bff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f240015c420 0x7f24000bc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:31.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.311+0000 7f2407bff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24001ebf90 con 0x7f240015c420 2023-06-22T08:25:31.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.311+0000 7f2407bff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24000bb580 con 0x7f24000c5620 2023-06-22T08:25:31.314 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.311+0000 7f2407bff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24000bbe20 con 0x7f240015c050 2023-06-22T08:25:31.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.312+0000 7f240599b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24000c5620 0x7f24001eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:31.315 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.312+0000 7f240599b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24000c5620 0x7f24001eea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:34306/0 (socket says 172.21.15.23:34306) 2023-06-22T08:25:31.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.312+0000 7f240599b700 1 -- 172.21.15.23:0/3599592903 learned_addr learned my addr 172.21.15.23:0/3599592903 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:31.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.312+0000 7f240519a700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f240015c050 0x7f240015aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:31.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.312+0000 7f240619c700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f240015c420 0x7f24000bc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:31.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.313+0000 7f240599b700 1 -- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f240015c050 msgr2=0x7f240015aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:31.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.313+0000 7f240599b700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f240015c050 0x7f240015aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:31.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.313+0000 7f240599b700 1 -- 172.21.15.23:0/3599592903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f240015c420 msgr2=0x7f24000bc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:31.316 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.313+0000 7f240599b700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f240015c420 0x7f24000bc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:31.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.313+0000 7f240599b700 1 -- 172.21.15.23:0/3599592903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc0094a0 con 0x7f24000c5620 2023-06-22T08:25:31.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.313+0000 7f240599b700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24000c5620 0x7f24001eea00 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009fd0 tx=0x7f23fc004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:31.317 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.313+0000 7f23f67fc700 1 -- 172.21.15.23:0/3599592903 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc01a970 con 0x7f24000c5620 2023-06-22T08:25:31.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.313+0000 7f23f67fc700 1 -- 172.21.15.23:0/3599592903 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc01aad0 con 0x7f24000c5620 2023-06-22T08:25:31.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.313+0000 7f23f67fc700 1 -- 172.21.15.23:0/3599592903 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc01ae40 con 0x7f24000c5620 2023-06-22T08:25:31.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.313+0000 7f2407bff700 1 -- 172.21.15.23:0/3599592903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24001eef40 con 0x7f24000c5620 2023-06-22T08:25:31.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.314+0000 7f2407bff700 1 -- 172.21.15.23:0/3599592903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24000bd130 con 0x7f24000c5620 2023-06-22T08:25:31.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.315+0000 7f2407bff700 1 -- 172.21.15.23:0/3599592903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240006bb90 con 0x7f24000c5620 2023-06-22T08:25:31.318 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.316+0000 7f23f67fc700 1 -- 172.21.15.23:0/3599592903 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f23fc0243f0 con 0x7f24000c5620 2023-06-22T08:25:31.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.317+0000 7f23f67fc700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23e406ace0 0x7f23e406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:31.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.317+0000 7f23f67fc700 1 -- 172.21.15.23:0/3599592903 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8129+0+0 (secure 0 0 0) 0x7f23fc08b160 con 0x7f24000c5620 2023-06-22T08:25:31.320 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.318+0000 7f240519a700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23e406ace0 0x7f23e406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.326+0000 7f240519a700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23e406ace0 0x7f23e406d190 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f23f0001270 tx=0x7f23f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:31.334 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.327+0000 7f23f67fc700 1 -- 172.21.15.23:0/3599592903 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f23fc055a70 con 0x7f24000c5620 2023-06-22T08:25:31.692 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.690+0000 7f2407bff700 1 -- 172.21.15.23:0/3599592903 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f240004ea20 con 0x7f24000c5620 2023-06-22T08:25:31.693 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.690+0000 7f23f67fc700 1 -- 172.21.15.23:0/3599592903 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7f23fc0150a0 con 0x7f24000c5620 2023-06-22T08:25:31.693 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:25:31.695 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 -- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23e406ace0 msgr2=0x7f23e406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23e406ace0 0x7f23e406d190 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f23f0001270 tx=0x7f23f0009040 comp rx=0 tx=0).stop 2023-06-22T08:25:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 -- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24000c5620 msgr2=0x7f24001eea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24000c5620 0x7f24001eea00 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009fd0 tx=0x7f23fc004750 comp rx=0 tx=0).stop 2023-06-22T08:25:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 -- 172.21.15.23:0/3599592903 shutdown_connections 2023-06-22T08:25:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f240015c420 0x7f24000bc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f23e406ace0 0x7f23e406d190 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24000c5620 0x7f24001eea00 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 --2- 172.21.15.23:0/3599592903 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f240015c050 0x7f240015aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 -- 172.21.15.23:0/3599592903 >> 172.21.15.23:0/3599592903 conn(0x7f24000cb070 msgr2=0x7f24000c82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 -- 172.21.15.23:0/3599592903 shutdown_connections 2023-06-22T08:25:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.693+0000 7f2407bff700 1 -- 172.21.15.23:0/3599592903 wait complete. 2023-06-22T08:25:31.723 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-06-22T08:25:31.991 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.989+0000 7f30763ff700 1 Processor -- start 2023-06-22T08:25:31.992 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.989+0000 7f30763ff700 1 -- start start 2023-06-22T08:25:31.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.990+0000 7f30763ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f30701593f0 0x7f30700c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:31.994 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.990+0000 7f30763ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f30700c4650 0x7f30700c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.990+0000 7f30763ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30700c9aa0 con 0x7f3070152ba0 2023-06-22T08:25:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.990+0000 7f30763ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30700c4a60 con 0x7f30701593f0 2023-06-22T08:25:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.990+0000 7f30763ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307006aed0 con 0x7f30700c4650 2023-06-22T08:25:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.990+0000 7f306ffff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3070152ba0 0x7f30700c9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:50302/0 (socket says 172.21.15.23:50302) 2023-06-22T08:25:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f307499c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f30700c4650 0x7f30700c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f306ffff700 1 -- 172.21.15.23:0/1239285483 learned_addr learned my addr 172.21.15.23:0/1239285483 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:25:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f306f7fe700 1 --2- 172.21.15.23:0/1239285483 >> v2:172.21.15.23:3300/0 conn(0x7f30701593f0 0x7f30700c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:31.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f306effd700 1 -- 172.21.15.23:0/1239285483 <== mon.1 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328400926 0 0) 0x7f30700c9aa0 con 0x7f3070152ba0 2023-06-22T08:25:31.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f306effd700 1 -- 172.21.15.23:0/1239285483 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30580035b0 con 0x7f3070152ba0 2023-06-22T08:25:31.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f307499c700 1 -- 172.21.15.23:0/1239285483 >> v2:172.21.15.23:3300/0 conn(0x7f30701593f0 msgr2=0x7f30700c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:31.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f307499c700 1 --2- 172.21.15.23:0/1239285483 >> v2:172.21.15.23:3300/0 conn(0x7f30701593f0 0x7f30700c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:31.996 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f307499c700 1 -- 172.21.15.23:0/1239285483 >> v1:172.21.15.67:6789/0 conn(0x7f3070152ba0 legacy=0x7f30700c9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:31.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f307499c700 1 -- 172.21.15.23:0/1239285483 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307006b1b0 con 0x7f30700c4650 2023-06-22T08:25:31.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f306f7fe700 1 --2- 172.21.15.23:0/1239285483 >> v2:172.21.15.23:3300/0 conn(0x7f30701593f0 0x7f30700c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T08:25:31.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.991+0000 7f307499c700 1 --2- 172.21.15.23:0/1239285483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 0x7f30700c9ee0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f307008b390 tx=0x7f306800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60e4654961db4dbb server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:31.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f306effd700 1 -- 172.21.15.23:0/1239285483 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306800e070 con 0x7f30700c4650 2023-06-22T08:25:31.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f306effd700 1 -- 172.21.15.23:0/1239285483 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306800bb20 con 0x7f30700c4650 2023-06-22T08:25:31.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f306effd700 1 -- 172.21.15.23:0/1239285483 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068012430 con 0x7f30700c4650 2023-06-22T08:25:31.998 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f30763ff700 1 -- 172.21.15.23:0/1239285483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 msgr2=0x7f30700c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f30763ff700 1 --2- 172.21.15.23:0/1239285483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 0x7f30700c9ee0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f307008b390 tx=0x7f306800b330 comp rx=0 tx=0).stop 2023-06-22T08:25:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f30763ff700 1 -- 172.21.15.23:0/1239285483 shutdown_connections 2023-06-22T08:25:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f30763ff700 1 --2- 172.21.15.23:0/1239285483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 0x7f30700c9ee0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f30763ff700 1 --2- 172.21.15.23:0/1239285483 >> v2:172.21.15.23:3300/0 conn(0x7f30701593f0 0x7f30700c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f30763ff700 1 -- 172.21.15.23:0/1239285483 >> 172.21.15.23:0/1239285483 conn(0x7f3070065b90 msgr2=0x7f3070065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f30763ff700 1 -- 172.21.15.23:0/1239285483 shutdown_connections 2023-06-22T08:25:32.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.992+0000 7f30763ff700 1 -- 172.21.15.23:0/1239285483 wait complete. 2023-06-22T08:25:32.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.994+0000 7f30763ff700 1 Processor -- start 2023-06-22T08:25:32.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.994+0000 7f30763ff700 1 -- start start 2023-06-22T08:25:32.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.995+0000 7f30763ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 0x7f30700ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:32.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.995+0000 7f30763ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3070152ba0 0x7f30701532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:32.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.995+0000 7f30763ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30701593f0 0x7f3070153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:32.000 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.995+0000 7f30763ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3070155a30 con 0x7f30701593f0 2023-06-22T08:25:32.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.995+0000 7f30763ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3070156020 con 0x7f30700c4650 2023-06-22T08:25:32.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.996+0000 7f30763ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30701f13d0 con 0x7f3070152ba0 2023-06-22T08:25:32.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.996+0000 7f306ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 0x7f30700ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:32.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.996+0000 7f306f7fe700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3070152ba0 0x7f30701532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:32.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.996+0000 7f306ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 0x7f30700ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.23:34324/0 (socket says 172.21.15.23:34324) 2023-06-22T08:25:32.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.996+0000 7f306ffff700 1 -- 172.21.15.23:0/2894830876 learned_addr learned my addr 172.21.15.23:0/2894830876 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:32.001 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.996+0000 7f307499c700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30701593f0 0x7f3070153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:32.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.996+0000 7f306ffff700 1 -- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3070152ba0 msgr2=0x7f30701532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:32.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.996+0000 7f306ffff700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3070152ba0 0x7f30701532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:32.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.996+0000 7f306ffff700 1 -- 172.21.15.23:0/2894830876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30701593f0 msgr2=0x7f3070153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:32.002 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.996+0000 7f306ffff700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30701593f0 0x7f3070153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:32.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.997+0000 7f306ffff700 1 -- 172.21.15.23:0/2894830876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30640094a0 con 0x7f30700c4650 2023-06-22T08:25:32.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.997+0000 7f306ffff700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 0x7f30700ca6a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3060001da0 tx=0x7f306000c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:32.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.998+0000 7f306cff9700 1 -- 172.21.15.23:0/2894830876 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060005d70 con 0x7f30700c4650 2023-06-22T08:25:32.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.998+0000 7f306cff9700 1 -- 172.21.15.23:0/2894830876 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3060015960 con 0x7f30700c4650 2023-06-22T08:25:32.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.998+0000 7f306cff9700 1 -- 172.21.15.23:0/2894830876 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060015cf0 con 0x7f30700c4650 2023-06-22T08:25:32.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.999+0000 7f30763ff700 1 -- 172.21.15.23:0/2894830876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3068009a70 con 0x7f30700c4650 2023-06-22T08:25:32.003 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:31.999+0000 7f30763ff700 1 -- 172.21.15.23:0/2894830876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30701f1620 con 0x7f30700c4650 2023-06-22T08:25:32.004 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.001+0000 7f30763ff700 1 -- 172.21.15.23:0/2894830876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30700cbed0 con 0x7f30700c4650 2023-06-22T08:25:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.002+0000 7f306cff9700 1 -- 172.21.15.23:0/2894830876 <== mon.1 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f306001b020 con 0x7f30700c4650 2023-06-22T08:25:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.004+0000 7f306cff9700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f305c06adb0 0x7f305c06d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.004+0000 7f306cff9700 1 -- 172.21.15.23:0/2894830876 <== mon.1 v2:172.21.15.67:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8129+0+0 (secure 0 0 0) 0x7f3060091610 con 0x7f30700c4650 2023-06-22T08:25:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.004+0000 7f306f7fe700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f305c06adb0 0x7f305c06d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:32.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.005+0000 7f306f7fe700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f305c06adb0 0x7f305c06d260 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3064009d70 tx=0x7f3064009210 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:32.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.010+0000 7f306cff9700 1 -- 172.21.15.23:0/2894830876 <== mon.1 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f306005bdf0 con 0x7f30700c4650 2023-06-22T08:25:32.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/rpmatch.c' 2023-06-22T08:25:32.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/sha1.c' 2023-06-22T08:25:32.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/vfprintf.c' 2023-06-22T08:25:32.353 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/isnanf-nolibm.h' 2023-06-22T08:25:32.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/c-strtod.c' 2023-06-22T08:25:32.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/getpass.c' 2023-06-22T08:25:32.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/fnmatch.in.h' 2023-06-22T08:25:32.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/isnand.c' 2023-06-22T08:25:32.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/write-any-file.c' 2023-06-22T08:25:32.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/c-strcasecmp.c' 2023-06-22T08:25:32.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/stdint.in.h' 2023-06-22T08:25:32.354 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/dirname-lgpl.c' 2023-06-22T08:25:32.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/lseek.c' 2023-06-22T08:25:32.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/lib/strerror.c' 2023-06-22T08:25:32.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/lib' 2023-06-22T08:25:32.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.tarball-version' 2023-06-22T08:25:32.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_error_message_uppercase' 2023-06-22T08:25:32.355 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_space_tab' 2023-06-22T08:25:32.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/ChangeLog-2007' 2023-06-22T08:25:32.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_require_config_h_first' 2023-06-22T08:25:32.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/ABOUT-NLS' 2023-06-22T08:25:32.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/aclocal.m4' 2023-06-22T08:25:32.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/bootstrap' 2023-06-22T08:25:32.356 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/NEWS' 2023-06-22T08:25:32.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/ChangeLog-2006' 2023-06-22T08:25:32.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_unmarked_diagnostics' 2023-06-22T08:25:32.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/Makefile.in' 2023-06-22T08:25:32.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/ChangeLog-2005' 2023-06-22T08:25:32.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/timespec.m4' 2023-06-22T08:25:32.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/xfts.m4' 2023-06-22T08:25:32.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/jm-macros.m4' 2023-06-22T08:25:32.357 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getugroups.m4' 2023-06-22T08:25:32.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/unicodeio.m4' 2023-06-22T08:25:32.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/lib-link.m4' 2023-06-22T08:25:32.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stpcpy.m4' 2023-06-22T08:25:32.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/math_h.m4' 2023-06-22T08:25:32.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/printf.m4' 2023-06-22T08:25:32.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/locale-fr.m4' 2023-06-22T08:25:32.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sha1.m4' 2023-06-22T08:25:32.358 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/safe-write.m4' 2023-06-22T08:25:32.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/double-slash-root.m4' 2023-06-22T08:25:32.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/close-stream.m4' 2023-06-22T08:25:32.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/linkat.m4' 2023-06-22T08:25:32.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/nl_langinfo.m4' 2023-06-22T08:25:32.359 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/host-os.m4' 2023-06-22T08:25:32.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/priv-set.m4' 2023-06-22T08:25:32.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/write.m4' 2023-06-22T08:25:32.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/lchmod.m4' 2023-06-22T08:25:32.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/read-file.m4' 2023-06-22T08:25:32.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/netinet_in_h.m4' 2023-06-22T08:25:32.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/00gnulib.m4' 2023-06-22T08:25:32.360 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/utimbuf.m4' 2023-06-22T08:25:32.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/euidaccess.m4' 2023-06-22T08:25:32.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/cloexec.m4' 2023-06-22T08:25:32.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/argmatch.m4' 2023-06-22T08:25:32.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mountlist.m4' 2023-06-22T08:25:32.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/string_h.m4' 2023-06-22T08:25:32.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/utimens.m4' 2023-06-22T08:25:32.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/warn-on-use.m4' 2023-06-22T08:25:32.361 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/codeset.m4' 2023-06-22T08:25:32.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/unlink.m4' 2023-06-22T08:25:32.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fclose.m4' 2023-06-22T08:25:32.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/vfprintf-posix.m4' 2023-06-22T08:25:32.362 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/yield.m4' 2023-06-22T08:25:32.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/localcharset.m4' 2023-06-22T08:25:32.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/isnanf.m4' 2023-06-22T08:25:32.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/idcache.m4' 2023-06-22T08:25:32.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/readlink.m4' 2023-06-22T08:25:32.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mbrlen.m4' 2023-06-22T08:25:32.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getgroups.m4' 2023-06-22T08:25:32.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getopt.m4' 2023-06-22T08:25:32.363 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getndelim2.m4' 2023-06-22T08:25:32.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strnlen.m4' 2023-06-22T08:25:32.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mgetgroups.m4' 2023-06-22T08:25:32.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/signalblocking.m4' 2023-06-22T08:25:32.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stat.m4' 2023-06-22T08:25:32.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/gettext.m4' 2023-06-22T08:25:32.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/uname.m4' 2023-06-22T08:25:32.364 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/xnanosleep.m4' 2023-06-22T08:25:32.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strtoimax.m4' 2023-06-22T08:25:32.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/locale-zh.m4' 2023-06-22T08:25:32.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mode_t.m4' 2023-06-22T08:25:32.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/printf-frexp.m4' 2023-06-22T08:25:32.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strcspn.m4' 2023-06-22T08:25:32.365 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/wchar_h.m4' 2023-06-22T08:25:32.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/file-type.m4' 2023-06-22T08:25:32.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/usleep.m4' 2023-06-22T08:25:32.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stdlib-safer.m4' 2023-06-22T08:25:32.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/setenv.m4' 2023-06-22T08:25:32.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stat-prog.m4' 2023-06-22T08:25:32.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/write-any-file.m4' 2023-06-22T08:25:32.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/yesno.m4' 2023-06-22T08:25:32.366 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sys_ioctl_h.m4' 2023-06-22T08:25:32.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/lchown.m4' 2023-06-22T08:25:32.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/intlmacosx.m4' 2023-06-22T08:25:32.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strsignal.m4' 2023-06-22T08:25:32.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mbsinit.m4' 2023-06-22T08:25:32.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/memcasecmp.m4' 2023-06-22T08:25:32.367 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/freopen.m4' 2023-06-22T08:25:32.400 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.398+0000 7f30763ff700 1 -- 172.21.15.23:0/2894830876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f30700c3820 con 0x7f30700c4650 2023-06-22T08:25:32.404 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.401+0000 7f306cff9700 1 -- 172.21.15.23:0/2894830876 <== mon.1 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3060023070 con 0x7f30700c4650 2023-06-22T08:25:32.406 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2023-06-22T08:25:32.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.407+0000 7f30763ff700 1 -- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f305c06adb0 msgr2=0x7f305c06d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:32.409 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.407+0000 7f30763ff700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f305c06adb0 0x7f305c06d260 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3064009d70 tx=0x7f3064009210 comp rx=0 tx=0).stop 2023-06-22T08:25:32.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.407+0000 7f30763ff700 1 -- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 msgr2=0x7f30700ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:32.410 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.407+0000 7f30763ff700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 0x7f30700ca6a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3060001da0 tx=0x7f306000c960 comp rx=0 tx=0).stop 2023-06-22T08:25:32.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.408+0000 7f30763ff700 1 -- 172.21.15.23:0/2894830876 shutdown_connections 2023-06-22T08:25:32.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.408+0000 7f30763ff700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30701593f0 0x7f3070153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:32.412 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.408+0000 7f30763ff700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f305c06adb0 0x7f305c06d260 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:32.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.408+0000 7f30763ff700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30700c4650 0x7f30700ca6a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:32.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.408+0000 7f30763ff700 1 --2- 172.21.15.23:0/2894830876 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3070152ba0 0x7f30701532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:32.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.408+0000 7f30763ff700 1 -- 172.21.15.23:0/2894830876 >> 172.21.15.23:0/2894830876 conn(0x7f3070065b90 msgr2=0x7f3070156500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:32.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.409+0000 7f30763ff700 1 -- 172.21.15.23:0/2894830876 shutdown_connections 2023-06-22T08:25:32.413 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.409+0000 7f30763ff700 1 -- 172.21.15.23:0/2894830876 wait complete. 2023-06-22T08:25:32.461 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-06-22T08:25:33.005 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.998+0000 7fae983ff700 1 Processor -- start 2023-06-22T08:25:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:32.998+0000 7fae983ff700 1 -- start start 2023-06-22T08:25:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.000+0000 7fae983ff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae90155560 0x7fae90152b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.000+0000 7fae983ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae90155a50 con 0x7fae900c7900 2023-06-22T08:25:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.002+0000 7fae983ff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9006d0c0 con 0x7fae900c5500 2023-06-22T08:25:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.002+0000 7fae983ff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9006d360 con 0x7fae90155560 2023-06-22T08:25:33.006 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.002+0000 7fae9619b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fae900c5500 0x7fae900c58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.23:50306/0 (socket says 172.21.15.23:50306) 2023-06-22T08:25:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.002+0000 7fae9619b700 1 -- 172.21.15.23:0/4047951188 learned_addr learned my addr 172.21.15.23:0/4047951188 (peer_addr_for_me v1:172.21.15.23:0/0) 2023-06-22T08:25:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.003+0000 7fae9599a700 1 --2- 172.21.15.23:0/4047951188 >> v2:172.21.15.67:3300/0 conn(0x7fae90155560 0x7fae90152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.003+0000 7fae9599a700 1 -- 172.21.15.23:0/4047951188 >> v1:172.21.15.67:6789/0 conn(0x7fae900c5500 legacy=0x7fae900c58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:33.007 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.003+0000 7fae9599a700 1 -- 172.21.15.23:0/4047951188 >> v1:172.21.15.23:6789/0 conn(0x7fae900c7900 legacy=0x7fae901518a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:33.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.003+0000 7fae9599a700 1 -- 172.21.15.23:0/4047951188 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9006d640 con 0x7fae90155560 2023-06-22T08:25:33.008 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.004+0000 7fae9599a700 1 --2- 172.21.15.23:0/4047951188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae90155560 0x7fae90152b50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0097c0 tx=0x7fae8c009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe9b8888afd769e7 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.005+0000 7fae95199700 1 -- 172.21.15.23:0/4047951188 <== mon.1 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c004230 con 0x7fae90155560 2023-06-22T08:25:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.005+0000 7fae95199700 1 -- 172.21.15.23:0/4047951188 <== mon.1 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c004390 con 0x7fae90155560 2023-06-22T08:25:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.005+0000 7fae95199700 1 -- 172.21.15.23:0/4047951188 <== mon.1 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c00f740 con 0x7fae90155560 2023-06-22T08:25:33.009 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.006+0000 7fae983ff700 1 -- 172.21.15.23:0/4047951188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae90155560 msgr2=0x7fae90152b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:33.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.006+0000 7fae983ff700 1 --2- 172.21.15.23:0/4047951188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae90155560 0x7fae90152b50 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fae8c0097c0 tx=0x7fae8c009ad0 comp rx=0 tx=0).stop 2023-06-22T08:25:33.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.007+0000 7fae983ff700 1 -- 172.21.15.23:0/4047951188 shutdown_connections 2023-06-22T08:25:33.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.007+0000 7fae983ff700 1 --2- 172.21.15.23:0/4047951188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae90155560 0x7fae90152b50 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:33.010 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.007+0000 7fae983ff700 1 -- 172.21.15.23:0/4047951188 >> 172.21.15.23:0/4047951188 conn(0x7fae900cb070 msgr2=0x7fae900c6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:33.012 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.007+0000 7fae983ff700 1 -- 172.21.15.23:0/4047951188 shutdown_connections 2023-06-22T08:25:33.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.007+0000 7fae983ff700 1 -- 172.21.15.23:0/4047951188 wait complete. 2023-06-22T08:25:33.013 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.010+0000 7fae983ff700 1 Processor -- start 2023-06-22T08:25:33.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.011+0000 7fae983ff700 1 -- start start 2023-06-22T08:25:33.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.012+0000 7fae983ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae900c5500 0x7fae90070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:33.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.012+0000 7fae983ff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fae900c7900 0x7fae90150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:33.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.012+0000 7fae983ff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae90155560 0x7fae90150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:33.018 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.012+0000 7fae983ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae900c0f60 con 0x7fae900c5500 2023-06-22T08:25:33.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.012+0000 7fae983ff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9014fda0 con 0x7fae90155560 2023-06-22T08:25:33.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.012+0000 7fae983ff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae900c9950 con 0x7fae900c7900 2023-06-22T08:25:33.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.014+0000 7fae9619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae900c5500 0x7fae90070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:33.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.014+0000 7fae9619b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae900c5500 0x7fae90070ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38756/0 (socket says 172.21.15.23:38756) 2023-06-22T08:25:33.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.014+0000 7fae9619b700 1 -- 172.21.15.23:0/1226736410 learned_addr learned my addr 172.21.15.23:0/1226736410 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:33.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.014+0000 7fae9619b700 1 -- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fae900c7900 msgr2=0x7fae90150560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-06-22T08:25:33.019 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.014+0000 7fae9619b700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fae900c7900 0x7fae90150560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:33.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.014+0000 7fae9619b700 1 -- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae90155560 msgr2=0x7fae90150aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-22T08:25:33.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.014+0000 7fae9619b700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae90155560 0x7fae90150aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:33.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.015+0000 7fae9619b700 1 -- 172.21.15.23:0/1226736410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0094a0 con 0x7fae900c5500 2023-06-22T08:25:33.020 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.015+0000 7fae9619b700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae900c5500 0x7fae90070ee0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fae88002070 tx=0x7fae8800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:33.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.017+0000 7fae86ffd700 1 -- 172.21.15.23:0/1226736410 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8800ae60 con 0x7fae900c5500 2023-06-22T08:25:33.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.017+0000 7fae86ffd700 1 -- 172.21.15.23:0/1226736410 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae880070d0 con 0x7fae900c5500 2023-06-22T08:25:33.021 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.017+0000 7fae86ffd700 1 -- 172.21.15.23:0/1226736410 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88007460 con 0x7fae900c5500 2023-06-22T08:25:33.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.019+0000 7fae983ff700 1 -- 172.21.15.23:0/1226736410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae900c06c0 con 0x7fae900c5500 2023-06-22T08:25:33.023 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.019+0000 7fae983ff700 1 -- 172.21.15.23:0/1226736410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae900c0c80 con 0x7fae900c5500 2023-06-22T08:25:33.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.025+0000 7fae86ffd700 1 -- 172.21.15.23:0/1226736410 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7fae880075c0 con 0x7fae900c5500 2023-06-22T08:25:33.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.026+0000 7fae86ffd700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fae7406ade0 0x7fae7406d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:33.029 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.026+0000 7fae86ffd700 1 -- 172.21.15.23:0/1226736410 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8129+0+0 (secure 0 0 0) 0x7fae88089e80 con 0x7fae900c5500 2023-06-22T08:25:33.030 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.028+0000 7fae9599a700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fae7406ade0 0x7fae7406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:33.031 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.028+0000 7fae983ff700 1 -- 172.21.15.23:0/1226736410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae900bd360 con 0x7fae900c5500 2023-06-22T08:25:33.038 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.034+0000 7fae9599a700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fae7406ade0 0x7fae7406d290 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fae90155980 tx=0x7fae8c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:33.042 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.040+0000 7fae86ffd700 1 -- 172.21.15.23:0/1226736410 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7fae88018030 con 0x7fae900c5500 2023-06-22T08:25:33.529 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.526+0000 7fae983ff700 1 -- 172.21.15.23:0/1226736410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fae9014fc00 con 0x7fae900c5500 2023-06-22T08:25:33.530 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.527+0000 7fae86ffd700 1 -- 172.21.15.23:0/1226736410 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v112) v1 ==== 103+0+11 (secure 0 0 0) 0x7fae9014fc00 con 0x7fae900c5500 2023-06-22T08:25:33.533 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2023-06-22T08:25:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.534+0000 7fae983ff700 1 -- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fae7406ade0 msgr2=0x7fae7406d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.534+0000 7fae983ff700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fae7406ade0 0x7fae7406d290 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fae90155980 tx=0x7fae8c008030 comp rx=0 tx=0).stop 2023-06-22T08:25:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.534+0000 7fae983ff700 1 -- 172.21.15.23:0/1226736410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae900c5500 msgr2=0x7fae90070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:33.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.534+0000 7fae983ff700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae900c5500 0x7fae90070ee0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fae88002070 tx=0x7fae8800a340 comp rx=0 tx=0).stop 2023-06-22T08:25:33.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.535+0000 7fae983ff700 1 -- 172.21.15.23:0/1226736410 shutdown_connections 2023-06-22T08:25:33.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.535+0000 7fae983ff700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7fae7406ade0 0x7fae7406d290 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:33.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.535+0000 7fae983ff700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae90155560 0x7fae90150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:33.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.535+0000 7fae983ff700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae900c5500 0x7fae90070ee0 secure :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fae88002070 tx=0x7fae8800a340 comp rx=0 tx=0).stop 2023-06-22T08:25:33.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.535+0000 7fae983ff700 1 --2- 172.21.15.23:0/1226736410 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fae900c7900 0x7fae90150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:33.538 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.535+0000 7fae983ff700 1 -- 172.21.15.23:0/1226736410 >> 172.21.15.23:0/1226736410 conn(0x7fae900cb070 msgr2=0x7fae900c1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.535+0000 7fae983ff700 1 -- 172.21.15.23:0/1226736410 shutdown_connections 2023-06-22T08:25:33.539 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:33.535+0000 7fae983ff700 1 -- 172.21.15.23:0/1226736410 wait complete. 2023-06-22T08:25:33.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-06-22T08:25:34.001 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/iconv.m4' 2023-06-22T08:25:34.002 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mbrtowc.m4' 2023-06-22T08:25:34.002 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/ioctl.m4' 2023-06-22T08:25:34.002 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sockpfaf.m4' 2023-06-22T08:25:34.002 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/thread.m4' 2023-06-22T08:25:34.002 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fflush.m4' 2023-06-22T08:25:34.002 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/assert.m4' 2023-06-22T08:25:34.003 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stpncpy.m4' 2023-06-22T08:25:34.003 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strftime.m4' 2023-06-22T08:25:34.003 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/wcrtomb.m4' 2023-06-22T08:25:34.003 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/langinfo_h.m4' 2023-06-22T08:25:34.003 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/safe-read.m4' 2023-06-22T08:25:34.004 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/clock_time.m4' 2023-06-22T08:25:34.005 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/tls.m4' 2023-06-22T08:25:34.005 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/memcoll.m4' 2023-06-22T08:25:34.005 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/readutmp.m4' 2023-06-22T08:25:34.005 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stdio_h.m4' 2023-06-22T08:25:34.005 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/ftello.m4' 2023-06-22T08:25:34.005 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/memcmp.m4' 2023-06-22T08:25:34.005 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/canon-host.m4' 2023-06-22T08:25:34.006 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/gmp.m4' 2023-06-22T08:25:34.006 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/environ.m4' 2023-06-22T08:25:34.006 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fdopendir.m4' 2023-06-22T08:25:34.006 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stdarg.m4' 2023-06-22T08:25:34.006 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stdbool.m4' 2023-06-22T08:25:34.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/intmax_t.m4' 2023-06-22T08:25:34.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/isapipe.m4' 2023-06-22T08:25:34.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/ldexpl.m4' 2023-06-22T08:25:34.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/remove.m4' 2023-06-22T08:25:34.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fpurge.m4' 2023-06-22T08:25:34.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getusershell.m4' 2023-06-22T08:25:34.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/memcpy.m4' 2023-06-22T08:25:34.007 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fcntl.m4' 2023-06-22T08:25:34.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/tempname.m4' 2023-06-22T08:25:34.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/xstrtol.m4' 2023-06-22T08:25:34.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fseeko.m4' 2023-06-22T08:25:34.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fnmatch.m4' 2023-06-22T08:25:34.008 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/save-cwd.m4' 2023-06-22T08:25:34.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/config-h.m4' 2023-06-22T08:25:34.009 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/filemode.m4' 2023-06-22T08:25:34.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/errno_h.m4' 2023-06-22T08:25:34.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/eealloc.m4' 2023-06-22T08:25:34.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strdup.m4' 2023-06-22T08:25:34.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/signal_h.m4' 2023-06-22T08:25:34.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/regex.m4' 2023-06-22T08:25:34.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mbchar.m4' 2023-06-22T08:25:34.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strtoull.m4' 2023-06-22T08:25:34.010 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stdint.m4' 2023-06-22T08:25:34.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/wctype_h.m4' 2023-06-22T08:25:34.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/realloc.m4' 2023-06-22T08:25:34.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/printf-frexpl.m4' 2023-06-22T08:25:34.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/open.m4' 2023-06-22T08:25:34.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/inline.m4' 2023-06-22T08:25:34.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/warnings.m4' 2023-06-22T08:25:34.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/vasprintf-posix.m4' 2023-06-22T08:25:34.011 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/xstrndup.m4' 2023-06-22T08:25:34.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sys_utsname_h.m4' 2023-06-22T08:25:34.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/locale-ja.m4' 2023-06-22T08:25:34.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/snprintf.m4' 2023-06-22T08:25:34.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/po.m4' 2023-06-22T08:25:34.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/hash.m4' 2023-06-22T08:25:34.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sig2str.m4' 2023-06-22T08:25:34.012 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/ctype.m4' 2023-06-22T08:25:34.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strnumcmp.m4' 2023-06-22T08:25:34.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/rpmatch.m4' 2023-06-22T08:25:34.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/md5.m4' 2023-06-22T08:25:34.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/boottime.m4' 2023-06-22T08:25:34.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/unistd_h.m4' 2023-06-22T08:25:34.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fd-reopen.m4' 2023-06-22T08:25:34.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/cycle-check.m4' 2023-06-22T08:25:34.013 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/ls-mntd-fs.m4' 2023-06-22T08:25:34.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mkancesdirs.m4' 2023-06-22T08:25:34.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/manywarnings.m4' 2023-06-22T08:25:34.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/ChangeLog-2007' 2023-06-22T08:25:34.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getcwd-path-max.m4' 2023-06-22T08:25:34.014 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/modechange.m4' 2023-06-22T08:25:34.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stdlib_h.m4' 2023-06-22T08:25:34.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/gethostname.m4' 2023-06-22T08:25:34.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/physmem.m4' 2023-06-22T08:25:34.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fileblocks.m4' 2023-06-22T08:25:34.015 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/socklen.m4' 2023-06-22T08:25:34.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/st_dm_mode.m4' 2023-06-22T08:25:34.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/ungetc.m4' 2023-06-22T08:25:34.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/posixver.m4' 2023-06-22T08:25:34.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getdtablesize.m4' 2023-06-22T08:25:34.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/check-decl.m4' 2023-06-22T08:25:34.016 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/threadlib.m4' 2023-06-22T08:25:34.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sys_socket_h.m4' 2023-06-22T08:25:34.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getaddrinfo.m4' 2023-06-22T08:25:34.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/float_h.m4' 2023-06-22T08:25:34.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/btowc.m4' 2023-06-22T08:25:34.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/userspec.m4' 2023-06-22T08:25:34.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/posixtm.m4' 2023-06-22T08:25:34.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/alloca.m4' 2023-06-22T08:25:34.017 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/lstat.m4' 2023-06-22T08:25:34.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/lock.m4' 2023-06-22T08:25:34.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/readtokens.m4' 2023-06-22T08:25:34.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/locale-tr.m4' 2023-06-22T08:25:34.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/wctob.m4' 2023-06-22T08:25:34.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/canonicalize.m4' 2023-06-22T08:25:34.018 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/servent.m4' 2023-06-22T08:25:34.660 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mbswidth.m4' 2023-06-22T08:25:34.660 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/memrchr.m4' 2023-06-22T08:25:34.660 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/autobuild.m4' 2023-06-22T08:25:34.660 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strtol.m4' 2023-06-22T08:25:34.661 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/closein.m4' 2023-06-22T08:25:34.661 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sha256.m4' 2023-06-22T08:25:34.661 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mbslen.m4' 2023-06-22T08:25:34.661 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/lseek.m4' 2023-06-22T08:25:34.661 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/base64.m4' 2023-06-22T08:25:34.661 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/iconv_h.m4' 2023-06-22T08:25:34.662 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/longlong.m4' 2023-06-22T08:25:34.662 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/dirfd.m4' 2023-06-22T08:25:34.662 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/d-ino.m4' 2023-06-22T08:25:34.662 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/ssize_t.m4' 2023-06-22T08:25:34.662 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/gnulib-comp.m4' 2023-06-22T08:25:34.662 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/memset.m4' 2023-06-22T08:25:34.662 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getcwd-abort-bug.m4' 2023-06-22T08:25:34.663 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/c-strtod.m4' 2023-06-22T08:25:34.663 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fts.m4' 2023-06-22T08:25:34.663 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/lib-prefix.m4' 2023-06-22T08:25:34.663 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mmap-anon.m4' 2023-06-22T08:25:34.663 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/close.m4' 2023-06-22T08:25:34.663 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/lib-ignore.m4' 2023-06-22T08:25:34.663 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/gnulib-common.m4' 2023-06-22T08:25:34.663 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/time_h.m4' 2023-06-22T08:25:34.664 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/nls.m4' 2023-06-22T08:25:34.664 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/dirname.m4' 2023-06-22T08:25:34.664 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fcntl_h.m4' 2023-06-22T08:25:34.664 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stddef_h.m4' 2023-06-22T08:25:34.664 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mkstemp.m4' 2023-06-22T08:25:34.664 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/link-follow.m4' 2023-06-22T08:25:34.665 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/wcwidth.m4' 2023-06-22T08:25:34.665 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/memchr.m4' 2023-06-22T08:25:34.665 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strerror.m4' 2023-06-22T08:25:34.665 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/iconv_open.m4' 2023-06-22T08:25:34.665 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/exponentf.m4' 2023-06-22T08:25:34.665 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/pathmax.m4' 2023-06-22T08:25:34.665 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stdio-safer.m4' 2023-06-22T08:25:34.666 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/acl.m4' 2023-06-22T08:25:34.666 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fpending.m4' 2023-06-22T08:25:34.666 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/putenv.m4' 2023-06-22T08:25:34.666 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/include-exclude-prog.m4' 2023-06-22T08:25:34.666 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mktime.m4' 2023-06-22T08:25:34.666 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getdate.m4' 2023-06-22T08:25:34.666 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fsync.m4' 2023-06-22T08:25:34.667 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/symlink.m4' 2023-06-22T08:25:34.667 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/quote.m4' 2023-06-22T08:25:34.667 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getloadavg.m4' 2023-06-22T08:25:34.667 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/gettime.m4' 2023-06-22T08:25:34.667 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/quotearg.m4' 2023-06-22T08:25:34.667 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/inttypes.m4' 2023-06-22T08:25:34.667 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/dos.m4' 2023-06-22T08:25:34.667 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/gettimeofday.m4' 2023-06-22T08:25:34.668 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/hostent.m4' 2023-06-22T08:25:34.668 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/chdir-long.m4' 2023-06-22T08:25:34.668 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/selinux-selinux-h.m4' 2023-06-22T08:25:34.668 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fcntl-safer.m4' 2023-06-22T08:25:34.668 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/inttostr.m4' 2023-06-22T08:25:34.668 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getline.m4' 2023-06-22T08:25:34.669 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/posix-shell.m4' 2023-06-22T08:25:34.669 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/d-type.m4' 2023-06-22T08:25:34.669 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/inet_pton.m4' 2023-06-22T08:25:34.669 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sigaction.m4' 2023-06-22T08:25:34.669 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mkdir.m4' 2023-06-22T08:25:34.669 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getdelim.m4' 2023-06-22T08:25:34.669 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/xsize.m4' 2023-06-22T08:25:34.670 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/symlinkat.m4' 2023-06-22T08:25:34.670 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/dirent-safer.m4' 2023-06-22T08:25:34.670 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/include_next.m4' 2023-06-22T08:25:34.670 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getcwd.m4' 2023-06-22T08:25:34.670 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sys_time_h.m4' 2023-06-22T08:25:34.670 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fprintftime.m4' 2023-06-22T08:25:34.670 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/prereq.m4' 2023-06-22T08:25:34.671 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/netdb_h.m4' 2023-06-22T08:25:34.671 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/perl.m4' 2023-06-22T08:25:34.671 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/extensions.m4' 2023-06-22T08:25:34.671 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/closeout.m4' 2023-06-22T08:25:34.671 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/jm-winsz2.m4' 2023-06-22T08:25:34.671 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/settime.m4' 2023-06-22T08:25:34.671 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/progtest.m4' 2023-06-22T08:25:34.672 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/atexit.m4' 2023-06-22T08:25:34.672 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/savedir.m4' 2023-06-22T08:25:34.672 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/isnand.m4' 2023-06-22T08:25:34.672 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/arpa_inet_h.m4' 2023-06-22T08:25:34.672 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/root-dev-ino.m4' 2023-06-22T08:25:34.672 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/hard-locale.m4' 2023-06-22T08:25:34.673 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/multiarch.m4' 2023-06-22T08:25:34.673 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/gnu-make.m4' 2023-06-22T08:25:34.673 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stdint_h.m4' 2023-06-22T08:25:34.673 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/openat.m4' 2023-06-22T08:25:34.673 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/nproc.m4' 2023-06-22T08:25:34.673 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/link.m4' 2023-06-22T08:25:34.673 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/faccessat.m4' 2023-06-22T08:25:34.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/uptime.m4' 2023-06-22T08:25:34.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fpieee.m4' 2023-06-22T08:25:34.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mkdir-p.m4' 2023-06-22T08:25:34.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/xvasprintf.m4' 2023-06-22T08:25:34.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strpbrk.m4' 2023-06-22T08:25:34.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/rename.m4' 2023-06-22T08:25:34.674 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/wchar_t.m4' 2023-06-22T08:25:35.094 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.091+0000 7f03d5fff700 1 Processor -- start 2023-06-22T08:25:35.098 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.092+0000 7f03d5fff700 1 -- start start 2023-06-22T08:25:35.100 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.098+0000 7f03d5fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f03d0069eb0 0x7f03d006d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:35.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.098+0000 7f03d5fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f03d015d9b0 0x7f03d006a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:35.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.098+0000 7f03d5fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d00c5aa0 0x7f03d006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:35.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.098+0000 7f03d5fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d006a7c0 con 0x7f03d0069eb0 2023-06-22T08:25:35.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.098+0000 7f03d5fff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d00c4d70 con 0x7f03d015d9b0 2023-06-22T08:25:35.101 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.098+0000 7f03d5fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00c5610 con 0x7f03d00c5aa0 2023-06-22T08:25:35.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.105+0000 7f03cffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d00c5aa0 0x7f03d006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:35.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.105+0000 7f03cffff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d00c5aa0 0x7f03d006fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.23:46802/0 (socket says 172.21.15.23:46802) 2023-06-22T08:25:35.107 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.105+0000 7f03cffff700 1 -- 172.21.15.23:0/2242696801 learned_addr learned my addr 172.21.15.23:0/2242696801 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:35.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.105+0000 7f03cffff700 1 -- 172.21.15.23:0/2242696801 >> v2:172.21.15.67:3300/0 conn(0x7f03d015d9b0 msgr2=0x7f03d006a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:35.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.105+0000 7f03cffff700 1 --2- 172.21.15.23:0/2242696801 >> v2:172.21.15.67:3300/0 conn(0x7f03d015d9b0 0x7f03d006a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.105+0000 7f03cffff700 1 -- 172.21.15.23:0/2242696801 >> v2:172.21.15.23:3300/0 conn(0x7f03d0069eb0 msgr2=0x7f03d006d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T08:25:35.108 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.105+0000 7f03cffff700 1 --2- 172.21.15.23:0/2242696801 >> v2:172.21.15.23:3300/0 conn(0x7f03d0069eb0 0x7f03d006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.105+0000 7f03cffff700 1 -- 172.21.15.23:0/2242696801 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0071340 con 0x7f03d00c5aa0 2023-06-22T08:25:35.109 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.106+0000 7f03cffff700 1 --2- 172.21.15.23:0/2242696801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d00c5aa0 0x7f03d006fcb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f03d00c1c10 tx=0x7f03c800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cf81d1c969f0339 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:35.118 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.113+0000 7f03ceffd700 1 -- 172.21.15.23:0/2242696801 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c800e070 con 0x7f03d00c5aa0 2023-06-22T08:25:35.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.113+0000 7f03ceffd700 1 -- 172.21.15.23:0/2242696801 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c800b9c0 con 0x7f03d00c5aa0 2023-06-22T08:25:35.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.113+0000 7f03ceffd700 1 -- 172.21.15.23:0/2242696801 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c800bd70 con 0x7f03d00c5aa0 2023-06-22T08:25:35.119 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.116+0000 7f03d5fff700 1 -- 172.21.15.23:0/2242696801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d00c5aa0 msgr2=0x7f03d006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:35.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.116+0000 7f03d5fff700 1 --2- 172.21.15.23:0/2242696801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d00c5aa0 0x7f03d006fcb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f03d00c1c10 tx=0x7f03c800b330 comp rx=0 tx=0).stop 2023-06-22T08:25:35.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.116+0000 7f03d5fff700 1 -- 172.21.15.23:0/2242696801 shutdown_connections 2023-06-22T08:25:35.120 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.116+0000 7f03d5fff700 1 --2- 172.21.15.23:0/2242696801 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d00c5aa0 0x7f03d006fcb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.116+0000 7f03d5fff700 1 --2- 172.21.15.23:0/2242696801 >> v2:172.21.15.23:3300/0 conn(0x7f03d0069eb0 0x7f03d006d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.129 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.116+0000 7f03d5fff700 1 --2- 172.21.15.23:0/2242696801 >> v2:172.21.15.67:3300/0 conn(0x7f03d015d9b0 0x7f03d006a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.130 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.116+0000 7f03d5fff700 1 -- 172.21.15.23:0/2242696801 >> 172.21.15.23:0/2242696801 conn(0x7f03d00ca3f0 msgr2=0x7f03d00ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:35.132 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.130+0000 7f03d5fff700 1 -- 172.21.15.23:0/2242696801 shutdown_connections 2023-06-22T08:25:35.133 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.131+0000 7f03d5fff700 1 -- 172.21.15.23:0/2242696801 wait complete. 2023-06-22T08:25:35.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.134+0000 7f03d5fff700 1 Processor -- start 2023-06-22T08:25:35.137 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.134+0000 7f03d5fff700 1 -- start start 2023-06-22T08:25:35.141 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.138+0000 7f03d5fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03d00c5aa0 0x7f03d014dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:35.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.138+0000 7f03d5fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d015d9b0 0x7f03d00c3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:35.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.138+0000 7f03d5fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d0150eb0 0x7f03d00c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:35.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.138+0000 7f03d5fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d014fa40 con 0x7f03d00c5aa0 2023-06-22T08:25:35.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.138+0000 7f03d5fff700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d0207ed0 con 0x7f03d0150eb0 2023-06-22T08:25:35.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.138+0000 7f03d5fff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d014fdc0 con 0x7f03d015d9b0 2023-06-22T08:25:35.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.138+0000 7f03cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03d00c5aa0 0x7f03d014dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:35.142 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.139+0000 7f03cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03d00c5aa0 0x7f03d014dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38766/0 (socket says 172.21.15.23:38766) 2023-06-22T08:25:35.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.139+0000 7f03cf7fe700 1 -- 172.21.15.23:0/909626223 learned_addr learned my addr 172.21.15.23:0/909626223 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-06-22T08:25:35.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.139+0000 7f03c6ffd700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d015d9b0 0x7f03d00c3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:35.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.139+0000 7f03cffff700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d0150eb0 0x7f03d00c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:35.143 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.139+0000 7f03cf7fe700 1 -- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d015d9b0 msgr2=0x7f03d00c3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:35.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.139+0000 7f03cf7fe700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d015d9b0 0x7f03d00c3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.139+0000 7f03cf7fe700 1 -- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d0150eb0 msgr2=0x7f03d00c3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:35.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.139+0000 7f03cf7fe700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d0150eb0 0x7f03d00c3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.139+0000 7f03cf7fe700 1 -- 172.21.15.23:0/909626223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c8009b30 con 0x7f03d00c5aa0 2023-06-22T08:25:35.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.140+0000 7f03cf7fe700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03d00c5aa0 0x7f03d014dfd0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f03c000b4a0 tx=0x7f03c000b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:35.144 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.140+0000 7f03ccff9700 1 -- 172.21.15.23:0/909626223 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0011830 con 0x7f03d00c5aa0 2023-06-22T08:25:35.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.140+0000 7f03ccff9700 1 -- 172.21.15.23:0/909626223 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c0011990 con 0x7f03d00c5aa0 2023-06-22T08:25:35.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.140+0000 7f03ccff9700 1 -- 172.21.15.23:0/909626223 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0011d40 con 0x7f03d00c5aa0 2023-06-22T08:25:35.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.142+0000 7f03d5fff700 1 -- 172.21.15.23:0/909626223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d01500a0 con 0x7f03d00c5aa0 2023-06-22T08:25:35.145 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.142+0000 7f03d5fff700 1 -- 172.21.15.23:0/909626223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d0156110 con 0x7f03d00c5aa0 2023-06-22T08:25:35.148 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.146+0000 7f03ccff9700 1 -- 172.21.15.23:0/909626223 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 6) v1 ==== 89037+0+0 (secure 0 0 0) 0x7f03c001a400 con 0x7f03d00c5aa0 2023-06-22T08:25:35.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.147+0000 7f03ccff9700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f03b006ad30 0x7f03b006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:35.149 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.147+0000 7f03ccff9700 1 -- 172.21.15.23:0/909626223 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 8129+0+0 (secure 0 0 0) 0x7f03c0089be0 con 0x7f03d00c5aa0 2023-06-22T08:25:35.150 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.147+0000 7f03c6ffd700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f03b006ad30 0x7f03b006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T08:25:35.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.148+0000 7f03c6ffd700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f03b006ad30 0x7f03b006d1e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f03d006a760 tx=0x7f03b8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T08:25:35.151 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.148+0000 7f03d5fff700 1 -- 172.21.15.23:0/909626223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b40052f0 con 0x7f03d00c5aa0 2023-06-22T08:25:35.162 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.160+0000 7f03ccff9700 1 -- 172.21.15.23:0/909626223 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176884 (secure 0 0 0) 0x7f03c0015020 con 0x7f03d00c5aa0 2023-06-22T08:25:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.555+0000 7f03d5fff700 1 -- 172.21.15.23:0/909626223 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f03b40055a0 con 0x7f03d00c5aa0 2023-06-22T08:25:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.555+0000 7f03ccff9700 1 -- 172.21.15.23:0/909626223 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 28 entries v112) v1 ==== 81+0+1702 (secure 0 0 0) 0x7f03c00544d0 con 0x7f03d00c5aa0 2023-06-22T08:25:35.558 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6837/1502668188 2023-06-23T08:24:44.731195+0000 2023-06-22T08:25:35.558 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6836/1502668188 2023-06-23T08:24:44.731195+0000 2023-06-22T08:25:35.558 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6839/298860192 2023-06-23T08:23:34.705291+0000 2023-06-22T08:25:35.558 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6838/298860192 2023-06-23T08:23:34.705291+0000 2023-06-22T08:25:35.558 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6834/3080866617 2023-06-23T08:20:29.589258+0000 2023-06-22T08:25:35.559 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6835/2064462140 2023-06-23T08:18:54.558433+0000 2023-06-22T08:25:35.559 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6834/1707022233 2023-06-23T08:16:24.518029+0000 2023-06-22T08:25:35.559 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6834/3365097525 2023-06-23T08:05:56.351872+0000 2023-06-22T08:25:35.559 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6835/934828076 2023-06-23T08:22:04.631962+0000 2023-06-22T08:25:35.559 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6832/3365097525 2023-06-23T08:05:56.351872+0000 2023-06-22T08:25:35.559 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6835/3798178226 2023-06-23T08:03:59.299349+0000 2023-06-22T08:25:35.559 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6839/567663545 2023-06-23T08:11:04.427068+0000 2023-06-22T08:25:35.559 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6833/934828076 2023-06-23T08:22:04.631962+0000 2023-06-22T08:25:35.560 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6837/2064462140 2023-06-23T08:18:54.558433+0000 2023-06-22T08:25:35.560 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6836/2058668008 2023-06-23T08:14:04.481860+0000 2023-06-22T08:25:35.560 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6832/1707022233 2023-06-23T08:16:24.518029+0000 2023-06-22T08:25:35.560 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6836/1762110734 2023-06-23T08:03:09.287156+0000 2023-06-22T08:25:35.560 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6835/3894174037 2023-06-23T08:07:19.357578+0000 2023-06-22T08:25:35.560 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6837/1762110734 2023-06-23T08:03:09.287156+0000 2023-06-22T08:25:35.560 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6833/3894174037 2023-06-23T08:07:19.357578+0000 2023-06-22T08:25:35.561 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6834/1962314516 2023-06-23T08:08:34.377921+0000 2023-06-22T08:25:35.561 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6837/2058668008 2023-06-23T08:14:04.481860+0000 2023-06-22T08:25:35.561 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6836/1962314516 2023-06-23T08:08:34.377921+0000 2023-06-22T08:25:35.561 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6837/3798178226 2023-06-23T08:03:59.299349+0000 2023-06-22T08:25:35.561 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6838/567663545 2023-06-23T08:11:04.427068+0000 2023-06-22T08:25:35.561 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6835/1551460358 2023-06-23T08:13:09.469013+0000 2023-06-22T08:25:35.561 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6836/3080866617 2023-06-23T08:20:29.589258+0000 2023-06-22T08:25:35.561 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.67:6837/1551460358 2023-06-23T08:13:09.469013+0000 2023-06-22T08:25:35.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 -- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f03b006ad30 msgr2=0x7f03b006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:35.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f03b006ad30 0x7f03b006d1e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f03d006a760 tx=0x7f03b8009190 comp rx=0 tx=0).stop 2023-06-22T08:25:35.562 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 -- 172.21.15.23:0/909626223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03d00c5aa0 msgr2=0x7f03d014dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T08:25:35.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03d00c5aa0 0x7f03d014dfd0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f03c000b4a0 tx=0x7f03c000b7b0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 -- 172.21.15.23:0/909626223 shutdown_connections 2023-06-22T08:25:35.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:6832/3180103139,v1:172.21.15.67:6833/3180103139] conn(0x7f03b006ad30 0x7f03b006d1e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d0150eb0 0x7f03d00c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03d00c5aa0 0x7f03d014dfd0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 --2- 172.21.15.23:0/909626223 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f03d015d9b0 0x7f03d00c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:35.563 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 -- 172.21.15.23:0/909626223 >> 172.21.15.23:0/909626223 conn(0x7f03d00ca3f0 msgr2=0x7f03d0156570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T08:25:35.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.558+0000 7f03d5fff700 1 -- 172.21.15.23:0/909626223 shutdown_connections 2023-06-22T08:25:35.564 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:35.559+0000 7f03d5fff700 1 -- 172.21.15.23:0/909626223 wait complete. 2023-06-22T08:25:35.564 INFO:teuthology.orchestra.run.smithi023.stderr:listed 28 entries 2023-06-22T08:25:35.585 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-06-22T08:25:35.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:25:35.586 DEBUG:teuthology.orchestra.run.smithi023:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-06-22T08:25:35.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/glibc21.m4' 2023-06-22T08:25:35.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/perror.m4' 2023-06-22T08:25:35.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/backupfile.m4' 2023-06-22T08:25:35.711 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/xstrtod.m4' 2023-06-22T08:25:35.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/xattr.m4' 2023-06-22T08:25:35.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fsusage.m4' 2023-06-22T08:25:35.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/human.m4' 2023-06-22T08:25:35.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mbsrtowcs.m4' 2023-06-22T08:25:35.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/tm_gmtoff.m4' 2023-06-22T08:25:35.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strtod.m4' 2023-06-22T08:25:35.712 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/selinux-context-h.m4' 2023-06-22T08:25:35.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strtoumax.m4' 2023-06-22T08:25:35.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/inttypes-pri.m4' 2023-06-22T08:25:35.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/bison.m4' 2023-06-22T08:25:35.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/filenamecat.m4' 2023-06-22T08:25:35.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/i-ring.m4' 2023-06-22T08:25:35.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strstr.m4' 2023-06-22T08:25:35.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/calloc.m4' 2023-06-22T08:25:35.713 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getpass.m4' 2023-06-22T08:25:35.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/error.m4' 2023-06-22T08:25:35.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/vasnprintf.m4' 2023-06-22T08:25:35.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/exponentd.m4' 2023-06-22T08:25:35.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/unlocked-io.m4' 2023-06-22T08:25:35.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/nocrash.m4' 2023-06-22T08:25:35.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fstypename.m4' 2023-06-22T08:25:35.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/gethrxtime.m4' 2023-06-22T08:25:35.714 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/vasprintf.m4' 2023-06-22T08:25:35.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/isblank.m4' 2023-06-22T08:25:35.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/jm-winsz1.m4' 2023-06-22T08:25:35.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/memmove.m4' 2023-06-22T08:25:35.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/unlinkdir.m4' 2023-06-22T08:25:35.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sockets.m4' 2023-06-22T08:25:35.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/savewd.m4' 2023-06-22T08:25:35.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/frexpl.m4' 2023-06-22T08:25:35.715 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fopen.m4' 2023-06-22T08:25:35.716 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/unistd-safer.m4' 2023-06-22T08:25:35.716 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/utimecmp.m4' 2023-06-22T08:25:35.716 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mempcpy.m4' 2023-06-22T08:25:35.716 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mpsort.m4' 2023-06-22T08:25:35.716 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sys_stat_h.m4' 2023-06-22T08:25:35.716 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/exponentl.m4' 2023-06-22T08:25:35.716 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sha512.m4' 2023-06-22T08:25:35.716 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/time_r.m4' 2023-06-22T08:25:35.717 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sys_select_h.m4' 2023-06-22T08:25:35.717 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/vprintf-posix.m4' 2023-06-22T08:25:35.717 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/malloc.m4' 2023-06-22T08:25:35.717 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/tzset.m4' 2023-06-22T08:25:35.717 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/memxfrm.m4' 2023-06-22T08:25:35.718 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/flexmember.m4' 2023-06-22T08:25:35.718 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fcntl-o.m4' 2023-06-22T08:25:35.718 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/getpagesize.m4' 2023-06-22T08:25:35.718 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/unlink-busy.m4' 2023-06-22T08:25:35.718 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/fchdir.m4' 2023-06-22T08:25:35.718 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/malloca.m4' 2023-06-22T08:25:35.718 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/lib-ld.m4' 2023-06-22T08:25:35.718 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/euidaccess-stat.m4' 2023-06-22T08:25:35.719 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/xgetcwd.m4' 2023-06-22T08:25:35.719 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/dirent_h.m4' 2023-06-22T08:25:35.719 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/dup2.m4' 2023-06-22T08:25:35.719 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/locale_h.m4' 2023-06-22T08:25:35.719 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/stat-time.m4' 2023-06-22T08:25:35.719 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/select.m4' 2023-06-22T08:25:35.720 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/same.m4' 2023-06-22T08:25:35.720 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/freading.m4' 2023-06-22T08:25:35.720 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strtoll.m4' 2023-06-22T08:25:35.720 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/group-member.m4' 2023-06-22T08:25:35.720 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/inttypes_h.m4' 2023-06-22T08:25:35.720 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/long-options.m4' 2023-06-22T08:25:35.720 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/chown.m4' 2023-06-22T08:25:35.721 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/xalloc.m4' 2023-06-22T08:25:35.721 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/nanosleep.m4' 2023-06-22T08:25:35.721 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/utimes.m4' 2023-06-22T08:25:35.721 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/size_max.m4' 2023-06-22T08:25:35.721 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/frexp.m4' 2023-06-22T08:25:35.721 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/exclude.m4' 2023-06-22T08:25:35.721 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/wint_t.m4' 2023-06-22T08:25:35.722 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strndup.m4' 2023-06-22T08:25:35.722 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/lib-check.m4' 2023-06-22T08:25:35.723 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mbiter.m4' 2023-06-22T08:25:35.723 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/mbstate_t.m4' 2023-06-22T08:25:35.723 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/rmdir.m4' 2023-06-22T08:25:35.723 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/inet_ntop.m4' 2023-06-22T08:25:35.723 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/version-etc.m4' 2023-06-22T08:25:35.723 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/isnanl.m4' 2023-06-22T08:25:35.723 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/signbit.m4' 2023-06-22T08:25:35.724 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/strtoul.m4' 2023-06-22T08:25:35.724 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/m4/sleep.m4' 2023-06-22T08:25:35.724 INFO:tasks.workunit.client.0.smithi023.stdout:removed directory 'k/coreutils-8.5/m4' 2023-06-22T08:25:35.724 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.version' 2023-06-22T08:25:35.724 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/.x-sc_prohibit_fail_0' 2023-06-22T08:25:35.724 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/Makefile.am' 2023-06-22T08:25:35.724 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/INSTALL' 2023-06-22T08:25:35.725 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/maint.mk' 2023-06-22T08:25:35.725 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/bootstrap.conf' 2023-06-22T08:25:35.725 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/gnulib-tests/test-xstrtoimax.c' 2023-06-22T08:25:35.725 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/gnulib-tests/test-update-copyright.sh' 2023-06-22T08:25:35.725 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/gnulib-tests/connect.c' 2023-06-22T08:25:35.725 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/gnulib-tests/test-signbit.c' 2023-06-22T08:25:35.725 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/gnulib-tests/test-stdlib.c' 2023-06-22T08:25:35.809 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/gnulib-tests/test-wcrtomb.c' 2023-06-22T08:25:35.809 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/gnulib-tests/test-sameacls.c' 2023-06-22T08:25:35.810 INFO:tasks.workunit.client.0.smithi023.stdout:removed 'k/coreutils-8.5/gnulib-tests/test-isnanf-nolibm.c' 2023-06-22T08:25:35.810 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-wcwidth.c': Input/output error 2023-06-22T08:25:35.810 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-argmatch.c': Input/output error 2023-06-22T08:25:35.810 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-dirname.c': Input/output error 2023-06-22T08:25:35.810 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fseeko.c': Input/output error 2023-06-22T08:25:35.810 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-sys_select.c': Input/output error 2023-06-22T08:25:35.811 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freopen-safer.c': Input/output error 2023-06-22T08:25:35.811 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbrtowc.c': Input/output error 2023-06-22T08:25:35.811 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fseterr.c': Input/output error 2023-06-22T08:25:35.811 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-vasprintf.c': Input/output error 2023-06-22T08:25:35.811 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fopen.h': Input/output error 2023-06-22T08:25:35.811 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-ftello.sh': Input/output error 2023-06-22T08:25:35.811 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsinit.sh': Input/output error 2023-06-22T08:25:35.811 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-dup-safer.c': Input/output error 2023-06-22T08:25:35.812 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsstr3.c': Input/output error 2023-06-22T08:25:35.812 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-vprintf-posix.sh': Input/output error 2023-06-22T08:25:35.812 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-userspec.c': Input/output error 2023-06-22T08:25:35.812 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-isnand.h': Input/output error 2023-06-22T08:25:35.812 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freopen.c': Input/output error 2023-06-22T08:25:35.812 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-file-has-acl.sh': Input/output error 2023-06-22T08:25:35.812 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbrtowc4.sh': Input/output error 2023-06-22T08:25:35.813 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsrtowcs2.sh': Input/output error 2023-06-22T08:25:35.813 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fseeko.sh': Input/output error 2023-06-22T08:25:35.813 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/nan.h': Input/output error 2023-06-22T08:25:35.813 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-sys_ioctl.c': Input/output error 2023-06-22T08:25:35.813 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-btowc.c': Input/output error 2023-06-22T08:25:35.813 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getopt.h': Input/output error 2023-06-22T08:25:35.813 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mkdirat.c': Input/output error 2023-06-22T08:25:35.813 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-stdint.c': Input/output error 2023-06-22T08:25:35.814 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-c-strcasecmp.c': Input/output error 2023-06-22T08:25:35.814 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsstr3.sh': Input/output error 2023-06-22T08:25:35.814 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-readlink.h': Input/output error 2023-06-22T08:25:35.814 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbrtowc1.sh': Input/output error 2023-06-22T08:25:35.814 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-exclude.c': Input/output error 2023-06-22T08:25:35.814 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freadptr.c': Input/output error 2023-06-22T08:25:35.814 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fflush2.sh': Input/output error 2023-06-22T08:25:35.814 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-inttypes.c': Input/output error 2023-06-22T08:25:35.815 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-sys_utsname.c': Input/output error 2023-06-22T08:25:35.815 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-math.c': Input/output error 2023-06-22T08:25:35.815 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/signature.h': Input/output error 2023-06-22T08:25:35.815 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/w32sock.h': Input/output error 2023-06-22T08:25:35.815 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-wchar.c': Input/output error 2023-06-22T08:25:35.815 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/uniwidth': Input/output error 2023-06-22T08:25:35.815 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-lstat.c': Input/output error 2023-06-22T08:25:35.815 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/unlinkdir.h': Input/output error 2023-06-22T08:25:35.816 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbscasecmp.c': Input/output error 2023-06-22T08:25:35.816 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-netdb.c': Input/output error 2023-06-22T08:25:35.816 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-stat.c': Input/output error 2023-06-22T08:25:35.816 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-open.c': Input/output error 2023-06-22T08:25:35.816 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-strsignal.c': Input/output error 2023-06-22T08:25:35.816 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/accept.c': Input/output error 2023-06-22T08:25:35.816 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xfprintf-posix.c': Input/output error 2023-06-22T08:25:35.816 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-sockets.c': Input/output error 2023-06-22T08:25:35.817 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xalloc-die.c': Input/output error 2023-06-22T08:25:35.817 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fchdir.c': Input/output error 2023-06-22T08:25:35.817 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-sigaction.c': Input/output error 2023-06-22T08:25:35.817 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-i-ring.c': Input/output error 2023-06-22T08:25:35.817 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-read-file.c': Input/output error 2023-06-22T08:25:35.817 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-unlinkat.c': Input/output error 2023-06-22T08:25:35.817 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-stddef.c': Input/output error 2023-06-22T08:25:35.817 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-linkat.c': Input/output error 2023-06-22T08:25:35.818 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-openat-safer.c': Input/output error 2023-06-22T08:25:35.818 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-gettimeofday.c': Input/output error 2023-06-22T08:25:35.818 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-remove.c': Input/output error 2023-06-22T08:25:35.818 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-strtod.c': Input/output error 2023-06-22T08:25:35.818 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-unlink.h': Input/output error 2023-06-22T08:25:35.818 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-exclude6.sh': Input/output error 2023-06-22T08:25:35.818 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-argv-iter.c': Input/output error 2023-06-22T08:25:35.819 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-lstat.h': Input/output error 2023-06-22T08:25:35.819 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/unistr': Input/output error 2023-06-22T08:25:35.819 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/setsockopt.c': Input/output error 2023-06-22T08:25:35.819 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-exclude3.sh': Input/output error 2023-06-22T08:25:35.819 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/ioctl.c': Input/output error 2023-06-22T08:25:35.819 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/locale': Input/output error 2023-06-22T08:25:35.820 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-c-strncasecmp.c': Input/output error 2023-06-22T08:25:35.820 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fpurge.c': Input/output error 2023-06-22T08:25:35.820 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-exclude5.sh': Input/output error 2023-06-22T08:25:35.820 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-arpa_inet.c': Input/output error 2023-06-22T08:25:35.820 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fopen-safer.c': Input/output error 2023-06-22T08:25:35.820 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-nl_langinfo.c': Input/output error 2023-06-22T08:25:35.820 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-time.c': Input/output error 2023-06-22T08:25:35.821 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-netinet_in.c': Input/output error 2023-06-22T08:25:35.821 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fopen.c': Input/output error 2023-06-22T08:25:35.821 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fnmatch.c': Input/output error 2023-06-22T08:25:35.821 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/bind.c': Input/output error 2023-06-22T08:25:35.821 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freadahead.sh': Input/output error 2023-06-22T08:25:35.821 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-lutimens.h': Input/output error 2023-06-22T08:25:35.821 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getndelim2.c': Input/output error 2023-06-22T08:25:35.821 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-printf-posix.h': Input/output error 2023-06-22T08:25:35.822 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xprintf-posix.c': Input/output error 2023-06-22T08:25:35.822 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-environ.c': Input/output error 2023-06-22T08:25:35.822 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-uname.c': Input/output error 2023-06-22T08:25:35.822 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/glthread': Input/output error 2023-06-22T08:25:35.822 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/gnulib.mk': Input/output error 2023-06-22T08:25:35.823 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-areadlink.c': Input/output error 2023-06-22T08:25:35.823 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-link.h': Input/output error 2023-06-22T08:25:35.824 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-nl_langinfo.sh': Input/output error 2023-06-22T08:25:35.824 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-quotearg.c': Input/output error 2023-06-22T08:25:35.824 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-vasnprintf.c': Input/output error 2023-06-22T08:25:35.825 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-atexit.sh': Input/output error 2023-06-22T08:25:35.825 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/sleep.c': Input/output error 2023-06-22T08:25:35.825 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fpending.c': Input/output error 2023-06-22T08:25:35.825 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-isblank.c': Input/output error 2023-06-22T08:25:35.825 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xalloc-die.sh': Input/output error 2023-06-22T08:25:35.826 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-wctype.c': Input/output error 2023-06-22T08:25:35.826 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-utimens.h': Input/output error 2023-06-22T08:25:35.826 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-file-has-acl.c': Input/output error 2023-06-22T08:25:35.826 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-set-mode-acl.sh': Input/output error 2023-06-22T08:25:35.827 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-btowc2.sh': Input/output error 2023-06-22T08:25:35.827 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-sys_time.c': Input/output error 2023-06-22T08:25:35.827 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-areadlink.h': Input/output error 2023-06-22T08:25:35.828 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freadptr2.sh': Input/output error 2023-06-22T08:25:35.828 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsrtowcs3.sh': Input/output error 2023-06-22T08:25:35.828 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-vfprintf-posix.c': Input/output error 2023-06-22T08:25:35.828 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fsync.c': Input/output error 2023-06-22T08:25:35.829 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-vc-list-files-cvs.sh': Input/output error 2023-06-22T08:25:35.829 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-alloca-opt.c': Input/output error 2023-06-22T08:25:35.830 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-yesno.sh': Input/output error 2023-06-22T08:25:35.830 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-memrchr.c': Input/output error 2023-06-22T08:25:35.830 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-c-strcase.sh': Input/output error 2023-06-22T08:25:35.831 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-select-out.sh': Input/output error 2023-06-22T08:25:35.831 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-binary-io.sh': Input/output error 2023-06-22T08:25:35.832 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-wcrtomb.sh': Input/output error 2023-06-22T08:25:35.832 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-ftello2.sh': Input/output error 2023-06-22T08:25:35.832 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mkdir.h': Input/output error 2023-06-22T08:25:35.832 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xstrtoul.c': Input/output error 2023-06-22T08:25:35.832 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-printf-frexp.c': Input/output error 2023-06-22T08:25:35.832 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-snprintf.c': Input/output error 2023-06-22T08:25:35.832 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-isnand-nolibm.c': Input/output error 2023-06-22T08:25:35.833 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-readlink.c': Input/output error 2023-06-22T08:25:35.833 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-sleep.c': Input/output error 2023-06-22T08:25:35.833 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/usleep.c': Input/output error 2023-06-22T08:25:35.833 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xstrtoimax.sh': Input/output error 2023-06-22T08:25:35.833 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-stdbool.c': Input/output error 2023-06-22T08:25:35.833 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-frexpl.c': Input/output error 2023-06-22T08:25:35.834 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-alignof.c': Input/output error 2023-06-22T08:25:35.834 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fflush.c': Input/output error 2023-06-22T08:25:35.834 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-ctype.c': Input/output error 2023-06-22T08:25:35.834 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/socket.c': Input/output error 2023-06-22T08:25:35.834 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-yesno.c': Input/output error 2023-06-22T08:25:35.834 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/Makefile.in': Input/output error 2023-06-22T08:25:35.835 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-filenamecat.c': Input/output error 2023-06-22T08:25:35.835 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-link.c': Input/output error 2023-06-22T08:25:35.835 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-nanosleep.c': Input/output error 2023-06-22T08:25:35.835 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-sys_socket.c': Input/output error 2023-06-22T08:25:35.835 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsrtowcs4.sh': Input/output error 2023-06-22T08:25:35.835 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-copy-acl.c': Input/output error 2023-06-22T08:25:35.835 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-rmdir.h': Input/output error 2023-06-22T08:25:35.836 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-isnanl.h': Input/output error 2023-06-22T08:25:35.836 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-signal.c': Input/output error 2023-06-22T08:25:35.836 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbrtowc2.sh': Input/output error 2023-06-22T08:25:35.836 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-iconv-h.c': Input/output error 2023-06-22T08:25:35.836 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-exclude2.sh': Input/output error 2023-06-22T08:25:35.836 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-perror.sh': Input/output error 2023-06-22T08:25:35.836 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-langinfo.c': Input/output error 2023-06-22T08:25:35.837 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-vprintf-posix.c': Input/output error 2023-06-22T08:25:35.837 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-rmdir.c': Input/output error 2023-06-22T08:25:35.837 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fcntl.c': Input/output error 2023-06-22T08:25:35.837 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-dirent.c': Input/output error 2023-06-22T08:25:35.837 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-chown.h': Input/output error 2023-06-22T08:25:35.837 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/read-file.c': Input/output error 2023-06-22T08:25:35.837 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-usleep.c': Input/output error 2023-06-22T08:25:35.837 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-base64.c': Input/output error 2023-06-22T08:25:35.838 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-atexit.c': Input/output error 2023-06-22T08:25:35.838 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-memchr2.c': Input/output error 2023-06-22T08:25:35.838 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fdopendir.c': Input/output error 2023-06-22T08:25:35.838 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-stdio.c': Input/output error 2023-06-22T08:25:35.839 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-vfprintf-posix.sh': Input/output error 2023-06-22T08:25:35.839 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getdtablesize.c': Input/output error 2023-06-22T08:25:35.839 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xstrtoumax.c': Input/output error 2023-06-22T08:25:35.839 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fpending.sh': Input/output error 2023-06-22T08:25:35.839 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-version-etc.c': Input/output error 2023-06-22T08:25:35.839 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getgroups.c': Input/output error 2023-06-22T08:25:35.839 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freadptr2.c': Input/output error 2023-06-22T08:25:35.840 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-select-fd.c': Input/output error 2023-06-22T08:25:35.840 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fcntl-safer.c': Input/output error 2023-06-22T08:25:35.840 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fflush2.c': Input/output error 2023-06-22T08:25:35.840 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-select-in.sh': Input/output error 2023-06-22T08:25:35.840 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-dup2.c': Input/output error 2023-06-22T08:25:35.840 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-frexp.c': Input/output error 2023-06-22T08:25:35.840 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsstr2.sh': Input/output error 2023-06-22T08:25:35.840 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/Makefile.am': Input/output error 2023-06-22T08:25:35.841 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-select-stdin.c': Input/output error 2023-06-22T08:25:35.841 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getdelim.c': Input/output error 2023-06-22T08:25:35.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T08:25:35.842 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-btowc1.sh': Input/output error 2023-06-22T08:25:35.842 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-locale.c': Input/output error 2023-06-22T08:25:35.842 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-md5.c': Input/output error 2023-06-22T08:25:35.843 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-rename.h': Input/output error 2023-06-22T08:25:35.843 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-utimens.c': Input/output error 2023-06-22T08:25:35.843 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-strerror.c': Input/output error 2023-06-22T08:25:35.843 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-inet_pton.c': Input/output error 2023-06-22T08:25:35.843 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-priv-set.c': Input/output error 2023-06-22T08:25:35.843 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-open.h': Input/output error 2023-06-22T08:25:35.843 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-unlink.c': Input/output error 2023-06-22T08:25:35.843 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/macros.h': Input/output error 2023-06-22T08:25:35.844 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-rename.c': Input/output error 2023-06-22T08:25:35.844 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xstrtoumax.sh': Input/output error 2023-06-22T08:25:35.844 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/inet_pton.c': Input/output error 2023-06-22T08:25:35.844 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-memcmp.c': Input/output error 2023-06-22T08:25:35.844 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xvasprintf.c': Input/output error 2023-06-22T08:25:35.844 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freadahead.c': Input/output error 2023-06-22T08:25:35.845 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-printf-posix.output': Input/output error 2023-06-22T08:25:35.845 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-openat.c': Input/output error 2023-06-22T08:25:35.845 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-c-ctype.c': Input/output error 2023-06-22T08:25:35.845 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-chown.c': Input/output error 2023-06-22T08:25:35.845 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-sys_stat.c': Input/output error 2023-06-22T08:25:35.845 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-closein.sh': Input/output error 2023-06-22T08:25:35.845 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/init.sh': Input/output error 2023-06-22T08:25:35.845 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-filevercmp.c': Input/output error 2023-06-22T08:25:35.846 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getaddrinfo.c': Input/output error 2023-06-22T08:25:35.846 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-areadlinkat.c': Input/output error 2023-06-22T08:25:35.846 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-isnanl-nolibm.c': Input/output error 2023-06-22T08:25:35.846 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-lchown.h': Input/output error 2023-06-22T08:25:35.846 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getline.c': Input/output error 2023-06-22T08:25:35.846 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-hash.c': Input/output error 2023-06-22T08:25:35.846 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-canonicalize.c': Input/output error 2023-06-22T08:25:35.847 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fchownat.c': Input/output error 2023-06-22T08:25:35.847 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-striconv.c': Input/output error 2023-06-22T08:25:35.847 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-binary-io.c': Input/output error 2023-06-22T08:25:35.847 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-malloca.c': Input/output error 2023-06-22T08:25:35.847 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freadseek.c': Input/output error 2023-06-22T08:25:35.847 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-closein.c': Input/output error 2023-06-22T08:25:35.847 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-unistd.c': Input/output error 2023-06-22T08:25:35.847 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/listen.c': Input/output error 2023-06-22T08:25:35.848 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-posixtm.c': Input/output error 2023-06-22T08:25:35.848 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbrtowc3.sh': Input/output error 2023-06-22T08:25:35.848 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsstr2.c': Input/output error 2023-06-22T08:25:35.848 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/unlinkdir.c': Input/output error 2023-06-22T08:25:35.848 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-version-etc.sh': Input/output error 2023-06-22T08:25:35.848 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-lseek.sh': Input/output error 2023-06-22T08:25:35.848 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/nap.h': Input/output error 2023-06-22T08:25:35.849 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-gethostname.c': Input/output error 2023-06-22T08:25:35.849 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freading.c': Input/output error 2023-06-22T08:25:35.849 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-areadlink-with-size.c': Input/output error 2023-06-22T08:25:35.849 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freadseek.sh': Input/output error 2023-06-22T08:25:35.849 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-exclude7.sh': Input/output error 2023-06-22T08:25:35.849 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsalign.c': Input/output error 2023-06-22T08:25:35.849 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/wctob.c': Input/output error 2023-06-22T08:25:35.849 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-quotearg.sh': Input/output error 2023-06-22T08:25:35.850 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbscasecmp.sh': Input/output error 2023-06-22T08:25:35.850 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/binary-io.h': Input/output error 2023-06-22T08:25:35.850 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getopt.c': Input/output error 2023-06-22T08:25:35.850 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-select.c': Input/output error 2023-06-22T08:25:35.850 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-u64.c': Input/output error 2023-06-22T08:25:35.850 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-stat.h': Input/output error 2023-06-22T08:25:35.850 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-cloexec.c': Input/output error 2023-06-22T08:25:35.850 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-freadptr.sh': Input/output error 2023-06-22T08:25:35.851 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-lseek.c': Input/output error 2023-06-22T08:25:35.851 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-perror.c': Input/output error 2023-06-22T08:25:35.851 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-bitrotate.c': Input/output error 2023-06-22T08:25:35.851 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-stat-time.c': Input/output error 2023-06-22T08:25:35.851 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-ftello.c': Input/output error 2023-06-22T08:25:35.851 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getcwd.c': Input/output error 2023-06-22T08:25:35.851 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-symlink.h': Input/output error 2023-06-22T08:25:35.851 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-inet_ntop.c': Input/output error 2023-06-22T08:25:35.852 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/zerosize-ptr.h': Input/output error 2023-06-22T08:25:35.852 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-sha1.c': Input/output error 2023-06-22T08:25:35.852 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fcntl-h.c': Input/output error 2023-06-22T08:25:35.852 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fseeko2.sh': Input/output error 2023-06-22T08:25:35.852 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-printf-frexpl.c': Input/output error 2023-06-22T08:25:35.852 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsrtowcs.c': Input/output error 2023-06-22T08:25:35.852 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-futimens.h': Input/output error 2023-06-22T08:25:35.852 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mkdir.c': Input/output error 2023-06-22T08:25:35.853 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsrtowcs1.sh': Input/output error 2023-06-22T08:25:35.853 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-exclude4.sh': Input/output error 2023-06-22T08:25:35.853 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-exclude1.sh': Input/output error 2023-06-22T08:25:35.853 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-symlink.c': Input/output error 2023-06-22T08:25:35.853 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-unsetenv.c': Input/output error 2023-06-22T08:25:35.853 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xstrtol.c': Input/output error 2023-06-22T08:25:35.853 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsstr1.c': Input/output error 2023-06-22T08:25:35.853 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-lchown.c': Input/output error 2023-06-22T08:25:35.854 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-symlinkat.c': Input/output error 2023-06-22T08:25:35.854 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-set-mode-acl.c': Input/output error 2023-06-22T08:25:35.854 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xprintf-posix.sh': Input/output error 2023-06-22T08:25:35.854 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-lock.c': Input/output error 2023-06-22T08:25:35.854 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-copy-acl.sh': Input/output error 2023-06-22T08:25:35.854 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/sys_ioctl.in.h': Input/output error 2023-06-22T08:25:35.854 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/read-file.h': Input/output error 2023-06-22T08:25:35.854 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fstatat.c': Input/output error 2023-06-22T08:25:35.855 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-xstrtol.sh': Input/output error 2023-06-22T08:25:35.855 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-isnanf.h': Input/output error 2023-06-22T08:25:35.855 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getopt_long.h': Input/output error 2023-06-22T08:25:35.855 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-memchr.c': Input/output error 2023-06-22T08:25:35.855 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-fprintf-posix.h': Input/output error 2023-06-22T08:25:35.855 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-setenv.c': Input/output error 2023-06-22T08:25:35.855 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-iconv.c': Input/output error 2023-06-22T08:25:35.856 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-getdate.c': Input/output error 2023-06-22T08:25:35.856 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-dirent-safer.c': Input/output error 2023-06-22T08:25:35.856 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-vasprintf-posix.c': Input/output error 2023-06-22T08:25:35.856 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-errno.c': Input/output error 2023-06-22T08:25:35.856 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-vc-list-files-git.sh': Input/output error 2023-06-22T08:25:35.856 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-tls.c': Input/output error 2023-06-22T08:25:35.856 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-mbsinit.c': Input/output error 2023-06-22T08:25:35.857 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-string.c': Input/output error 2023-06-22T08:25:35.857 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/perror.c': Input/output error 2023-06-22T08:25:35.857 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/gnulib-tests/test-utimens-common.h': Input/output error 2023-06-22T08:25:35.857 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/AUTHORS': Input/output error 2023-06-22T08:25:35.857 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/.x-sc_po_check': Input/output error 2023-06-22T08:25:35.857 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/dist-check.mk': Input/output error 2023-06-22T08:25:35.857 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/ChangeLog-2008': Input/output error 2023-06-22T08:25:35.858 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/.x-sc_trailing_blank': Input/output error 2023-06-22T08:25:35.858 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/.x-sc_file_system': Input/output error 2023-06-22T08:25:35.858 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/README': Input/output error 2023-06-22T08:25:35.858 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/configure.ac': Input/output error 2023-06-22T08:25:35.858 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/.x-sc_system_h_headers': Input/output error 2023-06-22T08:25:35.858 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/THANKStt.in': Input/output error 2023-06-22T08:25:35.858 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/old': Input/output error 2023-06-22T08:25:35.858 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/.x-sc_sun_os_names': Input/output error 2023-06-22T08:25:35.859 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/THANKS-to-translators': Input/output error 2023-06-22T08:25:35.859 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/build-aux': Input/output error 2023-06-22T08:25:35.859 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/configure': Input/output error 2023-06-22T08:25:35.859 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/.vg-suppressions': Input/output error 2023-06-22T08:25:35.859 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/.x-sc_obsolete_symbols': Input/output error 2023-06-22T08:25:35.859 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/.x-sc_prohibit_atoi_atof': Input/output error 2023-06-22T08:25:35.859 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/COPYING': Input/output error 2023-06-22T08:25:35.859 INFO:tasks.workunit.client.0.smithi023.stderr:rm: cannot remove 'k/coreutils-8.5/cfg.mk': Input/output error 2023-06-22T08:25:35.860 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-06-22T08:25:35.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-06-22T08:25:36.046 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T08:25:36.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:25:36.046 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-06-22T08:25:37.538 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' 2023-06-22T08:25:37.607 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6e13860d1eca4ac58c3c9dc45a4e8024 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' 2023-06-22T08:25:37.610 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-06-22T08:25:37.620 INFO:tasks.mds_thrash:joining mds_thrasher 2023-06-22T08:25:37.623 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 1 2023-06-22T08:25:37.624 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2023-06-22T08:25:37.624 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 13 2023-06-22T08:25:37.624 INFO:tasks.mds_thrash:done joining 2023-06-22T08:25:37.625 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-06-22T08:25:37.691 INFO:tasks.kclient:Unmounting kernel clients... 2023-06-22T08:25:37.691 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-06-22T08:25:37.692 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2023-06-22T08:25:37.736 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345736k,nr_inodes=4086434,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275012k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 2023-06-22T08:25:37.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:25:37.737 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2023-06-22T08:25:37.840 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-06-22T08:25:37.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:25:37.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-06-22T08:25:37.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-22T08:25:37.912 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2023-06-22T08:25:38.022 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-06-22T08:25:38.031 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-06-22T08:25:38.031 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-06-22T08:25:38.096 INFO:tasks.ceph.osd.0:Sent signal 15 2023-06-22T08:25:38.097 INFO:tasks.ceph.osd.1:Sent signal 15 2023-06-22T08:25:38.097 INFO:tasks.ceph.osd.2:Sent signal 15 2023-06-22T08:25:38.098 INFO:tasks.ceph.osd.3:Sent signal 15 2023-06-22T08:25:38.098 INFO:tasks.ceph.osd.4:Sent signal 15 2023-06-22T08:25:38.098 INFO:tasks.ceph.osd.5:Sent signal 15 2023-06-22T08:25:38.099 INFO:tasks.ceph.osd.6:Sent signal 15 2023-06-22T08:25:38.099 INFO:tasks.ceph.osd.7:Sent signal 15 2023-06-22T08:25:38.100 INFO:tasks.ceph.mds.a:Sent signal 15 2023-06-22T08:25:38.100 INFO:tasks.ceph.mds.c:Sent signal 15 2023-06-22T08:25:38.100 INFO:tasks.ceph.mds.e:Sent signal 15 2023-06-22T08:25:38.101 INFO:tasks.ceph.mds.d:Sent signal 15 2023-06-22T08:25:38.101 INFO:tasks.ceph.mds.f:Sent signal 15 2023-06-22T08:25:38.102 INFO:tasks.ceph.mon.a:Sent signal 15 2023-06-22T08:25:38.102 INFO:tasks.ceph.mon.b:Sent signal 15 2023-06-22T08:25:38.102 INFO:tasks.ceph.mon.c:Sent signal 15 2023-06-22T08:25:38.103 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-06-22T08:25:38.103 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-06-22T08:25:38.104 INFO:tasks.ceph.osd.5.smithi067.stderr:2023-06-22T08:25:38.098+0000 7f2e2eb7d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 130915) UID: 0 2023-06-22T08:25:38.104 INFO:tasks.ceph.osd.5.smithi067.stderr:2023-06-22T08:25:38.098+0000 7f2e2eb7d700 -1 osd.5 112 *** Got signal Terminated *** 2023-06-22T08:25:38.104 INFO:tasks.ceph.osd.5.smithi067.stderr:2023-06-22T08:25:38.098+0000 7f2e2eb7d700 -1 osd.5 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T08:25:38.105 INFO:tasks.ceph.osd.7.smithi067.stderr:2023-06-22T08:25:38.098+0000 7f5910891700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 130920) UID: 0 2023-06-22T08:25:38.105 INFO:tasks.ceph.osd.7.smithi067.stderr:2023-06-22T08:25:38.098+0000 7f5910891700 -1 osd.7 112 *** Got signal Terminated *** 2023-06-22T08:25:38.105 INFO:tasks.ceph.osd.7.smithi067.stderr:2023-06-22T08:25:38.098+0000 7f5910891700 -1 osd.7 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T08:25:38.105 INFO:tasks.ceph.osd.6.smithi067.stderr:2023-06-22T08:25:38.098+0000 7f4a240f2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 130919) UID: 0 2023-06-22T08:25:38.106 INFO:tasks.ceph.osd.6.smithi067.stderr:2023-06-22T08:25:38.098+0000 7f4a240f2700 -1 osd.6 112 *** Got signal Terminated *** 2023-06-22T08:25:38.106 INFO:tasks.ceph.osd.6.smithi067.stderr:2023-06-22T08:25:38.098+0000 7f4a240f2700 -1 osd.6 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T08:25:38.106 INFO:tasks.ceph.osd.4.smithi067.stderr:2023-06-22T08:25:38.099+0000 7ffbdfe8b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 130913) UID: 0 2023-06-22T08:25:38.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2023-06-22T08:25:38.099+0000 7ffbdfe8b700 -1 osd.4 112 *** Got signal Terminated *** 2023-06-22T08:25:38.107 INFO:tasks.ceph.osd.4.smithi067.stderr:2023-06-22T08:25:38.099+0000 7ffbdfe8b700 -1 osd.4 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T08:25:38.107 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T08:25:38.100+0000 7f683cc89700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i d (PID: 138498) UID: 0 2023-06-22T08:25:38.108 INFO:tasks.ceph.mds.d.smithi067.stderr:2023-06-22T08:25:38.100+0000 7f683cc89700 -1 mds.d *** got signal Terminated *** 2023-06-22T08:25:38.108 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:25:38.101+0000 7f6a18470700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i f (PID: 138902) UID: 0 2023-06-22T08:25:38.108 INFO:tasks.ceph.mds.f.smithi067.stderr:2023-06-22T08:25:38.101+0000 7f6a18470700 -1 mds.f *** got signal Terminated *** 2023-06-22T08:25:38.109 INFO:tasks.ceph.mon.b.smithi067.stderr:2023-06-22T08:25:38.101+0000 7fc2d350e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 130251) UID: 0 2023-06-22T08:25:38.109 INFO:tasks.ceph.mon.b.smithi067.stderr:2023-06-22T08:25:38.101+0000 7fc2d350e700 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-06-22T08:25:38.109 INFO:tasks.ceph.mon.c.smithi067.stderr:2023-06-22T08:25:38.101+0000 7f9b3c151700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 130254) UID: 0 2023-06-22T08:25:38.110 INFO:tasks.ceph.mon.c.smithi067.stderr:2023-06-22T08:25:38.101+0000 7f9b3c151700 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-06-22T08:25:38.110 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T08:25:38.102+0000 7f95aa8f7700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 130257) UID: 0 2023-06-22T08:25:38.110 INFO:tasks.ceph.mgr.x.smithi067.stderr:2023-06-22T08:25:38.102+0000 7f95aa8f7700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-06-22T08:25:38.111 INFO:tasks.ceph.osd.0.smithi023.stderr:2023-06-22T08:25:38.096+0000 7f7cf6ff9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 129936) UID: 0 2023-06-22T08:25:38.111 INFO:tasks.ceph.osd.0.smithi023.stderr:2023-06-22T08:25:38.097+0000 7f7cf6ff9700 -1 osd.0 112 *** Got signal Terminated *** 2023-06-22T08:25:38.111 INFO:tasks.ceph.osd.0.smithi023.stderr:2023-06-22T08:25:38.097+0000 7f7cf6ff9700 -1 osd.0 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T08:25:38.112 INFO:tasks.ceph.osd.1.smithi023.stderr:2023-06-22T08:25:38.097+0000 7f3e2d9f9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 129938) UID: 0 2023-06-22T08:25:38.112 INFO:tasks.ceph.osd.1.smithi023.stderr:2023-06-22T08:25:38.097+0000 7f3e2d9f9700 -1 osd.1 112 *** Got signal Terminated *** 2023-06-22T08:25:38.112 INFO:tasks.ceph.osd.1.smithi023.stderr:2023-06-22T08:25:38.097+0000 7f3e2d9f9700 -1 osd.1 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T08:25:38.113 INFO:tasks.ceph.osd.2.smithi023.stderr:2023-06-22T08:25:38.096+0000 7fd0e51f9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 129944) UID: 0 2023-06-22T08:25:38.113 INFO:tasks.ceph.osd.2.smithi023.stderr:2023-06-22T08:25:38.096+0000 7fd0e51f9700 -1 osd.2 112 *** Got signal Terminated *** 2023-06-22T08:25:38.113 INFO:tasks.ceph.osd.2.smithi023.stderr:2023-06-22T08:25:38.096+0000 7fd0e51f9700 -1 osd.2 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T08:25:38.114 INFO:tasks.ceph.osd.3.smithi023.stderr:2023-06-22T08:25:38.097+0000 7f3f239f9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 129945) UID: 0 2023-06-22T08:25:38.114 INFO:tasks.ceph.osd.3.smithi023.stderr:2023-06-22T08:25:38.097+0000 7f3f239f9700 -1 osd.3 112 *** Got signal Terminated *** 2023-06-22T08:25:38.114 INFO:tasks.ceph.osd.3.smithi023.stderr:2023-06-22T08:25:38.097+0000 7f3f239f9700 -1 osd.3 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-22T08:25:38.115 INFO:tasks.ceph.mds.e.smithi023.stderr:2023-06-22T08:25:38.101+0000 7f61057f9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i e (PID: 161822) UID: 0 2023-06-22T08:25:38.115 INFO:tasks.ceph.mds.e.smithi023.stderr:2023-06-22T08:25:38.101+0000 7f61057f9700 -1 mds.e *** got signal Terminated *** 2023-06-22T08:25:38.115 INFO:tasks.ceph.mon.a.smithi023.stderr:2023-06-22T08:25:38.101+0000 7f2a197ff700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 129638) UID: 0 2023-06-22T08:25:38.116 INFO:tasks.ceph.mon.a.smithi023.stderr:2023-06-22T08:25:38.101+0000 7f2a197ff700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-06-22T08:25:38.116 INFO:tasks.ceph.mds.c.smithi023.stderr:2023-06-22T08:25:38.101+0000 7f4bd5ff9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 155288) UID: 0 2023-06-22T08:25:38.116 INFO:tasks.ceph.mds.c.smithi023.stderr:2023-06-22T08:25:38.101+0000 7f4bd5ff9700 -1 mds.c *** got signal Terminated *** 2023-06-22T08:25:38.117 INFO:tasks.ceph.mds.a.smithi023.stderr:2023-06-22T08:25:38.101+0000 7fd0c41f9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i a (PID: 159180) UID: 0 2023-06-22T08:25:38.118 INFO:tasks.ceph.mds.a.smithi023.stderr:2023-06-22T08:25:38.101+0000 7fd0c41f9700 -1 mds.a *** got signal Terminated *** 2023-06-22T08:25:38.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.261+0000 7f0f9efff700 1 Processor -- start 2023-06-22T08:25:38.263 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.261+0000 7f0f9efff700 1 -- start start 2023-06-22T08:25:38.264 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.262+0000 7f0f9efff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 0x7f0f980c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:38.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.262+0000 7f0f9efff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 0x7f0f98160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:38.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.262+0000 7f0f9efff700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f9806d050 2023-06-22T08:25:38.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.262+0000 7f0f9efff700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f9815a320 2023-06-22T08:25:38.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.262+0000 7f0f9efff700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f980c3a40 2023-06-22T08:25:38.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.262+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 msgr2=0x7f0f98160ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:38.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.262+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 msgr2=0x7f0f980c3e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:38.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.262+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f9806d050 legacy=0x7f0f98159c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:38.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.262+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 0x7f0f98160ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:38.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.262+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 0x7f0f980c3e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:38.304 INFO:tasks.ceph.mgr.y.smithi023.stderr:daemon-helper: command crashed with signal 15 2023-06-22T08:25:38.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.463+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 msgr2=0x7f0f98160ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:38.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.463+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 msgr2=0x7f0f980c3e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:38.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.463+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 0x7f0f98160ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:38.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.463+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 0x7f0f980c3e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:38.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.463+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f9806d050 legacy=0x7f0f98159c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:38.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.864+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 msgr2=0x7f0f980c3e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.864+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 msgr2=0x7f0f98160ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.864+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f9806d050 legacy=0x7f0f98159c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.864+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 0x7f0f980c3e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:38.864+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 0x7f0f98160ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:39.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:39.665+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 msgr2=0x7f0f98160ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:39.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:39.665+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 msgr2=0x7f0f980c3e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:39.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:39.665+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f9806d050 legacy=0x7f0f98159c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:39.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:39.665+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 0x7f0f98160ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:39.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:39.665+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 0x7f0f980c3e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 msgr2=0x7f0f980c3e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f980c3a40 0x7f0f980c3e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 msgr2=0x7f0f98160ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f9815a320 0x7f0f98160ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f9806d050 legacy=0x7f0f98159c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 0x7f0f7c005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c006210 2023-06-22T08:25:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c003750 2023-06-22T08:25:41.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:41.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c006210 legacy=0x7f0f7c0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:41.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.262+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:41.465 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.463+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c006210 legacy=0x7f0f7c0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:41.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.463+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:41.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.463+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:41.866 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.864+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:41.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.864+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c006210 legacy=0x7f0f7c0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:41.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:41.864+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:42.667 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:42.665+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:42.665+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c006210 legacy=0x7f0f7c0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:42.665+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:44.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:44.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c003750 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:44.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c006210 legacy=0x7f0f7c0086c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:44.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:25:44.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:44.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 0x7f0f7c0086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:44.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f9806d050 2023-06-22T08:25:44.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c00aae0 2023-06-22T08:25:44.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c00a670 2023-06-22T08:25:44.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 msgr2=0x7f0f7c0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:44.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 0x7f0f7c0086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:44.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f9806d050 legacy=0x7f0f7c009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:44.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f97fff700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:44.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.263+0000 7f0f97fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:44.466 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.464+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 msgr2=0x7f0f7c0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:44.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.464+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 0x7f0f7c0086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:44.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.464+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f9806d050 legacy=0x7f0f7c009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:44.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.464+0000 7f0f97fff700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:44.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.464+0000 7f0f97fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:44.867 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.865+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 msgr2=0x7f0f7c0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:44.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.865+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 0x7f0f7c0086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:44.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.865+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f9806d050 legacy=0x7f0f7c009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:44.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.865+0000 7f0f97fff700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:44.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:44.865+0000 7f0f97fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:45.668 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:45.666+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 msgr2=0x7f0f7c0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:45.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:45.666+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 0x7f0f7c0086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:45.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:45.666+0000 7f0f97fff700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:45.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:45.666+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f9806d050 legacy=0x7f0f7c009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:45.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:45.666+0000 7f0f97fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:47.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.263+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:47.266 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.263+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.263+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 msgr2=0x7f0f7c0086c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.263+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c00aae0 0x7f0f7c0086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.263+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f9806d050 legacy=0x7f0f7c009f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.263+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.263+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c009810 2023-06-22T08:25:47.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.263+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c008e70 2023-06-22T08:25:47.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.263+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c009810 legacy=0x7f0f7c012220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:47.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.264+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:47.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.264+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:47.278 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:25:47.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:25:47.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.465+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c009810 legacy=0x7f0f7c012220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:47.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.465+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:47.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.465+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:47.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.866+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c009810 legacy=0x7f0f7c012220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:47.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.866+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:47.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:47.866+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:48.667+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c009810 legacy=0x7f0f7c012220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:48.667+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:48.667+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:50.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:50.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:50.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c009810 legacy=0x7f0f7c012220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:50.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:25:50.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:50.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c008e70 2023-06-22T08:25:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c0086c0 2023-06-22T08:25:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c00a670 2023-06-22T08:25:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e70 legacy=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.264+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:50.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.465+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e70 legacy=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.465+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:50.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.465+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:50.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.465+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:50.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.465+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:50.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.866+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:50.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.866+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:50.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.866+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e70 legacy=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:50.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.866+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:50.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:50.866+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:51.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:51.667+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:51.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:51.667+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:51.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:51.667+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e70 legacy=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:51.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:51.667+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:51.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:51.667+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:53.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:53.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:53.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:53.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:53.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e70 legacy=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 0x7f0f7c013de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c0143b0 2023-06-22T08:25:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c013980 2023-06-22T08:25:53.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0143b0 legacy=0x7f0f7c012220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:53.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 msgr2=0x7f0f7c013de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:53.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.264+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 0x7f0f7c013de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.465+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 msgr2=0x7f0f7c013de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.465+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0143b0 legacy=0x7f0f7c012220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.465+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 0x7f0f7c013de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:53.868 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.866+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0143b0 legacy=0x7f0f7c012220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:53.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.866+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 msgr2=0x7f0f7c013de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:53.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:53.866+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 0x7f0f7c013de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:54.669 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:54.667+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0143b0 legacy=0x7f0f7c012220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:54.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:54.667+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 msgr2=0x7f0f7c013de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:54.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:54.667+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 0x7f0f7c013de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:56.267 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 msgr2=0x7f0f7c013de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:56.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c013980 0x7f0f7c013de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:56.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0143b0 legacy=0x7f0f7c012220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:56.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:25:56.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 0x7f0f7c016db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:56.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:56.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c008e70 2023-06-22T08:25:56.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c0149d0 2023-06-22T08:25:56.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c0086c0 2023-06-22T08:25:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 msgr2=0x7f0f7c016db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:56.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 0x7f0f7c016db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:56.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:56.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e70 legacy=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:56.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.265+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:56.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.466+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 msgr2=0x7f0f7c016db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:56.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.466+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 0x7f0f7c016db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:56.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.466+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:56.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.466+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e70 legacy=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:56.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.466+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:56.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.867+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 msgr2=0x7f0f7c016db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:56.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.867+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 0x7f0f7c016db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:56.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.867+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:56.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.867+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e70 legacy=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:56.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:56.867+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:25:57.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:57.668+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 msgr2=0x7f0f7c016db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:25:57.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:57.668+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 0x7f0f7c016db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:57.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:57.668+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:25:57.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:57.668+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e70 legacy=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:25:57.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:57.668+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:25:59.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:59.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0086c0 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:59.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 msgr2=0x7f0f7c016db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:59.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0149d0 0x7f0f7c016db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:25:59.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e70 legacy=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:25:59.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:25:59.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c0143b0 2023-06-22T08:25:59.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c00a670 2023-06-22T08:25:59.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0143b0 legacy=0x7f0f7c0101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:59.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:59.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.265+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:25:59.468 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.466+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0143b0 legacy=0x7f0f7c0101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:59.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.466+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:59.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.466+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:25:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.867+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0143b0 legacy=0x7f0f7c0101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:25:59.869 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.867+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:25:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:25:59.867+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:00.670 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:00.668+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0143b0 legacy=0x7f0f7c0101f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:00.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:00.668+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:00.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:00.668+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:02.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.265+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:02.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.265+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:02.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.265+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0143b0 legacy=0x7f0f7c0101f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:02.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.265+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:02.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.265+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:26:02.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.265+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:02.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.265+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:02.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.265+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c0143b0 2023-06-22T08:26:02.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.265+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c008e70 2023-06-22T08:26:02.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.265+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c00a460 con 0x7f0f7c00a670 2023-06-22T08:26:02.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.266+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c003bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:02.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.266+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:02.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.266+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 msgr2=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:02.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.266+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:02.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.266+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:02.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.266+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c003bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.467+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 msgr2=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.467+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.467+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c003bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.467+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.467+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c003bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.467+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.868+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 msgr2=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.868+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.868+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.868+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c003bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:02.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.868+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:02.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:02.868+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c003bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:03.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:03.669+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 msgr2=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:03.669+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:03.669+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c003bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:03.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:03.669+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:03.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:03.669+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c003bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:03.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:03.669+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:05.268 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.266+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c003bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:05.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.266+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c003bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:05.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.266+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:05.269 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.266+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e70 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:05.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.266+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 msgr2=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:05.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.266+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:05.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.266+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c00d840 2023-06-22T08:26:05.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.266+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c0086c0 2023-06-22T08:26:05.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.266+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00d840 legacy=0x7f0f7c0111c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:05.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.266+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0086c0 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:05.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.467+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00d840 legacy=0x7f0f7c0111c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:05.469 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.467+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0086c0 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:05.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.868+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00d840 legacy=0x7f0f7c0111c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:05.870 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:05.868+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0086c0 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:06.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:06.669+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00d840 legacy=0x7f0f7c0111c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:06.671 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:06.669+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0086c0 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0086c0 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00d840 legacy=0x7f0f7c0111c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:26:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:08.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:08.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c0159a0 2023-06-22T08:26:08.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c0143b0 2023-06-22T08:26:08.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c00a670 2023-06-22T08:26:08.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0159a0 legacy=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:08.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 msgr2=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:08.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:08.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:08.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.267+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:08.470 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.468+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 msgr2=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:08.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.468+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:08.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.468+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0159a0 legacy=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:08.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.468+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:08.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.468+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:08.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.869+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 msgr2=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:08.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.869+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:08.871 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.869+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0159a0 legacy=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:08.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.869+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:08.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:08.869+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:09.670+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 msgr2=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:09.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:09.670+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:09.670+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0159a0 legacy=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:09.670+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:09.670+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:11.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:11.270 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:11.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 msgr2=0x7f0f7c009f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:11.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c0143b0 0x7f0f7c009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:11.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c0159a0 legacy=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:11.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:11.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c008620 2023-06-22T08:26:11.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c008e50 2023-06-22T08:26:11.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008620 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:11.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 msgr2=0x7f0f7c005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:11.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.268+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:11.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.469+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008620 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:11.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.469+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 msgr2=0x7f0f7c005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:11.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.469+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.870+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008620 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.870+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 msgr2=0x7f0f7c005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:11.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:11.870+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:12.671+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008620 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:12.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:12.671+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 msgr2=0x7f0f7c005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:12.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:12.671+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:14.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 msgr2=0x7f0f7c005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:14.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:14.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008620 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:14.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:26:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c008e50 2023-06-22T08:26:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c0111c0 2023-06-22T08:26:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c00a670 2023-06-22T08:26:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:14.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:14.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0111c0 legacy=0x7f0f7c0150a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:14.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.268+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:14.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.469+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:14.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.469+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:14.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.469+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0111c0 legacy=0x7f0f7c0150a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:14.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.469+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:14.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.870+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:14.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.870+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:14.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.870+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0111c0 legacy=0x7f0f7c0150a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:14.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:14.870+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:15.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:15.671+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:15.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:15.671+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:15.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:15.671+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0111c0 legacy=0x7f0f7c0150a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:15.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:15.671+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:17.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0111c0 legacy=0x7f0f7c0150a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:17.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:17.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0167c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c009f20 2023-06-22T08:26:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c014380 2023-06-22T08:26:17.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0167c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0167c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 msgr2=0x7f0f7c005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:17.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.268+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:17.403 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:26:17.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:26:17.471 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.469+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0167c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.469+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0167c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.469+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 msgr2=0x7f0f7c005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.469+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:17.872 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.870+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0167c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:17.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.870+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0167c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:17.941 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.870+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 msgr2=0x7f0f7c005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:17.870+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:18.673 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:18.671+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0167c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:18.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:18.671+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0167c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:18.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:18.671+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 msgr2=0x7f0f7c005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:18.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:18.671+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:20.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 msgr2=0x7f0f7c005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:20.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:20.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0167c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:20.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0167c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:20.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:26:20.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:20.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:20.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c0111c0 2023-06-22T08:26:20.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c009f20 2023-06-22T08:26:20.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c00a670 2023-06-22T08:26:20.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:20.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:20.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0111c0 legacy=0x7f0f7c0150a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:20.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:20.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.269+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:20.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.470+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:20.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.470+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:20.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.470+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:20.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.470+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0111c0 legacy=0x7f0f7c0150a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:20.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.470+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:20.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.871+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:20.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.871+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:20.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.871+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0111c0 legacy=0x7f0f7c0150a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:20.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.871+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:20.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:20.871+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:21.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:21.672+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:21.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:21.672+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:21.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:21.672+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0111c0 legacy=0x7f0f7c0150a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:21.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:21.672+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:21.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:21.672+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:23.271 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c0092d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:23.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c0092d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:23.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:23.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:23.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c0111c0 legacy=0x7f0f7c0150a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:23.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c014990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:23.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c008e50 2023-06-22T08:26:23.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c014380 2023-06-22T08:26:23.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c0157b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:23.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 msgr2=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:23.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.269+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c014990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:23.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.470+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c0157b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:23.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.470+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 msgr2=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:23.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.470+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c014990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:23.873 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.871+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c0157b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:23.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.871+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 msgr2=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:23.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:23.871+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c014990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:24.674 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:24.672+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c0157b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:24.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:24.672+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 msgr2=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:24.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:24.672+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c014990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:26.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 msgr2=0x7f0f7c014990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:26.272 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c014380 0x7f0f7c014990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:26.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c0157b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:26.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:26:26.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:26.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:26.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c008e50 2023-06-22T08:26:26.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c009f20 2023-06-22T08:26:26.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c00a670 2023-06-22T08:26:26.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c0157b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:26.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:26.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:26.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f9d59c700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:26.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.270+0000 7f0f9d59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:26.472 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.471+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c0157b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:26.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.471+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:26.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.471+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:26.473 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.471+0000 7f0f9d59c700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:26.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.471+0000 7f0f9d59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:26.874 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.872+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:26.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.872+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c0157b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:26.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.872+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:26.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.872+0000 7f0f9d59c700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:26.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:26.872+0000 7f0f9d59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:27.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:27.673+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:27.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:27.673+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c0157b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:27.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:27.673+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:27.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:27.673+0000 7f0f9d59c700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:27.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:27.673+0000 7f0f9d59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:29.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 msgr2=0x7f0f7c005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:29.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c00a670 0x7f0f7c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:29.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:29.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:29.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c0157b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:29.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:29.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c014990 2023-06-22T08:26:29.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c0142d0 2023-06-22T08:26:29.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c014990 legacy=0x7f0f7c014d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:29.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:29.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.270+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:29.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.471+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.471+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c014990 legacy=0x7f0f7c014d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.471+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:29.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.872+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c014990 legacy=0x7f0f7c014d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:29.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.872+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:29.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:29.872+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:30.675 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:30.673+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c014990 legacy=0x7f0f7c014d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:30.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:30.673+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:30.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:30.673+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:32.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:32.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:32.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c014990 legacy=0x7f0f7c014d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:32.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:32.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:26:32.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c018ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:32.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c008e50 2023-06-22T08:26:32.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c009f20 2023-06-22T08:26:32.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c0142d0 2023-06-22T08:26:32.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:32.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:32.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c019420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:32.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 msgr2=0x7f0f7c018ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:32.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.271+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c018ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:32.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.472+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c019420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:32.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.472+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:32.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.472+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:32.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.472+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 msgr2=0x7f0f7c018ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:32.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.472+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c018ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:32.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.873+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c019420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:32.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.873+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:32.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.873+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:32.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.873+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 msgr2=0x7f0f7c018ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:32.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:32.873+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c018ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:33.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:33.674+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c019420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:33.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:33.674+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:33.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:33.674+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:33.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:33.674+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 msgr2=0x7f0f7c018ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:33.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:33.674+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c018ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:35.273 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 msgr2=0x7f0f7c018ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:35.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c0142d0 0x7f0f7c018ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:35.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:35.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:35.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c019420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:35.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:35.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c00a670 2023-06-22T08:26:35.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c0111c0 2023-06-22T08:26:35.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:35.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:35.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.271+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.472+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.472+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:35.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.472+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.873+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.873+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:35.873+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:36.676 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:36.674+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:36.674+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:36.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:36.674+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:38.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:38.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:38.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:38.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:38.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:26:38.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:38.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c014990 2023-06-22T08:26:38.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c009f20 2023-06-22T08:26:38.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c00a670 2023-06-22T08:26:38.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:38.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c014990 legacy=0x7f0f7c003b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:38.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:38.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:38.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.271+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:38.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.472+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c014990 legacy=0x7f0f7c003b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:38.474 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.472+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.472+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.472+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.472+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:38.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.873+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c014990 legacy=0x7f0f7c003b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:38.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.873+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:38.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.873+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:38.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.873+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:38.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:38.873+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:39.674+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:39.674+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c014990 legacy=0x7f0f7c003b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:39.674+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:39.674+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:39.674+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:41.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:41.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:41.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:41.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:41.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c014990 legacy=0x7f0f7c003b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:41.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:41.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c005270 2023-06-22T08:26:41.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c008e50 2023-06-22T08:26:41.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005270 legacy=0x7f0f7c005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:41.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:41.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.272+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:41.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.473+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:41.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.473+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005270 legacy=0x7f0f7c005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:41.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.473+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:41.875 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.874+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005270 legacy=0x7f0f7c005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:41.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.874+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:41.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:41.874+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:42.675+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005270 legacy=0x7f0f7c005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:42.675+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:42.675+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:44.274 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:44.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:44.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005270 legacy=0x7f0f7c005670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:44.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:26:44.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:44.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:44.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c005270 2023-06-22T08:26:44.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c009f20 2023-06-22T08:26:44.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c00a670 2023-06-22T08:26:44.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:44.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:44.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005270 legacy=0x7f0f7c005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:44.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:44.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.272+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:44.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.473+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:44.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.473+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:44.475 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.473+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005270 legacy=0x7f0f7c005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:44.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.473+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:44.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.473+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:44.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.874+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:44.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.874+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005270 legacy=0x7f0f7c005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:44.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.874+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:44.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.874+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:44.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:44.874+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:45.677 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:45.675+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:45.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:45.675+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:45.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:45.675+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005270 legacy=0x7f0f7c005670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:45.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:45.675+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:45.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:45.675+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:47.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:47.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:47.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:47.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:47.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005270 legacy=0x7f0f7c005670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:47.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:47.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c008e50 2023-06-22T08:26:47.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c0111c0 2023-06-22T08:26:47.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c003b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:47.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:47.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.272+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:47.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.473+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c003b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:47.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.473+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:47.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.473+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:47.526 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:26:47.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:26:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.874+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c003b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.874+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:47.874+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:48.675+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c003b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:48.675+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:48.675+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:50.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:50.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:50.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c008e50 legacy=0x7f0f7c003b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:50.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:50.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:26:50.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:50.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:50.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c014990 2023-06-22T08:26:50.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c008e50 2023-06-22T08:26:50.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c00a460 con 0x7f0f7c0111c0 2023-06-22T08:26:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.271+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.272+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.272+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.272+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:50.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.272+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:50.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.272+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:50.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.473+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:50.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.473+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:50.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.473+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:50.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.473+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:50.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.473+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:50.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.473+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:50.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.874+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:50.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.874+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:50.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.874+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:50.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.874+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:50.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.874+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:50.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:50.874+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:51.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:51.676+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:51.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:51.676+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:51.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:51.676+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:51.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:51.676+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:51.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:51.676+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:51.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:51.676+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:53.275 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.272+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.272+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.272+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.272+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.272+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.272+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.272+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c00a670 2023-06-22T08:26:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.272+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c009f20 2023-06-22T08:26:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.272+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c0108c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:53.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.272+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c0101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:53.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.473+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c0108c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:53.476 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.473+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c0101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.874+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c0108c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:53.877 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:53.874+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c0101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:54.678 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:54.675+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c0108c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:54.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:54.675+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c0101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c0101b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c0108c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:56.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:56.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:26:56.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:56.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c00a670 2023-06-22T08:26:56.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c014990 2023-06-22T08:26:56.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c0111c0 2023-06-22T08:26:56.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:56.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:56.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:56.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c0108c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:56.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.273+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.474+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c0108c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.474+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.474+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.474+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:56.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.474+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:56.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.875+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c0108c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:56.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.875+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:56.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.875+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.875+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:56.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:56.875+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:26:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:57.676+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c0108c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:26:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:57.676+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:26:57.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:57.676+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:57.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:57.676+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:26:57.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:57.676+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:26:59.276 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.273+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:59.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.273+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c0111c0 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:59.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.273+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:59.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.273+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:26:59.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.273+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c0108c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:26:59.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.273+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:26:59.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.273+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c009f20 2023-06-22T08:26:59.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.273+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c008e50 2023-06-22T08:26:59.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.274+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c0101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:59.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.274+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:59.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.274+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:26:59.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.475+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:59.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.475+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c0101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:59.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.475+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:26:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.876+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c0101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:26:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.876+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:26:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:26:59.876+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:00.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:00.678+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c0101b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:00.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:00.678+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:00.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:00.678+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:02.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:02.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:02.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c0101b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:02.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:02.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:27:02.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:02.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:02.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c014990 2023-06-22T08:27:02.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c008e50 2023-06-22T08:27:02.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c00a460 con 0x7f0f7c009f20 2023-06-22T08:27:02.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:02.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:02.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:02.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f9d59c700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:02.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:02.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.274+0000 7f0f9d59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:02.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.475+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:02.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.475+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:02.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.475+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:02.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.475+0000 7f0f9d59c700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:02.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.475+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:02.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.475+0000 7f0f9d59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.876+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.876+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.876+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.876+0000 7f0f9d59c700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.876+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:02.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:02.876+0000 7f0f9d59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:03.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:03.677+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:03.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:03.677+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:03.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:03.677+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:03.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:03.677+0000 7f0f9d59c700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:03.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:03.677+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:03.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:03.677+0000 7f0f9d59c700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:05.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.274+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c0092b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:05.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.274+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c0092b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:05.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.274+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:05.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.274+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:05.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.274+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:05.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.274+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:05.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.274+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c00a670 2023-06-22T08:27:05.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.274+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9814d690 con 0x7f0f7c005b30 2023-06-22T08:27:05.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.274+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:05.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.274+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005b30 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:05.477 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.475+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:05.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.475+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005b30 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:05.878 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.876+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:05.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:05.876+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005b30 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:06.679 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:06.677+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:06.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:06.677+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005b30 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:08.277 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.274+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005b30 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:08.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.274+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:08.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:27:08.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:08.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:08.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c00a670 2023-06-22T08:27:08.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c014990 2023-06-22T08:27:08.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c008e50 2023-06-22T08:27:08.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:08.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:08.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:08.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:08.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.275+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:08.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.476+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:08.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.476+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:08.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.476+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:08.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.476+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:08.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.476+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:08.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.877+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:08.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.877+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:08.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.877+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:08.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.877+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:08.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:08.877+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:09.680 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:09.678+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:09.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:09.678+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:09.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:09.678+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:09.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:09.678+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:09.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:09.678+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c008e50 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:11.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:11.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:11.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:11.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c009f20 2023-06-22T08:27:11.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c005b30 2023-06-22T08:27:11.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f97fff700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:11.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:11.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.275+0000 7f0f97fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:11.478 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.476+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:11.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.476+0000 7f0f97fff700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:11.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.476+0000 7f0f97fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:11.879 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.877+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:11.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.877+0000 7f0f97fff700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:11.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:11.877+0000 7f0f97fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:12.678+0000 7f0f97fff700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:12.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:12.678+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:12.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:12.678+0000 7f0f97fff700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:14.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:14.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:14.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c009f20 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:14.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:14.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:27:14.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:14.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c00a670 2023-06-22T08:27:14.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c014990 2023-06-22T08:27:14.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c005b30 2023-06-22T08:27:14.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:14.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.275+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:14.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.276+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:14.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.276+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:14.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.276+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:14.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.477+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:14.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.477+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:14.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.477+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:14.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.477+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:14.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.477+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:14.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.878+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:14.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.878+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:14.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.878+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:14.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.878+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:14.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:14.878+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:15.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:15.679+0000 7f0f97fff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:15.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:15.679+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:15.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:15.679+0000 7f0f97fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:15.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:15.679+0000 7f0f9d59c700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:15.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:15.679+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:17.278 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:17.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:17.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:17.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c014990 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:17.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:17.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:17.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c008e00 2023-06-22T08:27:17.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c009f20 2023-06-22T08:27:17.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:17.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e00 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:17.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.276+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.477+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:17.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.477+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e00 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:17.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.477+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:17.656 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:27:17.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:27:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.878+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.878+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e00 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:17.878+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:18.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:18.679+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e00 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:18.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:18.679+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:18.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:18.679+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:20.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:20.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:20.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c008e00 legacy=0x7f0f7c00f8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:20.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:27:20.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:20.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:20.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c00a670 2023-06-22T08:27:20.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c005b30 2023-06-22T08:27:20.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c009f20 2023-06-22T08:27:20.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:20.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:20.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:20.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:20.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.276+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:20.479 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.477+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:20.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.477+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:20.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.477+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:20.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.477+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:20.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.477+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:20.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.878+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:20.880 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.878+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:20.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.878+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:20.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.878+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:20.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:20.878+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:21.681 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:21.679+0000 7f0f9d59c700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:21.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:21.679+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:21.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:21.679+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:21.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:21.679+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:21.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:21.679+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:23.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:23.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c005b30 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:23.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00a670 legacy=0x7f0f7c00e1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:23.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:23.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c00f8c0 2023-06-22T08:27:23.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c008e00 2023-06-22T08:27:23.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:23.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:23.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.277+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00f8c0 legacy=0x7f0f7c00fcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:23.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.478+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:23.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.478+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:23.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.478+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00f8c0 legacy=0x7f0f7c00fcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:23.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.879+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:23.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.879+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:23.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:23.879+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00f8c0 legacy=0x7f0f7c00fcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:24.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:24.680+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:24.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:24.680+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:24.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:24.680+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00f8c0 legacy=0x7f0f7c00fcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:26.279 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:26.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:26.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00f8c0 legacy=0x7f0f7c00fcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:26.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:26.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:27:26.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:26.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c00f8c0 2023-06-22T08:27:26.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c008e00 2023-06-22T08:27:26.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c009f20 2023-06-22T08:27:26.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:26.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00f8c0 legacy=0x7f0f7c00fcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:26.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:26.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:26.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.277+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:26.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.478+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:26.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.478+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:26.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.478+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:26.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.478+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00f8c0 legacy=0x7f0f7c00fcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:26.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.478+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:26.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.879+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:26.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.879+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:26.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.879+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:26.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.879+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00f8c0 legacy=0x7f0f7c00fcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:26.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:26.879+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:27.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:27.680+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:27.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:27.680+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:27.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:27.680+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:27.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:27.680+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00f8c0 legacy=0x7f0f7c00fcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:27.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:27.680+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:29.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.277+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:29.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.277+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:29.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.277+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:29.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.277+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:29.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.277+0000 7f0f967fc700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c00f8c0 legacy=0x7f0f7c00fcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:29.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.277+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:29.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.277+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c00e1a0 2023-06-22T08:27:29.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.277+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c00a670 2023-06-22T08:27:29.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.277+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00e1a0 legacy=0x7f0f7c00e570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:29.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.277+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:29.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.278+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:29.480 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.478+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00e1a0 legacy=0x7f0f7c00e570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.479+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.479+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:29.881 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.879+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00e1a0 legacy=0x7f0f7c00e570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:29.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.880+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:29.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:29.880+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:30.682 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:30.680+0000 7f0f97fff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00e1a0 legacy=0x7f0f7c00e570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:30.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:30.681+0000 7f0f9cd9b700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:30.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:30.681+0000 7f0f9cd9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:32.280 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f967fc700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00e1a0 legacy=0x7f0f7c00e570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f9d59c700 1 -- reap_dead start 2023-06-22T08:27:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T08:27:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f967fc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98157570 con 0x7f0f7c008e00 2023-06-22T08:27:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f967fc700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980c4350 con 0x7f0f7c00a670 2023-06-22T08:27:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f967fc700 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c00a460 con 0x7f0f7c009f20 2023-06-22T08:27:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.278+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T08:27:32.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.479+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:32.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.479+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.479+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.479+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.479+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.479+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-22T08:27:32.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.880+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:32.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.880+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:32.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.880+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:32.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.880+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:32.883 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.880+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:32.884 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:32.880+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-22T08:27:33.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:33.681+0000 7f0f9d59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2023-06-22T08:27:33.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:33.681+0000 7f0f9d59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:33.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:33.681+0000 7f0f97fff700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-06-22T08:27:33.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:33.681+0000 7f0f9cd9b700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-06-22T08:27:33.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:33.681+0000 7f0f9cd9b700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:33.685 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:33.681+0000 7f0f97fff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-22T08:27:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.278+0000 7f0f967fc700 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 msgr2=0x7f0f7c00d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.278+0000 7f0f967fc700 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0f7c009f20 0x7f0f7c00d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.278+0000 7f0f967fc700 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 msgr2=0x7f0f7c009290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:35.281 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.278+0000 7f0f967fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0f7c00a670 0x7f0f7c009290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.278+0000 7f0f967fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 msgr2=0x7f0f7c014260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T08:27:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.278+0000 7f0f967fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f7c008e00 0x7f0f7c014260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T08:27:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.278+0000 7f0f967fc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c00dc40 con 0x7f0f7c00fcc0 2023-06-22T08:27:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.278+0000 7f0f967fc700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980c6a60 con 0x7f0f7c005b30 2023-06-22T08:27:35.282 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.278+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00fcc0 legacy=0x7f0f7c0100d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:35.283 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.278+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005b30 legacy=0x7f0f7c00f1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:35.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.479+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00fcc0 legacy=0x7f0f7c0100d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:35.481 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.479+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005b30 legacy=0x7f0f7c00f1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.880+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00fcc0 legacy=0x7f0f7c0100d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:35.880+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005b30 legacy=0x7f0f7c00f1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:36.683 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:36.681+0000 7f0f9cd9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0f7c00fcc0 legacy=0x7f0f7c0100d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2023-06-22T08:27:36.684 INFO:teuthology.orchestra.run.smithi023.stderr:2023-06-22T08:27:36.681+0000 7f0f97fff700 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f0f7c005b30 legacy=0x7f0f7c00f1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-06-22T08:27:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-06-22T08:27:38.133 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-06-22T08:27:38.143 INFO:teuthology.misc:Shutting down mds daemons... 2023-06-22T08:27:38.143 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-22T08:27:38.144 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.146 INFO:tasks.ceph.mds.a:Stopped 2023-06-22T08:27:38.147 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-22T08:27:38.147 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.148 INFO:tasks.ceph.mds.c:Stopped 2023-06-22T08:27:38.148 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-06-22T08:27:38.149 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.149 INFO:tasks.ceph.mds.e:Stopped 2023-06-22T08:27:38.150 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-22T08:27:38.150 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T08:27:38.151 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b' 2023-06-22T08:27:38.161 INFO:tasks.ceph.mds.b:Stopped 2023-06-22T08:27:38.161 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-22T08:27:38.162 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.162 INFO:tasks.ceph.mds.d:Stopped 2023-06-22T08:27:38.163 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-06-22T08:27:38.163 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.164 INFO:tasks.ceph.mds.f:Stopped 2023-06-22T08:27:38.164 INFO:teuthology.misc:Shutting down osd daemons... 2023-06-22T08:27:38.165 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-06-22T08:27:38.165 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.166 INFO:tasks.ceph.osd.0:Stopped 2023-06-22T08:27:38.166 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-06-22T08:27:38.167 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.167 INFO:tasks.ceph.osd.1:Stopped 2023-06-22T08:27:38.168 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-06-22T08:27:38.168 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.169 INFO:tasks.ceph.osd.2:Stopped 2023-06-22T08:27:38.169 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-06-22T08:27:38.170 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.170 INFO:tasks.ceph.osd.3:Stopped 2023-06-22T08:27:38.170 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-06-22T08:27:38.171 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.171 INFO:tasks.ceph.osd.4:Stopped 2023-06-22T08:27:38.172 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-06-22T08:27:38.172 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.173 INFO:tasks.ceph.osd.5:Stopped 2023-06-22T08:27:38.173 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-06-22T08:27:38.174 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.174 INFO:tasks.ceph.osd.6:Stopped 2023-06-22T08:27:38.175 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-06-22T08:27:38.175 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.175 INFO:tasks.ceph.osd.7:Stopped 2023-06-22T08:27:38.176 INFO:teuthology.misc:Shutting down mgr daemons... 2023-06-22T08:27:38.176 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-06-22T08:27:38.177 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T08:27:38.178 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi023 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2023-06-22T08:27:38.189 INFO:tasks.ceph.mgr.y:Stopped 2023-06-22T08:27:38.189 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-06-22T08:27:38.190 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.190 INFO:tasks.ceph.mgr.x:Stopped 2023-06-22T08:27:38.191 INFO:teuthology.misc:Shutting down mon daemons... 2023-06-22T08:27:38.191 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-06-22T08:27:38.191 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.192 INFO:tasks.ceph.mon.a:Stopped 2023-06-22T08:27:38.192 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-06-22T08:27:38.193 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.194 INFO:tasks.ceph.mon.b:Stopped 2023-06-22T08:27:38.195 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-06-22T08:27:38.195 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T08:27:38.196 INFO:tasks.ceph.mon.c:Stopped 2023-06-22T08:27:38.196 INFO:tasks.ceph:Checking cluster log for badness... 2023-06-22T08:27:38.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | head -n 1 2023-06-22T08:27:38.308 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T08:27:38.309 DEBUG:teuthology.orchestra.run.smithi023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-06-22T08:27:38.566 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T08:27:38.567 DEBUG:teuthology.orchestra.run.smithi023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-06-22T08:27:38.867 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T08:27:38.868 DEBUG:teuthology.orchestra.run.smithi023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-06-22T08:27:39.133 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T08:27:39.134 DEBUG:teuthology.orchestra.run.smithi023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-06-22T08:27:39.392 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T08:27:39.393 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-06-22T08:27:39.561 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T08:27:39.562 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-06-22T08:27:39.653 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T08:27:39.653 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-06-22T08:27:39.743 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T08:27:39.744 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-06-22T08:27:39.858 INFO:tasks.ceph:Archiving mon data... 2023-06-22T08:27:39.861 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/data/mon.a.tgz 2023-06-22T08:27:39.861 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2023-06-22T08:27:39.920 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.JVbNVTlgcp 2023-06-22T08:27:39.921 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.JVbNVTlgcp 2023-06-22T08:27:40.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.JVbNVTlgcp 2023-06-22T08:27:40.213 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.JVbNVTlgcp is 447KB 2023-06-22T08:27:40.309 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.JVbNVTlgcp 2023-06-22T08:27:40.351 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/data/mon.b.tgz 2023-06-22T08:27:40.351 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2023-06-22T08:27:40.387 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.a2guNneE0F 2023-06-22T08:27:40.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.a2guNneE0F 2023-06-22T08:27:40.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.a2guNneE0F 2023-06-22T08:27:40.638 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.a2guNneE0F is 449KB 2023-06-22T08:27:40.706 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.a2guNneE0F 2023-06-22T08:27:40.723 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/data/mon.c.tgz 2023-06-22T08:27:40.723 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2023-06-22T08:27:40.778 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.GPByHfLXLd 2023-06-22T08:27:40.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.GPByHfLXLd 2023-06-22T08:27:40.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.GPByHfLXLd 2023-06-22T08:27:41.017 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.GPByHfLXLd is 449KB 2023-06-22T08:27:41.092 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.GPByHfLXLd 2023-06-22T08:27:41.109 INFO:tasks.ceph:Cleaning ceph cluster... 2023-06-22T08:27:41.110 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-22T08:27:41.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-22T08:27:41.181 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-06-22T08:27:41.182 DEBUG:teuthology.orchestra.run.smithi023:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-22T08:27:41.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-22T08:27:41.247 INFO:teuthology.orchestra.run.smithi067.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-22T08:27:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-22T08:27:41.269 INFO:tasks.ceph:Archiving crash dumps... 2023-06-22T08:27:41.271 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/remote/smithi023/crash 2023-06-22T08:27:41.272 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-22T08:27:41.339 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/remote/smithi067/crash 2023-06-22T08:27:41.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-22T08:27:41.413 INFO:tasks.ceph:Shutting down logrotate 2023-06-22T08:27:41.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:27:41.418 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T08:27:41.535 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-22T08:27:41.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-22T08:27:41.629 INFO:tasks.ceph:Compressing logs... 2023-06-22T08:27:41.630 DEBUG:teuthology.orchestra.run.smithi023:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-06-22T08:27:41.635 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-06-22T08:27:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-06-22T08:27:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130620.log 2023-06-22T08:27:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-06-22T08:27:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.130620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130575.log 2023-06-22T08:27:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-06-22T08:27:41.708 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.130575.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-06-22T08:27:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130710.log 2023-06-22T08:27:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-06-22T08:27:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.130710.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-06-22T08:27:41.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130665.log 2023-06-22T08:27:41.710 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145219.log 2023-06-22T08:27:41.710 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129644.log 2023-06-22T08:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.4.log 2023-06-22T08:27:41.712 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.130665.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-06-22T08:27:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-06-22T08:27:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130809.log 2023-06-22T08:27:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130528.log 2023-06-22T08:27:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.130809.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-06-22T08:27:41.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.130528.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-06-22T08:27:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130763.log 2023-06-22T08:27:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-06-22T08:27:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.130763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130483.log 2023-06-22T08:27:41.715 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152554.log 2023-06-22T08:27:41.715 INFO:teuthology.orchestra.run.smithi023.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.145219.log.gz 2023-06-22T08:27:41.716 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.129644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154296.log 2023-06-22T08:27:41.716 INFO:teuthology.orchestra.run.smithi023.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.129644.log.gz 2023-06-22T08:27:41.716 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152554.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.152554.log.gz 2023-06-22T08:27:41.716 INFO:teuthology.orchestra.run.smithi023.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.135816.log 2023-06-22T08:27:41.716 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138316.log 2023-06-22T08:27:41.717 INFO:teuthology.orchestra.run.smithi023.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.154296.log.gz 2023-06-22T08:27:41.717 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144822.log 2023-06-22T08:27:41.718 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150061.log 2023-06-22T08:27:41.718 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.144822.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.144822.log.gz 2023-06-22T08:27:41.718 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160043.log 2023-06-22T08:27:41.718 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150061.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.150061.log.gz-5 2023-06-22T08:27:41.719 INFO:teuthology.orchestra.run.smithi023.stderr:--verbose -- /var/log/ceph/ceph-client.admin.147846.log 2023-06-22T08:27:41.719 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160043.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.160043.log.gz 2023-06-22T08:27:41.720 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153624.log 2023-06-22T08:27:41.720 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147846.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.147846.log.gz 2023-06-22T08:27:41.720 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148991.log 2023-06-22T08:27:41.721 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153624.log: gzip 88.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.153624.log.gz-- 2023-06-22T08:27:41.721 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148337.log 2023-06-22T08:27:41.722 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148991.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.148991.log.gz 2023-06-22T08:27:41.722 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139741.log 2023-06-22T08:27:41.723 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148337.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.138316.log.gz 2023-06-22T08:27:41.723 INFO:teuthology.orchestra.run.smithi023.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.135816.log.gz 2023-06-22T08:27:41.723 INFO:teuthology.orchestra.run.smithi023.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.148337.log.gz 2023-06-22T08:27:41.723 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146485.log 2023-06-22T08:27:41.724 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139741.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.139741.log.gz 2023-06-22T08:27:41.724 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158905.log 2023-06-22T08:27:41.725 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146485.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146485.log.gz 2023-06-22T08:27:41.725 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145541.log 2023-06-22T08:27:41.726 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158905.log: gzip -5 --verbose 88.5%-- /var/log/ceph/ceph-client.admin.151418.log -- replaced with /var/log/ceph/ceph-client.admin.158905.log.gz 2023-06-22T08:27:41.726 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-06-22T08:27:41.727 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145541.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.145541.log.gz 2023-06-22T08:27:41.727 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161953.log 2023-06-22T08:27:41.728 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158562.log 2023-06-22T08:27:41.728 INFO:teuthology.orchestra.run.smithi023.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151418.log.gz 2023-06-22T08:27:41.728 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161953.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.161953.log.gz 2023-06-22T08:27:41.729 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145173.log 2023-06-22T08:27:41.729 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mds.b.log: /var/log/ceph/ceph-client.admin.130483.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.130483.log.gz 2023-06-22T08:27:41.730 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158562.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.158562.log.gz 2023-06-22T08:27:41.730 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136376.log 2023-06-22T08:27:41.731 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145173.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.145173.log.gzgzip -5 2023-06-22T08:27:41.731 INFO:teuthology.orchestra.run.smithi023.stderr:--verbose -- /var/log/ceph/ceph-client.admin.139264.log 2023-06-22T08:27:41.731 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158813.log 2023-06-22T08:27:41.732 INFO:teuthology.orchestra.run.smithi023.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.136376.log.gz 2023-06-22T08:27:41.732 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139264.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.139264.log.gz 2023-06-22T08:27:41.733 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150835.log 2023-06-22T08:27:41.733 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158813.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158813.log.gz 2023-06-22T08:27:41.734 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147015.log 2023-06-22T08:27:41.734 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150835.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150835.log.gz 2023-06-22T08:27:41.735 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136144.log 2023-06-22T08:27:41.735 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147015.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.147015.log.gz 2023-06-22T08:27:41.735 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145449.log 2023-06-22T08:27:41.736 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136144.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.136144.log.gz 2023-06-22T08:27:41.736 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153317.log 2023-06-22T08:27:41.737 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145449.log: gzip -5 --verbose 89.3%-- /var/log/ceph/ceph-client.admin.138890.log -- replaced with /var/log/ceph/ceph-client.admin.145449.log.gz 2023-06-22T08:27:41.737 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-06-22T08:27:41.738 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153317.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.153317.log.gz 2023-06-22T08:27:41.738 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158459.log 2023-06-22T08:27:41.739 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138890.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.138890.log.gz 2023-06-22T08:27:41.739 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139473.log 2023-06-22T08:27:41.740 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158459.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158459.log.gz 2023-06-22T08:27:41.740 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-06-22T08:27:41.741 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153762.log 2023-06-22T08:27:41.741 INFO:teuthology.orchestra.run.smithi023.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.139473.log.gz 2023-06-22T08:27:41.742 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135322.log 2023-06-22T08:27:41.743 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153762.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.153762.log.gz 2023-06-22T08:27:41.743 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139665.log 2023-06-22T08:27:41.744 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135322.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.135322.log.gz 2023-06-22T08:27:41.745 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145788.log 2023-06-22T08:27:41.745 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139665.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.139665.log.gz 2023-06-22T08:27:41.745 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157288.log 2023-06-22T08:27:41.747 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145788.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.145788.log.gz 2023-06-22T08:27:41.747 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.141400.log 2023-06-22T08:27:41.747 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157288.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.157288.log.gz 2023-06-22T08:27:41.748 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151832.log 2023-06-22T08:27:41.749 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.0.141400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.141400.log.gz 2023-06-22T08:27:41.749 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139266.log 2023-06-22T08:27:41.750 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151832.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.151832.log.gz 2023-06-22T08:27:41.750 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161039.log 2023-06-22T08:27:41.751 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139266.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.139266.log.gz 2023-06-22T08:27:41.752 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149040.log 2023-06-22T08:27:41.752 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161039.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.161039.log.gz 2023-06-22T08:27:41.753 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140249.log 2023-06-22T08:27:41.753 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.130620.log.gz 2023-06-22T08:27:41.753 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.130665.log.gz 2023-06-22T08:27:41.754 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.130710.log.gz 2023-06-22T08:27:41.754 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.130763.log.gz 2023-06-22T08:27:41.755 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.130809.log.gz 2023-06-22T08:27:41.755 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149040.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149040.log.gz 2023-06-22T08:27:41.756 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157840.log 2023-06-22T08:27:41.756 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140249.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.140249.log.gz 2023-06-22T08:27:41.756 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142557.log 2023-06-22T08:27:41.757 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157840.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.157840.log.gz 2023-06-22T08:27:41.757 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148086.log 2023-06-22T08:27:41.758 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142557.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.142557.log.gz 2023-06-22T08:27:41.758 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-06-22T08:27:41.758 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148086.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.148086.log.gz 2023-06-22T08:27:41.759 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137564.log 2023-06-22T08:27:41.759 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153455.log 2023-06-22T08:27:41.761 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137564.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.137564.log.gz 2023-06-22T08:27:41.761 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138885.log 2023-06-22T08:27:41.762 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161566.log 2023-06-22T08:27:41.762 INFO:teuthology.orchestra.run.smithi023.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.153455.log.gz 2023-06-22T08:27:41.763 INFO:teuthology.orchestra.run.smithi023.stderr: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-06-22T08:27:41.763 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138885.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.138885.log.gz 2023-06-22T08:27:41.764 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141774.log 2023-06-22T08:27:41.764 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161566.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.161566.log.gz 2023-06-22T08:27:41.765 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138684.log 2023-06-22T08:27:41.765 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.141774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153271.log 2023-06-22T08:27:41.766 INFO:teuthology.orchestra.run.smithi023.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141774.log.gz 2023-06-22T08:27:41.766 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147800.log 2023-06-22T08:27:41.767 INFO:teuthology.orchestra.run.smithi023.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138684.log.gz 2023-06-22T08:27:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153271.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.153271.log.gz 2023-06-22T08:27:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140806.log 2023-06-22T08:27:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147800.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.147800.log.gz 2023-06-22T08:27:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138270.log 2023-06-22T08:27:41.769 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154342.log 2023-06-22T08:27:41.770 INFO:teuthology.orchestra.run.smithi023.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.140806.log.gz 2023-06-22T08:27:41.770 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145701.log 2023-06-22T08:27:41.771 INFO:teuthology.orchestra.run.smithi023.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.138270.log.gz 2023-06-22T08:27:41.772 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154342.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.154342.log.gz 2023-06-22T08:27:41.772 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137150.log 2023-06-22T08:27:41.772 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145701.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.145701.log.gz 2023-06-22T08:27:41.773 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140596.log 2023-06-22T08:27:41.774 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137951.log 2023-06-22T08:27:41.775 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156026.log 2023-06-22T08:27:41.775 INFO:teuthology.orchestra.run.smithi023.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.140596.log.gz 2023-06-22T08:27:41.776 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137951.log: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.137150.log.gz 2023-06-22T08:27:41.776 INFO:teuthology.orchestra.run.smithi023.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.137951.log.gz 2023-06-22T08:27:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159787.log 2023-06-22T08:27:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156026.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.156026.log.gz 2023-06-22T08:27:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159997.log 2023-06-22T08:27:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.159787.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.159787.log.gz 2023-06-22T08:27:41.779 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141958.log 2023-06-22T08:27:41.779 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.159997.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.159997.log.gz 2023-06-22T08:27:41.779 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146802.log 2023-06-22T08:27:41.780 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.141958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145495.log 2023-06-22T08:27:41.781 INFO:teuthology.orchestra.run.smithi023.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.141958.log.gz 2023-06-22T08:27:41.781 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146802.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.146802.log.gz 2023-06-22T08:27:41.782 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138546.log 2023-06-22T08:27:41.782 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145495.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.145495.log.gz 2023-06-22T08:27:41.783 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135583.log 2023-06-22T08:27:41.783 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135724.log 2023-06-22T08:27:41.784 INFO:teuthology.orchestra.run.smithi023.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.138546.log.gz 2023-06-22T08:27:41.784 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135583.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135583.log.gz 2023-06-22T08:27:41.785 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152600.log 2023-06-22T08:27:41.785 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138592.log 2023-06-22T08:27:41.786 INFO:teuthology.orchestra.run.smithi023.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.135724.log.gz 2023-06-22T08:27:41.786 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152600.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152600.log.gz 2023-06-22T08:27:41.787 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148891.log 2023-06-22T08:27:41.787 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138592.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.138592.log.gz 2023-06-22T08:27:41.788 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161269.log 2023-06-22T08:27:41.788 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148891.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.148891.log.gz 2023-06-22T08:27:41.789 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135862.log 2023-06-22T08:27:41.789 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138044.log 2023-06-22T08:27:41.790 INFO:teuthology.orchestra.run.smithi023.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.161269.log.gz 2023-06-22T08:27:41.790 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135862.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135862.log.gz 2023-06-22T08:27:41.791 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155085.log 2023-06-22T08:27:41.791 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138044.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.138044.log.gz 2023-06-22T08:27:41.792 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-06-22T08:27:41.792 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155085.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.155085.log.gz 2023-06-22T08:27:41.793 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149960.log 2023-06-22T08:27:41.793 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142188.log 2023-06-22T08:27:41.794 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149960.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.149960.log.gz 2023-06-22T08:27:41.795 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157647.log 2023-06-22T08:27:41.796 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142188.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.142188.log.gz 2023-06-22T08:27:41.796 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157380.log 2023-06-22T08:27:41.797 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157647.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.157647.log.gz 2023-06-22T08:27:41.797 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139259.log 2023-06-22T08:27:41.798 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157380.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.157380.log.gz 2023-06-22T08:27:41.798 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160142.log 2023-06-22T08:27:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139259.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.139259.log.gz 2023-06-22T08:27:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142372.log 2023-06-22T08:27:41.800 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160142.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.160142.log.gz 2023-06-22T08:27:41.800 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152019.log 2023-06-22T08:27:41.801 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142372.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.142372.log.gz 2023-06-22T08:27:41.801 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160442.log 2023-06-22T08:27:41.802 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152019.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152019.log.gz 2023-06-22T08:27:41.803 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153133.log 2023-06-22T08:27:41.803 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160442.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.160442.log.gz 2023-06-22T08:27:41.804 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153716.log 2023-06-22T08:27:41.805 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153133.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.153133.log.gz 2023-06-22T08:27:41.805 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135368.log 2023-06-22T08:27:41.806 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153716.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.153716.log.gz 2023-06-22T08:27:41.806 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145265.log 2023-06-22T08:27:41.807 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135368.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.135368.log.gz 2023-06-22T08:27:41.807 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141866.log 2023-06-22T08:27:41.808 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145265.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.145265.log.gz 2023-06-22T08:27:41.809 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161177.log 2023-06-22T08:27:41.809 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.141866.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.141866.log.gz 2023-06-22T08:27:41.810 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148245.log 2023-06-22T08:27:41.811 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161177.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.161177.log.gz 2023-06-22T08:27:41.811 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148540.log 2023-06-22T08:27:41.812 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148245.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148245.log.gz 2023-06-22T08:27:41.812 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-06-22T08:27:41.813 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148540.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.148540.log.gz 2023-06-22T08:27:41.814 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139263.log 2023-06-22T08:27:41.814 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158958.log 2023-06-22T08:27:41.815 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139263.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.139263.log.gz 2023-06-22T08:27:41.816 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154692.log 2023-06-22T08:27:41.817 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154455.log 2023-06-22T08:27:41.817 INFO:teuthology.orchestra.run.smithi023.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158958.log.gz 2023-06-22T08:27:41.818 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154692.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.154692.log.gz 2023-06-22T08:27:41.818 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145975.log 2023-06-22T08:27:41.819 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151510.log 2023-06-22T08:27:41.820 INFO:teuthology.orchestra.run.smithi023.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.154455.log.gz 2023-06-22T08:27:41.821 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145975.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145975.log.gz 2023-06-22T08:27:41.821 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139251.log 2023-06-22T08:27:41.822 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146068.log 2023-06-22T08:27:41.822 INFO:teuthology.orchestra.run.smithi023.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.151510.log.gz 2023-06-22T08:27:41.823 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139251.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.139251.log.gz 2023-06-22T08:27:41.824 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155821.log 2023-06-22T08:27:41.825 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146206.log 2023-06-22T08:27:41.825 INFO:teuthology.orchestra.run.smithi023.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.146068.log.gz 2023-06-22T08:27:41.826 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155821.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.155821.log.gz 2023-06-22T08:27:41.826 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147615.log 2023-06-22T08:27:41.827 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154409.log 2023-06-22T08:27:41.828 INFO:teuthology.orchestra.run.smithi023.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146206.log.gz 2023-06-22T08:27:41.828 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147615.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.147615.log.gz 2023-06-22T08:27:41.829 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152295.log 2023-06-22T08:27:41.830 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154409.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-06-22T08:27:41.830 INFO:teuthology.orchestra.run.smithi023.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.154409.log.gz 2023-06-22T08:27:41.831 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152295.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152295.log.gz 2023-06-22T08:27:41.832 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141912.log 2023-06-22T08:27:41.833 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150317.log 2023-06-22T08:27:41.834 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.141912.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.141912.log.gz 2023-06-22T08:27:41.834 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146577.log 2023-06-22T08:27:41.835 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156072.log 2023-06-22T08:27:41.836 INFO:teuthology.orchestra.run.smithi023.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.150317.log.gz 2023-06-22T08:27:41.836 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146577.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.146577.log.gz 2023-06-22T08:27:41.837 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149429.log 2023-06-22T08:27:41.838 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156072.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.156072.log.gz 2023-06-22T08:27:41.839 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149178.log 2023-06-22T08:27:41.840 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141682.log 2023-06-22T08:27:41.841 INFO:teuthology.orchestra.run.smithi023.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.149429.log.gz 2023-06-22T08:27:41.841 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149178.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149178.log.gz 2023-06-22T08:27:41.842 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154593.log 2023-06-22T08:27:41.843 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.141682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157507.log 2023-06-22T08:27:41.843 INFO:teuthology.orchestra.run.smithi023.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.141682.log.gz 2023-06-22T08:27:41.844 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154593.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.154593.log.gz 2023-06-22T08:27:41.845 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156778.log 2023-06-22T08:27:41.846 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157507.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.157507.log.gz 2023-06-22T08:27:41.846 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137518.log 2023-06-22T08:27:41.847 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156778.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.156778.log.gz 2023-06-22T08:27:41.848 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-06-22T08:27:41.849 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137518.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137518.log.gz 2023-06-22T08:27:41.849 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152508.log 2023-06-22T08:27:41.851 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151020.log 2023-06-22T08:27:41.852 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152508.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.152508.log.gz 2023-06-22T08:27:41.853 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138638.log 2023-06-22T08:27:41.853 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151020.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151020.log.gz 2023-06-22T08:27:41.855 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137702.log 2023-06-22T08:27:41.855 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138638.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.138638.log.gz 2023-06-22T08:27:41.856 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145357.log 2023-06-22T08:27:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137702.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.137702.log.gz 2023-06-22T08:27:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155177.log 2023-06-22T08:27:41.859 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145357.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145357.log.gz 2023-06-22T08:27:41.860 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142649.log 2023-06-22T08:27:41.860 INFO:teuthology.orchestra.run.smithi023.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.155177.log.gz 2023-06-22T08:27:41.861 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161658.log 2023-06-22T08:27:41.863 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142649.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.142649.log.gz 2023-06-22T08:27:41.863 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150008.log 2023-06-22T08:27:41.865 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161658.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.161658.log.gz 2023-06-22T08:27:41.865 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157175.log 2023-06-22T08:27:41.866 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150008.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150008.log.gz 2023-06-22T08:27:41.866 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149822.log 2023-06-22T08:27:41.868 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157175.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.157175.log.gz 2023-06-22T08:27:41.869 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158406.log 2023-06-22T08:27:41.869 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149822.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149822.log.gz 2023-06-22T08:27:41.870 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154972.log 2023-06-22T08:27:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158406.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.158406.log.gz 2023-06-22T08:27:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139611.log 2023-06-22T08:27:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154972.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.154972.log.gz 2023-06-22T08:27:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137426.log 2023-06-22T08:27:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139611.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.139611.log.gz 2023-06-22T08:27:41.875 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139249.log 2023-06-22T08:27:41.876 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137426.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.137426.log.gz 2023-06-22T08:27:41.877 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151204.log 2023-06-22T08:27:41.878 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139249.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.139249.log.gz 2023-06-22T08:27:41.878 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135956.log 2023-06-22T08:27:41.879 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151204.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.151204.log.gz 2023-06-22T08:27:41.879 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144776.log 2023-06-22T08:27:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135956.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.135956.log.gz 2023-06-22T08:27:41.882 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156663.log 2023-06-22T08:27:41.882 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.144776.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.144776.log.gz 2023-06-22T08:27:41.883 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150224.log 2023-06-22T08:27:41.887 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156663.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.156663.log.gz 2023-06-22T08:27:41.887 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161826.log 2023-06-22T08:27:41.888 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150224.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150224.log.gz 2023-06-22T08:27:41.888 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147569.log 2023-06-22T08:27:41.889 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161826.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.161826.log.gz 2023-06-22T08:27:41.890 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147569.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.147569.log.gz 2023-06-22T08:27:41.890 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149086.log 2023-06-22T08:27:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153900.log 2023-06-22T08:27:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149086.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.149086.log.gz 2023-06-22T08:27:41.894 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153900.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.153900.log.gz 2023-06-22T08:27:41.894 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153179.log 2023-06-22T08:27:41.896 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158767.log 2023-06-22T08:27:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153179.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.153179.log.gz 2023-06-22T08:27:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158767.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.158767.log.gz 2023-06-22T08:27:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160534.log 2023-06-22T08:27:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146361.log 2023-06-22T08:27:41.902 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160534.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.160534.log.gz 2023-06-22T08:27:41.903 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146361.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.146361.log.gz 2023-06-22T08:27:41.904 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146669.log 2023-06-22T08:27:41.905 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140295.log 2023-06-22T08:27:41.909 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146669.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.146669.log.gz 2023-06-22T08:27:41.909 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148845.log 2023-06-22T08:27:41.909 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140295.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.140295.log.gz 2023-06-22T08:27:41.910 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137859.log 2023-06-22T08:27:41.910 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148845.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.148845.log.gz 2023-06-22T08:27:41.911 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137859.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.137859.log.gz 2023-06-22T08:27:41.911 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152065.log 2023-06-22T08:27:41.913 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146114.log 2023-06-22T08:27:41.914 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152065.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.152065.log.gz 2023-06-22T08:27:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146114.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146114.log.gz 2023-06-22T08:27:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154784.log 2023-06-22T08:27:41.918 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-06-22T08:27:41.919 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154784.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.154784.log.gz 2023-06-22T08:27:41.924 INFO:teuthology.orchestra.run.smithi067.stderr: 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-06-22T08:27:41.929 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140155.log 2023-06-22T08:27:41.931 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159365.log 2023-06-22T08:27:41.933 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140155.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.140155.log.gz 2023-06-22T08:27:41.934 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.159365.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.159365.log.gz 2023-06-22T08:27:41.934 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140433.log 2023-06-22T08:27:41.937 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140433.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.140433.log.gz 2023-06-22T08:27:41.938 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154639.log 2023-06-22T08:27:41.941 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154639.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154639.log.gz 2023-06-22T08:27:41.941 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152646.log 2023-06-22T08:27:41.944 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152646.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152646.log.gz 2023-06-22T08:27:41.944 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153946.log 2023-06-22T08:27:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153946.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.153946.log.gz 2023-06-22T08:27:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153854.log 2023-06-22T08:27:41.950 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153854.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.153854.log.gz 2023-06-22T08:27:41.951 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152876.log 2023-06-22T08:27:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152876.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.152876.log.gz 2023-06-22T08:27:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153501.log 2023-06-22T08:27:41.957 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153501.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.153501.log.gz 2023-06-22T08:27:41.958 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157702.log 2023-06-22T08:27:41.960 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157702.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.157702.log.gz 2023-06-22T08:27:41.961 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161131.log 2023-06-22T08:27:41.964 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161131.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.161131.log.gz 2023-06-22T08:27:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151648.log 2023-06-22T08:27:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151648.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151648.log.gz 2023-06-22T08:27:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141636.log 2023-06-22T08:27:41.969 INFO:teuthology.orchestra.run.smithi067.stderr: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-06-22T08:27:41.970 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.141636.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141636.log.gz 2023-06-22T08:27:41.971 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142004.log 2023-06-22T08:27:41.974 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159054.log 2023-06-22T08:27:41.974 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142004.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.142004.log.gz 2023-06-22T08:27:41.975 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140666.log 2023-06-22T08:27:41.978 INFO:teuthology.orchestra.run.smithi067.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-06-22T08:27:41.978 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.159054.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.159054.log.gz 2023-06-22T08:27:41.978 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140666.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.140666.log.gz 2023-06-22T08:27:41.979 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148799.log 2023-06-22T08:27:41.981 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160813.log 2023-06-22T08:27:41.981 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148799.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.148799.log.gz 2023-06-22T08:27:41.983 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160813.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.160813.log.gz 2023-06-22T08:27:41.983 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149383.log 2023-06-22T08:27:41.986 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149383.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.149383.log.gz 2023-06-22T08:27:41.986 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.141414.log 2023-06-22T08:27:41.989 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.0.141414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.141414.log.gz 2023-06-22T08:27:41.989 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137997.log 2023-06-22T08:27:41.992 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137997.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.137997.log.gz 2023-06-22T08:27:41.993 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157932.log 2023-06-22T08:27:41.995 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157932.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.157932.log.gz 2023-06-22T08:27:41.996 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138872.log 2023-06-22T08:27:41.999 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138872.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.138872.log.gz 2023-06-22T08:27:41.999 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161382.log 2023-06-22T08:27:42.002 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161382.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.161382.log.gz 2023-06-22T08:27:42.002 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151258.log 2023-06-22T08:27:42.005 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151258.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.151258.log.gz 2023-06-22T08:27:42.006 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160626.log 2023-06-22T08:27:42.008 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160626.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160626.log.gz 2023-06-22T08:27:42.009 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152738.log 2023-06-22T08:27:42.012 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152738.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.152738.log.gz 2023-06-22T08:27:42.012 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152692.log 2023-06-22T08:27:42.015 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152692.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.152692.log.gz 2023-06-22T08:27:42.015 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147892.log 2023-06-22T08:27:42.018 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147892.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147892.log.gz 2023-06-22T08:27:42.019 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161758.log 2023-06-22T08:27:42.021 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139787.log 2023-06-22T08:27:42.022 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161758.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.161758.log.gz 2023-06-22T08:27:42.025 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139787.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.139787.log.gz 2023-06-22T08:27:42.025 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138132.log 2023-06-22T08:27:42.028 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138132.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138132.log.gz 2023-06-22T08:27:42.028 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152203.log 2023-06-22T08:27:42.031 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152203.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152203.log.gz 2023-06-22T08:27:42.032 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149776.log 2023-06-22T08:27:42.034 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149776.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.149776.log.gz 2023-06-22T08:27:42.035 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156125.log 2023-06-22T08:27:42.038 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156125.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.156125.log.gz 2023-06-22T08:27:42.038 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138408.log 2023-06-22T08:27:42.041 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138408.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.138408.log.gz 2023-06-22T08:27:42.042 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162137.log 2023-06-22T08:27:42.044 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.162137.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.162137.log.gz 2023-06-22T08:27:42.045 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142511.log 2023-06-22T08:27:42.048 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142511.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.142511.log.gz 2023-06-22T08:27:42.048 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142234.log 2023-06-22T08:27:42.051 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142234.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.142234.log.gz 2023-06-22T08:27:42.052 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139265.log 2023-06-22T08:27:42.054 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139265.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.139265.log.gz 2023-06-22T08:27:42.055 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150178.log 2023-06-22T08:27:42.058 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150178.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.150178.log.gz 2023-06-22T08:27:42.058 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151372.log 2023-06-22T08:27:42.061 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151372.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151372.log.gz 2023-06-22T08:27:42.061 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139247.log 2023-06-22T08:27:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139247.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139247.log.gz 2023-06-22T08:27:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158360.log 2023-06-22T08:27:42.067 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158360.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158360.log.gz 2023-06-22T08:27:42.068 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147707.log 2023-06-22T08:27:42.071 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147707.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.147707.log.gz 2023-06-22T08:27:42.071 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147661.log 2023-06-22T08:27:42.074 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147661.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.147661.log.gz 2023-06-22T08:27:42.074 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148753.log 2023-06-22T08:27:42.077 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148753.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.148753.log.gz 2023-06-22T08:27:42.078 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.96267.log 2023-06-22T08:27:42.080 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph.tmp-client.admin.96267.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.96267.log.gz 2023-06-22T08:27:42.081 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140201.log 2023-06-22T08:27:42.083 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140201.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.140201.log.gz 2023-06-22T08:27:42.084 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154738.log 2023-06-22T08:27:42.087 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154738.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.154738.log.gz 2023-06-22T08:27:42.087 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136236.log 2023-06-22T08:27:42.090 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-06-22T08:27:42.091 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136236.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.136236.log.gz 2023-06-22T08:27:42.093 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph.audit.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-06-22T08:27:42.094 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152989.log 2023-06-22T08:27:42.097 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152989.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152989.log.gz 2023-06-22T08:27:42.097 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161085.log 2023-06-22T08:27:42.100 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161085.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.161085.log.gz 2023-06-22T08:27:42.101 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136050.log 2023-06-22T08:27:42.103 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136050.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.136050.log.gz 2023-06-22T08:27:42.104 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160234.log 2023-06-22T08:27:42.107 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153409.log 2023-06-22T08:27:42.107 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160234.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.160234.log.gz 2023-06-22T08:27:42.108 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153409.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.153409.log.gz 2023-06-22T08:27:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150455.log 2023-06-22T08:27:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150455.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150455.log.gz 2023-06-22T08:27:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157748.log 2023-06-22T08:27:42.115 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157748.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.157748.log.gz 2023-06-22T08:27:42.116 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160672.log 2023-06-22T08:27:42.118 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160672.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160672.log.gz 2023-06-22T08:27:42.119 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149132.log 2023-06-22T08:27:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149132.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.149132.log.gz 2023-06-22T08:27:42.122 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140017.log 2023-06-22T08:27:42.125 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140017.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.140017.log.gz 2023-06-22T08:27:42.125 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148153.log 2023-06-22T08:27:42.132 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148199.log 2023-06-22T08:27:42.133 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148153.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148153.log.gz 2023-06-22T08:27:42.133 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148199.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.148199.log.gz 2023-06-22T08:27:42.133 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155775.log 2023-06-22T08:27:42.134 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155775.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.155775.log.gz 2023-06-22T08:27:42.134 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150974.log 2023-06-22T08:27:42.137 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155867.log 2023-06-22T08:27:42.137 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150974.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.150974.log.gz 2023-06-22T08:27:42.138 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155867.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.155867.log.gz 2023-06-22T08:27:42.139 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137656.log 2023-06-22T08:27:42.141 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137656.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.137656.log.gz 2023-06-22T08:27:42.142 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156718.log 2023-06-22T08:27:42.145 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156718.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.156718.log.gz 2023-06-22T08:27:42.145 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138891.log 2023-06-22T08:27:42.148 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138891.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.138891.log.gz 2023-06-22T08:27:42.149 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149475.log 2023-06-22T08:27:42.151 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149475.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.149475.log.gz 2023-06-22T08:27:42.152 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159184.log 2023-06-22T08:27:42.156 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136004.log 2023-06-22T08:27:42.156 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.159184.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159184.log.gz 2023-06-22T08:27:42.156 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136004.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.136004.log.gz 2023-06-22T08:27:42.157 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130064.log 2023-06-22T08:27:42.159 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.130064.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.130064.log.gz 2023-06-22T08:27:42.160 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158654.log 2023-06-22T08:27:42.163 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158654.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.158654.log.gz 2023-06-22T08:27:42.163 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146315.log 2023-06-22T08:27:42.166 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146315.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146315.log.gz 2023-06-22T08:27:42.167 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156217.log 2023-06-22T08:27:42.169 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156217.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.156217.log.gz 2023-06-22T08:27:42.170 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152157.log 2023-06-22T08:27:42.172 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152157.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152157.log.gz 2023-06-22T08:27:42.173 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162091.log 2023-06-22T08:27:42.176 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.162091.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.162091.log.gz 2023-06-22T08:27:42.176 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158130.log 2023-06-22T08:27:42.178 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.130575.log.gz 2023-06-22T08:27:42.179 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158130.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158130.log.gz 2023-06-22T08:27:42.180 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141820.log 2023-06-22T08:27:42.182 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.141820.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141820.log.gz 2023-06-22T08:27:42.183 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138876.log 2023-06-22T08:27:42.186 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138876.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.138876.log.gz 2023-06-22T08:27:42.186 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153087.log 2023-06-22T08:27:42.189 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153087.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.153087.log.gz 2023-06-22T08:27:42.190 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154062.log 2023-06-22T08:27:42.192 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154062.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.154062.log.gz 2023-06-22T08:27:42.193 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150107.log 2023-06-22T08:27:42.197 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139519.log 2023-06-22T08:27:42.197 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150107.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150107.log.gz 2023-06-22T08:27:42.198 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139519.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.139519.log.gz 2023-06-22T08:27:42.198 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149337.log 2023-06-22T08:27:42.201 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151066.log 2023-06-22T08:27:42.202 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149337.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.149337.log.gz 2023-06-22T08:27:42.202 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151066.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.151066.log.gz 2023-06-22T08:27:42.203 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150363.log 2023-06-22T08:27:42.205 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150363.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.150363.log.gz 2023-06-22T08:27:42.206 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147754.log 2023-06-22T08:27:42.209 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147754.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.147754.log.gz 2023-06-22T08:27:42.209 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156412.log 2023-06-22T08:27:42.213 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151694.log 2023-06-22T08:27:42.213 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156412.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.156412.log.gz 2023-06-22T08:27:42.214 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151694.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.151694.log.gz 2023-06-22T08:27:42.214 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158721.log 2023-06-22T08:27:42.217 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158721.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.158721.log.gz 2023-06-22T08:27:42.218 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157886.log 2023-06-22T08:27:42.220 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157886.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.157886.log.gz 2023-06-22T08:27:42.221 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141728.log 2023-06-22T08:27:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.141728.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.141728.log.gz 2023-06-22T08:27:42.224 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156171.log 2023-06-22T08:27:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156171.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156171.log.gz 2023-06-22T08:27:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157221.log 2023-06-22T08:27:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157221.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.157221.log.gz 2023-06-22T08:27:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152349.log 2023-06-22T08:27:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152349.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152349.log.gz 2023-06-22T08:27:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154879.log 2023-06-22T08:27:42.237 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150501.log 2023-06-22T08:27:42.237 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153808.log 2023-06-22T08:27:42.238 INFO:teuthology.orchestra.run.smithi023.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.154879.log.gz 2023-06-22T08:27:42.240 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150501.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150501.log.gz 2023-06-22T08:27:42.241 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138454.log 2023-06-22T08:27:42.241 INFO:teuthology.orchestra.run.smithi023.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.153808.log.gz 2023-06-22T08:27:42.243 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150550.log 2023-06-22T08:27:42.244 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138454.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.138454.log.gz 2023-06-22T08:27:42.244 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150550.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.150550.log.gz 2023-06-22T08:27:42.245 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136190.log 2023-06-22T08:27:42.247 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136190.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.136190.log.gz 2023-06-22T08:27:42.248 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153225.log 2023-06-22T08:27:42.252 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136096.log 2023-06-22T08:27:42.252 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153225.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.153225.log.gz 2023-06-22T08:27:42.253 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136096.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.136096.log.gz 2023-06-22T08:27:42.253 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160488.log 2023-06-22T08:27:42.256 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160488.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.160488.log.gz 2023-06-22T08:27:42.256 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156312.log 2023-06-22T08:27:42.259 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156312.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.156312.log.gz 2023-06-22T08:27:42.260 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160746.log 2023-06-22T08:27:42.262 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160746.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.160746.log.gz 2023-06-22T08:27:42.263 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139879.log 2023-06-22T08:27:42.265 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139879.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.139879.log.gz 2023-06-22T08:27:42.266 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157553.log 2023-06-22T08:27:42.268 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157553.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.157553.log.gz 2023-06-22T08:27:42.269 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161704.log 2023-06-22T08:27:42.272 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161704.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.161704.log.gz 2023-06-22T08:27:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151112.log 2023-06-22T08:27:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151112.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.151112.log.gz 2023-06-22T08:27:42.276 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148707.log 2023-06-22T08:27:42.279 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148707.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.148707.log.gz 2023-06-22T08:27:42.279 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160930.log 2023-06-22T08:27:42.282 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160930.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.160930.log.gz 2023-06-22T08:27:42.283 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153363.log 2023-06-22T08:27:42.286 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149914.log 2023-06-22T08:27:42.286 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153363.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.153363.log.gz 2023-06-22T08:27:42.287 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149914.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149914.log.gz 2023-06-22T08:27:42.288 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-06-22T08:27:42.294 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139971.log 2023-06-22T08:27:42.297 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139971.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.139971.log.gz 2023-06-22T08:27:42.304 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161612.log 2023-06-22T08:27:42.308 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161612.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.161612.log.gz 2023-06-22T08:27:42.308 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150409.log 2023-06-22T08:27:42.311 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150409.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150409.log.gz 2023-06-22T08:27:42.312 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146418.log 2023-06-22T08:27:42.315 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146418.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.146418.log.gz 2023-06-22T08:27:42.315 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157129.log 2023-06-22T08:27:42.319 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157129.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.157129.log.gz 2023-06-22T08:27:42.319 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139925.log 2023-06-22T08:27:42.322 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139925.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.139925.log.gz 2023-06-22T08:27:42.323 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155360.log 2023-06-22T08:27:42.326 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155360.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.155360.log.gz 2023-06-22T08:27:42.327 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139565.log 2023-06-22T08:27:42.330 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139565.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.139565.log.gz 2023-06-22T08:27:42.331 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140109.log 2023-06-22T08:27:42.334 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140109.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.140109.log.gz 2023-06-22T08:27:42.334 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151158.log 2023-06-22T08:27:42.337 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151158.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.151158.log.gz 2023-06-22T08:27:42.338 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151326.log 2023-06-22T08:27:42.341 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151326.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151326.log.gz 2023-06-22T08:27:42.342 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144914.log 2023-06-22T08:27:42.345 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.144914.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.144914.log.gz 2023-06-22T08:27:42.345 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142280.log 2023-06-22T08:27:42.347 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152416.log 2023-06-22T08:27:42.349 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142280.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.142280.log.gz 2023-06-22T08:27:42.353 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152416.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.152416.log.gz 2023-06-22T08:27:42.353 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140387.log 2023-06-22T08:27:42.356 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140387.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.140387.log.gz 2023-06-22T08:27:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155223.log 2023-06-22T08:27:42.360 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155223.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.155223.log.gz 2023-06-22T08:27:42.360 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156571.log 2023-06-22T08:27:42.364 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156571.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.156571.log.gz 2023-06-22T08:27:42.364 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158268.log 2023-06-22T08:27:42.367 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158268.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.158268.log.gz 2023-06-22T08:27:42.368 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146531.log 2023-06-22T08:27:42.371 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146531.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146531.log.gz 2023-06-22T08:27:42.372 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147984.log 2023-06-22T08:27:42.375 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147984.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.147984.log.gz 2023-06-22T08:27:42.375 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160188.log 2023-06-22T08:27:42.379 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160188.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.160188.log.gz 2023-06-22T08:27:42.379 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149544.log 2023-06-22T08:27:42.382 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149544.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.149544.log.gz 2023-06-22T08:27:42.383 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138889.log 2023-06-22T08:27:42.386 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138889.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.138889.log.gz 2023-06-22T08:27:42.387 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138224.log 2023-06-22T08:27:42.390 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138224.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.138224.log.gz 2023-06-22T08:27:42.390 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153670.log 2023-06-22T08:27:42.394 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153670.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.153670.log.gz 2023-06-22T08:27:42.394 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142465.log 2023-06-22T08:27:42.397 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142465.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.142465.log.gz 2023-06-22T08:27:42.397 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154501.log 2023-06-22T08:27:42.401 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154501.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.154501.log.gz 2023-06-22T08:27:42.401 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155131.log 2023-06-22T08:27:42.405 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155131.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.155131.log.gz 2023-06-22T08:27:42.405 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153992.log 2023-06-22T08:27:42.408 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153992.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.153992.log.gz 2023-06-22T08:27:42.409 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149868.log 2023-06-22T08:27:42.412 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149868.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.149868.log.gz 2023-06-22T08:27:42.413 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160884.log 2023-06-22T08:27:42.416 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160884.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.160884.log.gz 2023-06-22T08:27:42.416 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153557.log 2023-06-22T08:27:42.419 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153557.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.153557.log.gz 2023-06-22T08:27:42.420 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158222.log 2023-06-22T08:27:42.423 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158222.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.158222.log.gz 2023-06-22T08:27:42.424 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135182.log 2023-06-22T08:27:42.427 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135182.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.135182.log.gz 2023-06-22T08:27:42.428 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135629.log 2023-06-22T08:27:42.430 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138887.log 2023-06-22T08:27:42.431 INFO:teuthology.orchestra.run.smithi023.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135629.log.gz 2023-06-22T08:27:42.435 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138887.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.138887.log.gz 2023-06-22T08:27:42.436 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147938.log 2023-06-22T08:27:42.447 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.147938.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.147938.log.gz 2023-06-22T08:27:42.448 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156266.log 2023-06-22T08:27:42.450 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152111.log 2023-06-22T08:27:42.450 INFO:teuthology.orchestra.run.smithi023.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.156266.log.gz 2023-06-22T08:27:42.453 INFO:teuthology.orchestra.run.smithi067.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.130528.log.gz 2023-06-22T08:27:42.454 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152111.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152111.log.gz 2023-06-22T08:27:42.454 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146160.log 2023-06-22T08:27:42.457 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146160.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146160.log.gz 2023-06-22T08:27:42.458 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140479.log 2023-06-22T08:27:42.461 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140479.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.140479.log.gz 2023-06-22T08:27:42.462 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146715.log 2023-06-22T08:27:42.465 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146715.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146715.log.gz 2023-06-22T08:27:42.466 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156504.log 2023-06-22T08:27:42.469 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156504.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.156504.log.gz 2023-06-22T08:27:42.469 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142603.log 2023-06-22T08:27:42.473 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142603.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.142603.log.gz 2023-06-22T08:27:42.473 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155980.log 2023-06-22T08:27:42.476 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155980.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.155980.log.gz 2023-06-22T08:27:42.477 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150881.log 2023-06-22T08:27:42.480 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150881.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.150881.log.gz 2023-06-22T08:27:42.480 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150928.log 2023-06-22T08:27:42.484 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150928.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150928.log.gz 2023-06-22T08:27:42.484 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136468.log 2023-06-22T08:27:42.488 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136468.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.136468.log.gz 2023-06-22T08:27:42.488 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136567.log 2023-06-22T08:27:42.492 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136567.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.136567.log.gz 2023-06-22T08:27:42.492 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158176.log 2023-06-22T08:27:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158176.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158176.log.gz 2023-06-22T08:27:42.496 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146857.log 2023-06-22T08:27:42.499 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146857.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146857.log.gz 2023-06-22T08:27:42.500 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155934.log 2023-06-22T08:27:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155934.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155934.log.gz 2023-06-22T08:27:42.503 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140736.log 2023-06-22T08:27:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140736.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.140736.log.gz 2023-06-22T08:27:42.507 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157334.log 2023-06-22T08:27:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157334.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.157334.log.gz 2023-06-22T08:27:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158314.log 2023-06-22T08:27:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158314.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158314.log.gz 2023-06-22T08:27:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129952.log 2023-06-22T08:27:42.519 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.129952.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129952.log.gz 2023-06-22T08:27:42.519 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161474.log 2023-06-22T08:27:42.522 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161474.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.161474.log.gz 2023-06-22T08:27:42.523 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135480.log 2023-06-22T08:27:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135480.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.135480.log.gz 2023-06-22T08:27:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148586.log 2023-06-22T08:27:42.530 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148586.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.148586.log.gz 2023-06-22T08:27:42.530 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155292.log 2023-06-22T08:27:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155292.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.155292.log.gz 2023-06-22T08:27:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149730.log 2023-06-22T08:27:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149730.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149730.log.gz 2023-06-22T08:27:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142419.log 2023-06-22T08:27:42.541 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142419.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.142419.log.gz 2023-06-22T08:27:42.541 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160375.log 2023-06-22T08:27:42.545 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160375.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160375.log.gz 2023-06-22T08:27:42.545 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140525.log 2023-06-22T08:27:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140525.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.140525.log.gz 2023-06-22T08:27:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148405.log 2023-06-22T08:27:42.555 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148405.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.148405.log.gz 2023-06-22T08:27:42.555 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162429.log 2023-06-22T08:27:42.561 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.162429.log: 95.7% -- replaced with /var/log/ceph/ceph-client.admin.162429.log.gz 2023-06-22T08:27:42.561 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151556.log 2023-06-22T08:27:42.564 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151556.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151556.log.gz 2023-06-22T08:27:42.565 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140341.log 2023-06-22T08:27:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140341.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.140341.log.gz 2023-06-22T08:27:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148634.log 2023-06-22T08:27:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148634.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.148634.log.gz 2023-06-22T08:27:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142326.log 2023-06-22T08:27:42.576 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142326.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.142326.log.gz 2023-06-22T08:27:42.576 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152249.log 2023-06-22T08:27:42.579 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140948.log 2023-06-22T08:27:42.580 INFO:teuthology.orchestra.run.smithi023.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.152249.log.gz 2023-06-22T08:27:42.583 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140948.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.140948.log.gz 2023-06-22T08:27:42.584 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145006.log 2023-06-22T08:27:42.587 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145006.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145006.log.gz 2023-06-22T08:27:42.588 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148291.log 2023-06-22T08:27:42.591 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148291.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.148291.log.gz 2023-06-22T08:27:42.591 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158608.log 2023-06-22T08:27:42.595 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158608.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.158608.log.gz 2023-06-22T08:27:42.595 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145929.log 2023-06-22T08:27:42.599 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145929.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.145929.log.gz 2023-06-22T08:27:42.599 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146623.log 2023-06-22T08:27:42.602 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146623.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.146623.log.gz 2023-06-22T08:27:42.603 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155018.log 2023-06-22T08:27:42.606 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155018.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.155018.log.gz 2023-06-22T08:27:42.606 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153038.log 2023-06-22T08:27:42.610 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.153038.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.153038.log.gz 2023-06-22T08:27:42.610 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151786.log 2023-06-22T08:27:42.614 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151786.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.151786.log.gz 2023-06-22T08:27:42.614 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137905.log 2023-06-22T08:27:42.617 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137905.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.137905.log.gz 2023-06-22T08:27:42.618 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137610.log 2023-06-22T08:27:42.621 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137610.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.137610.log.gz 2023-06-22T08:27:42.621 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142142.log 2023-06-22T08:27:42.625 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142142.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.142142.log.gz 2023-06-22T08:27:42.625 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145106.log 2023-06-22T08:27:42.629 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145106.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.145106.log.gz 2023-06-22T08:27:42.629 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151740.log 2023-06-22T08:27:42.633 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151740.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151740.log.gz 2023-06-22T08:27:42.633 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159859.log 2023-06-22T08:27:42.637 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.159859.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.159859.log.gz 2023-06-22T08:27:42.637 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161223.log 2023-06-22T08:27:42.640 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161223.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.161223.log.gz 2023-06-22T08:27:42.641 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138362.log 2023-06-22T08:27:42.644 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138362.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.138362.log.gz 2023-06-22T08:27:42.644 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151602.log 2023-06-22T08:27:42.648 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151602.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.151602.log.gz 2023-06-22T08:27:42.648 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138500.log 2023-06-22T08:27:42.651 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138500.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138500.log.gz 2023-06-22T08:27:42.652 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156617.log 2023-06-22T08:27:42.655 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156617.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.156617.log.gz 2023-06-22T08:27:42.656 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161520.log 2023-06-22T08:27:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161520.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.161520.log.gz 2023-06-22T08:27:42.659 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144868.log 2023-06-22T08:27:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.144868.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.144868.log.gz 2023-06-22T08:27:42.663 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154833.log 2023-06-22T08:27:42.667 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154833.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.154833.log.gz 2023-06-22T08:27:42.667 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137767.log 2023-06-22T08:27:42.671 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137767.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.137767.log.gz 2023-06-22T08:27:42.671 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160283.log 2023-06-22T08:27:42.674 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160283.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.160283.log.gz 2023-06-22T08:27:42.675 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135770.log 2023-06-22T08:27:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135770.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.135770.log.gz 2023-06-22T08:27:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152830.log 2023-06-22T08:27:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152830.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.152830.log.gz 2023-06-22T08:27:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154547.log 2023-06-22T08:27:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154547.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154547.log.gz 2023-06-22T08:27:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142050.log 2023-06-22T08:27:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142050.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.142050.log.gz 2023-06-22T08:27:42.690 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135678.log 2023-06-22T08:27:42.693 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135678.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.135678.log.gz 2023-06-22T08:27:42.693 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138178.log 2023-06-22T08:27:42.697 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138178.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.138178.log.gz 2023-06-22T08:27:42.697 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145311.log 2023-06-22T08:27:42.701 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149224.log 2023-06-22T08:27:42.701 INFO:teuthology.orchestra.run.smithi023.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145311.log.gz 2023-06-22T08:27:42.705 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149224.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.149224.log.gz 2023-06-22T08:27:42.705 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156458.log 2023-06-22T08:27:42.708 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156458.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.156458.log.gz 2023-06-22T08:27:42.708 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154925.log 2023-06-22T08:27:42.712 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.154925.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154925.log.gz 2023-06-22T08:27:42.712 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136284.log 2023-06-22T08:27:42.716 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136284.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.136284.log.gz 2023-06-22T08:27:42.716 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160089.log 2023-06-22T08:27:42.720 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160089.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.160089.log.gz 2023-06-22T08:27:42.720 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146261.log 2023-06-22T08:27:42.723 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146261.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.146261.log.gz 2023-06-22T08:27:42.724 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148945.log 2023-06-22T08:27:42.727 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148945.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148945.log.gz 2023-06-22T08:27:42.728 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152784.log 2023-06-22T08:27:42.731 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152784.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152784.log.gz 2023-06-22T08:27:42.731 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155729.log 2023-06-22T08:27:42.735 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.155729.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.155729.log.gz 2023-06-22T08:27:42.735 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161999.log 2023-06-22T08:27:42.739 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161999.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.161999.log.gz 2023-06-22T08:27:42.739 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158513.log 2023-06-22T08:27:42.743 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158513.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.158513.log.gz 2023-06-22T08:27:42.743 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142695.log 2023-06-22T08:27:42.746 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142695.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.142695.log.gz 2023-06-22T08:27:42.747 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150271.log 2023-06-22T08:27:42.750 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150271.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.150271.log.gz 2023-06-22T08:27:42.750 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-06-22T08:27:42.755 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145587.log 2023-06-22T08:27:42.771 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145587.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.145587.log.gz 2023-06-22T08:27:42.771 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148032.log 2023-06-22T08:27:42.784 INFO:teuthology.orchestra.run.smithi023.stderr:gzip /var/log/ceph/ceph-client.admin.148032.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.136330.log 2023-06-22T08:27:42.784 INFO:teuthology.orchestra.run.smithi023.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.148032.log.gz 2023-06-22T08:27:42.797 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160992.log 2023-06-22T08:27:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136330.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.136330.log.gz 2023-06-22T08:27:42.819 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160580.log 2023-06-22T08:27:42.820 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160992.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.160992.log.gz 2023-06-22T08:27:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142096.log 2023-06-22T08:27:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160580.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.160580.log.gz 2023-06-22T08:27:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.142096.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.142096.log.gz 2023-06-22T08:27:42.854 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159905.log 2023-06-22T08:27:42.865 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149291.log 2023-06-22T08:27:42.866 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.159905.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.159905.log.gz 2023-06-22T08:27:42.887 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149291.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.149291.log.gz 2023-06-22T08:27:42.887 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161315.log 2023-06-22T08:27:42.908 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146021.log 2023-06-22T08:27:42.909 INFO:teuthology.orchestra.run.smithi023.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.161315.log.gz 2023-06-22T08:27:42.933 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137472.log 2023-06-22T08:27:42.934 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.146021.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.146021.log.gz 2023-06-22T08:27:42.936 INFO:teuthology.orchestra.run.smithi067.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-06-22T08:27:42.950 INFO:teuthology.orchestra.run.smithi023.stderr:gzip /var/log/ceph/ceph-client.admin.137472.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.159951.log 2023-06-22T08:27:42.951 INFO:teuthology.orchestra.run.smithi023.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.137472.log.gz 2023-06-22T08:27:42.961 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.159951.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.159951.log.gz 2023-06-22T08:27:42.972 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157794.log 2023-06-22T08:27:42.983 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160329.log 2023-06-22T08:27:42.984 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157794.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.157794.log.gz 2023-06-22T08:27:42.994 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152922.log 2023-06-22T08:27:42.995 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.160329.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.160329.log.gz 2023-06-22T08:27:43.006 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138884.log 2023-06-22T08:27:43.017 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152922.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.152922.log.gz 2023-06-22T08:27:43.028 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.138884.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138884.log.gz 2023-06-22T08:27:43.029 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148494.log 2023-06-22T08:27:43.039 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144960.log 2023-06-22T08:27:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.148494.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148494.log.gz 2023-06-22T08:27:43.060 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.144960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151464.log 2023-06-22T08:27:43.060 INFO:teuthology.orchestra.run.smithi023.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.144960.log.gz 2023-06-22T08:27:43.073 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.151464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137813.log 2023-06-22T08:27:43.073 INFO:teuthology.orchestra.run.smithi023.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.151464.log.gz 2023-06-22T08:27:43.086 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.137813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150613.log 2023-06-22T08:27:43.086 INFO:teuthology.orchestra.run.smithi023.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137813.log.gz 2023-06-22T08:27:43.099 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135259.log 2023-06-22T08:27:43.102 INFO:teuthology.orchestra.run.smithi067.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-06-22T08:27:43.109 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.150613.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150613.log.gz 2023-06-22T08:27:43.121 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135259.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135259.log.gz 2023-06-22T08:27:43.121 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145052.log 2023-06-22T08:27:43.141 INFO:teuthology.orchestra.run.smithi023.stderr:gzip /var/log/ceph/ceph-client.admin.145052.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.152462.log 2023-06-22T08:27:43.151 INFO:teuthology.orchestra.run.smithi023.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.145052.log.gz 2023-06-22T08:27:43.163 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.152462.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.152462.log.gz 2023-06-22T08:27:43.163 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-06-22T08:27:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158859.log 2023-06-22T08:27:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.158859.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.158859.log.gz 2023-06-22T08:27:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140063.log 2023-06-22T08:27:43.210 INFO:teuthology.orchestra.run.smithi023.stderr: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-06-22T08:27:43.224 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.140063.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.140063.log.gz 2023-06-22T08:27:43.225 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135908.log 2023-06-22T08:27:43.242 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162229.log 2023-06-22T08:27:43.252 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.135908.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.135908.log.gz 2023-06-22T08:27:43.264 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.162229.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.162229.log.gz 2023-06-22T08:27:43.264 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161428.log 2023-06-22T08:27:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145633.log 2023-06-22T08:27:43.287 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.161428.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.161428.log.gz 2023-06-22T08:27:43.299 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145633.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.145633.log.gz 2023-06-22T08:27:43.299 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156358.log 2023-06-22T08:27:43.320 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162183.log 2023-06-22T08:27:43.330 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.156358.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.156358.log.gz 2023-06-22T08:27:43.342 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.162183.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.162183.log.gz 2023-06-22T08:27:43.343 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157999.log 2023-06-22T08:27:43.364 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145403.log 2023-06-22T08:27:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.157999.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.157999.log.gz 2023-06-22T08:27:43.386 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.145403.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.145403.log.gz 2023-06-22T08:27:43.386 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162045.log 2023-06-22T08:27:43.399 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139833.log 2023-06-22T08:27:43.409 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.162045.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.162045.log.gz 2023-06-22T08:27:43.422 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136422.log 2023-06-22T08:27:43.446 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.139833.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.139833.log.gz 2023-06-22T08:27:43.446 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.136422.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.136422.log.gz 2023-06-22T08:27:43.447 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149610.log 2023-06-22T08:27:43.474 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/ceph-client.admin.149610.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149610.log.gz 2023-06-22T08:27:44.343 INFO:teuthology.orchestra.run.smithi023.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-06-22T08:28:13.425 INFO:teuthology.orchestra.run.smithi023.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-06-22T08:28:23.797 INFO:teuthology.orchestra.run.smithi067.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-06-22T08:28:24.183 INFO:teuthology.orchestra.run.smithi023.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-06-22T08:28:26.238 INFO:teuthology.orchestra.run.smithi023.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-06-22T08:28:28.013 INFO:teuthology.orchestra.run.smithi023.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-06-22T08:28:29.224 INFO:teuthology.orchestra.run.smithi067.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-06-22T08:28:30.011 INFO:teuthology.orchestra.run.smithi023.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-06-22T08:28:32.645 INFO:teuthology.orchestra.run.smithi067.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-06-22T08:28:33.993 INFO:teuthology.orchestra.run.smithi023.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-06-22T08:28:35.884 INFO:teuthology.orchestra.run.smithi067.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-06-22T08:28:36.341 INFO:teuthology.orchestra.run.smithi023.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-06-22T08:28:36.345 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-06-22T08:28:36.345 INFO:teuthology.orchestra.run.smithi023.stderr:real 0m54.675s 2023-06-22T08:28:36.345 INFO:teuthology.orchestra.run.smithi023.stderr:user 4m58.405s 2023-06-22T08:28:36.345 INFO:teuthology.orchestra.run.smithi023.stderr:sys 0m22.579s 2023-06-22T08:28:43.913 INFO:teuthology.orchestra.run.smithi067.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-06-22T08:28:45.144 INFO:teuthology.orchestra.run.smithi067.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-06-22T08:29:09.947 INFO:teuthology.orchestra.run.smithi067.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-06-22T08:29:09.979 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-06-22T08:29:09.980 INFO:teuthology.orchestra.run.smithi067.stderr:real 1m28.294s 2023-06-22T08:29:09.980 INFO:teuthology.orchestra.run.smithi067.stderr:user 5m25.298s 2023-06-22T08:29:09.980 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m10.079s 2023-06-22T08:29:09.981 INFO:tasks.ceph:Archiving logs... 2023-06-22T08:29:09.981 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/remote/smithi023/log 2023-06-22T08:29:09.983 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-22T08:29:38.199 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/remote/smithi067/log 2023-06-22T08:29:38.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-22T08:30:12.849 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-06-22T08:30:12.860 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-06-22T08:30:12.875 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-06-22T08:30:12.879 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-06-22T08:30:12.880 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-22T08:30:12.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-22T08:30:13.029 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-06-22T08:30:13.041 INFO:teuthology.task.clock:Checking final clock skew... 2023-06-22T08:30:13.042 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T08:30:13.047 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T08:30:13.066 INFO:teuthology.orchestra.run.smithi067.stderr:bash: ntpq: command not found 2023-06-22T08:30:13.085 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2023-06-22T08:30:13.088 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T08:30:13.088 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2023-06-22T08:30:13.088 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +1526us[+1526us] +/- 81ms 2023-06-22T08:30:13.088 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -1206us[-1202us] +/- 65ms 2023-06-22T08:30:13.089 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 +120us[ +120us] +/- 87ms 2023-06-22T08:30:13.089 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T08:30:13.103 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T08:30:13.103 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2023-06-22T08:30:13.103 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +1507us[+1507us] +/- 81ms 2023-06-22T08:30:13.104 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -1237us[-1210us] +/- 65ms 2023-06-22T08:30:13.104 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +97us[ +124us] +/- 87ms 2023-06-22T08:30:13.104 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T08:30:13.105 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-06-22T08:30:13.114 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-06-22T08:30:13.116 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-06-22T08:30:13.129 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-22T08:30:13.334 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-22T08:30:13.448 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T08:30:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T08:30:13.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T08:30:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T08:30:13.560 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-06-22T08:30:13.570 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-06-22T08:30:13.584 INFO:teuthology.task.internal:Duration was 3011.263143 seconds 2023-06-22T08:30:13.585 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-06-22T08:30:13.600 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-06-22T08:30:13.600 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-22T08:30:13.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-22T08:30:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T08:30:13.736 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T08:30:14.069 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-06-22T08:30:14.070 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2023-06-22T08:30:14.070 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-22T08:30:14.128 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2023-06-22T08:30:14.129 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-22T08:30:14.155 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-06-22T08:30:14.156 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T08:30:14.160 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T08:30:14.251 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-06-22T08:30:14.252 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-22T08:30:14.256 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-22T08:30:14.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-06-22T08:30:14.418 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-06-22T08:30:14.419 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-22T08:30:14.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-22T08:30:14.539 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-06-22T08:30:14.553 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-22T08:30:14.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-22T08:30:14.594 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2023-06-22T08:30:14.639 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2023-06-22T08:30:14.676 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-22T08:30:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T08:30:14.718 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-22T08:30:14.735 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi067.front.sepia.ceph.com, flagging run as failed 2023-06-22T08:30:14.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-06-22T08:30:14.751 INFO:teuthology.task.internal:Transferring archived files... 2023-06-22T08:30:14.757 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/remote/smithi023 2023-06-22T08:30:14.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-22T08:30:14.835 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/remote/smithi067 2023-06-22T08:30:14.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-22T08:30:44.267 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-06-22T08:30:44.362 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116/remote/smithi067/coredump/1687422013.137549.core.gz: gzip compressed data, was "1687422013.137549.core", last modified: Thu Jun 22 08:30:14 2023, max compression 2023-06-22T08:30:44.364 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-06-22T08:30:46.195 INFO:teuthology.task.internal: core looks like: /tmp/tmpsz24t9e5: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i b', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2023-06-22T08:30:46.487 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i b 2023-06-22T08:30:46.489 DEBUG:teuthology.orchestra.run.smithi067:> which 'ceph-mds -f --cluster ceph -i b' 2023-06-22T08:30:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/which: invalid option -- 'f' 2023-06-22T08:30:46.512 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/which: unrecognized option '--cluster' 2023-06-22T08:30:46.512 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-mds 2023-06-22T08:30:46.513 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph 2023-06-22T08:30:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T08:30:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:/usr/bin/which: no b in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2023-06-22T08:30:46.515 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 1: "which 'ceph-mds -f --cluster ceph -i b'" 2023-06-22T08:30:46.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-06-22T08:30:46.544 INFO:teuthology.task.internal:Not uploading archives. 2023-06-22T08:30:46.545 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-06-22T08:30:46.555 INFO:teuthology.task.internal:Tidying up after the test... 2023-06-22T08:30:46.555 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-22T08:30:46.566 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-22T08:30:46.589 INFO:teuthology.orchestra.run.smithi067.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 22 08:30 /home/ubuntu/cephtest 2023-06-22T08:30:46.589 INFO:teuthology.orchestra.run.smithi067.stdout: 262170 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Jun 22 08:30 /home/ubuntu/cephtest/archive 2023-06-22T08:30:46.589 INFO:teuthology.orchestra.run.smithi067.stdout: 393222 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Jun 22 08:30 /home/ubuntu/cephtest/archive/audit 2023-06-22T08:30:46.590 INFO:teuthology.orchestra.run.smithi067.stdout: 393241 236 -rw------- 1 ubuntu ubuntu 240049 Jun 22 08:30 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-06-22T08:30:46.590 INFO:teuthology.orchestra.run.smithi067.stdout: 262172 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 22 08:20 /home/ubuntu/cephtest/archive/coredump 2023-06-22T08:30:46.590 INFO:teuthology.orchestra.run.smithi067.stdout: 262280 161164 -rw------- 1 root root 417800192 Jun 22 08:20 /home/ubuntu/cephtest/archive/coredump/1687422013.137549.core 2023-06-22T08:30:46.590 INFO:teuthology.orchestra.run.smithi067.stdout: 262173 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 22 08:30 /home/ubuntu/cephtest/archive/syslog 2023-06-22T08:30:46.590 INFO:teuthology.orchestra.run.smithi067.stdout: 262278 4 -rw-rw-rw- 1 ubuntu ubuntu 634 Jun 22 08:27 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-06-22T08:30:46.591 INFO:teuthology.orchestra.run.smithi067.stdout: 262181 108 -rw-rw-rw- 1 ubuntu ubuntu 106847 Jun 22 08:30 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-06-22T08:30:46.591 INFO:teuthology.orchestra.run.smithi067.stdout: 262183 104 -rw-rw-r-- 1 ubuntu ubuntu 104415 Jun 22 08:30 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-22T08:30:46.592 INFO:teuthology.orchestra.run.smithi067.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-06-22T08:30:46.610 INFO:teuthology.orchestra.run.smithi023.stdout: 524293 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 22 08:30 /home/ubuntu/cephtest 2023-06-22T08:30:46.610 INFO:teuthology.orchestra.run.smithi023.stdout: 524294 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jun 22 08:30 /home/ubuntu/cephtest/archive 2023-06-22T08:30:46.610 INFO:teuthology.orchestra.run.smithi023.stdout: 655763 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Jun 22 08:30 /home/ubuntu/cephtest/archive/audit 2023-06-22T08:30:46.611 INFO:teuthology.orchestra.run.smithi023.stdout: 655765 284 -rw------- 1 ubuntu ubuntu 290058 Jun 22 08:30 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-06-22T08:30:46.611 INFO:teuthology.orchestra.run.smithi023.stdout: 524296 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 22 08:30 /home/ubuntu/cephtest/archive/syslog 2023-06-22T08:30:46.611 INFO:teuthology.orchestra.run.smithi023.stdout: 524496 8 -rw-rw-rw- 1 ubuntu ubuntu 5224 Jun 22 08:27 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-06-22T08:30:46.611 INFO:teuthology.orchestra.run.smithi023.stdout: 524301 140 -rw-rw-rw- 1 ubuntu ubuntu 139615 Jun 22 08:30 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-06-22T08:30:46.611 INFO:teuthology.orchestra.run.smithi023.stdout: 524302 128 -rw-rw-r-- 1 ubuntu ubuntu 128576 Jun 22 08:30 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-22T08:30:46.612 INFO:teuthology.orchestra.run.smithi023.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-06-22T08:30:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T08:30:46.613 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2744, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2576, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 2342, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8627af3e0adcb765a3c249fcc209cba9f4873e1b/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi023 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-06-22T08:30:46.679 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-06-22T08:30:46.690 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-06-22T08:30:46.701 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-06-22T08:30:46.702 INFO:teuthology.nuke:Checking targets against current locks 2023-06-22T08:30:46.737 DEBUG:teuthology.nuke:shortname: smithi023 2023-06-22T08:30:46.738 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T08:30:46.766 DEBUG:teuthology.nuke:shortname: smithi067 2023-06-22T08:30:46.766 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T08:30:46.769 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 07:29:00.887135', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-06-22T08:30:46.792 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-06-21_23:38:17-fs-wip-rishabh-improvements-authmon-testing-default-smithi/7311116', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 07:29:00.888702', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-06-22T08:30:46.833 INFO:teuthology.orchestra.console:Power off smithi023 2023-06-22T08:30:46.835 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T08:30:46.860 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T08:30:46.878 INFO:teuthology.orchestra.console:Power off smithi067 2023-06-22T08:30:46.879 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T08:30:46.903 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T08:30:50.891 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T08:30:50.937 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T08:30:55.012 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T08:30:55.050 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T08:30:55.226 INFO:teuthology.orchestra.console:Power off for smithi023 completed 2023-06-22T08:30:59.166 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T08:30:59.379 INFO:teuthology.orchestra.console:Power off for smithi067 completed 2023-06-22T08:30:59.486 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1c622d92b6f18ce44a7c33802686ff61c3c273ab description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 3011.2631430625916 failure_reason: 'Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi023 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8627af3e0adcb765a3c249fcc209cba9f4873e1b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6e13860d1eca4ac58c3c9dc45a4e8024 status: fail success: false 2023-06-22T08:30:59.487 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T08:30:59.568 INFO:teuthology.run:FAIL